[[s-mysql]] ==== Highly available MySQL MySQL is the default database server used by many OpenStack services. Making the MySQL service highly available involves * configuring a DRBD device for use by MySQL, * configuring MySQL to use a data directory residing on that DRBD device, * selecting and assigning a virtual IP address (VIP) that can freely float between cluster nodes, * configuring MySQL to listen on that IP address, * managing all resources, including the MySQL daemon itself, with the Pacemaker cluster manager. NOTE: http://codership.com/products/mysql_galera[MySQL/Galera] is an alternative method of configuring MySQL for high availability. It is likely to become the preferred method of achieving MySQL high availability once it has sufficiently matured. At the time of writing, however, the Pacemaker/DRBD based approach remains the recommended one for OpenStack environments. ===== Configuring DRBD The Pacemaker based MySQL server requires a DRBD resource from which it mounts the +/var/lib/mysql+ directory. In this example, the DRBD resource is simply named +mysql+: .+mysql+ DRBD resource configuration (+/etc/drbd.d/mysql.res+) ---- include::includes/mysql.res[] ---- This resource uses an underlying local disk (in DRBD terminology, a _backing device_) named +/dev/data/mysql+ on both cluster nodes, +node1+ and +node2+. Normally, this would be an LVM Logical Volume specifically set aside for this purpose. The DRBD +meta-disk+ is +internal+, meaning DRBD-specific metadata is being stored at the end of the +disk+ device itself. The device is configured to communicate between IPv4 addresses 10.0.42.100 and 10.0.42.254, using TCP port 7700. Once enabled, it will map to a local DRBD block device with the device minor number 0, that is, +/dev/drbd0+. Enabling a DRBD resource is explained in detail in http://www.drbd.org/users-guide-8.3/s-first-time-up.html[the DRBD User's Guide]. In brief, the proper sequence of commands is this: ---- drbdadm create-md mysql <1> drbdadm up mysql <2> drbdadm -- --force primary mysql <3> ---- <1> Initializes DRBD metadata and writes the initial set of metadata to +/dev/data/mysql+. Must be completed on both nodes. <2> Creates the +/dev/drbd0+ device node, _attaches_ the DRBD device to its backing store, and _connects_ the DRBD node to its peer. Must be completed on both nodes. <3> Kicks off the initial device synchronization, and puts the device into the +primary+ (readable and writable) role. See http://www.drbd.org/users-guide-8.3/ch-admin.html#s-roles[Resource roles] (from the DRBD User's Guide) for a more detailed description of the primary and secondary roles in DRBD. Must be completed _on one node only,_ namely the one where you are about to continue with creating your filesystem. ===== Creating a file system Once the DRBD resource is running and in the primary role (and potentially still in the process of running the initial device synchronization), you may proceed with creating the filesystem for MySQL data. XFS is the generally recommended filesystem: ---- mkfs -t xfs /dev/drbd1 ---- You may also use the alternate device path for the DRBD device, which may be easier to remember as it includes the self-explanatory resource name: ---- mkfs -t xfs /dev/drbd/by-res/mysql ---- Once completed, you may safely return the device to the secondary role. Any ongoing device synchronization will continue in the background: ---- drbdadm secondary mysql ---- ===== Preparing MySQL for Pacemaker high availability In order for Pacemaker monitoring to function properly, you must ensure that MySQL's database files reside on the DRBD device. If you already have an existing MySQL database, the simplest approach is to just move the contents of the existing +/var/lib/mysql+ directory into the newly created filesystem on the DRBD device. WARNING: You must complete the next step while the MySQL database server is shut down. ---- node1:# mount /dev/drbd/by-res/mysql /mnt node1:# mv /var/lib/mysql/* /mnt node1:# umount /mnt ---- For a new MySQL installation with no existing data, you may also run the +mysql_install_db+ command: ---- node1:# mount /dev/drbd/by-res/mysql /mnt node1:# mysql_install_db --datadir=/mnt node1:# umount /mnt ---- Regardless of the approach, the steps outlined here must be completed on only one cluster node. ===== Adding MySQL resources to Pacemaker You may now proceed with adding the Pacemaker configuration for MySQL resources. Connect to the Pacemaker cluster with +crm configure+, and add the following cluster resources: ---- include::includes/pacemaker-mysql.crm[] ---- This configuration creates * +p_ip_mysql+, a virtual IP address for use by MySQL (192.168.42.101), * +p_fs_mysql+, a Pacemaker managed filesystem mounted to +/var/lib/mysql+ on whatever node currently runs the MySQL service, * +ms_drbd_mysql+, the _master/slave set_ managing the +mysql+ DRBD resource, * a service +group+ and +order+ and +colocation+ constraints to ensure resources are started on the correct nodes, and in the correct sequence. +crm configure+ supports batch input, so you may copy and paste the above into your live pacemaker configuration, and then make changes as required. For example, you may enter +edit p_ip_mysql+ from the +crm configure+ menu and edit the resource to match your preferred virtual IP address. Once completed, commit your configuration changes by entering +commit+ from the +crm configure+ menu. Pacemaker will then start the MySQL service, and its dependent resources, on one of your nodes. ===== Configuring OpenStack services for highly available MySQL Your OpenStack services must now point their MySQL configuration to the highly available, virtual cluster IP address -- rather than a MySQL server's physical IP address as you normally would. For OpenStack Image, for example, if your MySQL service IP address is 192.168.42.101 as in the configuration explained here, you would use the following line in your OpenStack Image registry configuration file (+glance-registry.conf+): ---- sql_connection = mysql://glancedbadmin:@192.168.42.101/glance ---- No other changes are necessary to your OpenStack configuration. If the node currently hosting your database experiences a problem necessitating service failover, your OpenStack services may experience a brief MySQL interruption, as they would in the event of a network hiccup, and then continue to run normally.