Setting up IP failover with Heartbeat and Pacemaker on Ubuntu Lucid

By:
Jun 16, 2010

One of Zivtech's clients recently asked us to renovate their server setup, with a focus on improving its availability.

This post will cover the highly available load balancer setup that currently serves 7 sites, with another 4 in the pipeline. Basically, we want to run the sites through two Linux-based software load balancer boxes, and be able to keep the sites up even if one of them fails.

The ingredients:

  • Two Ubuntu Lucid boxes, standard stripped down server install. We'll call them Bart and Lisa, because that's how server naming rolls at Zivtech
  • Heartbeat
  • Pacemaker
  • A service IP per site (an IP address that is not the primary IP of the box it lives on, so it can be moved)

Heartbeat and Pacemaker are the real stars here. Pacemaker is the "brains" of the two, providing the high level control of our little cluster, while Heartbeat does the dirty work - it provides the low level inter-node communication and command framework.

Getting started with Heartbeat and Pacemaker

All of the packages we need are available in Ubuntu repositories, so we can just apt-get them on both boxes.

root@bart:~# apt-get install heartbeat pacemaker

The first place to go for configuration information is the The Linux-HA User's Guide, followed by the excellent Pacemaker configuration website.

The first step is to create an initial Heartbeat configuration and auth key on one of the boxes and copy it to the other, and restart the service.
root@bart:~# cat /etc/ha.d/ha.cf
autojoin none
bcast eth0
warntime 3
deadtime 6
initdead 60
keepalive 1
node bart.example.com
node lisa.example.com
crm respawn
root@bart:~# scp /etc/ha.d/ha.cf lisa:/etc/ha.d/
root@bart:~# ( echo -ne "auth 1\n1 sha1 "; \
dd if=/dev/urandom bs=512 count=1 | openssl md5 ) \
> /etc/ha.d/authkeys
root@bart:~# chmod 0600 /etc/ha.d/authkeys
root@bart:~# scp /etc/ha.d/authkeys lisa:/etc/ha.d/
root@bart:~# ssh lisa chmod 0600 /etc/ha.d/authkeys
root@bart:~# /etc/init.d/heartbeat restart
....
root@bart:~# ssh lisa /etc/init.d/heartbeat restart

If all went well, both nodes of the cluster should be up and know about each other. To check, use one of the insanely helpful command line tools that come with the packages, crm_mon.

root@bart:~# crm_mon -1 | grep Online
Online: [ lisa.example.com bart.example.com ]

Ok, now we move on to setting up the IP addresses we would like Hearbeat and Pacemaker to keep available. There are a few ways to change the configuration of your cluster, but the easiest one is to use the crm tool (go here for the excellent documentation). Note, this is the cluster configuration, not the configuration for a single node - you only need to update this on one node, and the changes will propagate automatically.

To Pacemaker, an IP address is a resource (and it can handle many different types of resources, but that's beyond the scope of this post). Assuming we have two sites, with IP addresses of 192.168.1.111 and 192.168.1.222 respectively, we'd tell Pacemaker about them like this:

root@bart:~# crm configure
crm(live)configure# primitive site_one_ip IPaddr params ip=192.168.1.111 cidr_netmask="255.255.255.0" nic="eth0"
crm(live)configure# primitive site_two_ip IPaddr params ip=192.168.1.222 cidr_netmask="255.255.255.0" nic="eth0"
crm(live)configure# commit
crm(live)configure# exit

To check that your configuration is what you think it is as you go, you can use crm configure show. Right now, your configuration should look similar to this:

root@lisa:~# crm configure show
node $id="9f5e6cd6-2b75-4445-8d75-43c1a34fe431" bart.example.com
node $id="fb502469-691b-4021-b504-7da7a188bc63" lisa.example.com
primitive site_one_ip ocf:heartbeat:IPaddr \
params ip="192.168.1.111" cidr_netmask="255.255.255.0" nic="eth0"
primitive site_two_ip ocf:heartbeat:IPaddr \
params ip="192.168.1.222" cidr_netmask="255.255.255.0" nic="eth0"
...

The next thing we want to do is tell Pacemaker on which node we'd prefer the IPs to live when both nodes are up. We do this by setting a "location" for each IP resource. In the following configuration, we tell Pacemaker to keep one IP on each node when they are both up, which is a fairly typical setup for a pair of load balancers:

root@bart:~# crm configure
crm(live)configure# location site_one_ip_pref site_one_ip 100: bart.example.com
crm(live)configure# location site_two_ip_pref site_two_ip 100: lisa.example.com
crm(live)configure# commit
crm(live)configure# exit

Checking the configuration, we should see two lines like this added to our output of crm configure show:

location site_one_ip_pref site_one_ip 100: bart.example.com
location site_two_ip_pref site_two_ip 100: lisa.example.com

Nearly there! Now that Pacemaker knows about our IP address resources, and where they should live, we can setup monitoring, which is the real meat of this exercise:

root@bart:~# crm configure
crm(live)configure# monitor site_one_ip 40s:20s
crm(live)configure# monitor site_two_ip 40s:20s
crm(live)configure# commit
crm(live)configure# exit

Done! Now, to test your setup, turn off/pull out the network cable from one of your nodes, and watch the other one pick up the IPs. Connect the the node again, and Pacemaker will honour your preferences, and send the failed-over IP address back to the original node.

Now that we have highly available load-balancing layer, the next post in this series will show you how to setup Apache to balance requests between backend servers running PHP and Drupal.

Related posts

Jun 6, 2014 | Posted by: David Hamme
We are excited to annouce that Zivtech CEO Alex Urevick-Ackelsberg will be discussing the future of Drupal in business on a panel with other local Drupal shops including Context, Rock River Star, and Tabula Creative.    Businesses have long been searching for a platform that empowers...
Mar 24, 2014 | Posted by: David Hamme
Are you an aspiring web developer, fresh out of college and eager to dive into Philly’s exploding tech startup scene? Are you a growing startup or established company in need of talented developers to help take your business to the next level?   Zivtech was thrilled last week when...
Mar 10, 2014 | Posted by: Jason Moore
Hi, my name is Jason Moore and I am a Developer here at Zivtech. I recently started working for Zivtech full time, helping their talented team continue to build and maintain awesome Drupal websites and applications for their clients. For the three years prior to joining Zivtech, I built a name for...
Nov 5, 2013 | Posted by: David Hamme
A salute to the unsung modules that make our jobs a little easier. We at Zivtech have been working with Drupal for a long time, some of us for over 10 years, and in that time we’ve gotten to use (and sometimes build) a lot of really cool modules to develop stellar sites for our clients. We...
Sep 26, 2013 | Posted by: Laurence Liss
As a Senior Developer here at Zivtech, part of my job is to develop with an eye towards security. If you follow development and security news, you know that achieving security is pretty hard in complex systems. But, while correctly implementing security across a site is challenging, some aspects...
Sep 11, 2013 | Posted by: Alban Bailly
Drupal Page: https://drupal.org/project/bear_skin GitHub: https://github.com/alioso/bear-skin Live Demo: http://bearst.dev.zivtech.com/   My Name is Alban and I'm a front-end dev here at Zivtech. We use an installation profile called Bear and I have been working on a theme to go along...
Apr 16, 2013 | Posted by: Jody Hamilton
The quality assurance (QA) phase of a web development project is the last phase before launch. While the development team has a lot of experience with QA phases, oftentimes the client team is new to the process, which can lead to stress. Let's prevent this with a little Q&A on QA. What is...
Apr 15, 2013 | Posted by: David Marvin
Nominees for the 17th annual Webby Awards have been announced! And Zivtech-built sites are in the running for four awards! The recently-launched Greatist is nominated for best health website, while DoSomething.org is nominated for best Charitable Organizations/Non-Profit website,...
Feb 14, 2013 | Posted by: Alex Urevick-Ackelsberg
This weekend nine members of the Zivtech staff are participating and competing in Startup Weekend Health Philadelphia 2013, which, according to their site, "centers on building a web or mobile application that could be the basis for a credible business. After 500+ Startup Weekends world-wide...
Feb 11, 2013 | Posted by: Mason Wendell
We love Responsive Web Design, and we love Drupal. But do they love each other? After working on a number of RWD and Drupal projects this year, I'm happy to report that they get along just fine. Though "Love" might be stretching it a bit. Can Drupal and RWD be Pals? Drupal is a great platform...

Pages