Setting up IP failover with Heartbeat and Pacemaker on Ubuntu Lucid
One of Zivtech's clients recently asked us to renovate their server setup, with a focus on improving its availability.
This post will cover the highly available load balancer setup that currently serves 7 sites, with another 4 in the pipeline. Basically, we want to run the sites through two Linux-based software load balancer boxes, and be able to keep the sites up even if one of them fails.
- Two Ubuntu Lucid boxes, standard stripped down server install. We'll call them Bart and Lisa, because that's how server naming rolls at Zivtech
- A service IP per site (an IP address that is not the primary IP of the box it lives on, so it can be moved)
Heartbeat and Pacemaker are the real stars here. Pacemaker is the "brains" of the two, providing the high level control of our little cluster, while Heartbeat does the dirty work - it provides the low level inter-node communication and command framework.
Getting started with Heartbeat and Pacemaker
All of the packages we need are available in Ubuntu repositories, so we can just apt-get them on both boxes.
root@bart:~# apt-get install heartbeat pacemaker
The first step is to create an initial Heartbeat configuration and auth key on one of the boxes and copy it to the other, and restart the service.
root@bart:~# cat /etc/ha.d/ha.cf
root@bart:~# scp /etc/ha.d/ha.cf lisa:/etc/ha.d/
root@bart:~# ( echo -ne "auth 1\n1 sha1 "; \
dd if=/dev/urandom bs=512 count=1 | openssl md5 ) \
root@bart:~# chmod 0600 /etc/ha.d/authkeys
root@bart:~# scp /etc/ha.d/authkeys lisa:/etc/ha.d/
root@bart:~# ssh lisa chmod 0600 /etc/ha.d/authkeys
root@bart:~# /etc/init.d/heartbeat restart
root@bart:~# ssh lisa /etc/init.d/heartbeat restart
If all went well, both nodes of the cluster should be up and know about each other. To check, use one of the insanely helpful command line tools that come with the packages,
root@bart:~# crm_mon -1 | grep Online
Online: [ lisa.example.com bart.example.com ]
Ok, now we move on to setting up the IP addresses we would like Hearbeat and Pacemaker to keep available. There are a few ways to change the configuration of your cluster, but the easiest one is to use the
crm tool (go here for the excellent documentation). Note, this is the cluster configuration, not the configuration for a single node - you only need to update this on one node, and the changes will propagate automatically.
To Pacemaker, an IP address is a resource (and it can handle many different types of resources, but that's beyond the scope of this post). Assuming we have two sites, with IP addresses of 192.168.1.111 and 192.168.1.222 respectively, we'd tell Pacemaker about them like this:
root@bart:~# crm configure
crm(live)configure# primitive site_one_ip IPaddr params ip=192.168.1.111 cidr_netmask="255.255.255.0" nic="eth0"
crm(live)configure# primitive site_two_ip IPaddr params ip=192.168.1.222 cidr_netmask="255.255.255.0" nic="eth0"
To check that your configuration is what you think it is as you go, you can use
crm configure show. Right now, your configuration should look similar to this:
root@lisa:~# crm configure show
node $id="9f5e6cd6-2b75-4445-8d75-43c1a34fe431" bart.example.com
node $id="fb502469-691b-4021-b504-7da7a188bc63" lisa.example.com
primitive site_one_ip ocf:heartbeat:IPaddr \
params ip="192.168.1.111" cidr_netmask="255.255.255.0" nic="eth0"
primitive site_two_ip ocf:heartbeat:IPaddr \
params ip="192.168.1.222" cidr_netmask="255.255.255.0" nic="eth0"
The next thing we want to do is tell Pacemaker on which node we'd prefer the IPs to live when both nodes are up. We do this by setting a "location" for each IP resource. In the following configuration, we tell Pacemaker to keep one IP on each node when they are both up, which is a fairly typical setup for a pair of load balancers:
root@bart:~# crm configure
crm(live)configure# location site_one_ip_pref site_one_ip 100: bart.example.com
crm(live)configure# location site_two_ip_pref site_two_ip 100: lisa.example.com
Checking the configuration, we should see two lines like this added to our output of
crm configure show:
location site_one_ip_pref site_one_ip 100: bart.example.com
location site_two_ip_pref site_two_ip 100: lisa.example.com
Nearly there! Now that Pacemaker knows about our IP address resources, and where they should live, we can setup monitoring, which is the real meat of this exercise:
root@bart:~# crm configure
crm(live)configure# monitor site_one_ip 40s:20s
crm(live)configure# monitor site_two_ip 40s:20s
Done! Now, to test your setup, turn off/pull out the network cable from one of your nodes, and watch the other one pick up the IPs. Connect the the node again, and Pacemaker will honour your preferences, and send the failed-over IP address back to the original node.
Now that we have highly available load-balancing layer, the next post in this series will show you how to setup Apache to balance requests between backend servers running PHP and Drupal.
Are you an aspiring web developer, fresh out of college and eager to dive into Philly’s exploding tech startup scene? Are you a growing startup or established company in need of talented developers to help take your business to the next level?
Zivtech was thrilled last week when Philadelphia Mayor Michael A. Nutter announced that we were one of five winners of Startup PHL’s second round of Call for Ideas Grants, for a 6-week Web Development Bootcamp. We are hosting and teaching this bootcamp along with another awesome Philly web development firm--Neomind Labs--the goal of which is to allow us to share our expertise with, and help create job opportunities for, the next generation of Philly-based professional web developers.
We at Zivtech have been working with Drupal for a long time, some of us for over 10 years, and in that time we’ve gotten to use (and sometimes build) a lot of really cool modules to develop stellar sites for our clients. We have sifted through thousands of modules on Drupal.org, going far beyond the first couple pages of results and venturing deep into the unknown, unheard of, and ultimately under-appreciated modules. We sent out a question to our staff to ask for their favorite obscure modules, and we've compiled this list to share some of these modules that we have found particularly helpful, and to help ensure that they don't fall through the cracks. All of these modules have fewer than 8,000 active installs, and most have under 5,000.
As a Senior Developer here at Zivtech, part of my job is to develop with an eye towards security. If you follow development and security news, you know that achieving security is pretty hard in complex systems. But, while correctly implementing security across a site is challenging, some aspects of security are actually pretty easy and we've known what to do and what not to do for a long time.
Introducing Bear Skin, a comprehensive skin for Zivtech's Bear base theme.
The quality assurance (QA) phase of a web development project is the last phase before launch. While the development team has a lot of experience with QA phases, oftentimes the client team is new to the process, which can lead to stress. Let's prevent this with a little Q&A on QA.
Nominees for the 17th annual Webby Awards have been announced! And Zivtech-built sites are in the running for four awards!
This weekend 9 members of the Zivtech staff will be participating and competing in Startup Weekend Health Philadelphia 2013, which, according to their site, ”centers on building a web or mobile application that could be the basis for a credible business. After 500+ Startup Weekends world-wide, this will be the first dedicated solely to healthcare’s unique problems.”
We love Responsive Web Design, and we love Drupal. But do they love each other? After working on a number of RWD and Drupal projects this year, I'm happy to report that they get along just fine. Though "Love" might be stretching it a bit.
I've worked on many "Site Rescue" jobs, in which a client comes with a sick, but often brand-new and expensive Drupal site, desperately needing it fixed in a number of ways. Through this work I've seen a lot of patterns of "worst practices" employed. I enjoy the challenge of this cleanup work, while feeling bad for the clients who have been taken by incompetent vendors. I refer to these sites as "lemons" and I've made a study of their characteristics.