Frequently Asked Questions

See also

There are other FAQs in this wiki:

Are there mailing lists for Linux-HA?

Yes! There are two public mailing lists for Linux-HA. You can find out about them by visiting the contact page[1].

What is a cluster?

HA (High availability Cluster) - This is a cluster that allows a host (or hosts) to become Highly Available, that means if one node[2] goes down (or a service on that node goes down) another node can pick up the service or node and take over from the failed machine. http://linux-ha.org/[3]

Computing Cluster - This is what a Beowulf[4] cluster is. It allows distributed computing over off the shelf components. In this case it is usually cheap IA32 machines.

Load balancing clusters - This is what the Linux Virtual Server[5] project does. In this scenario you have one machine which load balances requests to a certain server (Apache for example) over a farm of servers.

All of these sites have HOWTOs etc. on them. For a general overview on clustering under Linux, look at the Clustering HOWTO.

What is a resource script?

Resource scripts are basically (extended) System V[6] init scripts. They have to support stop, start, and status operations. In the future we will also add support for a monitor operation for monitoring services as you requested. The IPaddr[7] script implements this new monitor operation now (but Heartbeat[8] doesn't use that function of it). For more info see Resource HOWTO.

Some of my resources don't get started and I see no error messages, why?

Most likely there is some problem in your resource agent script. Command "<resource script> status" should not print either "ok" or "running" , otherwise Heartbeat[8] will think the resource[9] is running and will not run "<resource script> start". For a detailed description of a Heartbeat resource agent, see http://wiki.linux-ha.org/HeartbeatResourceAgent[10]

How to monitor various resources?

If one of my resource[9]s stops working Heartbeat[8] doesn't do anything unless the server crashes. How do I monitor resources with Heartbeat?

For Heartbeat[8] version 2[11], please see the Frequently Asked Questions for Heartbeat Version 2[12] page.

In version 1.x, Heartbeat was not designed for monitoring various resources. The best answer is to upgrade to an R2-style CRM[13] configuration. If you need to monitor some resources with the 1.x software (for example, availability of WWW server) you need some third party software. Mon[14] is a reasonable solution.

1. Get Mon from http://kernel.org/software/mon/[15] . Note that you are encouraged to use a mirror, not kernel.org directly.

2. Get all required modules listed. You can find them at nearest mirror or

   at the CPAN archive[16]. I am not very familiar with Perl, so I downloaded them from CPAN archive as .tar.gz packages and installed them the usual way (perl Makefile.pl && make && make test && make install).

3. Mon[14] is software for monitoring different network resources. It can ping computers, connect to various ports, monitor WWW, MySQL etc. In case of dysfunction of some resources it triggers some scripts.

4. Unpack Mon[14] in some directory. The best starting point is the README file. Complete documentation is in <dir>/doc, where <dir> is the place where you unpacked the Mon package.

5. For a fast start do the following steps:

   1. copy all subdirs found in <dir> to /usr/lib/mon

   2. create directory /etc/mon

   3. copy auth.cf from <dir>/etc to /etc/mon

Now, Mon[14] is prepared to work. You need to create your own mon.cf file, where you should point to resource[9]s Mon should watch and actions Mon will start in case of dysfunction and when resources are available again. All monitoring scripts are in /usr/lib/mon/mon.d/. At the beginning of every script you can find explanation of how to use it. All alert scripts are placed in /usr/lib/mon/alert.d/. Those are scripts triggered in case something went wrong. In case you are using IPVS (IP Virtual Server) on their homepage (http://www.linuxvirtualserver.org[17]) you can find scripts for adding and removing servers from an IPVS list.

Consider using hb_standby[18] to cause services to fail over[19] to the other side in case of problems.

If one of my Ethernet connections goes away (cable severance, NIC failure, locusts), but my current primary node (the one with the services) is otherwise fine, no one can get to my services and I want to fail them over to my other cluster node. Is there a way to do this?

Yes!

For each interface you wish to monitor, specify one or more "ping nodes[20]" in your configuration. Each node[21] in your cluster[22] will monitor these ping nodes.

For R1-style resources Use the ipfail plug-in

Should one node detect a failure in one of these ping nodes, it will contact the other node in order to determine whether it or the ping node has the problem. If the cluster node has the problem, it will try to failover[19] its resource[9]s (if it has any).

To use ipfail[23] you will need to add the following to your /etc/ha.d/ha.cf[24] files:

 respawn hacluster /usr/lib/heartbeat/ipfail
 ping <IPaddr1> <IPaddr2> ... <IPaddrN>

IPaddr1..N are your ping nodes. NOTE: ipfail[23] requires the auto_failback[25] option to be set to on or off (not legacy).

See the ipfail page[23] for more information on ipfail. Note: ipfail[23] only works for R1-style resources.

For R2-style resources Use the pingd plug-in

For R2-style resources, use the pingd[26] daemon instead of ipfail[23]. The ping nodes are configured in the same way, but pingd[26] provides much more flexibility in terms of what should fail over under what conditions. Note: pingd[26] only works for R2-style resources.

Every time my machine releases an IP alias, it loses the whole interface (i.e. eth0)! How do I fix this?

This isn't a problem with Heartbeat[8], but rather is caused by various versions of net-tools. Upgrade to the most recent version of net-tools and it will go away. You can test it with ifconfig manually.

I want a lot of IP addresses as resources (more than 8). What's the best way?

Instead of failing over[19] many IP addresses, just fail over one router address. On your router, do the equivalent of "route add -net x.x.x.0/24 gw x.x.x.2", where x.x.x.2 is the cluster[22] IP address controlled by Heartbeat[8]. Then, make every address within x.x.x.0/24 that you wish to failover a permanent alias of lo0 on both cluster node[21]s. This is done via "ifconfig lo:2 x.x.x.3 netmask 255.255.255.255 -arp" etc...

The documentation indicates that a serial line is a good idea, is there really a drawback to using two Ethernet connections?

If anything makes your Ethernet / IP stack fail, you may lose both connections. You definitely should run the cables differently, depending on how important your data is... If you're running an ipchains firewall, then it's easy to accidentally cause a SplitBrain[27] occurrence if you make a mistake in the rules, and currently our serial ports aren't subject to firewall rules at all.

However, most CRM-style configurations don't work well with serial configurations, because of the amount of data being transferred. If you want to try this, make sure your baud rate is as high as you can set it.

How to use Heartbeat with Ipchains firewall?

To make Heartbeat[8] work with Ipchains[28], you must accept incoming and outgoing traffic on 694 UDP port. Add something like

/sbin/ipchains -A output -i ethN -p udp -s <source_IP> -d <dest_IP>  -j ACCEPT
/sbin/ipchains -A input -i ethN -p udp -s <source_IP> -d <dest_IP>  -j ACCEPT

I got this message "ERROR: No local heartbeat. Forcing shutdown" and then Heartbeat shut itself down for no reason at all!

First of all, Heartbeat[8] never shuts itself down for no reason at all. This kind of occurrence indicates that Heartbeat is not working properly, which in our experience can be caused by one of two things:

• System under heavy I/O load, or
• Kernel bug.

For how to deal with the first occurrence (heavy load), please read the answer to the next FAQ item. If your system was not under moderate to heavy load when it got this message, you probably have the kernel bug. The 2.4.18-2.4.20 Linux kernels had a bug in it which would cause it to not schedule Heartbeat for very long periods of time when the system was idle, or nearly so. If this is the case, you need to get a kernel that isn't broken.

How to tune Heartbeat on heavily loaded system to avoid split-brain?

"No local heartbeat" or "Cluster node returning after partition" under heavy load is typically caused by too small a deadtime[29] interval, or an older version of Heartbeat[8]. Make sure you're running at least version 1.2.0. Here is a suggestion for how to tune deadtime:

• Set deadtime[29] to 60 seconds or higher

• Set warntime[30] to 1/4 to 1/2 of whatever you *want* your deadtime to be.

• Run your system under heavy load for a few weeks.
• Look at your logs for the longest time either system went without hearing a heartbeat. If your never saw a "late heartbeat" message, then your chosen deadtime is fine - use it.
• Set your deadtime to 1.5-2 times that amount.

• Set warntime to keepalive[31]*2.

• Continue to monitor logs for warnings about long heartbeat times. If you don't do this, you may get "Cluster node ... returning after partition"

  which will cause Heartbeat to restart on all machines in the cluster[22]. This will almost certainly annoy you at a minimum.

Adding memory to the machine generally helps. Limiting workload on the machine generally helps. Newer versions of Heartbeat are a better about this than pre 1.2.x versions. Some customers report being able to set sub-second deadtimes in their applications with 1.2.3. YMMV(!).

When Heartbeat starts up I get this error message in my logs: "WARN: process_clustermsg: node [<hostname>] failed authentication". What does this mean?

In general, this message usually means that a packet has been corrupted. It's common to get a single mangled packet on your serial interface when Heartbeat[8] starts up. This is an indicator that happened. It's harmless in this scenario. It will happen occasionally under other circumstances, and as long as it only happens occasionally (like less than once an hour), it's harmless. If it happens continually, there is probably something else going on to cause packet corruption, and this cause should be investigated and corrected.

When I try to start Heartbeat I receive this message: "Starting High-Availability services: Heartbeat failure [rc=1]. Failed." and there is nothing in any of the log files and no messages. What is wrong?

It's probably a permissions problem on authkeys[32]. It wants it to be read only mode (400, 600 or 700). Depending on where and when it discovers the problem, the message will wind up in different places.

But, it tends to be in

1. stdout/stderr
2. wherever you specified in your setup

3. /var/log/messages

Newer releases (>= 1.0) are better about also putting out startup messages to stderr in addition to wherever you have configured them to go.

How to run multiple clusters on the same network segment?

Use multicast[33] and give each its own multicast group. If you need to/want to use broadcast, then run each cluster[22] on different port numbers. An example of a configuration using multicast would be to have the following line in your ha.cf[24] file:

 mcast eth0 224.1.2.3 694 1 0

This sets eth0 as the interface over which to send the multicast, 224.1.2.3 as the multicast group (will be same on each node[21] in the same cluster), UDP port 694 (Heartbeat default), time to live of 1 (limit multicast to local network segment and not propagate through routers), multicast loopback disabled (typical).

How to get the latest CVS version of Heartbeat?

There is a CVS[34] repository for Linux-HA. You can find it at http://cvs.linux-ha.org/viewcvs/viewcvs.cgi/linux-ha/[35] . Read-only access is via login guest, password guest, module name linux-ha. More details are to be found on the HeartbeatCVS[36] page.

Heartbeat on other OSes

Heartbeat[8] now uses automake and is generally quite portable at this point. Join the Linux-HA-dev mailing list[1] if you want to help port it to your favorite platform.

When I try to install the linux-ha.org Heartbeat RPMs, they complain of dependencies from packages I already have installed! Now what?

Due to distribution RPM[37] package name differences, this was unavoidable. If you're not using STONITH[38], use the "--nodeps" option with RPM. Otherwise, use the Heartbeat[8] source to build your own RPMs. You'll have the added dependencies of autoconf >= 2.53 and libnet (get it from http://www.packetfactory.net/libnet[39] ). Use the Heartbeat source RPM (preferred) or unpack the Heartbeat source and from the top directory, run ./ConfigureMe rpm. This will build RPMs and place them where it's customary for your particular distro. It may even tell you if you are missing some other required packages!

I don't want Heartbeat to fail over the cluster automatically. How can I require human confirmation before failing over?

You configure a "meatware" STONITH[38] device into the ha.cf[24] file. The meatware STONITH device asks the operator to go power reset the machine which has gone down. When the operator has reset the machine he or she then issues a command to tell the meatware STONITH plugin that the reset has taken place. Heartbeat will wait indefinitely until the operator acknowledges the reset has occurred. During this time, the resource[9]s will not be taken over, and nothing will happen.

What is STONITH? And why might I need it?

STONITH[38] is a form of fencing[40], and is an acronym standing for Shoot The Other Node In The Head. It allows one node[21] in the cluster[22] to reset the other. Fencing is essential if you're using shared disks, in order to protect the integrity of the disk data. Heartbeat[8] supports STONITH fencing, and resource[9]s which are self-fencing. You need to configure some kind of fencing whenever you have a cluster resource which might be permanently damaged if both machines tried to make it active at the same time. When in doubt check with the Linux-HA mailing list[1].

How do I figure out what STONITH devices are available, and how to configure them?

To get the list of supported STONITH[38] devices, issue this command:

stonith -L

To get all the gory details on exactly what these STONITH device names mean, and how to configure them, issue this command:

stonith -h

I want to use a shared disk, but I don't want to use STONITH. Any recommendations?

This is not something which Heartbeat[8] supports directly, however, there are a few kinds of resource[9]s which are "self-fencing". This means that activating the resource causes it to fence[40] itself off from the other node[21] naturally. Since this fencing happens in the resource agent, Heartbeat doesn't know (and doesn't have to know) about it. Two possible hardware candidates are IBM's ServeRAID[41] RAID controllers and ICP Vortex RAID controllers - but do your homework!!! When in doubt check with the mailing list[1].

Can Heartbeat be configured in an active/active configuration? If so, how do I do this, since the haresources script is supposed to be the same on each box so I do not know how this could be done.

Yes, Heartbeat has supported active/active[42] configurations since its first release. The key to configuring active/active[42] clusters[22] is to understand that each resource group[43] in the haresources[44] file is preceded by the name of the server which is normally supposed to run that service. When in an "auto_failback[25] yes (or legacy)" (or old-style "nice_failback off") configuration, when a cluster node[21] comes up, it will take over any resources for which it is listed as the "normal master" in the haresources[44] file. Below is an example of how to do this for an Apache/MySQL configuration.

server1 10.10.10.1 mysql
server2 10.10.10.2 apache

In this case, the IP address 10.10.10.1 should be replaced with the IP address you want to contact the MySQL server at, and 10.10.10.2 should be replaced with the IP address you want people to use to contact the web server. Any time server1 is up, it will run the MySQL service. Any time server2 is up, it will run the Apache service. If both server1 and server2 are up, both servers will be active. Note that this is contradictory with the old nice_failback on parameter. With the new release which supports hb_standby foreign, you can manually fail back[45] into an active/active[42] configuration if you have auto_failback off. This allows administrators more flexibility in failing back in a more customized way at more safe or convenient times.

Why are my interface names getting truncated when they're brought up and down?

Heartbeat[8] was written to use ifconfig to manage its interfaces. That's nice for portability for other platforms, but for some reasons ifconfig truncates interface names. If you want to have fewer than 10 aliases, then you need to limit your interface names to 7 characters, and 6 for fewer than 100 interfaces. Or, you can use the IPaddr2[46] resource script.

What is this auto_failback parameter? What happened to the old nice_failback parameter?

The auto_failback[25] parameter is a replacement for the old nice_failback parameter. The old value nice_failback on is replaced by auto_failback off. The old value nice_failback off is logically replaced by the new auto_failback on parameter. Unlike the old nice_failback off behavior, the new auto_failback on allows the use of the ipfail[23] and hb_standby[18] facilities.

During upgrades from nice_failback to auto_failback, it is sometimes necessary to set auto_failback to legacy, as described in the upgrade procedure below.

I am upgrading from a version of Linux-HA which supported nice_failback to one that supports auto_failback. How to I avoid a flash cut in this upgrade?

To upgrade from a pre-auto_failback version of Heartbeat[8] to one which supports auto_failback[25], the following procedures are recommended to avoid a flash cut on the whole cluster[22].

1. Stop Heartbeat on one node[21] in the cluster.

2. Upgrade this node. If the other node has nice_failback on in ha.cf[24] then set auto_failback off in the new ha.cf[24] file. If the other node in the cluster has nice_failback off then set auto_failback legacy in the new ha.cf[24] file.

3. Start the new version of Heartbeat on this node.
4. Stop Heartbeat on the other node in the cluster.
5. Upgrade this second node in the cluster with the new version of

   Heartbeat. Set auto_failback the same as it was set in the previous step.

6. Start Heartbeat on this second node in the cluster.

7. If you set auto_failback to on or off, then you are done. Congratulations!

8. If you set auto_failback legacy in your ha.cf[24] file, then continue as described below...

   1. Schedule a time to shut down the entire cluster for a few seconds.
   2. At the scheduled time, stop both nodes in the cluster, and then change

      the value of auto_failback to on in the ha.cf[24] file on both sides.

   3. Restart both nodes on the cluster at about the same time.

   4. Congratulations, you're done! You can now use ipfail[23], and can also use the hb_standby[18] command to cause manual resource[9] moves.

If nothing helps, what should I do?

I want to submit a patch, how do I do that?

We love to get good patches. Here's the preferred way:

• If you have any questions about the patch, please check with the

  Linux-HA-Dev mailing list[1] for answers before starting

• Make your changes against the current CVS[34] source

• Test them, and make sure they work

• Produce the patch this way:

   cvs -q diff -u >patchname.txt

• Send an email to the Linux-HA-Dev mailing list[1] with the patch as a [text/plain] attachment. If your mailer wants to zip it up for you, please fix it.

  Another good way is to submit the patch attached to a bug report. See the ContactUs[1] page for how to access our bug reporting system. There's nothing wrong with doing both - if you don't mind.

I reinstalled a machine, and now I'm getting "attempted replay attack" messages

We just reinstalled our master node (paul) and Heartbeat[8] (1.2.0) is saying this on the slave node[21] (silas - which has the resource[9]s):

Mar 16 19:31:43 silas heartbeat[12561]: ERROR: should_drop_message: attempted replay attack [paul]?  
              [gen = 1, curgen = 10] 
Mar 16 19:32:15 silas last message repeated 38 times 
Mar 16 19:33:17 silas last message repeated 62 times

What should we do to get the resources back on the master node?

Put 11 (curgen+1) in /var/lib/heartbeat/hb_generation on paul - from this log it should have a 1 (gen) in there now.

Basically, it should be one larger than the curgen number from the message above.

Then if you restart Heartbeat on the master node (paul), all should be well. This is the result of a feature called ReplayAttackProtection[50]. You can also just restart Heartbeat on both nodes, if you prefer.

So, if you put any number larger than curgen into the hb_generation file on paul, on the machine you reinstalled, and restart, Heartbeat will be happy.

I have multiple Ethernet cards in one machine, how can I have IP failover within a single node?

IP failover[19] within a single node can be set using bonding device. It is described in IpFailoverChannelBonding[51].

How do I use Linux-HA for high availability NFS servers?

How to configure HA NFS, and corresponding test results can be found in the HaNFS[52] page.

I got this message "TTY write timeout on [/dev/ttyxxx]" but both nodes are up and I tested my serial cable

If both node[21]s are up, and your serial cable passes data, then the most probable explanation for the problem is that the serial cable does not pass the CTS and RTS leads through from end to end properly. Heartbeat[8] requires these leads in order to avoid data loss. See the SerialCable[53] page for more details.

How do I change the names of hosts in a cluster without any service interruption?

This discussion assumes that your current machines are named A and B, and you want them to be named C and D respectively after the change.

• A: Make A (host that will become C) active for all services: hb_takeover[54].

  B: /etc/init.d/heartbeat stop; vi /etc/ha.d/ha.cf[24] /etc/ha.d/haresources[44] # add an entry for C and D so 4 hosts are listed B: /etc/init.d/heartbeat start

  B: hb_takeover[54]

  A: /etc/init.d/heartbeat stop; vi /etc/ha.d/ha.cf[24] # add an entry for C and D so 4 hosts are listed A: /etc/init.d/heartbeat start A: hostname C # changes the first host's name # don't forget to update /etc/hostname to C

  A: hb_takeover[54]

  B: /etc/init.d/heartbeat stop; vi /etc/ha.d/ha.cf[24] /etc/ha.d/haresources[44] # remove A and B leaving only C and D from both files B: hostname D # changes the second host's name, don't forget to update # /etc/hostname to D B: /etc/init.d/heartbeat start

  B: hb_takeover[54]

  A: /etc/init.d/heartbeat stop; vi /etc/ha.d/ha.cf[24] /etc/ha.d/haresources[44] # remove A and B leaving only C and D in both files A: /etc/init.d/heartbeat start

  A: hb_takeover[54] # Leave host A as the active load balancer, if you wish

How do I do live changes to resources?

The general case is somewhat painful for R1-style configurations, but I think these recipes should work starting with the 1.2.x series... But, the best answer is to use an R2-style CRM configuration, because then this all just works exactly like it should.

If you want to reorganize them, but not eliminate or add any, then this should work:

1. do an hb_standby[18] on either side...

2. update haresources[44] on both sides

If you don't like where the resource[9]s are now running...

1. do an hb_standby[18] foreign on side A (wait for it to complete)

2. do an hb_standby[18] foreign on side B

If you want to add or eliminate some, then this should work:

1. do an hb_standby[18] on the A side

2. update the A side's haresources[44] file

3. do an hb_standby[18] on the B side (wait for it to complete)

4. update the haresources[44] file on the B side

5. do an hb_standby[18] foreign on side A (wait for it to complete)

6. do an hb_standby[18] foreign on side B

Heartbeat is declaring both nodes dead, even though both of them are up. How do I fix this?

This is almost always because of firewall rules. You may be running firewall rules even if you don't think you are. Most current distributions ship with firewall rules enabled by default. So, if you are having this problem, please check for firewall rules before asking for further help.

To check firewall rules, try using the "iptables-save" command, which dumps your current set of firewall rules. If you have no firewall rules, you would see output similar to this:

  [2] guin:~# iptables-save
  # Generated by iptables-save v1.3.0 on Fri Dec 30 16:29:07 2005
  *filter
  :FORWARD ACCEPT [0:0]
  :INPUT ACCEPT [16:1935]
  :OUTPUT ACCEPT [19:1790]
  COMMIT
  # Completed on Fri Dec 30 16:29:07 2005
  [2] guin:~#

On Red Hat and Fedora Core systems you can disable firewall rules by running "/etc/init.d/iptables stop". This is recommended only for testing, you should always run a firewall on your systems except when explicitly testing to see if firewall rules are causing your problems.

If you have followed these steps and are sure that you have no firewall rule in place, but still are having problems, mention what you have done to ensure it's not firewall rules when asking for help. Pretty much every time this error is reported it has been because of a firewall, and in many cases the users insisted that there were no firewall rules. So, save yourself some time and make sure to check this and report the results in your request for community support.

---

CategoryHowto[55]

---

This information provided courtesy of the Linux-HA project at http://linux-ha.org/