6.2. Upgrading from CRM-enabled Heartbeat 2.1 clusters

This section outlines the steps necessary to upgrade a Heartbeat 2.1 cluster with the built-in CRM enabled, to Heartbeat 3.0 with Pacemaker.


When properly planned and executed, the upgrade procedure can be completed in a manner of minutes, with no application down time. It is strongly recommended to read and understand the steps outlined in this section before attempting a production cluster uqpgrade. All commands stated in this section must be run as root. Do not perform the individual steps in parallel on all of your cluster nodes. Instead, complete the procedure on each node before continuing with the next.

6.2.1. Placing the cluster in unmanaged mode

With this step, the cluster temporarily relinquishes control of its resources. This means that the cluster no longer monitors its nodes or resources for the duration of the upgrade, and will not rectify and application or node failures during this time. Currently running resources, however, will continue to run.

# crm_attribute -t crm_config -n is_managed_default -v false

In most configurations, individual resources do not set the is_managed attribute individually, and hence the cluster-wide attribute is_managed_default applies to all of them.

If in your specific configuration you do have resources that have this attribute set, you should remove it to make sure the default applies:

# crm_resource -t primitive -r <resource-id> -p is_managed -D

6.2.2. Backing up the CIB

At this point, is is important to save a copy of the Cluster Information Base (CIB). The CIB to save is stored in a file named cib.xml, normally located in /var/lib/heartbeat/crm.

# cp /var/lib/heartbeat/crm/cib.xml ~

You need to perform this step on only one node currently connected to the cluster. Do not delete this file, it will be restored later.

6.2.3. Stopping Heartbeat services

You may now stop Heartbeat with /etc/init.d/heartbeat stop or the preferred command to stop a system service on your distribution (service heartbeat stop, rcheartbeat stop, etc.).

In case you are running a legacy version of Heartbeat affected by a shutdown bug, then graceful crmd shutdown will not work properly in unmanaged mode.

In this case, after you have initiated a graceful service shutdown with the above command, kill the crmd process manually:

  • use ps -AHfww to retrieve the process ID of crmd;
  • kill crmd with a TERM signal.

6.2.4. Wiping files related to the CRM


Before proceeding with this section, verify that you have created a backup copy of the CIB on one of your cluster nodes, as outlined in Section 6.2.2, “Backing up the CIB”.

You should now wipe local CRM-related files from your node. To do so, remove all files from the directory where the CRM stores CIB information, commonly /var/lib/heartbeat/crm.

# rm /var/lib/heartbeat/crm/*

6.2.5. Restoring the CIB


You should only proceed with this step if Heartbeat is still stopped on all cluster nodes, and all cluster nodes have had their CIB contents wiped. If you still have remaining nodes that have a residual CIB configuration, proceed as outlined in Section 6.2.4, “Wiping files related to the CRM”.

Restoring the CIB means copying the CIB backup described in Section 6.2.2, “Backing up the CIB” to the /var/lib/heartbeat/crm directory.

# cp ~/cib.xml /var/lib/heartbeat/crm/cib.xml
# chown hacluster:haclient /var/lib/heartbeat/crm/cib.xml
# chmod 0600 /var/lib/heartbeat/crm/cib.xml

You must perform this step on one node only, namely the first node on which you are about to upgrade the cluster software. On all other nodes, the /var/lib/heartbeat/crm directory must remain empty — Pacemaker distributes the CIB automatically.

6.2.6. Upgrading software

While upgrading, it is important to recall that the monolithic Heartbeat 2.1 tree has been split up into modular parts. Thus you will replace Heartbeat with three individual pieces of software: Cluster Glue, Pacemaker, and Heartbeat 3 which comprises just the cluster messaging layer.

  • Upgrading from source: In the unpacked archive that you installed Heartbeat 2.1 from, run make uninstall. Then, install Cluster Glue and Heartbeat.
  • Upgrading using locally built packages: When installing packages manually, uninstall the heartbeat package first. Then install cluster-glue, the version 3 heartbeat package, resource-agents, and pacemaker.
  • Upgrading using a package repository: When upgrading using an APT, YUM, or Zypper repository, you should just be able to run the install command for heartbeat version 3 and pacemaker, and the dependencies will be resolved automatically.

If this is the last node to be upgraded in your cluster, and your package management system did not restart Heartbeat services after the software upgrade, you should now proceed to Section 6.2.7, “Restarting Heartbeat services”. Otherwise, you should move to the next node and proceed as outlined in Section 6.1.1, “Stopping Heartbeat services” Section 6.2.6, “Upgrading software”.

6.2.7. Restarting Heartbeat services


This step may be omitted in case your package management system automatically restarts Heartbeat services during post-install.

First, restart Heartbeat on the node where you restored the CIB (see Section 6.2.5, “Restoring the CIB”) with /etc/init.d/heartbeat start. Then, repeat this command on the remaining cluster nodes. At this time,

  • the cluster is still in unmanaged mode (meaning it does not start, stop, or monitor any resources),
  • the cluster redistributes the old CIB among its nodes, and
  • the cluster is still using the pre-upgrade CIB schema.

6.2.8. Returning the cluster to managed mode

Once the cluster software has been upgraded, it is recommended to return the cluster into managed mode:

# crm_attribute -t crm_config -n is_managed_default -v true

6.2.9. Upgrading the CIB schema

Although an upgraded cluster can theoretically operate on the pre-upgrade CIB schema indefinitely, it is strongly recommended to upgrade the CIB to the current schema. To do so, run the following command after cluster communications between all nodes have been re-established:

# cibadmin --upgrade --force