Contents

1. Frequently Asked Questions for Heartbeat Version 2
   1. Why does crmadmin think my resource is not active?
   2. How Do I Write an OCF Compliant Resource Agent?
   3. Is my Init Script Compatible with Heartbeat?
   4. How do I monitor a resource for failure?
   5. What happens when monitor detects the resource down?
   6. Can I load a new cib.xml config in Heartbeat without restarting Heartbeat?
   7. Is it a good idea to keep the cib.xml in sync with the other nodes?
   8. What's the difference between all those timers/timeouts?
   9. How do I know Heartbeat is installed correctly?
   10. Why doesn't hb_standby do work in version 2?
   11. How do I replace a cluster node?
   12. See Also

1. ja/v2/faq/forced failover ja
2. ja/v2/faq/manual recovery ja
3. ja/v2/faq/pingd ja
4. ja/v2/faq/resource too active ja
5. ja/v2/faq/stop resource ja
6. ja/v2/faq/time based failback ja
7. v2/faq/cib changes detected
8. v2/faq/forced failover
9. v2/faq/manual recovery
10. v2/faq/pingd
11. v2/faq/resource too active
12. v2/faq/stop resource
13. v2/faq/time based failback

How Do I Force a Resource to be Migrated After a Failure?

In Versions prior to 2.0.5 there is no way to do this.

Subsequent versions can use the resource-failure-stickiness property of a resource (or the global default: default-resource-failure-stickiness) to control when failover happens.

Example

In a cluster where:

• default-resource-failure-stickiness is -100

• default-resource-stickiness is 500

• Resource my_rsc prefers to run on nodeA with score 1500

• Resource my_rsc prefers to run on nodeB with score 1000

my_rsc can fail up to ten times on nodeA before being moved to nodeB.

(nodeA score - nodeB score + stickiness) / abs(failure stickiness)

==> (1500 - 1000 + 500) / abs(-100)

==> 1000 / 100

==> Answer: 10

However if default-resource-failure-stickiness was set to -1001 or less, it would be moved immediately.

(nodeA score - nodeB score + stickiness) / abs(failure stickiness)

==> (1500 - 1000 + 500) / abs(-1001)

==> 1000 / 1001

==> Answer: 0.999

NOTE: Failure stickiness should be a negative value.
NOTE: The rules for +/-INFINITY apply...

 INFINITY +/- -INFINITY : -INFINITY
 INFINITY +/-  int      :  INFINITY
-INFINITY +/-  int      : -INFINITY

Multiple Failures

If the combined score for my_rsc on a node is less than zero, it will never be able to run there again until the failure count is reset.

The current failure count (for a given resource and node) is multiplied by the resource's failure stickiness to produce a failover score. When the failover score exceeds the regular preference to a given node, the node will be excluded from running the resource again (until the failure count is reset).

In the first example above (where default-resource-failure_stickiness is -100) my_rsc can fail up to 15 times on nodeA before Heartbeat will no longer consider running it there. Likewise it can fail up to 10 times on nodeB before that node will no longer be consider either.

In the second example (where default-resource-failure-stickiness is less than -500), these values drop to 3 and 2 failures respectively.

Resetting Failure Counts

To reset the failure count for my_rsc on nodeA, the following command can be used:

• crm_failcount -D -U nodeA -r my_rsc

To query the current failure count, use:

• crm_failcount -G -U nodeA -r my_rsc

Why the Failure Count is not Automatically Reset When a Resource is Moved

The failure count is not automatically reset when a resource is moved to prevent the resource from bouncing between two or more "bad" nodes.

How this works out with groups

When you use groups or other things with mandatory colocation constraints, it gets more complicated. If you take everything which you force to be colocated on the same machine, then all those things will be treated just as though they were in a group together. So, keep that in mind in the explanation below.

For a group with n resources, the resource-stickiness of the group of n resources is the sum of all the stickinesses in the group. Unless you've overridden the default for the resources in the group, that would be n times the default-resource-stickiness, and it is the stickiness of the group as a whole which is used, rather than the stickiness of any member of the group. The resource-failure-stickiness value for a group is computed in a similar way.

In addition, if you want to give a different resource-stickiness or resource-failure-stickiness to every member in the group, you can give this attribute to the group itself. However, it will effectively be multiplied by n in the process of summing up all the stickiness values of the group members, because this value is inherited by each primitive resource in the group, and then subjected to the summing process described above.

Things to keep in mind

• You probably want every resource in the group to score the same, so that computing all these score differences for failures is simpler. A simple way to do this is to put your locational constraints on your groups rather than on individual resources in the group. Note that unlike the stickiness values, the scores for locational constraints on groups are not multiplied by n - so to make things balance out correctly, you may need to do this yourself when you create the corresponding CIB.

• If you want to reset the failure counts so that things can fail back, you probably want to reset the failure counts of every resource in the group, since you don't know which one failed without searching the logs (see item below).

What you cannot do

You cannot force a group to fail over if any given resource in the group fails 'n' times. Failure counts for groups are cumulative. If you set it up so that 3 failures will cause a failover, then 3 failures of any resource in the group will cause a failover, not 3 for a particular resource. If you have a web server, an IP address and a mount point, then it will fail over with 3 web server failures, or one failure of each resource in the group.

A worked example involving a Group

Problem Definition

The outline of the requirements to be met are:

• Two-node cluster

• One group using pingd[16] for pinging a gateway

• Fail over the group on the third failure of any resource in the group
• One node is preferred over the other
• These location criteria are listed in order of decreasing importance:

  1. Don't run on a node which has failed 3 times without resetting failure counts (resource-failure-stickiness)

  2. Run on a node with ping access according to pingd[16]

  3. Stay on the node you're on if it has ping access (resource-stickiness)

  4. Run the resource group on the designated (preferred) cluster node

Problem Solution

To be supplied

How do I Manually Recover a Resource?

Why is my Resource Listed as Unmanaged?

• There are two common reasons for a reasons for the cluster to consider a resource unmanaged:
  • The resource is reported as active on more than one node
  • The resource failed to stop when the cluster asked it to and STONITH is not available

The authoritative source for information about what resources are active and what state they're in is the LRM. So while you could change the contents of the CIB directly, these changes are likely to be overwritten with data from the LRM at some point in the future.

Once you have manually stopped and otherwise cleaned up the resource, you should use the -C option for crm_resource to remove the resource from the LRM. More information on its usage is available with crm_resource --help. The general form is crm_resource -C -r my_resource_name. By default it will operate on all cluster nodes unless you specify -H some_host as well.

Since 2.0.5 the command will automatically instruct the cluster to refresh itself with the latest details from the LRM. Prior to that, users should "wait a bit" and run crm_resource -R.

Prior to 2.0.6 (which has not yet been released as of writing this) users had to be careful choosing the right value to use with -r. This affects resources in groups and is due to a bad choice earlier in the CRM's history and due to backwards compatibility we are stuck with it being the default.

The following command will list for you the internal names used by the cluster for its resources:
cibadmin -Ql -o status | grep "lrm_resource " | awk '{print $2}' | sort -u

Look for the one that matches your resource and use that value when calling crm_resource.

How Do I Monitor Connectivity in Version 2

pingd, An ipfail Replacement

pingd is a replacement for ipfail that like the rest of version 2 allows for connectivity (and resource placement based on relative connectivity) to work in clusters with any number of nodes.

The role of pingd is to detect changes to a node's connectivity and ensure that updates of this information to the CIB[17] occur (at least effectively) simultaneously.

To locate your resources on the node(s) with the greatest connectivity, an admin needs to use the information placed in the CIB by pingd. This is achieved with the creation of resource location[18] constraints that reference the attribute created by pingd. See "Using pingd Output in Location Constraints" below.

Configuration Methods

There are two options for configuring pingd

1. The first is by adding a respawn directive to ha.cf eg:
   

   • respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s
     See "pingd Usage Information" below for details on the meaning of the options passed to pingd.

2. The other option is to create a clone[19] using the pingd OCFResourceAgent[20]. Since RA's are started as root, you need to add a line like "apiauth pingd uid=root" to your ha.cf[12] or - even better - add the parameter user=hacluster to your RA configuration to use the implicit apiauth directive like the respawn method does. With version 2.0.6 you also have to add parameter pidfile which points to somewhere the chosen user has write permission (e.g. /tmp/pingd-default).

Both methods need the ping nodes listed in your ha.cf[12] file. The host_list parameter of the RA can only use a subset of those, not some other hosts.

Both methods also require the addition of one-or-more colocation constraints to the CIB. See "Using pingd Output in Location Constraints" below.

The advantage of using the resource agent is that you can:

• limit the connectivity information to nodes
  • with particular properties (names, ids, node attributes)
  • running a particular resource
• dynamically alter any of the input or output options including
  • the list of ping nodes being monitored
  • the attribute name
  • the multiplier
  • the dampening delay

An equivalent resource for the respawn directive above would be:

<clone id="pingd-clone">
  <meta_attributes id="pingd-clone-ma">
    <attributes>
      <nvpair id="pingd-clone-1" name="globally_unique" value="false"/>
    </attributes>
  </meta_attributes>
  <primitive id="pingd-child" provider="heartbeat" class="ocf" type="pingd">
    <operations>
      <op id="pingd-child-monitor" name="monitor" interval="20s" timeout="60s" prereq="nothing"/>
      <op id="pingd-child-start" name="start" prereq="nothing"/>
    </operations>
    <instance_attributes id="pingd_inst_attr">
      <attributes>
         <nvpair id="pingd-1" name="dampen" value="5s"/>
         <nvpair id="pingd-2" name="multiplier" value="100"/>
      </attributes>
    </instance_attributes>
  </primitive>
</clone>

NOTE: Changing the attribute's location in the CIB, while possible, is discouraged. This is because you may end up with multiple copies of the attribute for each node... causing the cluster to behave differently than expected.

pingd Usage Information

usage: pingd [-V?p:a:d:s:S:h:Dm:]
        --help (-?)                     This text
        --daemonize (-D)                Run in daemon mode
        --pid-file (-p) <filename>      File in which to store the process' PID
                                        * Default=/tmp/pingd.pid
        --attr-name (-a) <string>       Name of the node attribute to set
                                        * Default=pingd
        --attr-set (-s) <string>        Name of the set in which to set the attribute
                                        * Default=cib-bootstrap-options
        --attr-section (-S) <string>    Which part of the CIB to put the attribute in
                                        * Default=status
        --ping-host (-h) <single_host_name> Monitor a subset of the ping nodes listed in ha.cf
                                            (can be specified multiple times)
        --attr-dampen (-d) <integer>        How long to wait for no further changes to occur before
                                            updating the CIB with a changed attribute
        --value-multiplier (-m) <integer>   For every connected node, add <integer> to the value set in the CIB
                                            * Default=1

Using pingd Output in Location Constraints

Example pingd Configuration

• respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s -a default_ping_set

Example CIB Contents

• c001n04	N/A	N/A
  c001n05	0	0

Example Constraint

<rsc_location id="my_resource_connected" rsc="my_resource">
    <rule id="my_resource_connected_rule" score_attribute="default_ping_set">
       <expression id="my_resource_connected_expr_defined" attribute="default_ping_set" operation="defined"/>
    </rule>
</rsc_location>

The above constraint:

• requires a value to be set for default_ping_set (c001n04 is unaltered)

• requires the value of default_ping_set to be greater than 100 (c001n05 is unaltered)

• increases the preference for my_resource to run on c001n01 by 500

• increases the preference for my_resource to run on c001n02 by 400

• increases the preference for my_resource to run on c001n03 by 500

• c001n04	N/A	N/A	0
  c001n05	0	0	0

If we also had the following constraint:

<rsc_location id="my_resource_preferred" rsc="my_resource">
    <rule id="my_resource_prefer_c001n01" score="100">
       <expression id="my_resource_prefer_c001n01_expr" attribute="#uname" operation="eq" value="c001n01"/>
    </rule>
    <rule id="my_resource_prefer_c001n02" score="200">
       <expression id="my_resource_prefer_c001n02_expr" attribute="#uname" operation="eq" value="c001n02"/>
    </rule>
    <rule id="my_resource_prefer_c001n03" score="300">
       <expression id="my_resource_prefer_c001n03_expr" attribute="#uname" operation="eq" value="c001n03"/>
    </rule>
    <rule id="my_resource_never" score="-INFINITY" boolean_op="or">
       <expression id="my_resource_never_c001n04_expr" attribute="#uname" operation="eq" value="c001n04"/>
       <expression id="my_resource_never_c001n05_expr" attribute="#uname" operation="eq" value="c001n05"/>
    </rule>
</rsc_location>

Then the updated scores for running the resource would be:

• c001n04	N/A	N/A	-INFINITY
  c001n05	0	0	-INFINITY

At this point, if the resource was not running or v2/dtd1.0/annotated#default resource stickiness[21] was set to zero, then the resource would be started on c001n03 with c001n01 and c001n02 equally preferred as a backup.

However if the resource was running on c001n02 and resource_stickiness was set to 1000, then the updated scores would be:

• c001n04	N/A	N/A	-INFINITY
  c001n05	0	0	-INFINITY

and the resource would be left running on c001n02.

Alternatively, if resource_stickiness was set to 100, then the scores would look like this:

• c001n04	N/A	N/A	-INFINITY
  c001n05	0	0	-INFINITY

and the resource would be moved to c001n03.

This should also adequately demonstrate the importance of correctly setting:

• pingd's --value-multiplier option

• default_resource_stickiness / resource_stickiness

• score in rsc_location constraints

Quickstart - Only Run my_resource on Nodes with Access to at Least One Ping Node

Add this to ha.cf

• respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s -a pingd

Add this constraint to the CIB:

<rsc_location id="my_resource:connected" rsc="my_resource">
  <rule id="my_resource:connected:rule" score="-INFINITY" boolean_op="or">
    <expression id="my_resource:connected:expr:undefined"
      attribute="pingd" operation="not_defined"/>
    <expression id="my_resource:connected:expr:zero"
      attribute="pingd" operation="lte" value="0"/>
  </rule>
</rsc_location>

Quickstart - Run my resource on the node with the best connectivity

This is probably one of the better ways to use pingd. In this method, the pingd attribute value becomes the score for the rule. So, the --value-multiplier you set will depend heavily on the scores you give other criteria. This rule will not stop a resource completely if all nodes lose connectivity to the outside world.

<rsc_location id="my_resource:connected" rsc="my_resource">
  <rule id="my_resource:connected:rule" score_attribute="pingd" >
    <expression id="my_resource:connected:expr:defined"
      attribute="pingd" operation="defined"/>
  </rule>
</rsc_location>

Resource X is (potentially) active on 2 nodes.

Heartbeat will try and determine what resources are active on a machine when it starts. To do this, it sends what we call a probe which uses the monitor operation of your ResourceAgent[23].

There are two common reasons for seeing this message:

1. Your resource really is active on more than one node
   • Check you are not starting it on boot
   • Did Heartbeat suffer an internal failure?

     If so, please check the ReportingProblems[11] page and report it

2. Your resource doesn't implement the monitor operation correctly

   • Make sure your ResourceAgent[23] conforms to the OCF-spec

     • return 0 only for active resource instances
     • return 7 for inactive resource instances
     • return anything else for failed resource instances

How do I Make the Cluster Stop Resource X?

As of version 2.0.5 it is now possible to ask the cluster to stop a resource and leave it stopped.

To do this, you need to set an instance attribute called target_role for the resource.

To do this, your resource definition should look like something this:

 <resources>
   <group id="cds_grp">
     <primitive class="ocf" id="cds_res" provider="heartbeat" type="cds">
       <instance_attributes id="cds-defaults">
          <attributes>
             <nvpair id="cds-role" name="target_role" value="Stopped"/>
          </attributes>
       </instance_attributes>
     </primitive>
   </group>
 </resources>

Once you have done whatever you needed to do and are ready to start it again, the following command will tell the cluster to start it.

cibadmin -M -o resources -X '<nvpair id="cds-role" value="Started"/>'

To later cause the resource to be stopped again, you would run:

cibadmin -M -o resources -X '<nvpair id="cds-role" value="Stopped"/>'

Describe v2/faq/time based failback here.

How Do I Prevent a Resource from Failing Over During the Week but not on Weekends?

You need to use multiple instance_attributes sets with rules[24] with date_expressions[25].

An example of an IP address that is allowed to be moved only on weekends is listed below.

<primitive id="my_ip" provider="heartbeat" class="OCF" type="IPaddr">
   <instance_attributes id="my_ip:weekend_override" score="100">
     <rule id="my_ip:failover" boolean_op="and">
       <date_expression id="my_ip:days" operation="date_spec">
         <date_spec id="my_ip:days" weekdays="6-7"/>
       </date_expression>
     </rule>
     <attributes>
       <nvpair id="sat-sun-sticky" name="resource_stickiness" value="0"/>
     </attributes>
   </instance_attributes>
   <instance_attributes id="my_ip" score="10">
      <attributes>
        <nvpair id="mon-fri-sticky" name="resource_stickiness" value="INFINITY"/>
      </attributes>
   </instance_attributes>
</primitive>

Note the use of score to determin the order in which the sets are processed in. Sets with a higher score will be processed before sets with lower scores.

To further restrict the failover to the hours of 2am to 5am on the weekend, you could construct something like the resource below:

<primitive id="my_ip" provider="heartbeat" class="OCF" type="IPaddr">
   <instance_attributes id="my_ip:weekend_override" score="100">
     <rule id="my_ip:failover" boolean_op="and">
       <date_expression id="my_ip:days" operation="date_spec">
         <date_spec id="my_ip:days" weekdays="6-7"/>
       </date_expression>
       <date_expression id="my_ip:times" operation="date_spec">
         <date_spec id="my_ip:times" hours="2-5"/>
       </date_expression>
     </rule>
     <attributes>
       <nvpair id="sat-sun-sticky" name="resource_stickiness" value="0"/>
     </attributes>
   </instance_attributes>
   <instance_attributes id="my_ip" score="10">
      <attributes>
        <nvpair id="mon-fri-sticky" name="resource_stickiness" value="INFINITY"/>
      </attributes>
   </instance_attributes>
</primitive>

References