Linux-HA Logo

Contents

  1. Frequently Asked Questions for Heartbeat Version 2
    1. Why does crmadmin think my resource is not active?
    2. How Do I Write an OCF Compliant Resource Agent?
    3. Is my Init Script Compatible with Heartbeat?
    4. How do I monitor a resource for failure?
    5. What happens when monitor detects the resource down?
    6. Can I load a new cib.xml config in Heartbeat without restarting Heartbeat?
    7. Is it a good idea to keep the cib.xml in sync with the other nodes?
    8. What's the difference between all those timers/timeouts?
    9. How do I know Heartbeat is installed correctly?
    10. Why doesn't hb_standby do work in version 2?
    11. How do I replace a cluster node?
    12. See Also

Other FAQ entries included at the end of this page:

  1. ja/v2/faq/forced failover ja
  2. ja/v2/faq/manual recovery ja
  3. ja/v2/faq/pingd ja
  4. ja/v2/faq/resource too active ja
  5. ja/v2/faq/stop resource ja
  6. ja/v2/faq/time based failback ja
  7. v2/faq/cib changes detected
  8. v2/faq/forced failover
  9. v2/faq/manual recovery
  10. v2/faq/pingd
  11. v2/faq/resource too active
  12. v2/faq/stop resource
  13. v2/faq/time based failback

Frequently Asked Questions for Heartbeat Version 2

Why does crmadmin think my resource is not active?

Chances are you are trying to check a resource[1] that is part of a group[2].

Try prepending the id of the group to the resource's name. Eg. crmadmin -W group_id:rsc_id

Version 2.0.3 allows you to use the short name (provided the short_resource_names option is set).

How Do I Write an OCF Compliant Resource Agent?

The OCF RA spec can be found here:
http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD[3]

Is my Init Script Compatible with Heartbeat?

Init scripts are handled as LSBResourceAgent[4]s. Problems with them are quite common and the LSBResourceAgent[4] page contains a series of steps to help you determine if the script is usable or not.

How do I monitor a resource for failure?

One of the most requested Heartbeat[5] features was the ability for it to detect when a resource[1] failed (not just the whole node).

To support this, the CRM[6] also knows about monitor actions.

If you wish Heartbeat to make sure the resource is running, then you must specify one or more monitor actions in the operations section of the resource.

Each monitor action must specify an interval which tells Heartbeat how often it should check the resource's status.

To find out more, refer to section two of ClusterInformationBase/Actions[7] and the section on actions in the annotated DTD[8]

What happens when monitor detects the resource down?

The node will try to restart the resource[1], but if this fails, it will fail over to an other node.

A feature that allows failover after N failures in a given period of time is planned.

Can I load a new cib.xml config in Heartbeat without restarting Heartbeat?

Indeed you can. You will need the replace option of the cibadmin[9] tool.

Is it a good idea to keep the cib.xml in sync with the other nodes?

Absolutely. However the ClusterInformationBase[10] is supposed to do this automatically. If you find a situation where the contents on two active nodes is NOT in sync, please file a bug[11] immediately.

What's the difference between all those timers/timeouts?

How do I know Heartbeat is installed correctly?

You can check your Heartbeat installation by running /usr/lib/heartbeat/BasicSanityCheck

Why doesn't hb_standby do work in version 2?

In VersionTwo[14] you'll need to use crm_standby. See v2/AdminTools[15] for more help on the available commands.

How Do I Force a Resource to be Migrated After a Failure?

In Versions prior to 2.0.5 there is no way to do this.

Subsequent versions can use the resource-failure-stickiness property of a resource (or the global default: default-resource-failure-stickiness) to control when failover happens.

Example

In a cluster where:

my_rsc can fail up to ten times on nodeA before being moved to nodeB.

(nodeA score - nodeB score + stickiness) / abs(failure stickiness)

==> (1500 - 1000 + 500) / abs(-100)

==> 1000 / 100

==> Answer: 10

However if default-resource-failure-stickiness was set to -1001 or less, it would be moved immediately.

(nodeA score - nodeB score + stickiness) / abs(failure stickiness)

==> (1500 - 1000 + 500) / abs(-1001)

==> 1000 / 1001

==> Answer: 0.999

NOTE: Failure stickiness should be a negative value.
NOTE: The rules for +/-INFINITY apply...

 INFINITY +/- -INFINITY : -INFINITY
 INFINITY +/-  int      :  INFINITY
-INFINITY +/-  int      : -INFINITY

Multiple Failures

If the combined score for my_rsc on a node is less than zero, it will never be able to run there again until the failure count is reset.

The current failure count (for a given resource and node) is multiplied by the resource's failure stickiness to produce a failover score. When the failover score exceeds the regular preference to a given node, the node will be excluded from running the resource again (until the failure count is reset).

In the first example above (where default-resource-failure_stickiness is -100) my_rsc can fail up to 15 times on nodeA before Heartbeat will no longer consider running it there. Likewise it can fail up to 10 times on nodeB before that node will no longer be consider either.

In the second example (where default-resource-failure-stickiness is less than -500), these values drop to 3 and 2 failures respectively.

Resetting Failure Counts

To reset the failure count for my_rsc on nodeA, the following command can be used:

To query the current failure count, use:

Why the Failure Count is not Automatically Reset When a Resource is Moved

The failure count is not automatically reset when a resource is moved to prevent the resource from bouncing between two or more "bad" nodes.

How this works out with groups

When you use groups or other things with mandatory colocation constraints, it gets more complicated. If you take everything which you force to be colocated on the same machine, then all those things will be treated just as though they were in a group together. So, keep that in mind in the explanation below.

For a group with n resources, the resource-stickiness of the group of n resources is the sum of all the stickinesses in the group. Unless you've overridden the default for the resources in the group, that would be n times the default-resource-stickiness, and it is the stickiness of the group as a whole which is used, rather than the stickiness of any member of the group. The resource-failure-stickiness value for a group is computed in a similar way.

In addition, if you want to give a different resource-stickiness or resource-failure-stickiness to every member in the group, you can give this attribute to the group itself. However, it will effectively be multiplied by n in the process of summing up all the stickiness values of the group members, because this value is inherited by each primitive resource in the group, and then subjected to the summing process described above.

Things to keep in mind

What you cannot do

You cannot force a group to fail over if any given resource in the group fails 'n' times. Failure counts for groups are cumulative. If you set it up so that 3 failures will cause a failover, then 3 failures of any resource in the group will cause a failover, not 3 for a particular resource. If you have a web server, an IP address and a mount point, then it will fail over with 3 web server failures, or one failure of each resource in the group.

A worked example involving a Group

Problem Definition

The outline of the requirements to be met are:

Problem Solution

To be supplied

How do I Manually Recover a Resource?

Why is my Resource Listed as Unmanaged?

The authoritative source for information about what resources are active and what state they're in is the LRM. So while you could change the contents of the CIB directly, these changes are likely to be overwritten with data from the LRM at some point in the future.

Once you have manually stopped and otherwise cleaned up the resource, you should use the -C option for crm_resource to remove the resource from the LRM. More information on its usage is available with crm_resource --help. The general form is crm_resource -C -r my_resource_name. By default it will operate on all cluster nodes unless you specify -H some_host as well.

Since 2.0.5 the command will automatically instruct the cluster to refresh itself with the latest details from the LRM. Prior to that, users should "wait a bit" and run crm_resource -R.

Prior to 2.0.6 (which has not yet been released as of writing this) users had to be careful choosing the right value to use with -r. This affects resources in groups and is due to a bad choice earlier in the CRM's history and due to backwards compatibility we are stuck with it being the default.

The following command will list for you the internal names used by the cluster for its resources:
cibadmin -Ql -o status | grep "lrm_resource " | awk '{print $2}' | sort -u

Look for the one that matches your resource and use that value when calling crm_resource.

How Do I Monitor Connectivity in Version 2

pingd, An ipfail Replacement

pingd is a replacement for ipfail that like the rest of version 2 allows for connectivity (and resource placement based on relative connectivity) to work in clusters with any number of nodes.

The role of pingd is to detect changes to a node's connectivity and ensure that updates of this information to the CIB[17] occur (at least effectively) simultaneously.

To locate your resources on the node(s) with the greatest connectivity, an admin needs to use the information placed in the CIB by pingd. This is achieved with the creation of resource location[18] constraints that reference the attribute created by pingd. See "Using pingd Output in Location Constraints" below.

Configuration Methods

There are two options for configuring pingd

  1. The first is by adding a respawn directive to ha.cf eg:

    • respawn root /usr/lib/heartbeat/pingd -m 100 -d 5s
      See "pingd Usage Information" below for details on the meaning of the options passed to pingd.

  2. The other option is to create a clone[19] using the pingd OCFResourceAgent[20]. Since RA's are started as root, you need to add a line like "apiauth pingd uid=root" to your ha.cf[12] or - even better - add the parameter user=hacluster to your RA configuration to use the implicit apiauth directive like the respawn method does. With version 2.0.6 you also have to add parameter pidfile which points to somewhere the chosen user has write permission (e.g. /tmp/pingd-default).

Both methods need the ping nodes listed in your ha.cf[12] file. The host_list parameter of the RA can only use a subset of those, not some other hosts.

Both methods also require the addition of one-or-more colocation constraints to the CIB. See "Using pingd Output in Location Constraints" below.

The advantage of using the resource agent is that you can:

An equivalent resource for the respawn directive above would be:

<clone id="pingd-clone">
  <meta_attributes id="pingd-clone-ma">
    <attributes>
      <nvpair id="pingd-clone-1" name="globally_unique" value="false"/>
    </attributes>
  </meta_attributes>
  <primitive id="pingd-child" provider="heartbeat" class="ocf" type="pingd">
    <operations>
      <op id="pingd-child-monitor" name="monitor" interval="20s" timeout="60s" prereq="nothing"/>
      <op id="pingd-child-start" name="start" prereq="nothing"/>
    </operations>
    <instance_attributes id="pingd_inst_attr">
      <attributes>
         <nvpair id="pingd-1" name="dampen" value="5s"/>
         <nvpair id="pingd-2" name="multiplier" value="100"/>
      </attributes>
    </instance_attributes>
  </primitive>
</clone>

NOTE: Changing the attribute's location in the CIB, while possible, is discouraged. This is because you may end up with multiple copies of the attribute for each node... causing the cluster to behave differently than expected.

pingd Usage Information

usage: pingd [-V?p:a:d:s:S:h:Dm:]
        --help (-?)                     This text
        --daemonize (-D)                Run in daemon mode
        --pid-file (-p) <filename>      File in which to store the process' PID
                                        * Default=/tmp/pingd.pid
        --attr-name (-a) <string>       Name of the node attribute to set
                                        * Default=pingd
        --attr-set (-s) <string>        Name of the set in which to set the attribute
                                        * Default=cib-bootstrap-options
        --attr-section (-S) <string>    Which part of the CIB to put the attribute in
                                        * Default=status
        --ping-host (-h) <single_host_name> Monitor a subset of the ping nodes listed in ha.cf
                                            (can be specified multiple times)
        --attr-dampen (-d) <integer>        How long to wait for no further changes to occur before
                                            updating the CIB with a changed attribute
        --value-multiplier (-m) <integer>   For every connected node, add <integer> to the value set in the CIB
                                            * Default=1

Using pingd Output in Location Constraints

Example pingd Configuration

Example CIB Contents

Node

Connected Ping Nodes

default_ping_set Value

c001n01

5

500

c001n02

4

400

c001n03

5

500

Example Constraint

<rsc_location id="my_resource_connected" rsc="my_resource">
    <rule id="my_resource_connected_rule" score_attribute="default_ping_set">
       <expression id="my_resource_connected_expr_defined" attribute="default_ping_set" operation="defined"/>
    </rule>
</rsc_location>

The above constraint:

Node

Connected Ping Nodes

default_ping_set Value

Combined Score

c001n01

5

500

500

c001n02

4

400

400

c001n03

5

500

500

If we also had the following constraint:

<rsc_location id="my_resource_preferred" rsc="my_resource">
    <rule id="my_resource_prefer_c001n01" score="100">
       <expression id="my_resource_prefer_c001n01_expr" attribute="#uname" operation="eq" value="c001n01"/>
    </rule>
    <rule id="my_resource_prefer_c001n02" score="200">
       <expression id="my_resource_prefer_c001n02_expr" attribute="#uname" operation="eq" value="c001n02"/>
    </rule>
    <rule id="my_resource_prefer_c001n03" score="300">
       <expression id="my_resource_prefer_c001n03_expr" attribute="#uname" operation="eq" value="c001n03"/>
    </rule>
    <rule id="my_resource_never" score="-INFINITY" boolean_op="or">
       <expression id="my_resource_never_c001n04_expr" attribute="#uname" operation="eq" value="c001n04"/>
       <expression id="my_resource_never_c001n05_expr" attribute="#uname" operation="eq" value="c001n05"/>
    </rule>
</rsc_location>

Then the updated scores for running the resource would be:

Node

Connected Ping Nodes

default_ping_set Value

Combined Score

c001n01

5

500

600

c001n02

4

400

600

c001n03

5

500

800

At this point, if the resource was not running or v2/dtd1.0/annotated#default resource stickiness[21] was set to zero, then the resource would be started on c001n03 with c001n01 and c001n02 equally preferred as a backup.

However if the resource was running on c001n02 and resource_stickiness was set to 1000, then the updated scores would be:

Node

Connected Ping Nodes

default_ping_set Value

Combined Score

c001n01

5

500

600

c001n02

4

400

1600

c001n03

5

500

800

and the resource would be left running on c001n02.

Alternatively, if resource_stickiness was set to 100, then the scores would look like this:

Node

Connected Ping Nodes

default_ping_set Value

Combined Score

c001n01

5

500

600

c001n02

4

400

700

c001n03

5

500

800

and the resource would be moved to c001n03.

This should also adequately demonstrate the importance of correctly setting:

Quickstart - Only Run my_resource on Nodes with Access to at Least One Ping Node

Add this to ha.cf

Add this constraint to the CIB:

It is sometimes desirable to shut a particular service down if ping connectivity is lost. This rule will prohibit the service from running anywhere that there is no ping connectivity to the outside world, and all nodes with some connectivity are treated as the same, regardless of how many ping nodes are accessible.

<rsc_location id="my_resource:connected" rsc="my_resource">
  <rule id="my_resource:connected:rule" score="-INFINITY" boolean_op="or">
    <expression id="my_resource:connected:expr:undefined"
      attribute="pingd" operation="not_defined"/>
    <expression id="my_resource:connected:expr:zero"
      attribute="pingd" operation="lte" value="0"/>
  </rule>
</rsc_location>

Of course, if you have configured the pingd[16] daemon to set some attribute name besides its default (pingd), then you need to change the name of the attribute above from pingd to whatever name you have configured the pingd[16] daemon to use.

Attention: Note that this will stop the resource everywhere if the pinged node(s) indeed go down or heartbeat loses connectivity to them (firewalls et cetera). Consider using the wiki:CIB/Idioms/PingdAttrAsScore[22] instead, which instead expresses a positive preference for the node with the best connectivity.

The respawn rule for pingd from above can be used, or virtually any method of starting pingd with any non-zero --value-multiplier factor. If you have more than one ping node, you can run the resource, if not you can't. It's that simple.

Quickstart - Run my resource on the node with the best connectivity

This is probably one of the better ways to use pingd. In this method, the pingd attribute value becomes the score for the rule. So, the --value-multiplier you set will depend heavily on the scores you give other criteria. This rule will not stop a resource completely if all nodes lose connectivity to the outside world.

It is often desirable to allow the value of the attribute that pingd[16] sets directly as a the score for a particular rule.

If you set the pingd[16] scaling factor to 100, then having access to one node is worth 100, 2 nodes is worth 200, and so on.

This way, if all else is equal, the node with the highest ping connectivity will be selected. If two or more eligible nodes have the same score, then they will be given equal weight according to the rule below.

<rsc_location id="my_resource:connected" rsc="my_resource">
  <rule id="my_resource:connected:rule" score_attribute="pingd" >
    <expression id="my_resource:connected:expr:defined"
      attribute="pingd" operation="defined"/>
  </rule>
</rsc_location>

Of course, if you have configured the pingd[16] daemon to set some attribute name besides its default (pingd), then you need to change the name of the score_attribute above from pingd to whatever attribute you have configured the pingd[16] daemon to use.

Resource X is (potentially) active on 2 nodes.

Heartbeat will try and determine what resources are active on a machine when it starts. To do this, it sends what we call a probe which uses the monitor operation of your ResourceAgent[23].

There are two common reasons for seeing this message:

  1. Your resource really is active on more than one node
    • Check you are not starting it on boot
    • Did Heartbeat suffer an internal failure?

      If so, please check the ReportingProblems[11] page and report it

  2. Your resource doesn't implement the monitor operation correctly
    • Make sure your ResourceAgent[23] conforms to the OCF-spec

      • return 0 only for active resource instances
      • return 7 for inactive resource instances
      • return anything else for failed resource instances

How do I Make the Cluster Stop Resource X?

As of version 2.0.5 it is now possible to ask the cluster to stop a resource and leave it stopped.

To do this, you need to set an instance attribute called target_role for the resource.

To do this, your resource definition should look like something this:

 <resources>
   <group id="cds_grp">
     <primitive class="ocf" id="cds_res" provider="heartbeat" type="cds">
       <instance_attributes id="cds-defaults">
          <attributes>
             <nvpair id="cds-role" name="target_role" value="Stopped"/>
          </attributes>
       </instance_attributes>
     </primitive>
   </group>
 </resources>

Once you have done whatever you needed to do and are ready to start it again, the following command will tell the cluster to start it.

cibadmin -M -o resources -X '<nvpair id="cds-role" value="Started"/>'

To later cause the resource to be stopped again, you would run:

cibadmin -M -o resources -X '<nvpair id="cds-role" value="Stopped"/>'

Describe v2/faq/time based failback here.

How Do I Prevent a Resource from Failing Over During the Week but not on Weekends?

You need to use multiple instance_attributes sets with rules[24] with date_expressions[25].

An example of an IP address that is allowed to be moved only on weekends is listed below.

<primitive id="my_ip" provider="heartbeat" class="OCF" type="IPaddr">
   <instance_attributes id="my_ip:weekend_override" score="100">
     <rule id="my_ip:failover" boolean_op="and">
       <date_expression id="my_ip:days" operation="date_spec">
         <date_spec id="my_ip:days" weekdays="6-7"/>
       </date_expression>
     </rule>
     <attributes>
       <nvpair id="sat-sun-sticky" name="resource_stickiness" value="0"/>
     </attributes>
   </instance_attributes>
   <instance_attributes id="my_ip" score="10">
      <attributes>
        <nvpair id="mon-fri-sticky" name="resource_stickiness" value="INFINITY"/>
      </attributes>
   </instance_attributes>
</primitive>

Note the use of score to determin the order in which the sets are processed in. Sets with a higher score will be processed before sets with lower scores.

To further restrict the failover to the hours of 2am to 5am on the weekend, you could construct something like the resource below:

<primitive id="my_ip" provider="heartbeat" class="OCF" type="IPaddr">
   <instance_attributes id="my_ip:weekend_override" score="100">
     <rule id="my_ip:failover" boolean_op="and">
       <date_expression id="my_ip:days" operation="date_spec">
         <date_spec id="my_ip:days" weekdays="6-7"/>
       </date_expression>
       <date_expression id="my_ip:times" operation="date_spec">
         <date_spec id="my_ip:times" hours="2-5"/>
       </date_expression>
     </rule>
     <attributes>
       <nvpair id="sat-sun-sticky" name="resource_stickiness" value="0"/>
     </attributes>
   </instance_attributes>
   <instance_attributes id="my_ip" score="10">
      <attributes>
        <nvpair id="mon-fri-sticky" name="resource_stickiness" value="INFINITY"/>
      </attributes>
   </instance_attributes>
</primitive>

How do I replace a cluster node?

Each cluster node has an UUID[26] which is usually created and assigned to the node from heartbeat on at the first start of the cluster framework. If you start heartbeat on the replacement node (even if it has the same host name and IP address like the original node) it will have a different UUID than the orginal node. The result is the replacement node will not show up in your cluster. Some interesting facts:

As usual there are more ways to reach the goal:

  1. you have the copy of hb_uuid of the node which should be replaced

  2. you do not have the copy of hb_uuid but you have a heartbeat release which can create a binary UUID (crm_uuid -w) from the ASCII UUID

  3. you assign the replaced node a new UUID

Preconditions:

Here are the steps replacing node2 of a cluster with 2 nodes (node1 and node2) for the different cases:

NOTE: heartbeat release 2.0.8 crm_uuid does not have the functionality creating the binary UUID (-w option). The following steps will work beginning with release 2.1.0.

  1. on node1 get the UUID of the orginal node2 in ASCII format (you can retrieve it from the CIB <nodes> section or from the file /var/lib/heartbeat/hostcache mentioned earlier). Here the <nodes> section of our example which tells us node2 had UUID a69b64a2-4de8-4d4a-b4ba-8107136eec4b.

   <nodes>
       <node id="a0e71041-8cbd-449a-a04b-0da3da3d82a6" uname="node1" type="normal">
         <instance_attributes id="nodes-a0e71041-8cbd-449a-a04b-0da3da3d82a6">
           <attributes>
             <nvpair id="standby-a0e71041-8cbd-449a-a04b-0da3da3d82a6" name="standby" value="off"/>
           </attributes>
         </instance_attributes>
       </node>
       <node id="a69b64a2-4de8-4d4a-b4ba-8107136eec4b" uname="node2" type="normal">
         <instance_attributes id="nodes-a69b64a2-4de8-4d4a-b4ba-8107136eec4b">
           <attributes>
             <nvpair id="standby-a69b64a2-4de8-4d4a-b4ba-8107136eec4b" name="standby" value="on"/>
           </attributes>
         </instance_attributes>
       </node>
   <nodes/>
  1. on node2 use crm_uuid -w to create the binary UUID of node2.

ATTENTION: make sure you do this on the right node because it will overwrite hb_uuid on the node where you call it

node2:/tmp# crm_uuid -w a69b64a2-4de8-4d4a-b4ba-8107136eec4b
  1. restart heartbeat on node2 - finished

  2. make sure heartbeat is automatically started after a system reboot

In case heartbeat was started on the replacement node2 before you could do the procedure explained above, heartbeat assigned a new UUID to node2. node2 will be recognized as different from the original node2. It will not be integrated into the cluster, even if hostname and IP address are the same as the original node. Perform following steps to recover from this situation:

  1. stop heartbeat on node2

  2. remove node2 from the cluster (see below)

  3. now create hb_uuid on node2 as explained above

  4. you assign the replaced node a new UUID
  5. remove node2 from the cluster (see below)

  6. on all nodes in the cluster delete /var/lib/heartbeat/delhostcache

  7. now run on any node attached to to cluster /var/lib/heartbeat/hb_addnode node2

  8. on node2 start heartbeat - finished

  9. in case you deactivated pingd activate it again and make sure heartbeat is started automatically when rebooting

  10. removing a node from the cluster

Note: heartbeat release 2.0.8 and earlier had a bug with pingd. The pingd CIB attribute value is reset to 0 when removing a cluster node with hb_delnode (even if the removed cluster node was not part of the configured ping group). Depending on your cluster configuration it may have unwanted effects. It is adviced to set the cluster into unmanaged mode during the replacement procedure to avoid unwanted resource stops/starts.

  1. in case pingd is used, stop it

  2. on any node attached to the cluster run /usr/lib/heartbeat/hb_delnode node2. In case node2 is already attached to the cluaster and it says "can't delete active node" or something similar then shutdown heartbeat on node2 and repeat the step again

  3. verify on the node where you run 'hb_delnode if node2 was removed from the cluster by checking /var/lib/heartbeat/hostache - there should be no entry for node2`.

    • in case there is still an entry left repeat step 2
  4. in case pingd was used, restart it

  5. if the removed cluster node shall not be readded with the same UUID again delete /var/lib/heartbeat/delhostcache on all cluster nodes

See Also

Heartbeat FAQ[28]


References

[1]http://www.linux-ha.org/resource
[2]http://www.linux-ha.org/ClusterInformationBase/ResourceGroups
[3]http://www.opencf.org/cgi-bin/viewcvs.cgi/specs/ra/resource-agent-api.txt?rev=HEAD
[4]http://www.linux-ha.org/LSBResourceAgent
[5]http://www.linux-ha.org/Heartbeat
[6]http://www.linux-ha.org/ClusterResourceManager
[7]http://www.linux-ha.org/ClusterInformationBase/Actions
[8]http://www.linux-ha.org/ClusterResourceManager/DTD1.0/Annotated#action
[9]http://www.linux-ha.org/v2/AdminTools/cibadmin
[10]http://www.linux-ha.org/ClusterInformationBase
[11]http://www.linux-ha.org/ReportingProblems
[12]http://www.linux-ha.org/ha.cf
[13]http://www.linux-ha.org/PingNode
[14]http://www.linux-ha.org/VersionTwo
[15]http://www.linux-ha.org/v2/AdminTools
[16]http://www.linux-ha.org/pingd
[17]http://www.linux-ha.org/CIB
[18]http://www.linux-ha.org/v2/dtd1.0/annotated#rsc%2Blocation
[19]http://www.linux-ha.org/v2/Concepts/Clones
[20]http://www.linux-ha.org/OCFResourceAgent
[21]http://www.linux-ha.org/v2/dtd1.0/annotated#default%2Bresource%2Bstickiness
[22]http://www.linux-ha.org/CIB/Idioms/PingdAttrAsScore
[23]http://www.linux-ha.org/ResourceAgent
[24]http://www.linux-ha.org/v2/dtd1.0/annotated#rule
[25]http://www.linux-ha.org/v2/dtd1.0/annotated#date_expression
[26]http://en.wikipedia.org/wiki/UUID
[27]http://www.linux-ha.org/AutojoinDirective
[28]http://www.linux-ha.org/FAQ


This information provided courtesy of the Linux-HA project at http://linux-ha.org/