Providing Open Source High-Availability Software for Linux and other OSes since 1999.

Introduction

Using Linux-HA

Links

Common Searches

7th - 10th October 2008 Linux-Kongress in Hamburg will have several sessions on Linux-HA - see you there!

18 August 2008 Heartbeat release 2.1.4 is now out Download it

11 October 2007 NEW educational HA/DR Blog hosted by Alan Robertson

9 April 2007 Check out the Cool Heartbeat Screencasts: Installation, Intro to the GUI Part of the Heartbeat Education project

Last site update:
2008-09-08 15:22:33

printer friendly view

SFEX (Shared Disk File EXclusiveness Control Program)

Download

sfex-1.3.tar.gz (79.1 KB) [md5sum: 6bc563ad4a22d39b5f0fcf8726138ae8]

Basic Consept

SFEX is resource which control ownership of shared disk.
SFEX uses an special partition on the shared disk, and maintains the following data.
- "status" shows whether a disk is owned by somebody.
- "node" shows the node name to own the disk.
- "count" is used for judgment that owner node is up or down.
Typically, resources which use data partition on the shared disk(like PostgreSQL) make resource group with SFEX.
Resource of node which is holding ownership can access data partition.

attachment:sfex0.PNG

When node can get ownership?
- case1: Nobody has ownership.
- case2: Node can judge another node is down.

sequence diagram

start up process

SFEX can start on the node which has the highest score in cib.xml because more than one nodes do not access shared disk at the same time.

Node A

SFEX reads data from shared disk, and get "status". Usually "status" is "NO_OWNED" because nobody has owned shared disk.
Writes data that include node=Node A and status=OWNED.
Reads data again, and get "node=Node A".
Compareses it with my node name. If node name has not been changed, Node A get ownership!!
SFEX increments "count" on the shared disk by monitor processing of heartbeat. This processing means the update of ownership.

attachment:seq1_start.png

Heartbeat communication failuer

Node A

SFEX updates ownership by HB monitor processing.

Node B

When heartbeat communication fail, standby node(Node starts resources.
SFEX reads data on the sheard disk.
Waits a while. Wait time should be longer than sfex monitor interval. By this wait time, it waits for periodical update from Node A and confirms that Node A maintains ownership.
Reads data again.
Checks value of new "count". When the values of two "count" are different, it is able to think that Node A is up.
SFEX starts up process is stopped.

attachment:seq2_HBdown.png

Active Node failure

Node A

Node A is downed by failure.

Node B

This Node B start up in the same way as HB communication failure.

Waits for a while. It waits for periodical update from Node A but confirms that Node A does not it.
SFEX reads data again.
Checks value of new "count". The values of two "count" are SAME, it is able to think that Node A is DOWN.
Writes data that include node=Node B and status=OWNED.
Reads data again.
Compareses it with my node name. If node name has not been changed, Node B get ownership!!
Afterwards, other resources start.

attachment:seq3_NodeDown.png

Disk access on the same time

This is hardly generated. However for example, this case occurs when multiple nodes start up at the same time without heartbeat communication.

Node A / Node B

Writing to shared disk is serialized finally because writable area is "one". As a result, the node name written at the last time remains. In this example, Node B remains.

Read data again
Node A: value of "owner" is changed. this node does not get ownership. Node B: value of "owner" is name of Node B. Node B get ownnership!!

attachment:seq4_ST_access.png

sample cib.xml

 <cib admin_epoch="0" epoch="1" have_quorum="false" cib_feature_revision="1.3">
  <configuration>
    <crm_config>
      <cluster_property_set id="set01">
        <attributes>
          <nvpair id="symmetric-cluster"
            name="symmetric-cluster" value="true"/>
          <nvpair id="no-quorum-policy"
            name="no-quorum-policy" value="ignore"/>
          <nvpair id="stonith-enabled"
            name="stonith-enabled" value="false"/>
          <nvpair id="short-resource-names"
            name="short-resource-names" value="true"/>
          <nvpair id="is-managed-default"
            name="is-managed-default" value="true"/>
          <nvpair id="default-resource-stickiness"
            name="default-resource-stickiness" value="INFINITY"/>
          <nvpair id="stop-orphan-resources"
            name="stop-orphan-resources" value="true"/>
          <nvpair id="stop-orphan-actions"
            name="stop-orphan-actions" value="true"/>
          <nvpair id="remove-after-stop"
            name="remove-after-stop" value="false"/>
          <nvpair id="default-resource-failure-stickiness"
            name="default-resource-failure-stickiness" value="-INFINITY"/>
          <nvpair id="stonith-action"
            name="stonith-action" value="reboot"/>
          <nvpair id="default-action-timeout"
            name="default-action-timeout" value="120s"/>
          <nvpair id="dc-deadtime"
            name="dc-deadtime" value="10s"/>
          <nvpair id="cluster-recheck-interval"
            name="cluster-recheck-interval" value="0"/>
          <nvpair id="election-timeout"
            name="election-timeout" value="2min"/>
          <nvpair id="shutdown-escalation"
            name="shutdown-escalation" value="20min"/>
          <nvpair id="crmd-integration-timeout"
            name="crmd-integration-timeout" value="3min"/>
          <nvpair id="crmd-finalization-timeout"
            name="crmd-finalization-timeout" value="10min"/>
          <nvpair id="cluster-delay"
            name="cluster-delay" value="180s"/>
          <nvpair id="pe-error-series-max"
            name="pe-error-series-max" value="-1"/>
          <nvpair id="pe-warn-series-max"
            name="pe-warn-series-max" value="-1"/>
          <nvpair id="pe-input-series-max"
            name="pe-input-series-max" value="-1"/>
          <nvpair id="startup-fencing"
            name="startup-fencing" value="true"/>
        </attributes>
      </cluster_property_set>
    </crm_config>
    <nodes/>
    <resources>
      <group id="grpPostgreSQLDB">
        <primitive id="prmExPostgreSQLDB" class="ocf" type="sfex" provider="heartbeat">
          <operations>
            <op id="exPostgreSQLDB_start"
              name="start" timeout="180s" on_fail="fence"/>
            <op id="exPostgreSQLDB_monitor"
              name="monitor" interval="10s" timeout="60s" on_fail="fence"/>
            <op id="exPostgreSQLDB_stop"
              name="stop" timeout="60s" on_fail="fence"/>
          </operations>
          <instance_attributes id="atrExPostgreSQLDB">
            <attributes>
              <nvpair id="dskPostgreSQLDB"
                name="device" value="/dev/cciss/c1d0p1"/>
              <nvpair id="idxPostgreSQLDB"
                name="index" value="1"/>
              <nvpair id="cltPostgreSQLDB"
                name="collision_timeout" value="1"/>
              <nvpair id="lctPostgreSQLDB"
                name="lock_timeout" value="70"/>
              <nvpair id="mntPostgreSQLDB"
                name="monitor_interval" value="10"/>
              <nvpair id="fckPostgreSQLDB"
                name="fsck" value="/sbin/fsck -p /dev/cciss/c1d0p2"/>
              <nvpair id="fcmPostgreSQLDB"
                name="fsck_mode" value="check"/>
              <nvpair id="hltPostgreSQLDB"
                name="halt" value="/sbin/halt -f -n -p"/>
            </attributes>
          </instance_attributes>
        </primitive>
        <primitive id="prmFsPostgreSQLDB" class="ocf" type="Filesystem" provider="heartbeat">
          <operations>
            <op id="fsPostgreSQLDB_start"
              name="start" timeout="60s" on_fail="fence"/>
            <op id="fsPostgreSQLDB_monitor"
              name="monitor" interval="10s" timeout="60s" on_fail="fence"/>
            <op id="fsPostgreSQLDB_stop"
              name="stop" timeout="60s" on_fail="fence"/>
          </operations>
          <instance_attributes id="atrFsPostgreSQLDB">
            <attributes>
              <nvpair id="devPostgreSQLDB"
                name="device" value="/dev/cciss/c1d0p2"/>
              <nvpair id="dirPostgreSQLDB"
                name="directory" value="/mnt/shared-disk"/>
              <nvpair id="fstPostgreSQLDB"
                name="fstype" value="ext3"/>
            </attributes>
          </instance_attributes>
        </primitive>
        <primitive id="prmIpPostgreSQLDB" class="ocf" type="IPaddr" provider="heartbeat">
          <operations>
            <op id="ipPostgreSQLDB_start"
              name="start" timeout="60s" on_fail="fence"/>
            <op id="ipPostgreSQLDB_monitor"
              name="monitor" interval="10s" timeout="60s" on_fail="fence"/>
            <op id="ipPostgreSQLDB_stop"
              name="stop" timeout="60s" on_fail="fence"/>
            </operations>
          <instance_attributes id="atrIpPostgreSQLDB">
            <attributes>
              <!-- chenge ip address attribute -->
              <nvpair id="ipPostgreSQLDB" name="ip" value="aaa.bbb.ccc.ddd"/>
              <nvpair id="maskPostgreSQLDB" name="netmask" value="nn"/>
              <nvpair id="nicPostgreSQLDB" name="nic" value="bond0"/>
            </attributes>
          </instance_attributes>
        </primitive>
        <primitive id="prmApPostgreSQLDB" class="ocf" type="pgsql" provider="heartbeat">
          <operations>
            <op id="apPostgreSQLDB_start"
              name="start" timeout="60s" on_fail="fence"/>
            <op id="apPostgreSQLDB_monitor"
              name="monitor" interval="30s" timeout="60s" on_fail="fence"/>
            <op id="apPostgreSQLDB_stop"
              name="stop" timeout="60s" on_fail="fence"/>
          </operations>
          <instance_attributes id="atrApPostgreSQLDB">
            <attributes>
              <nvpair id="pgctl01"
                name="pgctl" value="/usr/local/pgsql/bin/pg_ctl"/>
              <nvpair id="psql01"
                name="psql" value="/usr/local/pgsql/bin/psql"/>
              <nvpair id="pgdata01"
                name="pgdata" value="/mnt/shared-disk/pgsql/data"/>
              <nvpair id="pgdba01"
                name="pgdba" value="postgres"/>
              <nvpair id="pgdb01"
                name="pgdb" value="template1"/>
              <nvpair id="logfile01"
                name="logfile" value="/var/log/pgsql.log"/>
            </attributes>
          </instance_attributes>
        </primitive>
      </group>
    </resources>
    <constraints>
      <rsc_location id="rlcPostgreSQLDB" rsc="grpPostgreSQLDB">
        <rule id="rulPostgreSQLDB_node01" score="200">
          <expression id="expPostgreSQLDB_node01"
            attribute="#uname" operation="eq" value="sfex01" />
        </rule>
        <rule id="rulPostgreSQLDB_node02" score="100">
          <expression id="expPostgreSQLDB_node02"
            attribute="#uname" operation="eq" value="sfex02"/>
        </rule>
      </rsc_location>
      <rsc_location id="ping1:disconn" rsc="grpPostgreSQLDB">
        <rule id="ping1:disconn:rule" score="-INFINITY" boolean_op="and">
          <expression id="ping1:disconn:expr:defined"
            attribute="default_ping_set" operation="defined"/>
          <expression id="ping1:disconn:expr:positive"
            attribute="default_ping_set" operation="lt" value="100"/>
        </rule>
      </rsc_location>
    </constraints>
  </configuration>
  <status/>
 </cib>

Release Notes

sfex -- Ver1.3
- 2008/03/07

Linux-HA site built by the Linux-HA crowd. HTML4 / CSS2