RHCS Red Hat Cluster Suite reminder
id : ys6jl02d6n
category : computer
blog : unixlinux
created : 03/03/10 - 14:25:58

  • This example is based on a two nodes cluster :

  • Naming conventions for the example :
    • rhcs1 : node 1 / (x.x.x.1) (eth1)
    • rhcs2 : node 2 / (x.x.x.2) (eth1)
    • floating address : x.x.x.3
    • gateway address : x.x.x.254
    • a shared disk used for quorum disk (shared between nodes) : /dev/data/qdisk

  • Monitored ip, services, disk
    • disk on cluster mountpoints (shared between nodes) :
      • /dev/data/disk1
      • /dev/other_data/disk2
    • a service :
      • httpd for example
    • an ip :
      • x.x.x.3

  • Fences devices
    • ipmi card : x.x.x.20 on rhcs1 / login : admin / password : password
    • ipmi card : x.x.x.21 on rhcs2 / login : admin / password : password
    • Theses can be apc or ilo/drac card

Installation
  • First of all install RHCS software :
    • rhcs1 :
root@rhcs1# yum groupinstall "Clustering"

    • rhcs2 :
root@rhcs2# yum groupinstall "Clustering"


Configuration
Floting address
  • On active node enable your floating adress :
root@rhcs1# ip addr add x.x.x.3 dev eth1


Services
  • Enable usefull services at startup :
    • rhcs1 :
root@rhcs1# chkconfig --level 2345 cman on
root@rhcs1# chkconfig --level 2345 rgmanager on
root@rhcs1# chkconfig --level 2345 qdiskd on
root@rhcs1# chkconfig --level 2345 fenced on
root@rhcs1# chkconfig --level 2345 modclusterd on

    • rhcs2 :
root@rhcs2# chkconfig --level 2345 cman on
root@rhcs2# chkconfig --level 2345 rgmanager on
root@rhcs2# chkconfig --level 2345 qdiskd on
root@rhcs2# chkconfig --level 2345 fenced on
root@rhcs2# chkconfig --level 2345 modclusterd on


Quorom disk
  • Setup your quorom disk :
root@rhcs1#mkqdisk -c /dev/data/qdisk -l quorumdisk


/etc/hosts file
  • Edit your /etc/hosts file on each node :
127.0.0.1 localhost.localdomain localhost
x.x.x.1   rhcs1 rhcs1.chaosisme.com
x.x.x.2   rhcs2 rhcs2.chaosisme.com
#Floating ip
x.x.x.3   rhcs  rhcs.chaosisme.com


cluster.conf
  • Edit your /etc/cluster/cluster.conf on each node :
<?xml version="1.0"?> 
<cluster config_version="1" name="rhcs" alias="rhcs.chaosisme.com">                                       
 <quorumd interval="3" label="quorumdisk" min_score="1" tko="9" votes="1">                         
  <heuristic program="ping -c1 -t1 x.x.x.254" score="1"/>                            
 </quorumd>              
                                        
 <fence_daemon clean_start="0" post_fail_delay="0" post_join_delay="3"/>                           
 <clusternodes>
  
  <clusternode name="rhcs1.chaosisme.com" votes="1" nodeid="1">                             
   <fence>
    <method name="1">                                                         
     <device name="fence_rhcs1"/>                                      
    </method>
   </fence>
  </clusternode>

  <clusternode name="rhcs2.chaosisme.com" votes="1" nodeid="2">                             
   <fence>                                                                           
    <method name="1">                                                         
     <device name="fence_rhcs2"/>                                      
    </method>                                                                 
   </fence>                                                                          
  </clusternode>
 </clusternodes>                                                                                   
              
 <cman two_node="0" expected_votes="3"/>                                                           
        
 <fencedevices>                                                                                    
  <fencedevice agent="fence_ipmilan" ipaddr="x.x.x.20" login="admin" name="fence_rhcs1" passwd="password"/>
  <fencedevice agent="fence_ipmilan" ipaddr="x.x.x.21" login="admin" name="fence_rhcs2" passwd="password"/> 
 </fencedevices> 
                                
 <rm>
                    
  <failoverdomains>
   <failoverdomain name="fd-rhcs.chaosisme.com" ordered="1" restricted="1">          
    <failoverdomainnode name="rhcs1.chaosisme.com" priority="1"/>             
    <failoverdomainnode name="rhcs2.chaosisme.com" priority="1"/>             
   </failoverdomain>
  </failoverdomains>
                        
  <resources>
   <fs device="/dev/data/disk1" force_unmount="1" fstype="ext3" mountpoint="/mnt/disk1" name="rhcs-disk1" options="noatime"/>
   <fs device="/dev/other_data/disk2" force_unmount="1" fstype="ext3" mountpoint="/mnt/disk2" name="rhcs-disk2" options="noatime"/>
   <ip address="x.x.x.3" monitor_link="1"/>                                          
   <script file="/etc/init.d/httpd" name="httpd"/>                                   
  </resources>

  <service autostart="1" domain="fd-rhcs.chaosisme.com" exclusive="1" name="rhcs.chaosisme.com" max_restarts="0" recovery="restart" restart_expire_time="0">
   <fs ref="rhcs-disk1"/>                                                            
   <fs ref="rhcs-disk2"/>                                                            
   <ip ref="x.x.x.3"/>
   <script ref="httpd"/>                                                             
  </service>

 </rm>

</cluster>


Heuristic
    • In this case we can use an heuristic ping to monitor gateway, this is not really a good choice, if one ping fail, cluster will go on passive node.
    • You can use this script for heursitic (it will success if a percentage of ping is successfull)(/scripts/heuristic.sh) :
.01#!/bin/bash
.02
.03PING_BIN="/bin/ping"
.04PING_OPT="-q -c1 -t1"
.05
.06if [ $# != 3 ]; then
.07        echo "Usage : heuristic.sh ip_to_test time score"
.08        exit 1;
.09fi
.10
.11IP=$1
.12TIME=$2
.13SCORE=$3
.14NOTE=0
.15
.16for i in `seq 1 ${TIME}`; do
.17        ${PING_BIN} ${PING_OPT} ${IP}
.18        TEST=$?
.19        if [[ ${TEST} -eq 0 ]]; then
.20                NOTE=`expr ${NOTE} + 1`
.21        fi
.22done
.23
.24if [[ ${NOTE} -ge ${SCORE} ]]; then
.25        exit 0
.26else
.27        exit 1
.28fi

    • You can use this heuristic.sh script with 10 for time and 5 for score, script will success if 5 of this 10 pings success, modify your cluster.conf file :
  <quorumd interval="3" label="quorumdisk" min_score="1" tko="9" votes="1">
    <heuristic program="/scripts/heuristic.sh x.x.x.254 10 5" score="1"/>
  </quorumd>


Fence devices
  • ON CISCO SWICTHS DISABLE PORT AUTO-NEGOTIATION
  • DISABLE SELINUX ON EACH NODES

  • You can use APCS for fences devices (if your servers are double alimented)
    • APC1 (x.x.x.20) (login : admin / password : password)
      • power1 of rhcs1 plugged (port 1)
      • power1 of rhcs2 plugged (port 2)
    • APC2 (x.x.x.21 (login : admin / password : password)
      • power2 of rhcs1 plugged (port 1)
      • power2 of rhcs2 plugged (port 2)

  • Modify your cluster.conf
 <clusternode name="rhcs1.chaosisme.com" votes="1" nodeid="1">
  <fence>
    <method name="1">
      <device name="APC1" option="off" switch="1" port="1"/>
      <device name="ACP2" option="off" switch="1" port="1"/>
      <device name="ACP1" option="on" switch="1" port="1"/>
      <device name="APC2" option="on" switch="1" port="1"/>
    </method>
  </fence>
 </clusternode>
 <clusternode name="rhcs2.chaosisme.com" votes="1" nodeid="2">
   <fence>
    <method name="1">
      <device name="APC1" option="off" switch="1" port="2"/>
      <device name="APC2" option="off" switch="1" port="2"/>
      <device name="APC1" option="on" switch="1" port="2"/>
      <device name="APC2" option="on" switch="1" port="2"/>
    </method>
  </fence>
 </clusternode>

<fencedevices>
  <fencedevice agent="fence_apc" ipaddr="x.x.x.20" login="admin" name="APC1" passwd="password"/>
  <fencedevice agent="fence_apc" ipaddr="x.x.x.21" login="admin" name="APC2" passwd="password"/>
</fencedevices>


Logging
  • It can be usefull to log all cluster message to a specific log file :
    • Edit your /etc/cluster/cluster.conf'
    • Edit the ressource manager, and add :
      • log_level.
      • log_facility.
 <rm log_level="7" log_facility="local4">


  • Modify your syslog configuration and add :
local4.*                                                /var/log/cluster.log


  • Restart syslog :
root@rhcs1# service syslog restart


Using cluster
    • Halt the two nodes and restart one node, it'll automaticly boot the second node.

clustat
  • Use clustat to monitor cluster status
    • rgmanager must to be started on the two nodes
    • quoromdisk must be online
root@rhcs1# clustat
Cluster Status for rhcs @ Tue Mar  2 16:43:20 2010
Member Status: Quorate

 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 rhcs1.chaosisme.com                                             1 Online, Local, rgmanager
 rhcs2.chaosisme.com                                             2 Online, rgmanager
 /dev/mapper/data--qdisk                                         0 Online, Quorum Disk

 Service Name                                                     Owner (Last)                                                     State
 ------- ----                                                     ----- ------                                                     -----
 service:rhcs.chaosisme.com                                      rhcs1.chaoisme.com                                                started


  • Use clustat -i to monitor each x second, like watch command :
    • clustat -i 5

clusvcadm
  • Use clusvcadm to administrate cluster :
    • restart group :
root@rhcs1# clusvcadm -R rhcs.chaosisme.com

    • switch to node 2 (relocate group to node 2) :
root@rhcs1# clusvcadm -r rhcs.chaosisme.com -m rhcs2

    • switch to node 1 (relocate group to node 1) :
root@rhcs1# clusvcadmin -r rhcs.chaosime.com -m rhcs1

    • disable group :
root@rhcs1# clusvcadm -d rhcs.chaosisme.com

    • enable group :
root@rhcs1# clusvcadm -e rhcs.chaosisme.com


cman tool
  • When upgrading your /etc/cluster.conf NEVER forget to modifiy config version

root@rhcs1# cman_tool -r /etc/cluster/cluster.conf


  • You can use cman_tool to have serveral informations on nodes, use man page to go deeper.

fence_node
  • Use fence_node command to test your fence device
root@rhcs1# fence_node rhcs1

Stopping cluster service on one node
If you want to stop service on one node :
# service rgmanager stop
# service clvmd stop
# service cman stop

Do not want cluster start on bootup :
# chkconfig --level 2345 rgmanager off
# chkconfig --level 2345 clvmd off
# chkconfig --level 2345 cman stop