Voting Disks in Oracle 11gR2 RAC
Voting disks are important component of
Oracle Clusterware. Clusterware uses voting disk to determine which nodes are
members of the cluster. After ASM is
introduced to store these files, these are called as VOTING FILES.
Primary function of voting disks is to
manage node membership and prevent SPLITBRAIN Syndrome in which 2 or more
instances attempt to control the RAC database.
Ø These files can be stored either in ASM or on shared storage.
Ø If it is stored in ASM, no need to configure manually as the files
will be created depending on the redundancy in ASM.
Ø In shared storage system, we need to manually configure these files
with redundancy setup for high availability.
Ø We must have odd number of disks.
Ø Oracle recommends minimum of 3
and maximum of 5. In 10g, Clusterware can
supports 32 voting disks but in 11gR2 supports 15 voting disks.
Ø A node must be able to access more than half of the voting disks at
any time. For eg, if you have 5 voting
disks, a node must be able access atleast 3 of the voting disks. If it cannot
access the minimum of voting disks, then it is evicted/removed from the
cluster.
Ø All nodes in the RAC cluster register their heartbeat information in
the voting disks/files. RAC heartbeat is
the polling mechanism that is sent over the cluster interconnect to ensure all
RAC
nodes are available.
How
Voting Happens
The CKPT process updates the control
file every 3 seconds in an operation known as heartbeat.
CKPT writes to a single block that is
local to the node/each instance and intra instance coordination is not
required. This block is called checkpoint
progress record.
All members of the cluster attempt to
lock on the controlfile record for updating.
The instance which obtains the locks
tallies the votes from all members.
Then, the group membership must conform to the decided(voted) membership before allowing GCS/GES to proceed for
reconfiguration. The control file record is then stored in
the same block as the heartbeat in the controlfile
checkpoint progress record.
What is NETWORK and DISK HEARTBEAT and how it registers in VOTING
DISKS/FILES
1.
All nodes in the RAC cluster
register their heartbeat information in the voting disks/files. RAC heartbeat is the polling mechanism that
is sent over the cluster interconnect to ensure all RAC
a.
nodes are available.
b.
Voting disks/files are just
like attendance register where you have nodes mark their attendance
(heartbeats).
2.
CSSD process on every node
makes entries in the voting disk to ascertain the membership of the node. While marking their own presence, all the
nodes also register the information about their communicability with other
nodes in the voting disk. This is called
NETWORK HEARTBEAT.
3.
CSSD process in each RAC maintains the heart beat
in a block of size 1 OS block in the hot
block of voting disk at a specific offset. The written block has a header area
with the node name. The heartbeat
counter increments every second on every write call. Thus heartbeat of various nodes is recorded
at different offsets in the voting disk. This process is called DISK HEARTBEAT.
4.
In addition of maintaining its
own disk block, CSSD processes also monitors the disk block maintained by the
CSSD processes of other nodes in cluster. Healthy nodes will have continuous
network & disk heartbeats exchanged between the nodes. Break in heartbeats indicates a possible error
scenario.
5.
If the disk is not updated in a
short timeout period, the node is considered unhealthy and may be rebooted to
protect the database. In this case, a
message to this effect is written in the KILL
BLOCK of node. Each nodes reads its KILL BLOCK once per second, if the kill
block is not overwritten, node commits suicide.
6.
During reconfig (leaving or
joining), CSSD monitors all nodes heartbeat information and determines whether
the nodes has a disk heartbeat including those with no network heartbeat. If no disk heartbeat is detected, then node
is considered as dead.
What
Information is stored in VOTING DISK/FILE
It contains 2 types of data .
Static data : Info
about the nodes in cluster
Dynamic data: Disk
heartbeat logging
It contains the important details of the
cluster nodes membership like
a.
Which node is part of the
cluster.
b.
Which node is leaving the
cluster and
c.
Which node is joining the
cluster.
Purpose of Voting disk or Why
is Voting disk needed
Voting disks are used by clusterware for
health check.
Ø Used by CSS to determine which nodes are currently members of the
cluster.
Ø In concert with other cluster components like CRS to shutdown, fence
or reboot either single or multiple nodes whenever network communication is
lost between any node within the cluster, to prevent to split-brain condition
in which 2 or more instances attempt to
control the RAC database and thus protecting the database.
Ø Will be used by CSS to arbitrate (to take an authorized decision)
with peers that it is not able to see over the private interconnect in the
event of an outage, allowing it to salvage (rescue from loss) the largest
fully connected sub-cluster for further
operation. During this operation , node
membership (NM) will make an entry in the voting disk to inform its vote on
availability. Other instances in the cluster too do similar actions. The 3 voting disks configured also provide a
method to determine who in the cluster should survive.
Example : if
eviction of one of the node is necessiated due to unresponsive action, then the
node that has 2 voting disks with start evicting the other node. NM alternates
it action between the heartbeat and the voting disk to determine the
availability of the other nodes in cluster.
Possible scenarios in Voting disks
As we know now that voting disks is used
by CSSD. It contains both network & disk heartbeat from all nodes and if
any break in heartbeat will result in eviction of the node from cluster. There
are possible scenarios with missing heartbeats.
1.
Network heart beat is
successful, but disk heart beat is missed.
2.
Disk heart beat is successful,
but network heart beat is missed.
3.
Both heart beats failing.
When a cluster is involved with many
nodes, then few more scenarios are possible.
1.
Nodes have a split into N sets
of nodes., communicating within the sets, but not with the members in other
set.
2.
Just one node going
unhealthy. Nodes with quorum (minimum number of nodes to make
cluster valid) will maintain active membership of the cluster and other node(s)
will be fenced/rebooted.
Why should we have ODD number of voting diks ??
A node must be able to access more than
half of the voting disks at any time.
Example.
a.
Let us consider 2 node cluster
with even number of voting disks say 2.
b.
Let node 1 is able to access
Voting disk 1.
c.
Node 2 is able to access voting
disk 2.
d.
From the above steps, we see
that we don’t any common file where clusterware can check the heartbeat of both
the nodes.
e.
If we have 3 voting disks and
both the nodes are able to access more than half ie., 2 voting disks, there
will be atleast one disk which will be accessed by both the nodes. The
clusterware can use this disk to check the heartbeat of the nodes.
f.
A node not able to do so will
be evicted from the cluster by another node that has more than half the voting
disks to maintain the integrity of the cluster.
Where voting disks are stored
It can be stored in
a.
Raw devices
b.
Cluster file system supported
by Oracle RAC such as OCFS,Sun cluster or Veritas Cluster Filesystem
c.
ASM disks (in 11gR2).
When voting disk is stored in ASM, a
question is arised how the voting file on ASM can be accessed when we want to
add a new node to a cluster.
The answer is.
Oracle ASM reserves several blocks at
the fixed location for every Oracle ASM disk used for storing the voting files.
As a result, Oracle clusterware can access the voting disks present in ASM even
if the ASM instance is down and CSS can continue to maintain the Oracle cluster
even if the ASM has failed. The physical
location of the voting files in ASM disks are fixed i.e., the cluster stack
does not rely on a running ASM instance to access the files.
d.
If the ASM is stored in ASM,
the multiplexing of voting disk is decided by the redundancy of the diskgroup.
|
Redundancy
of the diskgroup |
#of copies of
voting disk |
( Minimum # of disks
in the diskgroup) |
|
External
|
1
|
1
|
|
Normal
|
3
|
3
|
|
High
|
5
|
5
|
Commands
to check the Votingdisk
Crsctl query css votedisk - for checking the file location
When to take voting disk backup
1.
Fresh installation
2.
Adding /deleting node
Voting
disk backup (In 10g)
dd if=<voting-disk-path>
of=<backup/path>
Voting
disk restore (In 10g)
dd
if=<backup/path>
of=<voting disk path>
In 11gR2, the voting files are backed up
automatically as part of OCR. Oracle
recommends NOT used dd command to backup or restore as this can lead to loss of
the voting disk.
Add/delete
vote disk
crsctl add css votedisk <path>
-adds a new voting disk
crsctl delete css votedisk <path>
-- deletes the voting disk
