crsctl - Cluster Ready Service Control

To Get help
$crsctl -h

To Get Active Cluster Version

$crsctl query crs activeversion

To Get cluster version of given node

$crsctl query crs softwareversion [node_name]

Ex:$crsctl query crs softwareversion rac1

#crsctl start crs

#crsctl stop crs

(or)

#/etc/init.d/init.crs start

#/etc/init.d/init.crs stop

#crsctl enable crs

#crsctl disable crs

(or)

#/etc/init.d/init.crs enable

#/etc/init.d/init.crs disable

To get cluster stack health status

$crsctl check crs

To get the viability of CSS across nodes

$crsctl check cluster [-node node_name]   -- 11gR2 RACcommand

#crsctl start cluster -n HostName -- 11g R2

#crsctl stop cluster -n HostName -- 11g R2

#crsctl stop cluster -all  -- 11g R2

$ps -ef | grep d.bin

$crsctl check cssd

$crsctl check crsd

$crsctl check evmd

$crsctl check oprocd

$crsctl check ctss

#/etc/init.d/init.cssd stop

#/etc/init.d/init.cssd start

#/etc/rc.d/init.d/init.evmd

#/etc/rc.d/init.d/init.cssd

#/etc/rc.d/init.d/init.crsd

#mv /etc/rc3.d/S96init.cssd /etc/rc3.d/_S96init.cssd -- to stop cssd from autostarting after reboot

#crsctl check css votedisk

#crsctl query css votedisk -- lists the voting disks used by CSS command in 11gR2

#crsctl add css votedisk PATH

#crsctl add css votedisk PATH -force -- if Clusterware is not running

#crsctl delete css votedisk PATH

#crsctl delete css votedisk PATH -force -- if Clusterware is not running

#crsctl set css parameter_name value -- set parameters on OCR

#crsctl set css misscount 100

#crsctl unset css parameter_name -- sets CSS parameter to its default

#crsctl unset css misscount

#crsctl get css parameter_name -- gets the value of a CSS parameter

#crsctl get css disktimeout

#crsctl get css misscount

#crsctl get css reboottime

#crsctl start resources -- starts Clusterware resources

#crsctl start resource ora.DATA.dg
#crsctl stop resources -- stops Clusterware resources

$crsctl status resource

$crsctl status resource -t

$crsctl stat resource -t

#crsctl lsmodules crs -- lists CRS modules that can be used for debugging( need root privilage)

List CRSD Debug Module: AGENT

List CRSD Debug Module: AGFW

List CRSD Debug Module: CLSFRAME

List CRSD Debug Module: CLSVER

List CRSD Debug Module: CLUCLS

List CRSD Debug Module: COMMCRS

List CRSD Debug Module: COMMNS

List CRSD Debug Module: CRSAPP

List CRSD Debug Module: CRSCCL

List CRSD Debug Module: CRSCEVT

List CRSD Debug Module: CRSCOMM

List CRSD Debug Module: CRSD

List CRSD Debug Module: CRSEVT

List CRSD Debug Module: CRSMAIN

List CRSD Debug Module: CRSOCR

List CRSD Debug Module: CRSPE

List CRSD Debug Module: CRSPLACE

List CRSD Debug Module: CRSRES

List CRSD Debug Module: CRSRPT

List CRSD Debug Module: CRSRTI

List CRSD Debug Module: CRSSE

List CRSD Debug Module: CRSSEC

List CRSD Debug Module: CRSTIMER

List CRSD Debug Module: CRSUI

List CRSD Debug Module: CSSCLNT

List CRSD Debug Module: OCRAPI

List CRSD Debug Module: OCRASM

List CRSD Debug Module: OCRCAC

List CRSD Debug Module: OCRCLI

List CRSD Debug Module: OCRMAS

List CRSD Debug Module: OCRMSG

List CRSD Debug Module: OCROSD

List CRSD Debug Module: OCRRAW

List CRSD Debug Module: OCRSRV

List CRSD Debug Module: OCRUTL

List CRSD Debug Module: SuiteTes

List CRSD Debug Module: UiServer

$crsctl lsmodules css -- lists CSS modules that can be used for debugging

The following are the Cluster Synchronization Services modules::

    CSSD

    COMMCRS

    COMMNS

    CLSF

    SKGFD

$crsctl lsmodules evm -- lists EVM modules that can be used for debugging

The following are the Cluster Synchronization Services modules::

    CSSD

    COMMCRS

    COMMNS

    CLSF

    SKGFD

$crsctl start has   (HAS - High Availability Services)
$crsctl stop has

$crsctl check has

CRS-4638 Oracle High Availability Service is online

OCR Modules -- cannot be listed with crsctl lsmodules command

OCRAPI

OCRCLI

OCRSRV

OCRMAS

OCRMSG

OCRCAC

OCRRAW

OCRUTL

OCROSD

#crsctl debug statedump crs -- dumps state info for crs objects

#crsctl debug statedump css -- dumps state info for css objects

#crsctl debug statedump evm -- dumps state info for evm objects

#crsctl debug log crs [module:level]{,module:level} ...

-- Turns on debugging for CRS

#crsctl debug log crs CRSEVT:5,CRSAPP:5,CRSTIMER:5,CRSRES:5,CRSRTI:1,CRSCOMM:2

#crsctl debug log css [module:level]{,module:level} ...

-- Turns on debugging for CSS

#crsctl debug log css CSSD:1

#crsctl debug log evm [module:level]{,module:level} ...

-- Turns on debugging for EVM

#crsctl debug log evm EVMCOMM:1

#crsctl debug trace crs -- dumps CRS in-memory tracing cache

#crsctl debug trace css -- dumps CSS in-memory tracing cache

#crsctl debug trace evm -- dumps EVM in-memory tracing cache

#crsctl debug log res resource_name:level -- turns on debugging for resources

#crsctl debug log res "ora.lnx04.vip:1"

#crsctl trace all_the_above_commands -- tracing by adding a "trace" argument.

#crsctl trace check css

#crsctl backup -h

#crsctl backup css votedisk

Here is the list of the options for CRSCTL in 11gR2:

       crsctl add       - add a resource, type or other entity

       crsctl backup    - back up voting disk for CSS

       crsctl check     - check a service, resource or other entity

       crsctl config    - output autostart configuration

       crsctl debug     - obtain or modify debug state

       crsctl delete    - delete a resource, type or other entity

       crsctl disable   - disable autostart

       crsctl discover  - discover DHCP server

       crsctl enable    - enable autostart

       crsctl get       - get an entity value

       crsctl getperm   - get entity permissions

       crsctl lsmodules - list debug modules

       crsctl modify    - modify a resource, type or other entity

       crsctl query     - query service state

       crsctl pin       - Pin the nodes in the nodelist

       crsctl relocate  - relocate a resource, server or other entity

       crsctl replace   - replaces the location of voting files

       crsctl release   - release a DHCP lease

       crsctl request   - request a DHCP lease

       crsctl setperm   - set entity permissions

       crsctl set       - set an entity value

       crsctl start     - start a resource, server or other entity

       crsctl status    - get status of a resource or other entity

       crsctl stop      - stop a resource, server or other entity

       crsctl unpin     - unpin the nodes in the nodelist

       crsctl unset     - unset a entity value, restoring its default

How do I identify the voting disk/file location?
#crsctl query css votedisk

How to take backup of voting file/disk?
crsctl backup css votedisk

Initialization parameters in Oracle RAC 11gR2

SPECIAL parameters in RAC:
instance_number
instance_group
thread
cluster_database
cluster_database_instances
cluster_interconnects
remote_listener
local_listener
parallel_instance
max_commit_propagation_delay

Parameters with SAME VALUE across all instances:
active_instance_count
archive_lag_target
compatible
cluster_database
cluster_database_instances
cluster_interconnects      
control_files
db_block_size
db_domain
db_files
db_name
db_recovery_file_dest
db_recovery_file_dest_size
db_unique_name
dml_locks                     -- when 0
instance_type                -- rdbms or asm
max_commit_propagation_delay
parallel_max_servers
remote_login_password_file
trace_enabled
undo_management

For example,
*.cluster_database=true
*.cluster_database_instances=2
*.compatible='11.2.0.2'
*.undo_management='AUTO'
 
Parameters with UNIQUE Values across all instances:
instance_number
instance_name
thread
undo_tablespace/rollback_segments

For example,
INST1.instance_number=1
INST1.instance_name=rac1
INST1.thread=1
INST1.undo_tablespace='UNDOTBS1'

INST2.instance_number=2
INST2.instance_name=rac2
INST2.thread=2
INST2.undo_tablespace='UNDOTBS2'

OCRCHECK Command in 11gR2 RAC

$ ocrcheck -h
Name:
        ocrcheck - Displays health of Oracle Cluster/Local Registry.

Synopsis:
        ocrcheck [-config] [-local]

  -config       Displays the configured locations of the Oracle Cluster Registry.
                This can be used with the -local option to display the configured
                location of the Oracle Local Registry
  -local        The operation will be performed on the Oracle Local Registry.

Notes:
        A log file will be created in
        $ORACLE_HOME/log/<hostname>/client/ocrcheck_<pid>.log.
        File creation privileges in the above directory are needed
        when running this tool.

$ ocrcheck
Status of Oracle Cluster Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2736
         Available space (kbytes) :     259384
         ID                       :   33615009
         Device/File Name         :      +DATA
                                    Device/File integrity check succeeded
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured
                                    Device/File not configured

         Cluster registry integrity check succeeded
         Logical corruption check succeeded

$ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2172
         Available space (kbytes) :     259948
         ID                       : 1558172427
         Device/File Name         : /u01/app/11.2.0/grid/cdata/rac3.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded
         Logical corruption check succeeded

$ ocrcheck -config
Oracle Cluster Registry configuration is :
         Device/File Name         :      +DATA

$ ocrcheck -local -config
Oracle Local Registry configuration is :
         Device/File Name         : /u01/app/11.2.0/grid/cdata/rac3.olr

NOTE:  Debugging can be controlled through $CRS_HOME/srvm/admin/ocrlog.in

$ ocrcheck -local    ---(need root permission)
PROTL-602: Failed to retrieve data from the local registry
PROCL-26: Error while accessing the physical storage Operating System error [Permission denied] [13]

# ocrcheck -local
Status of Oracle Local Registry is as follows :
         Version                  :          3
         Total space (kbytes)     :     262120
         Used space (kbytes)      :       2172
         Available space (kbytes) :     259948
         ID                       : 1558172427
         Device/File Name         : /u01/app/11.2.0/grid/cdata/rak3.olr
                                    Device/File integrity check succeeded

         Local registry integrity check succeeded
         Logical corruption check succeeded

Disktimeout/Misscount/reboottime for CSS in Oracle RAC

    1) Disktimeout:
   
    Disk Latencies in seconds from node-to-Votedisk. Default Value is 200. (Disk IO)
    $crsctl get css disktimeout

    2) Misscount:
   
    Network Latencies in second from node-to-node (Interconnect).
    Default Value is 60 Sec (Linux) and 30 Sec in Unix platform. (Network IO)
    Misscount < Disktimeout

    $crsctl get css misscount

    3) RebootTime :

    (default 3 seconds) -the amount of time allowed for a node to complete a reboot
    after the CSS daemon has been evicted.
    crsctl get css reboottime

    4) To Edit these values of Disktimeout/Misscount/reboottime
  
    1. shut down CRS on all nodes but one as root run crsctl on that remaining node
    2. $CRS_HOME/bin/crsctl set css misscount <n> [-force] (<n> is seconds)
    3. $CRS_HOME/bin/crsctl set css reboottime <r> [-force](<r> is seconds)
    4. $CRS_HOME/bin/crsctl set css disktimeout <d> [-force](<d> is seconds)
    5. reboot the remaining node (the node where you just made the change)

Time Difference between RAC Nodes

    If the time difference between the RAC nodes is out of sync (time difference > 30 sec) then it will result one of the following issues

    1. CRS installation failure on remote node
    2. RAC node reboots periodically
    3. CRS Application status UKNOWN or OFFLINE

    To avoid these issues, configure NTP (Network Time Protocol) on both nodes using any one of the following methods

    1. system-config-time or system-config-date or dateconfig

    Type command system-config-time or system-config-date or dateconfig at terminal --> Click “Network Time Protocol” ->check “Enable Network Time Protocol” and select NTP server --> Click OK

    2. date MMDDHHMMSYY
    Type command date with current date and time

    3. /etc/ntp.conf
    Update /etc/ntp.conf file with timeservers IP addresses and start or restart the ntp daemon
    $ /etc/init.d/ntp start
    or
    $ /etc/rc.d/init.d/ntp start

    Once RAC nodes are time sync you might need to shutdown and startup the CRS

    $ crs_stop -all
    $ crs_start -all

11gR2 RAC Backgroun Processor

Oracle RAC is composed of two or more database instances.
They are composed of memory structures and background processes same as the single instance database.

Oracle RAC instances are composed of following background processes:
ACMS    — Atomic Control file to Memory Service (ACMS)
GTX0-j  — Global Transaction Process
LMON    — Global Enqueue Service Monitor
LMD     — Global Enqueue Service Daemon
LMS     — Global Cache Service Process
LCK0    — Instance Enqueue Process
DIAG    — Diagnosability Daemon
RMSn    — Oracle RAC Management Processes (RMSn)
RSMN    — Remote Slave Monitor
DBRM    — Database Resource Manager (from 11g R2)
PING    — Response Time Agent (from 11g R2)
Above processes spawned for supporting the multi-instance coordination.

ACMS (from Oracle 11g)
ACMS stands for Atomic Control file Memory Service. In an Oracle RAC environment ACMS is an agent that ensures a distributed SGA memory update(ie) SGA updates are globally committed on success or globally aborted in event of a failure.

GTX0-j  (from Oracle 11g)
The process provides transparent support for XA global transactions in a RAC environment. The database auto tunes the number of these processes based on the workload of XA global transactions.

LMON
The Global Enqueue Service Monitor (LMON), monitors the entire cluster to manage the global enqueues and the resources and performs global enqueue recovery operations. LMON manages instance and process failures and the associated recovery for the Global Cache Service (GCS) and Global Enqueue Service (GES). In particular, LMON handles the part of recovery associated with global resources. LMON provided services are also known as cluster group services (CGS). Lock monitor manages global locks and resources. It handles the redistribution of instance locks whenever instances are started or shutdown. Lock monitor also recovers instance lock information prior to the instance recovery process. Lock monitor co-ordinates with the Process Monitor (PMON) to recover dead processes that hold instance locks.

LMDx
The Global Enqueue Service Daemon (LMD) is the lock agent process that manages enqueue manager service requests for Global Cache Service enqueues to control access to global enqueues and resources. This process manages incoming remote resource requests within each instance. The LMD process also handles deadlock detection and remote enqueue requests. Remote resource requests are the requests originating from another instance. LMDn processes manage instance locks that are used to share resources between instances. LMDn processes also handle deadlock detection and remote lock requests.

LMSx
The Global Cache Service Processes (LMSx) are the processes that handle remote Global Cache Service (GCS) messages. Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSx varies depending on the amount of messaging traffic among nodes in the cluster.

This process maintains statuses of datafiles and each cached block by recording information in a Global Resource Directory(GRD). This process also controls the flow of messages to remote instances and manages global data block access and transmits block images between the buffer caches of different instances. This processing is a part of cache fusion feature.

The LMSx handles the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, the LMSx will create a consistent read version of the block and send it to the requesting instance. The LMSx also controls the flow of messages to remote instances.

The LMSn processes handle the blocking interrupts from the remote instance for the Global Cache Service resources by:

    Managing the resource requests and cross-instance call operations for the shared resources.
    
    Building a list of invalid lock elements and validating the lock elements during recovery.
    
    Handling the global lock deadlock detection and Monitoring for the lock conversion timeouts.


LCKx
This process manages the global enqueue requests and the cross-instance broadcast. Workload is automatically shared and balanced when there are multiple Global Cache Service Processes (LMSx). This process is called as instance enqueue process. This process manages non-cache fusion resource requests such as library and row cache requests. The instance locks that are used to share resources between instances are held by the lock processes.

DIAG
Diagnosability Daemon – Monitors the health of the instance and captures the data for instance process failures.

RMSn
This process is called as Oracle RAC Management Service/Process. These processes perform manageability tasks for Oracle RAC. Tasks include creation of resources related Oracle RAC when new instances are added to the cluster.

RSMN
This process is called as Remote Slave Monitor. This process manages background slave process creation and communication on remote instances. This is a background slave process. This process performs tasks on behalf of a coordinating process running in another instance.

Oracle RAC instances use two processes GES(Global Enqueue Service), GCS(Global Cache Service) that enable cache fusion. The GES and GCS maintain records of the statuses of each datafile and each cached block using global resource directory (GRD). This process is referred to as cache fusion and helps in data integrity.

Oracle RAC is composed of two or more instances. When a block of data is read from datafile by an instance within the cluster and another instance is in need of the same block, it is easy to get the block image from the instance which has the block in its SGA rather than reading from the disk. To enable inter instance communication Oracle RAC makes use of interconnects. The Global Enqueue Service(GES) monitors and Instance enqueue process manages the cache fusion.

11gR2 RAC Wait Events

Wait-class “Cluster” indicates cluster-related waits related to Cache Fusion:

   
 RAC WaitEvent Naming Conventions

    - Wait Events that start with “GCS%” and “gc%” are Cache Fusion-related waits.
 In other word, they have to do with waits experienced in shipping current or consistent-read versions of blocks across instances in a RAC cluster

    - Events that start with “ges%’ are related to Global Enqueue Services gc current/cr multiblock request  is only a placeholder , until the block is obtained Block provided Immediate from global Cache

    - “gc current block 2-way” - Indicates that a current block was sent from either the resource master   requiring 1 message and 1 transfer

 - “gc current block 3-way” – Indicates that a current block was forwarded to a 3rd node from which it was sent, requiring 2 messages and one block transfer.

    - “gc cr block 2-way” - Indicates that a cr block was sent from either the resource master requiring 1 message and 1 transfer

    - “gc cr block 3-way” -Indicates that a current block was forwarded to a 3rd node from which it was sent, requiring 2 messages and one block transfer



 High Load wait Events

    - “gc current block congested”
    - “gc cr block congested”

    The load wait events indicate that a delay in processing has occurred in the GCS, which is usually caused by high load, CPU saturation,paging and would have to be solved by additional CPUs, load-balancing, off loading processing to different times or a new cluster node.

   
AWR Reports

    Check the values for:
    Average global cache cr block recieve time (ms)
    Average global cache current block recieve time (ms)

 RAC Specific Wait Event

A) Block-Oriented Waits
-----------------------
1. gc current block 2-way
2. gc current block 3-way
3. gc cr block 2-way
4. gc cr block 3-way

B) Message-Oriented Wais
-------------------------
1. gc current grant 2-way
2. gc current grant 3-way
3. gc cr grant 2-way
4. gc cr grant 3-way (Normally this wait event is not possible, but

"_cr_grant_local_role" --> turn 3-way CR grants off, make it automatic, or turn it on)

C) Contention-Oriented Waits
----------------------------
1. gc current block busy (cluster cache contention)
2. gc cr block busy
3. gc current buffer busy (local cache contention)

D) Load-Oriented Waits
----------------------
1. gc current block congested
2. gc cr block congested
3. gc current grant congested
4. gc cr grant congested

A.1. gc current block 2-way (write/write with 2 nodes)

Definition
- Requesting instance request any data block for dml(current) from Master.
- If master is the holder of that data block and also has already modified that block.
- Then master will retain PI block for itself.
- Master will also flush respective redo log to log file before sending CURRENT block to requesting instance.
- Meanwhile requesting instance will wait in "GC CURRENT BLOCK 2-WAY"

Reason
Before sending the current data block to the requesting instance master instance first flush respective redo log of the log file then it will prepare PI block and then  send CURRENT block to the requesting instance over the interconnect.

- This wait event appears in "TOP-5" timed events section of AWR Report.
- Analyze the contention using AWR REPORT.
- In AWR REPORT, analyze "Current Block Received" and identify the top contentious objects.
- You are getting contention at segment level because of bad database design, DB object layout and Space Management.
- LGWR is not efficient to write in redo log file and that's why requesting instance is waiting in "gc current block 2-way".
- Interconnect is having Network latency.
- Application Segregation is also a reason. (bad explain, 

Troubleshooting
 - Best Solution is to apply application Segregation means try to locate all select query on one node and all DML on another node.
- Tune LGWR
- Tune Interconnect

A.2. gc current block 3-way (write/write with 3 nodes)

Definition
- Requesting instance request any data block in CURRENT MODE for dml(current) from Master.
- If master is not holder of that data block and that data block is globally available on another instance.
_ Master will send a message to the current holding instance to relinquish ownership (Downgrade lock).
- The holding instance retain the PI of that data block and then serve to the requesting instance.
- Holding instance will also flush respective redo log to log file before sending CURRENT block to requesting instance.
- Meanwhile requesting instance will wait in "GC CURRENT BLOCK 3-WAY"

Reason
As you know before sending the current data block to the requesting instance master instance first flush respective redo log of the log file then it will prepare PI block and then CR block to send to the requesting instance over the interconnect.

- This wait event appears in "TOP-5" timed events section of AWR Report.
- Analyze the contention using AWR REPORT.
- In AWR REPORT, analyze "Current Block Recieved" and identify the top contentious objects.
- You are getting contention at segment level because of bad database design, DB object layout and Space Management.
- LGWR is not efficient to write in redo log file and thats why requesting instance is waiting in "gc current block 3-way".
- Interconnect is having N/W latency.
- Application Segregation is also a reason.

Troubleshooting
- Best Solution is to apply application Segregation means try to locate all select query on one node and all DML on another node.
- Tune LGWR
- Tune Interconnect

A.3. gc cr block 2-way (read/read or write/read with 2 nodes)

Definition

Case-1: WRITE/READ
- Requesting instance request any CR data block for select from Master.
- If master is the holder of that data block and also has already modified that block.
- Then master will prepare CR copy of that data block (using undo).
- Finally Master instance serve CR block to the requesting instance.
- Meanwhile requesting instance will wait in "gc cr block 2-way"

Case-2: READ/READ
- Requesting instance request any CR data block for select from Master.
- If master is the holder of that data block and has not already modified that block.
- Master instance serve CR block to the requesting instance immediately.
- Meanwhile requesting instance will wait in "gc cr block 2-way"

Reason
- In both the cases you will encounter this wait event in "TOP-5" section of AWR Report.
- Plan of action would be similer like "gc current block 2-way"
- No locks are maintained by GCS for CR blocks

-Long-pending transaction on highly accessed objects can lead to CR storm ( Run this sort of transactions at a less busy timeframe )

A.4. gc cr block 3-way (read/read or write/read with 3 nodes or more than 3 nodes)

This wait event is exactly same as "gc cr block 2-way", only the difference is that here 3 or more than 3 instances are involved.

B.1. gc current grant 2-way

Definition
- An instance request any data block in CURRENT MODE for dml(current) from Master.
- If the data block is not cached on any instance even on master too, then master instance will send a message to the requesting instance granting the EXCLUSIVE lock.
- Meanwhile requesting instance will wait in "GC CURRENT BLOCK 2-WAY"
- Requesting instace will read data block from disk and do the physical I/O.
- This wait event does not indicate any contention.

Reason
- This wait event appears in "TOP-5" timed events section of AWR Report.
- This wait event represent that requesting instance is spending a significant amount of time in obtaining the locks.
- Interconnect is having N/W latency (rare).

Troubleshooting
 - Best Solution is to tune your SQL Application so that it request less amout of data blocks.
- Because if requesting instance ask for more data blocks then master has to locking and holding instance related information in GRD before granting EXCLUSIVE LOCK, which will be high.
- Tune Interconnect (very rare).

B.2. gc cr grant 2-way

Definition
- An instance request any data block in CR MODE for select from Master.
- If the data block is not cached on any instance even on master too, then master instance will send a message to the requesting instance granting the SHARED lock.
- Meanwhile requesting instance will wait in "GC CR BLOCK 2-WAY"
- Requesting instance will read data block from disk and do the physical I/O.
- This wait event does not indicate any contention.

B.3. gc current grant 3-way

Definition
- An instacne request for any data block in current mode.
- Mater is not the holder of that data block, then master forward a message to provide the data block to the requesting instance.
- Current holding instance don't have the block because of aging out mechanism.
- Current holding instance grant exclusive lock to the requesting instance for physical IO.
- Meanwhile requesting instance will wait in "gc current grant 3-way".
- there is a hidden parameter to control this "_cr_grant_local_role"

C.1. gc current block busy

Definition
- An instance request for any data block in current mode, it send a request to the master.
- If master is the holder of that data block and also has already modified that block.
- Then master will retain PI block for itself.
- Master will also flush respective redo log to log file before sending CURRENT block to requesting instance.
- Now the block transfer delayed on requsting instance.
- Meanwhile requesting instance will wait in "gc current block busy"
- Busy events indicate that LMS needs to perform add. work before sending a block ( like apply Redo to build CR block )

Reason
- The block was being used by a session on another instance.
- The block transfer was delayed because the holding instance could not write the corresponding redo record to the online redo log file.

Troubleshooting
 - Tune LGWR
- Appropriate Application Partitioning
- Tune N/w

C.2. gc current buffer busy   /       C.3. gc cr block busy
This wait event appears mostly in SINGLE INSTANCE when more than one sessions are trying to access same data block.

Troubleshooting
 - Application tuning

D.1.  gc current block congested  /  D. 2.  gc cr block congested

Definition
- Whenever any instance (HOST-03) request for any data block in any mode, this request will be served by MASTER NODE (HOST-01) of that data block.
- LMS process running on master node will provide data block to the requesting instance LMS process.
- Now LMS process running on both the nodes are highly loaded so there would be wait event ""gc current block congested

- if LMS process did not process a request within 1ms than LMS marks the response to that block with the congestion wait event

Reason
- Highly loaded LMS process or CPU.

- LMS is suffering CPU scheduling, LMS is suffering resources like memory ( paging )

Troubleshooting
- Increase number of LMS Processes by setting "GCS_SERVER_PROCESS" in 9i RAC onward.
- Optionally you can also set "_LM_LSM" till OPS.

- As LMS processes are RT processes OS scheduling delays should be minimal
- Tune OS
- Add CPUs


D.3. gc current grant congested

Definition
- Whenever any instance (HOST-03) request for any data block in any mode, this request will be served by MASTER NODE of that data block.
- Meanwhile requesting instance (HOST-03) is waiting for approval from master instance to perform physical IO to read data block from DISK.
- This wait event is "gc current grant congested"

Some more ......

GC current request/GC CR request

• These are placeholder requests which should which should be mapped to one of the above wait events once the LMS responds
• If there is no response in 0.5s from LMS ( 6s on windows ) the acoounted time is added to GC Lost Block wait event

GC Log Flush Sync

• for details please  read following link..
• in a healthy database 90 % of GCS Log flush Sync requet should be finished in 2ms or less
• Always check related instances for LOG FILE SYNC waiut event as this event also reduces the available Redo I/O bandwith 
• Monitor Wait distribution/histogramms  with following script : @event_histogram_from_awr.sql

GC Buffer Busy Acquire/Release

• For RHS index growth you see GC Buffer Busy Acquire/Release and GC Buffer Busy events and gc current request for the same block
• If the block is on instance 1 all other sessions from instance 1 are waiting on gc buffer busy aquire whereas sessions on the other instances are waiting on gc buffer busy release
• Always monitor GV$session P1,P2,P3 to get detail information about what blocks are used

Scripts to Measure RAC Wait events

Measuring RAC Waits from V$SESSION_WAIT

set linesize 200
set pages 55
col sid format 99999
col name format a36
col p1 format 999999999 Head ‘P1’
col program format a25
col p2 format 999999999 Head ‘P2’
col p3 format 999999999 Head ‘P3’
col pgm format a15 head ‘What’
col state format a15
col wt format 9999999 head ‘Wait|Time’
col WaitEvent format a38 head ‘Wait Event’
col lc format 99999999999.99 head ‘last call’
select A.sid,
decode(A.event,’null event’,’CPU Exec’,A.event) WaitEvent,
A.p1,A.p2,A.p3,
decode(A.state,’WAITING’,’WTG’,
‘WAITED UNKNOWN TIME’,’UNK’,
‘WAITED SHORT TIME’,’WST’,
‘WAITED KNOWN TIME’,’WKT’) wait_type,
decode(A.state,’WAITING’,A.seconds_in_wait,
‘WAITED UNKNOWN TIME’,-999,
‘WAITED SHORT TIME’,A.wait_time,
‘WAITED KNOWN TIME’,A.WAIT_TIME) wt,
round((last_call_et/60),2) lc,
substr(nvl(b.module,b.program),1,15) pgm
from v$session_wait A,
v$session B
where 1=1
and (A.event like ‘gc%’
or A.event like ‘GC%’
or A.event like ‘ge%’)
and A.event not like ‘%remote message’
and A.event not like ‘%sleep’
and A.sid=B.sid
and B.status=’ACTIVE’
order by 1
/

 Sample output:
930


Measuring RAC Waits from  V$ACTIVE_SESSION_HISTORY
set lines 200
col samplestart format a30 head ‘Begin Time’
col sampleend format a30 head ‘End Time’
col aas format 999.99 head ‘AAS’
col aas_event format 999.99 head ‘AAS per |Event’
col dbt format 999999 head ‘DB Time’
col event format a30
col time_waited format 999999 head ‘Time (ms)’
col wait_pct format 999.99 head ‘% of DB Time’
COLUMN bt NEW_VALUE _bt NOPRINT
COLUMN et NEW_VALUE _et NOPRINT
select min(cast(sample_time as date)) bt,
sysdate et
from v$active_session_history;
with xtimes (xdate) as
(select to_date(‘&_bt’) xdate
from dual
union all
select xdate+(&&interval_mins/1440)
from xtimes
where xdate+(&&interval_mins/1440) < sysdate)
select to_char(s1.xdate,’DD-MON-RR HH24:MI:SS’) samplestart,
to_char(s1.xdate+(&&interval_mins/1440),’DD-MON-RR HH24:MI:SS’) sampleend,
s2.event,.001*((sum(s2.time_waited))) time_waited,
(count(s2.sample_id)/(60*&&interval_mins)) aas_event,
(count(s3.sample_id)) dbt,
round(100*(sum(s2.time_waited)/1000000/count(s3.sample_id)),2) as wait_pct
from xtimes s1,
     v$active_session_history s2,
     v$active_session_history s3
where s2.sample_time between s1.xdate and s1.xdate+(&&interval_mins/1440)
and s3.sample_time between s1.xdate and s1.xdate+(&&interval_mins/1440)
and s2.sample_id=s3.sample_id
and (s2.event  like ‘gc%’
or s2.event like ‘GC%’
or s2.event like ‘ge%’)
and s2.event not like ‘%remote message’
and s2.event not like ‘%sleep’
group by s1.xdate,s2.event
order by s1.xdate
/
undefine interval_mins
Sample output:

931


Measuring RAC Waits from DBA_HIST_ACTIVE_SESS_HISTORY
set lines 200
col samplestart format a30 head ‘Begin Time’
col sampleend format a30 head ‘End Time’
col aas format 999.99 head ‘AAS’
col aas_event format 999.99 head ‘AAS per |Event’
col event format a30
col time_waited format 999999 head ‘Time (ms)’
col wait_pct format 999.99 head ‘Wait %’
COLUMN bt NEW_VALUE _bt NOPRINT
COLUMN et NEW_VALUE _et NOPRINT
select min(cast(sample_time as date)) bt,
sysdate et
from v$active_session_history;
with xtimes (xdate) as
(select to_date(‘&_bt’) xdate
from dual
union all
select xdate+(&&interval_mins/1440)
from xtimes
where xdate+(&&interval_mins/1440) < sysdate)
select to_char(s1.xdate,’DD-MON-RR HH24:MI:SS’) samplestart,
to_char(s1.xdate+(&&interval_mins/1440),’DD-MON-RR HH24:MI:SS’) sampleend,
s2.event,
.001*((sum(s2.time_waited))) time_waited,
10*(count(s2.sample_id)/(60*&&interval_mins)) aas_event,
10*(count(s3.sample_id)) dbt,
round(100*(sum(s2.time_waited)/1000000/(10*(count(s3.sample_id)))),2) as wait_pct
from xtimes s1,
     dba_hist_active_sess_history s2,
     dba_hist_active_sess_history s3
where s2.sample_time between s1.xdate and s1.xdate+(&&interval_mins/1440)
and s3.sample_time between s1.xdate and s1.xdate+(&&interval_mins/1440)
and s2.sample_id=s3.sample_id
and (s2.event  like ‘gc%’
or s2.event like ‘GC%’
or s2.event like ‘ge%’)
and s2.event not like ‘%remote message’
and s2.event not like ‘%sleep’
group by s1.xdate,s2.event
order by s1.xdate
/
undefine interval_mins
Sample output:
932