OCR Backup &Recovery Types IN 11GR2 RAC With ASM

Managing OCR

Three tools used to manage: OCRCONFIG, OCRDUMP and OCRCHECK

How to check OCR recovery required for the cluster?

[root@rac1]# /u01/app/11.2.0/grid/bin/ocrcheck

always check the cluster registry integrity to verify the recovery of ocr required or not
 
[root@rac1 bin]# ./ocrcheck

Status of Oracle Cluster Registry is as follows :

Version : 3

Total space (kbytes) : 262120

Used space (kbytes) : 3316

Available space (kbytes) : 258804

ID : 579764850

Device/File Name : /u01/ocr/ocr

Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

If this command does not display the message Device/File integrity check succeeded for at least one copy of the OCR, then both the primary OCR and the OCR mirrors have failed.
You must restore the OCR from a backup. If there is at least one copy of the OCR available, you can use that copy to restore the other copies of the OCR

Types of OCR Backup & Recovery
1.Manual backup & Recovery of OCR

2.Manual Export & Import of OCR  

Types of OCR Backup & Recovery

1.Manual backup & (Automatic/manual)Recovery of OCR

2.Manual Export & Import of OCR

 1.Manual backup & Recovery of OCR

To perform manual backup of OCR:

• You can use the ocrconfig -manualbackup command to force Oracle Clusterware to perform a backup of the OCR at any time, rather than wait for the automatic backup that occurs at 4-hour intervals.
• The -manualbackup option is especially useful when you to need to obtain a binary backup on demand, such as before you make changes to the OCR.
• You need to be logged in as root user to take manual backups

To view the available OCR Manual backup
[root@rac1 bin]# ./ocrconfig -showbackup manual  

To perform manual backup
 
 [root@rac1 bin]# ./ocrconfig -manualbackup
[root@rac1 bin]# ./ocrconfig -showbackup manual

 
To perform manual recovery of OCR:

When restoring the OCR from automatically generated backups, you first have to determine which backup file you will use for the recovery. Identify the available OCR backups using the ocrconfig command while logged in as root user:
STEP 1: View the available list of OCR backup as root user and choose which OCR file needed for recovery
[root@rac2 bin]# ./ocrconfig -showbackup manual

STEP 2: review the content  available in the backup of OCR using “OCRDUMP” command
The command generates an  OCRDUMPFILE in ASCII file format.

[root@racl2 bin]# ./ocrdump -backupfile /u01/app/11.2.0.2/grid/cdata/rac2/backup_271012_2.ocr

[root@raclinux2 bin]# ls OCRDUMPFILE

 

OCRDUMPFILE

then view content in OCRDUMPFILE to verify the validity

(imagine that we have valid copy of the ocr available).

Now recovery OCR using following steps

Automatic (or) manual Restore of OCR:

When restoring the OCR from automatically/manually generated backups,

first we have to determine which backup file you will use for the recovery.

Identify the available OCR backups using the ocrconfig command while logged in as root user

Step1: view the list of available backup of OCR created by automatically/manually

[root@rac1 bin]#./ocrconfig -showbackup auto

(or)

[root@rac1 bin]#./ocrconfig -showbackup manual

Step 2: Point the available backup of ocr file either created automatically or manually to verify the validity and read the content in the OCRDUMPFILE created in ASCII format

[root@rac1 bin]#./ocrdump -backupfile /u01/app/11.2.0.2/grid/cdata/rac1/backup_20101219_201012.ocr

Step 3: As the root user, Restart Oracle Clusterware on all nodes  (for instance rac1, rac2) of the Oracle RAC cluster

On node:RAC1

[root@rac1 bin]#./crsctl stop crs

On node:RAC2

[root@rac2 bin]#./crsctl stop crs

The OCR is present on ASM so we need to start the ASM on the the particular node(rac1)(called "exclusive mode") where we will recover the OCR. After the startup of rac1 node verify that the diskgroup we need are mounted.

On node:RAC1

[root@rac1 bin]#./crsctl start crs -excl

SQL> select name, state from v$asm_diskgroup;

NAME STATE

—————————— ———–

DATA MOUNTED

DGDUP1 MOUNTED

DGDUP2 MOUNTED

DGDUP MOUNTED

PRIM MOUNTED

SEC MOUNTED

6 rows selected.

Step 4: check and stop crsd if  running on node rac1

[root@rac1 bin]#./crsctl stop resource ora.crsd -init

Note:(if you do not stop, it  will throw an error )

Step 5: As the root user, restore the OCR backup.

mention the file_name for which we have already verified the validity and  include the name of the OCR that you want to restore.

[root@rac1 bin]#./ocrconfig -restore

/u01/app/11.2.0.2/grid/cdata/rac1/backup_20101219_201012.ocr

PROT-19: Cannot proceed while the Cluster Ready Service is running

(it shows restore command failed due to the error the crs running so stop the crsd )

[root@rac1 bin]#./crsctl stop resource ora.crsd -init

CRS-2673: Attempting to stop ‘ora.crsd’ on ‘rac1′

CRS-2677: Stop of ‘ora.crsd’ on ‘rac1′ succeeded

(crs is sucessfully stopped)

Step 6: now crs is sucessfully stoped again run the restore command and check the integrity of OCR once again

[root@rac1 bin]#./ocrconfig -restore

/u01/app/11.2.0.2/grid/cdata/rac1/backup_20101219_201012.ocr

[root@rac1 bin]#

(above prompt shows restore process sucessfull)

Now verify the integrity of OCR in rac1 node

[root@rac1 bin]#./ocrcheck

Step 7: As root stop the Oracle clusterware as it is running exclusivly on node rac1 ( called" exclusive mode") and start clusterware on all nodes (rac1 ,rac2)

[root@rac1 bin]#./crsctl stop crs -f

.

.

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘rac1′ has completed

CRS-4133: Oracle High Availability Services has been stopped.

Now start the node “ rac1”

[root@rac1 bin]#./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

 Now start the node “ rac2”

[root@rac2 bin]#./crsctl start crs

CRS-4123: Oracle High Availability Services has been started.

Step 9: Using the Cluster Verification Utility(CVU) to verify the OCR integrity.

Run the following command, where the "-n all" argument retrieves a list of

all the cluster nodes that are configured as part of your cluster.

[root@rac1 bin]#./cluvfy comp ocr -n all [-verbose]

[root@rac2 bin]#./cluvfy comp ocr -n all [-verbose]

The output as show below confirms that OCR was successfully restored on rac1, rac2.

 

2.Manual Export & Import of OCR

Manual Export of OCR file:

By using ocrconfig -export file_name command to export the OCR content to a file format.

we can also export the OCR contents to a file before and after making significant configuration changes such as adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database.

Exporting the OCR contents to a file lets you restore the OCR if your configuration changes cause errors.
 
EXAMPLE,
if you have un-resolvable configuration problems
if you are unable to restart your cluster database after such changes like adding or deleting nodes from your environment, modifying Oracle Clusterware resources, or creating a database.
then you can restore your configuration by importing the saved OCR content from the valid configuration.

STEP1:
[root@rac2 bin]# ./ocrconfig -export /tmp/ocr_export120812.exp
[root@rac2 bin]# ls /tmp
ocr_export120612.exp
ocr_export120712.exp
ocr_export120812.exp

Manual Import of OCR file:

We have two node cluster RAC1, RAC2. To import the exported_ocr_backup_file
follow the steps below

STEP 1: stop Oracle Clusterware on all the nodes(rac1, rac2) in Oracle RAC cluster as root user
[root@rac1 bin]#./crsctl stop crs
 
[root@rac2 bin]#./crsctl stop crs

STEP 2: start the Oracle clusterware on only one node(rac2) in "exclusive mode" so that we are able to perform the import process.
a)[root@rac2 bin]# ./crsctl start crs -excl
(or)
b)[root@rac2 bin]# ./crsctl start crs -excl -nocrs

STEP 3: All ASM disk groups that needed for OCR import process are mounted by ASM. Before
starting the import from the "exported file".
SQL> select name, state from v$asm_diskgroup;
NAME STATE
—————————— ———–
DATA MOUNTED
DGDUP1 MOUNTED
DGDUP2 MOUNTED
DGDUP MOUNTED
PRIM MOUNTED
SEC MOUNTED
6 rows selected.

STEP 4: Check the ocr file from the exported location and perform the import by giving full path and file name
[root@rac2 bin]# ls /tmp
ocr_export120612.exp
ocr_export120712.exp
ocr_export120812.exp

[root@rac2 bin]#ocrconfig -import /tmp/tmp/ocr_export120812.exp
#################Note:we cannot use the ocrconfig command to import an OCR backup file created automatically, only exported OCR file can be imported

STEP 5: Now check the integrity Status of Oracle Cluster Registry as root user from rac2 node
[root@rac2 bin]# ./ocrcheck
Status of Oracle Cluster Registry is as follows :
......
Cluster registry integrity check succeeded
Logical corruption check succeeded.

STEP 6: As root user stop the CRS from rac2 node and bring it out from exclusive mode
[root@rac2 bin]# ./crsctl stop crs -f
.
.
.
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘rac2′ has completed
CRS-4133: Oracle High Availability Services has been stopped.

STEP 7:Logged in as the root user, restart Oracle Clusterware on all the nodes(RAC1, RAC2) in your cluster by restarting each node, or by running the following command:
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

[root@rac2 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

STEP 8: Verify the status of OCR integrity between the nodes (rac1, rac2) with option "-n all"
which helps to retrieves the information about list of all the cluster nodes that are configured as part of your cluster.

[oracle@rac1 bin]$ cluvfy comp ocr -n all -verbose
[oracle@rac2 bin]$ cluvfy comp ocr -n all -verbose

FAN VS ONS

FAN (Fast Application Notification)

FAN is a notification mechanism that RAC uses to notify other processes about configuration and service level information such as includes service status changes, such as UP or DOWN events. Applications can respond to FAN events and take immediate action. FAN UP and DOWN events can apply to instances, services, and nodes.

RAC publishes the FAN events the minute any changes are made to the cluster. So, instead of waiting for the application to check on individual nodes to detect an anomaly, the applications are notified by FAN events and are able to react immediately.
FAN also publishes load balancing advisory (LBA) events. Applications are in a position to take full advantage of the LBA FAN events to facilitate a smooth transition of connections to healthier nodes in a cluster.

One can take advantage of FAN is the following ways:

1. When using integrated Oracle Client, the applications can use FAN with no programming whatsoever. Oracle 11g JDBC, ODP.NET and OCI would be considered as the components of the integrated clients.
2. Programmatic changes in ONS API make it possible for applications to still subscribe to the FAN events and can execute the event handling actions appropriately.
3. FAN can be implemented with server-side callouts on your database tier.

For instance, a typical DOWN event will prevent any further disruption of service by cleanly terminating the sessions on that failed node and notifying the user. Moreover, a typical UP event can address and allocate extra resources for new incoming requests/ connections. There are however several additional benefits with the server-side callouts

Main utilization of  FAN is:

1. Logging
2. Paging/SMS the DBA and/or to open trouble tickets when the resources fail to (re)start
3. Change resource plans or to shut down services when the number of available instances decreases, thus preventing further load on the cluster and keeping the RAC running until another healthy node is added to the cluster.

Automate the fail service back to PREFERRED instances when required. 
 





ONS (Oracle Notification Services)

ONS allows users to send SMS messages, e-mails, voice notifications, and fax messages in an easy-to-access manner. Oracle Clusterware uses ONS to send notifications about the state of the database instances to mid-tier applications that use this information for load-balancing and for fast failure detection. ONS is a daemon process that communicates with other ONS daemons on other nodes which inform each other of the current state of the database components on the database server.

To add additional members or nodes that should receive notifications,  the hostname or IP address of the node should be added to the ons.config file.
The ONS configuration file is located in the $ORACLE_HOME/opmn/conf  directory and has the
Example:

Oracle Notification Server Local Port	6100	6100 -6199
Oracle Notification Server Remote Port	6200	6200 -6299
Oracle Notification Server Request	6003	6003 -6099

port referece here

[oracle@racdb4 oracle]$ cat $ORACLE_HOME/opmn/conf/ons.config
localport=6101
remoteport=6201
loglevel=3
useocr=on
nodes=racdb1.cluster.oracle.com:6101,racdb2.cluster.oracle.com:6101,
racdb3.cluster.oracle.com:6201,racdb4.cluster.oracle.com:6201,
onsclient1.cluster.oracle.com :6200,onsclient2.cluster.oracle.com :6200

1. The localport is the port that ONS binds to on the local host interface to talk to local clients.
2. The remoteport is the port that ONS binds to on all interfaces to talk to other ONS daemons.
3. The loglevel indicates the amount of logging that should be generated. Oracle supports logging levels from 1 through 9.
4. ONS logs are generated in the $ORACLE_HOME/opmn/logs directory on the respective instances.
5. The useocr parameter (valid values are on/off) indicates whether ONS should use the OCR to determine which instances and nodes are participating in the cluster.
6. The nodes listed in the nodes line are all nodes in the network that will need to receive or send event notifications. This includes client machines where ONS is also running to receive FAN events for applications.

How do you monitor and track when these events are fired?
Oracle has provided options where a script or utility or application (called server-side callouts), if placed in the $ORA_CRS_HOME/racg/usrco directory, will be executed automatically.
For example,
the following shell script, when placed in this directory will write out events generated by the Event Manager to the file defined by the symbol FAN_LOGFILE in the script:
[oracle@oradb4 oracle]$ CAT callout.sh
#! /bin/ksh
FAN_LOGFILE=/usr/app/oracle/product/11.2.0/crs/evm/admin/log/racdb4_FANuptime.log
echo >> $FAN_LOGFILE &

Transparent Application Failover (TAF) -Client side/Server side configuration

Transparent Application Failover (TAF) is a client-side feature that allows for clients to reconnect to surviving databases in the event of a failure of a database instance.

1.Notifications are used by the server to trigger TAF callbacks on the client-side.
2.TAF is configured using either client-side specified TNS connect string or using server-side service attributes. However, if both methods are used to configure TAF, the server-side service attributes will supersede the client-side settings. The server-side service attributes are the preferred way to set up TAF.
3.TAF can operate in one of two modes, Session Failover and Select Failover. Session Failover will re-create lost connections and sessions. Select Failover will replay queries that were in progress.
4.TAF automatically restores some or all of the following elements associated with active database connections.
5.TAF automatically reestablishes the connection using the same connect string or an alternate connect string that you specify when configuring failover.
6.TAF automatically logs a user in with the same user ID as was used prior to failure. If multiple users were using the connection, then TAF automatically logs them in as they attempt to process database commands.
7.TAF allows applications that began fetching rows from a cursor before failover to continue fetching rows after failover. This is called "select" failover.
8.Any active transactions are rolled back at the time of failure because TAF cannot preserve active transactions after failover.
9.Server-side program variables, such as PL/SQL package states, are lost during failures; TAF cannot recover them.

TAF Configuration on Client side:
 A backup connection can be pre-established. The initial and backup connections must be explicitly specified.
In Example, clients that use net service name taftest1 to connect to the listener on or-11 are also preconnected to or-12. If or-11 fails after the connection, Oracle Net fails over to or-12, preserving any SELECT statements in progress.

TAFTEST1 =(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=or-11) (PORT=1521))
  (CONNECT_DATA=(SERVICE_NAME=rac)(INSTANCE_NAME=rac1)
     (FAILOVER_MODE= (BACKUP=TAFTEST2)(TYPE=select)(METHOD=preconnect)))
)

################################

TAFTEST2 =(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp) (HOST=or-12) (PORT=1521))
  (CONNECT_DATA=(SERVICE_NAME=rac)(INSTANCE_NAME=rac2)
     (FAILOVER_MODE=(BACKUP=TAFTEST2)(TYPE=select)(METHOD=preconnect)))
)

TAF Verification Query
You can query FAILOVER_TYPE, FAILOVER_METHOD, and FAILED_OVER columns in the V$SESSION view to verify that TAF is correctly configured.

SELECT MACHINE, FAILOVER_TYPE, FAILOVER_METHOD, FAILED_OVER, COUNT(*)
FROM V$SESSION
GROUP BY MACHINE, FAILOVER_TYPE, FAILOVER_METHOD, FAILED_OVER;

TAF Configuration on Server side:

When you check the service configuration you find no values for failover method

Creating a service
srvctl add service -d rac -s server_taf -r "rac1,rac2" -P BASIC 

Start the service
srvctl start service -d rac -s server_taf 

Checking that the services are running.
srvctl config service -d rac

ractest PREF: rac1 rac2 AVAIL:
server_taf PREF: rac1 rac2 AVAIL: 

Getting the service ID value

sqlplus /nolog
Connect / as sysdba
SQL> select name,service_id from dba_services where name = 'server_taf';
NAME          SERVICE_ID
--------------------------------
server_taf            6 

Checking the setup

SQL>col name format a15
col failover_method format a11 heading 'METHOD'
col failover_type format a10 heading 'TYPE'
col failover_retries format 9999999 heading 'RETRIES'
col goal format a10
col clb_goal format a8
col AQ_HA_NOTIFICATIONS format a5 heading 'AQNOT'
 
SQL>select name, failover_method, failover_type, failover_retries,goal, clb_goal,aq_ha_notifications
from dba_services where service_id = 6

NAME      METHOD   TYPE RETRIES GOAL CLB_GOAL AQ_HA_NOT
--------------- ----------- ---------- -------- ---------- -------- -----
server_taf   LONG        NO 

Adding the server side failover parameter to the service

Server side TAF method is BASIC. BASIC is the only value currently supported. This means that a new connection is established at failure time. It is not possible to pre-establish a backup connection. (which is to say, PRECONNECT is not supported)

SQL> execute dbms_service.modify_service (service_name => 'server_taf' -
, aq_ha_notifications => true -
, failover_method => dbms_service.failover_method_basic -
, failover_type => dbms_service.failover_type_select -
, failover_retries => 180 -
, failover_delay => 5 -
, clb_goal => dbms_service.clb_goal_long);
PL/SQL procedure successfully completed.

Check that the service can now see the values for methods, types and retries

SQL>select name, failover_method, failover_type, failover_retries,goal, clb_goal,aq_ha_notifications
from dba_services where service_id = 6


NAME METHOD TYPE RETRIES GOAL CLB_GOAL AQ_HA_NOT
--------------- ----------- ---------- -------- ---------- -------- -----
server_taf BASIC SELECT 180 NONE LONG YES 

Check that the listener has the service registered
lsnrctl services 

Create a net service name. Here we have client load balancing between the two nodes.

SERVERTAF =
(DESCRIPTION =
(LOAD_BALANCE = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1.cluster.us.oracle.com)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2.cluster.us.oracle.com)(PORT = 1521))
(CONNECT_DATA =
(SERVICE_NAME = server_taf.rac.oracle.com)

Client-side load balancing/ SCAN in Oracle RAC 11GR2

Client-side load balancing
We have a four nodes cluster. Using a simple TNS descriptor for connection in prior 11gr2 rac

RACDB =
  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.us.oracle.com)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.us.oracle.com)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac3-vip.us.oracle.com)(PORT = 1521))    
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip.us.oracle.com)(PORT = 1521))    
  (CONNECT_DATA =
      (SERVICE_NAME = racdb)
    )
  )

In this case, client will connect to the first node (on rac1 local listener) (rac1-vip.us.oracle.com)
only (if it's available of-course). For avoiding this situation we had to use LOAD_BALANCE parameter.

RACDB =
  (DESCRIPTION =
   (LOAD_BALANCE=ON)
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.us.oracle.com)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.us.oracle.com)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac3-vip.us.oracle.com)(PORT = 1521))    
   (ADDRESS = (PROTOCOL = TCP)(HOST = rac4-vip.us.oracle.com)(PORT = 1521))    
  (CONNECT_DATA =
      (SERVICE_NAME = racdb)
    )
  )

Now, Oracle Net will choose one address from the list of addresses in a random sequence that will allow clients to balance the load on the all four listeners.

 
SCAN (Single Client Access Name)
In 11gR2 SCAN (Single Client Access Name) was introduced. The SCAN feature is a new 'layer' between clients and local listeners in cluster, that allows you to change the your cluster configuration (i.e add/remove nodes) without making configuration changes in their clients. It means that now you can use only one name(SCAN name) for connection to RAC instead of using a set of VIPs.

RACDB1 =
 (DESCRIPTION =
  (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.cluster.us.oracle.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVICE_NAME = racdb)
    )
  )

The SCAN is configured during the installation of Oracle Grid Infrastructure. There are 2 options for defining the SCAN:

1. Define the SCAN in your corporate DNS (Domain Name Service)
2. Use the Grid Naming Service (GNS). 
   

Let's have a look on each option.
1.In first option your network administrator has to create a single name in DNS server that resolves up to three IP addresses using a round-robin algorithm.
2.Three IP addresses are recommended considering load balancing and high availability requirements regardless of the number of servers in the cluster (but using two addresses is also possible).
3.The IP addresses must be on the same subnet as your public network in the cluster.


Note: This one scan name(rac-scan.cluster.us.oracle.com) carry 3 IP address
EX:

rac-scan.cluster.us.oracle.com IN A 192.168.1.141
                               IN A 192.168.1.142
                               IN A 192.168.1.143

That means, if you connect to RAC using RACDB1 descriptor and corporate DNS you will balance the load on DNS level (using round-robin algorithm).

Each time when you are resolving the rac-scan.cluster.us.oracle.com name DNS will send you different SCAN VIP.

You can also to install Grid Infrastructure without DNS. In this case, you would use a hosts-file entry to resolve the SCAN to one IP address. Since you are using only one (and only one) SCAN VIP there is no opportunity for load balancing on SCAN level, but you are able to use load balancing across nodes VIPs described above. Second option is using GNS. Using GNS assumes you have a DHCP service for delivering IP for SCAN VIPs (not only) and corporate DNS available on your public network. During the cluster configuration you are defining the deligated subdomain and GNS VIP for GNS. Before that you should set up the DNS for subdomain deligation.In this case, three IP addresses will be acquired from a DHCP service to create the SCAN and name resolution for the SCAN will be provided by the GNS.

One interesting point is that Oracle Client only resolves SCAN name and transforms it into following  SCAN-vipN - SCAN VIPs address

  (DESCRIPTION =
   (ADDRESS = (PROTOCOL = TCP)(HOST = SCAN-vip1)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = SCAN-vip2)(PORT = 1521))
   (ADDRESS = (PROTOCOL = TCP)(HOST = SCAN-vip3)(PORT = 1521))        
  (CONNECT_DATA =
      (SERVICE_NAME = racdb)
    )
  )

As yo can see, in this case, clients will connect to the first SCAN VIP only (SCAN-vip1 in the example). For avoiding this situation using LOAD_BALANCE parameter:

RACDB1 =
 (DESCRIPTION =
 (LOAD_BALANCE=ON)
  (ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.cluster.us.oracle.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVICE_NAME = racdb)
    )
  )

According to Oracle documentation, this parameter would have 'ON' default values only if the specified DESCRIPTION_LIST tag. Therefore, for load balancing in this case, you should set the LOAD_BALANCE parameter to 'ON' state, even if the SCAN is used.

Create mirror copy of OCR/VOTING DISK online

OCR:
OCR can be created at the time of Grid Installation. It’s store information to manage Oracle cluster-ware and it’s component such as RAC database, listener, VIP,Scan IP & Services.
Minimum 1 and maximum 5 copy of OCR is possible.


Some key points about OCR handling in 11gR2 ASM:

1. We can store OCR And Voting disk on ASM or certified cluster file system.
2. We can dynamically add or replace OCR.
3. OCR can be keep in same disk-group or different disk-group
4. OCR automatic backup kept together in a single file.
5. Automatic backup of OCR happen after every four hours, end of the day, end of the week
6. must have root or sudo privilege account to manage it.
   

To find location of Current OCR:
[oracle@rac1]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     262120
Used space (kbytes)      :       2964
Available space (kbytes) :     259156
ID                       : 1390115973
Device/File Name         :      +DATA
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check bypassed due to non-privileged user


Create mirror copy of OCR online
[oracle@rac1]$ sudo ocrconfig -add +CRS
Password:

Check location of OCR after mirror copy creation:
[root@rac1]# /u01/app/11.2.0/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version                  :          3
Total space (kbytes)     :     262120
Used space (kbytes)      :       2964
Available space (kbytes) :     259156
ID                       : 1390115973
Device/File Name         :      +DATA
Device/File integrity check succeeded
Device/File Name         :       +CRS
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded

Another file to find location of OCR:
[root@rac1]# cat /etc/oracle/ocr.loc
#Device/file  getting replaced by device +CRS
ocrconfig_loc=+DATA
ocrmirrorconfig_loc=+CRS

Voting Disk: 

It manage information about node membership. Each voting disk must be accessible by all nodes in the cluster. If any node is not passing heat-beat across other note or voting disk, then that node will be evicted by Voting disk.
Minimum 1 and maximum 15 copy of voting disk is possible.

Some key points about VOTING DISK handling in 11gR2:

1. Voting disks will be kept in an ASM diskgroup
2. We can dynamically add or replace voting disk
3. Redundancy of the Voting Disk depends on the Redundancy of that diskgroup
4. Voting disk can be keep in same disk-group or different disk-group
5. Voting disk automatic backup kept together in a single file
6. Backup of Voting disk using “dd” command not supported
7. Voting Disks are mirrored across Failuregroups
8. Automatic backup of Voting disk and OCR happen after every four hours, end of the day, end of the week
9. must have root or sudo privilege account to manage it
10. vote file is not mirrored and stripped like any other files (data, redo, control, OCR etc…) and it has a fixed assignment to a failure group in a disk. 
11. Vote disk can be accessed even if ASM is not up whereas another file cannot be accessed if ASM is not up and running.
12. we cannot mix vote files on ASM and vote files on non-ASM. Instead we can dedicate either one ASM DG for all you cluster vote disk files or we can use non-ASM but never both except for initial migration to 11.2 when previous versions were on non-ASM storage. 
13. We cannot use the ‘crsctl add css votedisk’ to add a vote disk on ASM disk group or ACFS file system

A common example would be the Diskgroup DATA with redundancy normal that has at least 3 Failuregroups and contains also the Database Area of a RAC database. In this case 3 Failuregoups would each store one voting disk unstriped on one disk.

To find current location of Voting disk:

SQL> set line 200
SQL> column path format a30
SQL> select name, path, header_status from v$asm_disk; 

[oracle@rac1]$ crsctl query css votedisk
##  STATE    File Universal Id File Name Disk group
–  —–    —— —— —–——— ———
1. ONLINE   6a60a2c3510c4fbfbff62dcdc279b247 (ORCL:DATA1) [DATA]

Relocate or multiplexing Voting disk to another disk-group (With normal redundancy)
[root@rac1]# /u01/app/11.2.0/grid/bin/crsctl replace votedisk +CRS
Successful addition of voting disk afb77b2693a24f1ebfe876784103e82a.
Successful addition of voting disk 3e2542c5b1154ffdbfc8b6dea7dce390.
Successful addition of voting disk 8e0f3c5921cc4f93bf223de1465d83cc.
Successful deletion of voting disk 6a60a2c3510c4fbfbff62dcdc279b247.
Successfully replaced voting disk group with +CRS.
CRS-4266: Voting file(s) successfully replaced

New location of Voting disk:
[root@rac1]# /u01/app/11.2.0/grid/bin/crsctl query css votedisk
##  STATE    File Universal Id                File Name Disk group
–  —–    —————–                ——— ———
1. ONLINE   afb77b2693a24f1ebfe876784103e82a (ORCL:DATA2) [CRS]
2. ONLINE   3e2542c5b1154ffdbfc8b6dea7dce390 (ORCL:DATA3) [CRS]
3. ONLINE   8e0f3c5921cc4f93bf223de1465d83cc (ORCL:DATA4) [CRS]
Located 3 voting disk(s).

[root@rac1]#
[root@rac1]# /u01/app/11.2.0/grid/bin/crsctl add css votedisk +data
CRS-4671: This command is not supported for ASM diskgroups.
CRS-4000: Command Add failed, or completed with errors.

[root@rac1]# /u01/app/11.2.0/grid/bin/crsctl add css votedisk /u01/vote_acfs.dsk
CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM
[root@raclinux2 bin]#

How ohasd will know which disk contains voting disk? 

Following is the ASM disk header metadata (dumped using kfed).
ohasd use the markers between vfstart & vfend.
If the markers between vfstart & vfend are 0 then disk does *NOT* contain voting disk

[root@rac1] kfed read ‘/dev/sdi’ | grep vf
kfdhdb.vfstart: 130; 0x0ec: 0×00000060
kfdhdb.vfend: 168; 0x0f0: 0x000000d0

[root@rac1] kfed read ‘/dev/sdg’ | grep vf
kfdhdb.vfstart: 0 ; 0x0ec: 0×00000000
kfdhdb.vfend: 0 ; 0x0f0: 0×00000000

[root@rac1] kfed read ‘/dev/sdh’ | grep vf
kfdhdb.vfstart: 0 ; 0x0ec: 0×00000000
kfdhdb.vfend: 0 ; 0x0f0: 0×00000000

So from above output, disk ‘/dev/sdi’ contains voting disk.

Conclusion: The way to handle Voting Disk and OCR has changed significantly – they can be kept inside of an ASM Diskgroup especially. 

Moving the 11R2 RAC database servers across datacenter with changed IP address

1) Shutdown the CRS and DISABLE CRS auto start
2) Move the servers to new data center
3) Start the CRS – only on first node(./crsctl start crs)
4) Modify the IP details on OCR using oifcfg
5) Update the GPGNP profile in 11gR2
6) Now bounce CRS on rac1 and start the CRS on rac2
.........................................
note :

PUB.LI.CI.PP -- public ip address (eth1)

PRI.VA.TE.IP -- private ip address ( eth2)

4) Modify the IP details on OCR using oifcfg

root@rac1 bin]# ./oifcfg delif -global eth1
root@rac1 bin]# ./oifcfg setif -global eth1/PUB.LI.CI.PP:public -- New ip address netmask
root@rac1 bin]# ./oifcfg delif -global eth2
root@rac1 bin]# ./oifcfg setif -global eth2/PRI.VA.TE.IP:cluster_interconnect --- New ip address netmask
root@rac1 bin]# ./oifcfg getif
eth1 PUB.LI.CI.PP global public
eth2 PRI.VA.TE.IP global cluster_interconnect
root@rac1 bin]# ./srvctl config nodeapps

----
VIP exists: /rac1-vip/<newip>/<old netmask>/255.255.255.128/bond0, hosting node rac1

VIP exists: /rac2-vip/<newip>/<old netmask>/255.255.255.128/bond0, hosting node rac2
----

[root@rac1 bin]# ./srvctl modify nodeapps -n rac1 -A rac1-vip.us.oracle.com/255.255.255.0/eth1
[root@rac1 bin]# ./srvctl modify nodeapps -n rac2 -A rac2-vip.us.oracle.com/255.255.255.0/eth1
[root@rac1 bin]# ./srvctl config nodeapps

----
VIP exists: /rac1-vip/<New IP address>/<PUB.LI.CI.PP>/255.255.255.0/eth1, hosting node rac1

VIP exists: /rac2-vip/<New IP address>/<PUB.LI.CI.PP>/255.255.255.0/eth1, hosting node rac2
----

when try to bounce CRS on rac1 and start the CRS on rac2
Now looking at 11gR2 with same scenario of moving RAC to new datacenter.
When we Start the CRS – we got below errors when we tried starting the CRS with server running on different subnet with new IP address.
” [crsd(21366)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-44: Error in network address and interface operations Network address and interface operations error [7]]. Details at (:CRSD00111:) in /u01/app/grid/11.2.0/log/rac1/crsd/crsd.log.”

What is the difference between 10g/11gR1 and 11gR2??
As of 11.2 Grid Infrastructure, the CRS daemon (crsd.bin) now has a dependency on the private network configuration stored in the gpnp profile and OCR. If the private network is not available or its definition is incorrect, the CRSD process will not start and any subsequent changes to the OCR will be impossible NOTE:1094024.1

5) Update the GPGNP profile in 11gR2

STEPS UPDATE TO 11GR2 GRID GPGNP PROFILE
(run all the steps on rac1 as ROOT unless specified dont run on other nodes)

STEP1: Start CRS on all the nodes (though CRSD won’t come up we need gpnp deamon running on all the cluster nodes)

[root@rac1 bin]# cd $GRID_HOME/bin/
[root@rac1 bin]#./crsctl start crs
[root@rac1 bin]# ps -ef | grep gpnp

STEP 2: On rac1: As the from 11gR2 – CRS startup is based on GPNP profile we need to update profile to reflect new IP address (to view current profile infomation

[root@rac1 bin]# cd $GRID_HOME/bin/
[root@rac1 bin]# ./gpnptool get >> we can notice the old ip and interface name

STEP 3: on rac1: Take backup of existing GPNP profile before editing (we are going have temp profile XML file and do our modification there in that file)

[root@rac1 bin]#mkdir /home/oracle/gpnp
[root@rac1 bin]#export GPNPDIR=/home/oracle/gpnp
[root@rac1 bin]# ./gpnptool get -o=$GPNPDIR/profile.original
[root@rac1 bin]# cp $GPNPDIR/profile.original $GPNPDIR/p.xml

STEP 4: on rac1: Get current profile sequence number – we need to increment it while editing profile
i.e current prf_sq +1

[root@rac1 bin]# ./gpnptool getpval -p=$GPNPDIR/p.xml -prf_sq -o- ===> this output +1 will be used for prf_sq during edit command

STEP 5: on rac1: Available network interfaces

[root@rac1 bin]#./oifcfg iflist

STEP 6: on rac1: Get the current NETWORK ID from profile and crosscheck the xml file

[root@rac1 bin]# ./gpnptool getpval -p=$GPNPDIR/p.xml -net -o-

exampleOUTPUT : 

[root@rac1 bin]#./gpnptool getpval -p=$GPNPDIR/p.xml -net -o-
net1 net2

[root@rac1 bin]#more $GPNPDIR/p.xml (compare the above net1/net2 with xml file) and append below command based on " (d), (e) and (f)"

[root@rac1 bin]#./gpnptool edit -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -prf_sq=2 -net1:net_use=public -net1:net_ip=PUB.LI.CI.PP -net1:net_ada=eth3

[root@rac1 bin]#./gpnptool edit -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -prf_sq=2 -net2:net_use=cluster_interconnect -net2:net_ip=PRI.VA.TE.IP -net2:net_ada=eth4

[root@rac1 bin]#more $GPNPDIR/p.xml >> we should be seeing new ip and interface names here

STEP 7: on rac1: Sign the XML file with GPNP tool and upload it to update the current profile

[root@rac1 bin]#./gpnptool sign -p=$GPNPDIR/p.xml -o=$GPNPDIR/p.xml -ovr -w=cw-fs:peer

[root@rac1 bin]#./gpnptool put -p=$GPNPDIR/p.xml >> At this step if GPNP deamon not running then profile won't get 
updated on that node so we need to make sure it running on all the cluster nodes

STEP 8: on All the nodes : Start the CRS and check Status

[root@rac1 bin]#./gpnptool get >> we will be seeing new IP and interface name
[root@rac1 bin]#./crsctl start res ora.crsd –init
[root@rac1 bin]#./crsctl status res -t

At this point all the CRS deamons will be up (make sure listener/DB not up)

STEP 9: Now we run oifcfg getif — interesting piece of warning/error which ignorable as we edited GPNP profile and its not matching with OCR

[root@rac1 bin]# ./oifcfg getif
eth3 PUB.LI.CI.PP global public
eth4 PRI.VA.TE.IP global cluster_interconnect
Only in OCR: bond0 SCA.NV.IP.IP global public
Only in OCR: bond2 YYY.YY.yy.yy global cluster_interconnect
PRIF-30: Network information in OCR and GPnP profile differs

STEP 10: update OCR configuration using oifcfg

[root@rac1 bin]# ./oifcfg delif -global bond0
[root@rac1 bin]# ./oifcfg delif -global bond2
[root@rac1 bin]# ./oifcfg setif -global eth3/PUB.LI.CI.PP:public
[root@rac1 bin]#./oifcfg setif -global eth4/PRI.VA.TE.IP:cluster_interconnect

STEP 11: Modifing SCAN IP is very simple compare to VIP/Interconnect changes.

[root@rac1 bin]# ./srvctl config scan -- Verifiying existing configuration
SCAN name: node-scan, Network: 1/PUB.LI.CI.PP/255.255.255.0/eth3
SCAN VIP name: scan1, IP: /node-scan/SCA.NV.IP.IP
SCAN VIP name: scan2, IP: /node-scan/SCA.NV.IP.IP
SCAN VIP name: scan3, IP: /node-scan/SCA.NV.IP.IP

[root@rac1 bin]# ./srvctl config scan_listener -- Verifiying existing configuration
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521

[root@rac1 bin]# ./srvctl status scan_listener -- Making sure scan listener is down
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is not running
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running

[root@rac1 bin]# ./srvctl modify scan -n node-scan -- Modifying SCAN IP's on OCR, this command will pickup the new ip addres from DNS and updates OCR

[root@rac1 bin]# ./srvctl config scan -- Verifying new ip's registered on OCR
SCAN name: node-scan, Network: 1/PUB.LI.CI.PP/255.255.255.0/eth3
SCAN VIP name: scan1, IP: /node-scan/PUB.LI.CI.PP
SCAN VIP name: scan2, IP: /node-scan/PUB.LI.CI.PP
SCAN VIP name: scan3, IP: /node-scan/PUB.LI.CI.PP

Do clean bounce of CRS on all the nodes using

• crsctl stop crs
  
• crsctl start crs
  
• crsctl enable crs
  

This concludes 11gR2 clusterware IP address modified

OIFCFG COMMANDS 11GR2

The Oracle Interface Configuration Tool (oifcfg) is used define and administer network interfaces such as the public and private interfaces.
The usage of oifcfg command is

1. Allocate/ Deallocate network interfaces to components
2. Direct components to use specific network interfaces
3. Display information of component network configuration
   

The following table lists the different oifcfg commands and their purpose:

Oifcfg command	Description
Iflist [-p [-n]]	Displays the interfaces available to the cluster on the node that the oifcfg command was executed on. The –p parameter displays the type of interface which can be PRIVATE, PUBLIC or UNKNOWN.
Setif	Assigns the interface type (PUBLIC or PRIVATE) to a given interface.
Getif	Displays the interfaces that have been allocated to the cluster, along with the type of interface (public, cluster_interconnect)
Delif	This command is used to remove a stored network configuration.

The following lists the parameters can be used with the oifcfg command:

Parameter_name	Description
-node node_name	This parameter defines the node that the command should be executed on. Valid with the setif, getif, and the delif commands
-global	Indicates that the interface should be stored as a global interface for the entire cluster. Used with the setif, getif and delif commands.
-if interface_name	This is the name of the interface on the system. Used with the getif, setif and delif commands. The interface name takes the form: interface_name/subnet:interface_type. An example might be eth0/192.168.1.101:cluster_interconnect.
Subnet	This is the subnet number for the interface. Used with setif, getif and delif commands.
-type interface_type	This is the type of interface. Valid entries are public or cluster_interconnect. Used with the getif and setif commands.
-help	Print the help information.

Examples of Using the Oicfg Command
Example 1 
 Listing the Names of Network Interfaces
You can use OIFCFG to list the interface names and the subnets of all of the interfaces available on the local node by executing the iflist keyword, as shown in this example:

oifcfg iflist
hme0     139.185.141.0
qfe0     204.152.65.0

Example 2   

Retrieving Network Information

You can also retrieve specific OIFCFG information with a getif command.

oifcfg getif [ [-global | -node nodename] [-if if_name[/subnet]] [-type if_type] ]

For example, after you install Oracle Clusterware, you can verify that the public and cluster interconnect have been set to the desired values by entering the following commands as root:

$ oifcfg getif

This command should return values for global public and global cluster_interconnect. For example:

en0 144.25.68.0 global public
hme0 139.185.141.0 global cluster_interconnect

Example 3   

Storing a New Global Interface

To store a new interface, use the setif keyword. For example, to store the interface hme0, with the subnet 139.185.141.0, as a global interface (to be used as an interconnect for all of the Oracle RAC instances in your cluster), you would use the command:

oifcfg setif -global hme0/139.185.141.0:cluster_interconnect

For a cluster interconnect that exists between only two nodes, for example rac1 and rac2, you could create the cms0 interface with the following commands, assuming 139.185.142.0 is the subnet number for the interconnect on rac1 and rac2, respectively:

oifcfg setif -global cms0/139.185.142.0:cluster_interconnect

Example 4   

Deleting the Stored Interface

Use the OIFCFG delif command to delete the stored configuration for global or node-specific interfaces. A specific node-specific or global interface can be deleted by supplying the interface name, with an optional subnet, on the command line. Without the -node or -global options, the delif keyword deletes either the given interface or all of the global and node-specific interfaces on all of the nodes in the cluster.
For example, the following command deletes the global interface named qfe0 for the subnet 204.152.65.0:

oifcfg delif -global qfe0/204.152.65.0

The following command deletes all of the global interfaces assigned with OIFCFG:

oifcfg delif -global

CRSCTL COMMAND IN 11GR2

CRSCTL Command Syntax Overview
CRSCTL command has

Command	Description
check	Displays the Oracle Restart status.
config	Displays the Oracle Restart configuration.
disable	Disables automatic restart of Oracle Restart.
enable	Enables automatic restart of Oracle Restart.
start	Starts Oracle Restart.
stop	Stops Oracle Restart.

1) Do not use CRSCTL commands on Oracle entities (such as resources, resource types, and server pools) that have names beginning with ora unless you are directed to do so by Oracle Support. The Server Control utility (SRVCTL) is the correct utility to use on Oracle entities.

2) in order to run crsctl command you must be connected as oracle user (for Linux, AIX, Unix servers)

3)  There are 3 categories of CRSCTL commands:

a)Dual Environment CRSCTL Commands
The following lists the various crsctl commands and their functions are available for both Oracle RAC and Oracle Restart environments:

crsctl add resource
crsctl add type
crsctl check css
crsctl delete resource
crsctl delete type
crsctl get hostname
crsctl getperm resource
crsctl getperm type
crsctl modify resource
crsctl modify type
crsctl setperm resource
crsctl setperm type
crsctl start resource
crsctl status resource
crsctl status type
crsctl stop resource

b)Oracle RAC Environment CRSCTL Commands. The commands listed in this section manage the Oracle Clusterware stack in an Oracle RAC environment, which consists of the following:  

1. Oracle Clusterware, the member nodes and server pools
2. Oracle ASM (if installed)
3. Cluster Synchronization Services
4. Cluster Time Synchronization Services

crsctl add crs administrator
crsctl add css votedisk
crsctl add serverpool
crsctl check cluster
crsctl check crs
crsctl check resource
crsctl check ctss
crsctl config crs
crsctl delete crs administrator
crsctl delete css votedisk
crsctl delete node
crsctl delete serverpool
crsctl disable crs
crsctl enable crs
crsctl get css
crsctl get css ipmiaddr
crsctl get nodename
crsctl getperm serverpool
crsctl lsmodules
crsctl modify serverpool
crsctl pin css
crsctl query crs administrator
crsctl query crs activeversion
crsctl query crs releaseversion
crsctl query crs softwareversion
crsctl query css ipmidevice
crsctl query css votedisk
crsctl relocate resource
crsctl relocate server
crsctl replace discoverystring
crsctl replace votedisk
crsctl set css
crsctl set css ipmiaddr
crsctl set css ipmiadmin
crsctl setperm serverpool
crsctl start cluster
crsctl start crs
crsctl status server
crsctl status serverpool
crsctl stop cluster
crsctl stop crs
crsctl unpin css
crsctl unset css

     c)Oracle Restart Environment CRSCTL Commands:
        These commands only impact the cluster node that they are run on

1. Oracle High Availability Services daemon (ohasd)
2. Oracle ASM 
3. Cluster Synchronization Services (OSS) if ASM is installed                      

crsctl check has
crsctl config has
crsctl disable has
crsctl enable has
crsctl query has releaseversion
crsctl query has softwareversion
crsctl start has
crsctl stop has

NOTE:

The following commands are deprecated in Oracle Clusterware 11g release 2

crs_stat
crs_register
crs_unregister
crs_start
crs_stop
crs_getperm
crs_profile
crs_relocate
crs_setperm

crsctl check crsd
crsctl check cssd
crsctl check evmd
crsctl debug log
crsctl set css votedisk
crsctl start resources
crsctl stop resources

SRVCTL COMMANDS IN 11gR2

srvctl  command object options

 

command

 
config            Displays the Oracle Restart configuration for a component.
relocate        Relocates a service from one named instance to another named
instance

add                 Adds a component to the Oracle Restart configuration.

remove           Removes a component from the Oracle Restart configuration.
modify           Modifies the Oracle Restart configuration for a component. 

enable            Re-enables management by Oracle Restart for a component.
disable           Disables management by Oracle Restart for a component.

start                Starts the specified component.
status              Displays the running status of the specified component.
stop                 Stops the specified component.

setenv             Sets environment variables in Oracle Restart configuration for a  database,  Oracle ASM instance, or listener.
getenv            Displays environment variables in the Oracle Restart configuration for a database, Oracle ASM instance, or listener.

unsetenv        Unsets environment variables in the Oracle Restart configuration for a database, Oracle ASM instance, or listener. 

Component(object)  Abbreviation                 Description
asm                         asm                                Oracle ASM instance
diskgroup               dg                                  Oracle ASM disk group
filesystem               filesystem                     Oracle ASM file system
database                 db                                   Database
Instance                    Instance                         Instance
home                       home                             Oracle home or Oracle Clusterware home
listener                     lsnr                               Oracle Net listener
service                     serv                               Database service
ons, eons                 ons, eons                       Oracle Notification Services (ONS)

nodeapps
vip
srvpool
server
scan
scan_listener
oc4j
gns

Commands	Objects	Comment
srvctl add srvctl modify srvctl remove	instance database service nodeapps asm ons listener	The OCR is modified.
srvctl relocate	service scan scan_listener gns oc4j server	You can reallocate a service from one named instance to another named instance.
srvctl start srvctl stop srvctl status	instance asm database diskgroup home service ons nodeapps
srvctl disable srvctl enable	instance database service asm diskgroup listener	enable = when the server restart the resource must be restarted disable = when the server restart the resource must NOT be restarted (perhaps we are working for some maintenance tasks)
srvctl config	database service asm nodeapps service ons	Lists configuration information from the OCR (Oracle Cluster Registry).
srvctl getenv srvctl setenv srvctl unsetenv	instance database service nodeapps asm listener	srvctl getenv = displays the environment variables stored in the OCR for target. srvctl setenv = allows these variables to be set srvctl unsetenv = allows these variables to be unset

The following abbrevations properties may change based on the usage of commnad
-d   :db_unique_name

-a :Display additional information

-v :Verbose output

-g :diskgroup_name 

-t :name_list (Comma-delimited list of names of environment variables to display.

     If omitted, SRVCTL displays all configured environment variables)

-t stop_options (SHUTDOWN command options for the database(Ex: NORMAL,  

              TRANSACTIONAL, IMMEDIATE, or ABORT). Default is IMMEDIATE.

-u :oracle_user 

-o :oracle home

-o :start_options (Options for database startup(EX: OPEN, MOUNT, NOMOUNT and so on)

                      if the database must be started first

-y :Suppresses the confirmation prompt and removes immediately

The frequently used SRVCTL commands are:

srvctl start database -d DBname
srvctl stop database -d DBname

If you don't know the DBname you might run  select name from v$database;

srvctl start instance -d DBname -i INSTANCEname
srvctl stop instance -d DBname -i INSTANCEname

srvctl start instance -d DBname -i INSTANCEname
srvctl stop instance -d DBname -i INSTANCEname

srvctl status database -d DBname

srvctl status instance -d DBname -i INSTANCEname

srvctl enable database -d DBname

srvctl disable database -d DBname

srvctl enable instance -d DBname -i INSTANCEname

srvctl disable instance -d DBname -i INSTANCEname

srvctl config database -d DBname      -> to get some information about the database from OCR.

To determine the VIP hostname, VIP address, VIP subnet mask, VIP interface names -

srvctl getenv nodeaps
srvctl config nodeapps -a
srvctl status nodeapps -n NODEname

Service Related Commands with example

Create Service
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/db_1 export PATH=$ORACLE_HOME/bin:$PATH

Adding a Database Service
srvctl add service -d <database_name> -s <Service_name> -r “<preferred list>”

$ srvctl add service -d ORCL -s SERVICE2 -r “RACl, RAC2″ -a “RAC3,RAC4″

Adding the ASM using Listener 

srvctl add asm [-l listener_name [-p spfile] [-d asm_diskstring]]

Starting a Database Service
srvctl start service -d <database_name> [-s "<service_name_list>" [-i <inst_name>]] [-0 <start_options>] [-c <connect_str> | -q]

$ srvctl start service -d ORCL -s “SERVICEl, SERVICE2″

Stopping a Database Service
srvctl stop service -d <database_name> [-s "<service_name_list>" [-i <inst_name>]] [-C <connect_str> | -q] [-f]

$ srvctl stop service -d ORCL -s “SERVICE2,SERVICE3″ -f

Checking the Current Database Service Configuration
srvctl config service -d <database_name> [-s <service_name>] [-a] [-S <level>]

$ srvctl config service -d ORCL -a
The -a option includes information about the configuration of TAF for the database service

Checking Current Database Service Status
srvctl status service -d <name> -s “<service_name_list>” [-f] [-v] [-S <level>]

$ srvctl status service -d ORCL -s “SERVICEl, SERVICE4″

Enabling and Disabling a Database Service
srvctl disable service -d <database_name> -s “<service_name_list>” [-i <inst_name>]

$ srvctl disable service -d ORCL -s SERVICE2 -i RAC4

srvctl enable service -d <database_name> -s “<service_name_list>” [-i <inst_name>]

$ srvctl enable service -d ORCL -s SERVICE2 -i RAC4

Removing a Database Service
srvctl remove service -d <database_name> -s <service_name> [ - i <inst_name>] [-f]

$ srvctl remove service -d ORCL -s SERVICE4

Relocating a Database Service
srvctl relocate service -d <database_name> -s <service_name> -i <old_inst_name> -r <new_inst_name> [-f]

$ srvctl relocate service -d ORCL -s SERVICES -i RAC2 -r RAC4

To upgrades the configuration of a database and all of its services to version of database home 

$ srvctl upgrade database -d db_unique_name -o 11gR2_oracle_home