Search

Sunday, November 18, 2012

SRVCONFIG command in RAC

$ srvconfig -help   (or) srvconfig -?
Usage: srvconfig [options]

where options include:
        -help        display command-line help message
        -?           same as -help option
        -init        initialize cluster registry (if not already initialized)
        -init -f     force intialization of configuration even if initialized
        -exp <file>  export cluster registry contents to given text file
        -imp <file>  import given text file contents to the cluster registry
        -upgrade -dbname <database> -orahome <oracle_home_path> upgrade the database configuration to Oracle Database 10.2
        -downgrade -dbname <database> -orahome <oracle_home_path> -version <versionstring> downgrade the database configuration to Oracle Database 10.0 or 9.2 based on value of versionstring

To exports the contents of the configuration information or cluster registry information to file
srvconfig -exp <file_name>

To Import the contents of the configuration information or cluster registry information from file to cluster

$srvconfig -imp file_name

To Initialize cluster registry (if not already initialized)
srvconfig -init

To Force initialization of configuration even if initialized
srvconfig -init -f -- force initialization of configuration even if initialized

To Upgrade the database configuration
$srvconfig -upgrade -dbname db_name -orahome ORACLE_HOME

To downgrade the database configuration
$srvconfig -downgrade -dbname db_name -orahome ORACLE_HOME -version version_str

Olsnode Commands in Oracle RAC

[oracle@rac3 bin]$ ./olsnodes -h
Usage: olsnodes [ [-n] [-i] [-s] [-t] [<node> | -l [-p]] | [-c] ] [-g] [-v]
        where
                -n print node number with the node name
                -p print private interconnect address for the local node
                -i print virtual IP address with the node name
                <node> print information for the specified node
                -l print information for the local node
                -s print node status - active or inactive
                -t print node type - pinned or unpinned
                -g turn on logging
                -v Run in debug mode; use at direction of Oracle Support only.
                -c print clusterware name

To view list nodes in cluster & its number
[oracle@rac3 bin]$ ./olsnodes -n
rac1    1
rac3    2

To view nodes & VIP address
[oracle@rac3 bin]$ ./olsnodes -i
rak1    192.168.1.181
rac3    192.168.1.183

[oracle@rac3 bin]$ ./olsnodes rac1 -v
lang init : Initializing LXL global
main: Initializing CLSS context
node: retrieving information for node=rac1
node: No of cluster members configured = 256
getnodenum: Retrieving the node number for node = rac1
getnodenum: node number = 1
print data: Printing the node data
rac1
main: olsnodes executed successfully
term: Terminating LSF


CRSCTL commands in Oracle RAC 10g RAC/11gr2 RAC



crsctl - Cluster Ready Service Control

To Get CRSCTL  help
$crsctl -h

To Get Active Cluster Version
$crsctl query crs activeversion

To Get cluster version of given node
$crsctl query crs softwareversion [node_name]

Ex:$crsctl query crs softwareversion rac1

#crsctl start crs
#crsctl stop crs

(or)

#/etc/init.d/init.crs start
#/etc/init.d/init.crs stop

#crsctl enable crs
#crsctl disable crs

(or)

#/etc/init.d/init.crs enable
#/etc/init.d/init.crs disable

To get cluster stack health status
$crsctl check crs

To get the viability of CSS across nodes
$crsctl check cluster [-node node_name]   -- 11gR2 RACcommand

#crsctl start cluster -n HostName -- 11g R2
#crsctl stop cluster -n HostName -- 11g R2
#crsctl stop cluster -all  -- 11g R2

$ps -ef | grep d.bin
$crsctl check cssd
$crsctl check crsd
$crsctl check evmd
$crsctl check oprocd
$crsctl check ctss

#/etc/init.d/init.cssd stop
#/etc/init.d/init.cssd start

#/etc/rc.d/init.d/init.evmd
#/etc/rc.d/init.d/init.cssd
#/etc/rc.d/init.d/init.crsd

#mv /etc/rc3.d/S96init.cssd /etc/rc3.d/_S96init.cssd -- to stop cssd from autostarting after reboot

#crsctl check css votedisk
#crsctl query css votedisk -- lists the voting disks used by CSS command in 11gR2

#crsctl add css votedisk PATH
#crsctl add css votedisk PATH -force -- if Clusterware is not running
#crsctl delete css votedisk PATH
#crsctl delete css votedisk PATH -force -- if Clusterware is not running

#crsctl set css parameter_name value -- set parameters on OCR
#crsctl set css misscount 100
#crsctl unset css parameter_name -- sets CSS parameter to its default
#crsctl unset css misscount
#crsctl get css parameter_name -- gets the value of a CSS parameter
#crsctl get css disktimeout
#crsctl get css misscount
#crsctl get css reboottime

#crsctl start resources -- starts Clusterware resources
#crsctl start resource ora.DATA.dg
#crsctl stop resources -- stops Clusterware resources

$crsctl status resource
$crsctl status resource -t
$crsctl stat resource -t
#crsctl lsmodules crs -- lists CRS modules that can be used for debugging( need root privilage)
List CRSD Debug Module: AGENT
List CRSD Debug Module: AGFW
List CRSD Debug Module: CLSFRAME
List CRSD Debug Module: CLSVER
List CRSD Debug Module: CLUCLS
List CRSD Debug Module: COMMCRS
List CRSD Debug Module: COMMNS
List CRSD Debug Module: CRSAPP
List CRSD Debug Module: CRSCCL
List CRSD Debug Module: CRSCEVT
List CRSD Debug Module: CRSCOMM
List CRSD Debug Module: CRSD
List CRSD Debug Module: CRSEVT
List CRSD Debug Module: CRSMAIN
List CRSD Debug Module: CRSOCR
List CRSD Debug Module: CRSPE
List CRSD Debug Module: CRSPLACE
List CRSD Debug Module: CRSRES
List CRSD Debug Module: CRSRPT
List CRSD Debug Module: CRSRTI
List CRSD Debug Module: CRSSE
List CRSD Debug Module: CRSSEC
List CRSD Debug Module: CRSTIMER
List CRSD Debug Module: CRSUI
List CRSD Debug Module: CSSCLNT
List CRSD Debug Module: OCRAPI
List CRSD Debug Module: OCRASM
List CRSD Debug Module: OCRCAC
List CRSD Debug Module: OCRCLI
List CRSD Debug Module: OCRMAS
List CRSD Debug Module: OCRMSG
List CRSD Debug Module: OCROSD
List CRSD Debug Module: OCRRAW
List CRSD Debug Module: OCRSRV
List CRSD Debug Module: OCRUTL
List CRSD Debug Module: SuiteTes
List CRSD Debug Module: UiServer

$crsctl lsmodules css -- lists CSS modules that can be used for debugging
The following are the Cluster Synchronization Services modules::
    CSSD
    COMMCRS
    COMMNS
    CLSF
    SKGFD

$crsctl lsmodules evm -- lists EVM modules that can be used for debugging
The following are the Cluster Synchronization Services modules::
    CSSD
    COMMCRS
    COMMNS
    CLSF
    SKGFD

$crsctl start has   (HAS - High Availability Services)
$crsctl stop has

$crsctl check has
CRS-4638 Oracle High Availability Service is online

OCR Modules -- cannot be listed with crsctl lsmodules command
OCRAPI
OCRCLI
OCRSRV
OCRMAS
OCRMSG
OCRCAC
OCRRAW
OCRUTL
OCROSD

#crsctl debug statedump crs -- dumps state info for crs objects
#crsctl debug statedump css -- dumps state info for css objects
#crsctl debug statedump evm -- dumps state info for evm objects

#crsctl debug log crs [module:level]{,module:level} ...

-- Turns on debugging for CRS
#crsctl debug log crs CRSEVT:5,CRSAPP:5,CRSTIMER:5,CRSRES:5,CRSRTI:1,CRSCOMM:2
#crsctl debug log css [module:level]{,module:level} ...

-- Turns on debugging for CSS
#crsctl debug log css CSSD:1
#crsctl debug log evm [module:level]{,module:level} ...
-- Turns on debugging for EVM
#crsctl debug log evm EVMCOMM:1

#crsctl debug trace crs -- dumps CRS in-memory tracing cache
#crsctl debug trace css -- dumps CSS in-memory tracing cache
#crsctl debug trace evm -- dumps EVM in-memory tracing cache

#crsctl debug log res resource_name:level -- turns on debugging for resources
#crsctl debug log res "ora.lnx04.vip:1"

#crsctl trace all_the_above_commands -- tracing by adding a "trace" argument.
#crsctl trace check css
#crsctl backup -h
#crsctl backup css votedisk

Here is the list of the options for CRSCTL in 11gR2:
       crsctl add       - add a resource, type or other entity
       crsctl backup    - back up voting disk for CSS
       crsctl check     - check a service, resource or other entity
       crsctl config    - output autostart configuration
       crsctl debug     - obtain or modify debug state
       crsctl delete    - delete a resource, type or other entity
       crsctl disable   - disable autostart
       crsctl discover  - discover DHCP server
       crsctl enable    - enable autostart
       crsctl get       - get an entity value
       crsctl getperm   - get entity permissions
       crsctl lsmodules - list debug modules
       crsctl modify    - modify a resource, type or other entity
       crsctl query     - query service state
       crsctl pin       - Pin the nodes in the nodelist
       crsctl relocate  - relocate a resource, server or other entity
       crsctl replace   - replaces the location of voting files
       crsctl release   - release a DHCP lease
       crsctl request   - request a DHCP lease
       crsctl setperm   - set entity permissions
       crsctl set       - set an entity value
       crsctl start     - start a resource, server or other entity
       crsctl status    - get status of a resource or other entity
       crsctl stop      - stop a resource, server or other entity
       crsctl unpin     - unpin the nodes in the nodelist
       crsctl unset     - unset a entity value, restoring its default

How do I identify the voting disk/file location?
#crsctl query css votedisk

How to take backup of voting file/disk?
crsctl backup css votedisk

How to Change the public IP Address in an Oracle RAC


 Though this task is well documented (see [1] and [2]) there are still some pitfalls and quirks that can cause problems. Such as..
 Should a particular task be done with the root or oracle user? Should it be performed on both nodes?
This is a step-by-step description how to perform the change of the public IP address in the Oracle two node RAC. The Oracle version is 10.2.0.3. The OS used in this example is Linux Centos 2.6.9-42.ELsmp

Note that due to the change of the private IP address the virtual IP address (VIP) may be required to change as well as it must remain in the same subnet as the public IP address.

The configuration



Host
Current IP Address
Changed IP Address
Private IP
hst1
192.168.1.101  
192.168.2.121  
Private IP
hst2
192.168.1.100  
192.168.2.120  
VIP
hst1-vip
192.168.1.111  
192.168.2.111  
VIP
hst2-vip
192.168.1.110  
192.168.2.110  



Note that the actual value of IP addresses and host names was changed.

The basic steps


The change of the private IP address is performed in the following basic steps

1) Shut down everything except the CRS stack
2) Change the public interface
3) Modify the VIP address
4) Shut down CRS
5) Modify IP address on OS level and reconfigure /etc/hosts, listener,..
6) Restart

The order of the steps is very important. E.g. step 3) can be performed only if the CRS stack is up. If you perform step 5) prematurely, CRS will go down automatically (probably trying some reboots beforehand). This will effectively block step 3) to be done.

Shut Down Everything Except the CRS Stack


The database and nodeapps (on all nodes) are stopped.


[oracle@hst1 ~]$ srvctl stop database -d mydb
[oracle@hst1 ~]$ srvctl stop nodeapps -n hst1
[oracle@hst1 ~]$ srvctl stop nodeapps -n hst2

Note that if you use ASM, it must be stopped as well.

After that we verify the status.

[oracle@hst1 ~]$ srvctl status database -d mydb
Instance MYDB1 is not running on node hst2
Instance MYDB2 is not running on node hst1
[oracle@hst1 ~]$ srvctl status nodeapps -n hst1
VIP is not running on node: hst1
GSD is not running on node: hst1
Listener is not running on node: hst1
ONS daemon is not running on node: hst1
[oracle@hst1 ~]$ srvctl status nodeapps -n hst2
VIP is not running on node: hst2
GSD is not running on node: hst2
Listener is not running on node: hst2
ONS daemon is not running on node: hst2

Everything is OK, we can go to the next step.

Change the Public Interface


First let us have a look on the actual status

[oracle@hst1 ~]$ oifcfg getif
eth0  192.168.1.0  global  public
eth1  192.168.2.0  global  cluster_interconnect

We need to change the interface eth0.
As there is no modify command, we will delete and redefine the interface.


[oraclu@hst1 ~]$ oifcfg delif -global eth0
[oraclu@hst1 ~]$ oifcfg setif -global eth0/192.168.2.0:public

the exact syntax of the setif command can be found in [3]

Note that the CRS installation user (here oraclu) must be used for this operation. Otherwise an error message is issued:

PROC-5: User does not have permission to perform a cluster registry operation on
 this key. Authentication error [User [oraclu] does not match with initialized u
ser] [0]
PRIF-11: cluster registry error

Finally, we check if the action was done successfully

[oracle@hst1 ~]$ oifcfg getif
eth0  192.168.2.0  global  public
eth1  192.168.2.0  global  cluster_interconnect


Modify the VIP Address


As already mentioned, due to the fact that we changed the subnet of the public IP address, we must change the VIP address as well.
The following modify statement should be used.

[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst2 -A 192.168.2.110/255.255.255.0/eth0
[root@hst1 ~]# /appl/oracle/product/10.2.0/db_1/bin/srvctl modify nodeapps -n hst1 -A 192.168.2.111/255.255.255.0/eth0

Note that the root user should be used for this action.
The variable ORACLE_HOHE must be initialised.
Otherwise one of the following errors will be raised.

PRKO-2117 : This command should be executed as the system privilege user.


****ORACLE_HOME environment variable not set!
    ORACLE_HOME should be set to the main
    directory that contains Oracle products.
    Set and export ORACLE_HOME, then re-run.


Shut Down CRS


The stop command must be performed on all nodes:

crsctl stop crs

Modify the IP Address on OS Level


Modify the public IP address (eth0). In Centos with application / system setting / network
Perform the change on all nodes.
Modify /etc/hosts and listener.ora files if required.

Restart


Reboot all nodes and verify the status.
Shortly after reboot we can see...

[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE
          inet addr:192.168.2.120  Bcast:192.168.2.255  Mask:255.255.255.0
eth0:1    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE
          inet addr:192.168.2.111  Bcast:192.168.2.255  Mask:255.255.255.0
eth0:2    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE
          inet addr:192.168.2.110  Bcast:192.168.2.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F1
          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

This means both VIP's eth0:1 and eth0:2 are switched to one host. This is not good, normally this appears if one host is down and the second host takes over the VIP.

Fortunately, after a few seconds we see..

[oracle@hst2 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE
          inet addr:192.168.2.120  Bcast:192.168.2.255  Mask:255.255.255.0
eth0:2    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:FE
          inet addr:192.168.2.110  Bcast:192.168.2.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F1
          inet addr:192.168.2.100  Bcast:192.168.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

[oracle@hst1 ~]$ /sbin/ifconfig -a | egrep '(eth|Mask)'
eth0      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F2
          inet addr:192.168.2.121  Bcast:192.168.2.255  Mask:255.255.255.0
eth0:1    Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F2
          inet addr:192.168.2.111  Bcast:192.168.2.255  Mask:255.255.255.0
eth1      Link encap:Ethernet  HWaddr CA:FE:CA:FE:CA:F3
          inet addr:192.168.2.101  Bcast:192.168.2.255  Mask:255.255.255.0
          inet addr:127.0.0.1  Mask:255.0.0.0

I.e. the VIP's are distributed correctly.
The configuration of IP addresses is as required.
Let’s have a final look at the status of the RAC.

[oracle@hst1 ~]$ srvctl status database -d mydb
Instance MYDB1 is running on node hst2
Instance MYDB2 is running on node hst1

[oracle@hst1 ~]$ srvctl status nodeapps -n hst1
VIP is running on node: hst1
GSD is running on node: hst1
Listener is running on node: hst1
ONS daemon is running on node: hst1

[oracle@hst2 ~]$ srvctl status database -d mydb
Instance MYDB1 is running on node hst2
Instance MYDB2 is running on node hst1
[oracle@hst2 ~]$ srvctl status nodeapps -n hst1
VIP is running on node: hst1
GSD is running on node: hst1
Listener is running on node: hst1
ONS daemon is running on node: hst1


[root@hst1 ~]#  /appl/oraclu/product/10.2.0/crs/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....B1.inst application    ONLINE    ONLINE    hst2
ora....B2.inst application    ONLINE    ONLINE    hst1
ora....DB1.srv application    ONLINE    ONLINE    hst2
ora....DB2.srv application    ONLINE    ONLINE    hst1
ora....BTAF.cs application    ONLINE    ONLINE    hst2
ora.MYDB.db    application    ONLINE    ONLINE    hst2
ora....B2.lsnr application    ONLINE    ONLINE    hst2
ora.hst2.gsd   application    ONLINE    ONLINE    hst2
ora.hst2.ons   application    ONLINE    ONLINE    hst2
ora.hst2.vip   application    ONLINE    ONLINE    hst2
ora....B1.lsnr application    ONLINE    ONLINE    hst1
ora.hst1.gsd   application    ONLINE    ONLINE    hst1
ora.hst1.ons   application    ONLINE    ONLINE    hst1
ora.hst1.vip   application    ONLINE    ONLINE    hst1


References


[2] Metalink Note:276434.1

[3] Metalink Note:283684.1
https://metalink.oracle.com/metalink/plsql/showdoc?db=NOT&id=283684.1&blackframe=1

[4] refrence
http://www.db-nemec.com/RAC_IP_Change.html