Search

Friday, August 12, 2016

OHASD does not start - OHASD RAC ISSUE



OHASD does not start


Understanding CW startup configuration in OEL 6  
OHASD Script location
[root@grac41 init.d]# find /etc |grep S96
/etc/rc.d/rc5.d/S96ohasd
/etc/rc.d/rc3.d/S96ohasd
[root@grac41 init.d]# ls -l /etc/rc.d/rc5.d/S96ohasd
lrwxrwxrwx. 1 root root 17 May  4 10:57 /etc/rc.d/rc5.d/S96ohasd -> /etc/init.d/ohasd
[root@grac41 init.d]# ls -l /etc/rc.d/rc3.d/S96ohasd
lrwxrwxrwx. 1 root root 17 May  4 10:57 /etc/rc.d/rc3.d/S96ohasd -> /etc/init.d/ohasd
--> Run level 3 and 5 should start ohasd daemon

Check status of init.ohasd process
[root@grac41 bin]# more /etc/init/oracle-ohasd.conf
# Copyright (c) 2001, 2011, Oracle and/or its affiliates. All rights reserved.
#
# Oracle OHASD startup
start on runlevel [35]
stop  on runlevel [!35]
respawn
exec /etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null

List current PID
[root@grac41 Desktop]#  initctl list | grep oracle-ohasd
oracle-ohasd start/running, process 27558
[root@grac41 Desktop]# ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root     27558     1  0  80   0 -  2878 wait   07:01 ?        00:00:02 /bin/sh /etc/init.d/init.ohasd run

Case #1 : OHASD does not start
Check your runlevel, running init.ohasd process and clusterware configuration
  # who -r
         run-level 5  2014-05-19 14:48
  # ps -elf | egrep "PID|d.bin|ohas" | grep -v grep
  F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
  4 S root      6098     1  0  80   0 -  2846 wait   04:44 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
  # crsctl config crs
  CRS-4622: Oracle High Availability Services autostart is enabled.

Case #2 : OLR not accessilbe - CW doesn't start - Error CRS-4124
Reported error in ocssd.log               :
Reported Clusterware Error in CW alert.log: CRS-0704:Oracle High Availability Service aborted due to Oracle Local Registry error
                                            [PROCL-33: Oracle Local Registry is not configured Storage layer error
                                            [Error opening olr.loc file. No such file or directory] [2]].
                                            Details at (:OHAS00106:) in /u01/app/11204/grid/log/grac41/ohasd/ohasd.log.
Testing scenario :
# mv  /etc/oracle/olr.loc  /etc/oracle/olr.loc_bck
# crsctl start crs
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

Clusterware status :
# crsi
NAME                           TARGET     STATE           SERVER       STATE_DETAILS   
-------------------------      ---------- ----------      ------------ ------------------
                               CRS-4639:  Could           not          contact Oracle
                               CRS-4000:  Command         Status       failed, or
[root@grac41 Desktop]# ps -elf | egrep "PID|d.bin|ohas" | grep -v grep
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root      6098     1  0  80   0 -  2846 wait   04:44 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
--> OHASD.bin does not start

Tracefile Details:
./grac41/alertgrac41.log
[ohasd(20436)]CRS-0704:Oracle High Availability Service aborted due to Oracle Local Registry error
     [PROCL-33: Oracle Local Registry is not configured Storage layer error [Error opening olr.loc file. No such file or directory] [2]].
     Details at (:OHAS00106:) in /u01/app/11204/grid/log/grac41/ohasd/ohasd.log.

./grac41/ohasd/ohasd.log
2014-05-11 11:45:42.892: [  CRSOCR][149608224] OCR context init failure.  Error: PROCL-33: Oracle Local Registry is not configured Storage layer error
                           [Error opening olr.loc file. No such file or directory] [2]
2014-05-11 11:45:42.893: [ default][149608224] Created alert : (:OHAS00106:) :  OLR initialization failed, error:
                            PROCL-33: Oracle Local Registry is not configured Storage layer error
                         [Error opening olr.loc file. No such file or directory] [2]
2014-05-11 11:45:42.893: [ default][149608224][PANIC] OHASD exiting; Could not init OLR
2014-05-11 11:45:42.893: [ default][149608224] Done.

OS log : /var/log/messages
May 24 09:21:53 grac41 clsecho: /etc/init.d/init.ohasd: Ohasd restarts 11 times in 2 seconds.
May 24 09:21:53 grac41 clsecho: /etc/init.d/init.ohasd: Ohasd restarts too rapidly. Stop auto-restarting.

Debugging steps
Verify your local Cluster repository
# ocrcheck -local -config
PROTL-604: Failed to retrieve the configured location of the local registry
Error opening olr.loc file. No such file or directory

# ocrcheck -local
PROTL-601: Failed to initialize ocrcheck
PROCL-33: Oracle Local Registry is not configured Storage layer error [Error opening olr.loc file. No such file or directory] [2]

# ls -l /etc/oracle/olr.loc
ls: cannot access /etc/oracle/olr.loc: No such file or directory

Note a working OLR should look like:
#  more /etc/oracle/olr.loc
olrconfig_loc=/u01/app/11204/grid/cdata/grac41.olr
crs_home=/u01/app/11204/grid
# ls -l /u01/app/11204/grid/cdata/grac41.olr
-rw-------. 1 root oinstall 272756736 May 24 09:15 /u01/app/11204/grid/cdata/grac41.olr

Verify your OLR configuration with cluvfy
[grid@grac41 ~]$  cluvfy comp olr -verbose
 ERROR:
Oracle Grid Infrastructure not configured.
You cannot run '/u01/app/11204/grid/bin/cluvfy' without the Oracle Grid Infrastructure.

Strace the command to get more details:
[grid@grac41 crsd]$ strace -t -f -o clu.trc cluvfy comp olr -verbose
clu.trc reports :
29993 09:32:19 open("/etc/oracle/olr.loc", O_RDONLY) = -1 ENOENT (No such file or directory)