Thursday, March 6, 2014

OHASD Service not automitically started - CRS-4124: Oracle High Availability Services startup failed.



CRS Startup failed with below error.

CRS Status and Star tup Error

# /u01/app/11.2.0/grid/bin/crsctl check crs
CRS-4639: Could not contact Oracle High Availability Services
22:10:31 root@racsrv1: /root
#  /u01/app/11.2.0/grid/bin/crsctl start crs
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
22:19:31 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1



Done all below listed Diagnostics and identified as OHASD service has not started automatically when server started.

Unix team has done diagnostics and identified one of the startup script in rc local has hang for ever and some of the startup scripts has not run which has resulted in OHASD service not started up, once we  start the service manually I was able to startup CRS and all look fine.


All oracleasm Disk are verified and are available .
# oracleasm listdisks
OCR01
ASMDISK02
ASMDISK03
ASMDISK04
ASMDISK05
ASMDISK06
ASMDISK07
ASMDISK08
ASMDISK09
ASMDISK10
ASMDISK11
ASMDISK12
ASMDISK13
ASMDISK14
ASMDISK15
ASMDISK16
ASMDISK17
ASMDISK18
ASMDISK19
ASMDISK20
ASMDISK21
22:26:51 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Instance alert log errors

2014-02-28 22:17:16.630
[client(24180)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:17:16.632
[client(24180)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:22:10.175
[client(25549)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:22:10.177
[client(25549)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:22:13.314
[client(25553)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:22:13.316
[client(25553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:22:16.418
[client(25555)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:22:16.419
[client(25555)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:27:10.495
[client(27302)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:27:10.497
[client(27302)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:27:13.630
[client(27308)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:27:13.631
[client(27308)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:27:16.745
[client(27310)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
2014-02-28 22:27:16.747
[client(27310)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.
2014-02-28 22:28:24.787
[client(27583)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).
22:29:23 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Identified OHASD server was not running.

# ps -ef | grep ohasd
root     10610 25484  0 05:57 pts/2    00:00:00 grep ohasd
root     19254     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot
root     24149     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0/grid/bin/ohasd.bin reboot
root     25669     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

05:57:23 root@racsrv1: /root

Started OHASD service in background and was able to start CRS as fine.

nohup /etc/init.d/init.ohasd run &
[1] 18124
06:20:21 root@racsrv1: /root
# nohup: appending output to `nohup.out'

06:20:25 root@racsrv1: /root


Checking CRS Status

# /u01/app/11.2.0.3/grid/bin/crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
06:21:35 root@racsrv1: /root
# /u01/app/11.2.0.3/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
06:23:28 root@racsrv1: /root