OHASD Service not automitically started - CRS-4124: Oracle High Availability Services startup failed.

CRS Startup failed with below error.

CRS Status and Star tup Error

# /u01/app/11.2.0/grid/bin/crsctl check crs

CRS-4639: Could not contact Oracle High Availability Services

22:10:31 root@racsrv1: /root

#  /u01/app/11.2.0/grid/bin/crsctl start crs

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

22:19:31 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Done all below listed Diagnostics and identified as OHASD service has not started automatically when server started.

Unix team has done diagnostics and identified one of the startup script in rc local has hang for ever and some of the startup scripts has not run which has resulted in OHASD service not started up, once we  start the service manually I was able to startup CRS and all look fine.

All oracleasm Disk are verified and are available .

# oracleasm listdisks

OCR01

ASMDISK02

ASMDISK03

ASMDISK04

ASMDISK05

ASMDISK06

ASMDISK07

ASMDISK08

ASMDISK09

ASMDISK10

ASMDISK11

ASMDISK12

ASMDISK13

ASMDISK14

ASMDISK15

ASMDISK16

ASMDISK17

ASMDISK18

ASMDISK19

ASMDISK20

ASMDISK21

22:26:51 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Instance alert log errors

2014-02-28 22:17:16.630

[client(24180)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:17:16.632

[client(24180)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:10.175

[client(25549)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:10.177

[client(25549)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:13.314

[client(25553)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:13.316

[client(25553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:16.418

[client(25555)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:16.419

[client(25555)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:10.495

[client(27302)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:10.497

[client(27302)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:13.630

[client(27308)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:13.631

[client(27308)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:16.745

[client(27310)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:16.747

[client(27310)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:28:24.787

[client(27583)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

22:29:23 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Identified OHASD server was not running.

# ps -ef | grep ohasd

root     10610 25484  0 05:57 pts/2    00:00:00 grep ohasd

root     19254     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

root     24149     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

root     25669     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

05:57:23 root@racsrv1: /root

Started OHASD service in background and was able to start CRS as fine.

nohup /etc/init.d/init.ohasd run &

[1] 18124

06:20:21 root@racsrv1: /root

# nohup: appending output to `nohup.out'

06:20:25 root@racsrv1: /root

Checking CRS Status

# /u01/app/11.2.0.3/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

06:21:35 root@racsrv1: /root

# /u01/app/11.2.0.3/grid/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

06:23:28 root@racsrv1: /root