ADOP - Prepare failed with - copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.

Adop prepare faile with below error. Issue was "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed", adop was not able to archive for cloning the filesystem, hence it failed. 
Issue was identified as Temp directory specified T2P_JAVA_OPTIONS was not existing which was preventing archiving. Created the directory and the prepare went fine.


Fix :

$ echo $T2P_JAVA_OPTIONS
-Djava.io.tmpdir=/prodsrv1/product/temp
$ ls -ld /prodsrv1/product/temp
ls: cannot access /prodsrv1/product/temp: No such file or directory
$ mkdir -p /prodsrv1/product/temp
$ ls -ld /prodsrv1/product/temp
drwxr-xr-x 2 approdsrv1 aaprodsrv1 2 Sep 10 08:50 /prodsrv1/product/temp
$



Issue :


Beginning application tier FSCloneStage - wlsConfig Sat Sep 10 05:55:29 2016

/prodsrv1/applmgr/fs1/EBSapps/comn/util/jdk32/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /prodsrv1/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/prodsrv1/applmgr/fs1/EBSapps/comn/java/classes:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/prodsrv1/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/prodsrv1/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneStageAppsTier -e /prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -targ /prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -stage /prodsrv1/applmgr/fs1/EBSapps/comn/adopclone_ebsprodsrv1 -tmp /tmp -component wlsConfig -nopromptmsg
Log file located at /prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/admin/log/clone/FSCloneStageAppsTier_09100555.log

ERROR while running FSCloneStage...
Sat Sep 10 05:56:05 2016
*******FATAL ERROR*******
PROGRAM : (/prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sat Sep 10 05:56:05 2016
FUNCTION: main::migrateCloneComponentStage [ Level 1 ]
ERRORMSG: /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.

    [UNEXPECTED]Error occurred running "perl /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -patchcontextfile=/prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -promptmsg=hide -console=off -mode=migrate -sessionid=6 -timestamp=20160910_055301 -outdir=/prodsrv1/applmgr/fs_ne/EBSapps/log/adop/6/prepare_20160910_055301/prodsrv1_ebsprodsrv1"
    [UNEXPECTED]occurred during CONFIG_CLONE Patch File System from Run File System, running command: "perl /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -patchcontextfile=/prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -promptmsg=hide -console=off -mode=migrate -sessionid=6 -timestamp=20160910_055301 -outdir=/prodsrv1/applmgr/fs_ne/EBSapps/log/adop/6/prepare_20160910_055301/prodsrv1_ebsprodsrv1".
    [UNEXPECTED]fs_clone has failed.
    [UNEXPECTED]Error calling runPendingConfigClone subroutine.
    Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10


== FSCloneStageAppsTier_09100725.log log ======

START: Creating WLS config archive.
Script Executed in 1803 milliseconds, returning status 255
ERROR: Script failed, exit code 255
====== CLONE2016-09-10_07-26-22_344033396.log =============================

---------------------------------------------------
               T2P Summary Begin
---------------------------------------------------
Error Message  :1
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20368  Domain pack failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20368  Make sure that values specified are correct.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20368  Check the T2P logs for more details.
Error Message  :2
  Sep 10, 2016 07:26:23 - SEVERE - CLONE-20963  "copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.
Error Message  :3
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20235   Adding "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20235   An internal operation failed.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20235   Check the clone log and error file for more details.
Error Message  :4
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20236   Archive creation has failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20236   An internal operation failed.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20236   Check the clone log and error file for more details.

---------------------------------------------------
               T2P Summary End
---------------------------------------------------

=== Clone error  log CLONE2016-09-10_07-26-22_344033396.error ===


SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20368  Domain pack failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20368  Make sure that values specified are correct.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20368  Check the T2P logs for more details.
java.lang.Exception: Domain pack failed.
        at oracle.as.clone.cloner.component.J2EEComponentCreateCloner.doClone(J2EEComponentCreateCloner.java:303)
        at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)
        at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:162)
        at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:94)
        at oracle.as.clone.request.CreateCloneRequest._clone(CreateCloneRequest.java:69)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
        at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:236)
        at oracle.as.clone.client.CloningClient.main(CloningClient.java:124)
SEVERE : Sep 10, 2016 07:26:23 - SEVERE - CLONE-20963  "copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20235   Adding "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20235   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20235   Check the clone log and error file for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20236   Archive creation has failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20236   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20236   Check the clone log and error file for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20218   Cloning is not successful.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20218   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20218   Provide the clone log and error file for investigation.

Unit used to calculate space in SYS.DBA_TABLESPACE_USAGE_METRICS

Plese find how the initial SYS.DBA_TABLESPACE_USAGE_METRICS output look like, generally we get confused as what unit is used in the view and how we can convert the data into MB or GB so that the data is readable.

ORADB01>CLNDV10>SELECT TABLESPACE_NAME TBSP_NAME
, USED_SPACE
, TABLESPACE_SIZE TBSP_SIZE

, USED_PERCENT

FROM SYS.DBA_TABLESPACE_USAGE_METRICS  2    3    4    5
 6  ;

TBSP_NAME                      USED_SPACE  TBSP_SIZE USED_PERCENT
------------------------------ ---------- ---------- ------------
SYSAUX                              60784     524288    11.5936279
SYSTEM                              65456      524288    12.4847412
TEMP                                  384           524288     .073242188
TOOLS                                   8           524288    .001525879
UNDOTBS1                         536         524288   .102233887
USERS                               74896        524288    14.2852783

The unit used here is in blocks.

Please find the block size of database.


ORADB01>sho parameter block

NAME                                 TYPE                             VALUE
------------------------------------ -------------------------------- ------------------------------
db_block_buffers                     integer                          0
db_block_checking                    string                           FALSE
db_block_checksum                    string                           TRUE
db_block_size                        integer                               8192
db_file_multiblock_read_count        integer                          32

Here block size is 8Kb, in this case if we have to convert SYSAUX TBSP_SIZE to MB we need to divide 1024/8=128. That is 128 Blocks makes a MB, so the tablespace max extendable size is 524288/128=4096 MB (4GB).

Please find the script that shows the tablespace in GB as below. 

ORADB01>SELECT TABLESPACE_NAME TBSP_NAME
, USED_SPACE/128/1024 TBSP_USEDSPS_gb
, TABLESPACE_SIZE/128/1024 TBSP_SIZE_GB
, USED_PERCENT
FROM SYS.DBA_TABLESPACE_USAGE_METRICS  2    3    4    5  ;



TBSP_NAME                      TBSP_USEDSPS_GB TBSP_SIZE_GB USED_PERCENT
------------------------------ --------------- ------------ ------------
SYSAUX                              .463745117            4   11.5936279
SYSTEM                                .499389648           4   12.4847412
TEMP                                    .002929688            4   .073242188
TOOLS                                  .000061035            4   .001525879
UNDOTBS1                           .004089355            4   .102233887

        USERS                                   .571411133             4   14.2

List files in sorted order - by file size

Many time we might encounter situation like the mount point is almost full and might need to clear some old logs to make space.

ls -lS -> will list files in sorted order by size
ls -lhS -> will list files in sorted order by size and will show the file size in readable format.
ls -RSh -> will list files in sorted order by size and shows files in directory and sub directory recursively. 

Command to copy last 3 days file to destination folder.

Below command can be used to copy past 3 days files to a destination folder. 
Below command can be modified to match your requirement.
* run the command from the file location.

find . -type f -mtime -3 | awk -F / '{print "cp  "$2 " /tmp/test/"}' | sh –x

OHASD Service not automitically started - CRS-4124: Oracle High Availability Services startup failed.

CRS Startup failed with below error.

CRS Status and Star tup Error

# /u01/app/11.2.0/grid/bin/crsctl check crs

CRS-4639: Could not contact Oracle High Availability Services

22:10:31 root@racsrv1: /root

#  /u01/app/11.2.0/grid/bin/crsctl start crs

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

22:19:31 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Done all below listed Diagnostics and identified as OHASD service has not started automatically when server started.

Unix team has done diagnostics and identified one of the startup script in rc local has hang for ever and some of the startup scripts has not run which has resulted in OHASD service not started up, once we  start the service manually I was able to startup CRS and all look fine.

All oracleasm Disk are verified and are available .

# oracleasm listdisks

OCR01

ASMDISK02

ASMDISK03

ASMDISK04

ASMDISK05

ASMDISK06

ASMDISK07

ASMDISK08

ASMDISK09

ASMDISK10

ASMDISK11

ASMDISK12

ASMDISK13

ASMDISK14

ASMDISK15

ASMDISK16

ASMDISK17

ASMDISK18

ASMDISK19

ASMDISK20

ASMDISK21

22:26:51 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Instance alert log errors

2014-02-28 22:17:16.630

[client(24180)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:17:16.632

[client(24180)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:10.175

[client(25549)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:10.177

[client(25549)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:13.314

[client(25553)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:13.316

[client(25553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:16.418

[client(25555)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:16.419

[client(25555)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:10.495

[client(27302)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:10.497

[client(27302)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:13.630

[client(27308)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:13.631

[client(27308)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:16.745

[client(27310)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:16.747

[client(27310)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:28:24.787

[client(27583)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

22:29:23 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Identified OHASD server was not running.

# ps -ef | grep ohasd

root     10610 25484  0 05:57 pts/2    00:00:00 grep ohasd

root     19254     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

root     24149     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

root     25669     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

05:57:23 root@racsrv1: /root

Started OHASD service in background and was able to start CRS as fine.

nohup /etc/init.d/init.ohasd run &

[1] 18124

06:20:21 root@racsrv1: /root

# nohup: appending output to `nohup.out'

06:20:25 root@racsrv1: /root

Checking CRS Status

# /u01/app/11.2.0.3/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

06:21:35 root@racsrv1: /root

# /u01/app/11.2.0.3/grid/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

06:23:28 root@racsrv1: /root

 

Script checks for Fusion Middleware OID process running or not and send an email. ..

#!/bin/ksh

 export MW_HOME=/u01/app/oracle/Middleware

export INSTANCE_HOME=/u01/app/oracle/Middleware/asinst_1

 ## Script compatable with FMW11g OID on Linux EL5 ###

oidps1=`ps -ef | grep oidldapd | grep "inst=1" | grep -v grep | awk '{print $2}'`

oidps2=`ps -ef | grep oidldapd | grep "control" |grep -v grep | awk '{print $2}'`

oidld1=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep $oidps1 | awk '{print $7}'`

oidld2=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep $oidps2 | awk '{print $7}'`

oidmon=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep oidmon| awk '{print $7}'`

oidem=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep EMAGENT | awk '{print $7}'`



if [ "$oidld1" == "Alive" ] && [ "$oidld2" == "Alive" ] && [ "$oidmon" == "Alive" ] && [ "$oidem" == "Alive" ];

then

echo "OID running"

echo "OID ldap Services are running " > /u01/app/oracle/ldap_scripts/ldap_running.txt

date >> /u01/app/oracle/ldap_scripts/ldap_running.txt

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status >> /u01/app/oracle/ldap_scripts/ldap_running.txt

mailx -s "OID processes are running on OID Server" test@test.com < /u01/app/oracle/ldap_scripts/ldap_running.txt

else

echo "failed"

mailx -s "OID processes are not running on OID Server" test@test.com < /u01/app/oracle/ldap_scripts/ldap_start_stop_process.txt

fi

====== ldap_start_stop_process.txt == File output ========

#######################################################################################
LDAP OID Services stop / Start procedure
#######################################################################################

INSTANCE_HOME=/u01/app/oracle/Middleware/asinst_1/bin/

MW_HOME=/u01/app/oracle/Middleware/

 ### LDAP Services Status Check ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status

### LDAP Services Stop all services ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl stopall

### LDAP Services Start all services ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl startall

===== Script End opmnctl status sample output ==============

oidsrv1$

oidsrv1$ /u01/app/oracle/Middleware/asinst_1/bin/opmnctl status

Processes in Instance: asinst_1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

oid1                             | oidldapd           |    4962 | Alive

oid1                             | oidldapd           |    4953 | Alive

oid1                             | oidmon             |    4914 | Alive

EMAGENT                  | EMAGENT     |    4912 | Alive

 oidsrv1$

Switchover and Switch back standby database using DGMGRL - log

----- Switch over ------

[oaracle@orasrv1](DB11@ORADB_1):/u01/app/oracle/product/11.2.0/db_1/dbs\>dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - itii_itio_ORADB

  Protection Mode: MaxPerformance
  Databases:
    ORADB   - Primary database
    ORADBdr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to ORADBdr;
Performing switchover NOW, please wait...
New primary database "ORADBdr" is opening...
Operation requires shutdown of instance "ORADB_1" on database "ORADB"
Shutting down instance "ORADB_1"...
ORACLE instance shut down.
Operation requires startup of instance "ORADB_1" on database "ORADB"
Starting instance "ORADB_1"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect de                                                                           scriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up and mount instance "ORADB_1" of database "ORADB"

DGMGRL> DGMGRL>

---- Startup the database ORADB
1. Startup nomount
2. Alter database mount standby database disconnect.

--- Switch back -----

[oracle@orasrv1](DB11@ORADB_1):/u01/app/oracle/product/11.2.0/db_1/dbs\>dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - itii_itio_ORADB

  Protection Mode: MaxPerformance
  Databases:
    ORADBdr - Primary database
    ORADB   - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to ORADB;
Performing switchover NOW, please wait...
New primary database "ORADB" is opening...
Operation requires shutdown of instance "ORADBDR_1" on database "ORADBdr"
Shutting down instance "ORADBDR_1"...
ORACLE instance shut down.
Operation requires startup of instance "ORADBDR_1" on database "ORADBdr"
Starting instance "ORADBDR_1"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up and mount instance "ORADBDR_1" of database "ORADBdr"

DGMGRL> exit

---- Startup the database ORADBDR

1. Startup nomount
2. Alter database mount standby database disconnect.