ADOP - Prepare failed with - copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.

Adop prepare faile with below error. Issue was "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed", adop was not able to archive for cloning the filesystem, hence it failed. 
Issue was identified as Temp directory specified T2P_JAVA_OPTIONS was not existing which was preventing archiving. Created the directory and the prepare went fine.


Fix :

$ echo $T2P_JAVA_OPTIONS
-Djava.io.tmpdir=/prodsrv1/product/temp
$ ls -ld /prodsrv1/product/temp
ls: cannot access /prodsrv1/product/temp: No such file or directory
$ mkdir -p /prodsrv1/product/temp
$ ls -ld /prodsrv1/product/temp
drwxr-xr-x 2 approdsrv1 aaprodsrv1 2 Sep 10 08:50 /prodsrv1/product/temp
$



Issue :


Beginning application tier FSCloneStage - wlsConfig Sat Sep 10 05:55:29 2016

/prodsrv1/applmgr/fs1/EBSapps/comn/util/jdk32/bin/java -Xmx600M -DCONTEXT_VALIDATED=false -Doracle.installer.oui_loc=/oui -classpath /prodsrv1/applmgr/fs1/FMW_Home/webtier/lib/xmlparserv2.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/jdbc/lib/ojdbc6.jar:/prodsrv1/applmgr/fs1/EBSapps/comn/java/classes:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/OraInstaller.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/ewt3.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/oui/jlib/share.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/../Oracle_EBS-app1/oui/jlib/srvm.jar:/prodsrv1/applmgr/fs1/FMW_Home/webtier/jlib/ojmisc.jar:/prodsrv1/applmgr/fs1/FMW_Home/wlserver_10.3/server/lib/weblogic.jar:/prodsrv1/applmgr/fs1/FMW_Home/oracle_common/jlib/obfuscatepassword.jar  oracle.apps.ad.clone.FSCloneStageAppsTier -e /prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -targ /prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -stage /prodsrv1/applmgr/fs1/EBSapps/comn/adopclone_ebsprodsrv1 -tmp /tmp -component wlsConfig -nopromptmsg
Log file located at /prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/admin/log/clone/FSCloneStageAppsTier_09100555.log

ERROR while running FSCloneStage...
Sat Sep 10 05:56:05 2016
*******FATAL ERROR*******
PROGRAM : (/prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl)
TIME    : Sat Sep 10 05:56:05 2016
FUNCTION: main::migrateCloneComponentStage [ Level 1 ]
ERRORMSG: /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/bin/adclone.pl did not go through successfully.

    [UNEXPECTED]Error occurred running "perl /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -patchcontextfile=/prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -promptmsg=hide -console=off -mode=migrate -sessionid=6 -timestamp=20160910_055301 -outdir=/prodsrv1/applmgr/fs_ne/EBSapps/log/adop/6/prepare_20160910_055301/prodsrv1_ebsprodsrv1"
    [UNEXPECTED]occurred during CONFIG_CLONE Patch File System from Run File System, running command: "perl /prodsrv1/applmgr/fs1/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPPreparePhaseSynchronize.pl -contextfile=/prodsrv1/inst/fs1/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -patchcontextfile=/prodsrv1/inst/fs2/inst/apps/prodsrv1_ebsprodsrv1/appl/admin/prodsrv1_ebsprodsrv1.xml -promptmsg=hide -console=off -mode=migrate -sessionid=6 -timestamp=20160910_055301 -outdir=/prodsrv1/applmgr/fs_ne/EBSapps/log/adop/6/prepare_20160910_055301/prodsrv1_ebsprodsrv1".
    [UNEXPECTED]fs_clone has failed.
    [UNEXPECTED]Error calling runPendingConfigClone subroutine.
    Stopping services on patch file system.
    Stopping admin server.

You are running adadminsrvctl.sh version 120.10.12020000.10


== FSCloneStageAppsTier_09100725.log log ======

START: Creating WLS config archive.
Script Executed in 1803 milliseconds, returning status 255
ERROR: Script failed, exit code 255
====== CLONE2016-09-10_07-26-22_344033396.log =============================

---------------------------------------------------
               T2P Summary Begin
---------------------------------------------------
Error Message  :1
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20368  Domain pack failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20368  Make sure that values specified are correct.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20368  Check the T2P logs for more details.
Error Message  :2
  Sep 10, 2016 07:26:23 - SEVERE - CLONE-20963  "copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.
Error Message  :3
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20235   Adding "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20235   An internal operation failed.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20235   Check the clone log and error file for more details.
Error Message  :4
  Sep 10, 2016 07:26:23 - ERROR - CLONE-20236   Archive creation has failed.
  Sep 10, 2016 07:26:23 - CAUSE - CLONE-20236   An internal operation failed.
  Sep 10, 2016 07:26:23 - ACTION - CLONE-20236   Check the clone log and error file for more details.

---------------------------------------------------
               T2P Summary End
---------------------------------------------------

=== Clone error  log CLONE2016-09-10_07-26-22_344033396.error ===


SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20368  Domain pack failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20368  Make sure that values specified are correct.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20368  Check the T2P logs for more details.
java.lang.Exception: Domain pack failed.
        at oracle.as.clone.cloner.component.J2EEComponentCreateCloner.doClone(J2EEComponentCreateCloner.java:303)
        at oracle.as.clone.cloner.Cloner.doFinalClone(Cloner.java:63)
        at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:162)
        at oracle.as.clone.request.CreateGenericArchive.doGenericArchive(CreateGenericArchive.java:94)
        at oracle.as.clone.request.CreateCloneRequest._clone(CreateCloneRequest.java:69)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:131)
        at oracle.as.clone.process.CloningExecutionProcess.execute(CloningExecutionProcess.java:114)
        at oracle.as.clone.client.CloningClient.executeT2PCommand(CloningClient.java:236)
        at oracle.as.clone.client.CloningClient.main(CloningClient.java:124)
SEVERE : Sep 10, 2016 07:26:23 - SEVERE - CLONE-20963  "copyConfig" operation of J2EE domain failed. Check clone log and error files for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20235   Adding "J2EECOMPONENT@EBS_domain_prodsrv1" to the archive has failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20235   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20235   Check the clone log and error file for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20236   Archive creation has failed.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20236   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20236   Check the clone log and error file for more details.
SEVERE : Sep 10, 2016 07:26:23 - ERROR - CLONE-20218   Cloning is not successful.
SEVERE : Sep 10, 2016 07:26:23 - CAUSE - CLONE-20218   An internal operation failed.
SEVERE : Sep 10, 2016 07:26:23 - ACTION - CLONE-20218   Provide the clone log and error file for investigation.

Unit used to calculate space in SYS.DBA_TABLESPACE_USAGE_METRICS

Plese find how the initial SYS.DBA_TABLESPACE_USAGE_METRICS output look like, generally we get confused as what unit is used in the view and how we can convert the data into MB or GB so that the data is readable.

ORADB01>CLNDV10>SELECT TABLESPACE_NAME TBSP_NAME
, USED_SPACE
, TABLESPACE_SIZE TBSP_SIZE

, USED_PERCENT

FROM SYS.DBA_TABLESPACE_USAGE_METRICS  2    3    4    5
 6  ;

TBSP_NAME                      USED_SPACE  TBSP_SIZE USED_PERCENT
------------------------------ ---------- ---------- ------------
SYSAUX                              60784     524288    11.5936279
SYSTEM                              65456      524288    12.4847412
TEMP                                  384           524288     .073242188
TOOLS                                   8           524288    .001525879
UNDOTBS1                         536         524288   .102233887
USERS                               74896        524288    14.2852783

The unit used here is in blocks.

Please find the block size of database.


ORADB01>sho parameter block

NAME                                 TYPE                             VALUE
------------------------------------ -------------------------------- ------------------------------
db_block_buffers                     integer                          0
db_block_checking                    string                           FALSE
db_block_checksum                    string                           TRUE
db_block_size                        integer                               8192
db_file_multiblock_read_count        integer                          32

Here block size is 8Kb, in this case if we have to convert SYSAUX TBSP_SIZE to MB we need to divide 1024/8=128. That is 128 Blocks makes a MB, so the tablespace max extendable size is 524288/128=4096 MB (4GB).

Please find the script that shows the tablespace in GB as below. 

ORADB01>SELECT TABLESPACE_NAME TBSP_NAME
, USED_SPACE/128/1024 TBSP_USEDSPS_gb
, TABLESPACE_SIZE/128/1024 TBSP_SIZE_GB
, USED_PERCENT
FROM SYS.DBA_TABLESPACE_USAGE_METRICS  2    3    4    5  ;



TBSP_NAME                      TBSP_USEDSPS_GB TBSP_SIZE_GB USED_PERCENT
------------------------------ --------------- ------------ ------------
SYSAUX                              .463745117            4   11.5936279
SYSTEM                                .499389648           4   12.4847412
TEMP                                    .002929688            4   .073242188
TOOLS                                  .000061035            4   .001525879
UNDOTBS1                           .004089355            4   .102233887

        USERS                                   .571411133             4   14.2

List files in sorted order - by file size

Many time we might encounter situation like the mount point is almost full and might need to clear some old logs to make space.

ls -lS -> will list files in sorted order by size
ls -lhS -> will list files in sorted order by size and will show the file size in readable format.
ls -RSh -> will list files in sorted order by size and shows files in directory and sub directory recursively. 

Command to copy last 3 days file to destination folder.

Below command can be used to copy past 3 days files to a destination folder. 
Below command can be modified to match your requirement.
* run the command from the file location.

find . -type f -mtime -3 | awk -F / '{print "cp  "$2 " /tmp/test/"}' | sh –x

OHASD Service not automitically started - CRS-4124: Oracle High Availability Services startup failed.

CRS Startup failed with below error.

CRS Status and Star tup Error

# /u01/app/11.2.0/grid/bin/crsctl check crs

CRS-4639: Could not contact Oracle High Availability Services

22:10:31 root@racsrv1: /root

#  /u01/app/11.2.0/grid/bin/crsctl start crs

CRS-4124: Oracle High Availability Services startup failed.

CRS-4000: Command Start failed, or completed with errors.

22:19:31 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Done all below listed Diagnostics and identified as OHASD service has not started automatically when server started.

Unix team has done diagnostics and identified one of the startup script in rc local has hang for ever and some of the startup scripts has not run which has resulted in OHASD service not started up, once we  start the service manually I was able to startup CRS and all look fine.

All oracleasm Disk are verified and are available .

# oracleasm listdisks

OCR01

ASMDISK02

ASMDISK03

ASMDISK04

ASMDISK05

ASMDISK06

ASMDISK07

ASMDISK08

ASMDISK09

ASMDISK10

ASMDISK11

ASMDISK12

ASMDISK13

ASMDISK14

ASMDISK15

ASMDISK16

ASMDISK17

ASMDISK18

ASMDISK19

ASMDISK20

ASMDISK21

22:26:51 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Instance alert log errors

2014-02-28 22:17:16.630

[client(24180)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:17:16.632

[client(24180)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:10.175

[client(25549)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:10.177

[client(25549)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:13.314

[client(25553)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:13.316

[client(25553)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:22:16.418

[client(25555)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:22:16.419

[client(25555)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:10.495

[client(27302)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:10.497

[client(27302)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:13.630

[client(27308)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:13.631

[client(27308)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:27:16.745

[client(27310)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

2014-02-28 22:27:16.747

[client(27310)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/racsrv1/client/emcrsp.log.

2014-02-28 22:28:24.787

[client(27583)]CRS-2302:Cannot get GPnP profile. Error CLSGPNP_NO_DAEMON (GPNPD daemon is not running).

22:29:23 root@racsrv1: /u01/app/11.2.0/grid/log/racsrv1

Identified OHASD server was not running.

# ps -ef | grep ohasd

root     10610 25484  0 05:57 pts/2    00:00:00 grep ohasd

root     19254     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

root     24149     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0/grid/bin/ohasd.bin reboot

root     25669     1  0 Feb28 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot

05:57:23 root@racsrv1: /root

Started OHASD service in background and was able to start CRS as fine.

nohup /etc/init.d/init.ohasd run &

[1] 18124

06:20:21 root@racsrv1: /root

# nohup: appending output to `nohup.out'

06:20:25 root@racsrv1: /root

Checking CRS Status

# /u01/app/11.2.0.3/grid/bin/crsctl start crs

CRS-4640: Oracle High Availability Services is already active

CRS-4000: Command Start failed, or completed with errors.

06:21:35 root@racsrv1: /root

# /u01/app/11.2.0.3/grid/bin/crsctl check crs

CRS-4638: Oracle High Availability Services is online

CRS-4537: Cluster Ready Services is online

CRS-4529: Cluster Synchronization Services is online

CRS-4533: Event Manager is online

06:23:28 root@racsrv1: /root

 

Script checks for Fusion Middleware OID process running or not and send an email. ..

#!/bin/ksh

 export MW_HOME=/u01/app/oracle/Middleware

export INSTANCE_HOME=/u01/app/oracle/Middleware/asinst_1

 ## Script compatable with FMW11g OID on Linux EL5 ###

oidps1=`ps -ef | grep oidldapd | grep "inst=1" | grep -v grep | awk '{print $2}'`

oidps2=`ps -ef | grep oidldapd | grep "control" |grep -v grep | awk '{print $2}'`

oidld1=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep $oidps1 | awk '{print $7}'`

oidld2=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep $oidps2 | awk '{print $7}'`

oidmon=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep oidmon| awk '{print $7}'`

oidem=`/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status | grep EMAGENT | awk '{print $7}'`



if [ "$oidld1" == "Alive" ] && [ "$oidld2" == "Alive" ] && [ "$oidmon" == "Alive" ] && [ "$oidem" == "Alive" ];

then

echo "OID running"

echo "OID ldap Services are running " > /u01/app/oracle/ldap_scripts/ldap_running.txt

date >> /u01/app/oracle/ldap_scripts/ldap_running.txt

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status >> /u01/app/oracle/ldap_scripts/ldap_running.txt

mailx -s "OID processes are running on OID Server" test@test.com < /u01/app/oracle/ldap_scripts/ldap_running.txt

else

echo "failed"

mailx -s "OID processes are not running on OID Server" test@test.com < /u01/app/oracle/ldap_scripts/ldap_start_stop_process.txt

fi

====== ldap_start_stop_process.txt == File output ========

#######################################################################################
LDAP OID Services stop / Start procedure
#######################################################################################

INSTANCE_HOME=/u01/app/oracle/Middleware/asinst_1/bin/

MW_HOME=/u01/app/oracle/Middleware/

 ### LDAP Services Status Check ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl status

### LDAP Services Stop all services ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl stopall

### LDAP Services Start all services ###

/u01/app/oracle/Middleware/asinst_1/bin/opmnctl startall

===== Script End opmnctl status sample output ==============

oidsrv1$

oidsrv1$ /u01/app/oracle/Middleware/asinst_1/bin/opmnctl status

Processes in Instance: asinst_1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

oid1                             | oidldapd           |    4962 | Alive

oid1                             | oidldapd           |    4953 | Alive

oid1                             | oidmon             |    4914 | Alive

EMAGENT                  | EMAGENT     |    4912 | Alive

 oidsrv1$

Switchover and Switch back standby database using DGMGRL - log

----- Switch over ------

[oaracle@orasrv1](DB11@ORADB_1):/u01/app/oracle/product/11.2.0/db_1/dbs\>dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - itii_itio_ORADB

  Protection Mode: MaxPerformance
  Databases:
    ORADB   - Primary database
    ORADBdr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to ORADBdr;
Performing switchover NOW, please wait...
New primary database "ORADBdr" is opening...
Operation requires shutdown of instance "ORADB_1" on database "ORADB"
Shutting down instance "ORADB_1"...
ORACLE instance shut down.
Operation requires startup of instance "ORADB_1" on database "ORADB"
Starting instance "ORADB_1"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect de                                                                           scriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up and mount instance "ORADB_1" of database "ORADB"

DGMGRL> DGMGRL>

---- Startup the database ORADB
1. Startup nomount
2. Alter database mount standby database disconnect.

--- Switch back -----

[oracle@orasrv1](DB11@ORADB_1):/u01/app/oracle/product/11.2.0/db_1/dbs\>dgmgrl
DGMGRL for Linux: Version 11.2.0.2.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys
Password:
Connected.
DGMGRL> show configuration

Configuration - itii_itio_ORADB

  Protection Mode: MaxPerformance
  Databases:
    ORADBdr - Primary database
    ORADB   - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> switchover to ORADB;
Performing switchover NOW, please wait...
New primary database "ORADB" is opening...
Operation requires shutdown of instance "ORADBDR_1" on database "ORADBdr"
Shutting down instance "ORADBDR_1"...
ORACLE instance shut down.
Operation requires startup of instance "ORADBDR_1" on database "ORADBdr"
Starting instance "ORADBDR_1"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps to finish switchover:
        start up and mount instance "ORADBDR_1" of database "ORADBdr"

DGMGRL> exit

---- Startup the database ORADBDR

1. Startup nomount
2. Alter database mount standby database disconnect.

Oracle Database CPU Patching Very quick steps

Oracle Database CPU Patching Very quick steps.

Note: for detailed steps please read Patch readme file, few patch require database to be started in upgrade mode.

1. Backup Oracle Home -

           

            cd $ORACLE_HOME

            tar -cvf /u03/PATCH/HOMEBKP/DB_102050_HOME_bkp.tar .

2. Check Patch Readme file to know current version of OPatch matching requirement.

            Check opatch version :

            which opatch  -> ensure it shows the opatch file from right Oracle Home

            opatch -version -> will show opatch version.

            To upgrade,

           

            move current opatch directory to new name

            cd $ORACLE_HOME

            mv OPatch Opatch.bak

            Download and unzip opatch latest version from download.oracle.com and unzip the folder in Oracle Home.

3. Set Environment Variables.

            export ORACLE_HOME=/u01/app/oracle/product/10.2.0.5

            export PATH=$ORACLE_HOME/OPatch:$ORACLE_HOME/bin:$PATH

            export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/network/lib

            export LIBPATH=$ORACLE_HOME/lib:$ORACLE_HOME/lib

4. Shutdown all the instances and listener running from this Oracle Home.

5. Download patch and unzip the patch file in desired location.

            cd /u03/PATCH/

            unzip p14727319_10205_Linux-x86-64.zip

6. Prerequisites Check - running below command

            $ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir 14727319

            Invoking OPatch 10.2.0.5.1

            Oracle Interim Patch Installer version 10.2.0.5.1

            Copyright (c) 2010, Oracle Corporation.  All rights reserved.

            PREREQ session

            Oracle Home       : /u01/app/oracle/product/10.2.0.5

            Central Inventory : /u01/app/oracle/product/10.2.0.5

               from           : /etc/oraInst.loc

            OPatch version    : 10.2.0.5.1

            OUI version       : 10.2.0.5.0

            OUI location      : /u01/app/oracle/product/10.2.0.5/oui

            Log file location : /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch2013-06-13_02-30-20AM.log

            Patch history file: /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch_history.txt

            Invoking prereq "checkconflictagainstohwithdetail"

            Prereq "checkConflictAgainstOHWithDetail" passed.

            OPatch succeeded.

7. Apply the patch.

            -> Before applying it is always better to check which opatch is being used run “which opatch “command to know.

            $ echo $ORACLE_HOME

            /u01/app/oracle/product/10.2.0.5

            $ which opatch

            ~/product/10.2.0.5/OPatch/opatch

            -> cd to Patch locatio and run opatch apply ( It will ask for few interactive questions answer them)

           

            $ cd 14727319

            $ opatch apply

            Invoking OPatch 10.2.0.5.1

            Oracle Interim Patch Installer version 10.2.0.5.1

            Copyright (c) 2010, Oracle Corporation.  All rights reserved.

            Oracle Home       : /u01/app/oracle/product/10.2.0.5

            Central Inventory : /u01/app/oracle/product/10.2.0.5

               from           : /etc/oraInst.loc

            OPatch version    : 10.2.0.5.1

            OUI version       : 10.2.0.5.0

            OUI location      : /u01/app/oracle/product/10.2.0.5/oui

            Log file location : /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch2013-06-13_02-33-03AM.log

            Patch history file: /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch_history.txt

            ApplySession applying interim patch '14727319' to OH '/u01/app/oracle/product/10.2.0.5'

            Running prerequisite checks...

            Patch 14727319: Optional component(s) missing : [ oracle.rdbms.dv, 10.2.0.5.0 ] , [ oracle.rdbms.dv.oc4j, 10.2.0.5.0 ] , [ oracle.network.cman, 10.2.0.5.0 ]

            Provide your email address to be informed of security issues, install and

            initiate Oracle Configuration Manager. Easier for you if you use your My

            Oracle Support Email address/User Name.

            Visit http://www.oracle.com/support/policies.html for details.

            Email address/User Name:

            You have not provided an email address for notification of security issues.

            Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y

            OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only.

            Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

            (Oracle Home = '/u01/app/oracle/product/10.2.0.5')

            Is the local system ready for patching? [y|n]

.

.

.

.

.

.

.

.

##########. Last few lines of Patch log...##############

            Patching component oracle.xdk.rsf, 10.2.0.5.0...

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libxml10.a"  with "lib/libxml10.a/lpxpar.o"

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib32/libxml10.a"  with "lib32/libxml10.a/lpxpar.o"

            Patching component oracle.precomp.common, 10.2.0.5.0...

            Patching component oracle.rdbms.rman, 10.2.0.5.0...

            Copying file to "/u01/app/oracle/product/10.2.0.5/rdbms/admin/recover.bsq"

            Patching component oracle.sdo.locator, 10.2.0.5.0...

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libordsdo10.a"  with "lib/libordsdo10.a/mdidx.o"

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libordsdo10.a"  with "lib/libordsdo10.a/mdrcr.o"

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libordsdo10.a"  with "lib/libordsdo10.a/mdrt.o"

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libordsdo10.a"  with "lib/libordsdo10.a/mdopp.o"

            Updating archive file "/u01/app/oracle/product/10.2.0.5/lib/libordsdo10.a"  with "lib/libordsdo10.a/mdgr.o"

            Patching component oracle.network.listener, 10.2.0.5.0...

            Patching component oracle.network.client, 10.2.0.5.0...

            Copying file to "/u01/app/oracle/product/10.2.0.5/bin/adapters"

            Running make for target client_sharedlib

            Running make for target ioracle

            Running make for target iwrap

            Running make for target client_sharedlib

            Running make for target proc

            Running make for target irman

            Running make for target itnslsnr

            ApplySession adding interim patch '14727319' to inventory

            Verifying the update...

            Inventory check OK: Patch ID 14727319 is registered in Oracle Home inventory with proper meta-data.

            Files check OK: Files from Patch ID 14727319 are present in Oracle Home.

            The local system has been patched and can be restarted.

            OPatch succeeded.

8. On each database on oracle home startup the database and run below listed scripts.

            For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows:

            cd $ORACLE_HOME/rdbms/admin

            sqlplus /nolog

            SQL> CONNECT / AS SYSDBA

            SQL> STARTUP

            SQL> @catbundle.sql psu apply

            SQL> -- Execute the next statement only if this is the first PSU applied for 10.2.0.5 or this is the first PSU applied since 10.2.0.5.3.

            SQL> @utlrp.sql

            SQL> QUIT

            Check the following log files in $ORACLE_HOME/cfgtoollogs/catbundle or $ORACLE_BASE/cfgtoollogs/catbundle for any errors:

            catbundle_PSU__APPLY_.log

            catbundle_PSU__GENERATE_.log

9. Check the Opatch latest patch version " opatch lsinventory"

            $ opatch lsinventory

            Invoking OPatch 10.2.0.5.1

            Oracle Interim Patch Installer version 10.2.0.5.1

            Copyright (c) 2010, Oracle Corporation.  All rights reserved.

            Oracle Home       : /u01/app/oracle/product/10.2.0.5

            Central Inventory : /u01/app/oracle/product/10.2.0.5

               from           : /etc/oraInst.loc

            OPatch version    : 10.2.0.5.1

            OUI version       : 10.2.0.5.0

            OUI location      : /u01/app/oracle/product/10.2.0.5/oui

            Log file location : /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch2013-06-13_02-35-14AM.log

            Patch history file: /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/opatch_history.txt

            Lsinventory Output file location : /u01/app/oracle/product/10.2.0.5/cfgtoollogs/opatch/lsinv/lsinventory2013-06-13_02-35-14AM.txt

            --------------------------------------------------------------------------------

            Installed Top-level Products (3):

            Oracle Database 10g                                                  10.2.0.1.0

            Oracle Database 10g Products                                         10.2.0.1.0

            Oracle Database 10g Release 2 Patch Set 4                            10.2.0.5.0

            There are 3 products installed in this Oracle Home.

            Interim patches (1) :

            Patch  14727319     : applied on Thu Jun 13 02:34:39 PDT 2013

            Unique Patch ID:  15760144

               Created on 13 Dec 2012, 08:27:13 hrs PST8PDT

               Bugs fixed:

                 8865718, 11790175, 13489660, 9020537, 9772888, 8650138, 8664189, 10091698

                 14275629, 14469008, 10092858, 12551710, 7519406, 13349665, 8771916

                 7509714, 8822531, 10139235, 10159846, 13257247, 8350262, 11792865

                 7119382, 13632738, 11724962, 8966823, 9320130, 13775862, 11674645

                 15877957, 7026523, 15877958, 15877959, 9399589, 14841459, 9672816

                 13503598, 9499302, 9150282, 9448311, 9659614, 13632743, 9949948, 8882576

                 10327179, 7612454, 7111619, 9711859, 9714832, 9735237, 9952230, 15877960

                 12780098, 15877961, 15877962, 14665116, 15877963, 8660422, 11066597

                 14546673, 14105702, 9713537, 14105703, 14105704, 13483152, 13737773

                 13737775, 14269955, 12925532, 12748240, 9694101, 14390396, 12862186

                 12862187, 10249537, 14727319, 9586877, 8211733, 6694396, 9548269, 7115910

                 7710224, 9337325, 8354642, 7602341, 14076510, 11856395, 10157402, 12565867

                 6402302, 10327190, 10269717, 14023636, 11693109, 10017048, 8546356

                 8394351, 9024850, 8224558, 9770451, 9360157, 8488233, 9109487, 10132870

                 9171933, 10173237, 9532911, 10068982, 10306945, 7361418, 11725006

                 8666117, 6157713, 9184754, 10214450, 14205448, 8544696, 9767674, 9323583

                 8277300, 9726739, 13343467, 8412426, 10326338, 10165083, 12419392

                 6651220, 10208905, 9145204, 13554409, 11076894, 7450366, 11893577

                 8970313, 14492313, 6011045, 14492314, 10162036, 11814891, 14492315

                 10248542, 14492316, 9469117, 13359623, 9952270, 9842573, 13343471

                 10324526, 14546638, 12419258, 9322219, 8636407, 10010310, 12828105

                 9689310, 9390484, 13736501, 13736502, 9824435, 13736503, 13736504

                 13736505, 13736506, 9963497, 9032322, 13736507, 12551700, 12551701

                 12551702, 14035825, 11858315, 12551703, 12551704, 12551705, 10076669

                 12551706, 14040433, 12551707, 6076890, 12551708, 14258925, 9308296

                 13916709, 12827745, 12880299, 14038805, 13923855, 9072105, 8528171, 11737047

            --------------------------------------------------------------------------------

            OPatch succeeded.

10. Check all databases and listener are started properly. Send our an email to users for check

Long time.......

There has been no post for a very long time.... inactive .. Will start posting from today again..

Keep watching.. all the post in the blogs will be useful for a Core Oracle DBAs for their day today DBA activities .

Regards
Siva Prasad

RMAN Backup Script with TSM Env setting

connect target /
connect catalog catuse/catuser@rmancatlogdb
CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 180 DAYS;
run {
allocate channel c3 type 'SBT_TAPE' parms 'ENV=(TDPO_OPTFILE=C:\Program Files\tivoli\TSM\AgentOBA\tdpo_dbname.opt)';
backup database format 'DB_%d_%s_%p_%t' include current controlfile filesperset 12;
sql 'alter system archive log current';
change archivelog all crosscheck;
backup archivelog all format 'AL_%d_%s_%p_%t' filesperset 32;
backup current controlfile format '%d_%T_%t_%s_%p.ctl';
release channel c3;
}

TSM RMAN Backup Implementation & dircetory permissions issue.

TSM Implementation

We have started implementing RMAN Backup with Tivoli Storage Manager.
Things were set, TSM Client installation was done, TDPO file also was created and we have tried to run a test backup but backup fails with below error.

RMAN-03009: failure of allocate command on c1 channel at 01/23/2011
11:12:42
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27000: skgfqsbi: failed to initialize storage subsystem (SBT) layer
Linux-x86_64 Error: 106: Transport endpoint is already connected
Additional information: 7011
ORA-19511: Error received from media manager layer, error text:
SBT error = 7011, errno = 106, sbtopen: system error


Solution :

Wile trying to run backup TSM Client tries to update to file tdpoerror.log for which the “ORACLE” user need to have read Wright access to below location.

/usr/tivoli/tms/client/oracle/bin64/

Or

/opt/Tivoli/tms/client/oracle/bin64/

After oracle user have read write access to oracle for TSM client installation path.

Upgrade database 10.2.0.1 to 10.2.0.4 with 32 bit to 64 bit conversion

Upgrade database 10.2.0.1 to 10.2.0.4 with 32 bit to 64 bit conversion

While consolidating the database servers we had to move a database from 10.2.0.1 to 10.2.0.4 home. Here we have taken cold backup of the database on the source server and copied to target server and tried to upgrade resulted in below error, as the source database was in 32bit and target database home was in 64bit.

Error Message:

No errors.
CREATE OR REPLACE FUNCTION version_script
*
ERROR at line 1:
ORA-06544: PL/SQL: internal error, arguments: [56319], [], [], [], [], [], [],
[]
ORA-06553: PLS-801: internal error [56319]


Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

Reason:

The reason was we are upgrading a database from 10.2.0.1 32 Bit home to 10.2.0.4 64 Bit home. Oracle bit upgrade 32bit database 64 bit is not automatic.

Solution :

There is one more script to be executed before we run catupgrd.sql.
There are two ways.
1. run the script utlip.sql as below

a. sqlplus > @?/rdbms/admin/utlip.sql

2. Uncomment the below line in catupgrd.sql script
vi catupgrd.sql change:
==> @@&utlip_file
to:==> @@utlip.sql

and run catupgrd.sql
3. SQL> @?/rdbms/admin/catupgrd.sql

This should solve the issue.

In active.. It been a long time ther is no post on the blog

In active.. It been a long time there is no post on the blog

was bit busy.. will keep posting here after..

Skip an archive log or mark an archive log unavailable in RMAN backup

run {
change archivelog from logseq 44937 until logseq 44947 unavailable;
}

run {
change archivelog 44937 unavailable;
}

Rotate listener log.

Rotate listener log.
If the listener log grow beyond our limit size and you want to rotate listener log without restarting listener service we might do something like this.
Go to listener log path.

Eg. Listener log file name like listener.log

cp listener.log listener.log.old; tail -500 listner.log.old > listener.log

- try to maintain this in single line so that the time gap should not be more
Small note :- whenever we try to move the existing file and add touch a new file mean time listener service will fail to Wright the log to listener.log file hence the service might stop. Whenever we move the file and create a new file. The I node number of the file changes and we might need to restart listener as the service will be writing log to the old inode file and that changes now. We might need to restart the service for service to identify the new listener.log file.

Move, copy or delete files of particular date.

Move, copy or delete files of particular date.
Quick and the good one.
Eg. To delete files with a particular date without using find command ( hardcore process)
ls -ltr | grep "May 23" | awk '{print "mv "$9" /disk1/oradata/arch/"}' | more
above will list all the files that are having May 23 date.
ls -ltr | grep "May 23" | awk '{print "mv "$9" /disk1/oradata/arch/"}' | sh –x
This will execute the output. Customize the output and sh –x will execute the output.

RMAN 8i commands

RMAN 8i Most commands
Create Recovery Catalog
First create a user to hold the recovery catalog.
CONNECT sys/password@TSH1

-- Create tablepsace to hold repository

CREATE TABLESPACE "TOOLS" DATAFILE E:\ORACLE\ORADATA\DDBA1\TOOLS01.DBF' SIZE 10M AUTOEXTEND ON NEXT 1024K EXTENT MANAGEMENT LOCAL;

-- Create rman schema owner

CREATE USER rman IDENTIFIED BY rman TEMPORARY TABLESPACE temp DEFAULT TABLESPACE tools QUOTA UNLIMITED ON tools;


GRANT connect, resource, recovery_catalog_owner TO rman;

-- Create the recovery catalog:

c:> rman catalog rman/rman@tsh1
RMAN> create catalog tablespace tools;
-- Register Database
Each database to be backed up by RMAN must be registered:
C:>rman target sys/password@tsh1 rcvcat rman/rman@dba1 msglog 'C:OracleBackupTSH1TSH1_Daily_Backup.log'
RMAN> register database;


-- Cold Backup
rman target sys/password@tsh1 rcvcat rman/rman@dba1
RMAN> run
{
allocate channel ch1 type disk format 'C:\Oracle\Backup\TSH1\%d_DB_%u_%s_%p';
backup database include current controlfile
release channel ch1;
# Open the database and Archive all logfiles including current
alter database open;
sql 'ALTER SYSTEM ARCHIVE LOG CURRENT';
# Backup outdated archlogs and delete them
allocate channel ch1 type disk format 'C:\Oracle\Backup\TSH1\%d_ARCH_%u_%s_%p';
backup archivelog until time 'Sysdate-2' all delete input;
release channel ch1;
# Backup remaining archlogs
allocate channel ch1 type disk format 'C:\Oracle\Backup\TSH1\%d_ARCH_%u_%s_%p';
backup archivelog all;
release channel ch1;
}


--recovery catalog should be resyncronized

RMAN> resync catalog;

-- Hot Backup
Hot backups using RMAN are very simple. There is no need to alter the tablespace or database mode.
run {
allocate channel ch1 type disk format 'd:\oracle\backup%d_DB_%u_%s_%p';
backup database;
backup archivelog all;
release channel ch1;
}

-- Restore & Recover The Whole Database
Recovering from a media failure is as simple as:
run {
startup mount pfile=c:\Oracle\Admin\TSH1\pfile\init.ora;
allocate channel ch1 type disk;
restore database;
recover database;
release channel ch1;
}

-- Restore & Recover A Subset Of The Database
A subset of the database can be restored in a similar fashion:
run {
sql 'ALTER TABLESPACE users OFFLINE IMMEDIATE';
restore tablespace users;
recover tablespace users;
sql 'ALTER TABLESPACE users ONLINE';
}
-- Incomplete Recovery
As you would expect, RMAN allows incomplete recovery to a specified time, SCN or sequence number:
run
{
set until time 'Nov 15 2000 09:00:00';
# set until scn 1000; # alternatively, you can specify SCN
# set until sequence 9923; # alternatively, you can specify log sequence number
restore database;
recover database;
}

alter database open resetlogs;
The incomplete recovery requires the database to be opened using the RESETLOGS option.

-- Lists And Reports
RMAN has extensive listing and reporting functionality allowing you to monitor you backups and maintain the recovery catalog. Here are a few useful commands:
# Show all backup details
list backup;

# Show items that beed 7 days worth of
# archivelogs to recover completely
report need backup days = 7 database;

# Show/Delete items not needed for recovery
report obsolete;
delete obsolete;

# Show/Delete items not needed for point-in-time
# recovery within the last week
report obsolete recovery window of 7 days;
delete obsolete recovery window of 7 days;

# Show/Delete items with more than 2 newer copies available
report obsolete redundancy = 2 device type disk;
delete obsolete redundancy = 2 device type disk;

# Show datafiles that connot currently be recovered
report unrecoverable database;
report unrecoverable tablespace 'USERS';

Using commands with ssh to execute on remote server

Using commands with ssh to execute on remote server.

If you want to execute a command on a remote server we need not login to server but we can execute the command directly from the command prompt only.
Eg. If we want to check corntab entries of few servers we canuse contab –l from the command prompt only.

ssh servername crontab –l

If we want to execute more than one command from command prompt we can do as below with “;” as command .

ssh servername “hostname ; crontab –l | grep /u01/app/oracle “

Script to connect all the database on a server and run a SQL Script

Script to connect all the database on a server and run a SQL Script
!#!/bin/sh
for SID in `ps -ef grep pmon grep -v grep awk -F_ '{print $3}' sort `
do
ORACLE_SID=$SID
export ORACLE_SID
ORACLE_HOME=`grep $SID /var/opt/oracle/oratab awk -F: '{print $2}'`
export ORACLE_HOME
sqlplus "/as sysdba" << EOF
select name from v\$database;
EOF
Done
Please run this script only for any select operations, any updates or modifications in the SQL script It is suggested do them manually.
Note :- The above script check for oratab file in /var/opt/oracle/ path, if you want to run the script in LINUX or any other UNIX OS please change the oratab location to /etc/oratab
Include your script in place of “select name from v$databae” ( between the two EOFs)

ORA-01114 ( ORA-27072 or ORA-27069)

ORA-01114: IO error writing block to file 1501 (block # 579517) :
ORA-27072: skgfdisp: I/O error :Linux Error: 25: Inappropriate ioctl for device :
Additional information: 579516 :
Or
ORA-01114: IO error writing block to file 1501 (block # 579517) :
ORA-27069 : skgfdisp: attempt to do I/O beyond the range of the file
Additional information: 16385
At the first attempt, quickly I had search for the file_id in the dba_data_files and could not find and tried with the block#, no luck.
We may have to try finding file_id parameter from dba_temp_files, you will find the id there. Temp file_id start from next value from db_files parameter.
Now you are clear with the culprit.
There are two causes, either you need to add some space to resolve the issue or you have enough space to expand and still you are getting the same issue, please add a temp tablespace , make that as default tablespace and drop old temp tablespace.

Illustrations :
SQL> show parameter db_files
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_files integer 1500

Here you will see the temp file_id starg from the 1501 and so on. Temp file_id numbering start from

SQL> select file_name,tablespace_name,file_id from dba_temp_files;

FILE_NAME TABLESPACE_NAME FILE_ID
-------------------------------------------------------------------------------------------------------------- ----------


/u02/oradata/TESTDB/temp01.db TEMP2 1501


SQL> CREATE TEMPORARY TABLESPACE temp2
2 TEMPFILE '/u02/oradata/TESTDB/temp2_01.dbf' SIZE 5M REUSE
3 AUTOEXTEND ON NEXT 1M MAXSIZE unlimited
4 EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M;

Tablespace created.


SQL> ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp2;

Database altered.


SQL> DROP TABLESPACE temp INCLUDING CONTENTS AND DATAFILES;

Tablespace dropped.

SQL> CREATE TEMPORARY TABLESPACE temp
2 TEMPFILE '/u02/oradata/TESTDB/temp01.dbf' SIZE 500M REUSE
3 AUTOEXTEND ON NEXT 100M MAXSIZE unlimited
4 EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M;

Tablespace created.


SQL> ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp;

Database altered.


SQL> DROP TABLESPACE temp2 INCLUDING CONTENTS AND DATAFILES;

Tablespace dropped.