ORA-27300: OS System Dependent Operation: Fork Failed With Status: 12

ALERT LOG ENTRY
Process startup failed, error stack:
Errors in file /u01/app/oracle/diag/rdbms/cbelive/PRODDB2/trace/PRODDB2_psp0_5898502.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
====================CONTENT OF TRACE FILE======================
Trace file /u01/app/oracle/diag/rdbms/cbelive/PRODDB2/trace/PRODDB2_psp0_5898502.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/11.2.0.3/oracle
System name:    AIX
Node name:      t24db2
Release:        1
Version:        6
Machine:        00C941664C00
Instance name: PRODDB2
Redo thread mounted by this instance: 2
Oracle process number: 3
Unix process pid: 5898502, image: oracle@t24db2 (PSP0)

*** 2015-01-27 13:30:39.519
*** SESSION ID:(379.1) 2015-01-27 13:30:39.519
*** CLIENT ID:() 2015-01-27 13:30:39.519
*** SERVICE NAME:(SYS$BACKGROUND) 2015-01-27 13:30:39.519
*** MODULE NAME:() 2015-01-27 13:30:39.519
*** ACTION NAME:() 2015-01-27 13:30:39.519

Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

*** 2015-01-27 13:32:50.138
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

*** 2015-01-27 13:33:24.157
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
~
===========================================================

CAUSE


Insufficient paging space defined for the system program using excessive amount of paging space.

SOLUTION

Increase the OS paging space.

For AIX, the next APAR is advised:

Fix pack for: IZ19670: CANNOT INCREASE PAGING SPACE UNDER LOAD APPLIES TO AIX 6100-01

Please note the specific patches below:
  5300-09 - use AIX APAR IZ19504
  6100-01 - use AIX APAR IZ19670
  6100-02 - use AIX APAR IZ20052

For more information, please visit:
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ19670

ORA-19809 LIMIT EXCEEDED FOR RECOVERY FILES

We encounter this error when the archivelog destination (specified by DB_RECOVERY_FILE_DEST parameter ) is full.
Solution
1. When we have time and resource consider adding disks to the diskgroup upon which the archive log destination resides and then increase the parameter DB_RECOVERY_FILE_DEST_SIZE parameter.
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=XXXG SCOPE=BOTH;
2.If the above solution doesn't work, use rman to delete the archive log which have been backed up to make some space in the destination.
export ORACLE_SID=TESTDB
rman target /
RMAN> delete noprompt archivelog all completed before 'sysdate - 1';
The above command deletes archivelogs which have been completed 24 hours ago if you have archivelog backup in last 24 hours then you are good to go otherwise change the sysdate - to larger number of days.
3. Consider configuring rman archive log deletion and retention policies.

11g DATAGURARD BROKER DGMGRL CONFIGURATION STEPS

Enterprise Manager Grid Control GUI or Command line interface can be used to configure Guard Broker. The following demonstration uses the command line version.
ASSUMPTION
There exists a function physical standby database ( i.e  The redo log shipping/transport is happening and the transported logs are being applied to make the standby database in sync with the production.)
Primary Database: PROD
Standby Database: PRODSTDBY

STEP 1. ON BOTH THE PRIMARY AND STANDBY DATABASES START THE DATAGUARD BROKER PROCESS

SQL> ALTER SYSTEM SET DG_BROKER_START=TRUE SCOPE=BOTH;
System altered.

STEP 2. EDIT THE listener.ora ON BOTH THE PRIMARY AND STANDBY NODES TO CONTAIN THE STATIC ENTRY FOR DBMGRL

Ensure that the GLOBAL_DBNAME is set to db_unique_name_DGMGRL.db_domain
SID_LIST_LISTENER =
  (SID_LIST =
 (SID_DESC =
        (GLOBAL_DBNAME = PROD_dgmgrl)
        (ORACLE_HOME = /u01/oracle/product/11.2.0.3/db_1)
            (SID_NAME = PROD)
        )
    )

STEP 3. CREATE THE CONFIGURATION

PROD:/u01/oracle$ dgmgrl
DGMGRL for IBM/AIX RISC System/6000: Version 11.1.0.7.0 - 64bit Production

Copyright (c) 2000, 2005, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys/xxxxxxx
Connected.
DGMGRL> CREATE CONFIGURATION 'ORIGINDR'
> AS
> PRIMARY DATABASE IS 'PROD'
> CONNECT IDENTIFIER IS 'PRODSTDBY'
> ;
Configuration "ORIGINDR" created with primary database "PROD"

STEP 4. ADD THE STANDBY DATABASE TO THE CONFIGURATION

DGMGRL> ADD DATABASE
> 'PRODSTDBY'
> AS
> CONNECT IDENTIFIER IS 'PRODSTDBY'
> ;
Database "PRODSTDBY" added

DGMGRL> SHOW CONFIGURATION

Configuration
  Name:                ORIGINDR
  Enabled:             NO
  Protection Mode:     MaxAvailability
  Databases:
    PROD - Primary database
    PRODSTDBY - Physical standby database

Fast-Start Failover: DISABLED

Current status for "ORIGINDR":
DISABLED

STEP 5. ENABLE THE CONFIGURATION

DGMGRL> ENABLE CONFIGURATION
Enabled.
DGMGRL> SHOW CONFIGURATION
Configuration
  Name:                ORIGINDR
  Enabled:             YES
  Protection Mode:     MaxAvailability
  Databases:
    PROD - Primary database
    PRODSTDBY - Physical standby database

Fast-Start Failover: DISABLED

Current status for "ORIGINDR":
SUCCESS

ORA-15000: command disallowed by current instance type

ORA-15000: command disallowed by current instance type

As the error itself suggests we are not using the right instance type. It occurs when we try to adminitrer the ASM storage system from RDBMS instance such as adding disks,dropping disks,etc
Solution
Connect to the ASM instance and execute the command connecting 'as sysasm'
for instance
$export ORACLE_SID=+ASM
$sqlplus / as sysasm
SQL>ALTER DISKGROUP DATA ADD DISK '/dev/rhdisk391' REBALANCE POWER 11;

REGISTERING A DATABASE WITH GRID CLUSTER

After database duplication some times you may not be able to stop,start or see the status of the database using srvctl utility. 
$ srvctl stop database -d TESTDB
PRCD-1120 : The resource for database TESTDB could not be found.
PRCR-1001 : Resource ora.TESTDB.db does not exist
Despite the above errors the database is up and running.
$ ps -fe|grep pmon
grid      3960     1  0 15:57 ?        00:00:00 asm_pmon_+ASM
oracle    4179     1  0 16:33 ?        00:00:00 ora_pmon_TESTDB
oracle    4284  3702  0 16:36 pts/1    00:00:00 grep pmon
$ echo $ORACLE_SID
TESTDB

$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg           
               ONLINE  ONLINE       dbsrvr1
               ONLINE  ONLINE       dbsrvr2   
ora.LISTENER.lsnr   
               ONLINE  ONLINE       dbsrvr1
            ONLINE  ONLINE       dbsrvr2
ora.asm       
               ONLINE  ONLINE       dbsrvr1                Started
            ONLINE  ONLINE       dbsrvr2                Started
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.proddb.db        
      1        ONLINE  ONLINE       dbsrvr1                open
     2        ONLINE  ONLINE       dbsrvr2                open
Please NOTE that there is no entry for TESTDB (  ora.testdb.db  )

SOLUTION
Register the database as follows
$ echo $ORACLE_HOME
/u01/app/oracle/11.2.0.3
$ srvctl add database -d TESTDB -o /u01/app/oracle/11.2.0.3
Try to bounce the database using the srvctl utility and check its status using crsctl now you should see an entry for TESTDB.
$ srvctl stop database -d TESTDB -o immediate
PRCC-1016 : TESTDB was already stopped
$ srvctl start database -d TESTDB
$ srvctl stop database -d TESTDB -o immediate
$ srvctl start database -d TESTDB
$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg           
                ONLINE  ONLINE       dbsrvr1
                ONLINE  ONLINE       dbsrvr2   
ora.LISTENER.lsnr   
                ONLINE  ONLINE       dbsrvr1
             ONLINE  ONLINE       dbsrvr2
ora.asm       
                ONLINE  ONLINE       dbsrvr1                Started
             ONLINE  ONLINE       dbsrvr2                Started
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.proddb.db        
      1        ONLINE  ONLINE       dbsrvr1                open
     2        ONLINE  ONLINE       dbsrvr2                open
ora.testdb.db        
      1        ONLINE  ONLINE       dbsrvr1                open
     2        ONLINE  ONLINE       dbsrvr2                open

STEP BY STEP GUIDE FOR CREATION OF SINGLE NODE STANDBY FROM 2 NODE RAC

• CRETE A PASSWORD FILE FROM RAC NODE 1 AND COPY IT TO ORACLE_HOME/dbs DIRECTORY TO NODE2 AND DISASTER RECOVERY SITE

PROD1$ cd $ORACLE_HOME/dbs

PROD1$orapwd file=orapwCATDB1 entries=5 force=y
PROD1$scp orapwCATDB1 oracle@PROD2:$ORACLE_HOME/dbs

PROD1$scp orapwCATDB1 oracle@STDBY:$ORACLE_HOME/dbs

                          ON SECOND NODE

PROD2$cd $ORACLE_HOME/dbs

PROD2$mv orapwCATDB1 orapwCATDB2

                          ON STANDBY NODE

STDBY$cd $ORACLE_HOME/dbs

STDBY$mv orapwCATDB1 orapwCATDBSTDBY

• MODIFY THE tnsnames.ora (found in $ORACLE_HOME/network/admin directories in each server) FILE  TO CONTAIN THE FOLLOWING ENTRIES

CATDB1 =

    (DESCRIPTION =

        (ADDRESS = (PROTOCOL = TCP)(HOST = PROD1)(PORT = 1521))

         (CONNECT_DATA =

            (SERVER = DEDICATED)

              (SERVICE_NAME = CATDB)

            (SID = CATDB1)

       )

  )

CATDB2 =    

     (DESCRIPTION =        

         (ADDRESS = (PROTOCOL = TCP)(HOST = PROD2)(PORT = 1521))

           (CONNECT_DATA = 

             (SERVER = DEDICATED)              

               (SERVICE_NAME = CATDB)           

            (SID = CATDB2)   

    )  

)

CATDBSTDBY =

    (DESCRIPTION =

        (ADDRESS = (PROTOCOL = TCP)(HOST = STDBY)(PORT = 1521))

         (CONNECT_DATA =

            (SERVER = DEDICATED)

              (SERVICE_NAME = CATDBSTDBY)

       )

  )

 

• TAKE BACKUP OF THE PRODUCTION DATABASE.

PROD1$export ORACLE_SID=CATDB1

PROD1$rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Tue Nov 18 11:12:24 2014

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CATDB (not mounted)

RMAN>run {
allocate channel c1 type disk;
allocate channel c2 type disk;
allocate channel c3 type disk;
allocate channel c4 type disk;
allocate channel c5 type disk;
allocate channel c6 type disk;
allocate channel c7 type disk;
allocate channel c8 type disk;
allocate channel c9 type disk;
allocate channel c10 type disk;
backup as compressed backupset database format '/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_%U';
backup as compressed backupset archivelog all format '/u03/backup/rman/CATDB/CATDB_Primary_archive_%U.arc';
backup current controlfile for standby format '/u03/backup/rman/CATDB/CATDB_standbycontrol.ctl';
}

• MOVE THE BACKUP TO THE DISASTER RECOVERY SITE

PROD1$cd /u03/backup/rman/CATDB

PROD1$ls

CATDB_Primary_databkp_standby_adpnto9u_1_1

CATDB_Primary_databkp_standby_aepnto9v_1_1
CATDB_Primary_databkp_standby_a8pnto9r_1_1
CATDB_Primary_databkp_standby_afpnto9v_1_1
CATDB_Primary_databkp_standby_a9pnto9s_1_1
CATDB_Primary_databkp_standby_agpntoa0_1_1
CATDB_Primary_databkp_standby_aapnto9s_1_1
CATDB_Primary_databkp_standby_abpnto9t_1_1
CATDB_Primary_databkp_standby_acpnto9t_1_1
CATDB_Primary_archive_ahpntof2_1_1.arc
CATDB_Primary_archive_ajpntof2_1_1.arc
CATDB_Primary_archive_aipntof2_1_1.arc
CATDB_Primary_archive_aopntof3_1_1.arc
CATDB_Primary_archive_ampntof3_1_1.arc
CATDB_Primary_archive_anpntof3_1_1.arc
CATDB_Primary_archive_alpntof3_1_1.arc
CATDB_Primary_archive_akpntof3_1_1.arc
CATDB_Primary_archive_appntof3_1_1.arc
CATDB_standbycontrol.ctl

PROD1$scp * oracle@STDBY:$PWD

• CHECK THAT ALL THE FILES ARRIVED ON THE INTENDED STANDBY SERVER

STDBY$cd /u03/backup/rman/CATDB

STDBY$ls
CATDB_Primary_databkp_standby_adpnto9u_1_1
CATDB_Primary_databkp_standby_aepnto9v_1_1
CATDB_Primary_databkp_standby_a8pnto9r_1_1
CATDB_Primary_databkp_standby_afpnto9v_1_1
CATDB_Primary_databkp_standby_a9pnto9s_1_1
CATDB_Primary_databkp_standby_agpntoa0_1_1
CATDB_Primary_databkp_standby_aapnto9s_1_1
CATDB_Primary_databkp_standby_abpnto9t_1_1
CATDB_Primary_databkp_standby_acpnto9t_1_1
CATDB_Primary_archive_ahpntof2_1_1.arc
CATDB_Primary_archive_ajpntof2_1_1.arc
CATDB_Primary_archive_aipntof2_1_1.arc
CATDB_Primary_archive_aopntof3_1_1.arc
CATDB_Primary_archive_ampntof3_1_1.arc
CATDB_Primary_archive_anpntof3_1_1.arc
CATDB_Primary_archive_alpntof3_1_1.arc
CATDB_Primary_archive_akpntof3_1_1.arc
CATDB_Primary_archive_appntof3_1_1.arc
CATDB_standbycontrol.ctl

• INITIATE THE RESTORE AND RECOVER PROCESS

STDBY$rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Tue Nov 18 11:12:51 2014

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CATDB (not mounted)

RMAN> set dbid=2344484010

executing command: SET DBID

RMAN> restore standby controlfile from '/u03/backup/rman/CATDB/CATDB_standbycontrol.ctl';

Starting restore at 11/18/14 11:14:19
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=3907 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:03
output file name=+DATA/catdbstdby/controlfile/control01.ctl
output file name=+ARCHIVE/catdbstdby/controlfile/control02.ctl
Finished restore at 11/18/14 11:14:24

RMAN> sql 'alter database mount';

sql statement: alter database mount
released channel: ORA_DISK_1

RMAN> run {
2> allocate channel c1 type disk;
3> allocate channel c2 type disk;
4> allocate channel c3 type disk;
5> allocate channel c4 type disk;
6> allocate channel c5 type disk;
7> allocate channel c6 type disk;
8> allocate channel c7 type disk;
9> allocate channel c8 type disk;
10> allocate channel c9 type disk;
11> allocate channel c10 type disk;
12> restore database;
13> }

allocated channel: c1
channel c1: SID=5167 device type=DISK

allocated channel: c2
channel c2: SID=5419 device type=DISK

allocated channel: c3
channel c3: SID=5545 device type=DISK

allocated channel: c4
channel c4: SID=5671 device type=DISK

allocated channel: c5
channel c5: SID=5797 device type=DISK

allocated channel: c6
channel c6: SID=5923 device type=DISK

allocated channel: c7
channel c7: SID=6049 device type=DISK

allocated channel: c8
channel c8: SID=6175 device type=DISK

allocated channel: c9
channel c9: SID=6301 device type=DISK

allocated channel: c10
channel c10: SID=6427 device type=DISK

Starting restore at 11/18/14 11:15:33
Starting implicit crosscheck backup at 11/18/14 11:15:33
Crosschecked 18 objects
Crosschecked 18 objects
Finished implicit crosscheck backup at 11/18/14 11:15:39

Starting implicit crosscheck copy at 11/18/14 11:15:39
Crosschecked 2 objects
Finished implicit crosscheck copy at 11/18/14 11:15:42

searching for all files in the recovery area
cataloging files...
no files cataloged

channel c1: starting datafile backup set restore
channel c1: specifying datafile(s) to restore from backup set
channel c1: restoring datafile 00001 to +REDO/catdb/datafile/system.260.798798557
channel c1: restoring datafile 00007 to +DATA/catdbstdby/datafile/t24distindex.483.800907503
channel c1: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_aapnto9s_1_1
channel c2: starting datafile backup set restore
channel c2: specifying datafile(s) to restore from backup set
channel c2: restoring datafile 00002 to +REDO/catdb/datafile/sysaux.261.798798559
channel c2: restoring datafile 00012 to +DATA/catdbstdby/datafile/t24distindex.286.800907661
channel c2: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_aepnto9v_1_1
channel c3: starting datafile backup set restore
channel c3: specifying datafile(s) to restore from backup set
channel c3: restoring datafile 00004 to +REDO/catdb/datafile/users.263.798798559
channel c3: restoring datafile 00009 to +DATA/catdbstdby/datafile/t24distdata.346.800907611
channel c3: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_adpnto9u_1_1
channel c4: starting datafile backup set restore
channel c4: specifying datafile(s) to restore from backup set
channel c4: restoring datafile 00003 to +REDO/catdb/datafile/undotbs1.262.798798559
channel c4: restoring datafile 00008 to +DATA/catdbstdby/datafile/t24distdata.479.800907597
channel c4: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_abpnto9t_1_1
channel c5: starting datafile backup set restore
channel c5: specifying datafile(s) to restore from backup set
channel c5: restoring datafile 00005 to +ARCHIVE/catdb/datafile/users.5407.800376881
channel c5: restoring datafile 00010 to +DATA/catdbstdby/datafile/t24distdata.288.800907625
channel c5: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_acpnto9t_1_1
channel c6: starting datafile backup set restore
channel c6: specifying datafile(s) to restore from backup set
channel c6: restoring datafile 00006 to +DATA/catdbstdby/datafile/t24distdata.295.800907475
channel c6: restoring datafile 00014 to +DATA/catdbstdby/datafile/undotbs2.285.831499529
channel c6: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_a8pnto9r_1_1
channel c7: starting datafile backup set restore
channel c7: specifying datafile(s) to restore from backup set
channel c7: restoring datafile 00011 to +DATA/catdbstdby/datafile/t24distdata.287.800907639
channel c7: restoring datafile 00013 to +DATA/catdbstdby/datafile/t24distindex.358.800907675
channel c7: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_a9pnto9s_1_1
channel c2: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_aepnto9v_1_1 tag=TAG20141118T110010
channel c2: restored backup piece 1
channel c2: restore complete, elapsed time: 00:00:57
channel c1: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_aapnto9s_1_1 tag=TAG20141118T110010
channel c1: restored backup piece 1
channel c1: restore complete, elapsed time: 00:01:18
channel c3: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_adpnto9u_1_1 tag=TAG20141118T110010
channel c3: restored backup piece 1
channel c3: restore complete, elapsed time: 00:02:18
channel c4: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_abpnto9t_1_1 tag=TAG20141118T110010
channel c4: restored backup piece 1
channel c4: restore complete, elapsed time: 00:02:28
channel c5: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_acpnto9t_1_1 tag=TAG20141118T110010
channel c5: restored backup piece 1
channel c5: restore complete, elapsed time: 00:03:47
channel c7: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_a9pnto9s_1_1 tag=TAG20141118T110010
channel c7: restored backup piece 1
channel c7: restore complete, elapsed time: 00:04:06
channel c6: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_databkp_standby_a8pnto9r_1_1 tag=TAG20141118T110010
channel c6: restored backup piece 1
channel c6: restore complete, elapsed time: 00:04:27
Finished restore at 11/18/14 11:20:13
released channel: c1
released channel: c2
released channel: c3
released channel: c4
released channel: c5
released channel: c6
released channel: c7
released channel: c8
released channel: c9
released channel: c10

RMAN> list backup of archivelog all;

If the last sequence number listed is n, choose n+1 for complete recovery. In my case the last sequence number listed by the above command is 8291
therefore 8292 is chosen for recovery.

RMAN> recover database until sequence 8292;

Starting recover at 11/18/14 11:22:47
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=5167 device type=DISK

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=1 sequence=8291
channel ORA_DISK_1: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_archive_aopntof3_1_1.arc
channel ORA_DISK_1: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_archive_aopntof3_1_1.arc tag=TAG20141118T110258
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+ARCHIVE/catdbstdby/archivelog/2014_11_18/thread_1_seq_8291.2837.863954589 thread=1 sequence=8291
channel ORA_DISK_1: starting archived log restore to default destination
channel ORA_DISK_1: restoring archived log
archived log thread=2 sequence=2686
channel ORA_DISK_1: reading from backup piece /u03/backup/rman/CATDB/CATDB_Primary_archive_appntof3_1_1.arc
channel ORA_DISK_1: piece handle=/u03/backup/rman/CATDB/CATDB_Primary_archive_appntof3_1_1.arc tag=TAG20141118T110258
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
archived log file name=+ARCHIVE/catdbstdby/archivelog/2014_11_18/thread_2_seq_2686.3707.863954589 thread=2 sequence=2686
channel default: deleting archived log(s)
archived log file name=+ARCHIVE/catdbstdby/archivelog/2014_11_18/thread_1_seq_8291.2837.863954589 RECID=1 STAMP=863954588
unable to find archived log
archived log thread=1 sequence=8292
Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: '+DATA/catdbstdby/datafile/system.1827.863954149'

RMAN-00571: =======================================================
RMAN-00569: ========= ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ======================================================
RMAN-03002: failure of recover command at 11/18/2014 11:23:11
RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 8292 and starting SCN of 22535896918

RMAN> exit


Recovery Manager complete.

Since we have commanded oracle to recover from a sequence which is not included in the backup, the above error is expected.

STDBY$sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Tue Nov 18 11:24:01 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options

SQL> select name,open_mode,database_role from v$database;

NAME      OPEN_MODE            DATABASE_ROLE
---------      --------------------            ------------------------
CATDB     MOUNTED               PHYSICAL STANDBY

SQL> ALTER DATABASE RECOVER  managed standby database disconnect using current logfile nodelay noparallel ;

Database altered.

SQL>

Check the alert log while you are making some logswitches from both primary instances, mrp process should apply the logfiles.

To check the time gap between the primary and the standby you can use the following query on the standby

SQL>select NAME,VALUE 
from 
v$dataguard_stats 
where name = 'apply lag';
NAME                                                             VALUE
-------------------------                                       ----------------------------
apply lag                                                      +00 00:06:34

Script for Compiling INVALID objects

SET ECHO        OFF
SET FEEDBACK    OFF
SET HEADING     OFF
SET LINESIZE    180
SET PAGESIZE    0
SET TERMOUT     ON
SET TIMING      OFF
SET TRIMOUT     ON
SET TRIMSPOOL   ON
SET VERIFY      OFF

CLEAR COLUMNS
CLEAR BREAKS
CLEAR COMPUTES

spool compile.sql

SELECT  'alter ' ||
       decode(object_type, 'PACKAGE BODY', 'package', object_type) ||
       ' ' ||
       object_name||
       ' compile' ||
       decode(object_type, 'PACKAGE BODY', ' body;', ';')
FROM   dba_objects
WHERE  status = 'INVALID'
/

spool off

SET ECHO        off
SET FEEDBACK    off
SET HEADING     off
SET LINESIZE    180
SET PAGESIZE    0
SET TERMOUT     on
SET TIMING      off
SET TRIMOUT     on
SET TRIMSPOOL   on
SET VERIFY      off

@compile

SET FEEDBACK    6
SET HEADING     ON