Tuesday, January 27, 2015

ORA-27300: OS System Dependent Operation: Fork Failed With Status: 12

ALERT LOG ENTRY
Process startup failed, error stack:
Errors in file /u01/app/oracle/diag/rdbms/cbelive/PRODDB2/trace/PRODDB2_psp0_5898502.trc:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
====================CONTENT OF TRACE FILE======================
Trace file /u01/app/oracle/diag/rdbms/cbelive/PRODDB2/trace/PRODDB2_psp0_5898502.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u01/app/11.2.0.3/oracle
System name:    AIX
Node name:      t24db2
Release:        1
Version:        6
Machine:        00C941664C00
Instance name: PRODDB2
Redo thread mounted by this instance: 2
Oracle process number: 3
Unix process pid: 5898502, image: oracle@t24db2 (PSP0)

*** 2015-01-27 13:30:39.519
*** SESSION ID:(379.1) 2015-01-27 13:30:39.519
*** CLIENT ID:() 2015-01-27 13:30:39.519
*** SERVICE NAME:(SYS$BACKGROUND) 2015-01-27 13:30:39.519
*** MODULE NAME:() 2015-01-27 13:30:39.519
*** ACTION NAME:() 2015-01-27 13:30:39.519

Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

*** 2015-01-27 13:32:50.138
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3

*** 2015-01-27 13:33:24.157
Process startup failed, error stack:
ORA-27300: OS system dependent operation:fork failed with status: 12
ORA-27301: OS failure message: Not enough space
ORA-27302: failure occurred at: skgpspawn3
~
===========================================================

CAUSE


Insufficient paging space defined for the system program using excessive amount of paging space.

SOLUTION

Increase the OS paging space.

For AIX, the next APAR is advised:

Fix pack for: IZ19670: CANNOT INCREASE PAGING SPACE UNDER LOAD APPLIES TO AIX 6100-01

Please note the specific patches below:
  5300-09 - use AIX APAR IZ19504
  6100-01 - use AIX APAR IZ19670
  6100-02 - use AIX APAR IZ20052

For more information, please visit:
https://www-304.ibm.com/support/docview.wss?uid=isg1IZ19670

ORA-19809 LIMIT EXCEEDED FOR RECOVERY FILES

We encounter this error when the archivelog destination (specified by DB_RECOVERY_FILE_DEST parameter ) is full.
Solution
1. When we have time and resource consider adding disks to the diskgroup upon which the archive log destination resides and then increase the parameter DB_RECOVERY_FILE_DEST_SIZE parameter.
ALTER SYSTEM SET DB_RECOVERY_FILE_DEST_SIZE=XXXG SCOPE=BOTH;
2.If the above solution doesn't work, use rman to delete the archive log which have been backed up to make some space in the destination.
export ORACLE_SID=TESTDB
rman target /
RMAN> delete noprompt archivelog all completed before 'sysdate - 1';
The above command deletes archivelogs which have been completed 24 hours ago if you have archivelog backup in last 24 hours then you are good to go otherwise change the sysdate - to larger number of days.
3. Consider configuring rman archive log deletion and retention policies.