Not able to start 10gR2 RAC Database Running on 11gR2 Grid infra

Written By askMLabs on Wednesday, September 1, 2010 | 6:50 AM

We have 2 node 10gRAC database running on 11gR2 grid infra structure.
When the database was created, both the instances were running good.
Recently one instance was down due to some reason and we are not able to start the instance successfully.

When we are trying to start the instance , it says , it cant identify the control file. But the other instance is able to start and stop successfully.

When we start the database , we get the following error ...

ORA-00205: error in identifying control file, check alert log for more info

SQL> startup
ORACLE instance started.Total System Global Area 1610612736 bytes
Fixed Size 2084296 bytes
Variable Size 369099320 bytes
Database Buffers 1224736768 bytes
Redo Buffers 14692352 bytes
ORA-00205: error in identifying control file, check alert log for more info

Alert Log content :
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '+RACDB_DATA/rac10g/control01.ctl'
ORA-15081: failed to submit an I/O operation to a disk
Mon Aug 30 07:40:52 2010
ORA-205 signalled during: ALTER DATABASE MOUNT...
Mon Aug 30 07:40:55 2010
Errors in file /u01/apps/oracle/admin/rac10g/bdump/rac10g2_mmon_32634.trc:
ORA-00600: internal error code, arguments: [kmgs_parameter_update_timeout_1], [15081], [], [], [], [], [], []
ORA-15081: failed to submit an I/O operation to a disk
Mon Aug 30 07:40:56 2010
Trace dumping is performing id=[cdmp_20100830074056]
Mon Aug 30 07:41:59 2010
Restarting dead background process MMON
MMON started with pid=16, OS id=329

Other Details :
SQL> select GROUP_NUMBER,name,state from v$asm_diskgroup;

GROUP_NUMBER NAME STATE
------------ ------------------------------ -----------
1 CRS MOUNTED
2 FRA MOUNTED
3 RACDB_DATA MOUNTED

SQL> select group_number,instance_name ,db_name , status from v$asm_client;

GROUP_NUMBER INSTANCE_NAME DB_NAME STATUS
------------ -------------------- -------------------- ----------
1 +ASM2 +ASM CONNECTED
3 +ASM2 +ASM CONNECTED

===> After starting the rac10g2 ( instance on rac02) with nomount.(the problematic instance) :
SQL> select group_number,instance_name ,db_name , status from v$asm_client;

GROUP_NUMBER INSTANCE_NAME DB_NAME STATUS
------------ -------------------- -------------------- ----------
1 +ASM2 +ASM CONNECTED
3 +ASM2 +ASM CONNECTED
3 rac10g2 rac10g CONNECTED

SQL>

SQL> sho parameter control

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +RACDB_DATA/rac10g/control01.c
tl, +RACDB_DATA/rac10g/control
02.ctl, +RACDB_DATA/rac10g/con
trol03.ctl

Troubleshooting
From the database owner ( oracle ) on all the nodes

[oracle@rac01 ~]$ hostname
rac01.example.com
[oracle@rac01 ~]$ whoami
oracle
[oracle@rac01 ~]$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle dba 112468374 Aug 17 10:47 /u01/apps/oracle/product/10.2.0/db_1/bin/oracle
[oracle@rac01 ~]$ echo $ORACLE_HOME
/u01/apps/oracle/product/10.2.0/db_1
[oracle@rac01 ~]$
[oracle@rac01 ~]$ whoami
oracle
[oracle@rac01 ~]$ id
uid=1659(oracle) gid=501(dba) groups=501(dba),503(oper),552(admin2),1000(oinstall),1201(asmdba)
[oracle@rac01 ~]$

[oracle@rac02 ~]$ hostname
rac02.example.com
[oracle@rac02 ~]$ whoami
oracle
[oracle@rac02 ~]$ ls -l $ORACLE_HOME/bin/oracle
-rwsr-s--x 1 oracle dba 112468374 Aug 17 10:47 /u01/apps/oracle/product/10.2.0/db_1/bin/oracle
[oracle@rac02 ~]$ echo $ORACLE_HOME
/u01/apps/oracle/product/10.2.0/db_1
[oracle@rac02 ~]$
[oracle@rac02 ~]$ whoami
oracle
[oracle@rac02 ~]$ id
uid=1659(oracle) gid=501(dba) groups=501(dba),552(admin2),1000(oinstall),1201(asmdba),1301(oper)
[oracle@rac02 ~]$

From the Grid owner ( grid ) on all the nodes

[grid@rac01 grid]$ hostname
rac01.example.com
[grid@rac01 grid]$
[grid@rac01 grid]$ whoami
grid
[grid@rac01 grid]$ echo $ORACLE_HOME;echo $ORACLE_SID
/o001/home/11.2.0/grid
+ASM1
[grid@rac01 grid]$ ls -l $ORACLE_HOME/bin/oracle
-rwsrws--x 1 grid dba 184286251 Apr 5 15:51 /o001/home/11.2.0/grid/bin/oracle*
[grid@rac01 grid]$ whoami
grid
[grid@rac01 grid]$ id
uid=1100(grid) gid=501(dba) groups=501(dba),1200(asmadmin),1201(asmdba),1202(asmoper)
[grid@rac01 grid]$

[grid@rac02 grid]$ hostname
rac02.example.com
[grid@rac02 grid]$ whoami
grid
[grid@rac02 grid]$ echo $ORACLE_HOME;echo $ORACLE_SID
/o001/home/11.2.0/grid
+ASM2
[grid@rac02 grid]$ ls -l $ORACLE_HOME/bin/oracle
-rwsrws--x 1 grid dba 184286251 Apr 5 15:54 /o001/home/11.2.0/grid/bin/oracle*
[grid@rac02 grid]$ whoami
grid
[grid@rac02 grid]$ id
uid=1100(grid) gid=501(dba) groups=501(dba),1200(asmadmin),1201(asmdba),1202(asmoper)
[grid@rac02 grid]$

Then we changed the group ownership of Oracle executable to OS ASM group using below command after logging as ROOT user :

cd /u01/apps/oracle/product/10.2.0/db_1/bin
chgrp asmadmin oracle
chmod 6751 oracle
ls -l oracle

"This allows the database instance to access the ASM disks without having to make the database software owner a member of the OS ASM group."

Then re-login newly as ORACLE user and then started ORACLE Database instance.

[oracle@rac01 bin]$ srvctl status database -d rac10g
Instance rac10g1 is running on node rac01
Instance rac10g2 is running on node rac02
[oracle@rac01 bin]$

Hope it helps
SRI

Share this article :

askMLabs