askMLabs

This is an informative article to troubleshoot an issue in 11gR2 RAC.
In this article, i want to present one of the issue we identified and solved in 11gR2 RAC. This issue was caused due to one DBA accidentally running 755 permission on all the files in GRID_HOME/bin directory. We observed that the database running on this node is down and it is not communicating with ASM instance running on the same node after running the permissions command.

The error messages when trying to start the database are :

[oracle@b600e6tr01 oracle]$ srvctl start database -d racdb
PRCR-1079 : Failed to start resource ora.racdb.db
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
ORA-01078: failure in processing system parameters
CRS-2674: Start of 'ora.racdb.db' on 'b600e6tr01' failed
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Process ID: 0
Session ID: 0 Serial number: 0
CRS-2528: Unable to place an instance of 'ora.racdb.db' as all possible servers are occupied by the resource already
[oracle@b600e6tr01 oracle]$

Alert log file showed the following content :

ORA-00210: cannot open the specified control file
ORA-00202: control file: '+FRA/racdb/controlfile/current.256.715627363'
ORA-17503: ksfdopn:2 Failed to open file +FRA/racdb/controlfile/current.256.715627363
ORA-15001: diskgroup "FRA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 32768
Additional information: 10
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RACDB_DATA/racdb/controlfile/current.256.715627363'
ORA-17503: ksfdopn:2 Failed to open file +RACDB_DATA/racdb/controlfile/current.256.715627363
ORA-15001: diskgroup "RACDB_DATA" does not exist or is not mounted
ORA-15055: unable to connect to ASM instance
ORA-01034: ORACLE not available
ORA-27123: unable to attach to shared memory segment
Linux-x86_64 Error: 13: Permission denied
Additional information: 32768
Additional information: 10
Starting background process ASMB

We started comparing the permissions on node 1 with permission on node 2 for all the files in GRID_HOME/bin directory. Then identified that some files in GRID_HOME/bin are missing suid and sgid permission on the executable. The executable to our interest here is "oracle". This executable is missing the suid and sgid permission.

[grid@b600e6tr01 bin]$ ls -lrt oracle
-rwxr-x--x 1 grid dba 184431149 Apr 19 11:54 oracle*
[grid@b600e6tr01 bin]$
[grid@b600e6tr02 bin]$ ls -lrt oracle
-rwsr-s--x 1 grid dba 184431149 Apr 19 12:36 oracle*
[grid@b600e6tr02 bin]$
[grid@b600e6tr01 bin]$ chmod ug+s oracle
[oracle@b600e6tr01 bin]$ srvctl start instance -d racdb -i racdb1
[oracle@b600e6tr01 bin]$ srvctl status database -d racdb
Instance racdb1 is running on node b600e6tr01
Instance racdb2 is running on node b600e6tr02
[oracle@b600e6tr01 bin]$

That solves the issue ...... :-)

Hope it helps
SRI