facebook twitter youtube facebook facebook facebook

E-Mail : info@askmlabs.com

Phone : +1.215.353.8306

Latest Post
2:10 PM

How To Restore Database From Different Incarnation

Written By askMLabs on Thursday, April 4, 2013 | 2:10 PM

What is incarnation?
When we open database with RESETLOGS option, the database starts a new incarnation. This new incarnation resets the log sequence number to 1, and then gives the online redo logs a new time stamp and SCN.

When the database starts a new incarnation, are we able to use the old backups taken in previous incarnation for restore and recovery purpose as the present log sequence number is reset to 1.

The present article discusses about the details of how to use the backups in old incarnation for restore and recovery purpose after the database has been opened with resetlogs for many times.

I will explain the details considering a practical approach in our training environment. First i will explain the existing environment and available backups.
Then i will take you to details of how these backups are used to restore and recover to the same server (or to different server) considering that the database has been opened many times with resetlogs option.

Environment Details :
Let us assume that we have a training environment where we have some gold backups for the database. Whenever we do any modification to the database, we will take a new goldbackup and we will keep restoring this latest new backup every week twice( Thursday night and Saturday night).

Following Details shows the different goldbackups available for the training environment.
RMAN> list backup summary;List of Backups
===============
Key     TY LV S Device Type Completion Time #Pieces #Copies Compressed Tag
------- -- -- - ----------- --------------- ------- ------- ---------- ---
327559  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327560  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327561  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327562  B  0  A SBT_TAPE    04-AUG-12       1       1       YES        NOEXPIRE
327785  B  F  A SBT_TAPE    04-AUG-12       1       1       NO         NOEXPIRE
351357  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351358  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351359  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351360  B  0  A SBT_TAPE    13-OCT-12       1       1       YES        NOEXPIRE
351583  B  F  A SBT_TAPE    13-OCT-12       1       1       NO         NOEXPIRE
355951  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355952  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355953  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
355954  B  0  A SBT_TAPE    27-OCT-12       1       1       YES        NOEXPIRE
356177  B  F  A SBT_TAPE    27-OCT-12       1       1       NO         NOEXPIRE
386046  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386047  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386048  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386049  B  0  A SBT_TAPE    26-JAN-13       1       1       YES        NOEXPIRE
386275  B  F  A SBT_TAPE    26-JAN-13       1       1       NO         NOEXPIRE
387232  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387233  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387234  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387235  B  0  A SBT_TAPE    29-JAN-13       1       1       YES        NOEXPIRE
387462  B  F  A SBT_TAPE    29-JAN-13       1       1       NO         NOEXPIRE
392057  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392058  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392059  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392060  B  0  A SBT_TAPE    09-FEB-13       1       1       YES        NOEXPIRE
392286  B  F  A SBT_TAPE    09-FEB-13       1       1       NO         NOEXPIRE
392347  B  F  A SBT_TAPE    09-FEB-13       1       1       NO         TAG20130209T021024

RMAN>


So everytime when we restore the database from the latest gold backup, we will open the database with resetlogs option and it starts a new incarnation.
These details are shown below ..
RMAN> list incarnation of database TRNGDB;
List of Database Incarnations
DB Key  Inc Key DB Name  DB ID            STATUS  Reset SCN  Reset Time
------- ------- -------- ---------------- --- ---------- ----------
......
.......
158938  325435  TRNGDB   3679252776       ORPHAN  88080533256 29-JUL-12
158938  326880  TRNGDB   3679252776       PARENT  88080533256 02-AUG-12
158938  323440  TRNGDB   3679252776       ORPHAN  88080533256 26-JUL-12
158938  330059  TRNGDB   3679252776       ORPHAN  88080774587 09-AUG-12
158938  332058  TRNGDB   3679252776       ORPHAN  88080774587 12-AUG-12
158938  335023  TRNGDB   3679252776       ORPHAN  88080774587 23-AUG-12
158938  334177  TRNGDB   3679252776       ORPHAN  88080774587 19-AUG-12
158938  350823  TRNGDB   3679252776       PARENT  88080774587 11-OCT-12
158938  350133  TRNGDB   3679252776       ORPHAN  88080774587 07-OCT-12
158938  337328  TRNGDB   3679252776       ORPHAN  88080774587 26-AUG-12
158938  338003  TRNGDB   3679252776       ORPHAN  88080774587 30-AUG-12
158938  339685  TRNGDB   3679252776       ORPHAN  88080774587 02-SEP-12
158938  340363  TRNGDB   3679252776       ORPHAN  88080774587 06-SEP-12
158938  342063  TRNGDB   3679252776       ORPHAN  88080774587 09-SEP-12
158938  342743  TRNGDB   3679252776       ORPHAN  88080774587 13-SEP-12
158938  344429  TRNGDB   3679252776       ORPHAN  88080774587 16-SEP-12
158938  346103  TRNGDB   3679252776       ORPHAN  88080774587 23-SEP-12
158938  347243  TRNGDB   3679252776       ORPHAN  88080774587 27-SEP-12
158938  348499  TRNGDB   3679252776       ORPHAN  88080774587 30-SEP-12
158938  349183  TRNGDB   3679252776       ORPHAN  88080774587 04-OCT-12
158938  352901  TRNGDB   3679252776       ORPHAN  88081045250 18-OCT-12
158938  354443  TRNGDB   3679252776       ORPHAN  88081045250 21-OCT-12
158938  355286  TRNGDB   3679252776       PARENT  88081045250 25-OCT-12
158938  364387  TRNGDB   3679252776       ORPHAN  88081234176 22-NOV-12
158938  363526  TRNGDB   3679252776       ORPHAN  88081234176 18-NOV-12
158938  362107  TRNGDB   3679252776       ORPHAN  88081234176 15-NOV-12
158938  361247  TRNGDB   3679252776       ORPHAN  88081234176 11-NOV-12
158938  359827  TRNGDB   3679252776       ORPHAN  88081234176 08-NOV-12
158938  358965  TRNGDB   3679252776       ORPHAN  88081234176 04-NOV-12
158938  357545  TRNGDB   3679252776       ORPHAN  88081234176 01-NOV-12
158938  365807  TRNGDB   3679252776       ORPHAN  88081234176 25-NOV-12
158938  384807  TRNGDB   3679252776       PARENT  88081234176 24-JAN-13
158938  383953  TRNGDB   3679252776       ORPHAN  88081234176 20-JAN-13
158938  382547  TRNGDB   3679252776       ORPHAN  88081234176 17-JAN-13
158938  381687  TRNGDB   3679252776       ORPHAN  88081234176 13-JAN-13
158938  380267  TRNGDB   3679252776       ORPHAN  88081234176 10-JAN-13
158938  379411  TRNGDB   3679252776       ORPHAN  88081234176 06-JAN-13
158938  378007  TRNGDB   3679252776       ORPHAN  88081234176 03-JAN-13
158938  377152  TRNGDB   3679252776       ORPHAN  88081234176 30-DEC-12
158938  375747  TRNGDB   3679252776       ORPHAN  88081234176 27-DEC-12
158938  374892  TRNGDB   3679252776       ORPHAN  88081234176 23-DEC-12
158938  373487  TRNGDB   3679252776       ORPHAN  88081234176 20-DEC-12
158938  372631  TRNGDB   3679252776       ORPHAN  88081234176 16-DEC-12
158938  371227  TRNGDB   3679252776       ORPHAN  88081234176 13-DEC-12
158938  366667  TRNGDB   3679252776       ORPHAN  88081234176 29-NOV-12
158938  368086  TRNGDB   3679252776       ORPHAN  88081234176 02-DEC-12
158938  368947  TRNGDB   3679252776       ORPHAN  88081234176 06-DEC-12
158938  370366  TRNGDB   3679252776       ORPHAN  88081234176 09-DEC-12
158938  391514  TRNGDB   3679252776       PARENT  88082816786 07-FEB-13
158938  387719  TRNGDB   3679252776       ORPHAN  88082816786 01-FEB-13
158938  390618  TRNGDB   3679252776       ORPHAN  88082816786 03-FEB-13
158938  404336  TRNGDB   3679252776       ORPHAN  88082894636 21-FEB-13
158938  403425  TRNGDB   3679252776       ORPHAN  88082894636 17-FEB-13
158938  401240  TRNGDB   3679252776       ORPHAN  88082894636 14-FEB-13
158938  416836  TRNGDB   3679252776       CURRENT 88082894636 21-MAR-13
158938  412180  TRNGDB   3679252776       ORPHAN  88082894636 10-MAR-13
158938  409274  TRNGDB   3679252776       ORPHAN  88082894636 03-MAR-13
158938  415096  TRNGDB   3679252776       ORPHAN  88082894636 17-MAR-13
158938  413356  TRNGDB   3679252776       ORPHAN  88082894636 14-MAR-13
158938  406349  TRNGDB   3679252776       ORPHAN  88082894636 24-FEB-13
158938  407256  TRNGDB   3679252776       ORPHAN  88082894636 28-FEB-13
158938  410176  TRNGDB   3679252776       ORPHAN  88082894636 07-MAR-13

RMAN>


As long as i am restoring the latest goldbackups to the training environment, it uses the current incarnation and able to restore the latest backups and able to open the database with resetlogs option.

But if i want to restore any previous backups(backups taken at a different incarnation) to the training environment,i may not be able to restore the backups to the current incarnation.

Let us assume that i have my latest backups available on 09-OCT-2013 as per the above details and my current incarnation is 416836 (on 21-MAR-2013).
Now i have a requirement where i need to restore the backups available on 27-OCT-2012 to the same server. So as long as my current incarnation is 416836, i will not be able to use these backups on 27-OCT-2012 to restore the database.
So i need to first change the incarnation to be able to restore the backups on 27-OCT-2012.

For your easy understanding to identify to which incarnation to change, i have mixed backup details and incarnation details and orderded the details date wise below ....
158938  349183  TRNGDB   3679252776       ORPHAN  88080774587 04-OCT-12
13-OCT-12  <<<== Date on which we have Backups
158938  350133  TRNGDB   3679252776       ORPHAN  88080774587 07-OCT-12
158938  350823  TRNGDB   3679252776       PARENT  88080774587 11-OCT-12
158938  352901  TRNGDB   3679252776       ORPHAN  88081045250 18-OCT-12
158938  354443  TRNGDB   3679252776       ORPHAN  88081045250 21-OCT-12
158938  355286  TRNGDB   3679252776       PARENT  88081045250 25-OCT-12
27-OCT-12 <<<== Date on which we have  Backups
158938  357545  TRNGDB   3679252776       ORPHAN  88081234176 01-NOV-12
158938  358965  TRNGDB   3679252776       ORPHAN  88081234176 04-NOV-12
158938  359827  TRNGDB   3679252776       ORPHAN  88081234176 08-NOV-12
158938  361247  TRNGDB   3679252776       ORPHAN  88081234176 11-NOV-12
158938  362107  TRNGDB   3679252776       ORPHAN  88081234176 15-NOV-12
158938  363526  TRNGDB   3679252776       ORPHAN  88081234176 18-NOV-12
158938  364387  TRNGDB   3679252776       ORPHAN  88081234176 22-NOV-12
158938  365807  TRNGDB   3679252776       ORPHAN  88081234176 25-NOV-12
158938  366667  TRNGDB   3679252776       ORPHAN  88081234176 29-NOV-12

When a database goes through multiple incarnations, some backups can become orphaned. The simple way to find non-orphaned backups and orphaned backups are to form a table to list the incarnation number and orphaned and non-orphaned backups to each incarnation.
In our case, in simple words, we need to restore backups from 27-OCT-12. So identify the incarnation number for which this backup is non-orphaned.

I am considering the incarnation before and after the 27-OCT-12 to identify orphaned and non-orphaned backups. We have incarnations 357545(on 01-NOV-12)  and 355286 (25-OCT-12).
For incarnation 357545(on 01-NOV-12) : All the backups taken after the date 01-NOV-2012 are non-orphaned and all the backups before this incarnation are orphaned backups assuming that there is no other incarnation after the incarnation 357545. So in our case the backups on 27-OCT-2012 are orphaned backups if i set incarnation 357545.


For incarnation 355286(on 25-OCT-12) : All the backups taken after the date 25-OCT-2012 are non-orphaned and all the backups before this incarnation are orphaned backups assuming that there is no other incarnation after the incarnation 355286. So in our case the backups on 27-OCT-2012 are non-orphaned backups if i set incarnation 355286.

Now we have the details, which incarnation we need to set for restoring the 27-OCT-2012 backups.

Also get the controlfile information from the 27-OCT-2012 backups using
RMAN> list backup of controlfile;

NOTE : In our case the controlfile details are '4vnoosbv_1_1'

Then use the following syntax to restore and recover the database...
RMAN> set dbid=3679252776
executing command: SET DBID
database name is "TRNGDB" and DBID is 3679252776

RMAN> RESET DATABASE TO INCARNATION 355286;

database reset to incarnation 355286

RMAN>  run {
2> allocate channel t1 type 'SBT_TAPE' parms="ENV=(TDPO_OPTFILE=/tmp/askm/TRNGDB_tdpo.opt)";
3> restore controlfile from '4vnoosbv_1_1';
4> }

allocated channel: t1
channel t1: SID=396 device type=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.5.2.0

Starting restore at 22-MAR-13

channel t1: restoring control file
channel t1: restore complete, elapsed time: 00:00:01
output file name=/u01/oradata/TRNGDB/control01.ctl
output file name=/u01/oradata/TRNGDB/control02.ctl
output file name=/u01/oradata/TRNGDB/control03.ctl
Finished restore at 22-MAR-13
released channel: t1

RMAN> run {
2> allocate channel t1 type 'SBT_TAPE' parms="ENV=(TDPO_OPTFILE=/tmp/askm/TRNGDB_tdpo.opt)";
3> restore controlfile from '4vnoosbv_1_1';
4> alter database mount;
5> restore database;
6> recover database noredo;
7> release channel t1;
8> }

allocated channel: t1
channel t1: SID=396 device type=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.5.2.0

Starting restore at 22-MAR-13

channel t1: restoring control file
channel t1: restore complete, elapsed time: 00:00:01
output file name=/u01/oradata/TRNGDB/control01.ctl
output file name=/u01/oradata/TRNGDB/control02.ctl
output file name=/u01/oradata/TRNGDB/control03.ctl
Finished restore at 22-MAR-13

database mounted

Starting restore at 22-MAR-13

channel t1: starting datafile backup set restore
channel t1: specifying datafile(s) to restore from backup set
channel t1: restoring datafile 00003 to /u01/oradata/TRNGDB/sysaux01.dbf
channel t1: restoring datafile 00004 to /u01/oradata/TRNGDB/psdefault.dbf
channel t1: restoring datafile 00009 to /u01/oradata/TRNGDB/aalarge.dbf
channel t1: restoring datafile 00011 to /u01/oradata/TRNGDB/adapp.dbf
channel t1: restoring datafile 00012 to /u01/oradata/TRNGDB/adapp001002.dbf
channel t1: restoring datafile 00014 to /u01/oradata/TRNGDB/avapp.dbf
channel t1: restoring datafile 00015 to /u01/oradata/TRNGDB/avapp001002.dbf
channel t1: restoring datafile 00017 to /u01/oradata/TRNGDB/avapp001004.dbf
channel t1: restoring datafile 00022 to /u01/oradata/TRNGDB/ccapp.dbf
channel t1: restoring datafile 00030 to /u01/oradata/TRNGDB/cularg3.dbf
channel t1: restoring datafile 00031 to /u01/oradata/TRNGDB/diapp.dbf
channel t1: restoring datafile 00036 to /u01/oradata/TRNGDB/faapp001002.dbf
channel t1: restoring datafile 00038 to /u01/oradata/TRNGDB/fgapp.dbf
channel t1: restoring datafile 00041 to /u01/oradata/TRNGDB/giapp.dbf
channel t1: restoring datafile 00043 to /u01/oradata/TRNGDB/glapp.dbf
channel t1: restoring datafile 00047 to /u01/oradata/TRNGDB/hpapp001002.dbf
channel t1: restoring datafile 00054 to /u01/oradata/TRNGDB/hrapp4001002.dbf
channel t1: restoring datafile 00063 to /u01/oradata/TRNGDB/hrwork001002.dbf
channel t1: restoring datafile 00065 to /u01/oradata/TRNGDB/htapp001002.dbf
channel t1: restoring datafile 00067 to /u01/oradata/TRNGDB/inlarge.dbf
channel t1: restoring datafile 00068 to /u01/oradata/TRNGDB/paapp.dbf
channel t1: restoring datafile 00069 to /u01/oradata/TRNGDB/paapp001001.dbf
channel t1: restoring datafile 00070 to /u01/oradata/TRNGDB/palarge.dbf
channel t1: restoring datafile 00077 to /u01/oradata/TRNGDB/piwork001002.dbf
channel t1: restoring datafile 00081 to /u01/oradata/TRNGDB/psindx001002.dbf
channel t1: restoring datafile 00085 to /u01/oradata/TRNGDB/psindx001006.dbf
channel t1: restoring datafile 00086 to /u01/oradata/TRNGDB/psindx001007.dbf
channel t1: restoring datafile 00089 to /u01/oradata/TRNGDB/psindx001010.dbf
channel t1: restoring datafile 00098 to /u01/oradata/TRNGDB/ptcmstar.dbf
channel t1: restoring datafile 00100 to /u01/oradata/TRNGDB/ptprc.dbf
channel t1: restoring datafile 00107 to /u01/oradata/TRNGDB/pttlrg001002.dbf
channel t1: restoring datafile 00115 to /u01/oradata/TRNGDB/pywork.dbf
channel t1: restoring datafile 00121 to /u01/oradata/TRNGDB/saapp001005.dbf
channel t1: restoring datafile 00125 to /u01/oradata/TRNGDB/salarge.dbf
channel t1: restoring datafile 00136 to /u01/oradata/TRNGDB/tlapp001002.dbf
channel t1: restoring datafile 00140 to /u01/oradata/TRNGDB/tlwork001001.dbf
channel t1: restoring datafile 00142 to /u01/oradata/TRNGDB/waapp.dbf
channel t1: restoring datafile 00147 to /u01/oradata/TRNGDB/rbks001002.dbf
channel t1: restoring datafile 00149 to /u01/oradata/TRNGDB/epapp001001.dbf
channel t1: restoring datafile 00161 to /u01/oradata/TRNGDB/eoeiapp001001.dbf
channel t1: restoring datafile 00162 to /u01/oradata/TRNGDB/eodsapp001001.dbf
channel t1: restoring datafile 00166 to /u01/oradata/TRNGDB/hrslarge001001.dbf
channel t1: restoring datafile 00170 to /u01/oradata/TRNGDB/eopplrg001001.dbf
channel t1: restoring datafile 00174 to /u01/oradata/TRNGDB/eocuapp001001.dbf
channel t1: restoring datafile 00178 to /u01/oradata/TRNGDB/eoeilrg001001.dbf
channel t1: restoring datafile 00182 to /u01/oradata/TRNGDB/eoiuwrk001001.dbf
channel t1: restoring datafile 00186 to /u01/oradata/TRNGDB/erwork001001.dbf
channel t1: restoring datafile 00190 to /u01/oradata/TRNGDB/hrlarg1001001.dbf
channel t1: restoring datafile 00192 to /u01/oradata/TRNGDB/psimgr001001.dbf
channel t1: restoring datafile 00196 to /u01/oradata/TRNGDB/pswork001001.dbf
channel t1: restoring datafile 00200 to /u01/oradata/TRNGDB/hrapp001004.dbf
channel t1: restoring datafile 00203 to /u01/oradata/TRNGDB/clnd001001.dbf
channel t1: restoring datafile 00205 to /u01/oradata/TRNGDB/clnd001002
channel t1: reading from backup piece backup_TRNGDB_3679252776_set155_piece1_copy1_20121027_4rnoorig_1_1
channel t1: piece handle=backup_TRNGDB_3679252776_set155_piece1_copy1_20121027_4rnoorig_1_1 tag=NOEXPIRE

..
....
RMAN> alter database open resetlogs;

database opened
new incarnation of database registered in recovery catalog
starting full resync of recovery catalog
full resync complete

RMAN> exit


So we are able to successfully restore the backups from a different incarnation. The same procedure can be applied if you plan to restore the database to a different server as long as the tsm configuration is properly done.

Hope It Helps

SRI
Labels: , ,
6:03 PM

Troubleshooting RAC Services Startup

Written By askMLabs on Wednesday, April 3, 2013 | 6:03 PM

Recently we had a DR(Disaster Recovery) testing going on and we had RAC replica at the DR site for DR testing. The RAC servers are copied to DR setup, bit by bit. Then tried to start the RAC services at DR site and could not start them due to some issues.

Then I tried to troubleshoot the issue .....

This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.

STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.

STEP 2: Verify storage on both rac servers
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su -
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh        8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:1:1 sdk        8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:1:0 sdj        8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg        8:96  [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su -
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf        8:80  [active][ready]
\_ 3:0:0:2 sdi        8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc        8:32  [active][ready]
\_ 3:0:1:2 sdl        8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk        8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde        8:64  [active][ready]
\_ 3:0:0:1 sdh        8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd        8:48  [active][ready]
\_ 3:0:0:0 sdg        8:96  [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj        8:144 [active][ready]
[root@askmdbrac02 ~]#

Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.

STEP 3: Check high availability services and cluster ready services on both nodes with the following commands
[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$

Observation : The high availability services are up , but crs services are down.

STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.

Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.

STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
.....
.....
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'askmdbrac01'.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource 'ora.cssd'; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.

Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can't find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.

Then i tried to see if i can see voting disks through command.

SETP 6 :  Check the voting disks availability ( 11gR2 has voting disks in asm. I don't have asm instance up now. So i will get error if i try to query the voting disks on asm)
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$

Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.

STEP 7 :  Tried re-enabling the asmlibs as follows on both the nodes ...
[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [  OK  ]
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#  /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#

Observation : Not able to start services even after restarting the asmlib.

STEP 8 : Tried to re-configure the asmlib as below .....
[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#

[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting <ENTER> without typing an
answer will keep that current value.  Ctrl-C will abort.

Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks:                              [  OK  ]
Shutting down the Oracle ASMLib driver:                    [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver:                     [  OK  ]
Scanning the system for Oracle ASMLib disks:               [  OK  ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#


Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.

STEP 9 : Stop and Start cluster services on both the nodes.
[root@askmdbrac01 ~]#  /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac01'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac01'
CRS-2676: Start of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac01'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.askmdbrac02.vip' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac02.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan2.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac02'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac02'
CRS-2677: Stop of 'ora.crf' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#

Observation : Cluster ready services started without any issues. Now i need to verify the services status.

STEP 10 :  Verify the status.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N1.lsnr ora....er.type ONLINE    OFFLINE
ora....N2.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N3.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....DATA.dg ora....up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora....SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora....01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora....l01.gsd application    OFFLINE   OFFLINE
ora....l01.ons application    ONLINE    ONLINE    askmdbrac01
ora....l01.vip ora....t1.type ONLINE    ONLINE    askmdbrac01
ora....SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora....02.lsnr application    ONLINE    OFFLINE
ora....l02.gsd application    OFFLINE   OFFLINE
ora....l02.ons application    ONLINE    OFFLINE
ora....l02.vip ora....t1.type ONLINE    OFFLINE
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    OFFLINE
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora....se.type ONLINE    ONLINE    askmdbrac01
ora....ice.svc ora....ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac01 ~]# sudo su - grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.CRS.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora.FRA.dg     ora....up.type ONLINE    ONLINE    askmdbrac01
ora....ER.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N1.lsnr ora....er.type ONLINE    ONLINE    askmdbrac02
ora....N2.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....N3.lsnr ora....er.type ONLINE    ONLINE    askmdbrac01
ora....DATA.dg ora....up.type ONLINE    ONLINE    askmdbrac01
ora.asm        ora.asm.type   ONLINE    ONLINE    askmdbrac01
ora.cvu        ora.cvu.type   ONLINE    ONLINE    askmdbrac01
ora.gsd        ora.gsd.type   OFFLINE   OFFLINE
ora....network ora....rk.type ONLINE    ONLINE    askmdbrac01
ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    askmdbrac01
ora.ons        ora.ons.type   ONLINE    ONLINE    askmdbrac01
ora....SM1.asm application    ONLINE    ONLINE    askmdbrac01
ora....01.lsnr application    ONLINE    ONLINE    askmdbrac01
ora....l01.gsd application    OFFLINE   OFFLINE
ora....l01.ons application    ONLINE    ONLINE    askmdbrac01
ora....l01.vip ora....t1.type ONLINE    ONLINE    askmdbrac01
ora....SM2.asm application    ONLINE    ONLINE    askmdbrac02
ora....02.lsnr application    ONLINE    ONLINE    askmdbrac02
ora....l02.gsd application    OFFLINE   OFFLINE
ora....l02.ons application    ONLINE    ONLINE    askmdbrac02
ora....l02.vip ora....t1.type ONLINE    ONLINE    askmdbrac02
ora....ry.acfs ora....fs.type ONLINE    OFFLINE
ora.scan1.vip  ora....ip.type ONLINE    ONLINE    askmdbrac02
ora.scan2.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.scan3.vip  ora....ip.type ONLINE    ONLINE    askmdbrac01
ora.racdb.db ora....se.type ONLINE    ONLINE    askmdbrac01
ora....ice.svc ora....ce.type ONLINE    ONLINE    askmdbrac01
[root@askmdbrac02 ~]# sudo su - grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource                                        Target     State
-----------                                        ------     -----
ora.CRS.dg                                         ONLINE     ONLINE on askmdbrac01
ora.FRA.dg                                         ONLINE     ONLINE on askmdbrac01
ora.LISTENER.lsnr                                  ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr                            ONLINE     ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr                            ONLINE     ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg                                 ONLINE     ONLINE on askmdbrac01
ora.asm                                            ONLINE     ONLINE on askmdbrac01
ora.cvu                                            ONLINE     ONLINE on askmdbrac01
ora.gsd                                            OFFLINE    OFFLINE
ora.net1.network                                   ONLINE     ONLINE on askmdbrac01
ora.oc4j                                           ONLINE     ONLINE on askmdbrac01
ora.ons                                            ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm                           ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr          ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.gsd                                OFFLINE    OFFLINE
ora.askmdbrac01.ons                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac01.vip                                ONLINE     ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm                           ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr          ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.gsd                                OFFLINE    OFFLINE
ora.askmdbrac02.ons                                ONLINE     ONLINE on askmdbrac02
ora.askmdbrac02.vip                                ONLINE     ONLINE on askmdbrac02
ora.registry.acfs                                  ONLINE     OFFLINE
ora.scan1.vip                                      ONLINE     ONLINE on askmdbrac02
ora.scan2.vip                                      ONLINE     ONLINE on askmdbrac01
ora.scan3.vip                                      ONLINE     ONLINE on askmdbrac01
ora.racdb.db                                     ONLINE     ONLINE on askmdbrac01
ora.racdb.racdb_service.svc                    ONLINE     ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$

This concludes the article on troubleshooting RAC services startup issue.

Thanks
SRI
Labels: , ,
 
Support :
Copyright © 2013. askMLabs - All Rights Reserved
Proudly powered by Blogger