Recently we had a DR(Disaster Recovery) testing going on and we had RAC replica at the DR site for DR testing. The RAC servers are copied to DR setup, bit by bit. Then tried to start the RAC services at DR site and could not start them due to some issues.
Then I tried to troubleshoot the issue .....
This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.
STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.
STEP 2: Verify storage on both rac servers
Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.
STEP 3: Check high availability services and cluster ready services on both nodes with the following commands
Observation : The high availability services are up , but crs services are down.
STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.
Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.
STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.
Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can't find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.
Then i tried to see if i can see voting disks through command.
SETP 6 : Check the voting disks availability ( 11gR2 has voting disks in asm. I don't have asm instance up now. So i will get error if i try to query the voting disks on asm)
Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.
STEP 7 : Tried re-enabling the asmlibs as follows on both the nodes ...
Observation : Not able to start services even after restarting the asmlib.
STEP 8 : Tried to re-configure the asmlib as below .....
Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.
STEP 9 : Stop and Start cluster services on both the nodes.
Observation : Cluster ready services started without any issues. Now i need to verify the services status.
STEP 10 : Verify the status.
This concludes the article on troubleshooting RAC services startup issue.
Thanks
SRI
Then I tried to troubleshoot the issue .....
This article gives you the troubleshooting steps for this particular issue. These steps may also help you to troubleshoot if you have issues starting RAC services.
STEP 1: Reboot the rac servers and Check services after re-booting the servers. The services are still down.
STEP 2: Verify storage on both rac servers
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su -
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf 8:80 [active][ready]
\_ 3:0:1:2 sdl 8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc 8:32 [active][ready]
\_ 3:0:0:2 sdi 8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde 8:64 [active][ready]
\_ 3:0:1:1 sdk 8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ 3:0:1:0 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg 8:96 [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su -
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf 8:80 [active][ready]
\_ 3:0:0:2 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc 8:32 [active][ready]
\_ 3:0:1:2 sdl 8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk 8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde 8:64 [active][ready]
\_ 3:0:0:1 sdh 8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ 3:0:0:0 sdg 8:96 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj 8:144 [active][ready]
[root@askmdbrac02 ~]#
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac01 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac01 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac01 ~]$ exit
logout
[oracle@askmdbrac01 ~]$ sudo su -
[root@askmdbrac01 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf 8:80 [active][ready]
\_ 3:0:1:2 sdl 8:176 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc 8:32 [active][ready]
\_ 3:0:0:2 sdi 8:128 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:0:1 sdh 8:112 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde 8:64 [active][ready]
\_ 3:0:1:1 sdk 8:160 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ 3:0:1:0 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:0:0 sdg 8:96 [active][ready]
[root@askmdbrac01 ~]# exit[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[grid@askmdbrac02 ~]$ /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[grid@askmdbrac02 ~]$ multipath -ll
-bash: multipath: command not found
[grid@askmdbrac02 ~]$ exit
logout
[oracle@askmdbrac02 ~]$ sudo su -
[root@askmdbrac02 ~]# multipath -ll
vpath2 (360050768019280fb280000000000055e) dm-10 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:2 sdf 8:80 [active][ready]
\_ 3:0:0:2 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:0:2 sdc 8:32 [active][ready]
\_ 3:0:1:2 sdl 8:176 [active][ready]
vpath1 (360050768019280fb280000000000055d) dm-12 IBM,2145
[size=200G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 3:0:1:1 sdk 8:160 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 2:0:1:1 sde 8:64 [active][ready]
\_ 3:0:0:1 sdh 8:112 [active][ready]
vpath0 (360050768019280fb280000000000055c) dm-11 IBM,2145
[size=10G][features=1 queue_if_no_path][hwhandler=0][rw]
\_ round-robin 0 [prio=50][active]
\_ 2:0:1:0 sdd 8:48 [active][ready]
\_ 3:0:0:0 sdg 8:96 [active][ready]
\_ round-robin 0 [prio=10][enabled]
\_ 3:0:1:0 sdj 8:144 [active][ready]
[root@askmdbrac02 ~]#
Observation : Storage is fine and i am able to see ASM disks from both the nodes. Storage is configured with multipathing. I also verified multipathing and status shows fine.
STEP 3: Check high availability services and cluster ready services on both nodes with the following commands
[grid@askmdbrac01 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac01 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$
[grid@askmdbrac01 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac01 ~]$[grid@askmdbrac02 ~]$ crsctl check has
CRS-4638: Oracle High Availability Services is online
[grid@askmdbrac02 ~]$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
[grid@askmdbrac02 ~]$ crsctl check cluster
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
[grid@askmdbrac02 ~]$
Observation : The high availability services are up , but crs services are down.
STEP 4: Do ping and nslookup for other rac nodes from each node to make sure that each node in cluster is accessible from every other node. Check this for private ips, virtual ips and for scan ips.
Observation : The ping and nslookup are working fine and all the nodes are reachable from every other node.
STEP 5 : Then verify the log files from grid infrastructure and see if you can find any error messages from log files. The cluster log files are located at $ORACLE_HOME/log/askmdbrac01 and $ORACLE_HOME/log/askmdbrac02.
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
.....
.....
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'askmdbrac01'.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource 'ora.cssd'; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
2013-04-02 19:38:27.425
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
2013-04-02 19:38:39.988
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:38:55.007
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:39:10.025
.....
.....
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.836
[/o001/home/11.2.0.2/grid/bin/cssdagent(11163)]CRS-5818:Aborted command 'start for resource: ora.cssd 1 1' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:27:4} in /o001/home/11.2.0.2/grid/log/askmdbrac01/agent/ohasd/oracssdagent_root/oracssdagent_root.log.
2013-04-02 19:48:23.837
[cssd(11175)]CRS-1656:The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:23.838
[cssd(11175)]CRS-1603:CSSD on node askmdbrac01 shutdown by user.
2013-04-02 19:48:25.730
[cssd(11175)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /o001/home/11.2.0.2/grid/log/askmdbrac01/cssd/ocssd.log
2013-04-02 19:48:29.863
[ohasd(6876)]CRS-2765:Resource 'ora.cssdmonitor' has failed on server 'askmdbrac01'.
2013-04-02 19:48:30.126
[ohasd(6876)]CRS-2771:Maximum restart attempts reached for resource 'ora.cssd'; will not restart.
2013-04-02 19:48:35.833
[ohasd(6876)]CRS-2765:Resource 'ora.diskmon' has failed on server 'askmdbrac01'.
Observation : Found from the cluster alert log file (alertaskmdbrac01.log) that the cluster services can't find voting disk. So it could be due the in-accessible asm storage or due to currupted voting disks.
Then i tried to see if i can see voting disks through command.
SETP 6 : Check the voting disks availability ( 11gR2 has voting disks in asm. I don't have asm instance up now. So i will get error if i try to query the voting disks on asm)
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$
Unable to communicate with the Cluster Synchronization Services daemon.
[root@askmdbrac01 ~]#[grid@askmdbrac02 ~]$ /o001/home/11.2.0.2/grid/bin/crsctl query css votedisk
Unable to communicate with the Cluster Synchronization Services daemon.
[grid@askmdbrac02 ~]$
Observation : I am not able to query the voting disks. Now i have only one option to look at the asm storage accessibility.
STEP 7 : Tried re-enabling the asmlibs as follows on both the nodes ...
[root@askmdbrac01 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: no
Checking if /dev/oracleasm is mounted: no
[root@askmdbrac01 ~]# /etc/init.d/oracleasm enable
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm restart
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [ OK ]
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm listdisks
CRSVOL1
DISK1
FRADISK1
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#
Observation : Not able to start services even after restarting the asmlib.
STEP 8 : Tried to re-configure the asmlib as below .....
[root@askmdbrac01 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#
Configuring the Oracle ASM library driver.This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]#
[root@askmdbrac01 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac01 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac01 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface [grid]:
Default group to own the driver interface [asmadmin]:
Start Oracle ASM library driver on boot (y/n) [y]:
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]#
[root@askmdbrac02 ~]# /etc/init.d/oracleasm stop
Dropping Oracle ASMLib disks: [ OK ]
Shutting down the Oracle ASMLib driver: [FAILED]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm start
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
[root@askmdbrac02 ~]# /etc/init.d/oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
[root@askmdbrac02 ~]#
Observation : Identified that the asm library drivers configuration is wrong and i re-configured with correct user and group. See the difference in the lines when it is asking for prompt on second node. I executed this command twice to show you the difference when it is properly configured and when it is not properly configured.
STEP 9 : Stop and Start cluster services on both the nodes.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac01'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac01'
CRS-2676: Start of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac01'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.askmdbrac02.vip' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac02.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan2.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac02'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac02'
CRS-2677: Stop of 'ora.crf' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac01'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac01' succeeded
CRS-2672: Attempting to start 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2676: Start of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac01'
CRS-2676: Start of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac01'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac01'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac01'
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.crf' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac01'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac01 ~]#[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl stop crs
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crsd' on 'askmdbrac02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.askmdbrac02.vip' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.oc4j' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.cvu' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.CRS.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.racdb.racdb_service.svc' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac02.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.racdb_service.svc' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.racdb.db' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.askmdbrac01.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.askmdbrac01.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.scan2.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.scan3.vip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.racdb.db' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.FRA.dg' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.PRDSHR_DATA.dg' on 'askmdbrac02'
CRS-2677: Stop of 'ora.FRA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.PRDSHR_DATA.dg' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.cvu' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.CRS.dg' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'askmdbrac02'
CRS-2677: Stop of 'ora.ons' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'askmdbrac02'
CRS-2677: Stop of 'ora.net1.network' on 'askmdbrac02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'askmdbrac02' has completed
CRS-2677: Stop of 'ora.crsd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.evmd' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.asm' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.evmd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.asm' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.cssd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'askmdbrac02'
CRS-2673: Attempting to stop 'ora.crf' on 'askmdbrac02'
CRS-2677: Stop of 'ora.crf' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.diskmon' on 'askmdbrac02' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'askmdbrac02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'askmdbrac02'
CRS-2677: Stop of 'ora.gpnpd' on 'askmdbrac02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'askmdbrac02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@askmdbrac02 ~]#
Observation : Cluster ready services started without any issues. Now i need to verify the services status.
STEP 10 : Verify the status.
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.FRA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N1.lsnr ora....er.type ONLINE OFFLINE
ora....N2.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N3.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....DATA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.asm ora.asm.type ONLINE ONLINE askmdbrac01
ora.cvu ora.cvu.type ONLINE ONLINE askmdbrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE askmdbrac01
ora.oc4j ora.oc4j.type ONLINE ONLINE askmdbrac01
ora.ons ora.ons.type ONLINE ONLINE askmdbrac01
ora....SM1.asm application ONLINE ONLINE askmdbrac01
ora....01.lsnr application ONLINE ONLINE askmdbrac01
ora....l01.gsd application OFFLINE OFFLINE
ora....l01.ons application ONLINE ONLINE askmdbrac01
ora....l01.vip ora....t1.type ONLINE ONLINE askmdbrac01
ora....SM2.asm application ONLINE ONLINE askmdbrac02
ora....02.lsnr application ONLINE OFFLINE
ora....l02.gsd application OFFLINE OFFLINE
ora....l02.ons application ONLINE OFFLINE
ora....l02.vip ora....t1.type ONLINE OFFLINE
ora....ry.acfs ora....fs.type ONLINE OFFLINE
ora.scan1.vip ora....ip.type ONLINE OFFLINE
ora.scan2.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.scan3.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.racdb.db ora....se.type ONLINE ONLINE askmdbrac01
ora....ice.svc ora....ce.type ONLINE ONLINE askmdbrac01
[root@askmdbrac01 ~]# sudo su - grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.FRA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N1.lsnr ora....er.type ONLINE ONLINE askmdbrac02
ora....N2.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N3.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....DATA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.asm ora.asm.type ONLINE ONLINE askmdbrac01
ora.cvu ora.cvu.type ONLINE ONLINE askmdbrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE askmdbrac01
ora.oc4j ora.oc4j.type ONLINE ONLINE askmdbrac01
ora.ons ora.ons.type ONLINE ONLINE askmdbrac01
ora....SM1.asm application ONLINE ONLINE askmdbrac01
ora....01.lsnr application ONLINE ONLINE askmdbrac01
ora....l01.gsd application OFFLINE OFFLINE
ora....l01.ons application ONLINE ONLINE askmdbrac01
ora....l01.vip ora....t1.type ONLINE ONLINE askmdbrac01
ora....SM2.asm application ONLINE ONLINE askmdbrac02
ora....02.lsnr application ONLINE ONLINE askmdbrac02
ora....l02.gsd application OFFLINE OFFLINE
ora....l02.ons application ONLINE ONLINE askmdbrac02
ora....l02.vip ora....t1.type ONLINE ONLINE askmdbrac02
ora....ry.acfs ora....fs.type ONLINE OFFLINE
ora.scan1.vip ora....ip.type ONLINE ONLINE askmdbrac02
ora.scan2.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.scan3.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.racdb.db ora....se.type ONLINE ONLINE askmdbrac01
ora....ice.svc ora....ce.type ONLINE ONLINE askmdbrac01
[root@askmdbrac02 ~]# sudo su - grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac01 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.FRA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N1.lsnr ora....er.type ONLINE OFFLINE
ora....N2.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N3.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....DATA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.asm ora.asm.type ONLINE ONLINE askmdbrac01
ora.cvu ora.cvu.type ONLINE ONLINE askmdbrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE askmdbrac01
ora.oc4j ora.oc4j.type ONLINE ONLINE askmdbrac01
ora.ons ora.ons.type ONLINE ONLINE askmdbrac01
ora....SM1.asm application ONLINE ONLINE askmdbrac01
ora....01.lsnr application ONLINE ONLINE askmdbrac01
ora....l01.gsd application OFFLINE OFFLINE
ora....l01.ons application ONLINE ONLINE askmdbrac01
ora....l01.vip ora....t1.type ONLINE ONLINE askmdbrac01
ora....SM2.asm application ONLINE ONLINE askmdbrac02
ora....02.lsnr application ONLINE OFFLINE
ora....l02.gsd application OFFLINE OFFLINE
ora....l02.ons application ONLINE OFFLINE
ora....l02.vip ora....t1.type ONLINE OFFLINE
ora....ry.acfs ora....fs.type ONLINE OFFLINE
ora.scan1.vip ora....ip.type ONLINE OFFLINE
ora.scan2.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.scan3.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.racdb.db ora....se.type ONLINE ONLINE askmdbrac01
ora....ice.svc ora....ce.type ONLINE ONLINE askmdbrac01
[root@askmdbrac01 ~]# sudo su - grid
[grid@askmdbrac01 ~]$ . ./profile.grid
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac01 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac01 ~]$[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@askmdbrac02 ~]# /o001/home/11.2.0.2/grid/bin/crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.CRS.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.FRA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora....ER.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N1.lsnr ora....er.type ONLINE ONLINE askmdbrac02
ora....N2.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....N3.lsnr ora....er.type ONLINE ONLINE askmdbrac01
ora....DATA.dg ora....up.type ONLINE ONLINE askmdbrac01
ora.asm ora.asm.type ONLINE ONLINE askmdbrac01
ora.cvu ora.cvu.type ONLINE ONLINE askmdbrac01
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE askmdbrac01
ora.oc4j ora.oc4j.type ONLINE ONLINE askmdbrac01
ora.ons ora.ons.type ONLINE ONLINE askmdbrac01
ora....SM1.asm application ONLINE ONLINE askmdbrac01
ora....01.lsnr application ONLINE ONLINE askmdbrac01
ora....l01.gsd application OFFLINE OFFLINE
ora....l01.ons application ONLINE ONLINE askmdbrac01
ora....l01.vip ora....t1.type ONLINE ONLINE askmdbrac01
ora....SM2.asm application ONLINE ONLINE askmdbrac02
ora....02.lsnr application ONLINE ONLINE askmdbrac02
ora....l02.gsd application OFFLINE OFFLINE
ora....l02.ons application ONLINE ONLINE askmdbrac02
ora....l02.vip ora....t1.type ONLINE ONLINE askmdbrac02
ora....ry.acfs ora....fs.type ONLINE OFFLINE
ora.scan1.vip ora....ip.type ONLINE ONLINE askmdbrac02
ora.scan2.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.scan3.vip ora....ip.type ONLINE ONLINE askmdbrac01
ora.racdb.db ora....se.type ONLINE ONLINE askmdbrac01
ora....ice.svc ora....ce.type ONLINE ONLINE askmdbrac01
[root@askmdbrac02 ~]# sudo su - grid
[grid@askmdbrac02 ~]$ . ./profile.grid
[grid@askmdbrac02 ~]$ Check_CRS.txt
HA Resource Target State
----------- ------ -----
ora.CRS.dg ONLINE ONLINE on askmdbrac01
ora.FRA.dg ONLINE ONLINE on askmdbrac01
ora.LISTENER.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN1.lsnr ONLINE ONLINE on askmdbrac02
ora.LISTENER_SCAN2.lsnr ONLINE ONLINE on askmdbrac01
ora.LISTENER_SCAN3.lsnr ONLINE ONLINE on askmdbrac01
ora.PRDSHR_DATA.dg ONLINE ONLINE on askmdbrac01
ora.asm ONLINE ONLINE on askmdbrac01
ora.cvu ONLINE ONLINE on askmdbrac01
ora.gsd OFFLINE OFFLINE
ora.net1.network ONLINE ONLINE on askmdbrac01
ora.oc4j ONLINE ONLINE on askmdbrac01
ora.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.ASM1.asm ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.LISTENER_askmdbrac01.lsnr ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.gsd OFFLINE OFFLINE
ora.askmdbrac01.ons ONLINE ONLINE on askmdbrac01
ora.askmdbrac01.vip ONLINE ONLINE on askmdbrac01
ora.askmdbrac02.ASM2.asm ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.LISTENER_askmdbrac02.lsnr ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.gsd OFFLINE OFFLINE
ora.askmdbrac02.ons ONLINE ONLINE on askmdbrac02
ora.askmdbrac02.vip ONLINE ONLINE on askmdbrac02
ora.registry.acfs ONLINE OFFLINE
ora.scan1.vip ONLINE ONLINE on askmdbrac02
ora.scan2.vip ONLINE ONLINE on askmdbrac01
ora.scan3.vip ONLINE ONLINE on askmdbrac01
ora.racdb.db ONLINE ONLINE on askmdbrac01
ora.racdb.racdb_service.svc ONLINE ONLINE on askmdbrac02
[grid@askmdbrac02 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node askmdbrac01
Instance racdb2 is running on node askmdbrac02
[grid@askmdbrac02 ~]$
This concludes the article on troubleshooting RAC services startup issue.
Thanks
SRI
Posts
+ comments + 3 comments
hi
thanks for sharing good doc.
thanks
nice article , it helped me a lot to resolve simillar kind of issues , could you please help me how to make doc like you prepared like take entire thing from putty to doc , and format is good.
please help me to make this.
Hi Prakash, i use div and pre html tags to format. It is very simple. Give it a try :-)
Post a Comment
Thank you for visiting our site and leaving your valuable comment.