askMLabs

In this post ,we are going to discuss about a recent issue we had after completing the exadata grid upgrade from 12.1.0.2 to 12.2.0.1. After completing the grid upgrade, we started seeing some databases status in UNKNOWN status with crsctl status command. The srvctl command show the status of the Database as not running though they are actually running. So let’s see the step by step troubleshooting of this issue.

NOTE : When you are troubleshooting any issue it is very important to first understand the issue and all its symptoms. We can’t just search for error string in google or in Oracle Support portal and start applying the solutions. This is not the right approach. We always suggest my blog followers and my students to first understand the issue and all its error messages. Let’s start discussing our issue….

After completing the GRID upgrade, the following symptoms are identified ….

• srvctl shows the database status as “not running” though the database is actually up and running
• Can’t manage the database(start or stop ) using the srvctl utility
• SQLPLUS is the only utility to start and stop any instance of the RAC database
• CRSCTL shows the database resource status as UNKNOWN.

With above mentioned symptoms of the issue, any experienced RAC should be able to tell you that the database is not properly talking to clusterware to update its status. That being said, the clusterware is unable to manage the database. These kind of situations arise when any cluster rac nodes gets rebooted abruptly. In our case, It may or may not be the grid upgrade that caused this issue, but there could be some previous occurances where the rac node got rebooted and there was a mismatch of Oracle database status in Oracle Clusterware. So how do we let the clusterware tell to reset the database status and start managing the database? Simple , just disable the resource and re-enable the resource using srvctl utility. But many DBAs are not convinced with the solution provided by the experience, they need a step by step troubleshooting steps …

Check 1:

Validate all the symptoms of the issue specified above. This confirms the status mismatch with Oracle Clusterware.

I am considering one database to show you the commands, but this issue is with many database on Exadata.

The following output shows thath the database is not running on any of the clusterware exadata compute node.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status database -d ASKMDB
Instance ASKMDB1 is not running on node exaaskmdb01
Instance ASKMDB2 is not running on node exaaskmdb02
Instance ASKMDB3 is not running on node exaaskmdb03
Instance ASKMDB4 is not running on node exaaskmdb04
Instance ASKMDB5 is not running on node exaaskmdb05
Instance ASKMDB6 is not running on node exaaskmdb06
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$

But the above output is not correct. The database instance is actually running on node2 exaaskmdb02.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ sqlplus '/as sysdba'

SQL*Plus: Release 11.2.0.4.0 Production on Fri Sep 14 00:36:16 2018

Copyright (c) 1982, 2013, Oracle.  All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> sho parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string      +ASKM_DG/ASKMDB/spfileASKMDB.ora

If I try to start this instance using srvctl , it gives me the following error messages …..

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl start instance -d ASKMDB -i ASKMDB2
PRCR-1013 : Failed to start resource ora.askmdb.db
PRCR-1064 : Failed to start resource ora.askmdb.db on node exaaskmdb02
CRS-2680: Clean of 'ora.askmdb.db' on 'exaaskmdb02' failed
CRS-5802: Unable to start the agent process

But the same instance can be started and stopped using sqlplus.

The CRSCTL status of this database resource shows the following UNKNOWN status ….

[grid@exaaskmdb01 ~]$ crsctl status res ora.askmdb.db -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.askmdb.db
       1        ONLINE  ONLINE       exaaskmdb02                 Open,HOME=/u01/app/o
                                                              racle/product/11.2.0
                                                              .4/dbhome_1,STABLE
       2        OFFLINE OFFLINE                               STABLE
       3        OFFLINE OFFLINE                               STABLE
       4        ONLINE  UNKNOWN      exaaskmdb02                 STABLE
       5        ONLINE  OFFLINE                               STABLE
       6        ONLINE  OFFLINE                               STABLE
--------------------------------------------------------------------------------
[grid@exaaskmdb01 ~]$

Check 2 :

With Check1, we confirmed that all the symptoms we specified are met. Now lets see what is being populated in logs and what logs files we need to review …….

All cluster related issues are logged to clusteware logs. These logs are located in /u01/app/grid/diag/crs/exaaskmdb02/crs/trace.  Check the logs alert.log , crsd.trc and crsd_oraagent_oracle.trc files to check the error messages that states the cluster resource status changed to UNKNOWN status with the reason for this status change. As a DBA we should be able to understand the log formatting messages of the clusterware trace files. 

Solution

The following steps are listed to let any DBA understand the sequence of steps to execute to correct the database UNKNOWN status.

1. List the database status and check all the instances status

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status database -d ASKMDB
Instance ASKMDB1 is not running on node exaaskmdb01
Instance ASKMDB2 is not running on node exaaskmdb02
Instance ASKMDB3 is not running on node exaaskmdb03
Instance ASKMDB4 is not running on node exaaskmdb04
Instance ASKMDB5 is not running on node exaaskmdb05
Instance ASKMDB6 is not running on node exaaskmdb06

2. List the services status and check where the services are running.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status service -d ASKMDB
Service ASKMDBSC is running on instance(s) ASKMDB2

3. If the database is running a single instance,
3.a. Enable new database instance on a different node using srvctl

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl enable instance -d ASKMDB -i ASKMDB3

3.b. Start this new instance  using srvctl

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl start instance -d ASKMDB -i ASKMDB3

3.c. Modify the database services configuration to make the new instance as the preferred instance

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl modify service -d ASKMDB -s ASKMDBSC -n -i ASKMDB3 -a ASKMDB1,ASKMDB2,ASKMDB4,ASKMDB5,ASKMDB6

3.d. Verify the database configuration that the new instance is listed as the preferred instance.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl config service -d ASKMDB -s ASKMDBSC
Service name: ASKMDBSC
Service is enabled
Server pool: ASKMDB_ASKMDBSC
Cardinality: 1
Disconnect: false
Service role: PRIMARY
Management policy: AUTOMATIC
DTP transaction: false
AQ HA notifications: false
Failover type: NONE
Failover method: NONE
TAF failover retries: 0
TAF failover delay: 0
Connection Load Balancing Goal: LONG
Runtime Load Balancing Goal: NONE
TAF policy specification: NONE
Edition:
Preferred instances: ASKMDB3
Available instances: ASKMDB1,ASKMDB2,ASKMDB4,ASKMDB5,ASKMDB6
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$

4. If the database is running multiple instances,
4.a. List the database configuration and check if the services have multiple instances as preferred instances.
4.b. Modify if we need to adjust the service configuration, to have atleast one running instance(Other than problematic instance) as preferred instance.
5. Relocate the database services to the new instance.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status service -d ASKMDB -s ASKMDBSC
Service ASKMDBSC is running on instance(s) ASKMDB3

6. Stop the problematic instance using srvctl.           <<< If there are any errors during this step, it's expected. Don't worry about the error messages, but proceed to the next step.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl stop instance -d ASKMDB -i ASKMDB2
PRCR-1133 : Failed to stop database ASKMDB and its running services
PRCR-1132 : Failed to stop resources using a filter
CRS-2680: Clean of 'ora.askmdb.db' on 'exaaskmdb02' failed
CRS-5807: Agent failed to process the message
CRS-2680: Clean of 'ora.askmdb.db' on 'exaaskmdb02' failed
CRS-5807: Agent failed to process the message
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$

7. Disable the instance using srvctl

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl disable instance -d ASKMDB -i ASKMDB2

8. Enable the instance using srvctl

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl enable instance -d ASKMDB -i ASKMDB2

9. Start the instance using srvctl

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status database -d ASKMDB
Instance ASKMDB1 is not running on node exaaskmdb01
Instance ASKMDB2 is not running on node exaaskmdb02
Instance ASKMDB3 is running on node exaaskmdb03
Instance ASKMDB4 is not running on node exaaskmdb04
Instance ASKMDB5 is not running on node exaaskmdb05
Instance ASKMDB6 is not running on node exaaskmdb06
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ ps -ef | grep -i pmon | grep -i ASKMDB
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl start instance -d ASKMDB -i ASKMDB2
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status database -d ASKMDB
Instance ASKMDB1 is not running on node exaaskmdb01
Instance ASKMDB2 is running on node exaaskmdb02
Instance ASKMDB3 is running on node exaaskmdb03
Instance ASKMDB4 is not running on node exaaskmdb04
Instance ASKMDB5 is not running on node exaaskmdb05
Instance ASKMDB6 is not running on node exaaskmdb06
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$

10. Now check the status of the database resource using crsctl

[grid@exaaskmdb02 ~]$ crsctl status res ora.askmdb.db -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.askmdb.db
       1        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                              ABLE
       2        OFFLINE OFFLINE                               Instance Shutdown,ST
                                                              ABLE
       3        OFFLINE OFFLINE                               STABLE
       4        ONLINE  ONLINE       exaaskmdb02                 Open,HOME=/u01/app/o
                                                              racle/product/11.2.0
                                                              .4/dbhome_1,STABLE
       5        ONLINE  OFFLINE                               STABLE
       6        ONLINE  OFFLINE                               STABLE
--------------------------------------------------------------------------------
[grid@exaaskmdb01 ~]$

11. Once it is confirmed that the database shows the correct status using crsctl, proceed to bring down the temporary instance that we started.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl stop instance -d ASKMDB -i ASKMDB3

12. Relocate the services back to the original instance once the problem is resolved with that instance.
13. Stop the temporary instance that we started.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status database -d ASKMDB
Instance ASKMDB1 is not running on node exaaskmdb01
Instance ASKMDB2 is running on node exaaskmdb02
Instance ASKMDB3 is not running on node exaaskmdb03
Instance ASKMDB4 is not running on node exaaskmdb04
Instance ASKMDB5 is not running on node exaaskmdb05
Instance ASKMDB6 is not running on node exaaskmdb06
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl status service -d ASKMDB
Service ASKMDBSC is running on instance(s) ASKMDB2
[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$

14. Disable the temporary instance that we started.

[oracle@exaaskmdb02.askmlabs.com:ASKMDB2]$ srvctl disable instance -d ASKMDB -i ASKMDB3

Hope this helps …

In this post, we will discuss  troubleshooting OEM LDAP authentication issue. Due to various reasons, you may see sometimes the OEM users (LDAP users) getting Authentication failures. Well there could be various reasons for the authentication failures. We need to see how can we enable the tracing/debug to populate the detailed error messages in logs. The procedure is also explained in the Oracle Document.

Our troubleshooting started after users compaining about the authentication issues. We narrowed it with various testings/accessing other enterprise applications that the issue is only with the OEM. Now we have started our focus on troubleshooting authentication issue.

Check1 :

Check if the issue is with all the users ( Local and External ). If the issue is only with the External users(LDAP users), the primary focus of this post, Please proceed with the troubleshooting.

Knowing that the issue is only with the LDAP users, We focussed looking around the external ldap system. In our case, we are using Microsoft Active Directory to authenticate users to OEM.

Check2:

Check if the service account used to integrate for authentication is locked. The security/identity team in your Organization should be able to help you if the accout is locked or not. If the account is locked, Please have this account unlocked it. Verify if the issue is resolved.  But if the issue is not with the account locked, proceed to the next step to continue troubleshooting ….

Check3:

If the issue is not with the account locked, try enabling the Weblogic Debug/Tracing as described below ……

From the WebLogic Server Console, enable logging, set the WebLogic debug log level and then the enable the debug flag.

1. Navigate to Environment->Servers->EMGC_OMS1 and choose the Logging tab.
2. Under Rotation, set the file rotation size to 64000.
3. Expand Advanced.
4. Under Message destinations(s) (Log file: region) set the Severity level to Trace.
5. Navigate to the Environment->Servers->EMGC_OMS1 and choose the Debug tab.
6. Check WebLogic->security->atn->DebugSecurityAtn.
7. Click Save.

Now these settings will create a logfile ldap_trace.logATN. This file ldap_trace.logATN is located at $OMS_HOME/../../gc_inst/user_projects/domains/GCDomain

Following log messages are populated when we try to access the OEM using LDAP user accounts….

[oracle@askmoem12c GCDomain]$ tail -f ldap_trace.logATN
17:31:47.729 ldc=6 Connected to ldaps://adauth.askmlabs.com:389
17:31:47.730 ldc=6 op=227 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:31:47.732 ldc=6 op=227 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 532, v2580}
17:32:30.334 ldc=7 Connected to ldaps://adauth.askmlabs.com:389
17:32:30.334 ldc=7 op=228 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:32:30.336 ldc=7 op=228 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 532, v2580}
17:35:00.485 ldc=8 Connected to ldaps://adauth.askmlabs.com:389
17:35:00.485 ldc=8 op=229 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:35:00.487 ldc=8 op=229 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 532, v2580}
17:37:06.171 ldc=9 Connected to ldaps://adauth.askmlabs.com:389
17:37:06.171 ldc=9 op=230 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:37:06.174 ldc=9 op=230 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 532, v2580}
17:39:45.033 ldc=10 Connected to ldaps://adauth.askmlabs.com:389
17:39:45.033 ldc=10 op=231 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:39:45.035 ldc=10 op=231 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 52e, v2580}
17:47:17.121 ldc=2 Connected to ldaps://adauth.askmlabs.com:389
17:47:17.122 ldc=2 op=223 BindRequest {version=3, name=CN=oem.service,OU=Users,OU=Accounts,DC=askmlabs,DC=com, authentication=********}
17:47:17.124 ldc=2 op=223 BindResponse {resultCode=49, errorMessage=80090308: LdapErr: DSID-0C09042F, comment: AcceptSecurityContext error, data 532, v2580}
^C

From the above error messages, the “resultCode=49” indicates that the bad password is used. Now we understand that the issue is to deal with the service account password. We changed the service account oem.service password and then updated the password in weblogic console as shown below.

Follow the navigation to update the password ….

Domain - > Environment - > Security Realms - > myrealm - > providers

Save the password. And then activate the changes.

Ideally we don’t  need to bounce the services to take these changes effect. But if you could offer the bounce, please bounce the complete OEM services.

Hope this information helps….

This post talks about an Exadata RAC ASM issue. This issue is observed after completing Exadata patching.  It is observed that the ACFS mount point is missing. The following steps explains how we identified the issue and How can we troubleshoot this issue.

This is a 8 node Exadata full rack. The ACFS mount point is missing on first node , but all other 7 nodes have this ASM mount point.

First try to get the missing mount point …

Use “df –h” on any node where you can see the ACFS mount point. In our case getting the ACFS mount point details from node2 exaaskmdb02

[root@exaaskmdb02 ~]# df -h /u01/app/askmacfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/askmdata-282  512G   64G  449G  13% /u01/app/askmacfs
[root@exaaskmdb01 ~]#

Check1 :

Using these details, try to manually mount the ACFS on node1 exaaskmdb01

[root@exaaskmdb01 ~]# mount -t acfs /dev/asm/askmdata-282  /u01/app/askmacfs
mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 2
mount.acfs: CLSU-00101: operating system error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/askmdata-282)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/askmdata-282. Verify the volume exists.
[root@exaaskmdb01 ~]#

We are not able to mount the ACFS mountpoint. The first approach for any troubleshooting is to analyze the error messages. Cetain times, there could be useful error messages, which can directly help to point to the issue. But sometimes, the error messages may not help to focus on the real issue , rather informational messages. Your experience will count a lot in these scenarios where you should be able to relate the error message to the issue.

Check2:

The error message which caught to our attention is “No such file or directory”. Now our next check will be to check the directories…

On exaaskmdb01 check the directories  /dev/asm/askmdata-282 and /u01/app/askmacfs

[root@exaaskmdb01 ]# ls -lrt /dev/asm/askmdata-282
ls: cannot access /dev/asm/askmdata-282: No such file or directory

[root@exaaskmdb01 ]# ls -ld /u01/app/askmacfs
drwxrwxr-x 11 grid oinstall 4096 May  9 12:00 /u01/app/askmacfs

Same thing, verify on any other node ( 2-8)

[root@exaaskmdb02 ]# ls -lrt /dev/asm/askmdata-282
brwxrwx--- 1 root asmadmin 248, 144385 Sep 12 15:33 /dev/asm/askmdata-282

[root@exaaskmdb02 ]# ls -ld /u01/app/askmacfs
drwxrwxr-x 11 grid oinstall 4096 May  9 12:00 /u01/app/askmacfs

From the above check, it is clear that node1 exaaskmdb01 is missing a directory. Now lets check further how to bring this directory. As these are ACFS mounts, there could be possibility that these are missing due to ACFS drivers not loading properly during node reboot. Let’s concentrate on the ACFS drivers….

Check3:

Check acfs modules to be loaded …

[root@exaaskmdb01 ~]# lsmod | grep oracle
oracleacfs           4825088  0
oracleadvm            806912  3
oracleoks             659456  2 oracleacfs,oracleadvm

This check looks good, the acfs drivers are loaded properly. The next check would be to check the ACFS volume informaiton.

Check4:

ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: DISABLED                               <<<< Check this. This volume is disabled.
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

From the above output, we get that the volume is disabled. Now let’s check the same thing on the other nodes exaaskmdb02-08

[grid@exaaskmdb02 ~]$ asmcmd
ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: ENABLED                     <<< It is enabled on node2
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

From the above check4, we understood that this is due to the reason that the volume is not enabled by default after patching. Now we need to enable this volume and mount this ACFS manually.

Solution

ASMCMD> volenable -G ASKM_DG -a
ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: ENABLED
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

ASMCMD> exit
[grid@exaaskmdb01 ~]$ ls -lrt /dev/asm/askmdata-282
brwxrwx--- 1 root asmadmin 248, 144385 Sep 12 15:56 /dev/asm/askmdata-282

[root@exaaskmdb01 ~]# ls -ld /u01/app/askmacfs
drwxrwxr-x 2 grid oinstall 4096 Mar 18  2016 /u01/app/askmacfs

[root@exaaskmdb01 ~]# mount -t acfs /dev/asm/askmdata-282 /u01/app/askmacfs

[root@exaaskmdb01 ~]# df -h /u01/app/askmacfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/askmdata-282  512G   64G  449G  13% /u01/app/askmacfs
[root@exaaskmdb01 ~]#

That’s it. We fixed the issue.

Questions to the readers

Q1) Where do you find the ASM related modules/drivers on Linux OS ?

Q2) What to do if you find any missing ASM modules/drivers on Linux OS ?

Q3) How to load the asm drivers if they were not loaded by default during node reboot ?

Refer to my other post <Place Holder> which is similar to this issue , but happened during our exa grid upgrade from 12.1.0.2 to 12.2.0.1.

Hope this information helps….