Unable to acquire a central Inventory lock by OPatch due to multiple opatch run

Unable to acquire a central Inventory lock by OPatch due to multiple opatch run

OPatch will sleep for few seconds, before re-trying to get the lock...

High Level steps:

1. Rollback a patch from Oracle Home using opatch utility/tool

2. Meantime when you are rolling back a patch from Oracle Home, Parallelly try to rollback a patch from GI(Grid Home) which will be waiting on acquiring lock.

3. Verify the OPatch process running from Oracle Home and another from GI Home

4. As soon as Oracle Home patch rollback completes then GI home opatch rollback proceed further which was waiting on acquiring lock.

1. Rollback a patch from Oracle Home using opatch utility/tool

[root@oraclelab2 ~]# su - oracle

Last login: Thu May  2 00:06:12 IST 2024 on pts/0

[oracle@oraclelab2 ~]$ . oraenv

ORACLE_SID = [oracle] ? TESTDB

The Oracle base has been set to /u01/app/oracle

[oracle@oraclelab2 ~]$

[oracle@oraclelab2 ~]$ /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068

Oracle Interim Patch Installer version 12.2.0.1.41

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/19.0.0.0/dbhome_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc

OPatch version    : 12.2.0.1.41

OUI version       : 12.2.0.7.0

Log file location : /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log

Patches will be rolled back in the following order:

   35042068

The following patch(es) will be rolled back: 35042068

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/oracle/product/19.0.0.0/dbhome_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Rolling back patch 35042068...

RollbackSession rolling back interim patch '35042068' from OH '/u01/app/oracle/product/19.0.0.0/dbhome_1'

Patching component oracle.rsf, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...

Patching component oracle.slax.rsf, 19.0.0.0.0...

Patching component oracle.ordim.jai, 19.0.0.0.0...

Patching component oracle.bali.jewt, 11.1.1.6.0...

Patching component oracle.bali.ewt, 11.1.1.6.0...

Patching component oracle.help.ohj, 11.1.1.7.0...

Patching component oracle.rdbms.locator, 19.0.0.0.0...

Patching component oracle.perlint.expat, 2.0.1.0.4...

Patching component oracle.rdbms.util, 19.0.0.0.0...

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.rdbms, 19.0.0.0.0...

Patching component oracle.assistants.acf, 19.0.0.0.0...

Patching component oracle.assistants.deconfig, 19.0.0.0.0...

Patching component oracle.assistants.server, 19.0.0.0.0...

2. Meantime when you are rolling back a patch from Oracle Home, Parallelly try to rollback a patch from GI(Grid Home) which will be waiting on acquiring lock.

Which will hang due unable to acquire a lock on central inventory since already rollback running from Oracle Home put a lock on central inventory

[oracle@oraclelab2 35037840]$ . oraenv

ORACLE_SID = [+ASM] ? +ASM

The Oracle base remains unchanged with value /u01/app/oracle

[oracle@oraclelab2 35037840]$

[oracle@oraclelab2 35037840]$

[oracle@oraclelab2 35037840]$ env |grep ORA

ORACLE_SID=+ASM

ORACLE_BASE=/u01/app/oracle

ORACLE_HOME=/u01/app/19.0.0.0/grid

[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch lspatches

34580338;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34580338)

34428761;ACFS RELEASE UPDATE 19.17.0.0.0 (34428761)

34444834;OCW RELEASE UPDATE 19.17.0.0.0 (34444834)

34419443;Database Release Update : 19.17.0.0.221018 (34419443)

OPatch succeeded.

[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338

Oracle Interim Patch Installer version 12.2.0.1.41

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/19.0.0.0/grid

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/19.0.0.0/grid/oraInst.loc

OPatch version    : 12.2.0.1.41

OUI version       : 12.2.0.7.0

Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log

Unable to lock Central Inventory.  OPatch will attempt to re-lock.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch will sleep for few seconds, before re-trying to get the lock...

3. Verify the OPatch process running from Oracle Home and another from GI Home

[root@oraclelab2 ~]# ps -ef|grep opatch

oracle    7329  7261  0 00:10 pts/0    00:00:00 /bin/sh /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068

oracle    7466  7329 69 00:10 pts/0    00:01:51 /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jre/bin/java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch -cp /u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraInstallerNet.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraPrereq.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/share.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/xmlparserv2.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/emCfg.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/ojmisc.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/ocm/lib/emocmclnt.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/opatch.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/opatchsdk.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/automation.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/oracle.opatch.classpath.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/activation.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jaxb-api.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jaxb-impl.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/OsysModel.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/osysmodel-utils.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/CRSProductDriver.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/oracle.oplan.classpath.jar -DCommonLog.LOG_SESSION_ID= -DCommonLog.COMMAND_NAME=rollback -DOPatch.ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1 -DOPatch.DEBUG=false -DOPatch.MAKE=false -DOPatch.RUNNING_DIR=/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch -DOPatch.MW_HOME= -DOPatch.WL_HOME= -DOPatch.COMMON_COMPONENTS_HOME= -DOPatch.OUI_LOCATION=/u01/app/oracle/product/19.0.0.0/dbhome_1/oui -DOPatch.FMW_COMPONENT_HOME= -DOPatch.OPATCH_CLASSPATH= -DOPatch.WEBLOGIC_CLASSPATH= -DOPatch.SKIP_OUI_VERSION_CHECK= -DOPatch.NEXTGEN_HOME_CHECK=false -DOPatch.PARALLEL_ON_FMW_OH= oracle/opatch/OPatch rollback -id 35042068 -invPtrLoc /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc

oracle    7979 31071  0 00:11 pts/1    00:00:00 /bin/sh /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338

oracle    8125  7979  0 00:11 pts/1    00:00:00 /u01/app/19.0.0.0/grid/OPatch/jre/bin/java -d64 -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/app/19.0.0.0/grid/cfgtoollogs/opatch -cp /u01/app/19.0.0.0/grid/oui/jlib/OraInstaller.jar:/u01/app/19.0.0.0/grid/oui/jlib/OraInstallerNet.jar:/u01/app/19.0.0.0/grid/oui/jlib/OraPrereq.jar:/u01/app/19.0.0.0/grid/oui/jlib/share.jar:/u01/app/19.0.0.0/grid/oui/jlib/orai18n-mapping.jar:/u01/app/19.0.0.0/grid/oui/jlib/xmlparserv2.jar:/u01/app/19.0.0.0/grid/oui/jlib/emCfg.jar:/u01/app/19.0.0.0/grid/oui/jlib/ojmisc.jar:/u01/app/19.0.0.0/grid/OPatch/ocm/lib/emocmclnt.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/opatch.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/opatchsdk.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/automation.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/oracle.opatch.classpath.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/activation.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jaxb-api.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jaxb-impl.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/OsysModel.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/osysmodel-utils.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/CRSProductDriver.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/oracle.oplan.classpath.jar -DCommonLog.LOG_SESSION_ID= -DCommonLog.COMMAND_NAME=rollback -DOPatch.ORACLE_HOME=/u01/app/19.0.0.0/grid -DOPatch.DEBUG=false -DOPatch.MAKE=false -DOPatch.RUNNING_DIR=/u01/app/19.0.0.0/grid/OPatch -DOPatch.MW_HOME= -DOPatch.WL_HOME= -DOPatch.COMMON_COMPONENTS_HOME= -DOPatch.OUI_LOCATION=/u01/app/19.0.0.0/grid/oui -DOPatch.FMW_COMPONENT_HOME= -DOPatch.OPATCH_CLASSPATH= -DOPatch.WEBLOGIC_CLASSPATH= -DOPatch.SKIP_OUI_VERSION_CHECK= -DOPatch.NEXTGEN_HOME_CHECK=false -DOPatch.PARALLEL_ON_FMW_OH= oracle/opatch/OPatch rollback -id 34580338 -invPtrLoc /u01/app/19.0.0.0/grid/oraInst.loc

root      8558  8496  0 00:13 pts/2    00:00:00 grep --color=auto opatch

[root@oraclelab2 ~]#

4. As soon as Oracle Home patch rollback completes then GI home opatch rollback proceed further which was waiting on acquiring lock. 

[oracle@oraclelab2 ~]$ /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068

Oracle Interim Patch Installer version 12.2.0.1.41

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/19.0.0.0/dbhome_1

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc

OPatch version    : 12.2.0.1.41

OUI version       : 12.2.0.7.0

Log file location : /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log

Patches will be rolled back in the following order:

   35042068

The following patch(es) will be rolled back: 35042068

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/oracle/product/19.0.0.0/dbhome_1')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Rolling back patch 35042068...

RollbackSession rolling back interim patch '35042068' from OH '/u01/app/oracle/product/19.0.0.0/dbhome_1'

Patching component oracle.rsf, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...

Patching component oracle.slax.rsf, 19.0.0.0.0...

Patching component oracle.ordim.jai, 19.0.0.0.0...

Patching component oracle.bali.jewt, 11.1.1.6.0...

Patching component oracle.bali.ewt, 11.1.1.6.0...

Patching component oracle.help.ohj, 11.1.1.7.0...

Patching component oracle.rdbms.locator, 19.0.0.0.0...

Patching component oracle.perlint.expat, 2.0.1.0.4...

Patching component oracle.rdbms.util, 19.0.0.0.0...

Patching component oracle.rdbms.rsf, 19.0.0.0.0...

Patching component oracle.rdbms, 19.0.0.0.0...

Patching component oracle.assistants.acf, 19.0.0.0.0...

Patching component oracle.assistants.deconfig, 19.0.0.0.0...

Patching component oracle.assistants.server, 19.0.0.0.0...

Patching component oracle.blaslapack, 19.0.0.0.0...

Patching component oracle.buildtools.rsf, 19.0.0.0.0...

Patching component oracle.ctx, 19.0.0.0.0...

Patching component oracle.dbdev, 19.0.0.0.0...

Patching component oracle.dbjava.ic, 19.0.0.0.0...

Patching component oracle.dbjava.jdbc, 19.0.0.0.0...

Patching component oracle.dbjava.ucp, 19.0.0.0.0...

Patching component oracle.duma, 19.0.0.0.0...

Patching component oracle.javavm.client, 19.0.0.0.0...

Patching component oracle.ldap.owm, 19.0.0.0.0...

Patching component oracle.ldap.rsf, 19.0.0.0.0...

Patching component oracle.ldap.security.osdt, 19.0.0.0.0...

Patching component oracle.marvel, 19.0.0.0.0...

Patching component oracle.network.rsf, 19.0.0.0.0...

Patching component oracle.odbc.ic, 19.0.0.0.0...

Patching component oracle.ons, 19.0.0.0.0...

Patching component oracle.ons.ic, 19.0.0.0.0...

Patching component oracle.oracore.rsf, 19.0.0.0.0...

Patching component oracle.perlint, 5.28.1.0.0...

Patching component oracle.precomp.common.core, 19.0.0.0.0...

Patching component oracle.precomp.rsf, 19.0.0.0.0...

Patching component oracle.rdbms.crs, 19.0.0.0.0...

Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...

Patching component oracle.rdbms.deconfig, 19.0.0.0.0...

Patching component oracle.rdbms.oci, 19.0.0.0.0...

Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0...

Patching component oracle.rdbms.scheduler, 19.0.0.0.0...

Patching component oracle.rhp.db, 19.0.0.0.0...

Patching component oracle.sdo, 19.0.0.0.0...

Patching component oracle.sdo.locator.jrf, 19.0.0.0.0...

Patching component oracle.sqlplus, 19.0.0.0.0...

Patching component oracle.sqlplus.ic, 19.0.0.0.0...

Patching component oracle.wwg.plsql, 19.0.0.0.0...

Patching component oracle.xdk.xquery, 19.0.0.0.0...

Patching component oracle.javavm.server, 19.0.0.0.0...

Patching component oracle.xdk.parser.java, 19.0.0.0.0...

Patching component oracle.odbc, 19.0.0.0.0...

Patching component oracle.ctx.rsf, 19.0.0.0.0...

Patching component oracle.oraolap, 19.0.0.0.0...

Patching component oracle.rdbms.hsodbc, 19.0.0.0.0...

Patching component oracle.network.client, 19.0.0.0.0...

Patching component oracle.ctx.atg, 19.0.0.0.0...

Patching component oracle.rdbms.install.common, 19.0.0.0.0...

Patching component oracle.oraolap.dbscripts, 19.0.0.0.0...

Patching component oracle.ldap.rsf.ic, 19.0.0.0.0...

Patching component oracle.install.deinstalltool, 19.0.0.0.0...

Patching component oracle.ldap.client, 19.0.0.0.0...

Patching component oracle.rdbms.rman, 19.0.0.0.0...

Patching component oracle.ovm, 19.0.0.0.0...

Patching component oracle.rdbms.drdaas, 19.0.0.0.0...

Patching component oracle.rdbms.hs_common, 19.0.0.0.0...

Patching component oracle.oraolap.api, 19.0.0.0.0...

Patching component oracle.network.listener, 19.0.0.0.0...

Patching component oracle.rdbms.dv, 19.0.0.0.0...

Patching component oracle.sdo.locator, 19.0.0.0.0...

Patching component oracle.nlsrtl.rsf, 19.0.0.0.0...

Patching component oracle.xdk.rsf, 19.0.0.0.0...

Patching component oracle.xdk, 19.0.0.0.0...

Patching component oracle.dbtoolslistener, 19.0.0.0.0...

Patching component oracle.rdbms.install.plugins, 19.0.0.0.0...

Patching component oracle.ldap.ssl, 19.0.0.0.0...

Patching component oracle.rdbms.lbac, 19.0.0.0.0...

Patching component oracle.mgw.common, 19.0.0.0.0...

Patching component oracle.precomp.lang, 19.0.0.0.0...

Patching component oracle.precomp.common, 19.0.0.0.0...

Patching component oracle.jdk, 1.8.0.201.0...

RollbackSession removing interim patch '35042068' from inventory

Inactive sub-set patch [34419443] has become active due to the rolling back of a super-set patch [35042068].

Please refer to Doc ID 2161861.1 for any possible further required actions.

Log file location: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log

OPatch succeeded.

[oracle@oraclelab2 ~]$

[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338

Oracle Interim Patch Installer version 12.2.0.1.41

Copyright (c) 2024, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/19.0.0.0/grid

Central Inventory : /u01/app/oraInventory

   from           : /u01/app/19.0.0.0/grid/oraInst.loc

OPatch version    : 12.2.0.1.41

OUI version       : 12.2.0.7.0

Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log

Unable to lock Central Inventory.  OPatch will attempt to re-lock.

Do you want to proceed? [y|n]

y

User Responded with: Y

OPatch will sleep for few seconds, before re-trying to get the lock...

Patches will be rolled back in the following order:

   34580338

The following patch(es) will be rolled back: 34580338

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/u01/app/19.0.0.0/grid')

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Rolling back patch 34580338...

RollbackSession rolling back interim patch '34580338' from OH '/u01/app/19.0.0.0/grid'

Patching component oracle.tomcat.crs, 19.0.0.0.0...

RollbackSession removing interim patch '34580338' from inventory

Inactive sub-set patch [29401763] has become active due to the rolling back of a super-set patch [34580338].

Please refer to Doc ID 2161861.1 for any possible further required actions.

Log file location: /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log

OPatch succeeded.

[oracle@oraclelab2 35037840]$