Unable to acquire a central Inventory lock by OPatch due to multiple opatch run
OPatch will sleep for few seconds, before re-trying to get the lock...
High Level steps:
1. Rollback a patch from Oracle Home using opatch utility/tool
2. Meantime when you are rolling back a patch from Oracle Home, Parallelly try to rollback a patch from GI(Grid Home) which will be waiting on acquiring lock.
3. Verify the OPatch process running from Oracle Home and another from GI Home
4. As soon as Oracle Home patch rollback completes then GI home opatch rollback proceed further which was waiting on acquiring lock.
1. Rollback a patch from Oracle Home using opatch utility/tool
[root@oraclelab2 ~]# su - oracle
Last login: Thu May 2 00:06:12 IST 2024 on pts/0
[oracle@oraclelab2 ~]$ . oraenv
ORACLE_SID = [oracle] ? TESTDB
The Oracle base has been set to /u01/app/oracle
[oracle@oraclelab2 ~]$
[oracle@oraclelab2 ~]$ /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068
Oracle Interim Patch Installer version 12.2.0.1.41
Copyright (c) 2024, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.41
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log
Patches will be rolled back in the following order:
35042068
The following patch(es) will be rolled back: 35042068
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0.0.0/dbhome_1')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Rolling back patch 35042068...
RollbackSession rolling back interim patch '35042068' from OH '/u01/app/oracle/product/19.0.0.0/dbhome_1'
Patching component oracle.rsf, 19.0.0.0.0...
Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...
Patching component oracle.slax.rsf, 19.0.0.0.0...
Patching component oracle.ordim.jai, 19.0.0.0.0...
Patching component oracle.bali.jewt, 11.1.1.6.0...
Patching component oracle.bali.ewt, 11.1.1.6.0...
Patching component oracle.help.ohj, 11.1.1.7.0...
Patching component oracle.rdbms.locator, 19.0.0.0.0...
Patching component oracle.perlint.expat, 2.0.1.0.4...
Patching component oracle.rdbms.util, 19.0.0.0.0...
Patching component oracle.rdbms.rsf, 19.0.0.0.0...
Patching component oracle.rdbms, 19.0.0.0.0...
Patching component oracle.assistants.acf, 19.0.0.0.0...
Patching component oracle.assistants.deconfig, 19.0.0.0.0...
Patching component oracle.assistants.server, 19.0.0.0.0...
2. Meantime when you are rolling back a patch from Oracle Home, Parallelly try to rollback a patch from GI(Grid Home) which will be waiting on acquiring lock.
Which will hang due unable to acquire a lock on central inventory since already rollback running from Oracle Home put a lock on central inventory
[oracle@oraclelab2 35037840]$ . oraenv
ORACLE_SID = [+ASM] ? +ASM
The Oracle base remains unchanged with value /u01/app/oracle
[oracle@oraclelab2 35037840]$
[oracle@oraclelab2 35037840]$
[oracle@oraclelab2 35037840]$ env |grep ORA
ORACLE_SID=+ASM
ORACLE_BASE=/u01/app/oracle
ORACLE_HOME=/u01/app/19.0.0.0/grid
[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch lspatches
34580338;TOMCAT RELEASE UPDATE 19.0.0.0.0 (34580338)
34428761;ACFS RELEASE UPDATE 19.17.0.0.0 (34428761)
34444834;OCW RELEASE UPDATE 19.17.0.0.0 (34444834)
34419443;Database Release Update : 19.17.0.0.221018 (34419443)
OPatch succeeded.
[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338
Oracle Interim Patch Installer version 12.2.0.1.41
Copyright (c) 2024, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/19.0.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.41
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log
Unable to lock Central Inventory. OPatch will attempt to re-lock.
Do you want to proceed? [y|n]
y
User Responded with: Y
OPatch will sleep for few seconds, before re-trying to get the lock...
3. Verify the OPatch process running from Oracle Home and another from GI Home
[root@oraclelab2 ~]# ps -ef|grep opatch
oracle 7329 7261 0 00:10 pts/0 00:00:00 /bin/sh /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068
oracle 7466 7329 69 00:10 pts/0 00:01:51 /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jre/bin/java -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch -cp /u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraInstaller.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraInstallerNet.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/OraPrereq.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/share.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/orai18n-mapping.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/xmlparserv2.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/emCfg.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/oui/jlib/ojmisc.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/ocm/lib/emocmclnt.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/opatch.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/opatchsdk.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/automation.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/jlib/oracle.opatch.classpath.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/activation.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jaxb-api.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jaxb-impl.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/OsysModel.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/osysmodel-utils.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/CRSProductDriver.jar:/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/oplan/jlib/oracle.oplan.classpath.jar -DCommonLog.LOG_SESSION_ID= -DCommonLog.COMMAND_NAME=rollback -DOPatch.ORACLE_HOME=/u01/app/oracle/product/19.0.0.0/dbhome_1 -DOPatch.DEBUG=false -DOPatch.MAKE=false -DOPatch.RUNNING_DIR=/u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch -DOPatch.MW_HOME= -DOPatch.WL_HOME= -DOPatch.COMMON_COMPONENTS_HOME= -DOPatch.OUI_LOCATION=/u01/app/oracle/product/19.0.0.0/dbhome_1/oui -DOPatch.FMW_COMPONENT_HOME= -DOPatch.OPATCH_CLASSPATH= -DOPatch.WEBLOGIC_CLASSPATH= -DOPatch.SKIP_OUI_VERSION_CHECK= -DOPatch.NEXTGEN_HOME_CHECK=false -DOPatch.PARALLEL_ON_FMW_OH= oracle/opatch/OPatch rollback -id 35042068 -invPtrLoc /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc
oracle 7979 31071 0 00:11 pts/1 00:00:00 /bin/sh /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338
oracle 8125 7979 0 00:11 pts/1 00:00:00 /u01/app/19.0.0.0/grid/OPatch/jre/bin/java -d64 -Xmx3072m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/u01/app/19.0.0.0/grid/cfgtoollogs/opatch -cp /u01/app/19.0.0.0/grid/oui/jlib/OraInstaller.jar:/u01/app/19.0.0.0/grid/oui/jlib/OraInstallerNet.jar:/u01/app/19.0.0.0/grid/oui/jlib/OraPrereq.jar:/u01/app/19.0.0.0/grid/oui/jlib/share.jar:/u01/app/19.0.0.0/grid/oui/jlib/orai18n-mapping.jar:/u01/app/19.0.0.0/grid/oui/jlib/xmlparserv2.jar:/u01/app/19.0.0.0/grid/oui/jlib/emCfg.jar:/u01/app/19.0.0.0/grid/oui/jlib/ojmisc.jar:/u01/app/19.0.0.0/grid/OPatch/ocm/lib/emocmclnt.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/opatch.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/opatchsdk.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/automation.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar:/u01/app/19.0.0.0/grid/OPatch/jlib/oracle.opatch.classpath.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/activation.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jaxb-api.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jaxb-impl.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/OsysModel.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/osysmodel-utils.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/CRSProductDriver.jar:/u01/app/19.0.0.0/grid/OPatch/oplan/jlib/oracle.oplan.classpath.jar -DCommonLog.LOG_SESSION_ID= -DCommonLog.COMMAND_NAME=rollback -DOPatch.ORACLE_HOME=/u01/app/19.0.0.0/grid -DOPatch.DEBUG=false -DOPatch.MAKE=false -DOPatch.RUNNING_DIR=/u01/app/19.0.0.0/grid/OPatch -DOPatch.MW_HOME= -DOPatch.WL_HOME= -DOPatch.COMMON_COMPONENTS_HOME= -DOPatch.OUI_LOCATION=/u01/app/19.0.0.0/grid/oui -DOPatch.FMW_COMPONENT_HOME= -DOPatch.OPATCH_CLASSPATH= -DOPatch.WEBLOGIC_CLASSPATH= -DOPatch.SKIP_OUI_VERSION_CHECK= -DOPatch.NEXTGEN_HOME_CHECK=false -DOPatch.PARALLEL_ON_FMW_OH= oracle/opatch/OPatch rollback -id 34580338 -invPtrLoc /u01/app/19.0.0.0/grid/oraInst.loc
root 8558 8496 0 00:13 pts/2 00:00:00 grep --color=auto opatch
[root@oraclelab2 ~]#
4. As soon as Oracle Home patch rollback completes then GI home opatch rollback proceed further which was waiting on acquiring lock.
[oracle@oraclelab2 ~]$ /u01/app/oracle/product/19.0.0.0/dbhome_1/OPatch/opatch rollback -id 35042068
Oracle Interim Patch Installer version 12.2.0.1.41
Copyright (c) 2024, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/19.0.0.0/dbhome_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/19.0.0.0/dbhome_1/oraInst.loc
OPatch version : 12.2.0.1.41
OUI version : 12.2.0.7.0
Log file location : /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log
Patches will be rolled back in the following order:
35042068
The following patch(es) will be rolled back: 35042068
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/19.0.0.0/dbhome_1')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Rolling back patch 35042068...
RollbackSession rolling back interim patch '35042068' from OH '/u01/app/oracle/product/19.0.0.0/dbhome_1'
Patching component oracle.rsf, 19.0.0.0.0...
Patching component oracle.nlsrtl.rsf.core, 19.0.0.0.0...
Patching component oracle.slax.rsf, 19.0.0.0.0...
Patching component oracle.ordim.jai, 19.0.0.0.0...
Patching component oracle.bali.jewt, 11.1.1.6.0...
Patching component oracle.bali.ewt, 11.1.1.6.0...
Patching component oracle.help.ohj, 11.1.1.7.0...
Patching component oracle.rdbms.locator, 19.0.0.0.0...
Patching component oracle.perlint.expat, 2.0.1.0.4...
Patching component oracle.rdbms.util, 19.0.0.0.0...
Patching component oracle.rdbms.rsf, 19.0.0.0.0...
Patching component oracle.rdbms, 19.0.0.0.0...
Patching component oracle.assistants.acf, 19.0.0.0.0...
Patching component oracle.assistants.deconfig, 19.0.0.0.0...
Patching component oracle.assistants.server, 19.0.0.0.0...
Patching component oracle.blaslapack, 19.0.0.0.0...
Patching component oracle.buildtools.rsf, 19.0.0.0.0...
Patching component oracle.ctx, 19.0.0.0.0...
Patching component oracle.dbdev, 19.0.0.0.0...
Patching component oracle.dbjava.ic, 19.0.0.0.0...
Patching component oracle.dbjava.jdbc, 19.0.0.0.0...
Patching component oracle.dbjava.ucp, 19.0.0.0.0...
Patching component oracle.duma, 19.0.0.0.0...
Patching component oracle.javavm.client, 19.0.0.0.0...
Patching component oracle.ldap.owm, 19.0.0.0.0...
Patching component oracle.ldap.rsf, 19.0.0.0.0...
Patching component oracle.ldap.security.osdt, 19.0.0.0.0...
Patching component oracle.marvel, 19.0.0.0.0...
Patching component oracle.network.rsf, 19.0.0.0.0...
Patching component oracle.odbc.ic, 19.0.0.0.0...
Patching component oracle.ons, 19.0.0.0.0...
Patching component oracle.ons.ic, 19.0.0.0.0...
Patching component oracle.oracore.rsf, 19.0.0.0.0...
Patching component oracle.perlint, 5.28.1.0.0...
Patching component oracle.precomp.common.core, 19.0.0.0.0...
Patching component oracle.precomp.rsf, 19.0.0.0.0...
Patching component oracle.rdbms.crs, 19.0.0.0.0...
Patching component oracle.rdbms.dbscripts, 19.0.0.0.0...
Patching component oracle.rdbms.deconfig, 19.0.0.0.0...
Patching component oracle.rdbms.oci, 19.0.0.0.0...
Patching component oracle.rdbms.rsf.ic, 19.0.0.0.0...
Patching component oracle.rdbms.scheduler, 19.0.0.0.0...
Patching component oracle.rhp.db, 19.0.0.0.0...
Patching component oracle.sdo, 19.0.0.0.0...
Patching component oracle.sdo.locator.jrf, 19.0.0.0.0...
Patching component oracle.sqlplus, 19.0.0.0.0...
Patching component oracle.sqlplus.ic, 19.0.0.0.0...
Patching component oracle.wwg.plsql, 19.0.0.0.0...
Patching component oracle.xdk.xquery, 19.0.0.0.0...
Patching component oracle.javavm.server, 19.0.0.0.0...
Patching component oracle.xdk.parser.java, 19.0.0.0.0...
Patching component oracle.odbc, 19.0.0.0.0...
Patching component oracle.ctx.rsf, 19.0.0.0.0...
Patching component oracle.oraolap, 19.0.0.0.0...
Patching component oracle.rdbms.hsodbc, 19.0.0.0.0...
Patching component oracle.network.client, 19.0.0.0.0...
Patching component oracle.ctx.atg, 19.0.0.0.0...
Patching component oracle.rdbms.install.common, 19.0.0.0.0...
Patching component oracle.oraolap.dbscripts, 19.0.0.0.0...
Patching component oracle.ldap.rsf.ic, 19.0.0.0.0...
Patching component oracle.install.deinstalltool, 19.0.0.0.0...
Patching component oracle.ldap.client, 19.0.0.0.0...
Patching component oracle.rdbms.rman, 19.0.0.0.0...
Patching component oracle.ovm, 19.0.0.0.0...
Patching component oracle.rdbms.drdaas, 19.0.0.0.0...
Patching component oracle.rdbms.hs_common, 19.0.0.0.0...
Patching component oracle.oraolap.api, 19.0.0.0.0...
Patching component oracle.network.listener, 19.0.0.0.0...
Patching component oracle.rdbms.dv, 19.0.0.0.0...
Patching component oracle.sdo.locator, 19.0.0.0.0...
Patching component oracle.nlsrtl.rsf, 19.0.0.0.0...
Patching component oracle.xdk.rsf, 19.0.0.0.0...
Patching component oracle.xdk, 19.0.0.0.0...
Patching component oracle.dbtoolslistener, 19.0.0.0.0...
Patching component oracle.rdbms.install.plugins, 19.0.0.0.0...
Patching component oracle.ldap.ssl, 19.0.0.0.0...
Patching component oracle.rdbms.lbac, 19.0.0.0.0...
Patching component oracle.mgw.common, 19.0.0.0.0...
Patching component oracle.precomp.lang, 19.0.0.0.0...
Patching component oracle.precomp.common, 19.0.0.0.0...
Patching component oracle.jdk, 1.8.0.201.0...
RollbackSession removing interim patch '35042068' from inventory
Inactive sub-set patch [34419443] has become active due to the rolling back of a super-set patch [35042068].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/oracle/product/19.0.0.0/dbhome_1/cfgtoollogs/opatch/opatch2024-05-02_00-10-50AM_1.log
OPatch succeeded.
[oracle@oraclelab2 ~]$
[oracle@oraclelab2 35037840]$ /u01/app/19.0.0.0/grid/OPatch/opatch rollback -id 34580338
Oracle Interim Patch Installer version 12.2.0.1.41
Copyright (c) 2024, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/19.0.0.0/grid
Central Inventory : /u01/app/oraInventory
from : /u01/app/19.0.0.0/grid/oraInst.loc
OPatch version : 12.2.0.1.41
OUI version : 12.2.0.7.0
Log file location : /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log
Unable to lock Central Inventory. OPatch will attempt to re-lock.
Do you want to proceed? [y|n]
y
User Responded with: Y
OPatch will sleep for few seconds, before re-trying to get the lock...
Patches will be rolled back in the following order:
34580338
The following patch(es) will be rolled back: 34580338
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/19.0.0.0/grid')
Is the local system ready for patching? [y|n]
y
User Responded with: Y
Rolling back patch 34580338...
RollbackSession rolling back interim patch '34580338' from OH '/u01/app/19.0.0.0/grid'
Patching component oracle.tomcat.crs, 19.0.0.0.0...
RollbackSession removing interim patch '34580338' from inventory
Inactive sub-set patch [29401763] has become active due to the rolling back of a super-set patch [34580338].
Please refer to Doc ID 2161861.1 for any possible further required actions.
Log file location: /u01/app/19.0.0.0/grid/cfgtoollogs/opatch/opatch2024-05-02_00-11-54AM_1.log
OPatch succeeded.
[oracle@oraclelab2 35037840]$