Pre checks and Post check for Platinum Bundle Patching
Before Exadata bundle patching we need to do certain prechks and once the patching completed we should do some post checks.
I. Pre Patching Verification Steps.
II. Post Patching Verification Steps.
Please follow below in details steps and execution outputs
I. Pre Patching Verification Steps.
YUM:
====
1. Verify the YUM image status
>dcli -l root -g dbs_group imageinfo
10.89.0.141:
10.89.0.141: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.141: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.141: Image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
10.89.0.141: Image activated: 2018-11-10 01:12:21 -0500
10.89.0.141: Image status: success ===============================================> Should be success
10.89.0.141: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.141:
10.89.0.142:
10.89.0.142: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.142: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.142: Image version: 18.1.7.0.0.180821
10.89.0.142: Image activated: 2018-11-10 01:15:28 -0500
10.89.0.142: Image status: success ===============================================> Should be latest version
10.89.0.142: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.142:
npexdbadm01.r02.xlgs.local:(root)-/root
>
2. Check uptime DB nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g dbs_group uptime
10.89.0.141: 04:45:13 up 2 days, 3:36, 2 users, load average: 5.18, 9.15, 8.69
10.89.0.142: 04:45:13 up 2 days, 1:36, 0 users, load average: 10.34, 6.83, 4.91
npexdbadm01.r02.xlgs.local:(root)-/root
>
CELL:
====
3. Verify the CELL image status
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group imageinfo
npexceladm01:
npexceladm01: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm01: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm01: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm01:
npexceladm01: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm01: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm01: Active image activated: 2018-11-10 04:38:16 -0500
npexceladm01: Active image status: success ===============================================> Should be success
npexceladm01: Active system partition on device: /dev/md6
npexceladm01: Active software partition on device: /dev/md8
npexceladm01:
npexceladm01: Cell boot usb partition: /dev/sdm1
npexceladm01: Cell boot usb version: 18.1.7.0.0.180821
npexceladm01:
npexceladm01: Inactive image version: 18.1.5.0.0.180506
npexceladm01: Inactive image activated: 2018-07-22 05:49:33 -0400
npexceladm01: Inactive image status: success
npexceladm01: Inactive system partition on device: /dev/md5
npexceladm01: Inactive software partition on device: /dev/md7
npexceladm01:
npexceladm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm01: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm01: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm01: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm01: Rollback to the inactive partitions: Possible
npexceladm02:
npexceladm02: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm02: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm02: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm02:
npexceladm02: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm02: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm02: Active image activated: 2018-11-10 04:33:15 -0500
npexceladm02: Active image status: success ===============================================> Should be success
npexceladm02: Active system partition on device: /dev/md6
npexceladm02: Active software partition on device: /dev/md8
npexceladm02:
npexceladm02: Cell boot usb partition: /dev/sdm1
npexceladm02: Cell boot usb version: 18.1.7.0.0.180821
npexceladm02:
npexceladm02: Inactive image version: 18.1.5.0.0.180506
npexceladm02: Inactive image activated: 2018-07-22 05:43:58 -0400
npexceladm02: Inactive image status: success
npexceladm02: Inactive system partition on device: /dev/md5
npexceladm02: Inactive software partition on device: /dev/md7
npexceladm02:
npexceladm02: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm02: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm02: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm02: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm02: Rollback to the inactive partitions: Possible
npexceladm03:
npexceladm03: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm03: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm03: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm03:
npexceladm03: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm03: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm03: Active image activated: 2018-11-10 04:37:24 -0500
npexceladm03: Active image status: success ===============================================> Should be success
npexceladm03: Active system partition on device: /dev/md6
npexceladm03: Active software partition on device: /dev/md8
npexceladm03:
npexceladm03: Cell boot usb partition: /dev/sdm1
npexceladm03: Cell boot usb version: 18.1.7.0.0.180821
npexceladm03:
npexceladm03: Inactive image version: 18.1.5.0.0.180506
npexceladm03: Inactive image activated: 2018-07-22 05:48:16 -0400
npexceladm03: Inactive image status: success
npexceladm03: Inactive system partition on device: /dev/md5
npexceladm03: Inactive software partition on device: /dev/md7
npexceladm03:
npexceladm03: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm03: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm03: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm03: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm03: Rollback to the inactive partitions: Possible
npexceladm04:
npexceladm04: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm04: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm04: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm04:
npexceladm04: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm04: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm04: Active image activated: 2018-11-10 05:34:52 -0500
npexceladm04: Active image status: success ===============================================> Should be success
npexceladm04: Active system partition on device: /dev/md24p5
npexceladm04: Active software partition on device: /dev/md24p7
npexceladm04:
npexceladm04: Cell boot usb partition: /dev/md25p1
npexceladm04: Cell boot usb version: 18.1.7.0.0.180821
npexceladm04:
npexceladm04: Inactive image version: 18.1.5.0.0.180506
npexceladm04: Inactive image activated: 2018-07-22 06:52:06 -0400
npexceladm04: Inactive image status: success
npexceladm04: Inactive system partition on device: /dev/md24p6
npexceladm04: Inactive software partition on device: /dev/md24p8
npexceladm04:
npexceladm04: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm04: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm04: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm04: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm04: Rollback to the inactive partitions: Possible
npexceladm05:
npexceladm05: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm05: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm05: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm05:
npexceladm05: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm05: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm05: Active image activated: 2018-11-10 05:32:22 -0500
npexceladm05: Active image status: success ===============================================> Should be success
npexceladm05: Active system partition on device: /dev/md24p5
npexceladm05: Active software partition on device: /dev/md24p7
npexceladm05:
npexceladm05: Cell boot usb partition: /dev/md25p1
npexceladm05: Cell boot usb version: 18.1.7.0.0.180821
npexceladm05:
npexceladm05: Inactive image version: 18.1.5.0.0.180506
npexceladm05: Inactive image activated: 2018-07-22 06:50:07 -0400
npexceladm05: Inactive image status: success
npexceladm05: Inactive system partition on device: /dev/md24p6
npexceladm05: Inactive software partition on device: /dev/md24p8
npexceladm05:
npexceladm05: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm05: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm05: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm05: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm05: Rollback to the inactive partitions: Possible
npexdbadm01.r02.xlgs.local:(root)-/root
>
4. Check uptime of CELLs
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group uptime
npexceladm01: 03:45:12 up 1 day, 23:12, 0 users, load average: 10.77, 8.09, 5.59
npexceladm02: 03:40:00 up 1 day, 23:12, 0 users, load average: 11.90, 8.49, 5.86
npexceladm03: 03:44:11 up 1 day, 23:12, 0 users, load average: 11.84, 8.96, 6.04
npexceladm04: 04:42:10 up 1 day, 23:11, 0 users, load average: 13.25, 10.07, 6.31
npexceladm05: 04:38:43 up 1 day, 23:10, 0 users, load average: 14.50, 10.90, 6.78
npexdbadm01.r02.xlgs.local:(root)-/root
>
IB Swicthes:
============
5. Check IB switch verison
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst version
npexsw-ibb01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-ibb01: Build time: Mar 2 2018 12:54:44
npexsw-ibb01: SP board info:
npexsw-ibb01: Manufacturing Date: N/A
npexsw-ibb01: Serial Number: "NED620379"
npexsw-ibb01: Hardware Revision: 0x0100
npexsw-ibb01: Firmware Revision: 0x0000
npexsw-ibb01: BIOS version: NUP1R918
npexsw-ibb01: BIOS date: 01/19/2016
npexsw-iba01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-iba01: Build time: Mar 2 2018 12:54:44
npexsw-iba01: SP board info:
npexsw-iba01: Manufacturing Date: N/A
npexsw-iba01: Serial Number: "NED620102"
npexsw-iba01: Hardware Revision: 0x0100
npexsw-iba01: Firmware Revision: 0x0000
npexsw-iba01: BIOS version: NUP1R918
npexsw-iba01: BIOS date: 01/19/2016
npexdbadm01.r02.xlgs.local:(root)-/root
>
6. Check uptime of IB Switches
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst uptime
npexsw-ibb01: 05:29:31 up 112 days, 22:12, 0 users, load average: 0.13, 0.16, 0.15
npexsw-iba01: 04:54:07 up 112 days, 21:29, 0 users, load average: 0.14, 0.13, 0.14
npexdbadm01.r02.xlgs.local:(root)-/root
>
BP:
===
7. Verify the opatch lsinventory
GI home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l grid -c "/u01/app/12.2.0.1/grid/OPatch/opatch lspatches -oh /u01/app/12.2.0.1/grid "'
DB Home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l orauat -c "/u01/app/oracle/product/11.2.0.4/DB1/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB1"'
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l obiuat -c "/u01/app/oracle/product/11.2.0.4/DB2/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB2"'
>
DB Catbudle:
============
8. Login to DB and check for catbundle status:
set pages 1000
set lines 1000
col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
DB1:
========
SQL> set pages 1000
SQL> set lines 1000
SQL> col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
SQL> SQL> SQL> SQL> SQL> SQL>
ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
------------------------------ -------------------- -------------------- ------------------------------ ---------- -------------------- --------------------------------------------------
20-FEB-13 12.51.48.044154 AM VIEW INVALIDATE 8289601 view invalidation
20-FEB-13 12.51.48.080024 AM UPGRADE SERVER 11.2.0.3.0 Upgraded from 11.1.0.7.0
20-FEB-13 11.29.14.796574 AM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
23-JAN-08 04.07.02.182324 PM CPU SERVER 10.2.0.3.0 6079591 CPUJul2007
26-NOV-08 11.39.09.067225 PM UPGRADE SERVER 11.1.0.6.0 Upgraded from 10.2.0.3.0
22-JAN-09 04.34.49.320298 AM UPGRADE SERVER 11.1.0.7.0 Upgraded from 11.1.0.6.0
20-FEB-13 03.26.12.509440 PM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
11-MAR-17 01.18.18.341094 PM VIEW INVALIDATE 8289601 view invalidation
11-MAR-17 01.18.18.375351 PM UPGRADE SERVER 11.2.0.4.0 Upgraded from 11.2.0.3.0
11-MAR-17 01.19.36.166175 PM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0
11-MAR-17 07.34.06.000000 PM APPLY 17501296 Patch 17501296 applied
11-MAR-17 07.37.56.000000 PM APPLY 22731026 Patch 22731026 applied
11-MAY-17 11.04.58.734791 AM APPLY SERVER 11.2.0.4 161018 EXA BP161018
10-FEB-18 04.24.17.618719 AM jvmpsu.sql SERVER 11.2.0.4.171017OJVMPSU 0 RAN jvmpsu.sql
10-FEB-18 04.24.17.651053 AM APPLY SERVER 11.2.0.4.171017OJVMPSU 0 OJVM PSU post-install
10-FEB-18 04.24.17.653268 AM APPLY 26635834 Patch 26635834 applied
10-FEB-18 04.25.36.182498 AM APPLY SERVER 11.2.0.4 171017 EXA BP171017
23-FEB-18 09.48.21.702054 PM jvmpsu.sql SERVER 11.2.0.4.180116OJVMPSU 0 RAN jvmpsu.sql
23-FEB-18 09.48.21.739476 PM APPLY SERVER 11.2.0.4.180116OJVMPSU 0 OJVM PSU post-install
23-FEB-18 09.48.21.741801 PM APPLY 26925532 Patch 26925532 applied
19-MAY-18 11.40.29.378510 PM jvmpsu.sql SERVER 11.2.0.4.180417OJVMPSU 0 RAN jvmpsu.sql
19-MAY-18 11.40.29.691650 PM APPLY SERVER 11.2.0.4.180417OJVMPSU 0 OJVM PSU post-install
19-MAY-18 11.40.29.694744 PM APPLY 27475598 Patch 27475598 applied
19-AUG-18 12.08.55.766831 AM APPLY SERVER 11.2.0.4 180417 EXA BP180417
19-AUG-18 02.38.28.101436 AM jvmpsu.sql SERVER 11.2.0.4.180717OJVMPSU 0 RAN jvmpsu.sql
19-AUG-18 02.38.28.352391 AM APPLY SERVER 11.2.0.4.180717OJVMPSU 0 OJVM PSU post-install
19-AUG-18 02.38.28.354748 AM APPLY 27923163 Patch 27923163 applied
05-NOV-18 03.42.35.835165 AM jvmpsu.sql SERVER 11.2.0.4.181016OJVMPSU 0 RAN jvmpsu.sql
05-NOV-18 03.42.36.277916 AM APPLY SERVER 11.2.0.4.181016OJVMPSU 0 OJVM PSU post-install
05-NOV-18 03.42.36.280072 AM APPLY 28440700 Patch 28440700 applied
10-NOV-18 06.05.44.941482 AM APPLY SERVER 11.2.0.4 180717 EXA BP180717
31 rows selected.
SQL>
--- Similarly perform on all DBs
Genaral checks:
===============
9. Check DB staus - should be up and running
10. Check Listener staus - should be up and running
---> SCAN listsneteners
---> Local listneters
11. Do clusterware health check on cluster nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm01 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm02 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm02 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
12. Verify the Patch asssessment provided by Oracle
13. Veriry the Patch Plan provided by Oracle.
14. Veriry the Prechks logs provided by Oracle.
15. Take root /u01 OS backup before the patching.
16. Run the exachk and verify the report.
17. verify the below OS comand outputs
df -h
lvs
cat /etc/fstab
18. Verify the applciation health check.
II. Post Patching Verification Steps.
YUM:
====
1. Verify the YUM image status
>dcli -l root -g dbs_group imageinfo
10.89.0.141:
10.89.0.141: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.141: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.141: Image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
10.89.0.141: Image activated: 2018-11-10 01:12:21 -0500
10.89.0.141: Image status: success ===============================================> Should be success
10.89.0.141: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.141:
10.89.0.142:
10.89.0.142: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.142: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.142: Image version: 18.1.7.0.0.180821
10.89.0.142: Image activated: 2018-11-10 01:15:28 -0500
10.89.0.142: Image status: success ===============================================> Should be latest version
10.89.0.142: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.142:
npexdbadm01.r02.xlgs.local:(root)-/root
>
2. Check uptime DB nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g dbs_group uptime
10.89.0.141: 04:45:13 up 2 days, 3:36, 2 users, load average: 5.18, 9.15, 8.69
10.89.0.142: 04:45:13 up 2 days, 1:36, 0 users, load average: 10.34, 6.83, 4.91
npexdbadm01.r02.xlgs.local:(root)-/root
>
CELL:
====
3. Verify the CELL image status
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group imageinfo
npexceladm01:
npexceladm01: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm01: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm01: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm01:
npexceladm01: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm01: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm01: Active image activated: 2018-11-10 04:38:16 -0500
npexceladm01: Active image status: success ===============================================> Should be success
npexceladm01: Active system partition on device: /dev/md6
npexceladm01: Active software partition on device: /dev/md8
npexceladm01:
npexceladm01: Cell boot usb partition: /dev/sdm1
npexceladm01: Cell boot usb version: 18.1.7.0.0.180821
npexceladm01:
npexceladm01: Inactive image version: 18.1.5.0.0.180506
npexceladm01: Inactive image activated: 2018-07-22 05:49:33 -0400
npexceladm01: Inactive image status: success
npexceladm01: Inactive system partition on device: /dev/md5
npexceladm01: Inactive software partition on device: /dev/md7
npexceladm01:
npexceladm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm01: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm01: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm01: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm01: Rollback to the inactive partitions: Possible
npexceladm02:
npexceladm02: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm02: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm02: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm02:
npexceladm02: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm02: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm02: Active image activated: 2018-11-10 04:33:15 -0500
npexceladm02: Active image status: success ===============================================> Should be success
npexceladm02: Active system partition on device: /dev/md6
npexceladm02: Active software partition on device: /dev/md8
npexceladm02:
npexceladm02: Cell boot usb partition: /dev/sdm1
npexceladm02: Cell boot usb version: 18.1.7.0.0.180821
npexceladm02:
npexceladm02: Inactive image version: 18.1.5.0.0.180506
npexceladm02: Inactive image activated: 2018-07-22 05:43:58 -0400
npexceladm02: Inactive image status: success
npexceladm02: Inactive system partition on device: /dev/md5
npexceladm02: Inactive software partition on device: /dev/md7
npexceladm02:
npexceladm02: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm02: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm02: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm02: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm02: Rollback to the inactive partitions: Possible
npexceladm03:
npexceladm03: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm03: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm03: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm03:
npexceladm03: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm03: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm03: Active image activated: 2018-11-10 04:37:24 -0500
npexceladm03: Active image status: success ===============================================> Should be success
npexceladm03: Active system partition on device: /dev/md6
npexceladm03: Active software partition on device: /dev/md8
npexceladm03:
npexceladm03: Cell boot usb partition: /dev/sdm1
npexceladm03: Cell boot usb version: 18.1.7.0.0.180821
npexceladm03:
npexceladm03: Inactive image version: 18.1.5.0.0.180506
npexceladm03: Inactive image activated: 2018-07-22 05:48:16 -0400
npexceladm03: Inactive image status: success
npexceladm03: Inactive system partition on device: /dev/md5
npexceladm03: Inactive software partition on device: /dev/md7
npexceladm03:
npexceladm03: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm03: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm03: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm03: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm03: Rollback to the inactive partitions: Possible
npexceladm04:
npexceladm04: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm04: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm04: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm04:
npexceladm04: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm04: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm04: Active image activated: 2018-11-10 05:34:52 -0500
npexceladm04: Active image status: success ===============================================> Should be success
npexceladm04: Active system partition on device: /dev/md24p5
npexceladm04: Active software partition on device: /dev/md24p7
npexceladm04:
npexceladm04: Cell boot usb partition: /dev/md25p1
npexceladm04: Cell boot usb version: 18.1.7.0.0.180821
npexceladm04:
npexceladm04: Inactive image version: 18.1.5.0.0.180506
npexceladm04: Inactive image activated: 2018-07-22 06:52:06 -0400
npexceladm04: Inactive image status: success
npexceladm04: Inactive system partition on device: /dev/md24p6
npexceladm04: Inactive software partition on device: /dev/md24p8
npexceladm04:
npexceladm04: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm04: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm04: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm04: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm04: Rollback to the inactive partitions: Possible
npexceladm05:
npexceladm05: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm05: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm05: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm05:
npexceladm05: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm05: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm05: Active image activated: 2018-11-10 05:32:22 -0500
npexceladm05: Active image status: success ===============================================> Should be success
npexceladm05: Active system partition on device: /dev/md24p5
npexceladm05: Active software partition on device: /dev/md24p7
npexceladm05:
npexceladm05: Cell boot usb partition: /dev/md25p1
npexceladm05: Cell boot usb version: 18.1.7.0.0.180821
npexceladm05:
npexceladm05: Inactive image version: 18.1.5.0.0.180506
npexceladm05: Inactive image activated: 2018-07-22 06:50:07 -0400
npexceladm05: Inactive image status: success
npexceladm05: Inactive system partition on device: /dev/md24p6
npexceladm05: Inactive software partition on device: /dev/md24p8
npexceladm05:
npexceladm05: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm05: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm05: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm05: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm05: Rollback to the inactive partitions: Possible
npexdbadm01.r02.xlgs.local:(root)-/root
>
4. Check uptime of CELLs
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group uptime
npexceladm01: 03:45:12 up 1 day, 23:12, 0 users, load average: 10.77, 8.09, 5.59
npexceladm02: 03:40:00 up 1 day, 23:12, 0 users, load average: 11.90, 8.49, 5.86
npexceladm03: 03:44:11 up 1 day, 23:12, 0 users, load average: 11.84, 8.96, 6.04
npexceladm04: 04:42:10 up 1 day, 23:11, 0 users, load average: 13.25, 10.07, 6.31
npexceladm05: 04:38:43 up 1 day, 23:10, 0 users, load average: 14.50, 10.90, 6.78
npexdbadm01.r02.xlgs.local:(root)-/root
>
IB Swicthes:
============
5. Check IB switch verison
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst version
npexsw-ibb01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-ibb01: Build time: Mar 2 2018 12:54:44
npexsw-ibb01: SP board info:
npexsw-ibb01: Manufacturing Date: N/A
npexsw-ibb01: Serial Number: "NED620379"
npexsw-ibb01: Hardware Revision: 0x0100
npexsw-ibb01: Firmware Revision: 0x0000
npexsw-ibb01: BIOS version: NUP1R918
npexsw-ibb01: BIOS date: 01/19/2016
npexsw-iba01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-iba01: Build time: Mar 2 2018 12:54:44
npexsw-iba01: SP board info:
npexsw-iba01: Manufacturing Date: N/A
npexsw-iba01: Serial Number: "NED620102"
npexsw-iba01: Hardware Revision: 0x0100
npexsw-iba01: Firmware Revision: 0x0000
npexsw-iba01: BIOS version: NUP1R918
npexsw-iba01: BIOS date: 01/19/2016
npexdbadm01.r02.xlgs.local:(root)-/root
>
6. Check uptime of IB Switches
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst uptime
npexsw-ibb01: 05:29:31 up 112 days, 22:12, 0 users, load average: 0.13, 0.16, 0.15
npexsw-iba01: 04:54:07 up 112 days, 21:29, 0 users, load average: 0.14, 0.13, 0.14
npexdbadm01.r02.xlgs.local:(root)-/root
>
BP:
===
7. Verify the opatch lsinventory
GI home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l grid -c "/u01/app/12.2.0.1/grid/OPatch/opatch lspatches -oh /u01/app/12.2.0.1/grid "'
DB Home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l orauat -c "/u01/app/oracle/product/11.2.0.4/DB1/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB1"'
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l obiuat -c "/u01/app/oracle/product/11.2.0.4/DB2/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB2"'
>
DB Catbudle:
============
8. Login to DB and check for catbundle status:
set pages 1000
set lines 1000
col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
DB1:
========
SQL> set pages 1000
SQL> set lines 1000
SQL> col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
SQL> SQL> SQL> SQL> SQL> SQL>
ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
------------------------------ -------------------- -------------------- ------------------------------ ---------- -------------------- --------------------------------------------------
20-FEB-13 12.51.48.044154 AM VIEW INVALIDATE 8289601 view invalidation
20-FEB-13 12.51.48.080024 AM UPGRADE SERVER 11.2.0.3.0 Upgraded from 11.1.0.7.0
20-FEB-13 11.29.14.796574 AM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
23-JAN-08 04.07.02.182324 PM CPU SERVER 10.2.0.3.0 6079591 CPUJul2007
26-NOV-08 11.39.09.067225 PM UPGRADE SERVER 11.1.0.6.0 Upgraded from 10.2.0.3.0
22-JAN-09 04.34.49.320298 AM UPGRADE SERVER 11.1.0.7.0 Upgraded from 11.1.0.6.0
20-FEB-13 03.26.12.509440 PM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
11-MAR-17 01.18.18.341094 PM VIEW INVALIDATE 8289601 view invalidation
11-MAR-17 01.18.18.375351 PM UPGRADE SERVER 11.2.0.4.0 Upgraded from 11.2.0.3.0
11-MAR-17 01.19.36.166175 PM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0
11-MAR-17 07.34.06.000000 PM APPLY 17501296 Patch 17501296 applied
11-MAR-17 07.37.56.000000 PM APPLY 22731026 Patch 22731026 applied
11-MAY-17 11.04.58.734791 AM APPLY SERVER 11.2.0.4 161018 EXA BP161018
10-FEB-18 04.24.17.618719 AM jvmpsu.sql SERVER 11.2.0.4.171017OJVMPSU 0 RAN jvmpsu.sql
10-FEB-18 04.24.17.651053 AM APPLY SERVER 11.2.0.4.171017OJVMPSU 0 OJVM PSU post-install
10-FEB-18 04.24.17.653268 AM APPLY 26635834 Patch 26635834 applied
10-FEB-18 04.25.36.182498 AM APPLY SERVER 11.2.0.4 171017 EXA BP171017
23-FEB-18 09.48.21.702054 PM jvmpsu.sql SERVER 11.2.0.4.180116OJVMPSU 0 RAN jvmpsu.sql
23-FEB-18 09.48.21.739476 PM APPLY SERVER 11.2.0.4.180116OJVMPSU 0 OJVM PSU post-install
23-FEB-18 09.48.21.741801 PM APPLY 26925532 Patch 26925532 applied
19-MAY-18 11.40.29.378510 PM jvmpsu.sql SERVER 11.2.0.4.180417OJVMPSU 0 RAN jvmpsu.sql
19-MAY-18 11.40.29.691650 PM APPLY SERVER 11.2.0.4.180417OJVMPSU 0 OJVM PSU post-install
19-MAY-18 11.40.29.694744 PM APPLY 27475598 Patch 27475598 applied
19-AUG-18 12.08.55.766831 AM APPLY SERVER 11.2.0.4 180417 EXA BP180417
19-AUG-18 02.38.28.101436 AM jvmpsu.sql SERVER 11.2.0.4.180717OJVMPSU 0 RAN jvmpsu.sql
19-AUG-18 02.38.28.352391 AM APPLY SERVER 11.2.0.4.180717OJVMPSU 0 OJVM PSU post-install
19-AUG-18 02.38.28.354748 AM APPLY 27923163 Patch 27923163 applied
05-NOV-18 03.42.35.835165 AM jvmpsu.sql SERVER 11.2.0.4.181016OJVMPSU 0 RAN jvmpsu.sql
05-NOV-18 03.42.36.277916 AM APPLY SERVER 11.2.0.4.181016OJVMPSU 0 OJVM PSU post-install
05-NOV-18 03.42.36.280072 AM APPLY 28440700 Patch 28440700 applied
10-NOV-18 06.05.44.941482 AM APPLY SERVER 11.2.0.4 180717 EXA BP180717
31 rows selected.
SQL>
--- Similarly perform on all DBs
Genaral checks:
===============
9. Check DB staus - should be up and running
10. Check Listener staus - should be up and running
---> SCAN listsneteners
---> Local listneters
11. Do clusterware health check on cluster nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm01 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm02 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm02 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
12. Start applications and do post checks
13. Run the post exachk report and upload it back to SR# so that oracle engineer will do post exachk review copaired with Pre-check report
Before Exadata bundle patching we need to do certain prechks and once the patching completed we should do some post checks.
I. Pre Patching Verification Steps.
II. Post Patching Verification Steps.
Please follow below in details steps and execution outputs
I. Pre Patching Verification Steps.
YUM:
====
1. Verify the YUM image status
>dcli -l root -g dbs_group imageinfo
10.89.0.141:
10.89.0.141: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.141: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.141: Image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
10.89.0.141: Image activated: 2018-11-10 01:12:21 -0500
10.89.0.141: Image status: success ===============================================> Should be success
10.89.0.141: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.141:
10.89.0.142:
10.89.0.142: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.142: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.142: Image version: 18.1.7.0.0.180821
10.89.0.142: Image activated: 2018-11-10 01:15:28 -0500
10.89.0.142: Image status: success ===============================================> Should be latest version
10.89.0.142: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.142:
npexdbadm01.r02.xlgs.local:(root)-/root
>
2. Check uptime DB nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g dbs_group uptime
10.89.0.141: 04:45:13 up 2 days, 3:36, 2 users, load average: 5.18, 9.15, 8.69
10.89.0.142: 04:45:13 up 2 days, 1:36, 0 users, load average: 10.34, 6.83, 4.91
npexdbadm01.r02.xlgs.local:(root)-/root
>
CELL:
====
3. Verify the CELL image status
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group imageinfo
npexceladm01:
npexceladm01: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm01: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm01: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm01:
npexceladm01: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm01: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm01: Active image activated: 2018-11-10 04:38:16 -0500
npexceladm01: Active image status: success ===============================================> Should be success
npexceladm01: Active system partition on device: /dev/md6
npexceladm01: Active software partition on device: /dev/md8
npexceladm01:
npexceladm01: Cell boot usb partition: /dev/sdm1
npexceladm01: Cell boot usb version: 18.1.7.0.0.180821
npexceladm01:
npexceladm01: Inactive image version: 18.1.5.0.0.180506
npexceladm01: Inactive image activated: 2018-07-22 05:49:33 -0400
npexceladm01: Inactive image status: success
npexceladm01: Inactive system partition on device: /dev/md5
npexceladm01: Inactive software partition on device: /dev/md7
npexceladm01:
npexceladm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm01: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm01: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm01: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm01: Rollback to the inactive partitions: Possible
npexceladm02:
npexceladm02: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm02: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm02: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm02:
npexceladm02: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm02: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm02: Active image activated: 2018-11-10 04:33:15 -0500
npexceladm02: Active image status: success ===============================================> Should be success
npexceladm02: Active system partition on device: /dev/md6
npexceladm02: Active software partition on device: /dev/md8
npexceladm02:
npexceladm02: Cell boot usb partition: /dev/sdm1
npexceladm02: Cell boot usb version: 18.1.7.0.0.180821
npexceladm02:
npexceladm02: Inactive image version: 18.1.5.0.0.180506
npexceladm02: Inactive image activated: 2018-07-22 05:43:58 -0400
npexceladm02: Inactive image status: success
npexceladm02: Inactive system partition on device: /dev/md5
npexceladm02: Inactive software partition on device: /dev/md7
npexceladm02:
npexceladm02: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm02: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm02: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm02: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm02: Rollback to the inactive partitions: Possible
npexceladm03:
npexceladm03: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm03: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm03: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm03:
npexceladm03: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm03: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm03: Active image activated: 2018-11-10 04:37:24 -0500
npexceladm03: Active image status: success ===============================================> Should be success
npexceladm03: Active system partition on device: /dev/md6
npexceladm03: Active software partition on device: /dev/md8
npexceladm03:
npexceladm03: Cell boot usb partition: /dev/sdm1
npexceladm03: Cell boot usb version: 18.1.7.0.0.180821
npexceladm03:
npexceladm03: Inactive image version: 18.1.5.0.0.180506
npexceladm03: Inactive image activated: 2018-07-22 05:48:16 -0400
npexceladm03: Inactive image status: success
npexceladm03: Inactive system partition on device: /dev/md5
npexceladm03: Inactive software partition on device: /dev/md7
npexceladm03:
npexceladm03: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm03: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm03: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm03: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm03: Rollback to the inactive partitions: Possible
npexceladm04:
npexceladm04: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm04: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm04: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm04:
npexceladm04: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm04: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm04: Active image activated: 2018-11-10 05:34:52 -0500
npexceladm04: Active image status: success ===============================================> Should be success
npexceladm04: Active system partition on device: /dev/md24p5
npexceladm04: Active software partition on device: /dev/md24p7
npexceladm04:
npexceladm04: Cell boot usb partition: /dev/md25p1
npexceladm04: Cell boot usb version: 18.1.7.0.0.180821
npexceladm04:
npexceladm04: Inactive image version: 18.1.5.0.0.180506
npexceladm04: Inactive image activated: 2018-07-22 06:52:06 -0400
npexceladm04: Inactive image status: success
npexceladm04: Inactive system partition on device: /dev/md24p6
npexceladm04: Inactive software partition on device: /dev/md24p8
npexceladm04:
npexceladm04: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm04: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm04: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm04: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm04: Rollback to the inactive partitions: Possible
npexceladm05:
npexceladm05: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm05: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm05: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm05:
npexceladm05: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm05: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm05: Active image activated: 2018-11-10 05:32:22 -0500
npexceladm05: Active image status: success ===============================================> Should be success
npexceladm05: Active system partition on device: /dev/md24p5
npexceladm05: Active software partition on device: /dev/md24p7
npexceladm05:
npexceladm05: Cell boot usb partition: /dev/md25p1
npexceladm05: Cell boot usb version: 18.1.7.0.0.180821
npexceladm05:
npexceladm05: Inactive image version: 18.1.5.0.0.180506
npexceladm05: Inactive image activated: 2018-07-22 06:50:07 -0400
npexceladm05: Inactive image status: success
npexceladm05: Inactive system partition on device: /dev/md24p6
npexceladm05: Inactive software partition on device: /dev/md24p8
npexceladm05:
npexceladm05: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm05: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm05: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm05: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm05: Rollback to the inactive partitions: Possible
npexdbadm01.r02.xlgs.local:(root)-/root
>
4. Check uptime of CELLs
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group uptime
npexceladm01: 03:45:12 up 1 day, 23:12, 0 users, load average: 10.77, 8.09, 5.59
npexceladm02: 03:40:00 up 1 day, 23:12, 0 users, load average: 11.90, 8.49, 5.86
npexceladm03: 03:44:11 up 1 day, 23:12, 0 users, load average: 11.84, 8.96, 6.04
npexceladm04: 04:42:10 up 1 day, 23:11, 0 users, load average: 13.25, 10.07, 6.31
npexceladm05: 04:38:43 up 1 day, 23:10, 0 users, load average: 14.50, 10.90, 6.78
npexdbadm01.r02.xlgs.local:(root)-/root
>
IB Swicthes:
============
5. Check IB switch verison
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst version
npexsw-ibb01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-ibb01: Build time: Mar 2 2018 12:54:44
npexsw-ibb01: SP board info:
npexsw-ibb01: Manufacturing Date: N/A
npexsw-ibb01: Serial Number: "NED620379"
npexsw-ibb01: Hardware Revision: 0x0100
npexsw-ibb01: Firmware Revision: 0x0000
npexsw-ibb01: BIOS version: NUP1R918
npexsw-ibb01: BIOS date: 01/19/2016
npexsw-iba01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-iba01: Build time: Mar 2 2018 12:54:44
npexsw-iba01: SP board info:
npexsw-iba01: Manufacturing Date: N/A
npexsw-iba01: Serial Number: "NED620102"
npexsw-iba01: Hardware Revision: 0x0100
npexsw-iba01: Firmware Revision: 0x0000
npexsw-iba01: BIOS version: NUP1R918
npexsw-iba01: BIOS date: 01/19/2016
npexdbadm01.r02.xlgs.local:(root)-/root
>
6. Check uptime of IB Switches
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst uptime
npexsw-ibb01: 05:29:31 up 112 days, 22:12, 0 users, load average: 0.13, 0.16, 0.15
npexsw-iba01: 04:54:07 up 112 days, 21:29, 0 users, load average: 0.14, 0.13, 0.14
npexdbadm01.r02.xlgs.local:(root)-/root
>
BP:
===
7. Verify the opatch lsinventory
GI home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l grid -c "/u01/app/12.2.0.1/grid/OPatch/opatch lspatches -oh /u01/app/12.2.0.1/grid "'
DB Home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l orauat -c "/u01/app/oracle/product/11.2.0.4/DB1/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB1"'
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l obiuat -c "/u01/app/oracle/product/11.2.0.4/DB2/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB2"'
>
DB Catbudle:
============
8. Login to DB and check for catbundle status:
set pages 1000
set lines 1000
col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
DB1:
========
SQL> set pages 1000
SQL> set lines 1000
SQL> col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
SQL> SQL> SQL> SQL> SQL> SQL>
ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
------------------------------ -------------------- -------------------- ------------------------------ ---------- -------------------- --------------------------------------------------
20-FEB-13 12.51.48.044154 AM VIEW INVALIDATE 8289601 view invalidation
20-FEB-13 12.51.48.080024 AM UPGRADE SERVER 11.2.0.3.0 Upgraded from 11.1.0.7.0
20-FEB-13 11.29.14.796574 AM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
23-JAN-08 04.07.02.182324 PM CPU SERVER 10.2.0.3.0 6079591 CPUJul2007
26-NOV-08 11.39.09.067225 PM UPGRADE SERVER 11.1.0.6.0 Upgraded from 10.2.0.3.0
22-JAN-09 04.34.49.320298 AM UPGRADE SERVER 11.1.0.7.0 Upgraded from 11.1.0.6.0
20-FEB-13 03.26.12.509440 PM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
11-MAR-17 01.18.18.341094 PM VIEW INVALIDATE 8289601 view invalidation
11-MAR-17 01.18.18.375351 PM UPGRADE SERVER 11.2.0.4.0 Upgraded from 11.2.0.3.0
11-MAR-17 01.19.36.166175 PM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0
11-MAR-17 07.34.06.000000 PM APPLY 17501296 Patch 17501296 applied
11-MAR-17 07.37.56.000000 PM APPLY 22731026 Patch 22731026 applied
11-MAY-17 11.04.58.734791 AM APPLY SERVER 11.2.0.4 161018 EXA BP161018
10-FEB-18 04.24.17.618719 AM jvmpsu.sql SERVER 11.2.0.4.171017OJVMPSU 0 RAN jvmpsu.sql
10-FEB-18 04.24.17.651053 AM APPLY SERVER 11.2.0.4.171017OJVMPSU 0 OJVM PSU post-install
10-FEB-18 04.24.17.653268 AM APPLY 26635834 Patch 26635834 applied
10-FEB-18 04.25.36.182498 AM APPLY SERVER 11.2.0.4 171017 EXA BP171017
23-FEB-18 09.48.21.702054 PM jvmpsu.sql SERVER 11.2.0.4.180116OJVMPSU 0 RAN jvmpsu.sql
23-FEB-18 09.48.21.739476 PM APPLY SERVER 11.2.0.4.180116OJVMPSU 0 OJVM PSU post-install
23-FEB-18 09.48.21.741801 PM APPLY 26925532 Patch 26925532 applied
19-MAY-18 11.40.29.378510 PM jvmpsu.sql SERVER 11.2.0.4.180417OJVMPSU 0 RAN jvmpsu.sql
19-MAY-18 11.40.29.691650 PM APPLY SERVER 11.2.0.4.180417OJVMPSU 0 OJVM PSU post-install
19-MAY-18 11.40.29.694744 PM APPLY 27475598 Patch 27475598 applied
19-AUG-18 12.08.55.766831 AM APPLY SERVER 11.2.0.4 180417 EXA BP180417
19-AUG-18 02.38.28.101436 AM jvmpsu.sql SERVER 11.2.0.4.180717OJVMPSU 0 RAN jvmpsu.sql
19-AUG-18 02.38.28.352391 AM APPLY SERVER 11.2.0.4.180717OJVMPSU 0 OJVM PSU post-install
19-AUG-18 02.38.28.354748 AM APPLY 27923163 Patch 27923163 applied
05-NOV-18 03.42.35.835165 AM jvmpsu.sql SERVER 11.2.0.4.181016OJVMPSU 0 RAN jvmpsu.sql
05-NOV-18 03.42.36.277916 AM APPLY SERVER 11.2.0.4.181016OJVMPSU 0 OJVM PSU post-install
05-NOV-18 03.42.36.280072 AM APPLY 28440700 Patch 28440700 applied
10-NOV-18 06.05.44.941482 AM APPLY SERVER 11.2.0.4 180717 EXA BP180717
31 rows selected.
SQL>
--- Similarly perform on all DBs
Genaral checks:
===============
9. Check DB staus - should be up and running
10. Check Listener staus - should be up and running
---> SCAN listsneteners
---> Local listneters
11. Do clusterware health check on cluster nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm01 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm02 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm02 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
12. Verify the Patch asssessment provided by Oracle
13. Veriry the Patch Plan provided by Oracle.
14. Veriry the Prechks logs provided by Oracle.
15. Take root /u01 OS backup before the patching.
16. Run the exachk and verify the report.
17. verify the below OS comand outputs
df -h
lvs
cat /etc/fstab
18. Verify the applciation health check.
II. Post Patching Verification Steps.
YUM:
====
1. Verify the YUM image status
>dcli -l root -g dbs_group imageinfo
10.89.0.141:
10.89.0.141: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.141: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.141: Image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
10.89.0.141: Image activated: 2018-11-10 01:12:21 -0500
10.89.0.141: Image status: success ===============================================> Should be success
10.89.0.141: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.141:
10.89.0.142:
10.89.0.142: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
10.89.0.142: Image kernel version: 4.1.12-94.8.4.el6uek
10.89.0.142: Image version: 18.1.7.0.0.180821
10.89.0.142: Image activated: 2018-11-10 01:15:28 -0500
10.89.0.142: Image status: success ===============================================> Should be latest version
10.89.0.142: System partition on device: /dev/mapper/VGExaDb-LVDbSys1
10.89.0.142:
npexdbadm01.r02.xlgs.local:(root)-/root
>
2. Check uptime DB nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g dbs_group uptime
10.89.0.141: 04:45:13 up 2 days, 3:36, 2 users, load average: 5.18, 9.15, 8.69
10.89.0.142: 04:45:13 up 2 days, 1:36, 0 users, load average: 10.34, 6.83, 4.91
npexdbadm01.r02.xlgs.local:(root)-/root
>
CELL:
====
3. Verify the CELL image status
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group imageinfo
npexceladm01:
npexceladm01: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm01: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm01: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm01:
npexceladm01: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm01: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm01: Active image activated: 2018-11-10 04:38:16 -0500
npexceladm01: Active image status: success ===============================================> Should be success
npexceladm01: Active system partition on device: /dev/md6
npexceladm01: Active software partition on device: /dev/md8
npexceladm01:
npexceladm01: Cell boot usb partition: /dev/sdm1
npexceladm01: Cell boot usb version: 18.1.7.0.0.180821
npexceladm01:
npexceladm01: Inactive image version: 18.1.5.0.0.180506
npexceladm01: Inactive image activated: 2018-07-22 05:49:33 -0400
npexceladm01: Inactive image status: success
npexceladm01: Inactive system partition on device: /dev/md5
npexceladm01: Inactive software partition on device: /dev/md7
npexceladm01:
npexceladm01: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm01: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm01: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm01: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm01: Rollback to the inactive partitions: Possible
npexceladm02:
npexceladm02: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm02: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm02: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm02:
npexceladm02: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm02: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm02: Active image activated: 2018-11-10 04:33:15 -0500
npexceladm02: Active image status: success ===============================================> Should be success
npexceladm02: Active system partition on device: /dev/md6
npexceladm02: Active software partition on device: /dev/md8
npexceladm02:
npexceladm02: Cell boot usb partition: /dev/sdm1
npexceladm02: Cell boot usb version: 18.1.7.0.0.180821
npexceladm02:
npexceladm02: Inactive image version: 18.1.5.0.0.180506
npexceladm02: Inactive image activated: 2018-07-22 05:43:58 -0400
npexceladm02: Inactive image status: success
npexceladm02: Inactive system partition on device: /dev/md5
npexceladm02: Inactive software partition on device: /dev/md7
npexceladm02:
npexceladm02: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm02: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm02: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm02: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm02: Rollback to the inactive partitions: Possible
npexceladm03:
npexceladm03: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm03: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm03: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm03:
npexceladm03: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm03: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm03: Active image activated: 2018-11-10 04:37:24 -0500
npexceladm03: Active image status: success ===============================================> Should be success
npexceladm03: Active system partition on device: /dev/md6
npexceladm03: Active software partition on device: /dev/md8
npexceladm03:
npexceladm03: Cell boot usb partition: /dev/sdm1
npexceladm03: Cell boot usb version: 18.1.7.0.0.180821
npexceladm03:
npexceladm03: Inactive image version: 18.1.5.0.0.180506
npexceladm03: Inactive image activated: 2018-07-22 05:48:16 -0400
npexceladm03: Inactive image status: success
npexceladm03: Inactive system partition on device: /dev/md5
npexceladm03: Inactive software partition on device: /dev/md7
npexceladm03:
npexceladm03: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm03: Inactive grub config for the rollback: /boot/grub2/grub.cfg.inactive
npexceladm03: Inactive usb grub config for the rollback: /boot/grub2/grub.cfg.usb.inactive
npexceladm03: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm03: Rollback to the inactive partitions: Possible
npexceladm04:
npexceladm04: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm04: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm04: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm04:
npexceladm04: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm04: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm04: Active image activated: 2018-11-10 05:34:52 -0500
npexceladm04: Active image status: success ===============================================> Should be success
npexceladm04: Active system partition on device: /dev/md24p5
npexceladm04: Active software partition on device: /dev/md24p7
npexceladm04:
npexceladm04: Cell boot usb partition: /dev/md25p1
npexceladm04: Cell boot usb version: 18.1.7.0.0.180821
npexceladm04:
npexceladm04: Inactive image version: 18.1.5.0.0.180506
npexceladm04: Inactive image activated: 2018-07-22 06:52:06 -0400
npexceladm04: Inactive image status: success
npexceladm04: Inactive system partition on device: /dev/md24p6
npexceladm04: Inactive software partition on device: /dev/md24p8
npexceladm04:
npexceladm04: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm04: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm04: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm04: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm04: Rollback to the inactive partitions: Possible
npexceladm05:
npexceladm05: Kernel version: 4.1.12-94.8.4.el6uek.x86_64 #2 SMP Sat May 5 16:14:51 PDT 2018 x86_64
npexceladm05: Cell version: OSS_18.1.7.0.0AUG_LINUX.X64_180821
npexceladm05: Cell rpm version: cell-18.1.7.0.0_LINUX.X64_180821-1.x86_64
npexceladm05:
npexceladm05: Active image version: 18.1.7.0.0.180821 ===============================================> Should be latest version
npexceladm05: Active image kernel version: 4.1.12-94.8.4.el6uek
npexceladm05: Active image activated: 2018-11-10 05:32:22 -0500
npexceladm05: Active image status: success ===============================================> Should be success
npexceladm05: Active system partition on device: /dev/md24p5
npexceladm05: Active software partition on device: /dev/md24p7
npexceladm05:
npexceladm05: Cell boot usb partition: /dev/md25p1
npexceladm05: Cell boot usb version: 18.1.7.0.0.180821
npexceladm05:
npexceladm05: Inactive image version: 18.1.5.0.0.180506
npexceladm05: Inactive image activated: 2018-07-22 06:50:07 -0400
npexceladm05: Inactive image status: success
npexceladm05: Inactive system partition on device: /dev/md24p6
npexceladm05: Inactive software partition on device: /dev/md24p8
npexceladm05:
npexceladm05: Inactive marker for the rollback: /boot/I_am_hd_boot.inactive
npexceladm05: Inactive grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.inactive
npexceladm05: Inactive usb grub config for the rollback: /boot/efi/EFI/redhat/grub.cfg.usb.inactive
npexceladm05: Inactive kernel version for the rollback: 4.1.12-94.8.4.el6uek.x86_64
npexceladm05: Rollback to the inactive partitions: Possible
npexdbadm01.r02.xlgs.local:(root)-/root
>
4. Check uptime of CELLs
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g cell_group uptime
npexceladm01: 03:45:12 up 1 day, 23:12, 0 users, load average: 10.77, 8.09, 5.59
npexceladm02: 03:40:00 up 1 day, 23:12, 0 users, load average: 11.90, 8.49, 5.86
npexceladm03: 03:44:11 up 1 day, 23:12, 0 users, load average: 11.84, 8.96, 6.04
npexceladm04: 04:42:10 up 1 day, 23:11, 0 users, load average: 13.25, 10.07, 6.31
npexceladm05: 04:38:43 up 1 day, 23:10, 0 users, load average: 14.50, 10.90, 6.78
npexdbadm01.r02.xlgs.local:(root)-/root
>
IB Swicthes:
============
5. Check IB switch verison
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst version
npexsw-ibb01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-ibb01: Build time: Mar 2 2018 12:54:44
npexsw-ibb01: SP board info:
npexsw-ibb01: Manufacturing Date: N/A
npexsw-ibb01: Serial Number: "NED620379"
npexsw-ibb01: Hardware Revision: 0x0100
npexsw-ibb01: Firmware Revision: 0x0000
npexsw-ibb01: BIOS version: NUP1R918
npexsw-ibb01: BIOS date: 01/19/2016
npexsw-iba01: SUN DCS 36p version: 2.2.9-3 ===============================================> Should be latest version
npexsw-iba01: Build time: Mar 2 2018 12:54:44
npexsw-iba01: SP board info:
npexsw-iba01: Manufacturing Date: N/A
npexsw-iba01: Serial Number: "NED620102"
npexsw-iba01: Hardware Revision: 0x0100
npexsw-iba01: Firmware Revision: 0x0000
npexsw-iba01: BIOS version: NUP1R918
npexsw-iba01: BIOS date: 01/19/2016
npexdbadm01.r02.xlgs.local:(root)-/root
>
6. Check uptime of IB Switches
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g ibswitches.lst uptime
npexsw-ibb01: 05:29:31 up 112 days, 22:12, 0 users, load average: 0.13, 0.16, 0.15
npexsw-iba01: 04:54:07 up 112 days, 21:29, 0 users, load average: 0.14, 0.13, 0.14
npexdbadm01.r02.xlgs.local:(root)-/root
>
BP:
===
7. Verify the opatch lsinventory
GI home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l grid -c "/u01/app/12.2.0.1/grid/OPatch/opatch lspatches -oh /u01/app/12.2.0.1/grid "'
DB Home:
========
npexdbadm01.r02.xlgs.local:(root)-/root
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l orauat -c "/u01/app/oracle/product/11.2.0.4/DB1/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB1"'
>dcli -l root -g /opt/oracle.SupportTools/onecommand/dbs_group 'su -l obiuat -c "/u01/app/oracle/product/11.2.0.4/DB2/OPatch/opatch lspatches -oh /u01/app/oracle/product/11.2.0.4/DB2"'
>
DB Catbudle:
============
8. Login to DB and check for catbundle status:
set pages 1000
set lines 1000
col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
DB1:
========
SQL> set pages 1000
SQL> set lines 1000
SQL> col ACTION_TIME for a30
col ACTION for a20
col NAMESPACE for a20
col VERSION for a30
col BUNDLE_SERIES for a20
col COMMENTS for a50
select * from dba_registry_history;
SQL> SQL> SQL> SQL> SQL> SQL>
ACTION_TIME ACTION NAMESPACE VERSION ID BUNDLE_SERIES COMMENTS
------------------------------ -------------------- -------------------- ------------------------------ ---------- -------------------- --------------------------------------------------
20-FEB-13 12.51.48.044154 AM VIEW INVALIDATE 8289601 view invalidation
20-FEB-13 12.51.48.080024 AM UPGRADE SERVER 11.2.0.3.0 Upgraded from 11.1.0.7.0
20-FEB-13 11.29.14.796574 AM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
23-JAN-08 04.07.02.182324 PM CPU SERVER 10.2.0.3.0 6079591 CPUJul2007
26-NOV-08 11.39.09.067225 PM UPGRADE SERVER 11.1.0.6.0 Upgraded from 10.2.0.3.0
22-JAN-09 04.34.49.320298 AM UPGRADE SERVER 11.1.0.7.0 Upgraded from 11.1.0.6.0
20-FEB-13 03.26.12.509440 PM APPLY SERVER 11.2.0.3 5 PSU PSU 11.2.0.3.5
11-MAR-17 01.18.18.341094 PM VIEW INVALIDATE 8289601 view invalidation
11-MAR-17 01.18.18.375351 PM UPGRADE SERVER 11.2.0.4.0 Upgraded from 11.2.0.3.0
11-MAR-17 01.19.36.166175 PM APPLY SERVER 11.2.0.4 0 PSU Patchset 11.2.0.2.0
11-MAR-17 07.34.06.000000 PM APPLY 17501296 Patch 17501296 applied
11-MAR-17 07.37.56.000000 PM APPLY 22731026 Patch 22731026 applied
11-MAY-17 11.04.58.734791 AM APPLY SERVER 11.2.0.4 161018 EXA BP161018
10-FEB-18 04.24.17.618719 AM jvmpsu.sql SERVER 11.2.0.4.171017OJVMPSU 0 RAN jvmpsu.sql
10-FEB-18 04.24.17.651053 AM APPLY SERVER 11.2.0.4.171017OJVMPSU 0 OJVM PSU post-install
10-FEB-18 04.24.17.653268 AM APPLY 26635834 Patch 26635834 applied
10-FEB-18 04.25.36.182498 AM APPLY SERVER 11.2.0.4 171017 EXA BP171017
23-FEB-18 09.48.21.702054 PM jvmpsu.sql SERVER 11.2.0.4.180116OJVMPSU 0 RAN jvmpsu.sql
23-FEB-18 09.48.21.739476 PM APPLY SERVER 11.2.0.4.180116OJVMPSU 0 OJVM PSU post-install
23-FEB-18 09.48.21.741801 PM APPLY 26925532 Patch 26925532 applied
19-MAY-18 11.40.29.378510 PM jvmpsu.sql SERVER 11.2.0.4.180417OJVMPSU 0 RAN jvmpsu.sql
19-MAY-18 11.40.29.691650 PM APPLY SERVER 11.2.0.4.180417OJVMPSU 0 OJVM PSU post-install
19-MAY-18 11.40.29.694744 PM APPLY 27475598 Patch 27475598 applied
19-AUG-18 12.08.55.766831 AM APPLY SERVER 11.2.0.4 180417 EXA BP180417
19-AUG-18 02.38.28.101436 AM jvmpsu.sql SERVER 11.2.0.4.180717OJVMPSU 0 RAN jvmpsu.sql
19-AUG-18 02.38.28.352391 AM APPLY SERVER 11.2.0.4.180717OJVMPSU 0 OJVM PSU post-install
19-AUG-18 02.38.28.354748 AM APPLY 27923163 Patch 27923163 applied
05-NOV-18 03.42.35.835165 AM jvmpsu.sql SERVER 11.2.0.4.181016OJVMPSU 0 RAN jvmpsu.sql
05-NOV-18 03.42.36.277916 AM APPLY SERVER 11.2.0.4.181016OJVMPSU 0 OJVM PSU post-install
05-NOV-18 03.42.36.280072 AM APPLY 28440700 Patch 28440700 applied
10-NOV-18 06.05.44.941482 AM APPLY SERVER 11.2.0.4 180717 EXA BP180717
31 rows selected.
SQL>
--- Similarly perform on all DBs
Genaral checks:
===============
9. Check DB staus - should be up and running
10. Check Listener staus - should be up and running
---> SCAN listsneteners
---> Local listneters
11. Do clusterware health check on cluster nodes
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm01 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm01 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm01 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm01 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm01.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm01.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t -init
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.asm
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crf
1 ONLINE ONLINE npexdbadm02 STABLE
ora.crsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.cssdmonitor
1 ONLINE ONLINE npexdbadm02 STABLE
ora.ctssd
1 ONLINE ONLINE npexdbadm02 OBSERVER,STABLE
ora.diskmon
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.acfs
1 ONLINE ONLINE npexdbadm02 STABLE
ora.drivers.oka
1 OFFLINE OFFLINE STABLE
ora.evmd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gipcd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.gpnpd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.mdnsd
1 ONLINE ONLINE npexdbadm02 STABLE
ora.storage
1 ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
npexdbadm02.r02.xlgs.local:(root)-/root
>crsctl stat res -t
--------------------------------------------------------------------------------
Name Target State Server State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFS_DG.ACFS_VOL1.advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ACFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ASMNET1LSNR_ASM.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DATAC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.DBFS_DG.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_EBSXDEV.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_IB.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_DB2.lsnr
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.RECOC1.dg
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.acfs_dg.acfs_vol1.acfs
ONLINE ONLINE npexdbadm01 mounted on /acfsmoun
ts/acfs1,STABLE
ONLINE ONLINE npexdbadm02 mounted on /acfsmoun
ts/acfs1,STABLE
ora.net1.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.net2.network
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.ons
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
ora.proxy_advm
ONLINE ONLINE npexdbadm01 STABLE
ONLINE ONLINE npexdbadm02 STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE npexdbadm02 STABLE
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE npexdbadm01 STABLE
ora.asm
1 ONLINE ONLINE npexdbadm01 Started,STABLE
2 ONLINE ONLINE npexdbadm02 Started,STABLE
ora.cvu
1 ONLINE ONLINE npexdbadm01 STABLE
ora.ebsbuat.db
1 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.ebsdba.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSDBA,STABLE
ora.ebsfdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSFDEV,STABLE
ora.ebsgold.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSGOLD,STABLE
ora.ebstdev.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Abnormal Termination
,HOME=/u01/app/oracl
e/product/11.2.0.4/E
BSTDEV,STABLE
ora.ebsxdev.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/EBSXDEV,STABLE
ora.DB1.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB1,STABLE
ora.npexdbadm01.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm01_2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.npexdbadm02.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.npexdbadm02_2.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.obipuat.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/OBIPUAT,STABLE
ora.obixaos.db
1 OFFLINE OFFLINE STABLE
2 OFFLINE OFFLINE Instance Shutdown,ST
ABLE
ora.DB2.db
1 ONLINE ONLINE npexdbadm01 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
2 ONLINE ONLINE npexdbadm02 Open,HOME=/u01/app/o
racle/product/11.2.0
.4/DB2,STABLE
ora.qosmserver
1 OFFLINE OFFLINE STABLE
ora.scan1.vip
1 ONLINE ONLINE npexdbadm02 STABLE
ora.scan2.vip
1 ONLINE ONLINE npexdbadm01 STABLE
ora.scan3.vip
1 ONLINE ONLINE npexdbadm01 STABLE
--------------------------------------------------------------------------------
npexdbadm02.r02.xlgs.local:(root)-/root
>
12. Start applications and do post checks
13. Run the post exachk report and upload it back to SR# so that oracle engineer will do post exachk review copaired with Pre-check report
Regards,
Mallik
No comments:
Post a Comment