Database RAC Dataguard RMAN Performance Tuning Feedback
Database Migration & Upgradation
DataGuard
ASM & RAC
Exadata
RMAN Backup & Recovery
Oracle Cloud AWS Azure Google Cloud
DevOps Ansible Kubernetes Terraform
Shell Scripting Powershell Scripting
Infomratica Others

Monday, September 27, 2021

sqlplus and rman and other binary executable are not working with error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory

Issue:

sqlplus and rman and other binary executable are not working with error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory

 
Error Message:

/orasw/oracle/product/12.1.0.2/db/bin/sqlplus: error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory
/orasw/oracle/product/12.1.0.2/db/bin/rman: error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory

[oracle@oracledb lib]$ sqlplus / as sysdba
sqlplus: error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory

[oracle@oracledb lib]$ rman target /
rman: error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory
[oracle@oracledb lib]$

Cause:

After Inverstigating found that the shared object file libclntshcore.so.12.1 was missing from the lib(/orasw/oracle/product/12.1.0.2/db/lib) location.

Solution:

[oracle@oracledb lib]$ pwd
/u01/app/oracle/product/12.2.0.1/dbhome_1/lib
[oracle@oracledb lib]$ ls -lrtr libclntshcore.so.12.1
ls: cannot access libclntshcore.so.12.1: No such file or directory
[oracle@oracledb lib]$

Copy from the existing Oracle Home and verify the file exist and has proper permission

[oracle@oracledb lib]$ ls -lrtr libclntshcore.so.12.1
-rwxr-xr-x. 1 oracle oinstall 8032696 Sep 27 19:28 libclntshcore.so.12.1
[oracle@oracledb lib]$

[oracle@oracledb lib]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Sep 27 19:29:18 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
[oracle@oracledb lib]$ rman target /

Recovery Manager: Release 12.2.0.1.0 - Production on Mon Sep 27 19:29:24 2021

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ORA12C (DBID=457334020)

RMAN>

Recovery Manager complete.
[oracle@oracledb lib]$

Regards,
Mallik
Posted by at No comments:

Friday, September 24, 2021

DBCA fails with errors: [FATAL] [DBT-50000] Unable to check for available memory

DBCA fails with errors: [FATAL] [DBT-50000] Unable to check for available memory


Recently I have installed Oracle 19c Database binary to windows machine. And after that I started DBCA to create database.

In “Configuration options” steps I got memory error. i.e. [DBT-50000] Unable to check for available memory

Earlier I selected 8GB for my SGA but after this error I had selected 2GB only. But still was facing this issue.

Here is the screenshot for the same.
 


After searching oracle docs for known issue, I found, it’s a bug. One can find details about this bug on following oracle notes.

DBCA fails with errors: [FATAL] [DBT-50000] Unable to check for available memory in “Specify configuration option” (Doc ID 2631718.1)

Workaround for this issue is,

Run dbca with “-J-Doracle.assistants.dbca.validate.ConfigurationParams=false” like a bellow command,

Open CMD (Run as Administrator)
Run database configuration assistant with the following command
dbca -J-Doracle.assistants.dbca.validate.ConfigurationParams=false

Regards,
Mallik
Posted by at 1 comment:

Monday, September 20, 2021

19c Oracle DBA Training || 30-Sep-2021 New Batch || Course Details & Demo Classes



Oracle Database Training!!! - 19c Oracle Database Training!!!

New DBA batch is starting on 30-Sep-2021. Don't forget to attend Demo classes on 30-Sep-2021 & 01-Oct-2021. 

Ping me on my WhatsApp Number +91 9880616848 for registration & more details.

Attend Demo Classes using below zoom Meeting:

https://us02web.zoom.us/j/86254664197?pwd=QVVCdEdVVlRoREQ2RGdYQUZyVVNFUT09

Timing: 

Sep 30, 2021 09:00 AM IST to 11:00 AM IST
Oct 01, 2021 09:00 AM IST to 11:00 AM IST

Check out this video for more details on course content:

https://youtu.be/BANrqNLiZqo

Check the below link for Course details:

https://mallik034.blogspot.com/p/database-training.html

Check the below link for Couse registration and Timing and fees details:

https://mallik034.blogspot.com/p/upcoming-batch-schedule.html

#19c #DBA #oracle #database #training #course #announcement #CareerOpportunities

Regards,
Mallik



Posted by at No comments:

Wednesday, September 1, 2021

Unable to Start Database Listener Due to Wrong VIP On /etc/hosts file

Unable to Start Database Listener Due to Wrong VIP On /etc/hosts file


1. When We tried to start the Database Listeber which was not starting on node1 and reporting below error message:

[oracle@oraclenode1 admin]$ lsnrctl start LISTENER_RACSB

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 01-SEP-2021 02:37:44

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/12.2.0.1/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oraclenode1/listener_racsb/alert/log.xml
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oraclenode1-vip.localdomain.com)(PORT=1523)))
TNS-12545: Connect failed because target host or object does not exist
 TNS-12560: TNS:protocol adapter error
  TNS-00515: Connect failed because target host or object does not exist
   Linux Error: 99: Cannot assign requested address

Listener failed to start. See the error message(s) above...
[oracle@oraclenode1 admin]$

2. When I checked on /etc/hosts file where node1 VIP was wrongly mentioned (10.38.4.168)

[root@oraclenode1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

############################################################################################
# Public
10.38.4.110 oraclenode1.localdomain.com oraclenode1
10.38.4.115 oraclenode2.localdomain.com oraclenode2

# Private
10.38.9.110 oraclenode1-priv.localdomain.com oraclenode1-priv
10.38.9.115 oraclenode2-priv.localdomain.com oraclenode2-priv

# Virtual
10.38.4.168 oraclenode1-vip.localdomain.com oraclenode1-vip
10.38.4.170 oraclenode2-vip.localdomain.com oraclenode2-vip

# SCAN
10.38.4.126 orclscan.localdomain.com orclscan
10.38.4.154 orclscan.localdomain.com orclscan
10.38.4.162 orclscan.localdomain.com orclscan
[root@oraclenode1 ~]#

3. When I checked node VIP using srvctl config found that correct VIP was 10.38.4.138

[root@oraclenode1 named]# srvctl config  vip -n oraclenode1
VIP exists: network number 1, hosting node oraclenode1
VIP IPv4 Address: 10.38.4.138
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
[root@oraclenode1 named]#

4. I have modified the /etc/hosts file and updated the correct VIP for node1

[root@oraclenode1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

############################################################################################
# Public
10.38.4.110 oraclenode1.localdomain.com oraclenode1
10.38.4.115 oraclenode2.localdomain.com oraclenode2

# Private
10.38.9.110 oraclenode1-priv.localdomain.com oraclenode1-priv
10.38.9.115 oraclenode2-priv.localdomain.com oraclenode2-priv

# Virtual
10.38.4.138 oraclenode1-vip.localdomain.com oraclenode1-vip
10.38.4.170 oraclenode2-vip.localdomain.com oraclenode2-vip

# SCAN
10.38.4.126 orclscan.localdomain.com orclscan
10.38.4.154 orclscan.localdomain.com orclscan
#10.38.4.165 orclscan.localdomain.com orclscan
10.38.4.162 orclscan.localdomain.com orclscan
[root@oraclenode1 ~]#

5. I was able to start the Database listener successfully

[oracle@oraclenode1 ~]$ cd /u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/
[oracle@oraclenode1 admin]$ lsnrctl start LISTENER_RACSB

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 01-SEP-2021 02:41:26

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Starting /u01/app/oracle/product/12.2.0.1/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 12.2.0.1.0 - Production
System parameter file is /u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/oraclenode1/listener_racsb/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.38.4.138)(PORT=1523)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=oraclenode1-vip.localdomain.com)(PORT=1523)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_RACSB
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                01-SEP-2021 02:41:26
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/product/12.2.0.1/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/oraclenode1/listener_racsb/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.38.4.138)(PORT=1523)))
Services Summary...
Service "RACSB" has 1 instance(s).
  Instance "RACSB", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@oraclenode1 admin]$

Regards,
Mallik

Posted by at No comments:

Modifying SCAN VIP In a RAC Cluster

How to change or modify or update new the SCAN VIP in a cluster?


1. Check cluster resource where you see ora.LISTENER.lsnr is OFFLINE and ora.LISTENER_SCAN3.lsnr is OFFLINE

[root@oraclenode1 named]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  OFFLINE      oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.chad
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.net1.network
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.ons
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  OFFLINE                               STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              Started,STABLE
      2        ONLINE  ONLINE       oraclenode2              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.oraclenode1.vip
      1        ONLINE  OFFLINE                               STABLE
ora.oraclenode2.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan3.vip
      1        ONLINE  OFFLINE                               STABLE
--------------------------------------------------------------------------------
[root@oraclenode1 named]# 

2. Try to start the ora.LISTENER_SCAN3.lsnr manually which is fail with below error message

[root@oraclenode1 named]# srvctl start scan_listener -scannumber 3
PRCR-1079 : Failed to start resource ora.LISTENER_SCAN3.lsnr
CRS-5005: IP Address: 10.38.4.165 is already in use in the network
CRS-2674: Start of 'ora.scan3.vip' on 'oraclenode1' failed
CRS-5005: IP Address: 10.38.4.165 is already in use in the network
CRS-2674: Start of 'ora.scan3.vip' on 'oraclenode2' failed
CRS-2632: There are no more servers to try to place resource 'ora.LISTENER_SCAN3.lsnr' on that would satisfy its placement policy
[root@oraclenode1 named]# 

3. Verify the scan status where it id indicates scan3 VIP is no running 

[root@oraclenode1 named]# srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is running on node oraclenode1
SCAN VIP scan2 is enabled
SCAN VIP scan2 is running on node oraclenode2
SCAN VIP scan3 is enabled
SCAN VIP scan3 is not running

4. Stop scan_listener & scan as shown below 

[root@oraclenode1 named]# srvctl stop scan_listener
[root@oraclenode1 named]# srvctl stop scan
[root@oraclenode1 named]# srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is not running
SCAN VIP scan2 is enabled
SCAN VIP scan2 is not running
SCAN VIP scan3 is enabled
SCAN VIP scan3 is not running

5. Verify the scan name which is configured or resolved by 3 IPs as shown below 

[root@oraclenode1 named]# srvctl config scan
SCAN name: orclscan, Network: 1
Subnet IPv4: 10.38.4.0/255.255.252.0/ens192, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 10.38.4.126
SCAN VIP is enabled.
SCAN 2 IPv4 VIP: 10.38.4.154
SCAN VIP is enabled.
SCAN 3 IPv4 VIP: 10.38.4.165
SCAN VIP is enabled.
[root@oraclenode1 named]# 

6. After consulting with Network team if is found out that one of the scan IP is reserved for some other server.

>>> Network team has replaced one of the scan IP with different IP (In my case scan IP 10.38.4.165 was replaced with 10.38.4.162)

Old SCAN:
[root@oraclenode1 ~]# nslookup orclscan.localdomain.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   orclscan.localdomain.com
Address: 10.38.4.154
Name:   orclscan.localdomain.com
Address: 10.38.4.126
Name:   orclscan.localdomain.com
Address: 10.38.4.165
[root@oraclenode1 ~]#

New SCAN:
[root@oraclenode1 ~]# nslookup orclscan.localdomain.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Name:   orclscan.localdomain.com
Address: 10.38.4.154
Name:   orclscan.localdomain.com
Address: 10.38.4.126
Name:   orclscan.localdomain.com
Address: 10.38.4.162
[root@oraclenode1 ~]#

7. Modify the scan configuration to reflect the new scan IP

[root@oraclenode1 named]# srvctl modify scan -scanname orclscan

[root@oraclenode1 named]# srvctl config scan
SCAN name: orclscan, Network: 1
Subnet IPv4: 10.38.4.0/255.255.252.0/ens192, static
Subnet IPv6:
SCAN 1 IPv4 VIP: 10.38.4.126
SCAN VIP is enabled.
SCAN 2 IPv4 VIP: 10.38.4.154
SCAN VIP is enabled.
SCAN 3 IPv4 VIP: 10.38.4.162
SCAN VIP is enabled.
[root@oraclenode1 named]#

8. Start the SCAN, SCAN listener and LISTENER

[root@oraclenode1 named]# srvctl start scan
[root@oraclenode1 named]# srvctl start scan_listener -scannumber 3
[root@oraclenode1 named]# srvctl start listener -l LISTENER

[root@oraclenode1 named]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  OFFLINE      oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.chad
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.net1.network
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.ons
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              Started,STABLE
      2        ONLINE  ONLINE       oraclenode2              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.oraclenode1.vip
      1        ONLINE  OFFLINE                               STABLE
ora.oraclenode2.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
--------------------------------------------------------------------------------
[root@oraclenode1 named]#

Regards,
Mallik
Posted by at No comments:

Modifying Node VIP In a RAC Cluster

How to update the new Node VIP on a cluster node?


1. Verify the cluster resource where it reported ora.LISTENER.lsnr is on OFFLINE on node1

[root@oraclenode1 named]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  OFFLINE      oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.chad
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.net1.network
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.ons
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              Started,STABLE
      2        ONLINE  ONLINE       oraclenode2              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.oraclenode1.vip
      1        ONLINE  OFFLINE                               STABLE
ora.oraclenode2.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
--------------------------------------------------------------------------------
[root@oraclenode1 named]# 

2. Tried starting the LISTENER which got failed to start on node1 and reported node1 VIP is already in use

[root@oraclenode1 named]# srvctl start listener -l LISTENER
PRCR-1079 : Failed to start resource ora.LISTENER.lsnr
CRS-5005: IP Address: 10.38.4.168 is already in use in the network
CRS-2674: Start of 'ora.oraclenode1.vip' on 'oraclenode1' failed
[root@oraclenode1 named]#

3.Verify the node1 VIP configuration and find that node1 has assigned with VIP 10.38.4.168
[root@oraclenode1 named]# srvctl config nodeapps -n oraclenode1
PRKO-2207 : Warning:-node option has been deprecated and will be ignored.
Network 1 exists
Subnet IPv4: 10.38.4.0/255.255.252.0/ens192, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node oraclenode1
VIP Name: oraclenode1-vip.localdomain.com
VIP IPv4 Address: 10.38.4.168
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL true
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
[root@oraclenode1 named]#

[root@oraclenode1 named]# srvctl config  vip -n oraclenode1
VIP exists: network number 1, hosting node oraclenode1
VIP Name: oraclenode1-vip.localdomain.com
VIP IPv4 Address: 10.38.4.168
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
[root@oraclenode1 named]# 

4. After consulting with network team got to know that node1 VIP(10.38.4.168) is reserved for some other server and network team has assigned the new VIP(10.38.4.138) for node1

Verify that newly assigned IP is not used anywhere in the network:

[root@oraclenode1 named]# ping 10.38.4.138
PING 10.38.4.138 (10.38.4.138) 56(84) bytes of data.
^C
--- 10.38.4.138 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
[root@oraclenode1 named]#

5. Stop the VIP on node1 if not already stopped   
[root@oraclenode1 named]# srvctl stop  vip -n oraclenode1
PRKO-3250 : Failed to stop VIPs on node oraclenode1 because no VIP was running on the given node
[root@oraclenode1 named]#

6. Verify on the server ifconfig output this VIP is not running/assigned to any inetrface or VLAN 
[root@oraclenode1 named]# ifconfig
ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.38.4.110  netmask 255.255.252.0  broadcast 10.38.7.255
        inet6 fe80::9dc9:2ec6:523a:9200  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9f:53:e8  txqueuelen 1000  (Ethernet)
        RX packets 125750  bytes 12533560 (11.9 MiB)
        RX errors 0  dropped 72  overruns 0  frame 0
        TX packets 21033  bytes 7756095 (7.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.38.4.162  netmask 255.255.252.0  broadcast 10.38.7.255
        ether 00:50:56:9f:53:e8  txqueuelen 1000  (Ethernet)

ens192:2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.38.4.154  netmask 255.255.252.0  broadcast 10.38.7.255
        ether 00:50:56:9f:53:e8  txqueuelen 1000  (Ethernet)

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.38.9.110  netmask 255.255.255.0  broadcast 10.38.9.255
        inet6 fe80::300e:be6a:d7ae:a0b8  prefixlen 64  scopeid 0x20<link>
        ether 00:50:56:9f:87:8c  txqueuelen 1000  (Ethernet)
        RX packets 336280  bytes 362673837 (345.8 MiB)
        RX errors 0  dropped 13  overruns 0  frame 0
        TX packets 224204  bytes 186974979 (178.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens224:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 169.254.21.65  netmask 255.255.224.0  broadcast 169.254.31.255
        ether 00:50:56:9f:87:8c  txqueuelen 1000  (Ethernet)

ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:50:56:9f:6c:01  txqueuelen 1000  (Ethernet)
        RX packets 2  bytes 120 (120.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30  bytes 3420 (3.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 49235  bytes 170353234 (162.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 49235  bytes 170353234 (162.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:f6:a4:0b  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@oraclenode1 named]#

7. Configure the new VIP on node1 using srvctl nodeapps as shown below which will fails at first attempt due to wrong subnet used. 


In second attempt we have corrected the correct subnet and which got added to cluster.

First Attempt:
[root@oraclenode1 named]# srvctl modify nodeapps -n oraclenode1 -A 10.38.4.138/255.255.255.0/ens192
PRCN-3024 : Mismatch between the subnet mask of the VIP (255.255.255.0) and that of the associated network (255.255.252.0)
[root@oraclenode1 named]# 

Second Atempet:
[root@oraclenode1 named]# srvctl modify nodeapps -n oraclenode1 -A 10.38.4.138/255.255.252.0/ens192
[root@oraclenode1 named]# srvctl config  vip -n oraclenode1
VIP exists: network number 1, hosting node oraclenode1
VIP IPv4 Address: 10.38.4.138
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
[root@oraclenode1 named]#

[root@oraclenode1 named]# srvctl config nodeapps -n oraclenode1
PRKO-2207 : Warning:-node option has been deprecated and will be ignored.
Network 1 exists
Subnet IPv4: 10.38.4.0/255.255.252.0/ens192, static
Subnet IPv6:
Ping Targets:
Network is enabled
Network is individually enabled on nodes:
Network is individually disabled on nodes:
VIP exists: network number 1, hosting node oraclenode1
VIP IPv4 Address: 10.38.4.138
VIP IPv6 Address:
VIP is enabled.
VIP is individually enabled on nodes:
VIP is individually disabled on nodes:
ONS exists: Local port 6100, remote port 6200, EM port 2016, Uses SSL true
ONS is enabled
ONS is individually enabled on nodes:
ONS is individually disabled on nodes:
[root@oraclenode1 named]#

8. Finally we can start the LISTENER 

[root@oraclenode1 named]# srvctl status listener -l LISTENER
Listener LISTENER is enabled
Listener LISTENER is running on node(s): oraclenode1,oraclenode2
[root@oraclenode1 named]#

[root@oraclenode1 named]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.LISTENER.lsnr
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.chad
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.net1.network
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
ora.ons
               ONLINE  ONLINE       oraclenode1              STABLE
               ONLINE  ONLINE       oraclenode2              STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.DATA.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.RECO.dg(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asm(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              Started,STABLE
      2        ONLINE  ONLINE       oraclenode2              Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.asmnet1.asmnetwork(ora.asmgroup)
      1        ONLINE  ONLINE       oraclenode1              STABLE
      2        ONLINE  ONLINE       oraclenode2              STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.oraclenode1.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.oraclenode2.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.qosmserver
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       oraclenode2              STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       oraclenode1              STABLE
--------------------------------------------------------------------------------
[root@oraclenode1 named]#

Regards,
Mallik
Posted by at No comments:

Manually Drop RAC Database and Remove From Clusterware

Manually Drop RAC Database and Remove From Clusterware


1. Verify RAC database status:

[oracle@oraclenode1 ~]$ . oraenv
ORACLE_SID = [oracle] ? RACSB1
The Oracle base has been set to /u01/app/oracle
[oracle@oraclenode1 ~]$ srvctl status database -d RACSB
Instance RACSB1 is running on node oraclenode1
Instance RACSB2 is running on node oraclenode2
[oracle@oraclenode1 ~]$

2. Stop RAC database and start in exclusive restrict mode

[oracle@oraclenode1 ~]$ srvctl stop database -d RACSB
[oracle@oraclenode1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed Sep 1 01:33:54 2021

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup mount exclusive restrict and try to drop DB which will fail;
ORACLE instance started.

Total System Global Area 3070230528 bytes
Fixed Size                  8625416 bytes
Variable Size             838861560 bytes
Database Buffers         2214592512 bytes
Redo Buffers                8151040 bytes
Database mounted.

SQL> drop database;
drop database
*
ERROR at line 1:
ORA-01586: database must be mounted EXCLUSIVE and not open for this operation

SQL> shut immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
SQL> startup nomount;
ORACLE instance started.

Total System Global Area 3070230528 bytes
Fixed Size                  8625416 bytes
Variable Size             838861560 bytes
Database Buffers         2214592512 bytes
Redo Buffers                8151040 bytes

### only way to open your RAC database is EXCLUSIVE session is to set cluster_database=FALSE ###

3. Set cluster_database=FALSE and start DB in exclusive restrict mode

SQL> alter system set cluster_database=FALSE scope=spfile sid='*';

System altered.

SQL> shut immediate;
ORA-01507: database not mounted

ORACLE instance shut down.
SQL> startup mount exclusive restrict;
ORACLE instance started.

Total System Global Area 3070230528 bytes
Fixed Size                  8625416 bytes
Variable Size             771752696 bytes
Database Buffers         2281701376 bytes
Redo Buffers                8151040 bytes
Database mounted.
SQL> select instance_name,status,logins from v$Instance;

INSTANCE_NAME    STATUS       LOGINS
---------------- ------------ ----------
RACSB1           MOUNTED      RESTRICTED

4. Drop Database

SQL> drop database;

Database dropped.

Disconnected from Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production
SQL> exit

5. Remove database from cluster

[oracle@oraclenode1 ~]$ srvctl remove database -d RACSB
Remove the database RACSB? (y/[n]) y
[oracle@oraclenode1 ~]$

Regards,
Mallik
Posted by at No comments:
Subscribe to: Posts (Atom)

Automation Script | Archivelog Generation Hourly Monitoring

1. List out all the running databases and pic one database where we want to monitore the archive log generation from last 1 month. [oracle@o...