ORA-15017: diskgroup "VOTE2" cannot be mounted on cluster nodes

Reuse disk and Create Diskgrup using old disk and mount diskgroup:

While creating disk group got error message"ORA-15017: diskgroup "VOTE2" cannot be mounted on cluster nodes" but diskgruoup created and mounted on only 1 node

Issue:

Unbale to mount diskgroup on node2

Cause: 

ASM disk is not visible on node2 

Error message:

ORA-15017: diskgroup "VOTE2" cannot be mounted on cluster nodes

Troubleshooting logs: 

Check alert log andassociate trace file (/u01/app/oracle/diag/crs/oraclelab2/crs/trace/crsd_oraagent_oracle.trc)

Solution:

after asm disk scan, ASM disks are visible on node2.

Mount the Diskgroup after ASM disks are visible  

[root@oraclelab1 ~]# ps -ef|grep smon

root      7423  7375  0 21:01 pts/1    00:00:00 grep --color=auto smon

root     23851     1  1 09:11 ?        00:10:18 /u01/app/19.0.0.0/grid/bin/osysmond.bin

oracle   24512     1  0 09:12 ?        00:00:00 asm_smon_+ASM1

oracle   25502     1  0 09:12 ?        00:00:00 ora_smon_DEVDB1

[root@oraclelab1 ~]# su - oracle

Last login: Sat Aug 13 20:46:30 IST 2022

[oracle@oraclelab1 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[oracle@oraclelab1 ~]$ asmcmd -p lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     20476    17524                0           17524              0             N  DATA/

MOUNTED  EXTERN  N         512             512   4096  4194304      3068     2744                0            2744              0             N  OCR/

MOUNTED  EXTERN  N         512             512   4096  4194304     10236     9252                0            9252              0             N  RECO/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      856                0             856              0             Y  VOTE1/

[oracle@oraclelab1 ~]$ oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASM_VOTEDISK1

[oracle@oraclelab1 ~]$

[oracle@oraclelab1 ~]$ cd /dev/

[oracle@oraclelab1 dev]$ ll sd*

brw-rw----. 1 root disk 8,  0 Aug 13 08:05 sda

brw-rw----. 1 root disk 8,  1 Aug 13 08:05 sda1

brw-rw----. 1 root disk 8,  2 Aug 13 08:05 sda2

brw-rw----. 1 root disk 8, 16 Aug 13 08:05 sdb

brw-rw----. 1 root disk 8, 17 Aug 13 08:05 sdb1

brw-rw----. 1 root disk 8, 32 Aug 13 08:05 sdc

brw-rw----. 1 root disk 8, 33 Aug 13 08:05 sdc1

brw-rw----. 1 root disk 8, 48 Aug 13 08:05 sdd

brw-rw----. 1 root disk 8, 49 Aug 13 08:05 sdd1

brw-rw----. 1 root disk 8, 64 Aug 13 08:05 sde

brw-rw----. 1 root disk 8, 65 Aug 13 08:05 sde1

brw-rw----. 1 root disk 8, 80 Aug 13 08:05 sdf

brw-rw----. 1 root disk 8, 81 Aug 13 08:05 sdf1

[oracle@oraclelab1 dev]$

[oracle@oraclelab1 dev]$ oracleasm createdisk ASM_VOTEDISK2 /dev/sdf1

Unable to open device "/dev/sdf1": Permission denied

[oracle@oraclelab1 dev]$

[oracle@oraclelab1 dev]$ exit

logout

[root@oraclelab1 ~]# oracleasm createdisk ASM_VOTEDISK2 /dev/sdf1

Device "/dev/sdf1" is already labeled for ASM disk ""

[root@oraclelab1 ~]#

[root@oraclelab1 ~]#

[root@oraclelab1 ~]# dd if=/dev/zero of=/dev/sdf1 bs=4096 count=100

100+0 records in

100+0 records out

409600 bytes (410 kB) copied, 0.00242461 s, 169 MB/s

[root@oraclelab1 ~]# oracleasm createdisk ASM_VOTEDISK2 /dev/sdf1

Writing disk header: done

Instantiating disk: done

[root@oraclelab1 ~]# oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASM_VOTEDISK1

ASM_VOTEDISK2

[root@oraclelab1 ~]#

Connect to +ASM1 and create disksgroupn in sql command prompt or use asmca to create diskgroup:

CREATE DISKGROUP VOTE2 EXTERNAL REDUNDANCY  DISK '/dev/oracleasm/disks/ASM_VOTEDISK2' SIZE 1023M

ATTRIBUTE 'compatible.asm'='19.0.0.0','au_size'='4M'

[oracle@oraclelab1 asmca]$ grep "CREATE DISKGROUP VOTE2" /u01/app/oracle/cfgtoollogs/asmca/asmca-220813PM090427.log

[Thread-83] [ 2022-08-13 21:08:40.050 IST ] [UsmcaLogger.logInfo:156]  SQL: CREATE DISKGROUP VOTE2 EXTERNAL REDUNDANCY  DISK '/dev/oracleasm/disks/ASM_VOTEDISK2' SIZE 1023M

[oracle@oraclelab1 asmca]$

While creating VOTE2 diskgroup get beloe error on asmca:

[DBT-30028] Generic failure interacting with CRS. Details PRCR-1079 : Failed to start resource ora.VOTE2.dg

CRS-5017: The resource action "ora.VOTE2.dg start" encountered the following error: 

ORA-15032: not all alterations performed

ORA-15017: diskgroup "VOTE2" cannot be mounted

ORA-15040: diskgroup is incomplete

. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/oraclelab2/crs/trace/crsd_oraagent_oracle.trc".

CRS-2674: Start of 'ora.VOTE2.dg' on 'oraclelab2' failed

[root@oraclelab2 ~]#tail -f /u01/app/oracle/diag/crs/oraclelab2/crs/trace/crsd_oraagent_oracle.trc

2022-08-13 21:08:45.424 : USRTHRD:916879104: [     INFO] {1:56316:8567} DgpAgent::fetchDgStatus(VOTE2) query:SELECT a.state, b.startup_time FROM v$asm_diskgroup_stat a, v$instance b WHERE a.name = :1  /* asm agent *//* {1:56316:8567} */ OCI error:1403 what:no data found

2022-08-13 21:08:45.424 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::queryDgStatus 122  updateDGSCache

2022-08-13 21:08:45.424 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::queryDgStatus 300 OCI error 1403 no data found

2022-08-13 21:08:45.424 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::queryDgStatus 300 cmdId:259 ckType:65535 dgs.m_dgName:VOTE2 dgs.m_dgpAgent:0x7f34501c43a0 dgs.m_isStatusCached:1

2022-08-13 21:08:45.424 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::queryDgStatus 310  no data found in v$asm_diskgroup_stat

2022-08-13 21:08:45.425 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::stopSingle 100 diskgroup VOTE2 already stopped clsagfw_res_status 1  exit }

2022-08-13 21:08:45.425 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::stop 900 s_DGStatusThread:0x7f344831ef60 m_pConnxn:0x7f3440056140

2022-08-13 21:08:45.425 : USRTHRD:916879104: [     INFO] {1:56316:8567} DgpAgent::fetchDgStatus(VOTE2) query:SELECT a.state, b.startup_time FROM v$asm_diskgroup_stat a, v$instance b WHERE a.name = :1  /* asm agent *//* {1:56316:8567} */ OCI error:1403 what:no data found

2022-08-13 21:08:45.426 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] ConnectionPool::signalEvent entry { this:0x7f3470091af0 s_ohSidEventMapLock:0x563118de55f0 action:3

2022-08-13 21:08:45.426 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] DgpAgent::stopSingle 999 status:2 }

2022-08-13 21:08:45.426 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] clean  }

2022-08-13 21:08:45.426 :CLSDYNAM:916879104: [ora.VOTE2.dg]{1:56316:8567} [clean] (:CLSN00106:) clsn_agent::clean }

2022-08-13 21:08:45.426 :    AGFW:916879104: [     INFO] {1:56316:8567} Command: clean for resource: ora.VOTE2.dg 2 1 completed with status: SUCCESS

[oracle@oraclelab1 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM1

The Oracle base has been set to /u01/app/oracle

[oracle@oraclelab1 ~]$ asmcmd -p lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     20476    17524                0           17524              0             N  DATA/

MOUNTED  EXTERN  N         512             512   4096  4194304      3068     2744                0            2744              0             N  OCR/

MOUNTED  EXTERN  N         512             512   4096  4194304     10236     9252                0            9252              0             N  RECO/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      856                0             856              0             Y  VOTE1/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      932                0             932              0             N  VOTE2/

[oracle@oraclelab1 ~]$

[root@oraclelab2 ~]# ps -ef|grep smon

root     24149     1  1 09:11 ?        00:09:23 /u01/app/19.0.0.0/grid/bin/osysmond.bin

oracle   25416     1  0 09:13 ?        00:00:00 asm_smon_+ASM2

oracle   25987     1  0 09:13 ?        00:00:00 ora_smon_DEVDB2

root     27062 23163  0 21:13 pts/2    00:00:00 grep --color=auto smon

[root@oraclelab2 ~]#

[root@oraclelab2 ~]# oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASM_VOTEDISK1

[root@oraclelab2 ~]# su - oracle

Last login: Sat Aug 13 20:10:43 IST 2022

[oracle@oraclelab2 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[oracle@oraclelab2 ~]$ asmcmd -p lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     20476    17524                0           17524              0             N  DATA/

MOUNTED  EXTERN  N         512             512   4096  4194304      3068     2744                0            2744              0             N  OCR/

MOUNTED  EXTERN  N         512             512   4096  4194304     10236     9252                0            9252              0             N  RECO/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      856                0             856              0             Y  VOTE1/

[oracle@oraclelab2 ~]$

[root@oraclelab2 ~]# oracleasm scandisks

Reloading disk partitions: done

Cleaning any stale ASM disks...

Scanning system for ASM disks...

Instantiating disk "ASM_VOTEDISK2"

[root@oraclelab2 ~]# oracleasm listdisks

ASMDISK1

ASMDISK2

ASMDISK3

ASM_VOTEDISK1

ASM_VOTEDISK2

[root@oraclelab2 ~]#

[oracle@oraclelab2 ~]$ . oraenv

ORACLE_SID = [oracle] ? +ASM2

The Oracle base has been set to /u01/app/oracle

[oracle@oraclelab2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Aug 13 21:24:30 2022

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0

SQL> ALTER DISKGROUP VOTE2 mount;

ALTER DISKGROUP VOTE2 mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15260: permission denied on ASM disk group

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0

[oracle@oraclelab2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Aug 13 21:25:31 2022

Version 19.3.0.0.0

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0

SQL> ALTER DISKGROUP VOTE2 mount;

Diskgroup altered.

SQL>

[oracle@oraclelab2 ~]$ asmcmd -p lsdg

State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name

MOUNTED  EXTERN  N         512             512   4096  4194304     20476    17524                0           17524              0             N  DATA/

MOUNTED  EXTERN  N         512             512   4096  4194304      3068     2744                0            2744              0             N  OCR/

MOUNTED  EXTERN  N         512             512   4096  4194304     10236     9252                0            9252              0             N  RECO/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      856                0             856              0             Y  VOTE1/

MOUNTED  EXTERN  N         512             512   4096  4194304      1020      888                0             888              0             N  VOTE2/

[oracle@oraclelab2 ~]$

[oracle@oraclelab1 asmca]$ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   834b0ae647534ff3bf237d71ebf0e33e (/dev/oracleasm/disks/ASM_VOTEDISK1) [VOTE1]

Located 1 voting disk(s).

[oracle@oraclelab1 asmca]$ crsctl replace votedisk +VOTE2

Successful addition of voting disk d5d34d250c634f14bf3dacc987d84bb7.

Successful deletion of voting disk 834b0ae647534ff3bf237d71ebf0e33e.

Successfully replaced voting disk group with +VOTE2.

CRS-4266: Voting file(s) successfully replaced

[oracle@oraclelab1 asmca]$ crsctl query css votedisk

##  STATE    File Universal Id                File Name Disk group

--  -----    -----------------                --------- ---------

 1. ONLINE   d5d34d250c634f14bf3dacc987d84bb7 (/dev/oracleasm/disks/ASM_VOTEDISK2) [VOTE2]

Located 1 voting disk(s).

[oracle@oraclelab1 asmca]$

Regards,

Mallik