How To Recreate The ASM Password File? (A Failure Story Part 3)

Is it possible to get credentials from crs?

I’ve moved my blog from https://insanedba.blogspot.com to https://dincosman.com Please update your bookmarks and follow/subscribe at the new address for all the latest updates and content. More up-to-date content of this post may be available there.

    In the last two posts, I have been talking about the catastrophic situation we experienced, which was triggered by mirrored disk failures. Up until now, we restored OCR config and recreated the lost ASM disk group which was hosting OCR before, but we could not start the CRS on more than one node.

    We got the error "CRS-5019: All OCR locations on ASM disk groups [DATA], and none of these disk groups are mounted". We recreated the asm password file.  (We should have restored it from the offline disk group.)

	[root@exadb02 trace]# vi /u01/app/oracle/diag/crs/exadb02/crs/trace/alert.log
	...
	2020-08-06 11:24:10.239 [ORAROOTAGENT(278746)]CRS-5019: All OCR locations are on ASM disk groups [DATA], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/oracle/diag/crs/exadb02/crs/trace/ohasd_orarootagent_root.trc".
	...
	[root@exadb02 trace]# tail -f /u01/app/oracle/diag/crs/exadb02/crs/trace/ohasd_orarootagent_root.trc
	...
	2020-08-06 11:29:37.160 : USRTHRD:697435904: [ INFO] {0:5:3} [ora.storage] Error [kgfoAl06] in [kgfokge] at kgfo.c:3169
	2020-08-06 11:29:37.160 : USRTHRD:697435904: [ INFO] {0:5:3} [ora.storage] ORA-01017: invalid username/password; logon denied
	...
	ASMCMD> pwcreate --asm +DATA/orapwASM welcome_1
	ASMCMD-9465: WARNING: passing password on command line is deprecated
	OPW-00010: Could not create the password file. This resource has a Password File.
	ASMCMD-9454: could not create new password file
	ASMCMD> pwdelete --asm
	OPW-00022: The password file does not exist.
	ASMCMD-9462: could not delete password file
	ASMCMD> pwcreate --asm +DATA/orapwASM welcome_1
	ASMCMD-9465: WARNING: passing password on command line is deprecated

view raw create_passwordfile_asm.txt hosted with ❤ by GitHub

Recreating it was not enough. There were some missing internal users.

According to "Doc ID 2341753.1, The users used in Flex ASM", CRSUSER__ASM_001 user is needed by crsd and it should have sysasm privilege, so we gave crs what it needed.

	We have this privileged database users in a healthy cluster :
	ASMCMD> lspwusr
	Username sysdba sysoper sysasm
	SYS TRUE TRUE TRUE
	CRSUSER__ASM_001 TRUE FALSE TRUE
	ASMSNMP TRUE FALSE FALSE
	What do we have :
	ASMCMD> lspwusr
	Username sysdba sysoper sysasm
	SYS TRUE TRUE FALSE
	We used below commands to fix it.
	[oracle@exadb01 ~]$ asmcmd orapwusr --grant sysasm SYS
	[oracle@exadb01 ~]$ asmcmd orapwusr --add CRSUSER__ASM_001
	Enter password: ********* ---> I used welcome1
	[oracle@exadb01 ~]$ asmcmd orapwusr --grant sysasm CRSUSER__ASM_001
	[oracle@exadb01 ~]$ asmcmd orapwusr --grant sysdba CRSUSER__ASM_001
	[oracle@exadb01 ~]$ asmcmd lspwusr
	Username sysdba sysoper sysasm
	SYS TRUE TRUE TRUE
	CRSUSER__ASM_001 TRUE FALSE TRUE
	[oracle@exadb01 ~]$ asmcmd orapwusr --add ASMSNMP
	Enter password: *********
	[oracle@exadb01 ~]$ asmcmd orapwusr --grant sysdba ASMSNMP
	[oracle@exadb01 ~]$ asmcmd lspwusr
	Username sysdba sysoper sysasm
	SYS TRUE TRUE TRUE
	CRSUSER__ASM_001 TRUE FALSE TRUE
	ASMSNMP TRUE FALSE FALSE

view raw flex_asm_users.txt hosted with ❤ by GitHub

We defined the CRSUSER__ASM_001 user's password on our own, but that was not the proper way. It is an internal user which is created at the grid installation part and its password is given internally. 

We used the ocrdump utility to view OCR and OLR contents by writing the content to a file to detect the CRSUSER__ASM_001 user's credential path and retrieved the CRSUSER__ASM_001 user password  according to the "Doc ID 2139591.1, ODA: CRS Could Not Start on Node Due to Invalid ASM Credentials for The "crsuser__asm_001" Clusterware User" and modified it.

	[oracle@exadb01 ~]$ ocrdump /tmp/ocr.dmp
	PROT-310: Not all keys were dumped due to permissions.
	[oracle@exadb01 ~]$ vi /tmp/ocr.dmp
	...
	[SYSTEM.ASM.CREDENTIALS.USERS.CRSUSER__ASM_001]
	ORATEXT : 8956aa1aaa66aa46aaaa2222a895aa5a:oracle
	...
	[oracle@exadb01 ~]$ crsctl get credmaint -path /ASM/Self/8956aa1aaa66aa46aaaa2222a895aa5a -credtype userpass -id 0 -attr passwd -local
	ZbV9Zx7F8BKxM2ZD4gUnKMjaH5Hnk
	[oracle@exadb01 ~]$ asmcmd orapwusr --modify CRSUSER__ASM_001
	Enter password: *****************************

view raw retrieve_credential.txt hosted with ❤ by GitHub

After modifying the CRSUSER__ASM_001 user password, CRS started normally. 

We raised IRON MAN again.

We also changed the backup location of the asm password file location and these are the commands we used.

	ASMCMD> pwcopy +DATA/orapwasm +RECO/orapwasm_backup
	copying +DATA/orapwasm -> +RECO/orapwasm_backup
	[oracle@exadb01 dbs]$ srvctl config asm
	ASM home: <CRS home>
	Password file: +DATA/orapwasm
	Backup of Password file: +DATA/orapwASM_backup
	ASM listener: LISTENER
	ASM instance count: ALL
	Cluster ASM listener: ASMNET1LSNR_ASM
	[oracle@exadb01 dbs]$ srvctl modify asm -pwfilebackup +RECO/orapwasm_backup
	[oracle@exadb01 dbs]$ srvctl config asm
	ASM home: <CRS home>
	Password file: +DATA/orapwasm
	Backup of Password file: +RECO/orapwasm_backup
	ASM listener: LISTENER
	ASM instance count: ALL
	Cluster ASM listener: ASMNET1LSNR_ASM

view raw modify_asm_pwbackup.txt hosted with ❤ by GitHub

That's the whole story to heal the cluster and asm disk groups, later we configured dataguard databases again.

Hope it helps.