Update Oracle RAC Database Using Fleet Maintenance #JoelKallmanDay

Step 4 - Patchin' Alive. (Oracle RAC Database Patching using Fleet Maintenance)

I’ve moved my blog from https://insanedba.blogspot.com to https://dincosman.com Please update your bookmarks and follow/subscribe at the new address for all the latest updates and content. More up-to-date content of this post may be available there.

 I will patch all my database servers database software from 19.16 to 19.20. As there are more than 20 servers to patch, we will use fleet patching. First, I have patched a 2-node cluster database homes by using out of place (OOP) patching methodology through runInstaller in silent mode in Step 2 - Patch My Breath Away(DB OOP Patching).

Now I will use patched an Oracle RAC database home to create a gold image for fleet patching of all Oracle RAC database homes.

Some useful references:

    * Primay Note for Database Patching Using Enterprise Manager 13c Cloud Control Fleet Maintenance (Doc Id 2435251.1) 

    * Prerequisite Checks For Setting Up Fleet Environment (Doc ID 2434264.1)

    * Enterprise Manager Fleet Maintenance FAQ (Doc ID 2272569.1)

    * How to Patch Or Upgrade RAC Database Using Fleet Maintenance (Doc ID 2448341.1)

First create a location on Software Library in EMCC. We will use it as a destination for storing gold image. 

      * Provisioning and Patching > Sofware Library 

* Name : Gold Image Profiles

* Description : Gold Image Profiles for Database and Grid Infrastructure

* Name : 19c

* Description : Version 19c Images

We will create a response file, but at first we need the manual oop (out of place) patched Oracle RAC database home (target name) to refer in response file.

	[oracle@oravt01 ~]$ /u01/app/oracle/em_agent/GoldImage_GI135v6/agent_13.5.0.0.0/bin/emctl config agent listtargets | grep oracle_home
	[OraGI19Home1_1_oravt01.bjk_3271, oracle_home]
	[OraDB19Home1_2_oravt01.bjk_6741, oracle_home]
	[OraHome2_4_oravt01.kkk.bjk_2369, oracle_home]
	[OraGI19Home2_5_oravt01.bjk_1255, oracle_home] --> OOP Patched new grid home to create gold image
	[OraDB19Home2_6_oravt01.bjk_5493, oracle_home] --> OOP Patched new database home to create gold image

view raw list_oracle_homes.txt hosted with ❤ by GitHub

    Here is the content of the response file.

	IMAGE_NAME=19.20DB
	IMAGE_DESCRIPTION=19.20 DB Infrastructure Image
	REF_TARGET_NAME=OraDB19Home2_6_oravt01.bjk_5493
	IMAGE_SWLIB_LOC=Gold Image Profiles/19c
	STORAGE_TYPE_FOR_SWLIB=OmsShared
	STORAGE_NAME_FOR_SWLIB=Software Location on ORAREPO
	REF_HOST_CREDENTIALS=ORACLE@ORAVT01-02:SYSMAN
	WORKING_DIRECTORY=/tmp
	VERSION_NAME=19.20DB

view raw db_create_gold_image.rsp hosted with ❤ by GitHub

    It is time to create the database gold image.

	[oracle@oracc oms]$ /u01/app/oracle/em135/oms/bin/emcli login -username=SYSMAN
	[oracle@s001oracc Setup]$ /u01/app/oracle/em135/oms/bin/emcli db_software_maintenance -createSoftwareImage \
	-input_file="data:/u01/app/Setup/db_create_gold_image.rsp"
	[oracle@oracc Setup]$ emcli db_software_maintenance -getImages
	================================================================================================================================================================
	IMAGE ID IMAGE NAME DESCRIPTION VERSION IMAGE TYPE CREATION DATE OWNER
	================================================================================================================================================================
	05161B2BF0C08F94E063CF14285FC25B 19.20GRID 19.20 Grid Infrastructure Image 19.0.0.0.0 Software Library 2023-09-11 16:16:32.0 SYSMAN
	----------------------------------------------------------------------------------------------------------------------------------------------------------------
	052A75E1737568DBE063CF14285F449D 19.20DB 19.20 DB Infrastructure Image 19.0.0.0.0 Software Library 2023-09-12 16:31:32.0 SYSMAN
	----------------------------------------------------------------------------------------------------------------------------------------------------------------
	Total Rows: 2

view raw create_db_gold_image.txt hosted with ❤ by GitHub

    Now we will subscribe to this gold image for sundb. (This is the database we want to patch with fleet maintenance). (This step can also be done from GUI/BUI, EMCC 13.5 RU16).

	[oracle@oracc Setup]$ /u01/app/oracle/em135/oms/bin/emcli db_software_maintenance -subscribeTarget -target_name=sundb.bjk \
	-target_type=rac_database -image_id=052A75E1737568DBE063CF14285F449D

view raw subscribe_db_gold_image.txt hosted with ❤ by GitHub

    We will deploy gold image to target database. (This step can also be done from GUI/BUI, EMCC 13.5).

	[oracle@oracc Setup]$ vi /u01/app/Setup/db_deploy_gold_image.txt
	NEW_ORACLE_HOME_LIST=/u01/app/oracle/product/19.20/dbhome_1
	workingDir=/tmp
	SKIP_PREREQUISITE_CHECKS=true
	SKIP_CVU_CHECK=true
	[oracle@oracc Setup]$ /u01/app/oracle/em135/oms/bin/emcli db_software_maintenance -performOperation -name=DB_DEPLOY_sundb \
	-purpose=DEPLOY_RAC_SOFTWARE -target_list=sundb.bjk -target_type=rac_database -normal_credential=ORACLE@ORAVT03-04:SYSMAN \
	-privilege_credential=ROOT@ORAVT03-04:SYSMAN -input_file="data:/u01/app/Setup/db_deploy_gold_image.txt"

view raw deploy_db_gold_image.txt hosted with ❤ by GitHub

     New Oracle database homes are deployed. There is only one step left. That is the migration of the RAC database and its instances targets running from the old Oracle Home to the newly patched Oracle Home. If you are using Unified Auditing, just make sure that it is relinked with on option. As we created a gold database image from a relinked with on option one. No additional steps required this time.

    

	[oracle@oracc ~]$ /u01/app/oracle/em135/oms/bin/emcli db_software_maintenance -performOperation -name=DB_UPDATE_sundb \
	-purpose=UPDATE_RACDB -target_list=sundb.bjk -target_type=rac_database -dataguard_role=primary \
	-normal_credential=@S001ORAVT03-04:SYSMAN -privilege_credential=ROOT@ORAVT03-04:SYSMAN -database_credential=SYS@SUNDB:SYSMAN \
	-rolling=true -node_list="oravt03.bjk,oravt04.bjk"

view raw update_target_database.txt hosted with ❤ by GitHub

All Oracle RAC database patching has been completed as easy as taking candy from a baby. tnsnames.ora, sqlnet.ora, and glogin.sql files are copied to the new database home, datapatch is also executed by fleet patching process. We will just update all our shell scripts, .bashrc etc files to refer to the new database home. Also, after patching completed, in EMCC - under Add targets menu, rediscover your updated hosts and promote newly discovered targets associated with the new database home.

 After database patching completed, you may see "PL/SQL package SYS.DBMS_BACKUP_RESTORE version is not current" messages in the alert.log file. If you are using RMAN catalog, connect recovery catalog schema through RMAN and run "UPGRADE CATALOG" command twice to send various SQL DDL statements to update the recovery catalog schema with new tables, views, columns, and so on. If you are not using an RMAN catalog then run below commands to recompile the RMAN packages and procedures.

	$ sqlplus / as sysdba
	SQL> @$ORACLE_HOME/rdbms/admin/dbmsrman.sql
	SQL> @$ORACLE_HOME/rdbms/admin/dbmsbkrs.sql
	SQL> @$ORACLE_HOME/rdbms/admin/prvtrmns.plb
	SQL> @$ORACLE_HOME/rdbms/admin/prvtbkrs.plb

view raw recompile_rman.sql hosted with ❤ by GitHub

    Hope it helps.

    See you on Step 5 - Another one Patches the dust (Cleaning old oracle homes)