Insane DBA...

Listener does not currently know of SID given in connect descriptor.  After upgrading Oracle grid infrastructure from 11.2.0.4 to 19c; we have detected that +ASM and +APX(ASM Proxy) instances used for advm are not registered with listener and OEM target status shows down.   They were only accessible from private-network(192.168.*.*).We have checked configuration attributes for asm instance with srvctl command. The easy and first step to take was to set it with "srvctl modify" command. Oops something went wrong and that is not possible.  It looks like this command is not working anymore, searched docs.oracle and checked oracle support but no easy way out. Checked asm proxy instance local_listener parameter and it was not set. According to the "Real Application Clusters Installation Guide for Linux and UNIX"  , If you do not set LOCAL_LISTENER, then the Database Agent process automatically updates the database associated with the local listener in the Grid ho

 Too many parse errors - how much do you mean? SQL syntax errors are always normal and commonly dealing with them is up to software developers.  But sometimes, they might bother database administrators too. When an sql is syntactically(E.g. syntax error) or semantically(E.g. projection of a nonexistent column) incorrect, its processing fails at parsing stage and it never gets executed. If that happens too often, it can have a negative impact on overall database performance. With 12.2 release, these failing sqls are recorded in alert.log as below if they are called excessively. In the above example, "select dual" statement is failing with "ORA-00923: FROM keyword not found where expected" error code. This is a syntax error, which is observed 100 times  within 4 minutes and it is recorded. Now the question is coming. How much amount of failure is required for an sql to got written in the alert.log file? According to the  Doc ID 16945190.8 , By default the diagnostic w

Is it possible to get credentials from crs?      In last two posts, I have been talking about the catastrophic situation we have experienced which was triggered with the mirrored disk failures. Up until now, we restored OCR config and recreated lost ASM disk group which was hosting OCR before. but we could not be able to start the CRS on more than one node.      We got the error "CRS-5019: All OCR locations on ASM disk groups [DATA], and none of these disk groups are mounted". We recreated asm password file.  (We should have restore it from offline disk group.) It was not enough to recreate it. There were some missing internal users. According to the " Doc ID 2341753.1, The users used in Flex ASM ", CRSUSER__ASM_001 user is needed by crsd and it should have sysasm privilege, so we gave crs what it needed. We defined CRSUSER__ASM_001 user's password on our own, but that was not the proper way. It is an internal user which is created at the grid installation part

OCR lost. Where is OCR backup? On ASM diskgroup. After Mirrored Disk Failure in Normal Redundancy Mode, CRS was down. We could not take the faulty disks and one disk group (DATA)  online again and decided to restore OCR config and change VOTING disk location from DATA to RECO.  Started CRS in exclusive mode and searched for backup locations, unfortunately we had no backups in local file system. OCR backups were on one of ASM disk group and this disk group could not be online. We mounted that disk group in restricted mode and tried copying latest ocr backup to local directory with the commands below, but could not achieve. We searched Oracle support and found, " Doc ID 2569847.1, How to Restore ASM Based OCR when OCR backup is located in ASM diskgroup . " According to the document, "amdu" command was the one we were looking for. We executed the commands below and restored latest ocr backup (file number 875) to our current working directory. We followed, " Doc I

 IRON MAN  WAS DOWN Lately, in our DR (Disaster Recovery) site, we have experienced two mirrored disk failure in normal redundancy mode which ended up with recreating of dataguard databases.  I will try to explain our problem in detail.     Our databases were down. CRS state was offline. ASM was down. Iron Man was down. We started diagnosing the issue with manually starting up ASM instance on one node. We have 3 diskgroups. 2 of them got mounted, but  one diskgroup(+DATA) could not get mounted. This diskgroup (+DATA) was holding ocrconfig and serving as voting disk.  This is the command and output.  We checked all ASM instances alert.logs to clarify chronological order of events. Let's examine the findings.      On 07:17:22, on exacel11 tried to offline disk 8. This was the first faulty disk.      On 07:17:25, after offlining the first faulty disk, Exadata disk worker process(XDWK) tried to access  partner disks of these ASM disks. All subsequent IOs to faulty ASM disks will be dir