Tricky Oracle data guard situation

Had an interesting situation configuring a Data Guard setup of 7TB Oracle 12c database at one of the clients. After a long and rounds of discussions, considering the constraints and limitations, agreed to use the traditional way of Data Guard setup. Backup the database, replicate the backups to DR site, create a standby controfile, restore the control file on standby host , do DB restore, recovery and configure the synchronization.

Restore failed due to unavailability of the data files location (same as production) on the DR host, because, all application tablespaces data files on PROD were not used OMF format, stored under ASM with .dbf extension. Due to this fact, only default tablespaces data files (which actually used OMF format in ASM) were restored. A PROD similar directory structure (using ALTER DISKGROUP ADD DIRECTORY) created and all the data files were restored successfully. The PROD similar directory (+DATA/PROD/DATAFILE)  contained only the soft links and data files were created under the Standby Database SID directory. The entire process took more than 24 hrs, and a subsequent incremental backup was taken to fill the gap and have the data in sync using the roll-forward procedure.

As part of the roll-forward, a new standby controlfile was created on PROD and restored on the DR DB. When CATALOG START WITH command was issued, only few data files (system related) were able to CATALOG, while the soft link data files weren't. This stopped us doing the roll-forward recovery. Tried all possible option, nothing was materialized.

We then cleaned-up the DR database data files, to have a fresh restore. Used the db_convert_file_name parameter and start over the traditional DR configuration procedure. This time, all the data files were restored under standby SID directory, also, soft links for the non-standard OMF data files, were created in the same directory, unlike the first attempt. Once the restore done, successfully performed the roll-forward procedure to make the PROD and DR DBs in sync.

If you have non-standard OMF files, with .dbf extension under ASM, ensure you use the db_file_name_convert to avoid the mess.

Well, if you have any alternative, you are most welcome to suggest.


Exadata X7 installation some Whitney+ issues

One of our DB teams from Europe highlighted the following issues while installation and configuring VM on Exadata X7.

"Today we started X7-2 Exadata installation. But we faced some issues. First problem is, when installing the cluster, resources are being up then suddenly, being down and there are some internal bugs for this problem:

Bug 27275655 - Running tcpdump on bondeth0 kills the Whitney LOM interface
Bug 27195117 - Finisar FTLX8574D3BCV-SU transceiver does not link up on X7-2 LOM

These are internal bugs required to re-image the whole stack and use the image Also, for the PXE(for re-imaging), there is another problem.

Without for those bugs when you are installing ExadataX7.


ORA-01033 when Connecting to the DB Via Scan Listener

Had a very interesting and tricky scenario a few days ago. It was something I never encountered in my DBA career. Hence, thought of sharing the detail of there store here today.


During mid-day, an application team reported that they are suddenly getting an ORA-01033 when connecting to a 3 node RAC database ( Quick basic validations reveals that the issue is happening only when connecting through SCAN IP, but, the VIP and direct (physical IP) connections were having no issues. So, it was clear that the issue is with the SCAN IPs. Verified all the configuration settings , network and firewall to ensure there is no issues accessing the SCAN IPS. To our surprise, everything was just perfectly okay. This really puzzles us.We also suspected the Data Guard configuration of this database, but, it wasn't the case either.


After a quick search over the internet, we come across of MOS Doc: ORA-01033 is Thrown When Connecting to the Database Via Scan Listener (Doc ID 1446132.1)

The issue was, one of the team members was restoring the database backup on a new host.  The tricky part here is, the new host is part of the same network/subnet where the 3 node RAC database is running, and can access to SCAN IPs too. Perhaps the instance that is restoring is registered with the SCAN, and whenever a new connection request is made through SCAN, the connection was referred to an instance which in mount state (restoring). Hence, an ORA-1033 error is thrown.


After reviving the note, the restore immediately stopped, and things got back to normal. Even nullifying the remote_listener parameter to de-register with SCAN would have been also worked in this case.

This issue can be prevented through configuring Class of Secure Transport (COST).

The probabilities of hitting the issue is very low, but, I felt its interesting and sharing it worth while.


ORA-01033 is Thrown When Connecting to the Database Via Scan Listener (Doc ID 1446132.1)
NOTE:1340831.1 - Using Class of Secure Transport (COST) to Restrict Instance Registration in Oracle RAC
NOTE:1453883.1 - Using Class of Secure Transport (COST) to Restrict Instance Registration 


ORA-19527: physical standby redo log must be renamed

Data Guard was configured a few days back at one of the customers site. An ORA-19527 error is continuously reported in the standby database alert.log (database name is changed)

Errors in file /u01/app/oracle/diag/rdbms/orcl/ORCL1/trace/ORCL1_rfs_3504.trc:
ORA-19527: physical standby redo log must be renamed
ORA-00312: online log 9 thread 1: '+DATAC1/ORCL/ONLINELOG/group_9.459.960055023'

Though this didn't interrupt the data guard replication, the customer wanted to get-rid of the error message. Starting with v10g, this was an expected behavior to improve the switchover and failover. With v10g, when MRP is started, it will attempt to clear the online log files rather than at the time of role transition.

This is also an expected behavior when the log_file_name_convert parameter is not set on the standby database. But, in our case, the log_file_name_convert was not required as PRIMARY and STANDBY has the same directory structure (DISK GROUPS for DATA and RECO).

The workaround to get rid of the message, when there is no different in the directory structure, is simply to set some dummy values to the parameter, as shown in the below example:

SQL> ALTER SYSTEM SET log_file_name_convert='dummy','dummy';

After the parameter was set, the ORA message was no longer seen in the alert.log

ORA-19527 reported in Standby Database when starting Managed Recovery (Doc ID 352879.1)
ORA-19527: Physical Standby Redo Log Must Be Renamed...during switchover (Doc ID 2194825.1)