Oracle EBS Suite blank Login page - Post Exadata migration

As part of EBS database migration on Exadata, we recently deployed a brand new Exadata X5-2L (eighth rack), and migrated an Oracle EBS Database from the typical storage/server technologies.

Below are the environment details:

EBS Suite 12.1.3 (running on two application servers with hardware LOAD BALANCER)
Database :, RAC database with 2 instance

After the database migration, buildxml and autoconfig procedures went well on both application and database tiers. However, when the EBS login page is launched, it came out as just blank page, plus, apps passwords were unable to change through the typical procedure. We wonder what went wrong, as none of the procedure gave any significant failure indications, all went fine and we could see the successful completion messages.

After a quick initial investigation, we found that there is an issue with the GUEST user, and also found that the profile was not loaded when the autoconfig was ran on the application server. In the autoconfig log file, we could see the process was failed to update the password (ORACLE). We then tried all workaround, the recommended on Oracle support and other websites. Unfortunately, none of the workarounds helped us.

After almost spending a whole day, investigating and analyzing the root cause, we looked at the DB components and status in dba_registry view. We found the JSserver Java Virtual Machine component INVALID. During Exadata software deployment at RDBMS patch installation phase, there was an error while applying the patch due to conflict between the patches.

Without further wasting a single second, we ran the @catbundle and followed by ultrp.sql.

Guess what, all issues disappeared. We run the autoconfig on the application servers. After which we could change the app user password and we could see the Login page too.

When you face EBS blank login page, and none of the workarounds fix the issues, its a good idea to have a look at dba_registry for any INVALID components.

It was quite a nice experience.

This really 


Exadata X5-2L deployment & Challenges

A brand new Exadata X5-2L eighth rack (I knew the latest is X7 now, but it was for a POC, so no worries) has been deployed recently at a customer for Oracle EBS Exadata migration POC purpose. Its wasn't an easy walk in the park as I initially presumed. There were some challenges (network, configuration) thrown during the migration, but, happily overcome and had it installed and EBS database migration completion.

So, I am going to share yet another Exadata bare metal deployment story explaining the challenges I have faced, and how they are fixed.

Issue 1) DB network cable issues:
After successful execution of the elasticConfig, all the Exadata factory IP addresses have been set to client IPs. Though the management network was accessible from the outside, client network was not accessible. When verified with the network team about enabling the ports on the corporate switch, they confirmed that the ports are enabled, however, the connection is showing as not active and asked to us investigate the network cables connected to the DB nodes. When we verified the network cables ports, we didn't find any lights flashing and after an extensive investigation (Switch ports, SFP on Exadata and Corporate switch, checking the cables status), it was found that the cables pin direction was not properly connected. Also, found that the network bonding interfaces (eth4 and eth5) were not up, confirmed from ethtool eth1 command. After fixing the cables, and bringing up the interfaces (ifup eth4 & eith5), we could see that cables are connected properly and we can also see the lights on the ports.

$ethtool eth4 (shows the interfaces were not connected)
Settings for eth4:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: Yes
        Advertised link modes:  100baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Speed: Unknown!
        Duplex: Unknown! (255)

        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no

Issue 2) Wrong Netmask selection for client network:
After fixing the cable issues, we then continued with onecommand execution. During the validation it failed because of different netmask for client network (under the cluster information section). The customer unfortunately made a mistake in the client network netmask selection for cluster settings, so there was a difference in the client netmask value for client and cluster. This was fixed by modifying the netmask value in the ifcfg-boneth0 file (/etc/sysconfig/network-scripts), restart the network services.

Issue 3) Failed eight Rack configuration (rack type and disk size):
Since the system was delivered somewhere during end of August 2015, no one actually knows exactly the disk size and rack model. The BOQ (Bill of quantity) for the order only shows X5-2 HC storage. So, there was wrong selection in the OEDA for Exadata rack and disk size. Instead of 4TB disk size, it was selected as 8TB disk size and instead of Elastic configuration, fixed Eighth rack was selected. This was fixed by rerunning the OEDA with the correct options.

Issue 4) Cell server IP issues:
There was another obstacle faced while doing the cell connectivity (part of onecommand). Cell server IPs were not modified by the elasticConfig. Fortunately, I found my friend blog on this topic and quickly fixed the issue. This is why I like to blog all the technical issues, who knows, this could solve someone pains.


Issue 5) SCAN Listener configuration:
Cluster validation failed due to inconsistent values for scan name. During the investigation of various issues, private, public & scan IPs are put in the /etc/hosts file. So, while configuring LISTENER_SCAN2 and SCAN3, this issue happened. This was fairly understandable. Due to 3 entries of scan values in the /etc/hosts file this happened. Upon a quick google about the issue, the following blog helped me to fix the issue


Finally, I have managed to deploy the Exadata successfully and perform the Oracle EBS database migration . No doubt, this experience really made me strong in network and other areas. So, every challenges comes with opportunity to learn.

I thank those individuals who really write blogs and share their experience to help Oracle community.

There is still one open issues which is yet to be resolved. A slow sqlplus and db startup. I presume this is due to heavy resource utilization over the server. Yet to resolve the mystery. Stay tuned for more updates.

Update : 06-Nov-2017.

sqlplus slowness resolved by increasing the hugepages from 26gb to 96GB.


Oracle R Enterprise Server Configuration

Oracle R Enterprise Server Overview

Oracle R Enterprise includes several components on the server. Together these components enable an Oracle R Enterprise client to interact with Oracle R Enterprise Server.
The server-side components of Oracle R Enterprise are: 
  • Oracle Database Enterprise Edition (64bit)
  • Oracle R Distribution or open source R
  • Oracle R Enterprise Server
Oracle R Enterprise Server consists of the following: 
    • The rqsys schema
    • Metadata and executable code in sys
    • Oracle R Enterprise Server libraries in $ORACLE_HOME/lib (Linux and UNIX) or %ORACLE_HOME%\bin (Windows)
    • Oracle R Enterprise R packages in $ORACLE_HOME/R/library (%ORACLE_HOME%\R\library on Windows)

Oracle R Enterprise Server Requirements

Before installing Oracle R Enterprise Server, verify your system environment, and ensure that your user ID has the proper permissions.

Oracle R Enterprise runs on 64-bit platforms only.
·        Linux x86-64
·        Oracle Solaris
·        IBM AIX
·        IBM AIX

Oracle Database must be installed and configured as described
·        Oracle R Enterprise requires the 64-bit version of Oracle Database Enterprise Edition.

Installing Oracle R Enterprise Server

verify if ORACLE_HOME, ORACLE_SID, R_HOME, PATH, and LD_LIBRARY_PATH environment variables are properly set. For example, you could specify values like the following in a bashrc file:

export ORACLE_HOME=/u01/app/oracle/product/
export R_HOME=/usr/lib64/R
export PATH=$PATH:$R_HOME/bin:$ORACLE_HOME/bin

Download Oracle R Enterprise Server

Go to the Oracle R Enterprise home page on the Oracle Technology Network:.
http://www.oracle.com/technetwork/database/options/advanced-analytics/r-enterprise/Opens a new window
Select Oracle R Enterprise Downloads. On the Downloads page, select Oracle R Enterprise Server and the Supporting Packages for Linux. The following files are downloaded for Oracle R Enterprise 1.4.1.


Login as root, and copy the installers for Oracle R Enterprise Server and the supporting packages across nodes. For example:

$ dcli -g nodes -l oracle mkdir -p /home/oracle/ORE
$ dcli -g nodes -l oracle -f ore-server-linux-x86-64-1.4.1.zip -d 
$ dcli -g nodes -l oracle -f ore-supporting-linux-x86-64-1.4.1.zip -d 

Unzip the supporting packages on each node:
$ dcli -t -g nodes -l oracle unzip   
     /home/oracle/ORE/ore-supporting-linux-x86-64-1.4.1.zip -d 

 Install Oracle R Enterprise server components:
$ dcli -t -g nodes -l oracle "cd /my_destination_directory; ./server.sh -y
      --admin --sys syspassword --perm permtablespace
      --temp temptablespace --rqsys rqsyspassword
      --user-perm usertablespace --user-temp usertemptablespace
      --pass rquserpassword --user RQUSER"

Installing Oracle R Enterprise Server interactively:
$ ./server.sh -i

Oracle R Enterprise 1.4.1 Server.

Copyright (c) 2012, 2014 Oracle and/or its affiliates. All rights reserved.

Checking platform .................. Pass
Checking R ......................... Pass
Checking R libraries ............... Pass
Checking ORACLE_HOME ............... Pass
Checking ORACLE_SID ................ Pass
Checking sqlplus ................... Pass
Checking ORACLE instance ........... Pass
Checking CDB/PDB ................... Pass
Checking ORE ....................... Pass

Choosing RQSYS tablespaces
  PERMANENT tablespace to use for RQSYS [list]: DATA
  TEMPORARY tablespace to use for RQSYS [list]: TEMP1
Choosing RQSYS password
  Password to use for RQSYS:

Choosing ORE user
  ORE user to use [list]:

ORE user to use [list]: OREUSER1

Current configuration
  R Version ........................ Oracle Distribution of R version 3.1.1  (--)
  R_HOME ........................... /usr/lib64/R
  R_LIBS_USER ...................... /u01/app/oracle/product/
  ORACLE_HOME ...................... /u01/app/oracle/product/
  ORACLE_SID ....................... OFSAPRD1

  Existing R Version ............... None
  Existing R_HOME .................. None
  Existing ORE data ................ None
  Existing ORE code ................ None
  Existing ORE libraries ........... None

  RQSYS PERMANENT tablespace ....... DATA
  RQSYS TEMPORARY tablespace ....... TEMP1

  ORE user type .................... Existing
  ORE user name .................... OREUSER1
  ORE user PERMANENT tablespace .... DATA
  Grant RQADMIN role ............... No

  Operation ........................ Install/Upgrade/Setup

Proceed? [yes] y

Removing R libraries ............... Pass
Installing R libraries ............. Pass
Installing ORE libraries ........... Pass
Installing RQSYS data .............. Pass
Configuring ORE .................... Pass
Installing RQSYS code .............. Pass
Installing ORE packages ............ Pass
Creating ORE script ................ Pass
Installing migration scripts ....... Pass
Installing supporting packages ..... Pass
Granting ORE privileges ............ Pass





How to install Oracle R Enterprise (ORE) on Exadata

Very recently, we have deployed ORE (R Distribution and R Enterprise) 3.1.1 packages on 4 node Exadata environment. This blog will discuss the prerequisites and procedure to deploy Oracle R Distribution v3.1.1.

Note: Ensure you have a latest system (root and /u01) backup before you deploy the packages on the db server.

What is R and Oracle Enterprise

R is third-party, open source software. Open source R is governed by GNU General Public License (GPL) and not by Oracle licensing. Oracle R Enterprise requires an installation of R on the server computer and on each client computer that interacts with the server.

Why Oracle R Distribution? 
  • Oracle R Distribution simplifies the installation of R for Oracle R Enterprise.
  • Oracle R Distribution is supported by Oracle for customers of Oracle Advanced Analytics, Oracle Linux, and Oracle Big Data Appliance.

What is needed for R Distribution deployment for Oracle Linux 6?

The Oracle R Distribution RPMs for Oracle Linux 6 are listed as follows:

If the following dependent RPM is not automatically included, then download and install it explicitly:

The picture below depicts the ORE client/server installation steps:

Description of Figure 1-2 follows

Oracle R Distribution on Oracle Linux Using RPMs

Oracle recommends that you use yum to install Oracle R Distribution, because yum automatically resolves RPM dependencies. However, if yum is not available, then you can install the RPMs directly and resolve the dependencies manually. Download the required rpms and its dependent rpms from below link:


To know more about rpms and its dependent rpms, visit the following Oracle website:


You can install the rpms in the following order:

yum localinstall libRmath-3.1.1-2.el6.x86_64.rpm
yum localinstall libRmath-devel-3.1.1-2.el6.x86_64.rpm
yum localinstall libRmath-static-3.1.1-2.el6.x86_64.rpm
yum localinstall R-core-3.1.1-2.el6.x86_64.rpm
yum localinstall R-devel-3.1.1-2.el6.x86_64.rpm
yum localinstall R-3.1.1-2.el6.x86_64.rpm

Once the rpms are installed, you can validate the installation , using the below procedure:

go to /usr/lib64/R directory on the database, as oracle user, type R:

You must see the output below:

type q() to exit from the R interface.

And repeat on the rest of the db nodes, if you are on RAC.

To install R distribution, use the procedure below:

rpm -e R-Rversion
rpm -e R-devel
rpm -e R-core
rpm -e libRmath-devel
rpm -e libRmath
 In the blog post, I will demonstrate how to configure Oracle R Enterprise.