Oracle RAC 10g Installation on Oracle Enterprise Linux Using VMware Server

This article provides a very low cost alternative to configure an Oracle10g RAC system using a single machine running Windows XP and VMware Server.
We will configure two virtual machines running Enterprise Linux, which is based on Red Hat Advanced Server Release 4 (Update 4) with additional bug fixes, is freely available for download.

This guide contains the following sections:
1. Hardware and Software Requirements and Overview
2. Configure the First Virtual Machine
3. Configure Enterprise Linux on the First Virtual Machine
4. Create and Configure the Second Virtual Machine
5. Configure Oracle Automatic Storage Management (ASM)
6. Configure Oracle Cluster File System (OCFS2)
7. Install Oracle Clusterware
8. Install Oracle Database 10g Release 2
9. Verify and Explore the RAC Cluster and the RAC Database Environment
10. Perform some DB Operations
11. Test Transparent Application Failover (TAF)
12. Database Backup and Recovery
13. Explore Oracle Enterprise Manager (OEM) Database Console
14. Common Issues
15. Improving VM and Oracle
16. References and more Information


1. Hardware and Software Requirements and Overview

In this article, I will not be providing details for installing VMware. If you need details on that, see the step-by-step details for this type of install at Installing VMware Workstation 5.0 - (Windows XP).
Just remember to:
- Choose the complete version without IIS support
- Disable the Windows firewall

Host Machine and Virtual Machine Details

Host Machine
Host Machine Name concord
Host Operating Environment Windows XP Professional Service Pack 3
WMware Version VMware Server Console 1.0.6 (Build 91891)
Host Machine Laptop Dell Latitude D620
Memory 2GB Installed
(Each virtual machine will take 700MB from this 2GB)
External Hard Drive 300GB
Processor 1.66 GHz.
Guest Machine
Virtual Machine Configuration #1
Guest Operating Environment Oracle Enterprise Linux 4 (32 bits)
Guest Machine Name rac1
Public Name/IP - (eth0) rac1.mycorpdomain.com- (192.168.2.131)
Interconnect Name/IP - (eth1) rac1-vip.mycorpdomain.com - (10.10.10.31)
Memory 700MB
Hard Drive 20GB
Virtual Machine Location E:\RAC\rac1
Guest Machine
Virtual Machine Configuration #2
Guest Operating Environment Oracle Enterprise Linux 4 (32 bits)
Guest Machine Name rac2
Public Name/IP - (eth0) rac1.mycorpdomain.com- (192.168.2.132)
Interconnect Name/IP - (eth1) rac2-vip.mycorpdomain.com - (10.10.10.32)
Memory 700MB
Hard Drive 20GB
Virtual Machine Location E:\RAC\rac2


Virtual Disk on Host
Virtual Disk on Host
Virtual Device Node
Size (GB)
Description
E:\RAC\rac1\localdisk.vmdk
/dev/sda1
/dev/sda2
/dev/sda3
SCSI 0:0
20
/
mountpoint
Swap space
Oracle binaries
E:\RAC\sharedstorage\ocfs2disk.vmdk
/dev/sdb
SCSI 1:0
0.5
OCFS2 disk
E:\RAC\sharedstorage\asmdisk1.vmdk
/dev/sdc
SCSI 1:1 3
ASM disk group 1
E:\RAC\sharedstorage\asmdisk2.vmdk /dev/sdd
SCSI 1:2 3
ASM disk group 2
E:\RAC\sharedstorage\asmdisk3.vmdk /dev/sde
SCSI 1:3 2
ASM flash recovery area

(To configure shared storage, the guest OS should not share the same SCSI bus with the shared storage. Use SCSI0 for the guest OS and SCSI1 for the shared disks.)
An overview of the RAC database environment:

Host Name
ASM Instance Name
RAC Instance Name
Database Name
Dabatase File Storage
OCR & Voting Disk
rac1
+ASM1
devdb1
devdb
ASM OCFS2
rac2
/+ASM2 devdb2
devdb
ASM OCFS2

You’ll install the Oracle Home on each node for redundancy. The ASM and Oracle RAC instances share the same Oracle Home on each node.


Software needed for Oracle RAC
• From OTN http://www.oracle.com/technology/software/products/database/index.html
10201_database_linux32.zip
10201_clusterware_linux32.zip

•From VMware http://www.vmware.com/download/server
VMware-server-installer-1.0.4-56528.exe

• From Oracle
Enterprise Linux from Oracle:
Enterprise-R4-U4-i386-disc1.iso
Enterprise-R4-U4-i386-disc2.iso
Enterprise-R4-U4-i386-disc3.iso
Enterprise-R4-U4-i386-disc4.is

• From http://oss.oracle.com/projects/ocfs2
Your specific ocfs2 kernel version using rpm

2. Configure the First Virtual Machine

To create and configure the first virtual machine, you will add virtual hardware devices such as disks and processors. Before proceeding with the install, create the windows folders to house the virtual machines and the shared storage.
E:\>mkdir E:\RAC\rac1
E:\>mkdir
E:\RAC\rac2
E:\>mkdir
E:\RAC\sharedstorage

Double-click on the VMware Server icon on your desktop to bring up the application:
1.  Press CTRL-N to create a new virtual machine.
2.  New Virtual Machine Wizard: Click on Next.
3.  Select the Appropriate Configuration:
        a.  Virtual machine configuration: Select Custom.
4.  Select a Guest Operating System:
        a.  Guest operating system: Select Linux.
        b.  Version: Select Red Hat Enterprise Linux 4.
5.  Name the Virtual Machine:
        a.  Virtual machine name: Enter “rac1.”
        b.  Location: Enter “d:\rac\rac1.”
6.  Set Access Rights:
        a.  Access rights: Select Make this virtual machine private.
7.  Startup / Shutdown Options:
        a.  Virtual machine account: Select User that powers on the virtual machine.
8.  Processor Configuration:
        a.  Processors: Select One.
9.  Memory for the Virtual Machine:
        a.  Memory: Select 700MB.
10.  Network Type:
        a.  Network connection: Select Use bridged networking.
11.  Select I/O Adapter Types:
        a.  I/O adapter types: Select LSI Logic.
12.  Select a Disk:
        a.  Disk: Select Create a new virtual disk.
13.  Select a Disk Type:
        a.  Virtual Disk Type: Select SCSI (Recommended).
14.  Specify Disk Capacity:
        a.  Disk capacity: Enter “20GB.”
        b.  Select Allocate all disk space now.
15.  Specify Disk File:
        a.  Disk file: Enter “localdisk.vmdk.”
b.  Click on Finish.
 

Now we will create the rest of shared disks:
16.  VMware Server Console: Click on Edit virtual machine settings.
17.  Virtual Machine Settings: Click on Add.
18.  Add Hardware Wizard: Click on Next.
19.  Hardware Type:
        a.  Hardware types: Select Hard Disk.
20.  Select a Disk:
        a.  Disk: Select Create a new virtual disk.
21.  Select a Disk Type:
        a.  Virtual Disk Type: Select SCSI (Recommended).
22.  Specify Disk Capacity:
        a.  Disk capacity: Enter “0.5GB.”
        b.  Select Allocate all disk space now.
23.  Specify Disk File:
        a.  Disk file: Enter “E:\RAC\sharedstorage\ocfs2disk.vmdk.”
        b.  Click on Advanced.
24.  Add Hardware Wizard:
a.  Virtual device node: Select SCSI 1:0.
b.  Mode: Select Independent, Persistent for all shared disks.
c.  Click on Finish.

Repeat steps 16 to 24 to create the rest of virtual SCSI hard disks - asmdisk1.vmdk(3GB), asmdisk2.vmdk (3GB), and asmdisk3.vmdk (2GB).


Finally, add an additional virtual network card for the private interconnects and remove the floppy drive, if any.
 25.  VMware Server Console: Click on Edit virtual machine settings.
26.  Virtual Machine Settings: Click on Add.
27.  Add Hardware Wizard: Click on Next.
28.  Hardware Type:
        a.  Hardware types: Ethernet Adapter.
29.  Network Type:
        a.  Host-only: A private network shared with the host
        b.  Click on Finish.
30.  Virtual Machine Settings:
        a.  Select Floppy and click on Remove.
31.  Add Hardware Wizard: Click on Next.
32. 
Hardware Type:
        a.  Hardware types: USB Controller
        b.  Click on Finish.
33.  Virtual Machine Settings: Click on OK.

At the end of this process, you should have something similar to:



Modify virtual machine configuration file.
Additional parameters are required to enable disk sharing between the two virtual RAC nodes. Open the configuration file, d:\rac\rac1\Red Hat EnterpriseLinux 4.vmx and add the bold parameters listed below.

config.version = "8"
virtualHW.version = "4"
scsi0.present = "TRUE"
scsi0.virtualDev = "lsilogic"
memsize = "600"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "localdisk.vmdk"
ide1:0.present = "TRUE"
ide1:0.fileName = "D:"
ide1:0.deviceType = "cdrom-raw"
floppy0.present = "FALSE"
Ethernet0.present = "TRUE"
displayName = "rac1"
guestOS = "rhel4"
priority.grabbed = "normal"
priority.ungrabbed = "normal"

disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
scsi1.sharedBus = "virtual"

scsi1.present = "TRUE"
scsi1:0.present = "TRUE"
scsi1:0.fileName = "E:\RAC\sharedstorage\ocfs2disk.vmdk"
scsi1:0.mode = "independent-persistent"
scsi1:0.deviceType = "disk"
scsi1:1.present = "TRUE"
scsi1:1.fileName = "E:\RAC\sharedstorage\asmdisk1.vmdk"
scsi1:1.mode = "independent-persistent"
scsi1:1.deviceType = "disk"
scsi1:2.present = "TRUE"
scsi1:2.fileName = "E:\RAC\sharedstorage\asmdisk2.vmdk"
scsi1:2.mode = "independent-persistent"
scsi1:2.deviceType = "disk"
scsi1:3.present = "TRUE"
scsi1:3.fileName = "E:\RAC\sharedstorage\asmdisk3.vmdk"
scsi1:3.mode = "independent-persistent"
scsi1:3.deviceType = "disk"
scsi1.virtualDev = "lsilogic"
ide1:0.autodetect = "FALSE"
Ethernet1.present = "TRUE"
Ethernet1.connectionType = "hostonly"
usb.present = "TRUE"
ide1:0.startConnected = "TRUE"
usb.autoConnect.device0 = ""

3. Install and Configure Enterprise Linux on the First Virtual
Machine
Download Enterprise Linux from Oracle and unzip the files:
Enterprise-R4-U4-i386-disc1.iso
Enterprise-R4-U4-i386-disc2.iso
Enterprise-R4-U4-i386-disc3.iso
Enterprise-R4-U4-i386-disc4.iso

1. On your VMware Server Console, double-click on the CD-ROM device on the right panel and select the ISO image for disk 1, Enterprise-R4-U4-i386-disc1.iso.
2. VMware Server console:
        Click on Start this virtual machine.
3. Hit Enter to install in graphical mode.
4. Skip the media test and start the installation.
5. Welcome to enterprise Linux: Click on Next.
6. Language Selection: <select your language preference>.
7. Keyboard Configuration: <select your keyboard preference>.
8. Installation Type: Custom.
9. Disk Partitioning Setup: Manually partition with Disk Druid.
        - Warning: Click on Yes to initialize each of the device – sda, sdb, sdc, sdd, and sde.
10.  Disk Setup: Allocate disk space on sda drive by double-clicking on /dev/sda free space for the mount points (/ and /u01) and swap space. You will configure the rest of the drives for OCFS2 and ASM later.
+ Add Partition:
Mount Point: /
File System Type: ext3
Start Cylinder: 1
End Cylinder: 910

File System Type: Swap
Start Cylinder: 911
End Cylinder: 1170

Mount Point: /u01
File System Type: ext3
Start Cylinder: 1171
End Cylinder: 2610




11. Boot Loader Configuration: Select only the default /dev/sda1 and leave the rest unchecked.
12. Network Configuration:
a. Network Devices
+ Select and edit eth0
1. De-select Configure Using DHCP.
2. Select Activate on boot.
3. IP Address: Enter “192.168.2.131
4. Netmask: Enter “255.255.255.0
+ Select and edit eth1
1. De-select Configure Using DHCP.
2. Select Activate on boot.
3. IP Address: Enter “10.10.10.31.”
4. Netmask: Enter “255.255.255.0.”
b. Hostname
Select manually and enter “rac1.mycorpdomain.com
c. Miscellaneous Settings
Gateway: Enter “192.168.2.1
Primary DNS: <optional>
Secondary DNS: <optional>
13.   Firewall Configuration:
a. Select No Firewall.
b. Enable SELinux?: Active.
14.   Warning – No Firewall: Click on Proceed.
15.   Additional Language Support: <select the desired language>.
16.   Time Zone Selection: <select your time zone>
17.   Set Root Password: <enter your root password>
18.   Package Group Selection:
a. Select X Window System.
b. Select GNOME Desktop Environment.
c. Select Editors.
Click on Details and select your preferred text editor.
d. Select Graphical Internet.
e. Select Text-based Internet.
f. Select Office/Productivity.
g. Select Sound and Video.
h. Select Graphics.
i. Select Server Configuration Tools.
j. Select FTP Server.
k. Select Legacy Network Server.
Click on Details.
1. Select rsh-server.
2. Select telnet-server.
l. Select Development Tools.
m. Select Legacy Software Development.
n. Select Administration Tools.
o. Select System Tools.
Click on Details. Select the following packages in addition to the default selected packages.
1. Select ocfs-2-2.6.9-42.0.0.0.1ELsmp
2. Select ocfs2-tools.
3. Select ocfs2console.
4. Select oracleasm-2.6.9-42.0.0.0.1ELsmp
5. Select sysstat.
p. Select Printing Support.
19.   About to Install: Click on Next.
20.   Required Install Media: Click on Continue.
21.   Change CD-ROM when needed: On your VMware Server Console, press CTRL-D to bring up the Virtual Machine Settings. Click on the CD-ROM device and select the ISO image for disk 2, then Enterprise-R4-U4-i386-disc3.iso.
22.   At the end of the installation:
a. On your VMware Server Console, press CTRL-D to bring up the Virtual Machine Settings. Click on the CD-ROM device and select Use physical drive.
b. Click on Reboot.
23.   Welcome: Click on Next.
24.   License Agreement: Select Yes, I agree to the License Agreement.
25.   Date and Time: Set the date and time.
26.   Display: <select your desired resolution>.
27.   System User: Leave the entries blank and click on Next.
28.   Additional CDs: Click on Next.
29.   Finish Setup: Click on Next

Now we finish the installation of  Enterprise Linux on VMware Server!


Installing VMware Tools within X
To install VMware Tools from X with the RPM installer:
  1. Choose [VM -> Install VMware Tools]. The guest operating system mounts the VMware Tools installation virtual CD.

  2. Double-click the VMware Tools CD icon on the desktop..

  3. Double-click the RPM installer in the root of the CD-ROM. VMwareTools-1.0.1-29996.i386.rpm

  4. Enter the root password (if prompted).

  5. Click [Continue]. The installer prepares the packages.

  6. Click [Continue] when the installer presents a dialog box saying "Completed System Preparation". A dialog appears for Updating system, with a progress bar. When the installer is done, VMware Tools are installed. There is no confirmation or finish button.

  7. In an X terminal, as root (su -), configure VMware Tools.
    # vmware-config-tools.pl
    Respond to any questions the installer displays on the screen. At the end of the configuration process, the program asks for the new screen resolution. You should pick the same screen resolution you selected during the CentOS Enterprise Linux install. I used option 2 ("800x600").

    NOTE: Be sure to respond [yes] if the installer offers to run the configuration program.

  8. After the configuration is complete, you must now reboot the virtual machine.
    # init 6

  9. After the virtual machine is rebooted and you log back in, you will notice that you no longer have to hit Ctrl-Alt to move between the virtual machine and the host operating system. Also, you will see that the mouse works more smoothly.

  10. Once you have verified the VMware Tools is installed and configured correctly, unmount the VMware Tools installation virtual CD. Choose [VM -> Cancel VMware Tools Install].

Synchronize Guest OS time with Host OS
When installing the Oracle Clusterware and Oracle Database software, the Oracle installer will initially install the software on the local node and then remotely copies the software to the remote node. If the date and time of both RAC nodes are not synchronized, you will likely receive errors similar to the one below:
"/bin/tar: ./inventory/Components21/oracle.ordim.server/10.2.0.1.0: time stamp  2006-11-04 06:24:04 is 25 s in the future"
To ensure a successful Oracle RAC installation, the time on the virtual machines has to synchronize with the host machine. Perform the steps below to synchronize the time as the root user.
1. Execute “vmware-toolbox” to bring up the VMware Tools Properties window. Under the Options tab, select Time synchronization between the virtual machine and the host operating system.
You should find the tools.syncTime = "TRUE" parameter appended to the virtual machine configuration file, E:\rac\rac1\Red Hat Enterprise Linux 4.vmx.
2. Edit /boot/grub/grub.conf and add the options, "clock=pit nosmp noapic nolapic" to the line that reads kernel /boot/. You have added the options to both kernels. You are only required to make the change to your specific kernel:
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title Enterprise (2.6.9-42.0.0.0.1.ELsmp)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.ELsmp ro root=LABEL=/ rhgb quiet clock=pit nosmp noapic nolapic
        initrd /boot/initrd-2.6.9-42.0.0.0.1.ELsmp.img
title Enterprise-up (2.6.9-42.0.0.0.1.EL)
        root (hd0,0)
        kernel /boot/vmlinuz-2.6.9-42.0.0.0.1.EL ro root=LABEL=/ rhgb quiet clock=pit nosmp noapic nolapic
        initrd /boot/initrd-2.6.9-42.0.0.0.1.EL.img
3. Reboot rac1.
# reboot

Setting the Correct Date and Time on Both Oracle RAC Nodes
During the installation of Oracle Clusterware, the Database, and the Companion CD, the Oracle Universal Installer (OUI) first installs the software to the local node running the installer (i.e. rac1). The software is then copied remotely to all of the remaining nodes in the cluster (i.e. rac2). During the remote copy process, the OUI will execute the UNIX "tar" command on each of the remote nodes to extract the files that were archived and copied over. If the date and time on the node performing the install is greater than that of the node it is copying to, the OUI will throw an error from the "tar" command indicating it is attempting to extract files stamped with a time in the future:

Error while copying directory 
/u01/app/oracle/product/crs with exclude file list 'null' to nodes 'rac2'.
[PRKC-1002 : All the submitted commands did not execute successfully]
---------------------------------------------
rac2:
/bin/tar: ./bin/lsnodes: time stamp 2007-02-19 09:21:34 is 735 s in the future
/bin/tar: ./bin/olsnodes: time stamp 2007-02-19 09:21:34 is 735 s in the future
...(more errors on this node)

Please note that although this would seem like a severe error from the OUI, it can safely be disregarded as a warning. The "tar" command DOES actually extract the files; however, when you perform a listing of the files (using ls -l) on the remote node, they will be missing the time field until the time on the server is greater than the timestamp of the file.
Before starting any of the above noted installations, ensure that each member node of the cluster is set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.
Accessing a Network Time Protocol server, however, may not always be an option. In this case, when manually setting the date and time for the nodes in the cluster, ensure that the date and time of the node you are performing the software installations from (rac1) is less than all other nodes in the cluster (rac2). I generally use a 20 second difference as shown in the following example:

Setting the date and time from rac1:

# date -s "2/19/2007 23:00:00"

Setting the date and time from rac2:

# date -s "2/19/2007 23:00:20"

The two-node RAC configuration described in this article does not make use of a Network Time Protocol server.



Create the oracle user.
As the root user, execute:

groupadd -g 116 oinstall
groupadd -g 115 dba
mkdir -p /export/home/oracle /ocfs
useradd -u 175 -d /export/home/oracle -g oinstall -G dba -s /bin/bash oracle
chown oracle:dba /export/home/oracle /u01
passwd oracle
New Password:
Re-enter new Password:
passwd: password successfully changed for oracle

I will be using the Oracle Cluster File System, Release 2 (OCFS2) to store the files required to be shared for the Oracle Clusterware software. When using OCFS2, the UID of the UNIX user "oracle" and GID of the UNIX group "dba" must be the same on all machines in the cluster. If either the UID or GID are different, the files on the OCFS2 file system will show up as "unowned" or may even be owned by a different user. For this article, I will use 175 for the "oracle" UID and 115 for the "dba" GID.


Create the oracle user environment file

su - oracle

vi /export/home/oracle/.bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
      . ~/.bashrc
fi

# User specific environment and startup programs
export PS1="`/bin/hostname -s`-> "
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/10.2.0/crs_1
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:$ORACLE_HOME/rdbms/admin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export EDITOR=vi

# Each RAC node must have a unique ORACLE_SID
export ORACLE_SID=devdb1

export JAVA_HOME=/usr/local/java
export PATH=.:${JAVA_HOME}/bin:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp

Create the filesystem directory structure. As the oracle user, execute
mkdir   -p $ORACLE_BASE/admin
mkdir   -p $ORACLE_HOME
mkdir   -p $ORA_CRS_HOME
mkdir -p /u01/oradata/devdb


Modifying the Shell Limits
Use a text editor and add the lines listed below to the following files:
To improve the performance of the software on Linux systems, Oracle recommends you increase the following shell limits for the oracle user:

vi /etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536

vi /etc/pam.d/login
session required /lib/security/pam_limits.so

vi /etc/profile
if [ $USER = "oracle" ]; then
   if [ $SHELL = "/bin/ksh" ]; then
      ulimit -p 16384
      ulimit -n 65536
   else
      ulimit -u 16384 -n 65536
   fi
fi



Install Enterprise Linux software packages
The following additional packages are required for Oracle software installation. If you have installed the 64-bit version of Enterprise Linux, the installer should have already installed these packages.
Extract the packages from the ISO CDs and execute the command below as the root user.
rpm -Uvh *.rpm
warning: libaio-0.3.105-2.i386.rpm: V3 DSA signature: NOKEY, key ID b38a8516
Preparing...               
########################################### [100%]
   1:openmotif21           
########################################### [ 50%]
   2:libaio               
########################################### [100%]



Configure the kernel parameters

As root, add the lines listed below to /etc/sysctl.conf.

vi /etc/sysctl.conf
kernel.shmall                = 2097152
kernel.shmmax                = 2147483648
kernel.shmmni                = 4096
kernel.sem                   = 250 32000 100 128
fs.file-max                  = 65536
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default        = 1048576
net.core.rmem_max            = 1048576
net.core.wmem_default        = 262144
net.core.wmem_max            = 262144

To make the changes effective immediately, execute /sbin/sysctl –p



Now an explanation of all these changes, if you don't want to understand the reason of this changes, go to the following section


Setting Shared Memory
Shared memory allows processes to access common structures and data by placing them in a shared memory segment. This is the fastest form of Inter-Process Communications (IPC) available - mainly due to the fact that no kernel involvement occurs when data is being passed between the processes. Data does not need to be copied between processes.
Oracle makes use of shared memory for its Shared Global Area (SGA) which is an area of memory that is shared by all Oracle backup and foreground processes. Adequate sizing of the SGA is critical to Oracle performance since it is responsible for holding the database buffer cache, shared SQL, access paths, and so much more.
To determine all shared memory limits, use the following:
# ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

Setting SHMMAX

The SHMMAX parameters defines the maximum size (in bytes) for a shared memory segment. The Oracle SGA is comprised of shared memory and it is possible that incorrectly setting SHMMAX could limit the size of the SGA. When setting SHMMAX, keep in mind that the size of the SGA should fit within one shared memory segment. An inadequate SHMMAX setting could result in the following:
ORA-27123: unable to attach to shared memory segment

You can determine the value of SHMMAX by performing the following:

# cat /proc/sys/kernel/shmmax
33554432
The default value for SHMMAX is 32MB. This is often too small to configure the Oracle SGA. I generally set the SHMMAX parameter to 2GB using the following methods:

Setting SHMMNI

We now look at the SHMMNI parameters. This kernel parameter is used to set the maximum number of shared memory segments system wide. The default value for this parameter is 4096.
You can determine the value of SHMMNI by performing the following:
# cat /proc/sys/kernel/shmmni
4096
The default setting for SHMMNI should be adequate for our Oracle10g Release 2 RAC installation.

Setting SHMALL

Finally, we look at the SHMALL shared memory kernel parameter. This parameter controls the total amount of shared memory (in pages) that can be used at one time on the system. In short, the value of this parameter should always be at least:
ceil(SHMMAX/PAGE_SIZE)
The default size of SHMALL is 2097152 and can be queried using the following command:
# cat /proc/sys/kernel/shmall
2097152
The default setting for SHMALL should be adequate for our Oracle10g Release 2 RAC installation.

  The page size in Red Hat Linux on the i386 platform is 4096 bytes. You can, however, use bigpages which supports the configuration of larger memory page sizes.


Setting Semaphores

Now that we have configured our shared memory settings, it is time to take care of configuring our semaphores. The best way to describe a semaphore is as a counter that is used to provide synchronization between processes (or threads within a process) for shared resources like shared memory. Semaphore sets are supported in System V where each one is a counting semaphore. When an application requests semaphores, it does so using "sets".

To determine all semaphore limits, use the following:

# ipcs -ls

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
You can also use the following command:
# cat /proc/sys/kernel/sem
250 32000 32 128

Setting SEMMSL

The SEMMSL kernel parameter is used to control the maximum number of semaphores per semaphore set.
Oracle recommends setting SEMMSL to the largest PROCESS instance parameter setting in the init.ora file for all databases on the Linux system plus 10. Also, Oracle recommends setting the SEMMSL to a value of no less than 100.

Setting SEMMNI

The SEMMNI kernel parameter is used to control the maximum number of semaphore sets in the entire Linux system.
Oracle recommends setting the SEMMNI to a value of no less than 100.

Setting SEMMNS

The SEMMNS kernel parameter is used to control the maximum number of semaphores (not semaphore sets) in the entire Linux system.
Oracle recommends setting the SEMMNS to the sum of the PROCESSES instance parameter setting for each database on the system, adding the largest PROCESSES twice, and then finally adding 10 for each Oracle database on the system.
Use the following calculation to determine the maximum number of semaphores that can be allocated on a Linux system. It will be the lesser of:
SEMMNS  -or-  (SEMMSL * SEMMNI)

Setting SEMOPM

The SEMOPM kernel parameter is used to control the number of semaphore operations that can be performed per semop system call.
The semop system call (function) provides the ability to do operations for multiple semaphores with one semop system call. A semaphore set can have the maximum number of SEMMSL semaphores per semaphore set and is therefore recommended to set SEMOPM equal to SEMMSL.
Oracle recommends setting the SEMOPM to a value of no less than 100.

Setting Semaphore Kernel Parameters

Finally, we see how to set all semaphore parameters. In the following, the only parameter I care about changing (raising) is SEMOPM. All other default settings should be sufficient for our example installation.


Setting File Handles

When configuring the Red Hat Linux server, it is critical to ensure that the maximum number of file handles is large enough. The setting for file handles denotes the number of open files that you can have on the Linux system.
Use the following command to determine the maximum number of file handles for the entire system:
# cat /proc/sys/fs/file-max
102563

Oracle recommends that the file handles for the entire system be set to at least 65536.

  You can query the current usage of file handles by using the following:
# cat /proc/sys/fs/file-nr
825 0 65536
The file-nr file displays three parameters:
  • Total allocated file handles
  • Currently used file handles
  • Maximum file handles that can be allocated

  If you need to increase the value in /proc/sys/fs/file-max, then make sure that the ulimit is set properly. Usually for Linux 2.4 and 2.6 it is set to unlimited. Verify the ulimit setting my issuing the ulimit command:
# ulimit
unlimited


Setting IP Local Port Range

Configure the system to allow a local port range of 1024 through 65000.
Use the following command to determine the value of ip_local_port_range:
# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
The default value for ip_local_port_range is ports 32768 through 61000. Oracle recommends a local port range of 1024 to 65000.




Modify the /etc/hosts file
Each node should have one static IP address for the public network and one static IP address for the private cluster interconnect. The private interconnect should only be used by Oracle to transfer Cluster Manager and Cache Fusion related data. Although it is possible to use the public network for the interconnect, this not recommended as it may cause degraded database performance (reducing the amount of bandwidth for Cache Fusion and Cluster Manager traffic).

vi /etc/hosts
127.0.0.1               localhost
# Public Network - (eth0)
192.168.2.131           rac1.mycorpdomain.com        rac1

192.168.2.132           rac2.mycorpdomain.com        rac2

# Public Virtual IP (VIP) addresses for - (eth0)
192.168.2.31            rac1-vip.mycorpdomain.com    rac1-vip
192.168.2.32            rac2-vip.mycorpdomain.com    rac2-vip

# Private Interconnect - (eth1)
10.10.10.31             rac1-priv.mycorpdomain.com   rac1-priv
10.10.10.32             rac2-priv.mycorpdomain.com   rac2-priv

Once the network if configured, you can use the ifconfig command to verify everything is working. The following example is from rac1:
$ /sbin/ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0C:29:07:E6:0B
inet addr:192.168.1.111 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe07:e60b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:170 errors:0 dropped:0 overruns:0 frame:0
TX packets:146 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14360 (14.0 KiB) TX bytes:11875 (11.5 KiB)
Interrupt:185 Base address:0x1400

eth1 Link encap:Ethernet HWaddr 00:0C:29:07:E6:15
inet addr:192.168.2.111 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe07:e615/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:197 errors:0 dropped:0 overruns:0 frame:0
TX packets:21 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:14618 (14.2 KiB) TX bytes:1386 (1.3 KiB)
Interrupt:169 Base address:0x1480

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1962 errors:0 dropped:0 overruns:0 frame:0
TX packets:1962 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:3226318 (3.0 MiB) TX bytes:3226318 (3.0 MiB)

sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)


About Virtual IP

Why do we have a Virtual IP (VIP) in 10g? Why does it just return a dead connection when its primary node fails?

It's all about availability of the application. When a node fails, the VIP associated with it is supposed to be automatically failed over to some other node. When this occurs, two things happen.

  1. The new node re-arps the world indicating a new MAC address for the address. For directly connected clients, this usually causes them to see errors on their connections to the old address.

  2. Subsequent packets sent to the VIP go to the new node, which will send error RST packets back to the clients. This results in the clients getting errors immediately.

This means that when the client issues SQL to the node that is now down, or traverses the address list while connecting, rather than waiting on a very long TCP/IP time-out (~10 minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case of connect, the next address in tnsnames is used.

Going one step further is making use of Transparent Application Failover (TAF). With TAF successfully configured, it is possible to completely avoid ORA-3113 errors alltogether!

Without using VIPs, clients connected to a node that died will often wait a 10 minute TCP timeout period before getting an error. As a result, you don't really have a good HA solution without using VIPs.

Source - Metalink: "RAC Frequently Asked Questions" (Note:220970.1)


Configure the hangcheck timer kernel module
The hangcheck timer kernel module monitors the system's health and restarts a failing RAC node. It uses two parameters, hangcheck_tick (defines the system checks frequency) and hangcheck_margin (defines the maximum hang delay before a RAC node is reset), to determine if a node is failing.
Add the following line in /etc/modprobe.conf to set the hangcheck kernel module parameters.

vi /etc/modprobe.conf
options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180

To load the module immediately, execute:
modprobe -v hangcheck-timer



Create disk partitions for OCFS2 and Oracle ASM shared storage
Prepare a set of raw disks for OCFS2 (/dev/sdb), and for Oracle ASM (/dev/sdc, /dev/sdd, /dev/sde).
The next step is to create a single partition on each of the five shared virtual drives. As mentioned earlier in this article, I will be using Oracle's Cluster File System, Release 2 (OCFS2) to store the two files to be shared for Oracle's Clusterware software. We will then be using Automatic Storage Management (ASM) to create two volumes; one for all physical database files (data/index files, online redo log files, and control files) and one for the Flash Recovery Area (RMAN backups and archived redo log files). We will create two ASM disk groups (+DG1 and +RECOVERYDEST) using NORMAL redundancy.
The fdisk command is used in Linux for creating (and removing) partitions
On rac1, as the root user, execute

Disk 1

fdisk /dev/sdb
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261): 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 2

fdisk /dev/sdc
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 3

fdisk /dev/sdd
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Disk 4

fdisk /dev/sde
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1566, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-1566, default 1566): 1566

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Verify new partitions
Use the fdisk -l command to verify the new partitions:
fdisk -l

Disk /dev/sda: 21.4 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 910 7309543+ 83 Linux
/dev/sda2 911 1170 2088450 82 Linux swap
/dev/sda3 1171 2610 11566800 83 Linux

Disk /dev/sdb: 536 MB, 536870912 bytes
64 heads, 32 sectors/track, 512 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 512 524272 83 Linux

Disk /dev/sdc: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 391 3140676 83 Linux

Disk /dev/sdd: 3221 MB, 3221225472 bytes
255 heads, 63 sectors/track, 391 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 391 3140676 83 Linux

Disk /dev/sde: 2147 MB, 2147483648 bytes
255 heads, 63 sectors/track, 261 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 261 2096451 83 Linux


Install oracleasmlib package
Download the ASM library from OTN and install the ASM RPM as the root user.

rpm -Uvh oracleasmlib-2.0.2-1.i386.rpm

Preparing...               
########################################### [100%]
   1:oracleasmlib         
########################################### [100%]

At this stage, you should already have the following ASM packages installed.
rpm -qa | grep oracleasm
oracleasm-support-2.0.3-2
oracleasm-2.6.9-42.0.0.0.1.ELsmp-2.0.3-2
oracleasmlib-2.0.2-1


Map raw devices for ASM disks
A raw device mapping is required only if you are planning on creating ASM disks using standard Linux I/O. An alternative to creating ASM disks is to use the ASM library driver provided by Oracle. You will configure ASM disks using ASM library driver later. Perform the following tasks to map the raw devices to the shared partitions created earlier. The raw devices have to bind with the block devices each time a cluster node boots.
Add the following lines in /etc/sysconfig/rawdevices.

vi /etc/sysconfig/rawdevices
/dev/raw/raw1 /dev/sdc1
/dev/raw/raw2 /dev/sdd1
/dev/raw/raw3 /dev/sde1

To make the mapping effective immediately, execute the following commands as the root user:
# /sbin/service rawdevices restart
Assigning devices:
           /dev/raw/raw1  -->   /dev/sdc1
/dev/raw/raw1:  bound to major 8, minor 33
           /dev/raw/raw2  -->   /dev/sdd1
/dev/raw/raw2:  bound to major 8, minor 49
           /dev/raw/raw3  -->   /dev/sde1
/dev/raw/raw3:  bound to major 8, minor 65
done

# chown oracle:dba /dev/raw/raw[1-3]
# chmod 660 /dev/raw/raw[1-3]

# ls -lat /dev/raw/raw*
crw-rw----  1 oracle dba 162, 3 Nov  4 07:04 /dev/raw/raw3
crw-rw----  1 oracle dba 162, 2 Nov  4 07:04 /dev/raw/raw2
crw-rw----  1 oracle dba 162, 1 Nov  4 07:04 /dev/raw/raw1


As the oracle user, execute
rac1-> ln -sf /dev/raw/raw1 /u01/oradata/devdb/asmdisk1
rac1-> ln -sf /dev/raw/raw2 /u01/oradata/devdb/asmdisk2
rac1-> ln -sf /dev/raw/raw3 /u01/oradata/devdb/asmdisk3


Modify the file /etc/udev/permissions.d/50-udev.permissions.
Raw devices are remapped on boot. The ownership of the raw devices will change to the root user by default upon boot. ASM will have problem accessing the shared partitions if the ownership is not the oracle user. Comment the original line,
“raw/*:root:disk:0660” in /etc/udev/permissions.d/50-udev.permissions
and add a new line,
“raw/*:oracle:dba:0660.”

vi /etc/udev/permissions.d/50-udev.permissions
# raw devices
ram*:root:disk:0660
#raw/*:root:disk:0660
raw/*:oracle:dba:0660


4. Create and Configure the Second Virtual Machine
To create the second virtual machine, simply shut down the first virtual machine (rac1), copy all the files in E:\RAC\rac1 to E:\RAC\rac2 and perform a few configuration changes.

Modify network configuration
1. As the root user on rac1,
# shutdown –h now
2. On your host system, copy all the files in rac1 folder to rac2.
E:\>copy E:\RAC\rac1 E:\RAC\rac2
3. On your VMware Server Console, press CTRL-O to open the second virtual machine,
E:\RAC\rac2\Red Hat Enterprise Linux 4.vmx.
4. Rename the virtual machine name from rac1 to rac2.
Right-click on the new rac1 tab you have just opened and select Settings.
- Select the Options tab.
1. Virtual machine name: Enter “rac2.” and click OK

5. Click on Start this virtual machine to start rac2, leaving rac1 powered off.
6. rac2 – Virtual Machine: Select Create a new identifier.

7. Log in as the root user and execute
system-config-network &

to modify the network configuration.

- IP Address: Double-click on each of the Ethernet devices and use the table below to make the necessary changes.
Device IP Address Subnet mask Default gateway
eth0 192.168.2.132 255.255.255.0 192.168.2.1
eth1 10.10.10.32 255.255.255.0 <leave empty>

- MAC Address: Navigate to the Hardware Device tab and probe for a new MAC address for each of the Ethernet device.
- Hostname and DNS: Use the table below to make the necessary changes to the entries in the DNS tab and press CTRL-S to save
Hostname Primary DNS
Secondary DNS
DNS with search Path
rac2.mycorpdomain.com Enter your DNS IP address or leave it empty Enter your DNS IP address or leave it empty Accepts the default or leave it empty

8. Finally, Activate each of the Ethernet device


Modify /etc/hosts. Add the following entry in /etc/hosts.
127.0.0.1 localhost

VIPCA will attempt to use the loopback address later during the Oracle Clusterware software installation.

Modify /export/home/oracle/.profile. Replace the value of ORACLE_SID with devdb2.



Configure RAC Nodes for Remote Acces
During the Cluster Ready Services (CRS) and RAC installation, the Oracle Universal Installer (OUI) has to be able to copy the software as oracle to all RAC nodes without being prompted for a password. This can be configured using either SSH or RSH where SSH is the preferred method.. In Oracle 10g, this can be accomplished using ssh instead of rsh.
If the Oracle Universal Installer in 10g does not detect the presence of the secure shell tools (ssh and scp), it will attempt to use the remote shell tools instead (rsh and rcp)..
Installing Oracle Clusterware and the Oracle Database software is only performed from one node in a RAC cluster. When running the Oracle Universal Installer (OUI) on that particular node, it will use the ssh and scp commands (or rsh and rcp commands if using remote shell) to run remote commands on and copy files (the Oracle software) to all other nodes within the RAC cluster. The "oracle" UNIX user account on the node running the OUI (runInstaller) must be trusted by all other nodes in your RAC cluster. This means that you must be able to run the secure shell commands (ssh or scp) or the remote shell commands (rsh and rcp) on the Linux server you will be running the OUI from against all other Linux servers in the RAC cluster without being prompted for a password.

The first step is to decide which method of remote access to use - secure shell or remote shell. Both of them have their pros and cons. Remote shell, for example, is extremely easy to setup and configure. It takes fewer steps to construct and is always available in the terminal session when logging on to the trusted node (the node you will be performing the install from). The connection to the remote nodes, however, is not secure during the installation and any patching process. Secure shell on the other hand does provide a secure connection when installing and patching but does require a greater number of steps. It also needs to be enabled in the terminal session each time the oracle user logs in to the trusted node. The official Oracle documentation only describes the steps for setting up secure shell and is considered the preferred method.

Using the Secure Shell Method (ssh and scp)
To establish user equivalence, generate the user's public and private keys as the oracle user on both nodes. Power on rac1 and perform the following tasks on both nodes as oracle.

On rac1:

rac1-> mkdir ~/.ssh
rac1-> chmod 700 ~/.ssh
rac1-> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_rsa.
Your public key has been saved in /export/home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
87:54:4f:92:ba:ed:7b:51:5d:1d:59:5b:f9:44:da:b6 oracle@rac1.mycorpdomain.com

rac1->ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_dsa.
Your public key has been saved in /export/home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
31:76:96:e6:fc:b7:25:04:fd:70:42:04:1f:fc:9a:26 oracle@rac1.mycorpdomain.com   


On rac2:

rac2-> mkdir ~/.ssh
rac2-> chmod 700 ~/.ssh
rac2-> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_rsa.
Your public key has been saved in /export/home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
29:5a:35:ac:0a:03:2c:38:22:3c:95:5d:68:aa:56:66 oracle@rac2.mycorpdomain.com

rac2-> ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/export/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /export/home/oracle/.ssh/id_dsa.
Your public key has been saved in /export/home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
4c:b2:5a:8d:56:0f:dc:7b:bc:e0:cd:3b:8e:b9:5c:7c oracle@rac2.mycorpdomain.com

On rac1,
rac1-> cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
rac1-> cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
rac1-> ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
The authenticity of host 'rac2 (192.168.2.132)' can't be established.
RSA key fingerprint is 63:d3:52:d4:4d:e2:cb:ac:8d:4a:66:9f:f1:ab:28:1f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'rac2,192.168.2.132' (RSA) to the list of known hosts.
oracle@rac2's password:

rac1-> ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
oracle@rac2's password:

rac1-> scp ~/.ssh/authorized_keys rac2:~/.ssh/authorized_keys
oracle@rac2's password:
authorized_keys                           100% 1716     1.7KB/s   00:00

Test the connection on each node. Verify that you are not prompted for password when you run the following the second time.
ssh rac1 date
ssh rac2 date
ssh rac1-priv date
ssh rac2-priv date
ssh rac1.mycorpdomain.com date
ssh rac2.mycorpdomain.com date
ssh rac1-priv.mycorpdomain.com date
ssh rac2-priv.mycorpdomain.com date

Remove any stty Commands
When installing the Oracle software, any hidden files on the system (i.e. .bashrc, .cshrc, .profile) will cause the installation process to fail if they contain stty commands.

To avoid this problem, you must modify these files to suppress all output on STDERR as in the following examples:


Using the Remote Shell Method
The services provided by remote shell are disabled by default on most Linux systems. This section describes the tasks required for enabling and configuring user equivalence for use by the Oracle Universal Installer when commands should be run and files copied to the remote nodes in the cluster using the remote shell tools. The goal is to enable the Oracle Universal Installer to use rsh and rcp to run commands and copy files to a remote node without being prompted for a password. Please note that using the remote shell method for configuring user equivalence is not secure.

The rsh daemon validates users using the /etc/hosts.equiv file or the .rhosts file found in the user's (oracle's) home directory.

First, let's make sure that we have the rsh RPMs installed on both of the Oracle RAC nodes in the cluster:

# rpm -q rsh rsh-server
rsh-0.17-25.4
rsh-server-0.17-25.4
From the above, we can see that we have the rsh and rsh-server installed.

  If rsh is not installed, run the following command from the CD where the RPM is located:
# su -
# rpm -ivh rsh-0.17-25.4.i386.rpm rsh-server-0.17-25.4.i386.rpm

To enable the "rsh" and "rlogin" services, the "disable" attribute in the /etc/xinetd.d/rsh file must be set to "no" and xinetd must be reloaded. This can be done by running the following commands on both Oracle RAC nodes in the cluster:

# su -
# chkconfig rsh on
# chkconfig rlogin on
# service xinetd reload
Reloading configuration: [ OK ]
To allow the "oracle" UNIX user account to be trusted among the RAC nodes, create the /etc/hosts.equiv file on both Oracle RAC nodes in the cluster:
# su -
# touch /etc/hosts.equiv
# chmod 600 /etc/hosts.equiv
# chown root.root /etc/hosts.equiv
Now add all RAC nodes to the /etc/hosts.equiv file similar to the following example for both Oracle RAC nodes in the cluster:
# cat /etc/hosts.equiv
+rac1 oracle
+rac2 oracle
+rac1-priv oracle
+rac2-priv oracle

  In the above example, the second field permits only the oracle user account to run rsh commands on the specified nodes. For security reasons, the /etc/hosts.equiv file should be owned by root and the permissions should be set to 600. In fact, some systems will only honor the content of this file if the owner of this file is root and the permissions are set to 600.

  Before attempting to test your rsh command, ensure that you are using the correct version of rsh. By default, Red Hat Linux puts /usr/kerberos/sbin at the head of the $PATH variable. This will cause the Kerberos version of rsh to be executed.

I will typically rename the Kerberos version of rsh so that the normal rsh command will be used. Use the following:

# su -

# which rsh
/usr/kerberos/bin/rsh

# mv /usr/kerberos/bin/rsh /usr/kerberos/bin/rsh.original
# mv /usr/kerberos/bin/rcp /usr/kerberos/bin/rcp.original
# mv /usr/kerberos/bin/rlogin /usr/kerberos/bin/rlogin.original

# which rsh
/usr/bin/rsh

You should now test your connections and run the rsh command from the node that will be performing the Oracle Clusterware and 10g RAC installation. I will be using the node rac1 to perform all installs so this is where I will run the following commands from:

# su - oracle

$ rsh rac1 ls -l /etc/hosts.equiv
-rw------- 1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac1-priv ls -l /etc/hosts.equiv
-rw------- 1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac2 ls -l /etc/hosts.equiv
-rw------- 1 root root 78 Feb 19 18:28 /etc/hosts.equiv

$ rsh rac2-priv ls -l /etc/hosts.equiv
-rw------- 1 root root 78 Feb 19 18:28 /etc/hosts.equiv

  Unlike when using secure shell, no other actions or commands are needed to enable user equivalence using the remote shell. User equivalence will be enabled for the "oracle" UNIX user account after successfully logging in to a terminal session.


5. Configure Oracle Automatic Storage Management (ASM)

Oracle ASM is tightly integrated with Oracle Database and works with Oracle’s suite of data management tools. It simplifies database storage management and provides the performance of raw disk I/O.

Configure ASMLib
Configure the ASMLib as the root user on both nodes.

# /etc/init.d/oracleasm configure
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration:           [  OK  ]
Loading module "oracleasm":                                [  OK  ]
Mounting ASMlib driver filesystem:                         [  OK  ]
Scanning system for ASM disks:                             [  OK  ]

Create ASM disks
Create the ASM disks on rac1 node as the root user.
# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk:                   [  OK  ]
# /etc/init.d/oracleasm createdisk VOL2 /dev/sdd1
Marking disk "/dev/sdd1" as an ASM disk:                   [  OK  ]
# /etc/init.d/oracleasm createdisk VOL3 /dev/sde1
Marking disk "/dev/sde1" as an ASM disk:                   [  OK  ]

Verify that the ASM disks are visible from every node.
# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks:                      [  OK  ]

# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3


Disable SELinux (RHEL4 U2 and higher)
Users of RHEL4 U2 and higher (CentOS 4.4 is based on RHEL4 U4) are advised that OCFS2 currently does not work with SELinux enabled. If you are using RHEL4 U2 or higher (which includes us since we are using CentOS 4.4) you will need to disable SELinux (using tool system-config-securitylevel) to get the O2CB service to execute. To disable SELinux, run the "Security Level Configuration" GUI utility as root:
# /usr/bin/system-config-securitylevel &

This will bring up the following screen:


Now, click the SELinux tab and check off the "Enabled" checkbox. After clicking [OK], you will be presented with a warning dialog. Simply acknowledge this warning by clicking "Yes". Your screen should now look like the following after disabling the SELinux option:


After making this change on both nodes in the cluster, each node will need to be rebooted to implement the change. SELinux must be disabled before you can continue with configuring OCFS2!
# init 6


6. Configure Oracle Cluster File System (OCFS2)


OCFS2 is a general-purpose cluster file system developed by Oracle and integrated with the Enterprise Linux kernel. It enables all nodes to share files concurrently on the cluster file system and thus eliminates the need to manage raw devices. Here you will house the OCR and Voting Disk in the OCFS2 file system. Additional information on OCFS2 can be obtained from OCFS2 User’s Guide.
This step will create the file /etc/ocfs2/cluster.conf . This will need to be done on both Oracle RAC nodes in the cluster as the root user account:
You should already have the OCFS2 RPMs installed during the Enterprise Linux installation. Verify that the RPMs have been installed on both nodes as the oracle user.

rac1-> rpm -qa | grep ocfs
ocfs2-tools-1.2.2-2
ocfs2console-1.2.2-2
ocfs2-2.6.9-42.0.0.0.1.ELsmp-1.2.3-2

rac2-> rpm -qa | grep ocfs
ocfs2-tools-1.2.2-2
ocfs2console-1.2.2-2
ocfs2-2.6.9-42.0.0.0.1.ELsmp-1.2.3-2

Create the OCFS2 configuration file
As the root user on rac1, execute:
# ocfs2console

This will bring up the GUI as shown below:

1.  Select [Cluster] -> [Configure Nodes...]. This will start the OCFS Cluster Stack and bring up the "Node Configuration" dialog: "The cluster stack has been started”: Click on Close.
2.  On the "Node Configuration" dialog, click the [Add] button. This will bring up the "Add Node" dialog
3. In the Add Node: Add the following nodes and then click on Apply.
Name: rac1
IP Address: 192.168.2.131
IP Port: 7777
Name: rac2
IP Address: 192.168.2.132
IP Port: 7777
4. Exit the Application
5. After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to the following.
# more /etc/ocfs2/cluster.conf
node:
        ip_port = 7777
        ip_address = 192.168.2.131
        number = 0
        name = rac1
        cluster = ocfs2
node:
        ip_port = 7777
        ip_address = 192.168.2.132
        number = 1
        name = rac2
        cluster = ocfs2
cluster:
        node_count = 2
        name = ocfs2
6. Propagate the configuration file to rac2. You can rerun the steps above on rac2 to generate the configuration file or select Cluster, Propagate Configuration on the OCFS2 Console on rac1 to propagate the configuration file to rac2. In any case, verify the existance of the file /etc/ocfs2/cluster.conf on rac2.


Configure the O2CB driver
O2CB is a set of clustering services that manages the communication between the nodes and the cluster file system. Below is a description of the individual services:
Perform the procedure below on both nodes to configure O2CB to start on boot.

When prompted for a value for the heartbeat dead threshold, you have to specify a value higher than 7 to prevent the nodes from crashing due to the slow IDE disk drive. The heartbeat dead threshold is a variable used to calculate the fence time.
Fence time (seconds) = (heartbeat dead threshold -1) * 2

A fence time of 601 works well in our environment. The value of heartbeat dead threshold should be the same on both nodes.

As the root user, execute:
# /etc/init.d/o2cb unload
Stopping O2CB cluster ocfs2: OK
Unmounting ocfs2_dlmfs filesystem: OK
Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK

# /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot.  The current values will be shown in brackets ('[]').  Hitting
 without typing an answer will keep that current value.  Ctrl-C will abort.
Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Specify heartbeat dead threshold (>=7) [7]: 601
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting O2CB cluster ocfs2: OK


Format the file system
We can now start to make use of the partitions we created before.
If the O2CB cluster is offline, start it. The format operation needs the cluster to be online, as it needs to ensure that the volume is not mounted on some node in the cluster.
Earlier in this document, we created the directory /ocfs. This section contains the commands to create and mount the file system to be used for the Cluster Manager /ocfs
Before proceeding with formatting and mounting the file system, verify that O2CB is online on both nodes; O2CB heartbeat is currently inactive because the file system is not mounted.

# /etc/init.d/o2cb status
Module "configfs": Loaded
Filesystem "configfs": Mounted
Module "ocfs2_nodemanager": Loaded
Module "ocfs2_dlm": Loaded
Module "ocfs2_dlmfs": Loaded
Filesystem "ocfs2_dlmfs": Mounted
Checking O2CB cluster ocfs2: Online
Checking O2CB heartbeat: Not active

You are only required to format the file system on one node. As the root user on rac1, execute:

# ocfs2console &
1. OCFS2 Console: Select Tasks, Format.
2. Format:
Available devices: /dev/sdb1
Volume label: oracle
Cluster size: Auto
Number of node slots: 4
Block size: Auto
3. OCFS2 Console: CTRL-Q to quit.

Mount the file system
To mount the file system, execute the command below on both nodes as root.
# mount -t ocfs2 -o datavolume,nointr /dev/sdb1 /ocfs

If the mount was successful, you will simply get your prompt back. We should, however, run the following checks to ensure the file system is mounted correctly. Let's use the mount command to ensure that the new file system is really mounted. This should be performed on both nodes in the RAC cluster:
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /ocfs type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)

To mount the file system on boot, add the following line in /etc/fstab on both nodes.

# vi /etc/fstab
/dev/sdb1 /ocfs ocfs2 _netdev,datavolume,nointr 0 0

Notice the "_netdev" option for mounting this file system. The _netdev mount option is a must for OCFS2 volumes. This mount option indicates that the volume is to be mounted after the network is started and dismounted before the network is shutdown.

Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file system will be mounted during the boot process.
If you have been following along with the examples in this article, the actions to load the kernel module and mount the OCFS2 file system should already be enabled. However, we should still check those options by running the following on both nodes in the RAC cluster as the root user account:

rac1-> su -
# chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The flags that I have marked in bold should be set to "on".


Create Oracle Clusterware directory
Create the directory in OCFS2 file system where the OCR and Voting Disk will reside. On rac1,

# mkdir /ocfs/clusterware
# chown -R oracle:dba /ocfs
# chmod 755 /ocfs

You have completed the set up of OCFS2. Verify that you can read and write files on the shared cluster file system from both nodes


Reboot Both Nodes

Before starting the next section, this would be a good place to reboot both of the nodes in the RAC cluster.
When the machines come up, ensure that the cluster stack services are being loaded and the new OCFS2 file system is being mounted:
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/sda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sdb1 on /ocfs type ocfs2 (rw,_netdev,datavolume,nointr,heartbeat=local)


If you modified the O2CB heartbeat threshold, you should verify that it is set correctly:

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
601

To determine which version of OCFS2 is running, use:
# cat /proc/fs/ocfs2/version
OCFS2 1.2.4 Thu Feb 1 15:04:31 PST 2007 (build a821ad1645e42e94b93ec4904c40dd10)


7. Install Oracle Clusterware


The following download procedures only need to be performed on one node in the cluster!

In this section, we will be downloading and extracting the required software from Oracle to only one of the Linux nodes in the RAC cluster - namely rac1. This is the machine where I will be performing all of the Oracle installs from. The Oracle installer will copy the required software packages to all other nodes in the RAC configuration using the remote access method we setup
Login to the node that you will be performing all of the Oracle installations from as the "oracle" user account. In this example, I will be downloading the required Oracle software to rac1 and saving them to "/u01/staging" directory.

First, download the Oracle Clusterware Release 2 for Linux x86.

  Oracle Clusterware Release 2 (10.2.0.1.0)


Next, we need to download the Oracle Database 10g Release 2 (10.2.0.1.0) Software for Linux x86. This can be downloaded from the same page used to download the Oracle Clusterware Release 2 software:

  Oracle Database 10g Release 2 (10.2.0.1.0)

As the oracle user account, extract the three packages you downloaded to a temporary directory. In this example, I will use "/u01/staging"

Extract the Clusterware package as follows:

# su - oracle
rac1-> cd /u01/staging
rac1-> unzip 10201_clusterware_linux32.zip

Then extract the Oracle10g Database Software:

rac1-> cd /u01/staging
rac1-> unzip 10201_database_linux32.zip

Pre-Installation Tasks for Oracle10g Release 2
Perform the following checks on both Oracle RAC nodes in the cluster!
When installing the Linux O/S, you should verify that all required RPMs for Oracle are installed. All of the required RPMs are on the Linux CDs/ISOs.
The next pre-installation step is to run the Cluster Verification Utility (CVU). CVU is a command-line utility provided on the Oracle Clusterware installation media. It is responsible for performing various system checks to assist you with confirming the Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle Real Application Clusters installation.
The CVU only needs to be run from the node you will be performing the Oracle installations from (rac1 in this article).


Check Required RPMs

The following packages (keeping in mind that the version number for your Linux distribution may vary slightly) must be installed:

binutils-2.15.92.0.2-21
compat-db-4.1.25-9
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
compat-libgcc-296-2.96-132.7.2
control-center-2.8.0-12.rhel4.5
cpp-3.4.6-3
gcc-3.4.6-3
gcc-c++-3.4.6-3
glibc-2.3.4-2.25
glibc-common-2.3.4-2.25
glibc-devel-2.3.4-2.25
glibc-headers-2.3.4-2.25
glibc-kernheaders-2.4-9.1.98.EL
gnome-libs-1.4.1.2.90-44.1
libaio-0.3.105-2
libstdc++-3.4.6-3
libstdc++-devel-3.4.6-3
make-3.80-6.EL4
openmotif-2.2.3-10.RHEL4.5
openmotif21-2.1.30-11.RHEL4.6
pdksh-5.2.14-30.3
setarch-1.6-1
sysstat-5.0.5-11.rhel4
xscreensaver-4.18-5.rhel4.11

  Note that the openmotif RPM packages are only required to install Oracle demos. This article does not cover the installation of Oracle demos.

To query package information (gcc and glibc-devel for example), use the "rpm -q <PackageName> [, <PackageName>]" command as follows:

# rpm -q gcc glibc-devel
gcc-3.4.6-3
glibc-devel-2.3.4-2.25
If you need to install any of the above packages, use the "rpm -Uvh <PackageName.rpm>" command. For example, to install the GCC gcc-3.4.6-3 package, use:
# rpm -Uvh gcc-3.4.6-3.i386.rpm

Prerequisites for Using Cluster Verification Utility

JDK 1.4.2

You must have JDK 1.4.2 installed on your system before you can run CVU. If you do not have JDK 1.4.2 installed on your system, and you attempt to run CVU, you will receive an error message similar to the following:
ERROR. Either CV_JDKHOME environment variable should be set
or /stagepath/cluvfy/jrepack.zip should exist.
If you do not have JDK 1.4.2 installed, then download it from the Sun Web site, and use the Sun instructions to install it. JDK 1.4.2 is available as a download from the following Web site: http://www.sun.com/java.

If you do have JDK 1.4.2 installed, then you must define the user environment variable CV_JDKHOME for the path to the JDK. For example, if JDK 1.4.2 is installed in /usr/local/j2re1.4.2_08, then log in as the user that you plan to use to run CVU, and enter the following commands:

CV_JDKHOME=/usr/local/j2re1.4.2_08
export CV_JDKHOME

Install cvuqdisk RPM (RHEL Users Only) in both nodes

The second pre-requisite for running the CVU is for Red Hat Linux users. If you are using Red Hat Linux, then you must download and install the Red Hat operating system package cvuqdisk to both of the Oracle RAC nodes in the cluster. This means you will need to install the cvuqdisk RPM to both rac1 and rac2. Without cvuqdisk, CVU will be unable to discover shared disks, and you will receive the error message "Package cvuqdisk not installed" when you run CVU.

The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm directory. For the purpose of this article, the Oracle Clusterware media was extracted to the /u01/staging/clusterware directory on  rac1. Note that before installing the cvuqdisk RPM, we need to set an environment variable named CVUQDISK_GRP to point to the group that will own the cvuqdisk utility. The default group is oinstall which is not the group we are using for the oracle UNIX user account in this article. Since we are using the dba group, we will need to set CVUQDISK_GRP=dba before attempting to install the cvuqdisk RPM.

Locate and copy the cvuqdisk RPM from rac1 to rac2 then perform the following steps on both Oracle RAC nodes to install:

$ su -
# cd /u01/staging/clusterware/rpm
# CVUQDISK_GRP=dba; export CVUQDISK_GRP

# rpm -iv cvuqdisk-1.0.1-1.rpm
Preparing packages for installation...
cvuqdisk-1.0.1-1

# ls -l /usr/sbin/cvuqdisk
-rwsr-x--- 1 root dba 4168 Jun 2 2005 /usr/sbin/cvuqdisk

Verify Remote Access / User Equivalence

The CVU should be run from rac1— the node we will be performing all of the Oracle installations from. Before running CVU, login as the oracle user account and verify remote access / user equivalence is configured to all nodes in the cluster. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:
# su - oracle
rac1-> exec /usr/bin/ssh-agent $SHELL
rac1-> /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa (/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa (/u01/app/oracle/.ssh/id_dsa)
When using the remote shell method, user equivalence is generally defined in the /etc/hosts.equiv file for the oracle user account and is enabled on all new terminal shell sessions.
Checking Pre-Installation Tasks for CRS with CVU
Once all prerequisites for using CVU have been met, we can start by checking that all pre-installation tasks for Oracle Clusterware (CRS) are completed by executing the following command as the "oracle" UNIX user account (with user equivalence enabled) from rac1:
rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre crsinst -n rac1,rac2 -verbose | more
Review the CVU report. Note that there are several errors you may ignore in this report.

The first error is with regards to membership of the user "oracle" in group "oinstall" [as Primary]. For the purpose of this article, the "oracle" user account will only be assigned to the "dba" group so this error can be safely ignored.

The second error is with regards to finding a suitable set of interfaces for VIPs. This is a bug documented in Metalink Note 338924.1:

Suitable interfaces for the private interconnect on subnet "192.168.1.0":
rac2 eth0:192.168.2.132
rac1 eth0:192.168.2.131

Suitable interfaces for the private interconnect on subnet "192.168.2.0":
rac2 eth1:192.168.2.32
rac1 eth1:192.168.2.31

ERROR:
Could not find a suitable set of interfaces for VIPs.

Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored.

The last set of errors that can be ignored deal with specific RPM package versions that do not exist in this Linux version For example:

While these specific packages are listed as missing in the CVU report, please ensure that the correct versions of the compat-* packages are installed on both of the Oracle RAC nodes in the cluster. For example, these would be:

Checking the Hardware and Operating System Setup with CVU
The next CVU check to run will verify the hardware and operating system setup. Again, run the following as the "oracle" UNIX user account from rac1:
$ cd /u01/staging/clusterware/cluvfy
$ ./runcluvfy.sh stage -post hwos -n rac1,rac2 -verbose | more
Review the CVU report. As with the previous check (pre-installation tasks for CRS), the check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored.

Also note that the check for shared storage accessibility will fail.

Checking shared storage accessibility...

WARNING:
Unable to determine the sharedness of /dev/sdf on nodes:
rac2,rac2,rac2,rac2,rac2,rac1,rac1,rac1,rac1,rac1


Shared storage check failed on nodes "rac2,rac1".
This too can be safely ignored. While we know the disks are visible and shared from both of our Oracle RAC nodes in the cluster, the check itself fails. Several reasons for this have been documented. The first came from Metalink indicating that cluvfy currently does not work with devices other than SCSI devices. This would include devices like EMC PowerPath and volume groups like those from Openfiler. At the time of this writing, no workaround exists other than to use manual methods for detecting shared devices. Another reason for this error was documented by Bane Radulovic at Oracle Corporation. His research shows that CVU calls smartclt on Linux, and the problem is that smartclt does not return the serial number from our virtual SCSI devices. The virtual SCSI devices from VMware so not support SMART. For example, a check against /dev/sdc shows:
# /usr/sbin/smartctl -i /dev/sdc
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: VMware, VMware Virtual S Version: 1.0
Device type: disk
Local Time is: Mon Feb 19 19:48:15 2007 EST
Device does not support SMART


Install Oracle10g Clusterware Software

So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster and database configuration metadata along with several system management features for RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the cluster. During normal operation, Oracle Clusterware will send messages (via a special ping operation) to all nodes configured in the cluster - often called the heartbeat. If the heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration files (on the shared disk) to distinguish between a real node failure and a network failure.

After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install the Oracle10g database software (next section) will automatically recognize these nodes. Like the Oracle Clusterware install we will be performing in this section, the Oracle10g database software only needs to be run from one node. The OUI will copy the software packages to all nodes configured in the RAC cluster.

Oracle Clusterware Shared Files

The two shared files used by Oracle Clusterware will be stored on the Oracle Cluster File System, Release 2 (OFCS2) we created earlier. The two shared Oracle Clusterware files are:

  It is not possible to use Automatic Storage Management (ASM) for the two shared Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files. The problem is that these files need to be in place and accessible BEFORE any Oracle instances can be started. For ASM to be available, the ASM instance would need to be run first.

  The two shared files could be stored on the OCFS2, shared RAW devices, or another vendor's clustered file system.


Verifying Terminal Shell Environment

Before starting the Oracle Universal Installer, you should first verify you are logged onto the server you will be running the installer from (i.e. rac1) then run the xhost command as root from the console to allow X Server connections. Next, login as the oracle user account. Finally, verify remote access / user equivalence to all nodes in the cluster:
# hostname
rac1

# xhost +
access control disabled, clients can connect from any host

Verify Remote Access / User Equivalence
# su - oracle
    Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell commands (rsh and rcp) on the Linux server you will be running the Oracle Universal Installer from against all other Linux servers in the cluster without being prompted for a password.
    When using the secure shell method, user equivalence will need to be enabled on any new terminal shell session before attempting to run the OUI. To enable user equivalence for the current terminal shell session, perform the following steps remembering to enter the pass phrase for each key that you generated when prompted:
rac1-> exec /usr/bin/ssh-agent $SHELL
rac1-> /usr/bin/ssh-add


Ready for the Oracle Clusterware Installation
As the oracle user on rac1, execute
rac1-> /u01/staging/clusterware/runInstaller
1. Welcome: Click on Next.
2. Specify Inventory directory and credentials:
Enter the full path of the inventory directory: /u01/app/oracle/oraInventory.
Specify Operating System group name: oinstall.
3. Specify Home Details:
Name: OraCrs10g_home
/u01/app/oracle/product/10.2.0/crs_1
4. Product-Specific Prerequisite Checks:
Ignore the warning on physical memory requirement.
5. Specify Cluster Configuration: Click on Add.
Public Node Name: rac2.mycorpdomain.com
Private Node Name: rac2-priv.mycorpdomain.com
Virtual Host Name: rac2-vip.mycorpdomain.com

6. Specify Network Interface Usage:
Interface Name: eth0
Subnet: 192.168.2.0
Interface Type: Public
Interface Name: eth1
Subnet: 10.10.10.0
Interface Type: Private

7. Specify Oracle Cluster Registry (OCR) Location: Select External Redundancy.
For simplicity, here you will not mirror the OCR. In a production environment, you may want to consider multiplexing the OCR for higher redundancy.
Specify OCR Location: /ocfs/clusterware/ocr
8. Specify Voting Disk Location: Select External Redundancy.
Similarly, for simplicity, we have chosen not to mirror the Voting Disk.
Voting Disk Location: /ocfs/clusterware/votingdisk
9. Summary: Click on Install.
10.   Execute Configuration scripts: Execute the scripts below as the root user sequentially, one at a time. Do not proceed to the next script until the current script completes.

Execute /u01/app/oracle/oraInventory/orainstRoot.sh on rac1 as root.
Execute /u01/app/oracle/oraInventory/orainstRoot.sh on rac2 as root.
Execute /u01/app/oracle/product/10.2.0/crs_1/root.sh on rac1 as root.
Execute /u01/app/oracle/product/10.2.0/crs_1/root.sh on rac2 as root.
The root.sh script on rac2 invoked the VIPCA automatically but it failed with the error "The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs."
As you are using a non-routable IP address (192.168.x.x) for the public interface, the Oracle Cluster Verification Utility (CVU) could not find a suitable public interface. A workaround (noted in Metalink article 338924.1) is to
run VIPCA manually.
11.   As the root user, manually invokes VIPCA on the second node.
# /u01/app/oracle/product/10.2.0/crs_1/bin/vipca
12.   Welcome: Click on Next.
13.   Network Interfaces: Select eth0.

14.   Virtual IPs for cluster nodes:
Node name: rac1
IP Alias Name: rac1-vip
IP address: 192.168.2.31
Subnet Mask: 255.255.255.0
Node name: rac2
IP Alias Name: rac2-vip
IP address: 192.168.2.32
Subnet Mask: 255.255.255.0

15.   Summary: Click on Finish


16.   Configuration Assistant Progress Dialog: After the configuration has completed, click on OK.
17.   Configuration Results: Click on Exit.
18.   Go back to the OUI on rac1 and acknowledge the "Execute Configuration scripts" dialog window and click on OK.
19.   At the end of the installation, exit from the OUI.

Verify Oracle Clusterware Installation
After the installation of Oracle Clusterware, we can run through several tests to verify the install was successful. Run the following commands on all nodes in the RAC cluster.

Verify that all checks are successful
The OUI does a Clusterware post-installation check at the end. If the CVU fails, correct the problem and re-run the following command as the oracle user:
rac1-> /u01/app/oracle/product/10.2.0/crs_1/bin/cluvfy stage -post crsinst -n rac1,rac2 | more

Check cluster nodes

rac1-> /u01/app/oracle/product/crs_1/bin/olsnodes -n
rac1 1
rac2 2
Check Oracle Clusterware Auto-Start Scripts
rac1-> ls -l /etc/init.d/init.*
-r-xr-xr-x 1 root root 1951 Feb 19 20:05 /etc/init.d/init.crs
-r-xr-xr-x 1 root root 4714 Feb 19 20:05 /etc/init.d/init.crsd
-r-xr-xr-x 1 root root 35394 Feb 19 20:05 /etc/init.d/init.cssd
-r-xr-xr-x 1 root root 3190 Feb 19 20:05 /etc/init.d/init.evmd

8. Install Oracle Database 10g Release 2

After successfully installing the Oracle Clusterware software, the next step is to install the Oracle10g Release 2 Database Software (10.2.0.1.0) with Real Application Clusters (RAC).
Perform the following action from ONE NODE only.

Run the Oracle Cluster Verification Utility
Before installing the Oracle Database Software, we should run the following database pre-installation check using the Cluster Verification Utility (CVU).
rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre dbinst -n rac1,rac2 -r 10gR2 -verbose | more
Review the CVU report. Note that this report will contain the same errors we received when checking pre-installation tasks for CRS — failure to find a suitable set of interfaces for VIPs and the failure to find specific RPM packages that do not exist in RHEL4 Update. These two errors can be safely ignored.

Install Oracle10g Release 2 Database Software
The following tasks are used to install the Oracle10g Release 2 Database Software:
rac1-> /u01/staging/database/runInstaller
1. Welcome: Click on Next.
2. Select Installation Type:
Select Enterprise Edition.
3. Specify Home Details:
Name: OraDb10g_home1
Path: /u01/app/oracle/product/10.2.0/db_1
4. Specify Hardware Cluster Installation Mode:
Select Cluster Installation.
Click on Select All. to select all servers: rac1 and rac2

5. Product-Specific Prerequisite Checks:
Ignore the warning on physical memory requirement.
6. Select Configuration Option:
Select: Install database Software only.
We will create the clustered database as a separate step using dbca
7. Summary: Click on Install.
8. Root Script Window: Execute the scripts below as the root user in all nodes.
Execute /u01/app/oracle/product/10.2.0/db_1/root.sh on rac1 as root.
Execute /u01/app/oracle/product/10.2.0/db_1/root.sh on rac2 as root.
9.   Return to the Execute Configuration scripts screen on rac1 and click on OK.
10.   End of Installation: Click on Exit.

Create TNS Listener Process

Perform the following configuration procedures from only one of the Oracle RAC nodes in the cluster (rac1)! The Network Configuration Assistant (NETCA) will setup the TNS listener in a clustered configuration on both of Oracle RAC nodes in the cluster.

The Database Configuration Assistant (DBCA) requires the Oracle TNS Listener process to be configured and running on all nodes in the RAC cluster before it can create the clustered database.
The process of creating the TNS listener only needs to be performed from one of the nodes in the RAC cluster. All changes will be made and replicated to both Oracle RAC nodes in the cluster. On one of the nodes (I will be using rac1) bring up the Network Configuration Assistant (NETCA) and run through the process of creating a new TNS listener process and to also configure the node for local access.

To start the NETCA, run the following:
rac1-> netca &

The following screenshots walk you through the process of creating a new Oracle listener for our RAC environment.

Screen Name Response
Select the Type of Oracle
Net Services Configuration
Select Cluster configuration
Select the nodes to configure Select all of the nodes: rac1 and rac2.
Type of Configuration Select Listener configuration.
Listener Configuration - Next 6 Screens The following screens are now like any other normal listener configuration. You can simply accept the default parameters for the next six screens:
   What do you want to do: Add
   Listener name: LISTENER
   Selected protocols: TCP
   Port number: 1521
   Configure another listener: No
   Listener configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Select Naming Methods configuration.
Naming Methods Configuration The following screens are:
   Selected Naming Methods: Local Naming
   Naming Methods configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Click Finish to exit the NETCA.


Verify TNS Listener Configuration
The Oracle TNS listener process should now be running on both nodes in the RAC cluster:
rac1-> hostname
rac1

rac1-> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_RAC1

=====================

rac1-> hostname
rac2

rac1-> ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print $9}'
LISTENER_RAC2

Create the Oracle Cluster Database

The database creation process should only be performed from one of the Oracle RAC nodes in the cluster (rac1)!
We will be using the Oracle Database Configuration Assistant (DBCA) to create the clustered database.
Before executing the Database Configuration Assistant, make sure that $ORACLE_HOME and $PATH are set appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment.
You should also verify that all services we have installed up to this point (Oracle TNS listener, Oracle Clusterware processes, etc.) are running before attempting to start the clustered database creation process.


Run the Oracle Cluster Verification Utility

Before creating the Oracle clustered database, we should run the following database configuration check using the Cluster Verification Utility (CVU).

rac1-> cd /u01/staging/clusterware/cluvfy
rac1-> ./runcluvfy.sh stage -pre dbcfg -n rac1,rac2 -d ${ORACLE_HOME} -verbose | more
Review the CVU report. Note that this report will contain the same error we received when checking pre-installation tasks for CRS — failure to find a suitable set of interfaces for VIPs. This error can be safely ignored.

Create the Clustered Database (this process takes 2 hours or so)
To start the database creation process, run the following as oracle on rac1:
rac1->  dbca &
Screen Name Response
Welcome Screen Select Oracle Real Application Clusters database.
Operations Select Create a Database.
Node Selection Click the Select All button to select all servers: rac1 and rac2.
Database Templates Select Custom Database
Database Identification Select:
   Global Database Name: devdb
   SID Prefix: devdb
Management Option Leave the default options here which is to Configure the Database with Enterprise Manager / Use Database Control for Database Management
Database Credentials I selected to Use the Same Password for All Accounts. Enter the password (twice) and make sure the password does not start with a digit number.
Storage Options For this article, we will select to use Automatic Storage Management (ASM).
Create ASM Instance Supply the SYS password to use for the new ASM instance. Also, starting with Oracle10g Release 2, the ASM instance server parameter file (SPFILE) needs to be on a shared disk. You will need to modify the default entry for "Create server parameter file (SPFILE)" to reside on the OCFS2 partition as follows: /ocfs/oradata/devdb/dbs/spfile+ASM.ora. All other options can stay at their defaults.

You will then be prompted with a dialog box asking if you want to create and start the ASM instance. Select the OK button to acknowledge this dialog.

The OUI will now create and start the ASM instance on all nodes in the RAC cluster.

ASM Disk Groups To start, click the Create New button. This will bring up the "Create Disk Group" window with the three volumes we configured earlier using ASMLib.

If the volumes we created earlier in this article do not show up in the "Select Member Disks" window: (ORCL:VOL1, ORCL:VOL2 and ORCL:VOL3) then click on the "Change Disk Discovery Path" button and input "ORCL:VOL*". You can also see the following:
/dev/raw/raw1
/dev/raw/raw2
/dev/raw/raw3

For the first "Disk Group Name", I used the string DG1. Select the first two ASM volumes (ORCL:VOL1 and ORCL:VOL2 OR /dev/raw/raw1 and /dev/raw/raw2 ) in the "Select Member Disks" window. Keep the "Redundancy" setting to Normal.

After verifying all values in this window are correct, click the OK button. This will present the "ASM Disk Group Creation" dialog. When the ASM Disk Group Creation process is finished, you will be returned to the "ASM Disk Groups" windows.

Click the Create New button again. For the second "Disk Group Name", I used the string RECOVERYDEST. Select the last volumes (ORCL:VOL3 OR /dev/raw/raw3) in the "Select Member Disks" window. Put the "Redundancy" setting to External.

After verifying all values in this window are correct, click the OK button. This will present the "ASM Disk Group Creation" dialog.

When the ASM Disk Group Creation process is finished, you will be returned to the "ASM Disk Groups" window with two disk groups created and selected. Select all of them and click Next to continue.

or

Database File Locations I selected to use the default which is Use Oracle-Managed Files:
   Database Area: +DG1
Recovery Configuration Check the option for Specify Flash Recovery Area.

For the Flash Recovery Area, click the [Browse] button and select the disk group name +RECOVERYDEST.

I used a Flash Recovery Area Size of 1500 MB.

Database Content Select or deselect the Sample Schemas..
Database Services Click on Next. You can always create or modify additional services later using DBCA or svctl
Initialization Parameters Select Custom.
Shared Memory Management: Automatic
SGA Size: 200MB
PGA Size: 25MB
Database Storage Change any parameters for your environment. I left them all at their default settings.
Creation Options Keep the default option Create Database selected and click Finish to start the database creation process.

Click OK on the "Summary" screen.

End of Database Creation At the end of the database creation, exit from the DBCA.

When exiting the DBCA, another dialog will come up indicating that it is starting all Oracle instances.This may take several minutes to complete. When finished, all windows and dialog boxes will disappear.

When the Oracle Database Configuration Assistant has completed, you will have a fully functional Oracle RAC cluster running!





9. Verify and Explore the RAC Cluster and RAC Database Environment

Now that you have successfully installed a virtual two-node RAC database, it’s time to do a little exploration of the environment you have just set up.
This section provides several srvctl commands and SQL queries that can be used to validate your Oracle10g RAC configuration
There are five node-level tasks defined for SRVCTL:


Check the status of application resources

crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.devdb.db   application    ONLINE    ONLINE    rac1
ora....b1.inst application    ONLINE    ONLINE    rac1
ora....b2.inst application    ONLINE    ONLINE    rac2
ora....SM1.asm application    ONLINE    ONLINE    rac1
ora....C1.lsnr application    ONLINE    ONLINE    rac1
ora.rac1.gsd   application    ONLINE    ONLINE    rac1
ora.rac1.ons   application    ONLINE    ONLINE    rac1
ora.rac1.vip   application    ONLINE    ONLINE    rac1
ora....SM2.asm application    ONLINE    ONLINE    rac2
ora....C2.lsnr application    ONLINE    ONLINE    rac2
ora.rac2.gsd   application    ONLINE    ONLINE    rac2
ora.rac2.ons   application    ONLINE    ONLINE    rac2
ora.rac2.vip   application    ONLINE    ONLINE    rac2

Status of all instances and services
srvctl status database -d devdb
Instance devdb1 is running on node rac1
Instance devdb2 is running on node rac2

Status of a single instance
srvctl status instance -d devdb -i devdb2
Instance devdb2 is running on node rac2

Status of node applications on a particular node
srvctl status nodeapps -n rac1
VIP is running on node: rac1
GSD is running on node: rac1
Listener is running on node: rac1
ONS daemon is running on node: rac1

srvctl status nodeapps -n rac2
VIP is running on node: rac2
GSD is running on node: rac2
Listener is running on node: rac2
ONS daemon is running on node: rac2

Status of an ASM instance
srvctl status asm -n rac1
ASM instance +ASM1 is running on node rac1.

srvctl status asm -n rac2
ASM instance +ASM2 is running on node rac2.   

List all configured databases
srvctl config database
devdb

Display configuration for our RAC database
srvctl config database -d devdb
rac1 devdb1 /u01/app/oracle/product/10.2.0/db_1
rac2 devdb2 /u01/app/oracle/product/10.2.0/db_1


Display the configuration for node applications - (VIP, GSD, ONS, Listener)
srvctl config nodeapps -n rac1 -a -g -s -l
VIP exists.: /rac1-vip/192.168.2.31/255.255.255.0/eth0
GSD exists.
ONS daemon exists.
Listener exists.


Display the configuration for the ASM instance(s)
srvctl config asm -n rac1
+ASM1 /u01/app/oracle/product/10.2.0/db_1


Check the status of Oracle Clusterware
rac1->crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy

rac2-> crsctl check crs
CSS appears healthy
CRS appears healthy
EVM appears healthy


Stopping the Oracle10g RAC Environment
The first step is to stop the Oracle instance. Once the instance (and related services) is down, then bring down the ASM instance. Finally, shutdown the node applications (Virtual IP, GSD, TNS Listener, and ONS).
$ export ORACLE_SID=devdb1
$ emctl stop dbconsole
$ srvctl stop instance -d devdb -i devdb1
$ srvctl stop asm -n rac1
$ srvctl stop nodeapps -n rac1


Starting the Oracle10g RAC Environment

The first step is to start the node applications (Virtual IP, GSD, TNS Listener, and ONS). Once the node applications are successfully started, then bring up the ASM instance. Finally, bring up the Oracle instance (and related services) and the Enterprise Manager Database console.
$ export ORACLE_SID=devdb1
$ srvctl start nodeapps -n rac1
$ srvctl start asm -n rac1
$ srvctl start instance -d devdb -i devdb1
$ emctl start dbconsole


Execute crsctl on the command line to check out all the available options.



List the RAC instances


SELECT inst_id, instance_number inst_no, instance_name inst_name, parallel,
status, database_status db_status, active_state state, host_name host
FROM gv$instance
ORDER BY inst_id;

INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST
-------- -------- ---------- --- ------- ------------ --------- --------
1 1 devdb1 YES OPEN ACTIVE NORMAL rac1
2 2 devdb2 YES OPEN ACTIVE NORMAL rac2

Check connectivity
Verify that you are able to connect to the instances and service on each node.
sqlplus system@devdb1
sqlplus system@devdb2
sqlplus system@devdb


Check database configuration

rac1->export ORACLE_SID=devdb1
rac1->sqlplus / as sysdba
show sga
Total System Global Area  209715200 bytes
Fixed Size                  1218556 bytes
Variable Size             104859652 bytes
Database Buffers          100663296 bytes
Redo Buffers                2973696 bytes

All data files which are in the disk group
select name from v$datafile
union
select member from v$logfile
union
select name from v$controlfile
union
select name from v$tempfile;


NAME
-------------------------------------------
++RECOVERYDEST/devdb/controlfile/current.256.578676737
++RECOVERYDEST/devdb/onlinelog/group_1.257.578676745
++RECOVERYDEST/devdb/onlinelog/group_2.258.578676759
++RECOVERYDEST/devdb/onlinelog/group_3.259.578682963
++RECOVERYDEST/devdb/onlinelog/group_4.260.578682987
++DG1/devdb/controlfile/current.256.578676735
++DG1/devdb/datafile/example.263.578676853
++DG1/devdb/datafile/indx.270.578685723
++DG1/devdb/datafile/sysaux.261.578676829
++DG1/devdb/datafile/system.259.578676767
++DG1/devdb/datafile/undotbs1.260.578676809
++DG1/devdb/datafile/undotbs1.271.578685941
++DG1/devdb/datafile/undotbs2.264.578676867
++DG1/devdb/datafile/undotbs2.272.578685977
++DG1/devdb/datafile/users.265.578676887
++DG1/devdb/datafile/users.269.578685653
++DG1/devdb/onlinelog/group_1.257.578676739
++DG1/devdb/onlinelog/group_2.258.578676753
++DG1/devdb/onlinelog/group_3.266.578682951
++DG1/devdb/onlinelog/group_4.267.578682977
++DG1/devdb/tempfile/temp.262.578676841

select file_name,bytes/1024/1024 from dba_data_files;
FILE_NAME                                   BYTES/1024/1024
------------------------------------------- ---------------
+DG1/devdb/datafile/users.259.606468449                   5
+DG1/devdb/datafile/sysaux.257.606468447                240
+DG1/devdb/datafile/undotbs1.258.606468449               30
+DG1/devdb/datafile/system.256.606468445                480
+DG1/devdb/datafile/undotbs2.264.606468677               25

select group#, type, member, is_recovery_dest_file
  from v$logfile
  order by group#;
GROUP# TYPE    MEMBER                                              IS_
------ ------- --------------------------------------------------- ---
     1 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_1.257.606468581 YES
     1 ONLINE  +DG1/devdb/onlinelog/group_1.261.606468575          NO
     2 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_2.258.606468589 YES
     2 ONLINE  +DG1/devdb/onlinelog/group_2.262.606468583          NO
     3 ONLINE  +DG1/devdb/onlinelog/group_3.265.606468865          NO
     3 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_3.259.606468875 YES
     4 ONLINE  +DG1/devdb/onlinelog/group_4.266.606468879          NO
     4 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_4.260.606468887 YES

rac1-> export ORACLE_SID=+ASM1
rac1-> sqlplus / as sysdba
SQL> show sga
Total System Global Area   92274688 bytes
Fixed Size                  1217884 bytes
Variable Size              65890980 bytes
ASM Cache                  25165824 bytes

SQL> show parameter asm_disk
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
asm_diskgroups                 string      DG1, RECOVERYDEST
asm_diskstring                 string

select group_number, name, allocation_unit_size alloc_unit_size, state,
         type, total_mb, usable_file_mb
  from v$asm_diskgroup;
                       ALLOC                        USABLE
 GROUP                  UNIT                 TOTAL    FILE
NUMBER NAME             SIZE STATE   TYPE       MB      MB
------ ------------ -------- ------- ------ ------ -------
     1 DG1           1048576 MOUNTED NORMAL   6134    1868
     2 RECOVERYDEST  1048576 MOUNTED EXTERN   2047    1713


select name, path, header_status, total_mb free_mb, trunc(bytes_read/1024/1024) read_mb,
         trunc(bytes_written/1024/1024) write_mb
from v$asm_disk;
NAME  PATH       HEADER_STATU    FREE_MB    READ_MB   WRITE_MB
----- ---------- ------------ ---------- ---------- ----------
VOL1  ORCL:VOL1  MEMBER             3067        229       1242
VOL2  ORCL:VOL2  MEMBER             3067        164       1242
VOL3  ORCL:VOL3  MEMBER             2047         11        354



Check flash recovery area space usage

select * from v$recovery_file_dest;

NAME          SPACE_LIMIT SPACE_USED SPACE_RECLAIMABLE NUMBER_OF_FILES
------------- ----------- ---------- ----------------- ---------------
+RECOVERYDEST  1572864000  331366400                 0               7

select * from v$flash_recovery_area_usage;
FILE_TYPE    PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
------------ ------------------ ------------------------- ---------------
CONTROLFILE                 .97                         0               1
ONLINELOG                    20                         0               6
ARCHIVELOG                    0                         0               0
BACKUPPIECE                   0                         0               0
IMAGECOPY                     0                         0               0
FLASHBACKLOG                  0                         0               0


Start and stop application resources
Follow the steps below to start and stop individual application resource.

srvctl start nodeapps -n <node1 hostname>
srvctl start nodeapps -n <node2 hostname>
srvctl start asm -n <node1 hostname>
srvctl start asm -n <node2 hostname>
srvctl start database -d <database name>
srvctl start service -d <database name> -s <service name>
crs_stat -t
srvctl stop service -d <database name> -s <service name>
srvctl stop database -d <database name>
srvctl stop asm -n <node1 hostname>
srvctl stop asm -n <node2 hostname>
srvctl stop nodeapps -n <node1 hostname>
srvctl stop nodeapps -n <node2 hostname>
crs_stat -t


10. Perform Some DB Operations

Create a tablespace

SQL> connect system/oracle@devdb
.
SQL> create tablespace test_d datafile '+DG1' size 10M;

The following query can be used to determine the file names for your environment:
select tablespace_name, file_name
from dba_data_files
union
select tablespace_name, file_name
from dba_temp_files;

TABLESPACE_NAME FILE_NAME
---------------- --------------------------------------------------
EXAMPLE
+DG1/devdb/datafile/example.263.578676853
SYSAUX
+DG1/devdb/datafile/sysaux.261.578676829
SYSTEM
+DG1/devdb/datafile/system.259.578676767
TEMP
+DG1/devdb/tempfile/temp.262.578676841
UNDOTBS1
+DG1/devdb/datafile/undotbs1.260.578676809
UNDOTBS2
+DG1/devdb/datafile/undotbs2.264.578676867
USERS
+DG1/devdb/datafile/users.265.578676887

select file_name, tablespace_name, bytes
  from dba_data_files
  where tablespace_name='TEST_D';
FILE_NAME                                TABLESPACE_NAME      BYTES
---------------------------------------- --------------- ----------
+DG1/devdb/datafile/test_d.269.606473423 TEST_D            10485760


$ sqlplus "/ as sysdba"

create user scott identified by tiger default tablespace users;
grant dba, resource, connect to scott;

alter database datafile '+DG1/devdb/datafile/users.265.578676887' resize 500m;
alter tablespace users add datafile '+DG1' size 250m autoextend off;

create tablespace indx datafile '+DG1' size 250m
autoextend on next 50m maxsize unlimited
extent management local autoallocate
segment space management auto;

alter database datafile '+DG1/devdb/datafile/system.259.578676767' resize 800m;

alter database datafile '+DG1/devdb/datafile/sysaux.261.578676829' resize 500m;

alter tablespace undotbs1 add datafile '+DG1' size 250m
autoextend on next 50m maxsize 2048m;

alter tablespace undotbs2 add datafile '+DG1' size 250m
autoextend on next 50m maxsize 2048m;

alter database tempfile '+DG1/devdb/tempfile/temp.262.578676841' resize 1024m;

Here is a snapshot of the tablespaces I have defined for my test database environment:

Status    Tablespace Name TS Type      Ext. Mgt.  Seg. Mgt.    Tablespace Size    Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- --------- ------------------ ------------------ ---------
ONLINE UNDOTBS1 UNDO LOCAL MANUAL 471,859,200 62,586,880 13
ONLINE SYSAUX PERMANENT LOCAL AUTO 524,288,000 278,724,608 53
ONLINE USERS PERMANENT LOCAL AUTO 786,432,000 131,072 0
ONLINE SYSTEM PERMANENT LOCAL MANUAL 838,860,800 502,726,656 60
ONLINE EXAMPLE PERMANENT LOCAL AUTO 157,286,400 83,820,544 53
ONLINE INDX PERMANENT LOCAL AUTO 262,144,000 65,536 0
ONLINE UNDOTBS2 UNDO LOCAL MANUAL 471,859,200 1,835,008 0
ONLINE TEMP TEMPORARY LOCAL MANUAL 1,073,741,824 27,262,976 3
------------------ ------------------ ---------
avg 23
sum 4,586,471,424 957,153,280

8 rows selected.

Create an online redo logfile group
SQL> connect system/oracle@devdb

alter database add logfile thread 1 group 5 size 50M;
alter database add logfile thread 2 group 6 size 50M;

select group#,thread#, bytes, members, status
from v$log;
    GROUP#    THREAD#      BYTES    MEMBERS STATUS
---------- ---------- ---------- ---------- ----------------
         1          1   52428800          2 CURRENT
         2          1   52428800          2 INACTIVE
         3          2   52428800          2 ACTIVE
         4          2   52428800          2 CURRENT
         5          1   52428800          2 UNUSED
         6          2   52428800          2 UNUSED

select group#, type, member, is_recovery_dest_file
 from v$logfile
 where group# in (5,6)
 order by group#;
GROUP# TYPE    MEMBER                                               IS_
------ ------- ---------------------------------------------------- ---
     5 ONLINE  +DG1/devdb/onlinelog/group_5.271.606473683           NO
     5 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_5.261.606473691  YES
     6 ONLINE  +DG1/devdb/onlinelog/group_6.272.606473697           NO
     6 ONLINE  +RECOVERYDEST/devdb/onlinelog/group_6.262.606473703  YES


11. Test Transparent Failover (TAF)

The failover mechanism in Oracle TAF enables any failed database connections to reconnect to another node within the cluster. The failover is transparent to the user. Oracle re-executes the query on the failed over instance and continues to display the remaining results to the user.

Create a new database service
Let’s begin by creating a new service called CRM. Database services can be created using either DBCA or the srvctl utility. Here you will use DBCA to create the CRM service on devdb1.
 
Service Name
Database Name
Preferred Instance
Available Instance
TAF Policy
CRM
devdb
devdb1
devdb2
BASIC

As the oracle user on rac1, execute

rac1-> dbca
1. Welcome: Select Oracle Real Application Clusters database.
2. Operations: Select Services Management.
3. List of cluster databases: Click on Next.
4. Database Services: Click on Add.
Add a Service: Enter “CRM.”
Select devdb1 as the Preferred instance.
Select devdb2 as the Available instance.
TAF Policy: Select Basic.
Click on Finish


5. Database Configuration Assistant: Click on No to exit


The Database Configuration Assistant creates the following CRM service name entry in tnsnames.ora.
Here is a FULL copy of it. You can include any of these entries on other client machines that need access to the clustered database.
LISTENERS_DEVDB =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
)

DEVDB2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = devdb)
(INSTANCE_NAME = devdb2)
)
)

DEVDB1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = devdb)
(INSTANCE_NAME = devdb1)
)
)

CRM =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = CRM)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)


DEVDB =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = devdb)
)
)

EXTPROC_CONNECTION_DATA =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
)
(CONNECT_DATA =
(SID = PLSExtProc)
(PRESENTATION = RO)
)
)



sqlplus system/oracle@devdb1
show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb, CRM

sqlplus system/oracle@devdb2
show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb
  
Connect the first session using the CRM service
If the returned output of failover_type and failover_mode is 'NONE', verify that the CRM service is configured correctly in tnsnames.ora.

sqlplus system/oracle@crm
select instance_number instance#, instance_name, host_name, status
   from v$instance;
    INSTANCE# INSTANCE_NAME    HOST_NAME             STATUS
   ---------- ---------------- --------------------- ------------
            1 devdb1           rac1.mycorpdomain.com OPEN
  
select failover_type, failover_method, failed_over
from v$session
where username='SYSTEM';
FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------------- ----------------
SELECT        BASIC           NO

Shut down the instance from another session
Connect as the sys user on CRM instance and shut down the instance.
rac1-> export ORACLE_SID=devdb1
rac1-> sqlplus / as sysdba
select instance_number instance#, instance_name, host_name, status
from v$instance;
INSTANCE# INSTANCE_NAME    HOST_NAME             STATUS
---------- ---------------- --------------------- ------------
         1 devdb1           rac1.mycorpdomain.com OPEN

shutdown abort;
ORACLE instance shut down.

Verify that the session has failed over
From the same CRM session you opened previously, execute the queries below to verify that the session has failed over to another instance.
select instance_number instance#, instance_name, host_name, status
from v$instance;
INSTANCE# INSTANCE_NAME    HOST_NAME             STATUS
---------- ---------------- --------------------- ------------
         2 devdb2           rac2.mycorpdomain.com OPEN

select failover_type, failover_method, failed_over
from v$session
where username='SYSTEM';
FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER
------------- --------------- ----------------
SELECT        BASIC           YES

Relocate the CRM service back to the preferred instance
After devdb1 is brought back up, the CRM service does not automatically relocate back to the preferred instance. You have to manually relocate the service to devdb1.
rac1-> export ORACLE_SID=devdb1
rac1-> sqlplus / as sysdba
SQL> startup
ORACLE instance started.
Total System Global Area  209715200 bytes
Fixed Size                  1218556 bytes
Variable Size             104859652 bytes
Database Buffers          100663296 bytes
Redo Buffers                2973696 bytes
Database mounted.
Database opened.

SQL> show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb

rac2-> export ORACLE_SID=devdb2
rac2-> sqlplus / as sysdba
SQL> show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb, CRM

rac1-> srvctl relocate service -d devdb -s crm -i devdb2 -t devdb1

SQL> connect system/oracle@devdb1
Connected.
SQL> show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb, CRM

SQL> connect system/oracle@devdb2
Connected.
SQL> show parameter service
NAME                           TYPE        VALUE
------------------------------ ----------- ------------------------
service_names                  string      devdb



12. Database Backup and Recovery

The backup and recovery procedure of an Oracle RAC database using RMAN is no different than that of a single instance database.
In this section you will follow a very simple backup and recovery scenario:
1. Perform a full database backup.
2. Create a table, mytable in the test_d tablespace.
3. At time t1, insert the first record into mytable.
4. At time t2, insert the second record into mytable.
5. At time t3, drop the table, mytable.
6. Recover the test_d tablespace to a point in time.
7. Verify the recovery.

1.Perform a full database backup.
rac1-> rman nocatalog target /
Recovery Manager: Release 10.2.0.1.0 - Production on Mon Nov 13 18:15:09 2006
Copyright (c) 1982, 2005, Oracle.  All rights reserved.
connected to target database: DEVDB (DBID=511198553)
using target database control file instead of recovery catalog
RMAN> configure controlfile autobackup on;
RMAN> backup database plus archivelog delete input;

2.Create a table, mytable in the test_d tablespace.
19:01:56  SQL> connect system/oracle@devdb2
19:02:01  SQL> create table mytable (col1 number) tablespace test_d;

3.At time, t1, insert the first record into mytable.
19:02:50  SQL> insert into mytable values (1);
19:02:59 SQL> commit;

4.At time, t2, insert the second record into mytable.
19:04:41 SQL>  insert into mytable values (2);
19:04:46  SQL> commit;

5.At time, t3, drop the table, mytable.
19:05:09 SQL> drop table mytable;

6.Recover the test_d tablespace to a point in time.
Create an auxiliary directory for the auxiliary database.
rac1-> mkdir /u01/app/oracle/aux

RMAN> recover tablespace test_d
until time "to_date('13-NOV-2006 19:03:10','DD-MON-YYYY HH24:MI:SS')"
auxiliary destination '/u01/app/oracle/aux';
RMAN> backup tablespace test_d;
RMAN> sql 'alter tablespace test_d online';

7.Verify the recovery.
19:15:09 SQL> connect system/oracle@devdb2
19:15:16 SQL> select * from mytable;
      COL1
----------
         1

13. Explore Oracle Enterprise Manager (OEM) Database
Console
Oracle Enterprise Manager Database Console provides a really nice integrated and comprehensive GUI interface to administering and managing your cluster database environment. You can perform virtually any tasks from within the console.
To access the Database Console, open a Web browser and enter the URL below. Log in as sysman and enter the password you have chosen earlier during the database installation.
http://rac1:1158/em



Start and stop the Database Console.
rac1-> emctl stop dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://rac1.mycorpdomain.com:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control ...
 ...  Stopped.

rac1-> emctl start dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://rac1.mycorpdomain.com:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 10g Database Control
................... started.
------------------------------------------------------------------
Logs are generated in directory
/u01/app/oracle/product/10.2.0/db_1/rac1_devdb1/sysman/log

Verify the status of Database Console.
rac1-> emctl status dbconsole
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0

Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
http://rac1.mycorpdomain.com:1158/em/console/aboutApplication
Oracle Enterprise Manager 10g is running.
------------------------------------------------------------------
Logs are generated in directory
/u01/app/oracle/product/10.2.0/db_1/rac1_devdb1/sysman/log
rac1-> emctl status agent
TZ set to US/Eastern
Oracle Enterprise Manager 10g Database Control Release 10.2.0.1.0
Copyright (c) 1996, 2005 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 10.1.0.4.1
OMS Version       : 10.1.0.4.0
Protocol Version  : 10.1.0.2.0
Agent Home        : /u01/app/oracle/product/10.2.0/db_1/rac1_devdb1
Agent binaries    : /u01/app/oracle/product/10.2.0/db_1
Agent Process ID  : 10263
Parent Process ID : 8171
Agent URL         : http://rac1.mycorpdomain.com:3938/emd/main
Started at        : 2006-11-12 08:10:01
Started by user   : oracle
Last Reload       : 2006-11-12 08:20:33
Last successful upload                       : 2006-11-12 08:41:53
Total Megabytes of XML files uploaded so far :     4.88
Number of XML files pending upload           :        0
Size of XML files pending upload(MB)         :     0.00
Available disk space on upload filesystem    :    71.53%
---------------------------------------------------------------
Agent is Running and Ready


14. Common Issues

Below is a summary list of issues and resolutions you may find useful.

Issue 1: Cannot activate Ethernet devices.
Error message, “Cannot activate network device eth0! Device eth0 has different MAC address than expected, ignoring.”
Resolution:
The MAC address reported by “ifconfig” does not match /etc/sysconfig/network-scripts/ifcfg-eth0. You can either update the file with the new MAC address or simply probe for the new MAC address via the system-config-network tool.


Issue 2: Cannot generate OCFS2 configuration file.
Error message, “Could not start cluster stack. This must be resolved before any OCFS2 filesystem can be mounted” when attempting to generate OCFS2 configuration file.
Resolution:
Execute ocfs2console as the root user instead of the oracle user.


Issue 3: Cannot install Oracle Clusterware or Oracle Database software on remote node.
Error message, “ /bin/tar: ./inventory/Components21/oracle.ordim.server/10.2.0.1.0: time stamp 2006-11-04 06:24:04 is 25 s in the future” during Oracle Clusterware software installation.
Resolution:
Synchronize the time between guest OS and host OS by installing VMware Tools and include theoptions, “clock=pit nosmp noapic nolapic” in /boot/grub/grub.conf. Refer to Section 3 for more information.


Issue 4: Cannot mount OCFS2 file system.
Error message, “mount.ocfs2: Transport endpoint is not connected while mounting” when attempting tomount the ocfs2 file system.
Resolution:
Execute /usr/bin/system-config-securitylevel to disable firewall.


Issue 5: Cannot start ONS resource.
Error message, “CRS-0215: Could not start resource ‘ora.rac2.ons’” when VIPCA attempts to start ONS application resource.
Resolution:
ONS attempts to access localhost but cannot resolve the IP address. Add the following entry in /etc/hosts.
127.0.0.1 localhost


OCFS2 - Configure O2CB to Start on Boot
With the releases of OCFS2 prior to 1.2.1, there is a bug that exists where the driver does not get loaded on each boot even after configuring the on-boot properties to do so. After attempting to configure the on-boot properties to start on each boot according to the official OCFS2 documentation, you will still get the following error on each boot:
...
Mounting other filesystems:
mount.ocfs2: Unable to access cluster service

Cannot initialize cluster mount.ocfs2:
Unable to access cluster service Cannot initialize cluster [FAILED]
...
Red Hat changed the way the service is registered between chkconfig-1.3.11.2-1 and chkconfig-1.3.13.2-1. The O2CB script used to work with the former.

Before attempting to configure the on-boot properties:

After resolving the bug I listed above, we can now continue to set the on-boot properties as follows:
# /etc/init.d/o2cb offline ocfs2
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure
Configuring the O2CB driver.

This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.

Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: OK


OCFS2 - Adjusting the O2CB Heartbeat Threshold

With previous versions of this article, I was able to install and configure OCFS2, format the new volume, and finally install Oracle Clusterware (with its two required shared files; the voting disk and OCR file), located on the new OCFS2 volume. While I was able to install Oracle Clusterware and see the shared virtual drive, however, I was receiving many lock-ups and hanging after about 15 minutes when the Clusterware software was running on both nodes. It always varied on which node would hang (either rac1 or rac2 in my example). It also didn't matter whether there was a high I/O load or none at all for it to crash (hang).

After looking through the trace files for OCFS2, it was apparent that access to the voting disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle Clusterware software (and the node) to crash. On the console would be a message similar to the following:

...
Index 0: took 0 ms to do submit_bio for read
Index 1: took 3 ms to do waiting for read completion
Index 2: took 0 ms to do bio alloc write
Index 3: took 0 ms to do bio add page write
Index 4: took 0 ms to do submit_bio for write
Index 5: took 0 ms to do checking slots
Index 6: took 4 ms to do waiting for write completion
Index 7: took 1993 ms to do msleep
Index 8: took 0 ms to do allocating bios for read
Index 9: took 0 ms to do bio alloc read
Index 10: took 0 ms to do bio add page read
Index 11: took 0 ms to do submit_bio for read
Index 12: took 10006 ms to do waiting for read completion
(13,3):o2hb_stop_all_regions:1888 ERROR: stopping heartbeat on all active regions.
Kernel panic - not syncing: ocfs2 is very sorry to be fencing this system by panicing

The solution I used was to increase the O2CB heartbeat threshold from its default value of 7, to 601. Some setups may require an even higher setting. This is a configurable parameter that is used to compute the time it takes for a node to "fence" itself. During the installation and configuration of OCFS2, we adjusted this value in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you encounter a kernel panic from OCFS2 and need to increase the heartbeat threshold, use the same procedures described in the section "Configure O2CB to Start on Boot and Adjust O2CB Heartbeat Threshold". If you are using an earlier version of OCFS2 tools (prior to ocfs2-tools release 1.2.2-1), the following describes how to manually adjust the O2CB heartbeat threshold.

First, let's see how to determine what the O2CB heartbeat threshold is currently set to. This can be done by querying the /proc file system as follows:

# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
7
We see that the value is 7, but what does this value represent? Well, it is used in the formula below to determine the fence time (in seconds):
[fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2
So, with an O2CB heartbeat threshold of 7, we would have a fence time of:
(7 - 1) * 2 = 12 seconds
If we want a larger threshold (say 1,200 seconds), we would need to adjust O2CB_HEARTBEAT_THRESHOLD to 601 as shown below:
(601 - 1) * 2 = 1,200 seconds

Let's see now how to manually increase the O2CB heartbeat threshold from 7 to 601. This task will need to be performed on all Oracle RAC nodes in the cluster. We first need to modify the file /etc/sysconfig/o2cb and set O2CB_HEARTBEAT_THRESHOLD to 601:

/etc/sysconfig/o2cb
# O2CB_ENABELED: 'true' means to load the driver on boot.
O2CB_ENABLED=true

# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.
O2CB_BOOTCLUSTER=ocfs2

# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.
O2CB_HEARTBEAT_THRESHOLD=601

After modifying the file /etc/sysconfig/o2cb, we need to alter the o2cb configuration. Again, this should be performed on all Oracle RAC nodes in the cluster.

# umount /ocfs/
# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure

Load O2CB driver on boot (y/n) [y]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Writing O2CB configuration: OK
Loading module "configfs": OK
Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Starting cluster ocfs2: OK
We can now check again to make sure the settings took place in for the o2cb cluster stack:
# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
601

  It is important to note that the value of 601 I used for the O2CB heartbeat threshold will not work for all configurations. In some cases, the O2CB heartbeat threshold value had to be increased to as high as 901 in order to prevent OCFS2 from panicking the kernel.


Clean remove Oracle Clusterware (CRS) 10GR2 from a RHEL4
/etc/init.d/init.evmd stop
/etc/init.d/init.evmd disable
/etc/init.d/init.cssd stop
/etc/init.d/init.cssd disable
/etc/init.d/init.crsd stop
/etc/init.d/init.crsd disable
/etc/init.d/init.crs stop
/etc/init.d/init.crs disable
rm -rf /etc/oracle /etc/oraInst.loc /etc/oratab
rm -rf /etc/init.d/init.crsd /etc/init.d/init.crs /etc/init.d/init.cssd /etc/init.d/init.evmd
rm -rf /etc/rc2.d/K96init.crs /etc/rc2.d/S96init.crs etc/rc3.d/K96init.crs /etc/rc3.d/S96init.crs /etc/rc4.d/K96init.crs /etc/rc4.d/S96init.crs /etc/rc5.d/K96init.crs /etc/rc5.d/S96init.crs /etc/rc.d/rc0.d/K96init.crs /etc/rc.d/rc1.d/K96init.crs /etc/rc.d/rc6.d/K96init.crs /etc/rc.d/rc4.d/K96init.crs
cp /etc/inittab.orig /etc/inittab
rm -rf /etc/inittab.crs /etc/inittab.no_crs
rm -rf /tmp/*
rm -rf /tmp/.oracle
rm -rf /usr/local/bin/dbhome /usr/local/bin/oraenv /usr/local/bin/coraenv
rm -rf /var/tmp/.oracle
rm -rf /u01/app/oracle/product/10.2.0/crs_1/*
echo
echo "Remove on one Node the Shared Devices"
echo "rm -rf /u03/oracrs/*"
echo
fi

15. Improving VM and Oracle

1. Obtain a baseline test for relative comparisons
2. On VM host, exclude VM clients from active, online virus scans
3. Remove Windows Indexing Service - because don’t need fast file searches
4. Remove other extraneous Windows services
5. Change VM host registry settings to improve file system IO for databases
6. Optimize the VM host configuration and options
7. Optimize the VM client OS configuration and options for Oracle database
8. Remove other extraneous VM client OS services or daemons
9. Change VM client file system settings to improve IO performance for databases
10. Adjust VM client file system block size to more closely match Oracle block size

I’m going to explain what choices #5 and #9 are. As a reminder:
   * #5 = Change VM host registry settings to improve file system IO for databases
   * #9 = Change VM client file system settings to improve IO performance for databases
In reality, this is one and the same improvement simply being applied to two different virtualization levels: the host and each client. Thus I’m simply going to present how to accomplish this technique for both Windows and Linux, and then leave it to the reader to make sure to apply it properly across all their various virtualization layers.

Both the Windows NTFS and Linux ext2/3 file systems maintain multiple "meta-data" information related to file access - such as date created, last time updated, etc. So an IO request might actually generate multiple physical IO’s - one for the data file and one or more for updating the related meta-data. In the case of the VMware server, we really don’t care to keep OS detailed file system information about the hosted clients’ data file access - it’s simply neither useful nor critical (unless doing snapshots). And at the Oracle database level, we know Oracle accesses its files every so many seconds - so why pay to update the meta-data with that information. Oracle keeps its own timestamps (i.e. SCN) in the file headers.

For Windows, we simply adjust the following registry entry:
   HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem\NtfsDisableLastAccessUpdate = 1

For Linux, there are several ways to accomplish the same result. We can set the attribute for the individual Oracle files as follows:
   chattr +A file_name
Or we can do it for an entire directory:
   chattr -R +A directory_name
However the best method (because it automatically handles any file additions) is to edit the /etc/fstab and add the NOATIME attribute:
/dev/sda6     /                 ext3      defaults,noatime   1 1
/dev/sda1    /boot            ext2      defaults,noatime   1 2
/dev/cdrom   /mnt/cdrom   iso9660  noauto,owner,ro   0 0
/dev/sda5    swap            swap     defaults               0 0

16. References:
http://www.fjandrade.com/?p=52 (Spanish) - Vmware for Windows, Oracle Enterprise Linux 4 and Oracle 10g)
Playing with RAC 11g and Oracle Enterprise Linux 5 (Spanish) - Vmware, Oracle Enterprise Linux 5 and Oracle 11g
http://www.idevelopment.info/data/Oracle/DBA_tips/VMware_Workstation_50/VMWARE_31.shtml
http://dbaworkshop.blogspot.com/search/label/Oracle%20Real%20Application%20Cluster

Patching Oracle RAC 10.2.0.1 on ASM with 10.2.0.4 Patch Set 3 for Linux x86
http://dbaworkshop.blogspot.com/2008/04/patching-oracle-rac-10201-on-asm-with.html

Oracle VM Articles
http://wiki.oracle.com/page/Installing+on+Oracle+VM
http://wiki.oracle.com/page/Oracle+VM?t=anon

Oracle VM 2.1.2 Media Pack
http://www.cyberciti.biz/tips/download-oracle-vm-software.html

Oracle VM Templates
http://www.oracle.com/technology/products/vm/templates.html

Build Your Own Oracle RAC 10g Cluster on Linux and FireWire  (U$1800)
http://www.oracle.com/technology/pub/articles/hunter_rac10g.html

Oracle 10gR2 RAC on Linux (CentOS 4) using VMware Server with Screen Shots
http://www.oracle-base.com/articles/10g/OracleDB10gR2RACInstallationOnCentos4UsingVMware.php

Tons of Articles on RAC
http://www.databasejournal.com/article.php/3559996


Sequences in RAC
Many applications have to use them for unique number generation .
- Use alongwithd bms_utility.current_instance if it is a PK
- Be prepared to have some gaps .
- Options CACHE and ORDER cannot be used for RAC
- If ORDER is used then Data Dictionary will be accessed every time -overhead
- GCS maintains the sequences across the instances



Post Installation
- Enable asynchronous I/O if available
   cd $ORACLE_HOME/rdbms/lib; make -f ins_rdbms.mk async_on ioracle
- Adjust UDP send / receive buffer size to 256K (Linux only)
- If Buffer Cache > 1.7GB required, use 64-bit platform.