在vmware上安装一个双节点的Redhat Linux + Oracle10g RAC + ASM的测试环境，按照Oracle官方网站的安装指南，基本都很顺利，只是在clusterware的安装过程中碰到一点小问题。在执行root.sh脚本时，一直卡在Startup will be queued to init within 90 seconds.过不去，手动执行/etc/init.d/init.cssd start也是同样的问题，两个节点都是同样的情况。重装了三次后终于发现问题所在，就是在前面指定ocr和voting disk的location的时候，我填入的是soft link而不是raw设备的路径，改成类似/dev/raw/raw1的字符设备路径就ok了，本来想建个link比较易于识别的，郁闷。
正常执行root.sh的结果
Node1：
./root.sh
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw3
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
CSS is inactive on these nodes.
rac2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Now formatting voting device: /dev/raw/raw3
Format of 1 voting devices complete.
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
CSS is inactive on these nodes.
rac2
Local node checking complete.
Run root.sh on remaining nodes to start CRS daemons.
Node 2:
./root.sh
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
rac2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
Checking to see if Oracle CRS stack is already configured
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/oracle/product/10.2.0' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 90 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
rac1
rac2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces should be used to configure virtual IPs.
发现在节点2上报”eth0″ is not public，这个在安装指南里有提到，只是由于公共接口使用不可路由的 IP 地址 (192.168.x.x)， Oracle 集群验证实用程序 (CVU) 将无法找到合适的公共接口，只要手动在Node2上以root用户执行vipca配置一下VIP即可。
顺便说一下，clusterware安装失败需要重装的话，需要删除很多垃圾，否则重装过程会报错无法进行：
1.删除crs home
2.删除ocr路径信息,/etc/oracle/ocr.loc
3.删除$ORACLE_BASE/oraInventory
4.清空ocr和voting disk对应的raw设备
5.清除启动脚本
mv -f /etc/init.d/init.cssd /etc/init.d/init.cssd.bak
mv -f /etc/init.d/init.crs /etc/init.d/init.crs.bak
mv -f /etc/init.d/init.crsd /etc/init.d/init.crsd.bak
mv -f /etc/init.d/init.evmd /etc/init.d/init.evmd.bak
mv -f /etc/inittab.crs /etc/inittab.crs.bak
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
mv -f /etc/inittab /etc/inittab.bak
cp -f /etc/inittab.orig /etc/inittab
mv -f /etc/init.d/init.crs /etc/init.d/init.crs.bak
mv -f /etc/init.d/init.crsd /etc/init.d/init.crsd.bak
mv -f /etc/init.d/init.evmd /etc/init.d/init.evmd.bak
mv -f /etc/inittab.crs /etc/inittab.crs.bak
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
mv -f /etc/inittab /etc/inittab.bak
cp -f /etc/inittab.orig /etc/inittab
/etc/inittab.orig是安装crs之前保留的备份
有意思. /dev/raw/raw1 已经通过 /etc/sysconfig/rawdevices 链接过一次磁盘设备了,你老老实实用吧.
这个是我在10.1.0.4上使用的清除脚本, 使用需谨慎!
Appendix d. Remove CRS
rm -f /etc/init.d/init.cssd
rm -f /etc/init.d/init.crs
rm -f /etc/init.d/init.crsd
rm -f /etc/init.d/init.evmd
rm -f /etc/rc2.d/K96init.crs
rm -f /etc/rc2.d/S96init.crs
rm -f /etc/rc3.d/K96init.crs
rm -f /etc/rc3.d/S96init.crs
rm -f /etc/rc5.d/K96init.crs
rm -f /etc/rc5.d/S96init.crs
#rm -rf /etc/oracle/scls_scr
rm -f /etc/inittab.crs
rm -rf /var/tmp/.oracle/*
rm -rf /tmp/.oracle
rm -rf /var/tmp/.oracle
rm -rf /usr/tmp/.oracle
rm -rf /etc/oracle/*
rm -rf /home/oracle/crs/*
rm -rf /home/oracle/oraInventory
#cp /etc/inittab.orig /etc/inittab
#/etc/init.d/init.crs disable
#/etc/init.d/init.crs stop
ls -a /var/tmp/.oracle # oracle user has write permission
chown -R oracle.dba /var/tmp/.oracle
chown -R oracle.oinstall /dev/raw/raw1
chown -R oracle.oinstall /dev/raw/raw2
chmod 777 /dev/raw/raw1
chmod 777 /dev/raw/raw2
raw绑定只是将块设备绑定成字符设备而已，呵呵。本来想通过soft link来使用raw设备，因为这样可以和现有规范保持一致，也容易管理些。
CRS本身提供了删除的脚本！忘了叫什么名字……