rac节点无法启动ORA-29702的问题及分析（70天)
时间:2022-05-04
本文章向大家介绍rac节点无法启动ORA-29702的问题及分析（70天)，主要内容包括其使用实例、应用技巧、基本知识点总结和需要注意事项，具有一定的参考价值，需要的朋友可以参考一下。
今天在虚拟机上启动rac,发现有一个节点怎么都起不了。另外一个节点没问题。
SQL> startup nomount
ORA-29702: error occurred in Cluster Group Service operation尝试使用crs_stat查看crs的组件状态，也报错了。
-bash-4.1$ crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.查看alert日志，发现在最后是因为29702的错误导致的。
SMON started with pid=20, OS id=12344
Sun May 11 04:10:28 2014
RECO started with pid=21, OS id=12346
Sun May 11 04:10:28 2014
MMON started with pid=22, OS id=12348
Sun May 11 04:10:28 2014
MMNL started with pid=23, OS id=12350
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
USER (ospid: 12242): terminating the instance due to error 29702
Instance terminated by USER, pid = 12242对于这个错误，oracle给出的解释如下。
-bash-4.1$ oerr ora 29702
29702, 00000, "error occurred in Cluster Group Service operation"
// *Cause: An unexpected error occurred while performing a CGS operation.
// *Action: Verify that the LMON process is still active.
// Check the Oracle LMON trace files for errors.
// Also, check the related CSS trace file for errors.查看lmon的日志如下：
Trace file /u04/app/11.2.0/db/diag/rdbms/racdb/RACDB1/trace/RACDB1_lmon_12324.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Oracle Label Security, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /u04/app/11.2.0/db/product/11.2.0/dbhome_1
System name: Linux
Node name: rac1
Release: 2.6.32-71.el6.x86_64
Version: #1 SMP Wed Sep 1 01:33:01 EDT 2010
Machine: x86_64
VM name: VMWare Version: 6
Instance name: RACDB1
Redo thread mounted by this instance: 0 <none>
Oracle process number: 11
Unix process pid: 12324, image: oracle@rac1 (LMON)
*** 2014-05-11 04:10:27.777
*** SESSION ID:(130.1) 2014-05-11 04:10:27.777
*** CLIENT ID:() 2014-05-11 04:10:27.777
*** SERVICE NAME:() 2014-05-11 04:10:27.777
*** MODULE NAME:() 2014-05-11 04:10:27.777
*** ACTION NAME:() 2014-05-11 04:10:27.777
GES resources 5720 pool 3
GES enqueues 8361
GES IPC: Receivers 2 Senders 2
GES IPC: Buffers Receive 1000 Send (i:1030 b:471) Reserve 301
GES IPC: Msg Size Regular 1176 Batch 8376
Batching factor: enqueue replay 206, ack 229
Batching factor: cache replay 128 size per lock 64
*** 2014-05-11 04:10:28.644
kjxggin: CGS tickets = 1000
kgxgncin: CLSS init failed with status 3
kgxgncin: return status 3 (1311719766 SKGXN not av) from CLSS
kjxgmin: kgxgncin fails - (2)
kjxggin: generic group layer init fails
*** 2014-05-11 04:10:28.655
Global Enqueue Service Shutdown对于该节点，使用crs_stat，crsctl的操作都无济于事。
-bash-4.1$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
-bash-4.1$ crs_start -all
CRS-0184: Cannot communicate with the CRS daemon.查看进程，确实都起来了。
-bash-4.1$ ps -ef|grep d.bin
root 2103 1 0 May10 ? 00:00:51 /u04/app/11.2.0/grid/bin/ohasd.bin reboot
grid 2297 1 0 May10 ? 00:00:32 /u04/app/11.2.0/grid/bin/oraagent.bin
grid 2309 1 0 May10 ? 00:00:01 /u04/app/11.2.0/grid/bin/mdnsd.bin
grid 2320 1 0 May10 ? 00:00:36 /u04/app/11.2.0/grid/bin/gpnpd.bin
root 2330 1 0 May10 ? 00:00:14 /u04/app/11.2.0/grid/bin/orarootagent.bin
grid 2333 1 0 May10 ? 00:02:39 /u04/app/11.2.0/grid/bin/gipcd.bin
root 2348 1 1 May10 ? 00:12:00 /u04/app/11.2.0/grid/bin/osysmond.bin
root 2569 1 0 May10 ? 00:03:55 /u04/app/11.2.0/grid/bin/ologgerd -M -d /u04/app/11.2.0/grid/crf/db/rac1
grid 12569 9580 0 04:25 pts/1 00:00:00 grep d.bin使用root用户来停掉crs。但是报了错。
root
[root@rac1 bin]# ./crsctl disable crs
CRS-4621: Oracle High Availability Services autostart is disabled.
[root@rac1 bin]# ./crsctl stop crs
CRS-2796: The command may not proceed when Cluster Ready Services is not running
CRS-4687: Shutdown command has completed with errors.
CRS-4000: Command Stop failed, or completed with errors.再次尝试启动，也是报错。
[root@rac1 bin]# ./crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@rac1 bin]# ./crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.最后看到mos上有一个workaround，可以手动Kill掉那些crs的进程。当然了，在正式环境中还是得把psu打上。
[root@rac1 bin]# ps -fea | grep ohasd.bin | grep -v grep
root 2103 1 0 May10 ? 00:00:52 /u04/app/11.2.0/grid/bin/ohasd.bin reboot
[root@rac1 bin]# ps -fea | grep gipcd.bin | grep -v grep
grid 2333 1 0 May10 ? 00:02:41 /u04/app/11.2.0/grid/bin/gipcd.bin
[root@rac1 bin]# ps -fea | grep mdnsd.bin | grep -v grep
grid 2309 1 0 May10 ? 00:00:01 /u04/app/11.2.0/grid/bin/mdnsd.bin
[root@rac1 bin]# ps -fea | grep gpnpd.bin | grep -v grep
grid 2320 1 0 May10 ? 00:00:37 /u04/app/11.2.0/grid/bin/gpnpd.bin
[root@rac1 bin]# ps -fea | grep evmd.bin | grep -v grep
[root@rac1 bin]# ps -fea | grep crsd.bin | grep -v grep
[root@rac1 bin]# kill -9 2103 2333 2309 2320
再次尝试启动crs
[root@rac1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
[root@rac1 bin]# ./crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.启动的时候有些慢，稍等一下，直接自己来启库了。这次起库就没有问题了。
-bash-4.1$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sun May 11 04:41:03 2014
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup nomount
ORACLE instance started.
Total System Global Area 638853120 bytes
Fixed Size 2231072 bytes
Variable Size 482346208 bytes
Database Buffers 146800640 bytes
Redo Buffers 7475200 bytes
SQL> alter database mount;
Database altered.
SQL> alter database open;
Database altered.
SQL>查看crs的状态，该起的都起了。两个节点创建了一个小表做测试，没有问题了。那个workaround的细节可以从MOS文档 ID 1233580.1里面查看。
-bash-4.1$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....ER.lsnr ora....er.type ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type ONLINE ONLINE rac2
ora.asm ora.asm.type OFFLINE OFFLINE
ora.cvu ora.cvu.type OFFLINE OFFLINE
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type OFFLINE OFFLINE
ora.ons ora.ons.type ONLINE ONLINE rac1
ora....SM1.asm application OFFLINE OFFLINE
ora....C1.lsnr application ONLINE ONLINE rac1
ora.rac1.gsd application OFFLINE OFFLINE
ora.rac1.ons application ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type ONLINE ONLINE rac1
ora....SM2.asm application OFFLINE OFFLINE
ora....C2.lsnr application ONLINE ONLINE rac2
ora.rac2.gsd application OFFLINE OFFLINE
ora.rac2.ons application ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type ONLINE ONLINE rac2
ora.racdb.db ora....se.type ONLINE ONLINE rac2
ora.scan1.vip ora....ip.type ONLINE ONLINE rac2
- SpringCloud实战小贴士：Zuul的路径匹配
- 程序员你为什么这么累【续】：编码习惯之参数校验和国际化规范
- 程序员你为什么这么累【续】：编码习惯-函数编写建议
- 那些年，我们一起碰到过的骗局
- Spring Security (五) 动手实现一个IP_Login
- 史上最全Linux提权后获取敏感信息方法
- Spring Security (四) 核心过滤器源码分析
- Spring Security (三) 核心配置解读
- Spring Cloud配置中心获取不到最新配置信息的问题
- 总是听别人说响应式布局，原来这么简单
- Spring Cloud Zuul重试机制探秘
- Eureka中RetryableClientQuarantineRefreshPercentage参数探秘
- Edgware.RC1中ZuulFallbackProvider的改进
- JPA的多表复杂查询：详细篇
- JavaScript 教程
- JavaScript 编辑工具
- JavaScript 与HTML
- JavaScript 与Java
- JavaScript 数据结构
- JavaScript 基本数据类型
- JavaScript 特殊数据类型
- JavaScript 运算符
- JavaScript typeof 运算符
- JavaScript 表达式
- JavaScript 类型转换
- JavaScript 基本语法
- JavaScript 注释
- Javascript 基本处理流程
- Javascript 选择结构
- Javascript if 语句
- Javascript if 语句的嵌套
- Javascript switch 语句
- Javascript 循环结构
- Javascript 循环结构实例
- Javascript 跳转语句
- Javascript 控制语句总结
- Javascript 函数介绍
- Javascript 函数的定义
- Javascript 函数调用
- Javascript 几种特殊的函数
- JavaScript 内置函数简介
- Javascript eval() 函数
- Javascript isFinite() 函数
- Javascript isNaN() 函数
- parseInt() 与 parseFloat()
- escape() 与 unescape()
- Javascript 字符串介绍
- Javascript length属性
- javascript 字符串函数
- Javascript 日期对象简介
- Javascript 日期对象用途
- Date 对象属性和方法
- Javascript 数组是什么
- Javascript 创建数组
- Javascript 数组赋值与取值
- Javascript 数组属性和方法