(1)一次失败的割接经历：麒麟linux6.5+oracle12crac打PSU补丁

作者：stong_lxm | 来源：互联网 | 2023-09-06 13:40

一次失败的割接经历：麒麟linux6.5+oracle12crac打PSU补丁根据安全扫描漏洞整改要求，需对某系统（麒麟linux6.5+oracle12crac）打22011

1.项目要求：

根据安全扫描漏洞整改要求，需对某系统（麒麟linux6.5+oracle12c rac）打220118版本的PSU补丁。

系统状态、磁盘空间等等检查都OK，根据保守经验，申请停机时间3小时

由于在9i/10g/11g/12c/19c rac+centos/redhat/solaris/aix等环境都有过打补丁经验。应该说这次打补丁差不多十拿九稳的事情，没想到很快就被打脸了:)

曾经在19c rac打过补丁踩坑记录，此时没有参考意义:(https://www.modb.pro/doc/46086

2.实施准备：

2.1介质主备：

下载PSU最新补丁集：Assistant: Download Reference for Oracle Database/GI Update, Revision, PSU, SPU(CPU), Bundle Patches, Patchsets and Base Releases (文档 ID 2118136.2)

下载最新opatch工具：How To Download And Install The Latest OPatch(6880880) Version (Doc ID 274526.1)

2.2两节点备份权限：

根据：Script to capture and restore file permission in a directory (for eg. ORACLE_HOME) (Doc ID 1515018.1)

chmod 755 permission.pl ./permission.pl /u01/app/12.2.0/grid cp restore-perm-二-9月-20-14-22-39-2022.cmd r.cmd

2.3两节点备份文件（注意保证空间足够）：

cd /u01/app tar -cvf grid.tar ./12.2.0 ./grid --exclude=./grid/admin --exclude=./grid/diag --exclude=./12.2.0/grid/network/log --exclude=./12.2.0/grid/log --exclude=./12.2.0/grid/rdbms/audit tar -cvf oracle.tar ./oracle --exclude=./oracle/admin --exclude=./oracle/diag tar -cvf oraInventory.tar ./oraInventory

2.4opatch工具解压和替换：

先/etc/profile增加环境变量，方便root用户执行和查看命令：

vi /etc/profile export PATH=/u01/app/12.2.0/grid/OPatch:/u01/app/12.2.0/grid/bin:$PATH; source /etc/profile

再替换oracle和grid下opatch工具：

cd /home/oracle/soft/2022 unzip p6880880_122010_Linux-x86-64-OPatch 12.2.0.1.32.zip rm -rf /u01/app/oracle/product/12.2.0/db_1/OPatch cp -rp OPatch /u01/app/oracle/product/12.2.0/db_1/ chown -R oracle:oinstall /u01/app/oracle/product/12.2.0/db_1/OPatch rm -rf /u01/app/12.2.0/grid/OPatch cp -rf OPatch /u01/app/12.2.0/grid/ chown -R grid:oinstall /u01/app/12.2.0/grid/OPatch export PATH=$PATH:/u01/app/12.2.0/grid/OPatch opatch version

2.5补丁冲突检查

根据readme执行即可，如下示例,省略部分：

grid用户下核查： $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33678030 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33116894 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/26839277 $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33610989 oracle用户下核查： $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/soft/2022/33559966/33583921/33587128 root用户下预执行分析：（此次预执行分析都成功，但实际执行却报错，继续往下看） opatchauto apply /home/oracle/soft/2022/33559966/33583921 -analyze

3.正式实施：

root用户下在节点1先执行：

opatchauto apply /home/oracle/soft/2022/33559966/33583921

可以看到db补丁成功，但开始应用grid补丁的时候失败了：

根据错误位置信息查看内容：

执行chmod -R 755 /u01/app/12.2.0/grid/jdk，然后opatchauto resume

继续报错jdk目录权限：

查到相关问题参考：

赋权chown -R grid:oinstall /u01/app/12.2.0/grid/jdk，然后opatchauto resume,还是报错bin、lib等目录权限，约9千行的错误：

根据错误日志，提取grep Destionation err.txt > 1.txt，然后排序去重，过滤出300多行的文件。

编辑脚本执行赋权：

chown grid:oinstall /u01/app/12.2.0/grid/bin/HiveMetadata.jar chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsccm chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsdriverstate chown grid:oinstall /u01/app/12.2.0/grid/bin/acfshanfs chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsload chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsregistrymount chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_apply.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_dupd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_initializer chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_monitor chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_preapply chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrepl_transport chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsreplcrs chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsrm chown grid:oinstall /u01/app/12.2.0/grid/bin/acfsroot chown grid:oinstall /u01/app/12.2.0/grid/bin/acfssinglefsmount chown grid:oinstall /u01/app/12.2.0/grid/bin/advmccb chown grid:oinstall /u01/app/12.2.0/grid/bin/afdboot chown grid:oinstall /u01/app/12.2.0/grid/bin/afddriverstate chown grid:oinstall /u01/app/12.2.0/grid/bin/afdload chown grid:oinstall /u01/app/12.2.0/grid/bin/afdroot chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool chown grid:oinstall /u01/app/12.2.0/grid/bin/afdtool.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/appagent.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg chown grid:oinstall /u01/app/12.2.0/grid/bin/appvipcfg.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/asmproxy chown grid:oinstall /u01/app/12.2.0/grid/bin/bdschecksw chown grid:oinstall /u01/app/12.2.0/grid/bin/cdpc.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutlo.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/cemutls.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/clscfg.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/clsdiaglsnr.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/clsecho.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/clsid.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/clssproxy.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_getperm.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_profile.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_register.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_relocate.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_setperm.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_start.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stat.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_stop.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crs_unregister.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crscdpd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crsctl.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crsd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/crsdiag.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/crsrename.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/crstmpl.scr chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdagent chown grid:oinstall /u01/app/12.2.0/grid/bin/cssdmonitor chown grid:oinstall /u01/app/12.2.0/grid/bin/cssvfupgd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/dbfs_client chown grid:oinstall /u01/app/12.2.0/grid/bin/diagcollection.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/diagsnap_setup.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/diskmon chown grid:oinstall /u01/app/12.2.0/grid/bin/dsml2ldif chown grid:oinstall /u01/app/12.2.0/grid/bin/emcrsp.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evminfo.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmlogger.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmkbin.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmmklib.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmpost.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmshow.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmsort.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evmwatch.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/evt.sh chown grid:oinstall /u01/app/12.2.0/grid/bin/fmputl chown grid:oinstall /u01/app/12.2.0/grid/bin/ghappctl.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/gipcd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd chown grid:oinstall /u01/app/12.2.0/grid/bin/gnsd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnpd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/gpnptool.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/gsd chown grid:oinstall /u01/app/12.2.0/grid/bin/jwcctl.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/kfod chown grid:oinstall /u01/app/12.2.0/grid/bin/kgpmon chown grid:oinstall /u01/app/12.2.0/grid/bin/lcsscan chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapadd chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapaddmt chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapbind chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapcompare chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapdelete chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmoddn chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodify chown grid:oinstall /u01/app/12.2.0/grid/bin/ldapmodifymt chown grid:oinstall /u01/app/12.2.0/grid/bin/lsdb.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes chown grid:oinstall /u01/app/12.2.0/grid/bin/lsnodes.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/lxchknlb chown grid:oinstall /u01/app/12.2.0/grid/bin/lxegen chown grid:oinstall /u01/app/12.2.0/grid/bin/lxinst chown grid:oinstall /u01/app/12.2.0/grid/bin/mdnsd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/mkstore chown grid:oinstall /u01/app/12.2.0/grid/bin/mtactl chown grid:oinstall /u01/app/12.2.0/grid/bin/ochadctl.pl chown grid:oinstall /u01/app/12.2.0/grid/bin/oclskd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/oclumon.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrcheck.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrconfig.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ocrdump.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ocssdrim.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/octssd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/odig chown grid:oinstall /u01/app/12.2.0/grid/bin/odig.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/odnsd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/ohasd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/oifcfg.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd chown grid:oinstall /u01/app/12.2.0/grid/bin/okacmd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/okadriverstate chown grid:oinstall /u01/app/12.2.0/grid/bin/okaload chown grid:oinstall /u01/app/12.2.0/grid/bin/okaroot chown grid:oinstall /u01/app/12.2.0/grid/bin/olfscmd chown grid:oinstall /u01/app/12.2.0/grid/bin/olfsroot chown grid:oinstall /u01/app/12.2.0/grid/bin/ologgerd chown grid:oinstall /u01/app/12.2.0/grid/bin/olsnodes.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/oprocd chown grid:oinstall /u01/app/12.2.0/grid/bin/oraagent.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnfs chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd chown grid:oinstall /u01/app/12.2.0/grid/bin/oradnssd.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/oranetmonitor.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/orapki chown grid:oinstall /u01/app/12.2.0/grid/bin/orarootagent.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/osysmond.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/owm chown grid:oinstall /u01/app/12.2.0/grid/bin/racgeut chown grid:oinstall /u01/app/12.2.0/grid/bin/racgevtf chown grid:oinstall /u01/app/12.2.0/grid/bin/racgmain chown grid:oinstall /u01/app/12.2.0/grid/bin/racgvip chown grid:oinstall /u01/app/12.2.0/grid/bin/schema chown grid:oinstall /u01/app/12.2.0/grid/bin/sclsspawn chown grid:oinstall /u01/app/12.2.0/grid/bin/scriptagent.bin chown grid:oinstall /u01/app/12.2.0/grid/bin/srvconfig chown grid:oinstall /u01/app/12.2.0/grid/bin/usrvip chown grid:oinstall /u01/app/12.2.0/grid/bin/xml chown grid:oinstall /u01/app/12.2.0/grid/crs/install/s_crsconfig_defs chown grid:oinstall /u01/app/12.2.0/grid/crs/install/tfa_setup chown grid:oinstall /u01/app/12.2.0/grid/jlib/gns.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/netcfg.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvm.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmasm.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/srvmhas.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster-common.jar chown grid:oinstall /u01/app/12.2.0/grid/jlib/supercluster.jar chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsdriverstate.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfshanfs.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfslib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsload.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsregistrymount.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsreplcrs.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfsroot.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfssinglefsmount.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/acfstoolsdriver.sh chown grid:oinstall /u01/app/12.2.0/grid/lib/afddriverstate.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/afdlib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/afdload.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/afdroot.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/afdtoolsdriver.sh chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdafd.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdambr.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdamdu.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdbase.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdpasswd.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdqg.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdshare.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdsys.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcmdug.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/asmcommand.xml chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_common.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/jwcctl_lib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/libCurveFit.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libNeticaJ.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libacfs12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libafd12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libagent12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libagfw12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libagtsh.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclnt12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmclntsh12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libasmperl12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_asym.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_base_non_fips.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_fips.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_accel_non_fips.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libccme_ecc_non_fips.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libcell12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcha12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libclient12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntsh.so.12.1 chown grid:oinstall /u01/app/12.2.0/grid/lib/libclntshcore.so.12.1 chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsce12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libclscred12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsnsjni12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsr12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libclsra12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcommon12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libcore12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libcorejava.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcrf_mdb12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.sig chown grid:oinstall /u01/app/12.2.0/grid/lib/libcryptocme.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libctx12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libctxc12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libdbcfg12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libeons.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libeonsserver.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libgeneric12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libgns12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libgnsjni12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libhasgen12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libipc1.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libjagent12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libkubsagt12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libldapclnt12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libmql1.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libn12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libnbeq12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libncrypt12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libnhost12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libnid.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnz12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libnnzst12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libntcps12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libocr12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrb12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libocrutl12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/liboevm.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libons.so chown grid:oinstall /u01/app/12.2.0/grid/lib/liborabz2.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libordsdo12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libowm2.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libplc12_pic.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libplp12_pic.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libpls12_pic.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libpsa12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/librdjni12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libserver12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpg.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libskgxpr.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libskjcx12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libsql12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libsqlplus.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmcred12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmhas12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libsrvmocr12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libuini12.so chown grid:oinstall /u01/app/12.2.0/grid/lib/libxml12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libzt12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/libztkg12.a chown grid:oinstall /u01/app/12.2.0/grid/lib/naect.o chown grid:oinstall /u01/app/12.2.0/grid/lib/naeet.o chown grid:oinstall /u01/app/12.2.0/grid/lib/nautab.o chown grid:oinstall /u01/app/12.2.0/grid/lib/okadriverstate.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/okalib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/okaload.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/okaroot.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/okatoolsdriver.sh chown grid:oinstall /u01/app/12.2.0/grid/lib/olfscmd.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/olfsroot.pl chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsdriverstate.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfshanfs.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfslib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsload.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsregistrymount.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfsroot.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_acfssinglefsmount.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afddriverstate.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdlib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdload.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_afdroot.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okadriverstate.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okalib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaload.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_okaroot.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_acfslib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_afdlib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/osds_unix_linux_okalib.pm chown grid:oinstall /u01/app/12.2.0/grid/lib/s_jwcctl_lib.pm chown grid:oinstall /u01/app/12.2.0/grid/srvm/jlib/srvctl.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable.pm chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/Text/ASCIITable/Wrap.pm chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/collectfiles.pl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/discover_ora_stack.sh chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/getppid.pl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/metric_iorm.pl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/patchtfa.sh chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/profiling.sh chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.pl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfactl.tmpl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/tfasetup.pl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/bin/uninstalltfa.sh chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/bin/tnt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/conf/tnt.prop.tmpl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-cli-1.2.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-lang3-3.1.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/commons-logging-1.1.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/tnt.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xmlparserv2.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/ext/tnt/lib/xternal.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/input chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/init.tfa.tmpl chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.aix chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.hpux chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.linux chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/inittab.sunos chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/oracle-tfa.conf chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/install/roottfa.sh chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/.buildid chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/internal/usableports.txt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/RATFA.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/Symlink.so chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/commons-io-2.1.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jdev-rt.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-4.1.27.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/je-5.0.84.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jewt4.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/jsch-0.1.51.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/ojdbc5.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/jlib/share.jar chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/public.jks chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/date_patterns.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/file_type_patterns.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignore_extensions.txt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/ignorefiles.txt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/mask_strings.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/problemset.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/searchStrings.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/resources/tasks.xml chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.jks chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa.md5 chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_directories.txt chown grid:oinstall /u01/app/12.2.0/grid/suptools/tfa/release/tfa_home/tfa_setup.txt mkdir -p /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/jdk mkdir -p /u01/app/12.2.0/grid/jdk/jre/legal/javafx mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited mkdir -p /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited mkdir -p /u01/app/12.2.0/grid/jdk/legal/jdk mkdir -p /u01/app/12.2.0/grid/jdk/legal/javafx chown -R grid:oinstall /u01/app/12.2.0/grid/usm/install/Novell/SLES12/x86_64/SP4/4.12.14-94.41/default/bin chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/jdk chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/legal/javafx chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/limited chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/jre/lib/security/policy/unlimited chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/jdk chown -R grid:oinstall /u01/app/12.2.0/grid/jdk/legal/javafx chown -R grid:oinstall /u01/app/12.2.0/grid/crs/install chown grid:oinstall /u01/app/12.2.0/grid/bin chown grid:oinstall /u01/app/12.2.0/grid/lib

继续opatchauto resume，还是报错

进程占用处理：

fuser -ck /u01/app/12.2.0/grid/bin/oracle fuser -ck /u01/app/12.2.0/grid/lib/libclntsh.so.12.1 fuser -ck /u01/app/12.2.0/grid/lib/libasmclntsh12.so

多次执行fuser删进程，确认没有进程了即可

继续opatchauto resume，提示grid patch成功，但crsctl start crs最终还是报错了：

第一个节点到此失败后尝试各种办法，均无法启动crsd集群服务，

比如根据CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)尝试：

/u01/app/12.2.0/grid/crs/install/rootcrs.sh -prepatch /u01/app/12.2.0/grid/crs/install/rootcrs.sh -postpatch opatchauto resume还是失败

由于时间紧迫，没有多余的时间再做研究分析，只能宣告此次打补丁失败，决定回退。

ps -ef|grep d.bin后kill掉相关进程。

然后tar -xvf解压原先备份的tar包。重启集群服务恢复正常。通知应用检查恢复业务。

收集好错误日志信息，第二天开始复盘分析问题。

由于手头没有麒麟linux6.5介质，所以尝试先用centos6.10搭相同的12c rac环境，补丁版本保持一致。（高效的做法：虚机环境+克隆+nfs共享存储+系统快照）

相关参考：

克隆的网卡问题https://sqlora.blog.csdn.net/article/details/126869607

nfs搭asmhttps://sqlora.blog.csdn.net/article/details/126951455

没想到同样的步骤很顺利的打上了PSU补丁，没有遇到任何错误。

此时怀疑麒麟系统是否存在兼容性bug问题？由于用户要求做安全整改必须打上补丁，所以只能找用户要来了麒麟linux6.5系统介质。

同样搭好相同的12c rac环境，补丁版本保持一致。希望能重现故障。

功夫不负有心人，节点1操作果然重现了生产环境打补丁的故障现象。于是小心翼翼的分析每一步错误，并做好记录。查阅大量资料及测试，最终顺利打上了集群补丁。

测试验证过程中，同样遇到了上述问题crs无法启动，通过查阅oracle资料发现：

CRS-6706: Oracle Clusterware Release patch level (‘nnn’) does not match Software patch level (‘mmm’) (Doc ID 1639285.1)其中说到节点两边补丁级别不一致的情况下，肯定是无法启动的，所以决定放弃尝试，先不管。

继续给第二个节点补丁

opatchauto apply /home/oracle/soft/2022/33559966/33583921

报错如下：

然后检查两节点的补丁情况，级别和补丁完全一致，但状态不对。

应该是【NORMAL】，但都显示【ROLLING PATCH】

虽然状态不对，但此时发现节点1和节点2的集群服务均是正常的，检查命令：

crsctl query crs activeversion -f kfod op=patches kfod op=patchlvl crsctl check crs crsctl stat res -t -init crsctl stat res -t [grid@rac1 ~]$ kfod op=patches --------------- List of Patches =============== 26839277 33116894 33587128 33610989 33678030 [grid@rac1 ~]$ kfod op=patchlvl ------------------- Current Patch level =================== 3975995681 [grid@rac1 ~]$ ssh rac2 Last login: Tue Sep 20 05:30:51 2022 from rac1 [grid@rac2 ~]$ kfod op=patches --------------- List of Patches =============== 26839277 33116894 33587128 33610989 33678030 [grid@rac2 ~]$ kfod op=patchlvl ------------------- Current Patch level =================== 3975995681

既然版本和补丁都一致，剩下就是状态的处理了：

参考：Software Patch Mismatch After Applying Bundle Patch or PSU on Grid Home and the Cluster Upgrade State is [ROLLING PATCH] (Doc ID 2176790.1)

在节点1执行如下命令后，节点1、2状态均恢复正常：

处理操作命令如下：

[root@rac1 ~]# clscfg -patch clscfg: -patch mode specified clscfg: EXISTING configuration version 5 detected. clscfg: version 5 is 12c Release 2. Successfully accumulated necessary OCR keys. Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. [root@rac1 ~]# crsctl query crs activeversion -f Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [927320293]. [root@rac1 ~]# crsctl stop rollingpatch CRS-1161: The cluster was successfully patched to patch level [3975995681]. [root@rac1 ~]# su - grid [grid@rac2 ~]$ crsctl query crs activeversion -f Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681]. [grid@rac2 ~]$ exit logout Connection to rac2 closed. [grid@rac1 ~]$ crsctl query crs activeversion -f Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [3975995681].