由于测试环境打19c补丁失败，导致节点2  d2 crs无法启动！打补丁没有有效备份，亲人两行泪

先删再加节点。

一、删除节点

参考
https://www.cnblogs.com/binliubiao/p/11996092.html
Deleting a Cluster Node on Linux and UNIX Systems
https://docs.oracle.com/en/database/oracle/oracle-database/19/cwadd/
adding-and-deleting-cluster-nodes.html#GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A
删除可以参考上述博客,添加节点建议参考MOS文档
Oracle E-Business Suite Release 12.2: Adding or Deleting 11gR2 or 12c1 Oracle RAC Nodes (Doc ID 1570554.1)

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0
测试环境RAC2节点,打补丁2节点安装失败,因此对节点2进行删除后,再添加。

删除实例，1.1都在正常的节点1执行操作 【由于节点2集群crs无法启动,因此删除操作在节点1执行] srvctl stop instance -db db_unique_name -node node_name
srvctl relocate server -servers "server_name_list" -serverpool Free
srvctl config database -db db_unique_name

oracle$
srvctl config database -db orcl
dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sysdba -sysDBAPassword oracle

--删除实例需要本地实例已启动！
[FATAL] [DBT-11502] The instance (orcl1) is not running on the local node.
CAUSE: A locally running instance is required to perform the selected operation.
ACTION: Specify a locally running database, or execute DBCA on a node where the database instance is running.
$ srvctl start instance -d orcl -instance orcl1
$ dbca -silent -deleteInstance -nodeList d2 -gdbName orcl -instanceName orcl2 Prepare for db operation
40% complete
Deleting instance
48% complete
52% complete
56% complete
60% complete
64% complete
68% complete
72% complete
76% complete
80% complete
Completing instance management.
100% complete
$ srvctl stop database -d orcl

本次测试环境3套DB 均为启动存活的节点1实例,进行删除实例操作。

SQL> alter system set enable_ddl_logging=true;
$ dbca -silent -deleteInstance -nodeList d2 -gdbName abc -instanceName abc2 --观察DB Alert日志,并没有ddl操作,而是上次了实例2的redo and undo，并且RAC相关参数设置了一下！
2020-04-15T06:57:54.304305+08:00
Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_3.309.1036713505
Deleted Oracle managed file +DATA/ABC/ONLINELOG/group_4.310.1036713537
Deleted Oracle managed file +DATA/ABC/DATAFILE/undotbs2.308.1036712911
2020-04-15T06:57:55.090943+08:00
ALTER SYSTEM RESET undo_tablespace SCOPE=SPFILE SID='abc2';
2020-04-15T06:57:55.098168+08:00
ALTER SYSTEM RESET instance_number SCOPE=SPFILE SID='abc2';
2020-04-15T06:57:55.107884+08:00

ALTER SYSTEM RESET thread SCOPE=SPFILE SID='abc2';

$ srvctl stop database -d abc

继续删除第三个实例
$ srvctl start instance -d nocdd -instance nocdd1
$ dbca -silent -deleteInstance -nodeList d2 -gdbName nocdd -instanceName nocdd2

检测交换分区,及产品软件清单目录 【节点2执行，删除的节点】
oracle$$ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d2}" -local Starting Oracle Universal Installer…
Checking swap space: must be greater than 500 MB. Actual 3072 MB Passed
The inventory pointer is located at /etc/oraInst.loc
[oracle@d2:/home/oracle]$ cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall

oracle$cd $ORACLE_HOME/deinstall
oracle$./deinstall -local
······
Specify the list of database names that are configured locally on this node for this Oracle home. Local
configurations of the discovered databases will be removed []: 回车，默认读取当前oracle_home目录 ······
The cluster node(s) on which the Oracle home deinstallation will be performed are:d2
Oracle Home selected for deinstall is: /u03/app/oracle/product/19.0.0/db_1
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Do you want to continue (y - yes, n - no)? [n]: y 继续,yes ······
Successfully detached Oracle home '/u03/app/oracle/product/19.0.0/db_1' from the central inventory on the local node.
Successfully deleted directory '/u03/app/oracle/product/19.0.0/db_1' on the local node.
Successfully deleted directory '/u03/app/oracle' on the local node.
Oracle Universal Installer cleanup was successful.
Oracle deinstall tool successfully cleaned up temporary directories.

**更新inventory

在保留任一节点执行命令

su** - oracle cd $ORACLE_HOME/oui/bin ./runInstaller -updateNodeList ORACLE_HOME=/u03/app/oracle/product/19.0.0/db_1 "CLUSTER_NODES={d1}"

检查集群节点是否被锁定,只需要在删除的节点操作即可，是检测是否被锁定
[grid@d1:/home/grid]$ olsnodes -s -t d1 Active Unpinned
[grid@d2:/home/grid]$ olsnodes -s -t PRCO-19: Failure retrieving list of nodes in the cluster
PRCO-2: Unable to communicate with the clusterware

如果节点被锁定，运行crsctl unpin css命令后继续下一步

crsctl unpin css -n delete_node

在不删除二进制文件的情况下取消配置Oracle Clusterware https://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/unconfiguring-oracle-clusterware-without-removing-binaries.html#
GUID-41C5E8EE-68F9-4849-B4B8-BD82D4A17A57
在删除的节点上,以root用户执行
su - root
cd /u01/app/19.0.0/grid/crs/install

/u01/app/19.0.0/grid/perl/bin/perl ./rootcrs.pl -deconfig -force

直接执行./rootcrs.pl提示./rootcrs.pl: line 88: =head1: command not found,perl无法执行脚本。
2020/04/15 17:03:56 CLSRSC-336: Successfully deconfigured Oracle Clusterware stack on this node

#若删除整个集群，在最后一个节点执行
#./rootcrs.pl -deconfig -force -lastnode
在删除的节点上执行删除软件命令操作d2，只删除本地节点2 su - grid cd $ORACLE_HOME/deinstall ./deinstall -local
ASM configuration was not detected in this Oracle home. Was ASM configured in this Oracle home (y|n) [n]: y
【asm 与grid安装在一起共享一个oracle_home]
ASM Diagnostic Destination : /u01/app/grid
ASM Diskgroups :
ASM diskstring : /dev/*_1*
Diskgroups will not be dropped
If you want to retain the existing diskgroups or if any of the information detected is incorrect, you can modify by entering 'y'.
Do you want to modify above information (y|n) [n]: n 【不修改上述对磁盘组的操作,不删除磁盘组] Oracle Grid Infrastructure Home is:
Oracle Home selected for deinstall is: /u01/app/19.0.0/grid
Inventory Location where the Oracle home registered is: /u01/app/oraInventory
Following Oracle Restart enabled listener(s) will be de-configured: ASMNET1LSNR_ASM,LISTENER,LISTENER_SCAN1
ASM instance will be de-configured from this Oracle home
Do you want to continue (y - yes, n - no)? [n]: y [确认清理上述资源】
####################### DEINSTALL CLEAN OPERATION SUMMARY #######################
Successfully detached Oracle home '/u01/app/19.0.0/grid' from the central inventory on the local node.
Failed to delete directory '/u01/app/19.0.0/grid' on the local node due to error : Either user has no permission
to delete or file is in use.
Review the permissions and manually delete '/u01/app/19.0.0/grid' on local node.
Failed to delete directory '/u01/app/oraInventory' on the local node due to error : Either user has no permission
to delete or file is in use.
Review the permissions and manually delete '/u01/app/oraInventory' on local node.
Oracle Universal Installer cleanup completed with errors.

Run 'rm -r /etc/oraInst.loc' as root on node(s) 'd2' at the end of the session. Run 'rm -r /opt/ORCLfmap' as root on node(s) 'd2' at the end of the session. Run 'rm -r /etc/oratab' as root on node(s) 'd2' at the end of the session. Review the permissions and contents of '/u01/app/grid' on nodes(s) 'd2'.
If there are no Oracle home(s) associated with '/u01/app/grid', manually delete '/u01/app/grid' and its contents.
Oracle deinstall tool successfully cleaned up temporary directories.

root# 删除上述目录! 四个目录均递归删除！

删除CRS配置【存活的节点执行,已删除的crs都没了】 su - root cd /u01/app/19.0.0/grid/bin ./crsctl delete node -n d2
[grid@d1:/u01/app/19.0.0/grid/crs/install]$ cluvfy stage -post nodedel -n 2 -verbose
Verifying Node Removal …
Verifying CRS Integrity …PASSED
Verifying Clusterware Version Consistency …PASSED
Verifying Node Removal …PASSED
Post-check for node removal was successful.
CVU operation performed: stage -post nodedel
Date: Apr 15, 2020 2:28:00 AM
CVU home: /u01/app/19.0.0/grid/
User: grid
**发现存在一个vip 未被删除FAILED OVER,STABLE

/u01/app/19.0.0/grid/bin/srvctl remove vip -vip d2 -force**

二、添加节点

前期准备工作，需要与安装RAC工作一样，本次环境都是好的【新增节点不需要再需要安装介质】
$ mkdir -p /u01/app/19.0./grid
$ ssh d1 date;ssh d2 date; ssh d1-priv date;ssh d2-priv date;
运行CVU程序
su - grid
cd ${ORACLE_HOME}/bin
cluvfy comp peer [-refnode ] -n node_list [-orainv orainventory_group] [-osdba osdba_group] [-verbose]
CVU用作检查与其他节点的兼容性的参考的节点。如果未指定此选项，则CVU报告节点列表中所有节点的值。
grid_node1$cluvfy comp peer -refnode d1 -n d2 -*verbose
*加节点（GI） grid_node1$/u01/app/19.0./grid/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={d2-priv}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={d2-vip}"

As a root user, execute the following script(s):
. /u01/app/oraInventory/orainstRoot.sh
. /u01/app/19.0./grid/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes: [d2]
Execute /u01/app/19.0./grid/****root.sh on the following nodes: [d2]
······
**
[root@d1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.

验证

$ cluvfy stage -post nodeadd -n d2

加节点（DB)
oracle_node1$/u03/app/oracle/product/19.0./db_1/addnode/addnode.sh -silent -ignoreSysPrereqs -ignorePrereqFailure "CLUSTER_NEW_NODES={d2}"
As a root user, execute the following script(s):
. /u03/app/oracle/product/19.0./db_1/root.sh
Execute /u03/app/oracle/product/19.0./db_1/root.sh on the following nodes: [d2]

--
Start the Instance on the New Node
The prerequisite steps to take prior to adding an instance using DBCA are as follows:.
Ensure that all the existing instances have been started using an spfile from the shared location.
Verify that all the instances are listening on the default grid listener. The following command displays all the existing instances:
$ $CRS_HOME/bin/lsnrctl status
Run $ORACLE_HOME/bin/dbca from any of the existing RAC instance Oracle homes.
Select "Oracle Real Application Clusters" database and click Next.
Select "Instance Management" and click Next.
Select "Add an Instance" and click Next.
Click Next to accept the default instance name (or change it, if necessary).
Check the summary window.

测试环境3套DB,启动单个实例后，静默方式添加节点2实例
$ srvctl start instance -d abc -instance abc1 $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName abc -instanceName abc2 -sysDBAUserName sys -sysDBAPassword oracle $ srvctl start database -d nocdd $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName nocdd -instanceName nocdd2 -sysDBAUserName sys -sysDBAPassword oracle $ srvctl start database -d orcl $dbca -ignorePreReqs -ignorePrereqFailure -silent -addInstance -nodeName d2 -gdbName orcl -instanceName orcl2 -sysDBAUserName sys -**sysDBAPassword oracle

DB_ALERT
**

2020-04-16T00:50:34.958060+08:00
CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192
2020-04-16T00:50:36.603592+08:00
Completed: CREATE SMALLFILE UNDO TABLESPACE "UNDOTBS2" DATAFILE SIZE 290M AUTOEXTEND ON NEXT 5120K MAXSIZE 32767M BLOCKSIZE 8192
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,
GROUP 4 SIZE 200M
2020-04-16T00:50:39.079100+08:00
Completed: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 SIZE 200M,
GROUP 4 SIZE 200M
ALTER DATABASE ENABLE PUBLIC THREAD 2
Completed: ALTER DATABASE ENABLE PUBLIC THREAD 2
2020-04-16T00:50:39.185364+08:00
ALTER SYSTEM SET instance_number=2 SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:50:39.213198+08:00
ALTER SYSTEM SET thread=2 SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:50:39.243485+08:00
ALTER SYSTEM SET undo_tablespace='UNDOTBS2' SCOPE=SPFILE SID='nocdd2';
2020-04-16T00:51:29.154327+08:00

补丁信息发现,添加节点补丁自动打上了！！！ [grid@d1:/u01/app/19.0./grid/OPatch]$ ./opatch lsinventory
oracle ······
SQL> select ACTION,NAMESPACE,VERSION,BUNDLE_SERIES,COMMENTS FROM dba_registry_history;

ACTION NAMESPACE VERSION BUND COMMENTS
---------- ---------- -------------------- ---- ----------------------------------------
BOOTSTRAP DATAPATCH 19 RDBMS_19.5.0.0.0DBRUR_LINUX.X64_191213
RU_APPLY SERVER 19.0.0.0.0 Patch applied on 19.3.0.0.0: Release_Update - 190410122720