Friday, March 4, 2016

Cluvfy returns "Unsuccessful" for most commands; trace has exectask.sh: "cannot execute" / "permission denied", or scp: "not found" [ID 549667.1]


________________________________________
       
In this Document
  Symptoms
     Examples:
  Changes
  Cause
     Cause #1 - Wrong permissions on exectask
     Cause #2 - scp in wrong location
  Solution
     Solution #1 - Adjust permissions on exectask.
     Solution #2 - Create a symlink to scp in the expected location.
     Scalability RAC Community
  References
________________________________________


Applies to:
Oracle Server - Standard Edition - Version: 10.2.0.1 to 11.2.0.2 - Release: 10.2 to 11.2
Oracle Server - Enterprise Edition - Version: 10.2.0.1 to 11.2.0.2   [Release: 10.2 to 11.2]
Information in this document applies to any platform.
***Checked for relevance on 19-Oct-2010***
Symptoms

Cluster is functioning perfectly, but running the Cluster Verification Utility (cluvfy) with any commands always returns "unsuccessful" with no further details, even with -verbose switch.
Examples:
1.
#cluvfy comp admprv -n mynode1,mynode2 -verbose -o crs_inst
Verifying administrative privileges
Checking user equivalence...
Check: User equivalence for user "root"
  Node Name                             Comment
  ------------------------------------  ------------------------
  mynode1                               passed
  mynode2                               passed
Result: User equivalence check passed for user "root".
Verification of administrative privileges was unsuccessful on all the nodes.

2.
#cluvfy comp nodecon -n mynode1,mynode2 -verbose
Verifying node connectivity
Verification of node connectivity was unsuccessful on all the nodes.

3.
#cluvfy comp crs -n mynode1,mynode2
Verifying CRS integrity
Verification of CRS integrity was unsuccessful on all the nodes.
4.
 ./runcluvfy.sh stage -post crsinst -n mynode1,mynode2 -verbose
Performing post-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "mynode1"
Destination                          Node Reachable?
------------------------------------ ------------------------
mynode1                              yes
mynode2                              yes
Result: Node reachability check passed from node "mynode1".


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name                        Comment
-------------------------------- ------------------------
mynode1                         passed
mynode2                         passed
Result: User equivalence check passed for user "oracle".

Post-check for cluster services setup was unsuccessful on all the nodes.
Changes
The issue can occur before, during or after installation.  After installation, the triggering event can be the application of a patch, or a change in the OS environment such as permissions change.  In some cases on 10.2 and 11.1 the issue was simply not noticed until after installation.
Cause
There are a couple of possible causes for this reason. To check for which cause you may be encountering, trace cvu and examine cvutrace.log .

For instructions on tracing cluvfy, please see this note:
Note 316817.1 - CLUSTER VERIFICATION UTILITY FAQ
under the section "How do I turn on tracing?"

If using runcluvfy.sh, see this note for tracing instructions:
Note 986822.1 - How to Collect CVU Trace / Debug Output Generated by RUNCLUVFY.SH
Cause #1 - Wrong permissions on exectask
After turning CVU tracing on, the following message can be seen in the cvutrace.log :
ksh: /tmp/CVU_10.2.0.1.0.1_dba/exectask.sh: cannot execute
Or, in an 11.2 install, in which the installer runs cluvfy, the installActionsDATE.log shows "exectask.sh: Execute permission denied", for example:
sh: /var/tmp/CVU_11.2.0.2.0_grid/exectask.sh: Execute permission denied.
Cause:
a) For already installed clusterware: Some CRS patches may change the permissions on $CRS_HOME/cv/remenv/exectask* .  If $CRS_HOME/cv/remenv/exectask* is not executable, then cluvfy will fail.
b) Previous runs of cvu can leave /tmp/CVU* or /var/tmp/CVU* directories in place; the permissions of exectask  must be correct in this location too, or cluvfy will fail.
Cause #2 - scp in wrong location
After turning CVU tracing on, the following message can be seen in the cvutrace.log :
exec exception:/usr/local/bin/scp: not found;
Cause: scp is not located in /usr/local/bin/scp but in a different place on the server.
Solution
Use either solution 1 or solution 2, depending on which of the above causes applies to your situation.
Solution #1 - Adjust permissions on exectask.

Steps:
1. Check the permissions with:
ls -l $CRS_HOME/cv/remenv/exectask*

2. Ensure that the files are owned by
oracle:oinstall
AND
permissions are 744 or 755

3.  Check for the existence of a /tmp/CVU* or /var/tmp/CVU* directory created by previous, failed run of cluvfy.  If the directory exists and exectask* exists in this directory, then make the same change to the permissions on these exectask* files as well.
Solution #2 - Create a symlink to scp in the expected location.

Steps:
1. Find scp
2. As root user, create a symlink to /usr/local/bin

eg. if scp is located in /usr/bin, create a symlink as follows:
# ln -s /usr/bin/scp /usr/local/bin/scp