Problem: Oozie fails smoke tests in secured cluster.
Workaround:
Download the following files attached to, https://issues.apache.org/jira/browse/AMBARI-2879:
check_oozie_status.sh
oozieSmoke.sh
Replace
/var/lib/ambari-agent/puppet/modules/hdp-nagios/files/check_oozie_status.sh
with the downloaded file.On the Nagios Server host machine, restart Nagios using the following command:
service nagios start
Replace
/var/lib/ambari-agent/puppet/modules/hdp-oozie/files/oozieSmoke.sh
with the downloaded file all the hosts in your cluster.Restart Oozie on the Oozie Server host machine using the following command:
sudo su -l $OOZIE_USER -c "cd $OOZIE_LOG_DIR/log; /usr/lib/oozie/bin/oozie-start.sh"
where:
$OOZIE_USER
is the Oozie Service user. For example,oozie
$OOZIE_LOG_DIR
is the directory where Oozie log files are stored (for example:/var/log/oozie
).
Problem:
TestBundleJobsFilter
test fails on RHEL v6.3, Oracle v6.3, and SUSE clusters with PostgreSQL.This issue is caused due to the strict typing of PostgreSQL which restricts the auto casting of
string integer
to aninteger
. The issue is reported when string representation of integer values is substituted into a query for PostgreSQL on the JPA layer.Problem: Delegation Token renewal exception in JobTracker logs.
The following exception is reported in the JobTracker log file when executing a long running job on Oozie in secure mode:
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:jt/hor1n22.gq1.ygridcore.net@HORTON.YGRIDCORE.NET cause:org.apache.hadoop.security.AccessContr olException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt 2013-04-25 15:09:41,543 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 06 68 72 74 5f 71 61 02 6a 74 34 6f 6f 7a 69 65 2f 68 6f 72 31 6e 32 34 2e 67 71 31 2e 79 67 72 69 64 63 6f 72 65 2e 6e 65 74 40 48 4f 52 54 4f 4e 2e 59 47 52 49 44 43 4f 52 45 2e 4e 45 54 8a 01 3e 41 b9 67 b8 8a 01 3e 65 c5 eb b8 8f 88 8f 9c, Kind: HDFS_DELEGATION_TOKEN, Service: 68.142.244.41:8020. Not rescheduled org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:678) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:6280) at org.apache.hadoop.hdfs.server.namenode.NameNode.renewDelegationToken(NameNode.java:652) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy7.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:676) ... 9 more
Workaround: Any new job job on secure cluster that runs longer than the validity of the Kerberos ticket (typically 24 hours) will fail as the delegation token will not be renewed.