TestBundleJobsFilter
test fails on RHEL v6.3, Oracle v6.3, and SUSE clusters with PostgreSQL.This issue is caused due to the strict typing of PostgreSQL which restricts the auto casting of
string integer
to aninteger
. The issue is reported when string representation of integer values is substituted into a query for PostgreSQL on the JPA layer.Delegation Token renewal exception in JobTracker logs.
Problem: The following exception is reported in the JobTracker log file when executing a long running job on Oozie in secure mode:
ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:jt/hor1n22.gq1.ygridcore.net@HORTON.YGRIDCORE.NET cause:org.apache.hadoop.security.AccessContr olException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt 2013-04-25 15:09:41,543 ERROR org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal: Exception renewing tokenIdent: 00 06 68 72 74 5f 71 61 02 6a 74 34 6f 6f 7a 69 65 2f 68 6f 72 31 6e 32 34 2e 67 71 31 2e 79 67 72 69 64 63 6f 72 65 2e 6e 65 74 40 48 4f 52 54 4f 4e 2e 59 47 52 49 44 43 4f 52 45 2e 4e 45 54 8a 01 3e 41 b9 67 b8 8a 01 3e 65 c5 eb b8 8f 88 8f 9c, Kind: HDFS_DELEGATION_TOKEN, Service: 68.142.244.41:8020. Not rescheduled org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:678) at org.apache.hadoop.security.token.Token.renew(Token.java:309) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:221) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask$1.run(DelegationTokenRenewal.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.mapreduce.security.token.DelegationTokenRenewal$RenewalTimerTask.run(DelegationTokenRenewal.java:216) at java.util.TimerThread.mainLoop(Timer.java:512) at java.util.TimerThread.run(Timer.java:462) Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.security.AccessControlException: Client mapred tries to renew a token with renewer specified as jt at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.renewToken(AbstractDelegationTokenSecretManager.java:267) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renewDelegationToken(FSNamesystem.java:6280) at org.apache.hadoop.hdfs.server.namenode.NameNode.renewDelegationToken(NameNode.java:652) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1405) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1401) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1195) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1118) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy7.renewDelegationToken(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:676) ... 9 more
Workaround: Any new job job on secure cluster that runs longer than the validity of the Kerberos ticket (typically 24 hours) will fail as the delegation token will not be renewed.