Long-running applications need additional configuration since the default settings only allow the hdfs user's delegation tokens a maximum lifetime of 7 days which is not always sufficient.
Long-running applications such as Spark Streaming jobs will need additional
configuration since the default settings only allow the
hdfs user's delegation tokens a maximum lifetime of
7 days which is not always sufficient.
You can work around this by configuring the ResourceManager as a proxy user for the
corresponding HDFS NameNode so that the ResourceManager can request new tokens when
the existing ones are past their maximum lifetime. YARN will then be able to
continue performing localization and log-aggregation on behalf of the
hdfs user.
-
In Cloudera Manager, select the YARN service.
-
Click the Configuration tab.
-
Search for yarn-site.xml.
-
Find the ResourceManager Advanced Configuration Snippet (Safety
Valve) for yarn-site.xml.
-
Click the plus icon and add the followoing:
- Name:
yarn.resourcemanager.proxy-user-privileges.enabled
- Value: true
-
Click Save Changes.
-
Go back to the home page by clicking the Cloudera Manager logo.
-
Select the HDFS service.
-
Click the Configuration tab.
-
Search for core-site.xml.
-
Find the Cluster-wide Advances Configuration Snippet (Safety Valve)
for core-site.xml.
-
Click the plus icon and add the following:
- Name: hadoop.proxyuser.yarn.hosts
- Value: *
-
Click the plus icon again and add the following:
- Name:
hadoop.proxyuser.yarn.groups
- Value: *
-
Click Save Changes.
-
Click the Stale Service Restart icon that is next to the service
to invoke the cluster restart wizard.
-
Click Restart Stale Services.
-
Select Re-deploy client configuration.
-
Click Restart Now.
