Configure Knox for WebHDFS HA

« Prev

	Important
	Before you can configure the Knox Gateway for WebHDFS HA (high availability), you must first configure WebHDFS for Knox.

Knox provides basic failover and retry functionality for REST API calls made to WebHDFS when HDFS HA has been configured and enabled.

To enable HA functionality for WebHDFS in Knox the following configuration must be added to the topology file.

<provider>
   <role>ha</role>
   <name>HaProvider</name>
   <enabled>true</enabled>
   <param>
       <name>WEBHDFS</name>
       <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>
   </param>
</provider>

The <role> and <name> of the provider must be as shown above. The <name> in the <param> section must match that of the service role name that is being configured for HA, and the <value> in the <param> section is the configuration for that particular service in HA mode. In this case the <name> is WEBHDFS.

The various configuration parameters are described below:

maxFailoverAttempts -- The maximum number of times a failover will be attempted. The current failover strategy is very simplistic in that the next URL in the list of URLs provided for the service is used, and the one that failed is put at the bottom of the list. If the list is exhausted and the maximum number of attempts has not been reached, the first URL that failed will be tried again (the list will start again from the original top entry).
failoverSleep -- The amount of time in milliseconds that the process will wait or sleep before attempting to failover.
maxRetryAttempts -- The maximum number of times that a retry request will be attempted. Unlike failover, the retry is done on the same URL that failed. This is a special case in HDFS when the node is in safe mode. The expectation is that the node will come out of safe mode, so a retry is desirable here as opposed to a failover.
retrySleep -- The amount of time in milliseconds that the process will wait or sleep before a retry is issued.
enabled - Flag to turn the particular service on or off for HA.

For the service configuration itself, the additional URLs for standby nodes should be added to the list. The active URL (at the time of configuration) should ideally be added at the top of the list. For example:

<service>
    <role>WEBHDFS</role>
    <url>http://{host1}:50070/webhdfs</url>
    <url>http://{host2}:50070/webhdfs</url>
</service>

​Configure Knox for WebHDFS HA