Cloudera Docs

Checklist for large clusters

If you have a large cluster with a large data volume, you should configure the Ambari NameNode restart timeout parameter on the Ambari Server host, to ensure that the Ambari requests for starting the NameNode do not timeout during an upgrade.

In a large cluster, NameNode startup processes can take a long time. NameNode startup time depends not only on host properties, but also on data volume and network parameters. To ensure that the Ambari requests for starting the NameNode do not timeout during an upgrade, you should configure the Ambari NameNode restart timeout parameter, upgrade.parameter.nn-restart.timeout in /etc/ambari-server/conf/ambari.properties on the Ambari Server host. You may need to add the restart timeout parameter and value to the Ambari server host following a default installation. For a large cluster, you should add 10% to the usual time (in seconds) required to restart your NameNode. Although no standard way to determine an appropriate value exists, you may use the following guidance:

For example, record the time (seconds) required to restart the active NameNode for your current Ambari server version. If restarting takes 10 minutes, (600 seconds), then add upgrade.parameter.nn-restart.timeout=660 to the /etc/ambari-server/conf/ambari.properties file on the Ambari Server host.

After adding or resetting the Ambari NameNode restart parameter, restart your Ambari server before starting the HDP upgrade.

ambari-server restart

For Ambari Upgrades

  • Review supported Ambari Server database versions using the Cloudera Support Matrix. Review the resources required for the database upgrade. You will upgrade your Ambari Server database during the Ambari upgrade procedure.

  • Ambari 7.1.x.x only supports the following operations when running against a HDP 3.1.5 cluster:

    • Run Service Checks

    • Start, Stop, Restart a Service

    • Change Configuration

    • Enable and Disable Maintenance Mode

    • Disable Auto Start

    • Remove Services

    • Remove Components

    • Remove and Decommission Hosts

To restore full management functionality, use Ambari 7.1.x.x to upgrade to HDP intermediate bits as soon as possible.

For HDP Cluster Upgrades

  • Ensure sufficient disk space on /usr/hdp/<version> (roughly 10GB for each additional HDP release).

  • Additional components are added to the cluster as part of the HDP intermediate bits upgrade including YARN ATSv2, YARN Registry DNS, and additional Hive clients required for the Spark history server. If your cluster has kerberos enabled, you must configure Ambari to manage the kerberos administrator credentials prior to the upgrade, so that the appropriate kerberos principals can be created during the upgrade process.

  • You must take a backup of the running topology processes if your HDP cluster includes the Storm component. You must stop and remove the Storm services during the upgrade. CDP Private Cloud Base cluster does not support Storm. Storm can be replaced with Cloudera Streaming Analytics (CSA) powered by Apache Flink. For more information on Component changes in CDP Private Cloud, see Updated HDP Components.