Software Requirements

Atlas - update Solr replication factor

In Solr, before upgrading, when you set the replication factor to 2, it implies 1 leader replica and 1 follower replica are available in each of the nodes. Later, when performing rolling upgrade/restart, you still have a follower replica in a node which can be used to serve Atlas without any downtime from the client (Atlas). Restart Atlas for the changes to take effect.

Atlas - update Kafka replication factor to the number of available Kafka brokers

You must perform certain prerequisite tasks based on whether Kafka topics are available and Atlas is installed on your cluster.

When Kafka topics are available, use the following procedure.

1. Stop the Atlas service.
2. Delete the topics (ATLAS_HOOK, ATLAS_ENTITIES, and ATLAS_SPARK_HOOK) manually. For more information, see Monitoring Kafka topics and Deleting a Kafka topic.
3. Set the replication factor to the number of available Kafka brokers in Kafka.
4. Restart Kafka.
5. Restart Atlas.
6. Check that the topics are created with a replication factor as set in Kafka.

Data Analytics Studio (Deprecated)

Cloudera recommends that you delete the DAS database to free up resources.

HDFS

You must ensure that the High Availability mode is enabled and more than one HDFS NameNode is running.

HMS

You must enable High Availability for Hive Metastore (HMS) and ensure that more than one instance of HMS is running. For details, see Configuring HMS for high availability.

Hive on Tez

Hue

Impala

You must enable High Availability (HA) for Impala Catalog server and StateStore server by configuring more than one Impala Catalog server and one StateStore server instance on different nodes. For details, seeConfiguring Impala for High Availability

Kafka

The Kafka service in CDP consists of multiple service role types. Each role type represents core features in Kafka. The following collects the ZDU prerequisites for each role type. You must ensure that these prerequisites are met to ensure that the service can be upgraded with zero downtime.

Although the Kafka service supports ZDU if the following prerequisites are met, Kafka service monitoring provided by SMM is briefly interrupted during the upgrade because SMM does not support ZDU.

Kafka Connect

• Ensure that your Kafka Connect roles are running in HA mode. That is, you have more than a single Kafka Connect role deployed in the cluster. Cloudera recommends a minimum of 3.
• If you have external clients that connect to the Kafka Connect REST API, ensure that you are using a load balancer to redirect REST API calls. If you do not have a load balancer, expect service downtime as Kafka Connect provides no automatic failover between its REST endpoints.

MirrorMaker

Ensure that your MirrorMaker roles are running in HA mode. That is, you have more than a single MirrorMaker role deployed in the cluster. Cloudera recommends a minimum of 3.

Key Trustee Server

1. Back up Key Trustee Server manually on both Active and Passive hosts
2. Back up the Key Trustee Server configuration directory on both Active and Passive hosts
3. Move the .backup files (keytrustee-db.zip and keytrustee-conf.zip) to a secure location on both Active and Passive hosts.

Kudu

Ensure that the High Aavailability mode is enabled and at least three Kudu master servers and three tablet servers are running and all the tables have a replication factor of three or more.

Livy

Omid

• Ensure that the HA mode is enabled and more than one OMID TSO server is running.
• Ensure that the OMID service supports the rolling restart functionality.
• Ensure that the Enable HA for Omid TSO server option is selected under Configurations in Cloudera Manager.

Phoenix

• Ensure that the HA mode is enabled and more than one PQS server is running.
• Ensure that the Phoenix service supports the rolling restart functionality.

Ranger KMS

Schema Registry

When upgrading from Cloudera Runtime 7.1.8 to 7.1.9 or higher, if you only have external clients connecting to the server, you can ensure zero downtime if you upgrade your external clients before upgrading the cluster. The actions you must take differ depending on the type client you are using:

• If you are using a generic HTTP client, ensure that retry and failover is implemented in the client code.
• If you are using the Cloudera distributed Schema Registry Java client, ensure that you upgrade your applications that use the client before you upgrade the cluster. You can access the latest version of the client in the Cloudera Manager Maven repository at https://repository.cloudera.com/artifactory/cloudera-repos/. Alternatively, you can choose to implement retry and failover yourself. The following code snippet is an example of the configuration that is required.

  Map<String, Object> retryParams = new HashMap<>();
       		retryParams.put("sleepTimeMs", SLEEP_TIME);
       		retryParams.put("maxAttempts", MAX_ATTEMPTS);
       		retryParams.put("timeoutMs", OVERALL_TIMEOUT);
  
          	Map<String, Object> retryConfig = new HashMap<>();
          	retryConfig.put("className", "com.hortonworks.registries.schemaregistry.retry.policy.ExponentialBackoffPolicy");
          	retryConfig.put("config", retryParams);
  		
  		Map<String, ?> clientConfig = new HashMap<>();
  		clientConfig.put("schema.registry.url", "url1,url2")
          	clientConfig.put("schema.registry.client.retry.policy", retryConfig);
  		clientConfig.put(...); // other client configuration
          	SchemaRegistryClient client = new SchemaRegistryClient(clientConfig);

Solr

When upgrading from Cloudera Runtime 7.1.9 or higher, you can ensure zero downtime by meeting the following prerequisites for your Solr services before initiating an upgrade with rolling restart:

• The Solr service must have more than one server to perform a rolling restart. Otherwise the service will not be available during the restart.

• You are highly recommended to change the workload Solr collections to read-only before performing a rolling restart. Revert this setting after the upgrade is finished.

• Ensure you have replication factor set to greater than one for all workload Solr collections. If this criteria is not met, your queries may fail at the time of upgrade and your application may have to re-send the queries once the upgrade is successful.

SRM

The Streams Replication Manager (SRM) service in CDP consists of multiple service role types. Each role type represents core features in SRM. You must ensure that these prerequisites are met to ensure that the service can be upgraded with zero downtime.

Although the SRM service supports ZDU if the following prerequisites are met, replication monitoring provided by SMM is briefly interrupted during the upgrade because SMM does not support ZDU.

SRM Driver

Ensure that your SRM Driver roles are running in HA mode. That is, you have more than a single SRM Driver role deployed in the cluster. Cloudera recommends a minimum of 3.

YARN

Upgrade from 7.1.8 or 7.1.9 or 7.1.9 SP1 to 7.3.1

For security reasons, CDP 7.3.1. has the mapreduce.shuffle.ssl.enabled option turned on. If this option is not set to true before the upgrade, already running MR2 applications fail during the upgrade in the shuffle phase with the java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out exception.

When no MR2 jobs are running, configure the mapreduce.shuffle.ssl.enabled option to true before the upgrade. Restart the YARN service and deploy the client configurations.

Upgrade from 7.1.8 to 7.1.9 or 7.1.9 SP1

Upgrading from Cloudera Runtime 7.1.8 to 7.1.9 or 7.1.9 SP1 displays the error TestDFSIO intermittently failing with the RuntimeException: native snappy library not available.

You must retry the job.