Apache Spark Post Upgrade Migration Steps

After upgrading to CDH 6, you might have multiple Spark services configured, each with their own set of configurations, including event log locations. Decide which service to keep and then manually merge the two services.
Manually merge your Spark services by performing the following steps:
  1. Copy all relevant configurations from the service you are removing to the service you are keeping. To view and edit the configurations:
    1. In the Cloudera Manager Admin Console, go to the Spark service you are removing.
    2. Click the Configuration tab.
    3. Note the configurations.
    4. Go to the Spark service you are keeping and replicate the configuration.
    5. Click Save Changes.
  2. To preserve historic event logs:
    1. Identify the location of the event logs for the service you are removing:
      1. In the Cloudera Manager Admin Console, go to the Spark service you are removing.
      2. Click the Configuration tab.
      3. Search for: spark.eventLog.dir
      4. Note the path.
    2. Log into a cluster host and run the following command: 
      hadoop fs -mv <old_Spark_Event_Log_dir>/* <new_location>/.
  3. Using Cloudera Manager, stop and delete the Spark Service you have selected for removal:
    1. In the Cloudera Manager Admin Console, click the drop-down arrow next to the Spark service you are removing, and then select Stop.
    2. Click the drop-down arrow next to the Spark service you are removing, and then select Delete.
  4. Restart the remaining Spark service: Click the drop-down arrow next to the Spark service, and then select Restart.