Post-upgrade Tasks

Atlas Migration and HBase Hook Settings

The settings for the Atlas migration need to be removed once the upgrade has been completed. On the Ambari Dashboard, select Atlas > Configs > Advanced > Custom application-properties, then click the red Remove button to remove the atlas.migration.data.filename property. Restart Atlas when prompted by Ambari.
When upgrading to HDP-3.1 from earlier versions, you may need to manually enable the HBase hook. On the Ambari dashboard, select HBase > Configs > Advanced hbase-env, then select the Enable Atlas Hook checkbox. Click Save, then restart HBase and any other services that require a restart.

Ambari Metrics, SmartSense, LogSearch

Take the Ambari Metrics, SmartSense, and Log Search Services out of Maintenance Mode by choosing Actions > Turn Off Maintenance Mode from each Service page.

Ambari Infra-Migrate & Restore

Follow the steps below to restore the data previously backed up:

SSH to the host where the migrationConfigGenerator.py was run prior to the HDP Upgrade. This will be from one of your Ambari Infra Solr instances. Please ensure you are in the current working directory containing the ambari_solr_migration.ini file.
Export the variable used to hold the path to the ini file.
export CONFIG_INI_LOCATION=ambari_solr_migration.ini
Migrate the data to ensure it’s in the right format to import into Solr 7. Please note that this script can take a long time to run depending on the volume of data backed up. It is recommended to run this script using the nohup command.
nohup /usr/lib/ambari-infra-solr-client/ambariSolrMigration.sh --ini-file $CONFIG_INI_LOCATION --mode migrate-restore
Re-index the migrated data into your current collections so the backed up data is visible in all of the tools using the Infra Solr instances. Please note that this script can take a long time to run depending on the volume of data backed up. It is recommended to run this script using the nohup command.
nohup /usr/lib/ambari-infra-solr-client/ambariSolrMigration.sh --ini-file $CONFIG_INI_LOCATION --mode transport

Migrate Ambari Metrics Data

Use the following steps to migrate data from the previous AMS schema to the new AMS schema.

	Note
	The migration code only migrates the metrics that are matching the ones present in the whitelist file. Review the whitelist file before the data migration to ensure that it contains all the required metrics.

Ensure the Ambari Metrics System is started. If it is not, in the Ambari Web UI, click Ambari Metrics, then select Actions > Start.
SSH into a host that is running a Metrics Collector. You can locate this host by going to the Ambari Web UI and clicking Hosts. Click on the Filter icon and type in "Metrics Collector: All" to find each host that has a Metrics Collector installed on it.
SU to the Ambari Metrics user. This is 'ams' by default, but if you don’t know which user is configured in your cluster go to the Ambari Web UI and click Cluster Admin > Service Accounts, and then look for "Ambari Metrics User".
# su ams

Run the command to migrate data from the old Ambari Metrics schema to the new.

$ /usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf/ upgrade_start /etc/ambari-metrics-collector/conf/metrics_whitelist

Note

The default time period for data migration is one month (2592000000 milli seconds), which can be overwritten if required. For overwriting the data migration to more than one month, you must provide the timeframe upto which data need to be migrated along with the command.

/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf/ upgrade_start /etc/ambari-metrics-collector/conf/metrics_whitelist "31556952000"

"31556952000" is the amount of days in milliseconds to be preserved and 31556952000 stands for 1 year.

Once the upgrade process has started, logs are available in the <ams-log-dir>/ambari-metrics-migration.log file.

Update Ranger Passwords

Ranger password validation has been updated in HDP 3.1, and to conform to these new password policies, the following Ranger passwords need to be updated to ensure that they have at least 8 characters with minimum one alphabet and one numeric. These passwords cannot contain the following special characters: " ' \ `

Ranger Admin

The following new passwords need to be populated with valid passwords that also conform to the password policy:

Ranger Usersync User’s Password
Ranger Tagsync User’s Password
Ranger KMS Keyadmin User’s Password

If Applicable: Configure Custom Spark SQL Warehouse Directory

Since Spark 2.0.0, Spark references spark.sql.warehouse.dir as the default Spark SQL Warehouse location. In HDP-2.6.x, the standard value for the spark.sql.warehouse.dir property is /apps/hive/warehouse. On an Ambari cluster with HDP-2.6.x, this property must be set manually both in "Custom spark2-defaults" and "Custom spark2-thrift-sparkconf".

When upgrading from HDP-2.6.x to HDP-3.x, Ambari migrates the spark.sql.warehouse.dir property to "Advanced spark2-defaults" and "Advanced spark2-thrift-sparkconf", and changes the value of this property to /apps/spark/warehouse. This is done to accommodate the separate Spark and Hive catalogs introduced in HDP-3.x.

If you used a custom setting for the spark.sql.warehouse.dir property in HDP-2.6.x, the Ambari upgrade to HDP-3.x ignores the custom setting and sets the value of the spark.sql.warehouse.dir property to /apps/spark/warehouse in both "Advanced spark2-defaults" and "Advanced spark2-thrift-sparkconf".

If you want to use a custom Spark SQL warehouse after upgrading to HDP-3.x, select Spark2 > Configs, then use Add Property in "Advanced spark2-defaults" and "Advanced spark2-thrift-sparkconf" to update the value of the spark.sql.warehouse.dir property with the custom setting.

If Hive is configured to run as a custom user, you must change the ownership of your new Hive warehouse directory to be owned by that custom user.

hdfs dfs -chown cstm-hive /warehouse/tablespace/managed/hive/

​Post-upgrade Tasks

Post-upgrade Tasks