Upgrading HDF to CFM on CDP Private Cloud Base

Cloudera provides an AM2CM tool to enable you to upgrade HDF cluster to CFM cluster on CDP Private Cloud Base easily. You need to download the AM2CM tool, configure settings for the tool, and generate Cloudera Manager template. After you generate the Cloudera Manager template, you need to configure the parcel, deploy Cloudera Manager, add CFM parcel, and activate the parcel through the Cloudera Manager UI.

  1. Ensure that you have the latest Ambari blueprint (JSON file).
  2. Download the AM2CM tool:
    wget https://archive.cloudera.com/am2cm/2.3.0.0/redhat7/yum/tars/am2cm/am2cm-tool-2.3.0.0-60.tar.gz
  3. Extract the TAR file:
    tar -xvf am2cm-tool-2.3.0.0-60.tar.gz
  4. Check the project files:
    # cd am2cm-2.3.0.0-60
                            # [root@ccycloud-1 am2cm-2.3.0.0-60]# ls
    am2cm-2.2.0.2.3.0.0-60.jar  am2cm.sh  conf  lib  ranger_policy_migration.py
     
     
    # [root@ccycloud-1 am2cm-2.3.0.0-60]# ls conf
    ambari_blueprint_hostgroups.josn  blueprint.json  cm-config-mapping-custom.ini  cm-config-mapping.ini  log4j.properties  ranger_policy_migration.py  service-config.ini  service-default-config.ini  user-settings.ini
     
     
    # mv conf/blueprint.json conf/blueprint_initial.json
    # [root@ccycloud-1 am2cm-2.3.0.0-60]# ls conf
    ambari_blueprint_hostgroups.josn  blueprint_initial.json  cm-config-mapping-custom.ini  cm-config-mapping.ini  log4j.properties  ranger_policy_migration.py  service-config.ini  service-default-config.ini  user-settings.ini
  5. Set JAVA_HOME if not already done:
    [root@ccycloud-1 am2cm-2.3.0.0-60]# echo $JAVA_HOME
    /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64
  6. Copy your Ambari blueprint to the right path, that is the conf directory of the AM2CM tool, from the Home directory:
    cp /data/backups/HDFTOCDF_blueprint_with_hosts_20201103144235.json /root/am2cm-2.3.0.0-60/conf
  7. Configure the user-settings.ini file:

    It is really important to specify the cluster.name and cluster.displayname to the same original Ambari cluster name. Also, it is worth to fill all password field.

    # Cluster details
    cluster.name=zt-HDFTOCDF
    cluster.displayname=HDFTOCDF
    
    # Update this with correct CM/CDP version before running the tool
    cluster.fullversion=7.1.7
    
    cluster.originating_source=AM2CM version 2.0
    
    # This is needed to update Ranger/Atlas policy service names.
    ambari.cluster.name=HDFTOCDF
    
    #This flag enables to translate Config Groups to Role config groups.
    cm.rolegroups.enable = false
    
    # FROM JDBC URL - tool reads only Hostname and port number - rest of the details like Database type, Database name, database user reads from from Blueprint.
    
    # Do not enclose double quotes for passwords, provide exact password.
    
    # Hive JDBC settings
    SERVICE_HIVE_hive_metastore_database_password=password
    SERVICE_HIVE_hive_jdbc_url_override=JDBC_URL
    
    # Oozie JDBC settings
    SERVICE_OOZIE_oozie_database_password=password
    SERVICE_OOZIE_oozie_service_JPAService_jdbc_url=JDBC_URL
    
    # Ranger JDBC settings
    SERVICE_RANGER_ranger_database_password=password
    SERVICE_RANGER_rangeradmin_user_password=password
    SERVICE_RANGER_rangerusersync_user_password=password
    SERVICE_RANGER_rangertagsync_user_password=password
    SERVICE_RANGER_rangerkeyadmin_user_password=password
    SERVICE_RANGER_ranger_usersync_ldap_ldapbindpassword=password
    
    SERVICE_RANGER_KMS_ranger_kms_master_key_password=password
    SERVICE_RANGER_KMS_ranger_kms_database_password=password
    
    #Knox Settings
    SERVICE_KNOX_gateway_master_secret=admin
  8. If your cluster contains Kafka service, please override the kafka-broker-ids.ini files with the one you created.
    For more details, see Extracting Kafka broker IDs.
  9. Run the command to generate the Cloudera Manager template:
    # cd am2cm-2.3.0.0-60
    # chmod +x ./am2cm.sh
    [root@ccycloud-1 am2cm-2.3.0.0-60]# ./am2cm.sh -bp conf/HDFTOCDF_blueprint_with_hosts_20201103144235.json -dt conf/cm_deployment_template.json --source-version=HDF352
     
    INPUT Ambari Blueprint : conf/HDFTOCDF_blueprint_with_hosts_20201103144235.json
    OUTPUT CM Template     : conf/cm_deployment_template.json
     
    Starting blueprint to CM Template migration
    Total number of hosts in blueprint: 4
     
    Your cluster has services (listed below) that are not handled by this migration tool.
    AMBARI_METRICS
    The tool will skip the above identified service related configs.
     
    ** This migration tool is a technical preview only **
     
    Do you want to proceed with migration (Y OR N)? (N):
    Y
    Processing: LIVY
    Processing: SOLR
    Processing: TEZ
    Processing: HDFS
    Processing: OOZIE
    Processing: SQOOP_CLIENT
    Processing: NIFIREGISTRY
    Processing: ZOOKEEPER
    Processing: HBASE
    Processing: YARN
    Processing: RANGER_KMS
    Processing: KNOX
    Processing: ATLAS
    Processing: HIVE_ON_TEZ
    Processing: RANGER
    Processing: HIVE
    Processing: KAFKA
    Processing: NIFI
    Processing: SPARK_ON_YARN
    Adding: QUEUEMANAGER
    CM Template is generated at : /root/am2cm-2.3.0.0-60/conf/cm_deployment_template.json
    Successfully completed
  10. Check for errors in the tool log for Cloudera Manager migration:
    less am2cm-2.3.0.0-60/cm_migration.log