Assumptions and prerequisites

Before you transition your cluster to CDP Private Cloud Base or migrating content from Navigator to Apache Atlas, ensure that you have collected all the credentials and set expectations for the time required for completing the transition. The prerequisites in this section help you to prepare in advance to transition.

In addition to the prerequisites outlined for the Cloudera Manager and CDP upgrades, you'll need the following for the Navigator to Atlas transition:

Deleted entities in Navigator. Check the Navigator Administration page to make sure that a successful purge has run recently. If it hasn't, consider running a purge before the transition. See Managing Metadata Storage with Purge.
Role to host assignments. Before you begin upgrading to CDP, make a plan for where you will install the Atlas server. In addition, Atlas depends upon HBase, Kafka, and Solr services; your plan should include host assignments for installing the components of these services. See Runtime Cluster Hosts and Role Assignments.
Resources for Atlas service. Atlas requires 16 GB of Java heap (Atlas Max Heapsize property) and 4 Solr shards (Initial Solr Shards for Atlas Collections property). Make sure the host you choose for Atlas has enough resources for all the services' requirements.

note
You must note about the default values for Initial Solr Shards for Atlas Collections in your Cloudera Manager UI. Before you commence the Atlas initialization process, based on your performance requirements, you must decide the actual (correct) values for Initial Solr Shards for Atlas Collections in your Cloudera Manager instance. Cloudera recommends to set 4 Solr shards (Initial Solr Shards for Atlas Collections property). You must also note that, you must not update or modify these values once the Atlas initialization has commenced. Additionally, note that once the Atlas initialization process is completed, modifying the value of Initial Solr Shards for Atlas Collections or Initial Solr Replication Factor for Collections will not have any effect on the collections for Atlas in Solr.
Resources for Solr service. During transition, Solr running to serve as Atlas' index requires 12 GB of Java heap (Java Heap Size of Solr Server in Bytes property). You can reset this back to make sure the host you choose for Atlas has enough resources for all the services' requirements.
Navigator credentials. The transition requires the username and password for a Navigator user with administrator privileges.
Local disk space needed for intermediate processing. The first two phases of the Navigator-to-Atlas transition produce intermediate files in /tmp in the local file system where Atlas is installed. See Estimating the time and resources needed for transition.
Local disk space for transition staging files. The first two phases of the Navigator-to-Atlas transition produce staging files on the local disk where Atlas is installed. See Estimating the time and resources needed for transition.
Time estimates for transition phases. Each phase of the transition runs independently from the upgrade. You can trigger them to run when convenient. See Estimating the time and resources needed for transition.

Estimating the time and resources needed for transition

While the cluster is starting up, you can plan for and start the transition process.

Inspect Navigator installation to determine the number of Navigator entities that will be transitioned. See How many Navigator entities are transitioned?

Estimate the time and disk space required for each phase of the transition.

The following transition rates are approximate and depend on the resources available on the Atlas host and other unknown factors. Note that the number of entities actually imported may be considerably less that the number of entities extracted. The transition process discards HDFS entities that are not referenced by processes that are transitioned (Hive, Impala, Spark).


Transition Phase	Transition Rate	Disk Space	Output File Size	Trial Data Points
Extraction	4 minutes / 1 million entities	100 MB / 1 million entities, less as volumes increase	65 MB / 1 million entities	10 million entities takes about 30 minutes; 256 million takes about 18 hours.
Transformation	1.5 minutes / 1 million entities	100 to 150 MB / 1 million entities, higher end of range with larger volumes	150 MB / 1 million entities	10 million entities takes about 20 minutes; 256 million takes about 6 hours.
Import	35 minutes / 1 million migrated entities	N/A	N/A	10 million entities takes about 4 hours; 256 million takes about 6 days.

Migration from Navigator to Atlas can be run only in non-HA mode

Migration import works only with a single Atlas instance.

If Atlas has been set up in HA mode before migration, you must remove the additional instances of Atlas, so that Atlas service has only one instance.

Later, start Atlas in the migration mode and complete the migration. Perform the necessary checks to verify if the data has been imported correctly.

Restart Atlas in non-migration mode.

If you have Atlas setup in HA mode, retain only one instance and remove the others.
Ensure that the ZIP files generated as output of Nav2Atlas conversion are placed at the same location where the Atlas node is present.

How many Navigator entities are transitioned?

When preparing to transition content from Navigator to Atlas, it helps in planning the transition to know how many Navigator entities will be extracted. Use Navigator's search facets to figure this out.

To determine the number of Navigator entities extracted for extraction and transformation phases of the transition:

Log into Navigator.
In the Cluster Group facet in the left panel, select the cluster you are migrating from.
The main panel displays the count of entities in that cluster. Use this value for estimating the extraction and transformation phase durations.

Not all Navigator entities are imported into Atlas. To estimate the subset of entities included in the import phase:

Log into Navigator.
In the Cluster Group facet in the left panel, select the cluster you are migrating from.
In the Source Type facet in the left panel, select "Hive", "Impala", and "Spark".
The main panel displays the count of entities in from these sources in this cluster.
Double the number from the search results to account for the physical files that correspond to the tables and jobs. The HDFS entities referenced by the Hive, Impala, and Spark entities are included in the transition.

The transition brings over all business metadata definitions and associations with transitioned entities. To determine the number of Navigator managed properties to transition:

Log into Navigator.
In the left Search panel, find the Tags facet.
This facet lists all the tags defined in Navigator. Navigator tags are imported into Atlas as labels.
Go to Administration > Managed Properties.
The Navigator namespaces are imported as Atlas business metadata collections. Each managed property is imported as a business metadata attribute.