Batch migration of projects
The Cloudera AI utility tool is primarily designed to facilitate the migration of individual projects. However, a wrapper script is available to enable batch migration of multiple projects.
The batch migration script reads the list of project names from
export-config.ini or the import-config.ini files. Each section
defined here corresponds to a specific project, with the section name corresponding to the
project name. You can include project-specific configurations within each respective section,
while configurations shared across multiple projects can be placed inside the
default section.
BATCH_SIZE
The BATCH_SIZE variable provided inside the script controls the number of
projects that can be exported or imported simultaneously. To prevent system errors like
running out of memory, you must select an appropriate batch size. Each export or import
operation of a project generates a distinct session on the workspace, utilizing 1 CPU and 0.5
GB of memory. Therefore, the batch size must be determined considering the available resources
on both the source and target workspaces.
Consider the following additional aspects before using the batch migration:
-
Before initiating the batch migration, ensure that enough disk space is available on the host machine for downloading all or a batch of projects.
-
In case of failure during batch migration, the script can be rerun. However the execution of the batch can be made quicker by deleting all the project names already exported or imported from the configuration file.
-
Logs for each project are collected inside the individual project directory.
