HDFS Replication

High latency among clusters can cause replication jobs to run more slowly, but does not cause them to fail. For best performance, latency between the source cluster NameNode and the destination cluster NameNode should be less than 80 milliseconds. (You can test latency using the Linux ping command.) Cloudera has successfully tested replications with latency of up to 360 milliseconds. As latency increases, replication performance degrades.

1. Verify that your cluster conforms to a supported replication scenario.
2. If you are using different Kerberos principals for the source and destination clusters, add the destination principal as a proxy user on the source cluster. For example, if you are using the hdfssrc principal on the source cluster and the hdfsdest principal on the destination cluster, add the following properties to the HDFS service Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml property on the source cluster:

   <property>
       <name>hadoop.proxyuser.hdfsdest.groups</name>
       <value>*</value>
   </property>
   <property>
       <name>hadoop.proxyuser.hdfsdest.hosts</name>
       <value>*</value>
   </property>

   Deploy the client configuration and restart all services on the source cluster.

3. If the source cluster is managed by a different Cloudera Manager server than the destination cluster, configure a peer relationship.
4. Do one of the following:
   1. Select Backup > Replications
   2. Click Schedule HDFS Replication.

   or

   1. Select Clusters > HDFS Service Name.
   2. Select Quick Links > Replication.
   3. Click Schedule HDFS Replication.

   The Create Replication dialog box displays.

5. Click the Source field and select the source HDFS service. You can select HDFS services managed by a peer Cloudera Manager Server, or local HDFS services (managed by the Cloudera Manager Server for the Admin Console you are logged into).
6. Enter the Path to the directory (or file) you want to replicate (the source).
7. Click the Destination field and select the destination HDFS service from the HDFS services managed by the Cloudera Manager Server for the Admin Console you are logged into.
8. Enter the Path where the source files should be saved.
9. Select a Schedule:
   • Immediate - Run the schedule Immediately.
   • Once - Run the schedule one time in the future. Set the date and time.
   • Recurring - Run the schedule periodically in the future. Set the date, time, and interval between runs.
10. Click the Add Exclusion link to exclude one or more paths from the replication.

    The Regular Expression-Based Path Exclusion field displays, where you can enter a regular expression-based path.

    Click to add additional regular expressions.

11. In the Advanced Options section, you can change the following parameters:
    • MapReduce Service - The MapReduce or YARN service to use.
    • Scheduler Pool - The name of a resource pool. The value you enter is used by the MapReduce Service you specified when Cloudera Manager executes the MapReduce job for the replication. The job specifies the value using one of these properties:
      • MapReduce - Fair scheduler: mapred.fairscheduler.pool
      • MapReduce - Capacity scheduler: queue.name
      • YARN - mapreduce.job.queuename
    • Run As - The user to run the job. By default this is hdfs. If you want to run the job as a different user, enter the user name here. If you are using Kerberos, you must provide a user name here, and it must be one with an ID greater than 1000. (You can also configure the minimum user ID number with the min.user.id property in the YARN or MapReduce service.) Verify that the user running the job has a home directory, /user/username, owned by username:supergroup in HDFS. This user must have permissions to read from the source directory and write to the destination directory.
      Note the following:

      • The User must not be present in the list of banned users specified with the Banned System Users property in the YARN configuration (Go to the YARN service, select Configuration tab and search for the property). For security purposes, the hdfs user is banned by default from running YARN containers.
      • The requirement for a user ID that is greater than 1000 can be overridden by adding the user to the "white list" of users that is specified with the Allowed System Users property. (Go to the YARN service, select Configuration tab and search for the property.)
    • Log path - An alternate path for the logs.
    • Maximum Map Slots and Maximum Bandwidth - Limits for the number of map slots and for bandwidth per mapper. The default for Maximum Bandwidth is 100 MB.
    • Error Handling You can select the following:
      • Abort on Error - Whether to abort the job on an error. If selected, files copied up to that point remain on the destination, but no additional files are copied. Abort on Error is off by default.
      • Skip Checksum Checks - Whether to skip checksum checks on the copied files. If checked, checksums are not validated. Checksums are checked by default.

      See Replication of Encrypted Data and HDFS Transparent Encryption.

    • Replication Strategy - Whether file replication tasks should be distributed among the mappers statically or dynamically. (The default is Dynamic.) Static replication distributes file replication tasks among the mappers up front to achieve a uniform distribution based on the file sizes. Dynamic replication distributes file replication tasks in small sets to the mappers, and as each mapper completes its tasks, it dynamically acquires and processes the next unallocated set of tasks. There are additional tuning options you can use to improve performance when using the Dynamic strategy. See HDFS Replication Tuning.
    • Delete Policy - Whether files that were deleted on the source should also be deleted from the destination directory. This policy also determines the handling of files in the destination location that are unrelated to the source. Options include:
      • Keep Deleted Files - Retains the destination files even when they no longer exist at the source. (This is the default.).
      • Delete to Trash - If the HDFS trash is enabled, files are moved to the trash folder.
      • Delete Permanently - Uses the least amount of space; use with caution.
    • Preserve - Whether to preserve the block size, replication count, permissions (including ACLs), and extended attributes (XAttrs) as they exist on the source file system, or to use the settings as configured on the destination file system. By default source system settings are preserved. When Permission is checked, and both the source and destination clusters support ACLs, replication preserves ACLs. Otherwise, ACLs are not replicated. When Extended attributes is checked, and both the source and destination clusters support extended attributes, replication preserves them.
    • Alerts - Whether to generate alerts for various state changes in the replication workflow. You can alert on failure, on start, on success, or when the replication workflow is aborted.
12. Click Save Schedule.

    The replication task now appears as a row in the Replications Schedule table. (It can take up to 15 seconds for the task to appear.)

To specify additional replication tasks, select Create > HDFS Replication.

If your cluster has clients installed on hosts with limited resources, HDFS replication may use these hosts to run commands for the replication, which can cause performance degradation. You can limit HDFS replication to run only on selected DataNodes by specifying a "whitelist" of DataNode hosts.

The Replications Schedules page displays a row of information about each scheduled replication job. Each row also displays recent messages regarding the last time the Replication job ran.

Replication Schedules Table

Only one job corresponding to a replication schedule can occur at a time; if another job associated with that same replication schedule starts before the previous one has finished, the second one is canceled.

You can limit the replication jobs that are displayed by selecting filters on the left. If you do not see an expected schedule, adjust or clear the filters.

• While a job is in progress, the Last Run column displays a spinner and progress bar, and each stage of the replication task is indicated in the message beneath the job's row. Click the Command Details link to view details about the execution of the command.
• If the job is successful, the number of files copied is indicated. If there have been no changes to a file at the source since the previous job, then that file is not copied. As a result, after the initial job, only a subset of the files may actually be copied, and this is indicated in the success message.
• If the job fails, the icon displays.
• To view more information about a completed job, select Actions > Show History. See Viewing Replication History.

You can view historical details about replication jobs on the Replication History page.

To view the history of a replication job:

1. Select Backup > Replication Schedules to go to the Replication Schedules page.
2. Locate the row for the job.
3. Click Actions > Show History.

Replication History Screen (HDFS)

Replication History Screen (Hive, Failed Replication)

The Replication History page displays a table of previously run replication jobs with the following columns:

Replication History Table

Column	Description
Start Time	Time when the replication job started. Click to expand the display and show details of the replication. In this screen, you can: Click the View link to open the Command Details page, which displays details and messages about each step in the execution of the command. Click to expand the display for a Step to: View the actual command string. View the Start time and duration of the command. Click the Context link to view the service status page relevant to the command. Select one of the tabs to view the Role Log, stdout, and stderr for the command. See Viewing Running and Recent Commands. Click Collect Diagnostic Data to open the Send Diagnostic Data screen, which allows you to collect replication-specific diagnostic data for this run of the schedule: Select Send Diagnostic Data to Cloudera to automatically send the bundle to Cloudera Support. You can also enter a ticket number and comments when sending the bundle. Click Collect and Send Diagnostic Data to generate the bundle and open the Replications Diagnostics Command screen. When the command finishes, click Download Result Data to download a zip file containing the bundle. (HDFS only) Link to view details on the MapReduce Job used for the replication. See Viewing and Filtering MapReduce Activities. (Dry Run only) View the number of Replicable Files. Displays the number of files that would be replicated during an actual replication. (Dry Run only) View the number of Replicable Bytes. Displays the number of bytes that would be replicated during an actual replication. Link to download a CSV file containing a Replication Report. This file lists the databases and tables that were replicated. View the number of Errors that occurred during the replication. View the number of Impala UDFs replicated. (Displays only for Hive replications where Replicate Impala Metadata is selected.) Click the link to download a CSV file containing a Download Listing. This file lists the files and directories that were replicated. Click the link to download a CSV file containing Download Status. If a user was specified in the Run as field when creating the replication job, the selected user displays. View messages returned from the replication job.
Duration	Amount of time the replication job took to complete.
Outcome	Indicates success or failure of the replication job.
Files Expected	Number of files expected to be copied, based on the parameters of the replication schedule.
Files Copied	Number of files actually copied during the replication.
Tables	(Hive only) Number of tables replicated.
Files Failed	Number of files that failed to be copied during the replication.
Files Deleted	Number of files that were deleted during the replication.
Files Skipped	Number of files skipped during the replication. The replication process skips files that already exist in the destination and have not changed.