Monitoring the performance of HDFS replication policies

You can monitor the progress of an HDFS replication policy using performance data that you download as a CSV file from the Cloudera Manager Admin console.

This file contains information about the files being replicated, the average throughput, and other details that can help diagnose performance issues during HDFS replications. You can view this performance data for running HDFS replication jobs and for completed jobs.

To view the performance data for a running HDFS replication policy, perform the following steps:

From Cloudera Manager, select Replication > Replication Policies.
Locate the row for the policy, select the policy, and click Actions. Select Show History.
Click Download CSV, and then choose one of the following options to view the performance report:
- Performance. Click to download a CSV file which contains a summary report about the performance of the replication job. The performance summary report includes the last performance sample for each mapper that is working on the replication job.
- Full Performance. Click to download the CSV file that contains the performance report of the job. The complete performance report includes all the samples taken for all mappers during the full execution of the replication job.
To view the data, open the file in a spreadsheet program such as Microsoft Excel.

The following table shows the columns that you can view in the CSV file:

Table 1. HDFS Performance Report Columns
Performance Data Columns	Description
Timestamp	Time when the performance data was collected.
Host	Name of the host where the YARN or MapReduce job was running.
Bytes Copied	Number of bytes copied for the file currently being copied.
Time Elapsed (ms)	Total time elapsed in milliseconds for the copy operation of the file currently being copied.
Files Copied	Number of files copied.
Avg Throughput (KB/s)	Average throughput since the start of the file currently being copied in kilobytes per second.
Last File (bytes)	File size of the last file in bytes.
Last File Time (ms)	Time taken to copy the last file in milliseconds.
Last file throughput (KB/s)	Throughput since the start of the last file being copied in kilobytes per second.

In addition to the performance reports, you can view the reports of files with errors, files that are deleted, and files that are skipped during the replication job. To view the reports, perform the following steps:

On the Replication Policies page, locate the policy and click Actions > Show History.
The Replication History page for the replication policy appears. Expand to view the replication job details.
Click Download CSV for the following options:
- Listing - Click to download the CSV file that contains the replication report. The file lists the list of files and directories copied during the replication job.
- Status - Click to download the CSV file that contains the complete status report. The file contains the full status report of the files where the status of the replication is one of the following:
  - ERROR – An error occurred and the file was not copied.
  - DELETED – A deleted file.
  - SKIPPED – A file where the replication was skipped because it was up-to-date.
- Error Status Only - Click to download the CSV file that contains the status report of all copied files with errors. The file lists the status, path, and message for the copied files with errors.
- Deleted Status Only - Click to download the CSV file that contains the status report of all deleted files. The file lists the status, path, and message for the databases and tables that were deleted.
- Skipped Status Only - Click to download the CSV file that contains the status report of all skipped files. The file lists the status, path, and message for the databases and tables that were skipped.
- Performance - Click to download a CSV file which contains a summary report about the performance of the running replication job. The performance summary report includes the last performance sample for each mapper that is working on the replication job.
- Full Performance - Click to download the CSV file that contains the performance report of the job. The performance report shows the samples taken for all the mappers during the full execution of the replication job.

To view the data, open the file in a spreadsheet program such as Microsoft Excel.

The performance data is collected every two minutes. Therefore, no data is available during the initial execution of a replication job because not enough samples are available to estimate throughput and other reported data.

A sample CSV file, as presented in Excel, is shown here:

Note the following limitations and known issues:

If you click the CSV download too soon after the replication job starts, Cloudera Manager returns an empty file or a CSV file that has columns headers only and a message to try later when performance data has actually been collected.
If you employ a proxy user with the form user@domain, performance data is not available through the links.
If the replication job only replicates small files that can be transferred in less than a few minutes, no performance statistics are collected.
For replication policies that specify the Dynamic Replication Strategy, statistics regarding the last file transferred by a MapReduce job hide previous transfers performed by that MapReduce job.
Only the last trace per MapReduce job is reported in the CSV file.