Troubleshooting Ozone replication policies
The troubleshooting scenarios in this topic help you to troubleshoot the Ozone replication policies in Replication Manager.
How to skip replicated files during subsequent Ozone replication policy runs?
The first Ozone replication policy job is always a bootstrap job where all the specified data is replicated from the source cluster to the target cluster. Some subsequent runs might also be bootstrap jobs if the incremental replication fails when the job runs fall back to bootstrap replication. In this scenario, the bootstrap replication jobs might replicate the files that were already replicated if the modification time is different for a file on the source and the target cluster. This scenario is unavoidable when the target filesystem does not support setting the modification time, and hence results in a performance issue.
- Ignore the modification time to skip replicated files.In this method, after you add the following advanced configuration snippet, the subsequent replication jobs for the specified Ozone replication policies ensure that the already-replicated files are skipped by the jobs. The advanced configuration snippet considers the relative file path, file name, and file size, and ignores the modification time to determine the replicated files to skip in subsequent job runs.
- Go to the tab.
- Search for Ozone Replication Advanced Configuration Snippet (Safety Valve) for core-site.xml.
- Add com.cloudera.enterprise.distcp.ozone-schedules-with-unsafe-equality-check = [***ENTER COMMA-SEPARATED LIST OF OZONE REPLICATION POLICIES’ ID or ENTER all TO APPLY TO ALL OZONE REPLICATION POLICIES***] key-value pair.
- Save the changes.
- Consider the modification time to skip replicated files.
When both of the advanced configuration snippets in the following task are configured, subsequent Ozone replication policy runs skip replicating a file when the relative file path, file name, and file size are equal, and when the source file’s modification time is less than or equal to the target file’s modification time. This reduces the risk of data loss, because if a file is modified on the source after a bootstrap replication job, the source file’s modification time would be higher than the target file’s modification time, therefore the file is not skipped during the subsequent replication job run.
- Go to the tab.
- Search for Ozone Replication Advanced Configuration Snippet (Safety Valve) for core-site.xml.
- Add com.cloudera.enterprise.distcp.ozone-schedules-with-unsafe-equality-check = [***ENTER COMMA-SEPARATED LIST OF OZONE REPLICATION POLICIES’ ID or ENTER all TO APPLY TO ALL OZONE REPLICATION POLICIES***] key-value pair.
- Add com.cloudera.enterprise.distcp.require-source-before-target-modtime-in-unsafe-equality-check = [***ENTER true OR false***]
- Save the changes.
