Starting the replication job

Learn how to start a Kudu replication job and how to resume a job from a specific checkpoint.

Starting a new replication job

You can start the replication job by submitting the shaded JAR file to a Flink cluster by using the flink run-application command. You must pass all configuration parameters as --key value flags.

To start a basic replication job on YARN, run the following command:

flink run-application -t yarn-application \
  -Dclassloader.parent-first-patterns.additional=org.apache.kudu \
  -c org.apache.kudu.replication.ReplicationJob \
  kudu-replication-<version>.jar \
  --job.sourceMasterAddresses source-master1:7051,source-master2:7051 \
  --job.sinkMasterAddresses sink-master1:7051,sink-master2:7051 \
  --job.tableName my_table \
  --job.checkpointsDirectory hdfs:///kudu-replication/checkpoints/my_table

Resuming from a checkpoint

If a job fails or you perform a planned restart, you can resume from a specific Flink checkpoint or savepoint. Flink retains checkpoints on cancellation by default by using the RETAIN_ON_CANCELLATION setting.

To resume the job, run the following command and include the -s flag with the path to the checkpoint directory:

flink run-application -t yarn-application \
  -s hdfs:///kudu-replication/checkpoints/my_table/<checkpoint-id> \
  -Dclassloader.parent-first-patterns.additional=org.apache.kudu \
  -c org.apache.kudu.replication.ReplicationJob \
  kudu-replication-<version>.jar \
  --job.sourceMasterAddresses source-master1:7051,source-master2:7051 \
  --job.sinkMasterAddresses sink-master1:7051,sink-master2:7051 \
  --job.tableName my_table \
  --job.checkpointsDirectory hdfs:///kudu-replication/checkpoints/my_table