Parameters in Hive replication policy definition JSON file

The Hive replication policy definition JSON file contains all the parameters required to create an Hive replication policy.

The following table lists the parameters in the policy definition JSON file that are required for an Hive replication policy:

Parameter Description
clusterCrn Enter the destination cluster CRN for Hive replication policy.
policyName Enter a unique name for the replication policy.
name Enter the unique name for the policy.
type Enter HIVE to create a Hive replication policy.
replicationStrategy Enter STATIC or DYNAMIC to determine whether the file replication tasks should be distributed among the mappers statically or dynamically. The default is DYNAMIC.

Static replication distributes file replication tasks among the mappers up front to achieve an uniform distribution based on the file sizes.

Dynamic replication distributes the file replication tasks in small sets to the mappers, and as each mapper completes its tasks, it dynamically acquires and processes the next unallocated set of tasks.

skipChecksumChecks Enter true to perform checksum checks. The default is true.
Checksums are used to perform the following tasks:
  • To skip replication of files that have already been copied. When set to true, the replication job skips copying a file if the file lengths and modification times are identical between the source and destination clusters. Otherwise, the job copies the file from the source to the destination.
  • To redundantly verify the integrity of data. However, checksums are not required to guarantee accurate transfers between clusters. HDFS data transfers are protected by checksums during transfer and storage hardware also uses checksums to ensure that data is accurately stored. These two mechanisms work together to validate the integrity of the copied data.
skipListingChecksumChecks Enter true to skip checksum check while comparing two files to determine whether they are the same or not. Otherwise, the file size and last modified time are used to determine if files are the same or not. Skipping the check improves performance during the mapper phase.
abortOnError Enter true to stop the policy job when an error occurs. This ensures that the files copied up to that point remain on the destination, but no additional files are copied. The default is false.
abortOnSnapshtDiffFailures Enter true to stop the replication job if a snapshot diff fails during replication.
preserve Enter true to preserve the block size, replication count, permissions (including ACLs), and extended attributes (XAttrs) as they exist on the source file system.
  • blockSize
  • replicationCount
  • permissions
  • extendedAtrributes

Enter false to use the settings as configured on the destination file system. By default, the source system settings are preserved.

deletePolicy Enter one of the following options:
  • KEEP_DELETED_FILES - Retains the destination files even when they no longer exist at the source. This is the default option.
  • DELETE_TO_TRASH - Moves files to the trash folder if the HDFS trash is enabled. (Not supported when replicating to S3 or ADLS.)
  • DELETE_PERMANENTLY - Uses the least amount of space; use with caution.
alert Configure the following parameters as required:
  • onFailure - Enter true to generate alerts when the replication job fails.
  • onStart - Enter true to generate alerts when the replication job starts.
  • onSuccess - Enter true to generate alerts when the replication job completes successfully.
  • onAbort - Enter true to generate alerts when the replication job is aborted.
  • database - Enter one or more database names to exclude from replication.
  • tablesIncludeRegex - Enter one or more regular expression-based paths to tables to include in replication.
  • tablesExcludeRegex - Enter one or more regular expression-based paths to tables to exclude in replication.
sentryPermissions Enter INCLUDE to import both Hive object and URL permissions.
skipUrlPermissions Enter true to import only the Hive object permissions.
numThreads Enter the number of threads to use during replication.
frequencyInSec Auto-populated after the policy runs successfully. Shows the time taken in seconds to complete the replication job.
targetDataset Auto-populated after the policy runs successfully. Shows the target location where the replicated files are available on the target cluster.
cloudCredentials Enter the cloud credentials.
sourceCluster Shows the source cluster name.
targetCluster Shows the target cluster name in the dataCenterName$cluster name format. For example, "DC-US$My Destination 17".
startTime Shows the start time of the replication job.
endTime Shows the end time of the replication job.
distcpMaxMaps Enter the maximum map slots to limit the number of map slots per mapper. The default value is 20.
distcpMapBandwidth Enter the maximum bandwidth to limit the bandwidth per mapper. The default is 100 MB.
queueName Enter the YARN queue name if not set to Default queue name. By default, the Default queue name is used.
tdeSameKey Enter true if the source and destination are encrypted with the same TDE key.
description Enter a description for the policy.
enableSnapshotBasedReplication Enter true to enable snapshot-based replication.
cloudEncryptionAlgorithm Enter the cloud encryption algorithm.
cloudEncryptionKey Enter the cloud encryption key.
plugins Enter the plugins to deploy on all the nodes in the cluster if you have multiple repositories configured in your environment.
hiveExternalTableBaseDirectory Enter the Hive external table base directory path.
cmPolicySubmitUser Enter the following options:
  • userName - Enter the user name that you are using to run the policy.
  • sourceUser - Enter the source cluster username, if any.