List of custom configurations to be manually migrated to CDW
You must be aware of the list of some custom configurations to be copied manually from CDP Private Cloud Base to CDW PvC 1.3.1.
If you have customized any of these configurations in your CDH environment to optimize performance, to mitigate the pressure on the network, to avoid resource usage spikes, to handle out-of-memory conditions or similar optimizations, you must manually import t
hese custom configurations from CDP Private Cloud Base to each VW.
default_file_format
anddefault_transactional_type
must be set same as CDP Private Cloud BaseCONVERT_LEGACY_HIVE_PARQUET_UTC_TIMESTAMPS
andUSE_LOCAL_TZ_FOR_UNIX_TIMESTAMP_CONVERSIONS
must be the same as CDP Private Cloud Base especially for older datasets to avoid hive vs impala timestamp issues
List of configurations to be copied from CDP Private Cloud Base Impala if they were set at
service or at pool level.
- Runtime filters for performance
RUNTIME_BLOOM_FILTER_SIZE,RUNTIME_FILTER_MAX_SIZE,RUNTIME_FILTER_MIN_SIZE, RUNTIME_FILTER_MODE,RUNTIME_FILTER_WAIT_TIME_MS
- Set
PARQUET_FALLBACK_SCHEMA_RESOLUTION
to handle any column order changes - Set
TIMEZONE
to your appropriate timezone as CDW uses UTC - Set
COMPRESSION_CODEC
to what the customers needs, LZ4 is recommended - Set
SCHEDULE_RANDOM_REPLICA
andREPLICA_PREFERENCE
to avoid hotspotting - Set
EXEC_TIME_LIMIT_S
,IDLE_SESSION_TIMEOUT
andQUERY_TIMEOUT_S
- Enable
ALLOW_ERASURE_CODED_FILES
and disableDECIMAL_V2
only if needed - Set
DISABLE_CODEGEN_ROWS_THRESHOLD
,EXEC_SINGLE_NODE_ROWS_THRESHOLD
,BROADCAST_BYTES_LIMIT
SCRATCH_LIMIT
if set to higher on CDP Private Cloud Base needs to be limited to a value between 300 and 600 GB