List of custom configurations to be manually migrated to CDW

You must be aware of the list of some custom configurations to be copied manually from CDP Private Cloud Base to CDW PvC 1.3.1.

If you have customized any of these configurations in your CDH environment to optimize performance, to mitigate the pressure on the network, to avoid resource usage spikes, to handle out-of-memory conditions or similar optimizations, you must manually import t

hese custom configurations from CDP Private Cloud Base to each VW.
  • default_file_format and default_transactional_type must be set same as CDP Private Cloud Base
  • CONVERT_LEGACY_HIVE_PARQUET_UTC_TIMESTAMPS and USE_LOCAL_TZ_FOR_UNIX_TIMESTAMP_CONVERSIONS must be the same as CDP Private Cloud Base especially for older datasets to avoid hive vs impala timestamp issues
List of configurations to be copied from CDP Private Cloud Base Impala if they were set at service or at pool level.
  • Runtime filters for performance
    RUNTIME_BLOOM_FILTER_SIZE,RUNTIME_FILTER_MAX_SIZE,RUNTIME_FILTER_MIN_SIZE,
            RUNTIME_FILTER_MODE,RUNTIME_FILTER_WAIT_TIME_MS
  • Set PARQUET_FALLBACK_SCHEMA_RESOLUTION to handle any column order changes
  • Set TIMEZONE to your appropriate timezone as CDW uses UTC
  • Set COMPRESSION_CODEC to what the customers needs, LZ4 is recommended
  • Set SCHEDULE_RANDOM_REPLICA and REPLICA_PREFERENCE to avoid hotspotting
  • Set EXEC_TIME_LIMIT_S, IDLE_SESSION_TIMEOUT and QUERY_TIMEOUT_S
  • Enable ALLOW_ERASURE_CODED_FILES and disable DECIMAL_V2 only if needed
  • Set DISABLE_CODEGEN_ROWS_THRESHOLD,EXEC_SINGLE_NODE_ROWS_THRESHOLD,BROADCAST_BYTES_LIMIT
  • SCRATCH_LIMIT if set to higher on CDP Private Cloud Base needs to be limited to a value between 300 and 600 GB