Cloudera Docs

Hive Configuration Property Changes

You need to know the property value changes made by the upgrade process as the change might impact your work. You might need to consider reconfiguring property value defaults that the upgrade changes.

Hive Configuration Property Values

The upgrade process changes the default values of some Hive configuration properties and adds new properties. The following list describes those changes that occur after upgrading from CDH or HDP to Cloudera.

datanucleus.connectionPool.maxPoolSize
Before upgrade: 30

After upgrade: 10

datanucleus.connectionPoolingType
Before upgrade: BONECP

After upgrade: HikariCP

hive.auto.convert.join.noconditionaltask.size
Before upgrade: 20971520

After upgrade: 52428800

Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.

hive.auto.convert.sortmerge.join
Before upgrade: FALSE in the old CDH; TRUE in the old HDP.

After upgrade: TRUE

hive.auto.convert.sortmerge.join.to.mapjoin
Before upgrade: FALSE

After upgrade: TRUE

hive.cbo.enable
Before upgrade: FALSE

After upgrade: TRUE

hive.cbo.show.warnings
Before upgrade: FALSE

After upgrade: TRUE

hive.compactor.worker.threads
Before upgrade: 0

After upgrade: 5

hive.compute.query.using.stats
Before upgrade: FALSE

After upgrade: TRUE

hive.conf.hidden.list
Before upgrade: 
javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,hive.metastore.dbaccess.ssl.truststore.password,fs.s3.awsAccessKeyId,fs.s3.awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.access.key,fs.s3a.secret.key,fs.s3a.proxy.password,dfs.adls.oauth2.credential,fs.adl.oauth2.credential,fs.azure.account.oauth2.client.secret
After upgrade: 
javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,hive.druid.metadata.password,hive.driver.parallel.compilation.global.limit
hive.conf.restricted.list
Before upgrade: 
hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.spark.client.connect.timeout,hive.spark.client.server.connect.timeout,hive.spark.client.channel.log.level,hive.spark.client.rpc.max.size,hive.spark.client.rpc.threads,hive.spark.client.secret.bits,hive.spark.client.rpc.server.address,hive.spark.client.rpc.server.port,hive.spark.client.rpc.sasl.mechanisms,hadoop.bin.path,yarn.bin.path,spark.home,bonecp.,hikaricp.,hive.driver.parallel.compilation.global.limit,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space
After upgrade: 
hive.security.authenticator.manager,hive.security.authorization.manager,hive.security.metastore.authorization.manager,hive.security.metastore.authenticator.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.security.authorization.enabled,hive.distcp.privileged.doAs,hive.server2.authentication.ldap.baseDN,hive.server2.authentication.ldap.url,hive.server2.authentication.ldap.Domain,hive.server2.authentication.ldap.groupDNPattern,hive.server2.authentication.ldap.groupFilter,hive.server2.authentication.ldap.userDNPattern,hive.server2.authentication.ldap.userFilter,hive.server2.authentication.ldap.groupMembershipKey,hive.server2.authentication.ldap.userMembershipKey,hive.server2.authentication.ldap.groupClassKey,hive.server2.authentication.ldap.customLDAPQuery,hive.privilege.synchronizer.interval,hive.spark.client.connect.timeout,hive.spark.client.server.connect.timeout,hive.spark.client.channel.log.level,hive.spark.client.rpc.max.size,hive.spark.client.rpc.threads,hive.spark.client.secret.bits,hive.spark.client.rpc.server.address,hive.spark.client.rpc.server.port,hive.spark.client.rpc.sasl.mechanisms,bonecp.,hive.druid.broker.address.default,hive.druid.coordinator.address.default,hikaricp.,hadoop.bin.path,yarn.bin.path,spark.home,hive.driver.parallel.compilation.global.limit,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space
hive.default.fileformat.managed
Before upgrade: None

After upgrade: ORC

hive.default.rcfile.serde
Before upgrade: org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe

After upgrade: org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe

Not supported in Impala. Impala cannot read Hive-created RC tables.

hive.driver.parallel.compilation
Before upgrade: FALSE

After upgrade: TRUE

hive.exec.dynamic.partition.mode
Before upgrade: strict

After upgrade: nonstrict

In Cloudera Base on premises, accidental use of dynamic partitioning feature is not prevented by default.

hive.exec.max.dynamic.partitions
Before upgrade: 1000

After upgrade: 5000

In Cloudera Base on premises, fewer restrictions on dynamic paritioning occur than in the pre-upgrade CDH or HDP cluster.
hive.exec.max.dynamic.partitions.pernode
Before upgrade: 100

After upgrade: 2000

In Cloudera Base on premises, fewer restrictions on dynamic paritioning occur than in the pre-upgrade CDH or HDP cluster.
hive.exec.post.hooks
Before upgrade: 
com.cloudera.navigator.audit.hive.HiveExecHookContext,org.apache.hadoop.hive.ql.hooks.LineageLogger

After upgrade: org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook

A prime number is recommended.
hive.exec.reducers.max
Before upgrade: 1099

After upgrade: 1009

Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default
hive.execution.engine
Before upgrade: mr

After upgrade: tez

Tez is now the only supported execution engine, existing queries that change execution mode to Spark or MapReduce within a session, for example, fail.
hive.fetch.task.conversion
Before upgrade: minimal

After upgrade: more

hive.fetch.task.conversion.threshold
Before upgrade: 256MB

After upgrade: 1GB

Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.

hive.hashtable.key.count.adjustment
Before upgrade: 1

After upgrade: 0.99

Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.
hive.limit.optimize.enable
Before upgrade: FALSE

After upgrade: TRUE

hive.limit.pushdown.memory.usage
Before upgrade: 0.1

After upgrade: 0.04

Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.
hive.mapjoin.hybridgrace.hashtable
Before upgrade: TRUE

After upgrade: FALSE

hive.mapred.reduce.tasks.speculative.execution
Before upgrade: TRUE

After upgrade: FALSE

hive.metastore.aggregate.stats.cache.enabled
Before upgrade: TRUE

After upgrade: FALSE

hive.metastore.disallow.incompatible.col.type.changes
Before upgrade: FALSE
After upgrade: TRUE
Schema evolution is more restrictive in Cloudera Base on premises than in CDH to avoid data corruption. The new default disallows column type changes if the old and new types are incompatible.
hive.metastore.dml.events
Before upgrade: FALSE

After upgrade: TRUE

hive.metastore.event.message.factory
Before upgrade: org.apache.hadoop.hive.metastore.messaging.json.ExtendedJSONMessageFactory

After upgrade: org.apache.hadoop.hive.metastore.messaging.json.gzip.GzipJSONMessageEncoder

hive.metastore.uri.selection
Before upgrade: SEQUENTIAL

After upgrade: RANDOM

hive.metastore.warehouse.dir
Before upgrade from CDH: /user/hive/warehouse
Before upgrade from HDP: /apps/hive/warehouse
After upgrade from CDH: /warehouse/tablespace/managed/hive
After upgrade from HDP: /warehouse/tablespace/managed/hive
For information about the location of old tables and new tables, which you create after the upgrade, see Changes to CDH Hive Tables or Changes to HDP Hive tables.
hive.optimize.metadataonly
Before upgrade: FALSE

After upgrade: TRUE

hive.optimize.point.lookup.min
Before upgrade: 31

After upgrade: 2

hive.prewarm.numcontainers
Before upgrade: 10

After upgrade: 3

hive.script.operator.env.blacklist
Before upgrade: hive.txn.valid.txns,hive.script.operator.env.blacklist
After upgrade: hive.txn.valid.txns,hive.txn.tables.valid.writeids,hive.txn.valid.writeids,hive.script.operator.env.blacklist
hive.security.authorization.sqlstd.confwhitelist
Before upgrade: 
hive\.auto\..*hive\.cbo\..*hive\.convert\..*hive\.exec\.dynamic\.partition.*hive\.exec\..*\.dynamic\.partitions\..*hive\.exec\.compress\..*hive\.exec\.infer\..*hive\.exec\.mode.local\..*hive\.exec\.orc\..*hive\.exec\.parallel.*hive\.explain\..*hive\.fetch.task\..*hive\.groupby\..*hive\.hbase\..*hive\.index\..*hive\.index\..*hive\.intermediate\..*hive\.join\..*hive\.limit\..*hive\.log\..*hive\.mapjoin\..*hive\.merge\..*hive\.optimize\..*hive\.orc\..*hive\.outerjoin\..*hive\.parquet\..*hive\.ppd\..*hive\.prewarm\..*hive\.server2\.proxy\.userhive\.skewjoin\..*hive\.smbjoin\..*hive\.stats\..*hive\.strict\..*hive\.tez\..*hive\.vectorized\..*mapred\.map\..*mapred\.reduce\..*mapred\.output\.compression\.codecmapred\.job\.queuenamemapred\.output\.compression\.typemapred\.min\.split\.sizemapreduce\.job\.reduce\.slowstart\.completedmapsmapreduce\.job\.queuenamemapreduce\.job\.tagsmapreduce\.input\.fileinputformat\.split\.minsizemapreduce\.map\..*mapreduce\.reduce\..*mapreduce\.output\.fileoutputformat\.compress\.codecmapreduce\.output\.fileoutputformat\.compress\.typeoozie\..*tez\.am\..*tez\.task\..*tez\.runtime\..*tez\.queue\.namehive\.transpose\.aggr\.joinhive\.exec\.reducers\.bytes\.per\.reducerhive\.client\.stats\.countershive\.exec\.default\.partition\.namehive\.exec\.drop\.ignorenonexistenthive\.counters\.group\.namehive\.default\.fileformat\.managedhive\.enforce\.bucketmapjoinhive\.enforce\.sortmergebucketmapjoinhive\.cache\.expr\.evaluationhive\.query\.result\.fileformathive\.hashtable\.loadfactorhive\.hashtable\.initialCapacityhive\.ignore\.mapjoin\.hinthive\.limit\.row\.max\.sizehive\.mapred\.modehive\.map\.aggrhive\.compute\.query\.using\.statshive\.exec\.rowoffsethive\.variable\.substitutehive\.variable\.substitute\.depthhive\.autogen\.columnalias\.prefix\.includefuncnamehive\.autogen\.columnalias\.prefix\.labelhive\.exec\.check\.crossproductshive\.cli\.tez\.session\.asynchive\.compathive\.exec\.concatenate\.check\.indexhive\.display\.partition\.cols\.separatelyhive\.error\.on\.empty\.partitionhive\.execution\.enginehive\.exec\.copyfile\.maxsizehive\.exim\.uri\.scheme\.whitelisthive\.file\.max\.footerhive\.insert\.into\.multilevel\.dirshive\.localize\.resource\.num\.wait\.attemptshive\.multi\.insert\.move\.tasks\.share\.dependencieshive\.support\.quoted\.identifiershive\.resultset\.use\.unique\.column\.nameshive\.analyze\.stmt\.collect\.partlevel\.statshive\.exec\.schema\.evolutionhive\.server2\.logging\.operation\.levelhive\.server2\.thrift\.resultset\.serialize\.in\.taskshive\.support\.special\.characters\.tablenamehive\.exec\.job\.debug\.capture\.stacktraceshive\.exec\.job\.debug\.timeouthive\.llap\.io\.enabledhive\.llap\.io\.use\.fileid\.pathhive\.llap\.daemon\.service\.hostshive\.llap\.execution\.modehive\.llap\.auto\.allow\.uberhive\.llap\.auto\.enforce\.treehive\.llap\.auto\.enforce\.vectorizedhive\.llap\.auto\.enforce\.statshive\.llap\.auto\.max\.input\.sizehive\.llap\.auto\.max\.output\.sizehive\.llap\.skip\.compile\.udf\.checkhive\.llap\.client\.consistent\.splitshive\.llap\.enable\.grace\.join\.in\.llaphive\.llap\.allow\.permanent\.fnshive\.exec\.max\.created\.fileshive\.exec\.reducers\.maxhive\.reorder\.nway\.joinshive\.output\.file\.extensionhive\.exec\.show\.job\.failure\.debug\.infohive\.exec\.tasklog\.debug\.timeouthive\.query\.id
After upgrade: 
hive\.auto\..*hive\.cbo\..*hive\.convert\..*hive\.druid\..*hive\.exec\.dynamic\.partition.*hive\.exec\.max\.dynamic\.partitions.*hive\.exec\.compress\..*hive\.exec\.infer\..*hive\.exec\.mode.local\..*hive\.exec\.orc\..*hive\.exec\.parallel.*hive\.exec\.query\.redactor\..*hive\.explain\..*hive\.fetch.task\..*hive\.groupby\..*hive\.hbase\..*hive\.index\..*hive\.index\..*hive\.intermediate\..*hive\.jdbc\..*hive\.join\..*hive\.limit\..*hive\.log\..*hive\.mapjoin\..*hive\.merge\..*hive\.optimize\..*hive\.materializedview\..*hive\.orc\..*hive\.outerjoin\..*hive\.parquet\..*hive\.ppd\..*hive\.prewarm\..*hive\.query\.redaction\..*hive\.server2\.thrift\.resultset\.default\.fetch\.sizehive\.server2\.proxy\.userhive\.skewjoin\..*hive\.smbjoin\..*hive\.stats\..*hive\.strict\..*hive\.tez\..*hive\.vectorized\..*hive\.query\.reexecution\..*reexec\.overlay\..*fs\.defaultFSssl\.client\.truststore\.locationdistcp\.atomicdistcp\.ignore\.failuresdistcp\.preserve\.statusdistcp\.preserve\.rawxattrsdistcp\.sync\.foldersdistcp\.delete\.missing\.sourcedistcp\.keystore\.resourcedistcp\.liststatus\.threadsdistcp\.max\.mapsdistcp\.copy\.strategydistcp\.skip\.crcdistcp\.copy\.overwritedistcp\.copy\.appenddistcp\.map\.bandwidth\.mbdistcp\.dynamic\..*distcp\.meta\.folderdistcp\.copy\.listing\.classdistcp\.filters\.classdistcp\.options\.skipcrccheckdistcp\.options\.mdistcp\.options\.numListstatusThreadsdistcp\.options\.mapredSslConfdistcp\.options\.bandwidthdistcp\.options\.overwritedistcp\.options\.strategydistcp\.options\.idistcp\.options\.p.*distcp\.options\.updatedistcp\.options\.deletemapred\.map\..*mapred\.reduce\..*mapred\.output\.compression\.codecmapred\.job\.queue\.namemapred\.output\.compression\.typemapred\.min\.split\.sizemapreduce\.job\.reduce\.slowstart\.completedmapsmapreduce\.job\.queuenamemapreduce\.job\.tagsmapreduce\.input\.fileinputformat\.split\.minsizemapreduce\.map\..*mapreduce\.reduce\..*mapreduce\.output\.fileoutputformat\.compress\.codecmapreduce\.output\.fileoutputformat\.compress\.typeoozie\..*tez\.am\..*tez\.task\..*tez\.runtime\..*tez\.queue\.namehive\.transpose\.aggr\.joinhive\.exec\.reducers\.bytes\.per\.reducerhive\.client\.stats\.countershive\.exec\.default\.partition\.namehive\.exec\.drop\.ignorenonexistenthive\.counters\.group\.namehive\.default\.fileformat\.managedhive\.enforce\.bucketmapjoinhive\.enforce\.sortmergebucketmapjoinhive\.cache\.expr\.evaluationhive\.query\.result\.fileformathive\.hashtable\.loadfactorhive\.hashtable\.initialCapacityhive\.ignore\.mapjoin\.hinthive\.limit\.row\.max\.sizehive\.mapred\.modehive\.map\.aggrhive\.compute\.query\.using\.statshive\.exec\.rowoffsethive\.variable\.substitutehive\.variable\.substitute\.depthhive\.autogen\.columnalias\.prefix\.includefuncnamehive\.autogen\.columnalias\.prefix\.labelhive\.exec\.check\.crossproductshive\.cli\.tez\.session\.asynchive\.compathive\.display\.partition\.cols\.separatelyhive\.error\.on\.empty\.partitionhive\.execution\.enginehive\.exec\.copyfile\.maxsizehive\.exim\.uri\.scheme\.whitelisthive\.file\.max\.footerhive\.insert\.into\.multilevel\.dirshive\.localize\.resource\.num\.wait\.attemptshive\.multi\.insert\.move\.tasks\.share\.dependencieshive\.query\.results\.cache\.enabledhive\.query\.results\.cache\.wait\.for\.pending\.resultshive\.support\.quoted\.identifiershive\.resultset\.use\.unique\.column\.nameshive\.analyze\.stmt\.collect\.partlevel\.statshive\.exec\.schema\.evolutionhive\.server2\.logging\.operation\.levelhive\.server2\.thrift\.resultset\.serialize\.in\.taskshive\.support\.special\.characters\.tablenamehive\.exec\.job\.debug\.capture\.stacktraceshive\.exec\.job\.debug\.timeouthive\.llap\.io\.enabledhive\.llap\.io\.use\.fileid\.pathhive\.llap\.daemon\.service\.hostshive\.llap\.execution\.modehive\.llap\.auto\.allow\.uberhive\.llap\.auto\.enforce\.treehive\.llap\.auto\.enforce\.vectorizedhive\.llap\.auto\.enforce\.statshive\.llap\.auto\.max\.input\.sizehive\.llap\.auto\.max\.output\.sizehive\.llap\.skip\.compile\.udf\.checkhive\.llap\.client\.consistent\.splitshive\.llap\.enable\.grace\.join\.in\.llaphive\.llap\.allow\.permanent\.fnshive\.exec\.max\.created\.fileshive\.exec\.reducers\.maxhive\.reorder\.nway\.joinshive\.output\.file\.extensionhive\.exec\.show\.job\.failure\.debug\.infohive\.exec\.tasklog\.debug\.timeouthive\.query\.idhive\.query\.tag
hive.security.command.whitelist
Before upgrade: set,reset,dfs,add,list,delete,reload,compile

After upgrade: set,reset,dfs,add,list,delete,reload,compile,llap

hive.server2.enable.doAs
Before upgrade: TRUE (in case of an insecure cluster only)

After upgrade: FALSE (in all cases)

Affects only insecure clusters by turning off impersonation. Permission issues are expected to arise for affected clusters.

hive.server2.idle.session.timeout
Before upgrade: 12 hours

After upgrade: 24 hours

Exception:Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.
hive.server2.max.start.attempts
Before upgrade: 30

After upgrade: 5

hive.server2.parallel.ops.in.session
Before upgrade: TRUE

After upgrade: FALSE

A Tez limitation requires disabling this property; otherwise, queries submitted concurrently on a single JDBC connection fail or execute slower.

hive.server2.support.dynamic.service.discovery
Before upgrade: FALSE

After upgrade: TRUE

hive.server2.tez.initialize.default.sessions
Before upgrade: FALSE

After upgrade: TRUE

hive.server2.thrift.max.worker.threads
Before upgrade: 100

After upgrade: 500

Exception: Preserves pre-upgrade value if the old default is overridden; otherwise, uses new default.

hive.server2.thrift.resultset.max.fetch.size
Before upgrade: 1000

After upgrade: 10000

hive.service.metrics.file.location
Before upgrade: /var/log/hive/metrics-hiveserver2/metrics.log

After upgrade: /var/log/hive/metrics-hiveserver2-hiveontez/metrics.log

This location change is due to a service name change.

hive.stats.column.autogather
Before upgrade: FALSE

After upgrade: TRUE

hive.stats.deserialization.factor
Before upgrade: 1

After upgrade: 10

hive.support.special.characters.tablename
Before upgrade: FALSE

After upgrade: TRUE

hive.tez.auto.reducer.parallelism
Before upgrade: FALSE

After upgrade: TRUE

hive.tez.bucket.pruning
Before upgrade: FALSE

After upgrade: TRUE

hive.tez.container.size
Before upgrade: -1

After upgrade: 4096

hive.tez.exec.print.summary
Before upgrade: FALSE

After upgrade: TRUE

hive.txn.manager
Before upgrade: org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager

After upgrade: org.apache.hadoop.hive.ql.lockmgr.DbTxnManager

hive.vectorized.execution.mapjoin.minmax.enabled
Before upgrade: FALSE

After upgrade: TRUE

hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled
Before upgrade: FALSE

After upgrade: TRUE

hive.vectorized.use.row.serde.deserialize
Before upgrade: FALSE

After upgrade: TRUE