Hive Configuration Property Changes
You need to know the property value changes made by the upgrade process as the change might impact your work. You might need to consider reconfiguring property value defaults that the upgrade changes.
Hive Configuration Property Values
The upgrade process changes the default values of some Hive configuration properties and adds new properties. The following list describes those changes that occur after upgrading from CDH or HDP to CDP.
- datanucleus.connectionPool.maxPoolSize
- Before upgrade:
30
After upgrade:
10
- datanucleus.connectionPoolingType
- Before upgrade:
BONECP
After upgrade:
HikariCP
- hive.auto.convert.join.noconditionaltask.size
- Before upgrade:
20971520
After upgrade:
52428800
Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.
- hive.auto.convert.sortmerge.join
- Before upgrade:
FALSE
in the old CDH;TRUE
in the old HDP.After upgrade:
TRUE
- hive.auto.convert.sortmerge.join.to.mapjoin
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.cbo.enable
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.cbo.show.warnings
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.compactor.worker.threads
- Before upgrade:
0
After upgrade:
5
- hive.compute.query.using.stats
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.conf.hidden.list
- Before upgrade:
javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,hive.metastore.dbaccess.ssl.truststore.password,fs.s3.awsAccessKeyId,fs.s3.awsSecretAccessKey,fs.s3n.awsAccessKeyId,fs.s3n.awsSecretAccessKey,fs.s3a.access.key,fs.s3a.secret.key,fs.s3a.proxy.password,dfs.adls.oauth2.credential,fs.adl.oauth2.credential,fs.azure.account.oauth2.client.secret
After upgrade:javax.jdo.option.ConnectionPassword,hive.server2.keystore.password,hive.druid.metadata.password,hive.driver.parallel.compilation.global.limit
- hive.conf.restricted.list
- Before upgrade:
hive.security.authenticator.manager,hive.security.authorization.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.spark.client.connect.timeout,hive.spark.client.server.connect.timeout,hive.spark.client.channel.log.level,hive.spark.client.rpc.max.size,hive.spark.client.rpc.threads,hive.spark.client.secret.bits,hive.spark.client.rpc.server.address,hive.spark.client.rpc.server.port,hive.spark.client.rpc.sasl.mechanisms,hadoop.bin.path,yarn.bin.path,spark.home,bonecp.,hikaricp.,hive.driver.parallel.compilation.global.limit,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space
After upgrade:hive.security.authenticator.manager,hive.security.authorization.manager,hive.security.metastore.authorization.manager,hive.security.metastore.authenticator.manager,hive.users.in.admin.role,hive.server2.xsrf.filter.enabled,hive.security.authorization.enabled,hive.distcp.privileged.doAs,hive.server2.authentication.ldap.baseDN,hive.server2.authentication.ldap.url,hive.server2.authentication.ldap.Domain,hive.server2.authentication.ldap.groupDNPattern,hive.server2.authentication.ldap.groupFilter,hive.server2.authentication.ldap.userDNPattern,hive.server2.authentication.ldap.userFilter,hive.server2.authentication.ldap.groupMembershipKey,hive.server2.authentication.ldap.userMembershipKey,hive.server2.authentication.ldap.groupClassKey,hive.server2.authentication.ldap.customLDAPQuery,hive.privilege.synchronizer.interval,hive.spark.client.connect.timeout,hive.spark.client.server.connect.timeout,hive.spark.client.channel.log.level,hive.spark.client.rpc.max.size,hive.spark.client.rpc.threads,hive.spark.client.secret.bits,hive.spark.client.rpc.server.address,hive.spark.client.rpc.server.port,hive.spark.client.rpc.sasl.mechanisms,bonecp.,hive.druid.broker.address.default,hive.druid.coordinator.address.default,hikaricp.,hadoop.bin.path,yarn.bin.path,spark.home,hive.driver.parallel.compilation.global.limit,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space,_hive.local.session.path,_hive.hdfs.session.path,_hive.tmp_table_space
- hive.default.fileformat.managed
- Before upgrade:
None
After upgrade:
ORC
- hive.default.rcfile.serde
- Before upgrade:
org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe
After upgrade:
org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe
Not supported in Impala. Impala cannot read Hive-created RC tables.
- hive.driver.parallel.compilation
- Before upgrade: FALSE
After upgrade: TRUE
- hive.exec.dynamic.partition.mode
- Before upgrade: strict
After upgrade: nonstrict
In CDP Private Cloud Base, accidental use of dynamic partitioning feature is not prevented by default.
- hive.exec.max.dynamic.partitions
- Before upgrade:
1000
After upgrade:
In CDP Private Cloud Base, fewer restrictions on dynamic paritioning occur than in the pre-upgrade CDH or HDP cluster.5000
- hive.exec.max.dynamic.partitions.pernode
- Before upgrade:
100
After upgrade:
In CDP Private Cloud Base, fewer restrictions on dynamic paritioning occur than in the pre-upgrade CDH or HDP cluster.2000
- hive.exec.post.hooks
- Before upgrade:
com.cloudera.navigator.audit.hive.HiveExecHookContext,org.apache.hadoop.hive.ql.hooks.LineageLogger
After upgrade:
A prime number is recommended.org.apache.hadoop.hive.ql.hooks.HiveProtoLoggingHook
- hive.exec.reducers.max
- Before upgrade:
1099
After upgrade:
Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default1009
- hive.execution.engine
- Before upgrade:
mr
After upgrade:
Tez is now the only supported execution engine, existing queries that change execution mode to Spark or MapReduce within a session, for example, fail.tez
- hive.fetch.task.conversion
- Before upgrade:
minimal
After upgrade:
more
- hive.fetch.task.conversion.threshold
- Before upgrade:
256MB
After upgrade:
1GB
Exception: Preserves pre-upgrade value if old default is overridden; otherwise, uses new default.
- hive.hashtable.key.count.adjustment
- Before upgrade:
1
After upgrade:
0.99
- hive.limit.optimize.enable
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.limit.pushdown.memory.usage
- Before upgrade:
0.1
After upgrade:
0.04
- hive.mapjoin.hybridgrace.hashtable
- Before upgrade:
TRUE
After upgrade:
FALSE
- hive.mapred.reduce.tasks.speculative.execution
- Before upgrade:
TRUE
After upgrade:
FALSE
- hive.metastore.aggregate.stats.cache.enabled
- Before upgrade:
TRUE
After upgrade:
FALSE
- hive.metastore.disallow.incompatible.col.type.changes
- Before upgrade:
FALSE
- hive.metastore.dml.events
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.metastore.event.message.factory
- Before upgrade:
org.apache.hadoop.hive.metastore.messaging.json.ExtendedJSONMessageFactory
After upgrade:
org.apache.hadoop.hive.metastore.messaging.json.gzip.GzipJSONMessageEncoder
- hive.metastore.uri.selection
- Before upgrade:
SEQUENTIAL
After upgrade:
RANDOM
- hive.metastore.warehouse.dir
- Before upgrade from CDH:
/user/hive/warehouse
- hive.optimize.metadataonly
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.optimize.point.lookup.min
- Before upgrade:
31
After upgrade:
2
- hive.prewarm.numcontainers
- Before upgrade:
10
After upgrade:
3
- hive.script.operator.env.blacklist
- Before upgrade:
hive.txn.valid.txns,hive.script.operator.env.blacklist
- hive.security.authorization.sqlstd.confwhitelist
- Before upgrade:
hive\.auto\..*hive\.cbo\..*hive\.convert\..*hive\.exec\.dynamic\.partition.*hive\.exec\..*\.dynamic\.partitions\..*hive\.exec\.compress\..*hive\.exec\.infer\..*hive\.exec\.mode.local\..*hive\.exec\.orc\..*hive\.exec\.parallel.*hive\.explain\..*hive\.fetch.task\..*hive\.groupby\..*hive\.hbase\..*hive\.index\..*hive\.index\..*hive\.intermediate\..*hive\.join\..*hive\.limit\..*hive\.log\..*hive\.mapjoin\..*hive\.merge\..*hive\.optimize\..*hive\.orc\..*hive\.outerjoin\..*hive\.parquet\..*hive\.ppd\..*hive\.prewarm\..*hive\.server2\.proxy\.userhive\.skewjoin\..*hive\.smbjoin\..*hive\.stats\..*hive\.strict\..*hive\.tez\..*hive\.vectorized\..*mapred\.map\..*mapred\.reduce\..*mapred\.output\.compression\.codecmapred\.job\.queuenamemapred\.output\.compression\.typemapred\.min\.split\.sizemapreduce\.job\.reduce\.slowstart\.completedmapsmapreduce\.job\.queuenamemapreduce\.job\.tagsmapreduce\.input\.fileinputformat\.split\.minsizemapreduce\.map\..*mapreduce\.reduce\..*mapreduce\.output\.fileoutputformat\.compress\.codecmapreduce\.output\.fileoutputformat\.compress\.typeoozie\..*tez\.am\..*tez\.task\..*tez\.runtime\..*tez\.queue\.namehive\.transpose\.aggr\.joinhive\.exec\.reducers\.bytes\.per\.reducerhive\.client\.stats\.countershive\.exec\.default\.partition\.namehive\.exec\.drop\.ignorenonexistenthive\.counters\.group\.namehive\.default\.fileformat\.managedhive\.enforce\.bucketmapjoinhive\.enforce\.sortmergebucketmapjoinhive\.cache\.expr\.evaluationhive\.query\.result\.fileformathive\.hashtable\.loadfactorhive\.hashtable\.initialCapacityhive\.ignore\.mapjoin\.hinthive\.limit\.row\.max\.sizehive\.mapred\.modehive\.map\.aggrhive\.compute\.query\.using\.statshive\.exec\.rowoffsethive\.variable\.substitutehive\.variable\.substitute\.depthhive\.autogen\.columnalias\.prefix\.includefuncnamehive\.autogen\.columnalias\.prefix\.labelhive\.exec\.check\.crossproductshive\.cli\.tez\.session\.asynchive\.compathive\.exec\.concatenate\.check\.indexhive\.display\.partition\.cols\.separatelyhive\.error\.on\.empty\.partitionhive\.execution\.enginehive\.exec\.copyfile\.maxsizehive\.exim\.uri\.scheme\.whitelisthive\.file\.max\.footerhive\.insert\.into\.multilevel\.dirshive\.localize\.resource\.num\.wait\.attemptshive\.multi\.insert\.move\.tasks\.share\.dependencieshive\.support\.quoted\.identifiershive\.resultset\.use\.unique\.column\.nameshive\.analyze\.stmt\.collect\.partlevel\.statshive\.exec\.schema\.evolutionhive\.server2\.logging\.operation\.levelhive\.server2\.thrift\.resultset\.serialize\.in\.taskshive\.support\.special\.characters\.tablenamehive\.exec\.job\.debug\.capture\.stacktraceshive\.exec\.job\.debug\.timeouthive\.llap\.io\.enabledhive\.llap\.io\.use\.fileid\.pathhive\.llap\.daemon\.service\.hostshive\.llap\.execution\.modehive\.llap\.auto\.allow\.uberhive\.llap\.auto\.enforce\.treehive\.llap\.auto\.enforce\.vectorizedhive\.llap\.auto\.enforce\.statshive\.llap\.auto\.max\.input\.sizehive\.llap\.auto\.max\.output\.sizehive\.llap\.skip\.compile\.udf\.checkhive\.llap\.client\.consistent\.splitshive\.llap\.enable\.grace\.join\.in\.llaphive\.llap\.allow\.permanent\.fnshive\.exec\.max\.created\.fileshive\.exec\.reducers\.maxhive\.reorder\.nway\.joinshive\.output\.file\.extensionhive\.exec\.show\.job\.failure\.debug\.infohive\.exec\.tasklog\.debug\.timeouthive\.query\.id
After upgrade:hive\.auto\..*hive\.cbo\..*hive\.convert\..*hive\.druid\..*hive\.exec\.dynamic\.partition.*hive\.exec\.max\.dynamic\.partitions.*hive\.exec\.compress\..*hive\.exec\.infer\..*hive\.exec\.mode.local\..*hive\.exec\.orc\..*hive\.exec\.parallel.*hive\.exec\.query\.redactor\..*hive\.explain\..*hive\.fetch.task\..*hive\.groupby\..*hive\.hbase\..*hive\.index\..*hive\.index\..*hive\.intermediate\..*hive\.jdbc\..*hive\.join\..*hive\.limit\..*hive\.log\..*hive\.mapjoin\..*hive\.merge\..*hive\.optimize\..*hive\.materializedview\..*hive\.orc\..*hive\.outerjoin\..*hive\.parquet\..*hive\.ppd\..*hive\.prewarm\..*hive\.query\.redaction\..*hive\.server2\.thrift\.resultset\.default\.fetch\.sizehive\.server2\.proxy\.userhive\.skewjoin\..*hive\.smbjoin\..*hive\.stats\..*hive\.strict\..*hive\.tez\..*hive\.vectorized\..*hive\.query\.reexecution\..*reexec\.overlay\..*fs\.defaultFSssl\.client\.truststore\.locationdistcp\.atomicdistcp\.ignore\.failuresdistcp\.preserve\.statusdistcp\.preserve\.rawxattrsdistcp\.sync\.foldersdistcp\.delete\.missing\.sourcedistcp\.keystore\.resourcedistcp\.liststatus\.threadsdistcp\.max\.mapsdistcp\.copy\.strategydistcp\.skip\.crcdistcp\.copy\.overwritedistcp\.copy\.appenddistcp\.map\.bandwidth\.mbdistcp\.dynamic\..*distcp\.meta\.folderdistcp\.copy\.listing\.classdistcp\.filters\.classdistcp\.options\.skipcrccheckdistcp\.options\.mdistcp\.options\.numListstatusThreadsdistcp\.options\.mapredSslConfdistcp\.options\.bandwidthdistcp\.options\.overwritedistcp\.options\.strategydistcp\.options\.idistcp\.options\.p.*distcp\.options\.updatedistcp\.options\.deletemapred\.map\..*mapred\.reduce\..*mapred\.output\.compression\.codecmapred\.job\.queue\.namemapred\.output\.compression\.typemapred\.min\.split\.sizemapreduce\.job\.reduce\.slowstart\.completedmapsmapreduce\.job\.queuenamemapreduce\.job\.tagsmapreduce\.input\.fileinputformat\.split\.minsizemapreduce\.map\..*mapreduce\.reduce\..*mapreduce\.output\.fileoutputformat\.compress\.codecmapreduce\.output\.fileoutputformat\.compress\.typeoozie\..*tez\.am\..*tez\.task\..*tez\.runtime\..*tez\.queue\.namehive\.transpose\.aggr\.joinhive\.exec\.reducers\.bytes\.per\.reducerhive\.client\.stats\.countershive\.exec\.default\.partition\.namehive\.exec\.drop\.ignorenonexistenthive\.counters\.group\.namehive\.default\.fileformat\.managedhive\.enforce\.bucketmapjoinhive\.enforce\.sortmergebucketmapjoinhive\.cache\.expr\.evaluationhive\.query\.result\.fileformathive\.hashtable\.loadfactorhive\.hashtable\.initialCapacityhive\.ignore\.mapjoin\.hinthive\.limit\.row\.max\.sizehive\.mapred\.modehive\.map\.aggrhive\.compute\.query\.using\.statshive\.exec\.rowoffsethive\.variable\.substitutehive\.variable\.substitute\.depthhive\.autogen\.columnalias\.prefix\.includefuncnamehive\.autogen\.columnalias\.prefix\.labelhive\.exec\.check\.crossproductshive\.cli\.tez\.session\.asynchive\.compathive\.display\.partition\.cols\.separatelyhive\.error\.on\.empty\.partitionhive\.execution\.enginehive\.exec\.copyfile\.maxsizehive\.exim\.uri\.scheme\.whitelisthive\.file\.max\.footerhive\.insert\.into\.multilevel\.dirshive\.localize\.resource\.num\.wait\.attemptshive\.multi\.insert\.move\.tasks\.share\.dependencieshive\.query\.results\.cache\.enabledhive\.query\.results\.cache\.wait\.for\.pending\.resultshive\.support\.quoted\.identifiershive\.resultset\.use\.unique\.column\.nameshive\.analyze\.stmt\.collect\.partlevel\.statshive\.exec\.schema\.evolutionhive\.server2\.logging\.operation\.levelhive\.server2\.thrift\.resultset\.serialize\.in\.taskshive\.support\.special\.characters\.tablenamehive\.exec\.job\.debug\.capture\.stacktraceshive\.exec\.job\.debug\.timeouthive\.llap\.io\.enabledhive\.llap\.io\.use\.fileid\.pathhive\.llap\.daemon\.service\.hostshive\.llap\.execution\.modehive\.llap\.auto\.allow\.uberhive\.llap\.auto\.enforce\.treehive\.llap\.auto\.enforce\.vectorizedhive\.llap\.auto\.enforce\.statshive\.llap\.auto\.max\.input\.sizehive\.llap\.auto\.max\.output\.sizehive\.llap\.skip\.compile\.udf\.checkhive\.llap\.client\.consistent\.splitshive\.llap\.enable\.grace\.join\.in\.llaphive\.llap\.allow\.permanent\.fnshive\.exec\.max\.created\.fileshive\.exec\.reducers\.maxhive\.reorder\.nway\.joinshive\.output\.file\.extensionhive\.exec\.show\.job\.failure\.debug\.infohive\.exec\.tasklog\.debug\.timeouthive\.query\.idhive\.query\.tag
- hive.security.command.whitelist
- Before upgrade:
set,reset,dfs,add,list,delete,reload,compile
After upgrade:
set,reset,dfs,add,list,delete,reload,compile,llap
- hive.server2.enable.doAs
- Before upgrade:
TRUE
(in case of an insecure cluster only)After upgrade:
FALSE
(in all cases)Affects only insecure clusters by turning off impersonation. Permission issues are expected to arise for affected clusters.
- hive.server2.idle.session.timeout
- Before upgrade:
12 hours
After upgrade:
24 hours
- hive.server2.max.start.attempts
- Before upgrade:
30
After upgrade:
5
- hive.server2.parallel.ops.in.session
- Before upgrade:
TRUE
After upgrade:
FALSE
A Tez limitation requires disabling this property; otherwise, queries submitted concurrently on a single JDBC connection fail or execute slower.
- hive.server2.support.dynamic.service.discovery
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.server2.tez.initialize.default.sessions
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.server2.thrift.max.worker.threads
- Before upgrade:
100
After upgrade:
500
Exception: Preserves pre-upgrade value if the old default is overridden; otherwise, uses new default.
- hive.server2.thrift.resultset.max.fetch.size
- Before upgrade:
1000
After upgrade:
10000
- hive.service.metrics.file.location
- Before upgrade: /
var/log/hive/metrics-hiveserver2/metrics.log
After upgrade:
/var/log/hive/metrics-hiveserver2-hiveontez/metrics.log
This location change is due to a service name change.
- hive.stats.column.autogather
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.stats.deserialization.factor
- Before upgrade:
1
After upgrade:
10
- hive.support.special.characters.tablename
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.tez.auto.reducer.parallelism
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.tez.bucket.pruning
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.tez.container.size
- Before upgrade:
-1
After upgrade:
4096
- hive.tez.exec.print.summary
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.txn.manager
- Before upgrade:
org.apache.hadoop.hive.ql.lockmgr.DummyTxnManager
After upgrade:
org.apache.hadoop.hive.ql.lockmgr.DbTxnManager
- hive.vectorized.execution.mapjoin.minmax.enabled
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.vectorized.execution.mapjoin.native.fast.hashtable.enabled
- Before upgrade:
FALSE
After upgrade:
TRUE
- hive.vectorized.use.row.serde.deserialize
- Before upgrade:
FALSE
After upgrade:
TRUE