Behavioral Changes

Behavioral changes denote a marked change in behavior from the previously released version to this version of HDP.

Table 1. Behavioral Changes
Bug ID	Apache Component	Apache JIRA	Summary	Details
BUG-118583	Ranger	N/A	Allow users to define policy level conditions	Previous behavior: Before this feature, users had provision to define policy conditions at individual policy items. New behavior: This allows users to define policy wide conditions. End users has flexibility to use both the features.
BUG-119330	Ranger	RANGER-2408	With introduction of Security zones in Ranger. Users of different roles in Ranger have different permissions based on they being Zone Admin or Zone Auditor for particular set of zones.	Previous behavior: Before this change, the user’s capabilities were not managed by their involvement in zones. So if user is not part of any zone, it works as usual. New behavior: This feature takes care of restricting access to different APIs based on individual’s role and their part in zone. See Apache Ranger Cwiki for more inforamtion.
N/A	Kafka	KIP-186	This change increases the default offset retention time from 1 day to 7 days. This makes it less likely to "lose" offsets in an application that commits infrequently. It also increases the active set of offsets and therefore can increase memory usage on the broker.	Previous behavior: Default offset retention time was one day. New behavior: The console consumer currently enables offset commit by default, and can be the source of a large number of offsets. Now, the offsets are preserved for 7 days instead of 1. You can retain the existing behavior by setting the broker configuration `offsets.retention.minutes` to 1440.
N/A	Kafka	N/A	Support for Java 7 has been dropped, Java 8 is now the minimum version required.	Previous behavior: Support for Java 7. New behavior: Java 8 is now the minimum version required.
N/A	Kafka	N/A	The default value for ssl.endpoint.identification.algorithm was changed to HTTPS, which performs hostname verification (man-in-the-middle attacks are possible otherwise).	New behavior: The default value for ssl.endpoint.identification.algorithm was changed to HTTPS. Expected customer action: Set `ssl.endpoint.identification.algorithm` to an empty string to restore the previous behavior.
KAFKA-5674	Kafka	N/A	Extends the lower interval of max.connections.per.ip minimum to zero and therefore allows IP-based filtering of inbound connections.	New behavior: Extends the lower interval of `max.connections.per.ip` minimum value to zero, and therefore allows IP-based filtering of inbound connections. Expected customer action: None.
N/A	Kafka	KIP-272	Added API version tag to the metric kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce\|FetchConsumer\|FetchFollower\|...}. This metric now becomes kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce\|FetchConsumer\|FetchFollower\|...},version={0\|1\|2\|3\|...}.	New behavior: This change will impact JMX monitoring tools that do not automatically aggregate. Expected customer action: To get the total count for a specific request type, the tool needs to be updated to aggregate across different versions.
N/A	Kafka	KIP-225	Changed the metric "records.lag" to use tags for topic and partition.	Previous behavior: The original version of the name format was "{topic}-{partition}.records-lag". New behavior: The original version with the name format "{topic}-{partition}.records-lag" has now been removed.
N/A	Kafka	N/A	The Scala consumers (deprecated since 0.11.0.0) are now removed.	Previous behavior: The Scala consumers has been deprecated since version 0.11.0.0. New behavior: This is now removed. Expected customer action: The Scala consumers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
N/A	Kafka	N/A	The Scala producers, which have been deprecated since Kafka version 0.10.0.0, have now been removed. The Java producer has been the recommended option since 0.9.0.0.	Previous behavior: The Scala producers has been deprecated since 0.10.0.0. New behavior: This is now removed.Note that the behavior of the default partitioner in the Java producer differs from the default partitioner in the Scala producers. Expected customer action: Users migrating should consider configuring a custom partitioner that retains the previous behavior. The Scala producers in 1.1.0 (and older) will continue to work even if the brokers are upgraded to 2.0.0.
N/A	Kakfa	N/A	MirrorMaker and ConsoleConsumer no longer support the Scala consumer, they always use the Java consumer.	New behavior: MirrorMaker and ConsoleConsumer always use the Java consumer. Expected customer action: None.
N/A	Kafka	N/A	The ConsoleProducer no longer supports the Scala producer, it always uses the Java producer.	New behavior: The ConsoleProducer always uses the Java producer. Expected customer action: None.
N/A	Kafka	N/A	A number of deprecated tools that rely on the Scala clients are now removed.	New behavior: Removed Scala clients: `ReplayLogProducer`, `SimpleConsumerPerformance`, `SimpleConsumerShell`, `ExportZkOffsets`, `ImportZkOffsets`, `UpdateOffsetsInZK`, `VerifyConsumerRebalance`. Expected customer action: None.
N/A	Kafka	N/A	The deprecated `kafka.tools.ProducerPerformance` parameter is removed.	New behavior: `kafka.tools.ProducerPerformance` parameter is removed. Expected customer action: Use `org.apache.kafka.tools.ProducerPerformance`
N/A	Kafka	KIP-284	Changed the retention time for Kafka Streams repartition topics.	New behavior: Changed the retention time for Kafka Streams repartition topics. by setting its default value to `Long.MAX_VALUE`. Expected customer action: Use `org.apache.kafka.tools.ProducerPerformance`
N/A	Kafka	KIP-266	Adds a new consumer configuration default.api.timeout.ms to specify the default timeout to use for KafkaConsumer APIs that could block. This change also adds overloads for such blocking APIs to support specifying a specific timeout to use for each of them instead of using the default timeout set by default.api.timeout.ms.	Previous behavior: The value of request.timeout.ms parameter was a little higher than 5 minutes to account for maximum time that a rebalance would take. New behavior: The default value of request.timeout.ms has been changed to 30 seconds. Now we treat the JoinGroup request in the rebalance as a special case and use a value derived from max.poll.interval.ms for the request timeout. All other request types use the timeout defined by request.timeout.ms. A new poll(Duration) API has been added which does not block for dynamic partition assignment. The old poll(long) API has been deprecated and will be removed in a future version. Overloads have also been added for other KafkaConsumer methods such as `partitionsFor`, `listTopics`, `offsetsForTimes`, `beginningOffsets`, `endOffsets`. Expected customer action: None.
N/A	Kafka	N/A	The internal method `kafka.admin.AdminClient.deleteRecordsBefore` is removed.	New behavior: The internal method kafka.admin.AdminClient.deleteRecordsBefore has been removed. Expected customer action: None.
N/A	Kafka	KIP-277	The `AclCommand` tool, producer convenience option uses the finer grained ACL on the given topic.	New behavior: The `AclCommand` tool, producer convenience option uses the finer grained ACL on the given topic. Expected customer action: None.
N/A	Kafka	KIP-176	Removes the `--new-consumer` option for all consumer based tools.	New behavior: The -`-new-consumer`option isa removed because it is redundant since the new consumer is automatically used if `--bootstrap-server` is defined. Expected customer action: None.
N/A	Kafka	KIP-283	Improved message down-conversion handling on Kafka broker, which has typically been a memory-intensive operation. New topic and broker configurations message.downconversion.enable and log.message.downconversion.enable are added to control whether down-conversion is enabled. When disabled, broker does not perform any down-conversion and instead sends an UNSUPPORTED_VERSION error to the client.	New behavior: Adds a mechanism by which the operation becomes less memory-intensive by down-converting chunks of partition data at a time which helps put an upper bound on memory consumption. With this improvement, there is a change in FetchResponse protocol behavior where the broker could send an oversized message batch towards the end of the response with an invalid offset. Such oversized messages must be ignored by consumer clients, as is done by KafkaConsumer. Expected customer action: None.
N/A	ZooKeeper	ZOOKEEPER-1045, ZOOKEEPER-2726	No authentication/authorization is enforced when a server attempts to join a quorum in Apache ZooKeeper	Previous behavior: The command doesn't check any permission and returns all information contained in the ACL Id field as plaintext string. New behavior: The command now does check any permission and returns all information contained in the ACL Id field as plaintext string. Expected customer action: None.