CDH 6 includes Apache Kafka as part of the core package. The documentation includes improved contents for how to set up, install, and administer your Kafka ecosystem. For more information, see the Cloudera Enterprise 6.0.x Apache Kafka Guide. We look forward to your feedback on both the existing and new documentation.

Issues Fixed in CDK 4.1.0 Powered By Apache Kafka

Upstream Issues Fixed

The following upstream issues are fixed in CDK 4.1.0:
  • [KAFKA-1120] - Controller could miss a broker state change
  • [KAFKA-2334] - Prevent HW from going back during leader failover
  • [KAFKA-3832] - Kafka Connect's JSON Converter never outputs a null value
  • [KAFKA-3832] - Kafka Connect's JSON Converter never outputs a null value
  • [KAFKA-4217] - KStream.transform equivalent of flatMap
  • [KAFKA-4453] - add request prioritization
  • [KAFKA-4468] - Correctly calculate the window end timestamp after read from state stores
  • [KAFKA-4850] - RocksDb cannot use Bloom Filters
  • [KAFKA-5117] - Kafka Connect REST endpoints reveal Password typed values
  • [KAFKA-5272] - Improve validation for Alter Configs (KIP-133)
  • [KAFKA-5462] - Add a configuration for users to specify a template for building a custom principal name
  • [KAFKA-5503] - Idempotent producer ignores shutdown while fetching ProducerId
  • [KAFKA-5503] - Idempotent producer ignores shutdown while fetching ProducerId
  • [KAFKA-5692] - Refactor PreferredReplicaLeaderElectionCommand to use AdminClient
  • [KAFKA-5950] - AdminClient should retry based on returned error codes
  • [KAFKA-5994] - Improve transparency of broker user ACL misconfigurations
  • [KAFKA-6036] - Enable logical materialization to physical materialization
  • [KAFKA-6388] - Error while trying to roll a segment that already exists
  • [KAFKA-6431] - Lock contention in Purgatory
  • [KAFKA-6478] - kafka-run-class.bat fails if CLASSPATH contains spaces
  • [KAFKA-6567] - KStreamWindowReduce can be replaced by KStreamWindowAggregate
  • [KAFKA-6774] - Improve default groupId behavior in consumer
  • [KAFKA-6833] - KafkaProducer throws "Invalid partition given with record" exception
  • [KAFKA-6928] - StreamsPartitionAssignor is double retrying within InternalTopicManager
  • [KAFKA-6970] - Kafka streams lets the user call init() and close() on a state store, when inside Processors
  • [KAFKA-7024] - Rocksdb state directory should be created before opening the DB
  • [KAFKA-7037] - delete topic command replaces '+' from the topic name which leads incorrect topic deletion
  • [KAFKA-7051] - Improve the efficiency of the ReplicaManager when there are many partitions
  • [KAFKA-7054] - Kafka describe command should throw topic doesn't exist exception.
  • [KAFKA-7096] - Consumer should drop the data for unassigned topic partitions
  • [KAFKA-7110] - Windowed changelog keys not deserialized properly by TimeWindowedSerde
  • [KAFKA-7165] - Error while creating ephemeral at /brokers/ids/BROKER_ID
  • [KAFKA-7181] - Kafka Streams State stuck in rebalancing after one of the StreamThread encounters IllegalStateException
  • [KAFKA-7235] - Use brokerZkNodeVersion to prevent broker from processing outdated controller request
  • [KAFKA-7253] - The connector type responded by worker is always null when creating connector
  • [KAFKA-7259] - Remove deprecated ZKUtils usage from ZkSecurityMigrator
  • [KAFKA-7312] - Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
  • [KAFKA-7352] - KIP-368: Allow SASL Connections to Periodically Re-Authenticate
  • [KAFKA-7367] - Streams should not create state store directories unless they are needed
  • [KAFKA-7391] - Introduce close(Duration) to Producer and AdminClient instead of close(long, TimeUnit)
  • [KAFKA-7402] - Kafka Streams should implement AutoCloseable where appropriate
  • [KAFKA-7412] - Bug prone response from producer.send(ProducerRecord, Callback) if Kafka broker is not running
  • [KAFKA-7418] - Add '--help' option to all available Kafka CLI commands (KIP-374)
  • [KAFKA-7420] - Global stores should be guarded as read-only for regular tasks
  • [KAFKA-7433] - Introduce broker options in TopicCommand to use AdminClient
  • [KAFKA-7443] - OffsetOutOfRangeException in restoring state store from changelog topic when start offset of local checkpoint is smaller than that of changelog topic
  • [KAFKA-7446] - Better error message to explain the upper limit of TimeWindow
  • [KAFKA-7449] - Kafka console consumer is not sending topic to deserializer
  • [KAFKA-7461] - Connect Values converter should have coverage of logical types
  • [KAFKA-7476] - SchemaProjector is not properly handling Date-based logical types
  • [KAFKA-7478] - Reduce OAuthBearerLoginModule verbosity
  • [KAFKA-7510] - KStreams RecordCollectorImpl leaks data to logs on error
  • [KAFKA-7518] - FutureRecordMetadata.get deadline calculation from timeout is not using timeunit
  • [KAFKA-7528] - Standardize on Min/Avg/Max Kafka metrics' default value
  • [KAFKA-7532] - Missing stringification in error message "Removed ArrayBuffer() from list of shutting down brokers. (kafka.controller.KafkaController)"
  • [KAFKA-7536] - TopologyTestDriver cannot pre-populate KTable or GlobalKTable
  • [KAFKA-7537] - Only include live brokers in the UpdateMetadataRequest sent to existing brokers if there is no change in the partition states
  • [KAFKA-7549] - Old ProduceRequest with zstd compression does not return error to client
  • [KAFKA-7551] - Refactor to create both producer & consumer in Worker
  • [KAFKA-7557] - optimize LogManager.truncateFullyAndStartAt()
  • [KAFKA-7560] - PushHttpMetricsReporter should not convert metric value to double
  • [KAFKA-7567] - Clean up internal metadata usage for consistency and extensibility
  • [KAFKA-7568] - Return leader epoch in ListOffsets responses
  • [KAFKA-7576] - Dynamic update of replica fetcher threads may fail to start/close fetchers
  • [KAFKA-7584] - StreamsConfig throws ClassCastException if max.in.flight.request.per.connect is specified as String
  • [KAFKA-7607] - NetworkClientUtils.sendAndReceive can take a long time to return during shutdown
  • [KAFKA-7610] - Detect consumer failures in initial JoinGroup
  • [KAFKA-7612] - Fix javac warnings and enable warnings as errors
  • [KAFKA-7616] - MockConsumer can return ConsumerRecords objects with a non-empty map but no records
  • [KAFKA-7620] - ConfigProvider is broken for KafkaConnect when TTL is not null
  • [KAFKA-7641] - Add `consumer.group.max.size` to cap consumer metadata size on broker
  • [KAFKA-7655] - Metadata spamming requests from Kafka Streams under some circumstances, potential DOS
  • [KAFKA-7657] - Invalid reporting of stream state in Kafka streams application
  • [KAFKA-7660] - Stream Metrics - Memory Analysis
  • [KAFKA-7671] - A KStream/GlobalKTable join shouldn't reset the repartition flag
  • [KAFKA-7672] - The local state not fully restored after KafkaStream rebalanced, resulting in data loss
  • [KAFKA-7673] - Upgrade RocksDB to include fix for WinEnvIO::GetSectorSize
  • [KAFKA-7678] - Failed to close producer due to java.lang.NullPointerException
  • [KAFKA-7687] - Print batch level information in DumpLogSegments when deep iterating
  • [KAFKA-7692] - updateFirstUnstableOffset NPE due to sequenceId overflow in ProducerStateManager.append
  • [KAFKA-7693] - "IllegalArgumentException: Invalid negative sequence number used" in Kafka Client
  • [KAFKA-7697] - Possible deadlock in kafka.cluster.Partition
  • [KAFKA-7702] - Prefixed ACLs don't work with single character prefix
  • [KAFKA-7704] - kafka.server.ReplicaFetechManager.MaxLag.Replica metric is reported incorrectly
  • [KAFKA-7709] - ConcurrentModificationException occurs when iterating through multiple partitions in Sender.getExpiredInflightBatches
  • [KAFKA-7712] - Handle exceptions from immediately connected channels in Selector
  • [KAFKA-7719] - Improve fairness in SocketServer processors
  • [KAFKA-7734] - Metrics tags should use LinkedHashMap to guarantee ordering
  • [KAFKA-7738] - Track partition leader epochs in client metadata
  • [KAFKA-7741] - Bad dependency via SBT
  • [KAFKA-7742] - DelegationTokenCache#hmacIdCache entry is not cleared when a token is removed using removeToken(String tokenId) API.
  • [KAFKA-7752] - zookeeper-security-migration.sh does not remove ACL on kafka-acl-extended
  • [KAFKA-7755] - Kubernetes - Kafka clients are resolving DNS entries only one time
  • [KAFKA-7758] - When Naming a Repartition Topic with Aggregations Reuse Repartition Graph Node for Multiple Operations
  • [KAFKA-7759] - Disable WADL output in Connect REST API
  • [KAFKA-7762] - KafkaConsumer uses old API in the javadocs
  • [KAFKA-7781] - Add validation check for Topic retention.ms property
  • [KAFKA-7786] - Fast update of leader epoch may stall partition fetching due to FENCED_LEADER_EPOCH
  • [KAFKA-7798] - Expose embedded client context from KafkaStreams threadMetadata
  • [KAFKA-7808] - AdminClient#describeTopics should not throw InvalidTopicException if topic name is not found
  • [KAFKA-7824] - Require member.id for initial join group request
  • [KAFKA-7837] - maybeShrinkIsr may not reflect OfflinePartitions immediately
  • [KAFKA-7838] - improve logging in Partition.maybeShrinkIsr()
  • [KAFKA-7873] - KafkaBasedLog's consumer should always seek to beginning when starting
  • [KAFKA-7880] - KafkaConnect should standardize worker thread name
  • [KAFKA-7890] - Invalidate ClusterConnectionState cache for a broker if the hostname of the broker changes.
  • [KAFKA-7895] - Ktable supress operator emitting more than one record for the same key per window
  • [KAFKA-7897] - Invalid use of epoch cache with old message format versions
  • [KAFKA-7902] - SASL/OAUTHBEARER can become unable to connect: javax.security.sasl.SaslException: Unable to find OAuth Bearer token in Subject's private credentials (size=2)
  • [KAFKA-7909] - Ensure timely rebalance completion after pending members rejoin or fail
  • [KAFKA-7915] - SASL authentication failures may return sensitive data to client
  • [KAFKA-7916] - Streams store cleanup: unify wrapping!!
  • [KAFKA-7920] - Do not permit zstd use until inter.broker.protocol.version is updated to 2.1
  • [KAFKA-7935] - UNSUPPORTED_COMPRESSION_TYPE if ReplicaManager.getLogConfig returns None
  • [KAFKA-7945] - ExpiringCredentialRefreshingLogin - timeout value is negative
  • [KAFKA-8002] - Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
  • [KAFKA-8012] - NullPointerException while truncating at high watermark can crash replica fetcher thread
  • [KAFKA-8040] - Streams needs to handle timeout in initTransactions
  • [KAFKA-8061] - Handle concurrent ProducerId reset and call to Sender thread shutdown
  • [KAFKA-8065] - Forwarding modified timestamps does not reset timestamp correctly
  • [KAFKA-8069] - Committed offsets get cleaned up right after the coordinator loading them back from __consumer_offsets in broker with old inter-broker protocol version (< 2.2)