Release Notes - Kafka - Version 3.4.0

Below is a summary of the JIRA issues addressed in the 3.4.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-10360] - Disabling JmxReporter registration
[KAFKA-13602] - Allow to broadcast a result record
[KAFKA-13715] - Add "generation" field into consumer protocol
[KAFKA-14286] - Time based cluster metadata snapshots

Improvement

[KAFKA-10149] - Do not prevent automatic preferred election when reassignment in progress
[KAFKA-12878] - Support --bootstrap-server kafka-streams-application-reset
[KAFKA-12960] - WindowStore and SessionStore do not enforce strict retention time
[KAFKA-13401] - Introduce a new Interface to manage Kafka resources in MM2
[KAFKA-13731] - Standalone Connect workers should not require connector configs to start
[KAFKA-13764] - Potential improvements for Connect incremental rebalancing logic
[KAFKA-13809] - FileStreamSinkConnector and FileStreamSourceConnector should propagate full configuration to tasks
[KAFKA-14017] - File source connector should implement KIP-618 APIs
[KAFKA-14095] - Improve handling of sync offset failures in MirrorMaker
[KAFKA-14097] - Separate configuration for producer ID expiry
[KAFKA-14098] - Internal Kafka clients used by Kafka Connect should have distinguishable client IDs
[KAFKA-14130] - Reduce RackAwarenessIntegrationTest to a unit test
[KAFKA-14146] - KIP-840: Config file option for MessageReader/MessageFormatter in ConsoleProducer/ConsoleConsumer
[KAFKA-14160] - Streamline clusterId retrieval in Connect
[KAFKA-14162] - HoistField and MaskField SMTs should not return immutable maps/lists
[KAFKA-14176] - Reduce visibility of WorkerConfig.lookupKafkaClusterId
[KAFKA-14206] - Upgrade zookeeper to 3.7.1 to address security vulnerabilities
[KAFKA-14212] - Fetch error response when hitting public OAuth/OIDC provider
[KAFKA-14239] - Merge StateRestorationIntegrationTest into RestoreIntegrationTest
[KAFKA-14293] - Basic Auth filter should set the SecurityContext after a successful login
[KAFKA-14294] - Kafka Streams should commit transaction when no records are processed
[KAFKA-14342] - KafkaOffsetBackingStore should clear offsets for source partitions on tombstone messages
[KAFKA-14344] - Improve MM2 integration test by building EmbeddedKafkaCluster with common configs used for all clients
[KAFKA-14346] - Remove static methods from internal Connect APIs for easier testing
[KAFKA-14375] - Remove use of "authorizer-properties" in EndToEndAuthorizationTest.scala
[KAFKA-14392] - KRaft broker heartbeat timeout should not exceed broker.session.timeout.ms
[KAFKA-14398] - Update EndToEndAuthorizerTest.scala to test with ZK and KRAFT quorum servers
[KAFKA-14414] - Remove unnecessary usage of ObjectSerializationCache
[KAFKA-14430] - optimize: -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT

Bug

[KAFKA-4852] - ByteBufferSerializer not compatible with offsets
[KAFKA-12476] - Worker can block for longer than scheduled rebalance delay and/or session key TTL
[KAFKA-12495] - Unbalanced connectors/tasks distribution will happen in Connect's incremental cooperative assignor
[KAFKA-12497] - Source task offset commits continue even after task has failed
[KAFKA-12610] - Resources on classpath break PluginClassLoader isolation
[KAFKA-12699] - Streams no longer overrides the java default uncaught exception handler
[KAFKA-13518] - Update gson dependency
[KAFKA-13546] - Explicitly specifying default topic creation groups should not make connector fail
[KAFKA-13586] - ConfigExceptions thrown by FileConfigProvider during connector/task startup crash worker
[KAFKA-13632] - MirrorMaker 2.0 NPE and Warning "Failure to commit records" for filtered records
[KAFKA-13702] - Connect RestClient overrides response status code on request failure
[KAFKA-13725] - KIP-768 OAuth code mixes public and internal classes in same package
[KAFKA-13730] - OAuth access token validation fails if it does not contain the "sub" claim
[KAFKA-13769] - KTable FK join can miss records if an upstream non-key-changing operation changes key serializer
[KAFKA-13927] - Kafka Connect Sink Connector Success after RetriableException, no commit offset to remote.
[KAFKA-13952] - Infinite retry timeout is not working
[KAFKA-13971] - Atomicity violations caused by improper usage of ConcurrentHashMap
[KAFKA-13985] - MirrorSourceTask commitRecord throws NPE if SMT is filtering out source record
[KAFKA-13989] - Get connectors call fails when plugin removed
[KAFKA-14007] - Connect header converters are never closed
[KAFKA-14009] - Rebalance timeout should be updated when static member rejoins
[KAFKA-14012] - passing a "method" into the `Utils.closeQuietly` method cause NPE
[KAFKA-14015] - ConfigProvider with ttl fails to restart tasks
[KAFKA-14039] - Fix KRaft AlterConfigPolicy usage
[KAFKA-14079] - Source task will not commit offsets and develops memory leak if "error.tolerance" is set to "all"
[KAFKA-14099] - No REST API request logs in Kafka connect
[KAFKA-14111] - Dynamic config update fails for "password" configs in KRaft
[KAFKA-14115] - Password configs are logged in plaintext in KRaft
[KAFKA-14122] - Flaky test DynamicBrokerReconfigurationTest.testKeyStoreAlter
[KAFKA-14136] - AlterConfigs in KRaft does not generate records for unchanged values
[KAFKA-14194] - NPE in Cluster.nodeIfOnline
[KAFKA-14225] - lazy val exemptSensor Could Cause Deadlock
[KAFKA-14236] - ListGroups request produces too much Denied logs in authorizer
[KAFKA-14260] - InMemoryKeyValueStore iterator still throws ConcurrentModificationException
[KAFKA-14270] - Kafka Streams logs exception on startup
[KAFKA-14282] - RecordCollector throws exception on message processing
[KAFKA-14292] - KRaft broker controlled shutdown can be delayed indefinitely
[KAFKA-14296] - Partition leaders are not demoted during kraft controlled shutdown
[KAFKA-14300] - KRaft controller snapshot not trigger after resign
[KAFKA-14303] - Producer.send without record key and batch.size=0 goes into infinite loop
[KAFKA-14314] - MirrorSourceConnector throwing NPE during `isCycle` check
[KAFKA-14316] - NoSuchElementException in feature control iterator
[KAFKA-14320] - Upgrade Jackson for CVE fix
[KAFKA-14324] - [CVE-2018-25032] introduced by rocksdbjni:6.29.4.1
[KAFKA-14325] - NullPointer in ProcessorParameters.toString
[KAFKA-14334] - DelayedFetch purgatory not completed when appending as follower
[KAFKA-14337] - topic name with "." cannot be created after deletion
[KAFKA-14339] - Source task producers commit transactions even if offsets cannot be serialized
[KAFKA-14355] - Integer overflow occurs in kafka-producer-perf-test.sh (class ProducerProfrmance ).
[KAFKA-14358] - Users should not be able to create a regular topic name __cluster_metadata
[KAFKA-14360] - Documentation: Streams Security page has broken links
[KAFKA-14372] - RackAwareReplicaSelector should choose a replica from the isr
[KAFKA-14379] - consumer should refresh preferred read replica on update metadata
[KAFKA-14382] - StreamThreads can miss rebalance events when processing records during a rebalance
[KAFKA-14388] - NPE When Retrieving StateStore with new Processor API
[KAFKA-14406] - Double iteration of records in batches to be restored
[KAFKA-14413] - Separate MirrorMaker configurations for each connector
[KAFKA-14415] - ThreadCache is getting slower with every additional state store
[KAFKA-14422] - Consumer rebalance stuck after new static member joins a group with members not supporting static members
[KAFKA-14432] - RocksDBStore relies on finalizers to not leak memory
[KAFKA-14443] - Mirror Maker Connectors leak admin clients used for topic creation
[KAFKA-14457] - Inconsistent in quorum controller fenced broker metric
[KAFKA-14496] - Wrong Base64 encoder used by OIDC OAuthBearerLoginCallbackHandler
[KAFKA-14531] - KRaft controller time-based snapshots are too frequent
[KAFKA-14532] - Correctly handle failed fetch when partitions unassigned
[KAFKA-14557] - Missing .lock file when using metadata.log.dir
[KAFKA-14571] - ZkMetadataCache.getClusterMetadata is missing rack information in aliveNodes
[KAFKA-14612] - Topic config records written to log even when topic creation fails
[KAFKA-14618] - Off by one error in generated snapshot IDs causes misaligned fetching
[KAFKA-14623] - OAuth's HttpAccessTokenRetriever potentially leaks secrets in logging
[KAFKA-14637] - Upgrade to 3.4 from old versions (< 0.10) are failing due to incompatible meta.properties check
[KAFKA-14646] - SubscriptionWrapper is of an incompatible version (Kafka Streams 3.2.3 -> 3.3.2)
[KAFKA-14656] - Brokers rejecting LISR during ZK migration

Task

[KAFKA-13559] - The broker's ProduceResponse may be delayed for 300ms
[KAFKA-13881] - Add package.java for public package javadoc
[KAFKA-14279] - Add 3.3.1 to broker/client and stream upgrade/compatibility tests

Test

[KAFKA-12935] - Flaky Test RestoreIntegrationTest.shouldRecycleStateFromStandbyTaskPromotedToActiveTaskAndNotRestore
[KAFKA-14248] - Flaky test PlaintextAdminIntegrationTest.testCreateTopicsReturnsConfigs
[KAFKA-14498] - flaky org.apache.kafka.tools.MetadataQuorumCommandTest

Sub-task

[KAFKA-13036] - Replace EasyMock and PowerMock with Mockito for RocksDBMetricsRecorderTest
[KAFKA-13060] - Replace EasyMock and PowerMock with Mockito in WorkerGroupMemberTest.java
[KAFKA-13133] - Replace EasyMock and PowerMock with Mockito for AbstractHerderTest
[KAFKA-13158] - Replace EasyMock and PowerMock with Mockito for ConnectClusterStateImplTest
[KAFKA-13982] - Replace EasyMock and PowerMock with Mockito for WorkerConfigTransformerTest
[KAFKA-14058] - Replace EasyMock and PowerMock with Mockito in ExactlyOnceWorkerSourceTaskTest
[KAFKA-14134] - Replace EasyMock with Mockito for WorkerConnectorTest
[KAFKA-14307] - KRaft controller time based snapshots
[KAFKA-14352] - Support rack-aware partition assignment for Kafka consumers
[KAFKA-14393] - Change default size based retention for the cluster metadata partition
[KAFKA-14427] - Add support for ZK migration multi-ops transaction
[KAFKA-14428] - Add Records, RPCs and Configs for KIP-866
[KAFKA-14448] - ZK brokers register with KRaft during migration
[KAFKA-14611] - ZK broker should not send epoch during registration