Release Notes - Kafka - Version 2.0.0

Below is a summary of the JIRA issues addressed in the 2.0.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at http://kafka.apache.org/documentation.html.

New Feature

[KAFKA-6576] - Configurable Quota Management (KIP-257)
[KAFKA-6841] - Add support for Prefixed ACLs
[KAFKA-6886] - Externalize Secrets for Kafka Connect Configurations
[KAFKA-6935] - KIP-295 Add Streams Config for Optional Optimization

Improvement

[KAFKA-1894] - Avoid long or infinite blocking in the consumer
[KAFKA-2061] - Offer a --version flag to print the kafka version
[KAFKA-3379] - Update tools relying on old producer to use new producer.
[KAFKA-3473] - KIP-237: More Controller Health Metrics
[KAFKA-3591] - JmxTool should exit out if a provided query matches no values
[KAFKA-3806] - Adjust default values of log.retention.hours and offsets.retention.minutes
[KAFKA-3834] - Consumer should not block in poll on coordinator discovery
[KAFKA-4292] - KIP-86: Configurable SASL callback handlers
[KAFKA-4914] - Partition re-assignment tool should check types before persisting state in ZooKeeper
[KAFKA-4936] - Allow dynamic routing of output records
[KAFKA-5058] - Add a sensor to KafkaStreams to track records that have been dropped due to having a null key
[KAFKA-5327] - Console Consumer should only poll for up to max messages
[KAFKA-5370] - Replace uses of old consumer with the new consumer
[KAFKA-5523] - ReplayLogProducer not using the new Kafka consumer
[KAFKA-5529] - ConsoleProducer uses deprecated BaseProducer
[KAFKA-5674] - max.connections.per.ip minimum value to be zero to allow IP address blocking
[KAFKA-5907] - Support aggregatedJavadoc in Java 9
[KAFKA-5965] - Remove Deprecated AdminClient from Streams Resetter Tool
[KAFKA-5987] - Kafka metrics templates used in document generation should maintain order of tags
[KAFKA-6024] - Consider moving validation in KafkaConsumer ahead of call to acquireAndEnsureOpen()
[KAFKA-6028] - Improve the quota throttle communication.
[KAFKA-6057] - Users forget `--execute` in the offset reset tool
[KAFKA-6058] - KIP-222: Add "describe consumer groups" and "list consumer groups" to KafkaAdminClient
[KAFKA-6106] - Postpone normal processing of tasks within a thread until restoration of all tasks have completed
[KAFKA-6184] - report a metric of the lag between the consumer offset and the start offset of the log
[KAFKA-6376] - Improve Streams metrics for skipped records
[KAFKA-6424] - QueryableStateIntegrationTest#queryOnRebalance should accept raw text
[KAFKA-6454] - Allow timestamp manipulation in Processor API
[KAFKA-6473] - Add MockProcessorContext to public test-utils
[KAFKA-6481] - Improving performance of the function ControllerChannelManager.addUpdateMetadataRequestForBrokers
[KAFKA-6486] - TimeWindows causes unordered calls to windowed aggregation functions
[KAFKA-6514] - Add API version as a tag for the RequestsPerSec metric
[KAFKA-6515] - Add toString() method to kafka connect Field class
[KAFKA-6526] - Update controller to handle changes to unclean.leader.election.enable
[KAFKA-6535] - Set default retention ms for Streams repartition topics to Long.MAX_VALUE
[KAFKA-6538] - Enhance ByteStore exceptions with more context information
[KAFKA-6546] - Add ENDPOINT_NOT_FOUND_ON_LEADER error code for missing listener
[KAFKA-6560] - Use single-point queries than range queries for windowed aggregation operators
[KAFKA-6562] - KIP-255: OAuth Authentication via SASL/OAUTHBEARER
[KAFKA-6608] - Add TimeoutException to KafkaConsumer#position()
[KAFKA-6611] - Re-write simple benchmark in system tests with JMXTool
[KAFKA-6615] - Add scripts for DumpLogSegments
[KAFKA-6628] - RocksDBSegmentedBytesStoreTest does not cover time window serdes
[KAFKA-6630] - Speed up the processing of TopicDeletionStopReplicaResponseReceived events on the controller
[KAFKA-6640] - Improve efficiency of KafkaAdminClient.describeTopics()
[KAFKA-6657] - Add StreamsConfig prefix for different consumers
[KAFKA-6659] - Improve error message if state store is not found
[KAFKA-6670] - Implement a Scala wrapper library for Kafka Streams
[KAFKA-6673] - Segment and Stamped implement Comparable, but don't override equals.
[KAFKA-6677] - Remove EOS producer config max.in.flight.request.per.connection=1
[KAFKA-6685] - Connect deserialization log message should distinguish key from value
[KAFKA-6688] - The Trogdor coordinator should track task statuses
[KAFKA-6726] - KIP-277 - Fine Grained ACL for CreateTopics API
[KAFKA-6727] - org.apache.kafka.clients.admin.Config has broken equals and hashCode method.
[KAFKA-6730] - Simplify state store recovery
[KAFKA-6738] - Kafka Connect handling of bad data
[KAFKA-6750] - Add listener name to AuthenticationContext
[KAFKA-6760] - responses not logged properly in controller
[KAFKA-6772] - Broker should load credentials from ZK before requests are allowed
[KAFKA-6776] - Connect Rest Extension Plugin
[KAFKA-6795] - Add unit test for ReplicaAlterLogDirsThread
[KAFKA-6802] - Improve logging when topics aren't known and assignments skipped
[KAFKA-6813] - Remove deprecated APIs from KIP-120 and KIP-182 in Streams
[KAFKA-6849] - Add transformValues() method to KTable
[KAFKA-6850] - KIP-244: Add Record Header support to Kafka Streams Processor API
[KAFKA-6896] - add producer metrics exporting in KafkaStreams.java
[KAFKA-6905] - Document that Processor objects can be reused
[KAFKA-6913] - Add primitive numeric converters to Connect
[KAFKA-6930] - Update KafkaZkClient debug log
[KAFKA-6938] - Add documentation for accessing Headers on Kafka Streams Processor API
[KAFKA-6955] - Use Java AdminClient in DeleteRecordsCommand
[KAFKA-6957] - Add getter to AbstractStream class to make internalTopologyBuilder accessible outside of package
[KAFKA-6979] - Add max.block.ms to consumer for default timeout behavior
[KAFKA-6997] - Kafka run class doesn't exclude test-sources jar
[KAFKA-7000] - KafkaConsumer.position should wait for assignment metadata
[KAFKA-7009] - Mute logger for reflections.org at the warn level in system tests
[KAFKA-7050] - Decrease consumer request timeout to 30s
[KAFKA-7066] - Make Streams Runtime Error User Friendly in Case of Serialisation exception
[KAFKA-7091] - AdminClient should handle FindCoordinatorResponse errors
[KAFKA-7111] - Review the NetworkClient log level used

Bug

[KAFKA-3177] - Kafka consumer can hang when position() is called on a non-existing partition.
[KAFKA-3417] - Invalid characters in config properties not being validated?
[KAFKA-3665] - Default ssl.endpoint.identification.algorithm should be https
[KAFKA-3899] - Consumer.poll() stuck in loop if wrong credentials are supplied
[KAFKA-4041] - kafka unable to reconnect to zookeeper behind an ELB
[KAFKA-4831] - Extract WindowedSerde to public APIs
[KAFKA-4879] - KafkaConsumer.position may hang forever when deleting a topic
[KAFKA-4883] - invalid client sasl.jaas.config triggers NullPointerException
[KAFKA-5253] - TopologyTestDriver must handle streams created with patterns
[KAFKA-5540] - Deprecate and remove internal converter configs
[KAFKA-5588] - Remove deprecated new-consumer option for tools
[KAFKA-5624] - Unsafe use of expired sensors
[KAFKA-5697] - StreamThread.shutdown() need to interrupt the stream threads to break the loop
[KAFKA-5807] - Check Connector.config() and Transformation.config() returns a valid ConfigDef
[KAFKA-5919] - Adding checks on "version" field for tools using it
[KAFKA-5974] - Removed unused parameter ProcessorContext
[KAFKA-6052] - Windows: Consumers not polling when isolation.level=read_committed
[KAFKA-6054] - ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1
[KAFKA-6238] - Issues with protocol version when applying a rolling upgrade to 1.0.0
[KAFKA-6253] - Improve sink connector topic regex validation
[KAFKA-6264] - Log cleaner thread may die on legacy segment containing messages whose offsets are too large
[KAFKA-6288] - Broken symlink interrupts scanning the plugin path
[KAFKA-6292] - KafkaConsumer ran into Unknown error fetching data for topic-partition caused by integer overflow in FileLogInputStream
[KAFKA-6299] - Fix AdminClient error handling when metadata changes
[KAFKA-6351] - libs directory has duplicate javassist jars
[KAFKA-6361] - Fast leader fail over can lead to log divergence between leader and follower
[KAFKA-6378] - NullPointerException on KStream-GlobalKTable leftJoin when KeyValueMapper returns null
[KAFKA-6386] - Deprecate KafkaStreams constructor taking StreamsConfig parameter
[KAFKA-6390] - Update ZooKeeper to 3.4.12, Gradle and other minor updates
[KAFKA-6394] - Prevent misconfiguration of advertised listeners
[KAFKA-6427] - Inconsistent exception type from KafkaConsumer.position
[KAFKA-6445] - Remove deprecated metrics in 2.0
[KAFKA-6446] - KafkaProducer with transactionId endless waits when bootstrap server is down
[KAFKA-6503] - Connect: Plugin scan is very slow
[KAFKA-6513] - New Connect header support doesn't define `converter.type` property correctly
[KAFKA-6530] - Use actual first offset of messages when rolling log segment for magic v2
[KAFKA-6534] - Consumer.poll may not trigger rebalance in time when there is a task migration
[KAFKA-6552] - “entity_type” not exactly in description of kafka-configs.sh
[KAFKA-6566] - SourceTask#stop() not called after exception raised in poll()
[KAFKA-6577] - Connect standalone SASL file source and sink test fails without explanation
[KAFKA-6581] - ConsumerGroupCommand hangs if even one of the partition is unavailable
[KAFKA-6637] - if set topic config segment.ms=0 Kafka broker won't be able to start
[KAFKA-6650] - The controller should be able to handle a partially deleted topic
[KAFKA-6656] - Use non-zero status code when kafka-configs.sh fails
[KAFKA-6658] - Fix RoundTripWorkload and make k/v generation configurable
[KAFKA-6672] - ConfigCommand failing to alter configs
[KAFKA-6697] - JBOD configured broker should not die if log directory is invalid
[KAFKA-6704] - Checking hasNext from SegementIterator could throw InvalidStateStoreException
[KAFKA-6709] - broker failed to handle request due to OOM
[KAFKA-6710] - Streams integration tests hang during shutdown
[KAFKA-6711] - GlobalStateManagerImpl should not write offsets of in-memory stores in checkpoint file
[KAFKA-6724] - ConsumerPerformance resets offsets on every startup
[KAFKA-6728] - Kafka Connect Header Null Pointer Exception
[KAFKA-6729] - KTable should use user source topics if possible and not create changelog topic
[KAFKA-6731] - waitOnState waits for the wrong state instead of the target one
[KAFKA-6737] - Is Kafka imapcted by critical vulnerqbilty CVE-2018-7489
[KAFKA-6739] - Down-conversion fails for records with headers
[KAFKA-6741] - Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded
[KAFKA-6742] - TopologyTestDriver error when dealing with stores from GlobalKTable
[KAFKA-6743] - ConsumerPerformance fails to consume all messages on topics with large number of partitions
[KAFKA-6747] - kafka-streams Invalid transition attempted from state READY to state ABORTING_TRANSACTION
[KAFKA-6748] - Scheduler cannot be cancelled from Punctuator
[KAFKA-6749] - TopologyTestDriver fails when topoloy under test uses EXACTLY_ONCE
[KAFKA-6752] - Unclean leader election metric no longer working
[KAFKA-6765] - Intermittent test failure in CustomQuotaCallbackTest
[KAFKA-6775] - AbstractProcessor created in SimpleBenchmark should call super#init
[KAFKA-6778] - DescribeConfigs does not return error for non-existent topic
[KAFKA-6782] - GlobalKTable GlobalStateStore never finishes restoring when consuming aborted messages
[KAFKA-6796] - Surprising UNKNOWN_TOPIC error for produce/fetch requests to non-replicas
[KAFKA-6809] - connections-created metric does not behave as expected
[KAFKA-6825] - DEFAULT_PRODUCTION_EXCEPTION_HANDLER_CLASS_CONFIG is private
[KAFKA-6826] - Avoid range scans when forwarding values in window store aggregations
[KAFKA-6829] - Consumer should retry when encountering unknown topic or partition error
[KAFKA-6834] - log cleaner should handle the case when the size of a message set is larger than the max message size
[KAFKA-6844] - Race condition between StreamThread and GlobalStreamThread stopping
[KAFKA-6853] - ResponseMetadata calculates latency incorrectly (and therefore ZooKeeperRequestLatencyMs is incorrect)
[KAFKA-6855] - Kafka fails to start with Java 10 due to faulty Java version detection
[KAFKA-6857] - LeaderEpochFileCache.endOffsetFor() should check for UNDEFINED_EPOCH explicitly
[KAFKA-6860] - NPE when reinitializeStateStores with eos enabled
[KAFKA-6870] - Concurrency conflicts in SampledStat
[KAFKA-6871] - KStreams Scala API: incorrect Javadocs and misleading parameter name
[KAFKA-6877] - Remove completedFetch upon a failed parse if it contains no records.
[KAFKA-6878] - NPE when querying global state store not in READY state
[KAFKA-6879] - Controller deadlock following session expiration
[KAFKA-6894] - Improve error message when connecting processor with a global store
[KAFKA-6897] - Mirrormaker waits to shut down forever on produce failure with abort.on.send.failure=true
[KAFKA-6906] - Kafka Streams does not commit transactions if data is produced via wall-clock punctuation
[KAFKA-6919] - Trogdor documentation points to wrong location for trogdor.sh
[KAFKA-6927] - Broker uses significant amount of memory during down-conversion
[KAFKA-6929] - ZkData - Consumers offsets Zookeeper path is not correct
[KAFKA-6936] - Scala API Wrapper for Streams uses default serializer for table aggregate
[KAFKA-6937] - In-sync replica delayed during fetch if replica throttle is exceeded
[KAFKA-6949] - alterReplicaLogDirs() should grab partition lock when accessing log of the future replica
[KAFKA-6967] - TopologyTestDriver does not allow pre-populating state stores that have change logging
[KAFKA-6973] - setting invalid timestamp causes Kafka broker restart to fail
[KAFKA-6975] - AdminClient.deleteRecords() may cause replicas unable to fetch from beginning
[KAFKA-6981] - Missing Connector Config (errors.deadletterqueue.topic.name) kills Connect Clusters
[KAFKA-6991] - Connect Rest Extension Plugin issue with Class Loader
[KAFKA-6993] - Fix defective documentations for KStream/KTable methods
[KAFKA-7012] - Performance issue upgrading to kafka 1.0.1 or 1.1
[KAFKA-7021] - Source KTable checkpoint is not correct
[KAFKA-7023] - Kafka Streams RocksDB bulk loading config may not be honored with customized RocksDBConfigSetter
[KAFKA-7028] - super.users doesn't work with custom principals
[KAFKA-7029] - ReplicaVerificationTool should not use the deprecated SimpleConsumer
[KAFKA-7031] - Kafka Connect API module depends on Jersey
[KAFKA-7032] - The TimeUnit is neglected by KakfaConsumer#close(long, TimeUnit)
[KAFKA-7039] - DelegatingClassLoader creates plugin instance even if its not Versioned
[KAFKA-7043] - Connect isolation whitelist does not include new primitive converters (KIP-305)
[KAFKA-7047] - Connect isolation whitelist does not include SimpleHeaderConverter
[KAFKA-7048] - NPE when creating connector
[KAFKA-7055] - Kafka Streams Processor API allows you to add sinks and processors without parent
[KAFKA-7056] - Connect's new numeric converters should be in a different package
[KAFKA-7058] - ConnectSchema#equals() broken for array-typed default values
[KAFKA-7064] - "Unexpected resource type GROUP" when describing broker configs using latest admin client
[KAFKA-7067] - ConnectRestApiTest fails assertion
[KAFKA-7068] - ConfigTransformer doesn't handle null values
[KAFKA-7076] - Broker startup could be inefficient when using old message format
[KAFKA-7082] - Concurrent createTopics calls may throw NodeExistsException
[KAFKA-7104] - ReplicaFetcher thread may die because of inconsistent log start offset in fetch response
[KAFKA-7112] - StreamThread does not check for state again after pollRequests()
[KAFKA-7136] - PushHttpMetricsReporter may deadlock when processing metrics changes
[KAFKA-7168] - Broker shutdown during SSL handshake may be handled as handshake failure
[KAFKA-7182] - SASL/OAUTHBEARER client response is missing %x01 separators
[KAFKA-7185] - getMatchingAcls throws StringIndexOutOfBoundsException for empty resource name
[KAFKA-7193] - ZooKeeper client times out with localhost due to random choice of ipv4/ipv6
[KAFKA-7194] - Error deserializing assignment after rebalance

Task

[KAFKA-2983] - Remove old Scala consumer and all related code, tests, and tools
[KAFKA-4423] - Drop support for Java 7
[KAFKA-5660] - Don't throw TopologyBuilderException during runtime
[KAFKA-5677] - Remove deprecated punctuate method
[KAFKA-6769] - Upgrade jetty library version
[KAFKA-6800] - Update documentation for SASL/PLAIN and SCRAM to use callbacks
[KAFKA-6805] - Allow dynamic broker configs to be configured in ZooKeeper before starting broker
[KAFKA-6810] - Enable dynamic reconfiguration of SSL truststores
[KAFKA-6854] - Log cleaner fails with transaction markers that are deleted during clean
[KAFKA-6911] - Incorrect check for keystore/truststore dynamic update
[KAFKA-6912] - Add authorization tests for custom principal types
[KAFKA-6916] - AdminClient does not refresh metadata on broker failure
[KAFKA-6917] - Request handler deadlocks attempting to acquire group metadata lock
[KAFKA-6921] - Remove old Scala producer and all related code, tests, and tools
[KAFKA-7001] - Rename errors.allowed.max property in Connect to errors.tolerance
[KAFKA-7002] - Allow replication factor to be set via a configuration property for the Connect DLQ topic
[KAFKA-7003] - Add headers with error context in messages written to the Connect DeadLetterQueue topic

Test

[KAFKA-2951] - Additional authorization test cases
[KAFKA-5739] - Rewrite KStreamPeekTest at processor level avoiding driver usage
[KAFKA-5944] - Add unit tests for handling of authentication failures in clients
[KAFKA-6716] - discardChannel should be released in MockSelector#completeSend

Sub-task

[KAFKA-2391] - Blocking call such as position(), partitionsFor(), committed() and listTopics() should have a timeout
[KAFKA-2837] - FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure
[KAFKA-3368] - Add the Message/Record set protocol to the protocol docs
[KAFKA-3387] - Update GetOffsetShell tool to not rely on old producer.
[KAFKA-4641] - Improve test coverage of StreamsThread
[KAFKA-4651] - Improve test coverage of Stores
[KAFKA-6447] - Add Delegation Token Operations to KafkaAdminClient
[KAFKA-7005] - Remove duplicate Java Resource class.
[KAFKA-7006] - Remove duplicate Scala ResourceNameType class
[KAFKA-7007] - Use JSON for /kafka-acl-extended-changes path
[KAFKA-7010] - Rename ResourceNameType.ANY to MATCH
[KAFKA-7011] - Investigate if its possible to drop the ResourceNameType field from Java Resource class.
[KAFKA-7030] - Add configuration to disable message down-conversion