Release Notes - Kafka - Version 0.8

Sub-task

[KAFKA-176] - Fix existing perf tools
[KAFKA-237] - create/delete ZK path for a topic in an admin tool
[KAFKA-239] - Wire existing producer and consumer to use the new ZK data structure
[KAFKA-240] - implement new producer and consumer request format
[KAFKA-329] - Remove the watches/broker for new topics and partitions and change create topic admin API to send start replica state change to all brokers
[KAFKA-335] - Implement an embedded controller
[KAFKA-336] - add an admin RPC to communicate state changes between the controller and the broker
[KAFKA-337] - upgrade ZKClient to allow conditional updates in ZK
[KAFKA-338] - controller failover
[KAFKA-339] - using MultiFetch in the follower
[KAFKA-342] - revisit the broker startup procedure according to V3 design
[KAFKA-343] - revisit the become leader and become follower state change operations using V3 design
[KAFKA-344] - migration tool from 0.7 to 0.8
[KAFKA-356] - Create a generic Kafka thread class that includes basic boiler plate code of instantiating and shutting down threads cleanly
[KAFKA-362] - ZookeeperConsumerConnector needs to connect to new leader after leadership change
[KAFKA-369] - remove ZK dependency on producer
[KAFKA-458] - remove errorcode from ByteBufferMessageSet
[KAFKA-482] - Make producer to run for the entire duration of the System Test
[KAFKA-488] - Port Mirroring System Test to this python system test framework
[KAFKA-492] - Sometimes the python system test framework doesn't terminate all running processes
[KAFKA-494] - Relative paths should be used for svg URLs in dashboards html
[KAFKA-502] - Simplify setup / initialization in replication_basic_test.py
[KAFKA-503] - Support "testcase_to_run" or "testcase_to_skip"
[KAFKA-507] - Shut down ZK last to avoid hanging brokers running processes
[KAFKA-513] - Add state change log to Kafka brokers
[KAFKA-571] - Add more test cases to System Test
[KAFKA-731] - ~/ivy2/cache should be a variable in the kafka-run-class bash script
[KAFKA-780] - Reassign partitions tool produces NPE in shutdown handler
[KAFKA-814] - Controller should not throw exception when a preferred replica is already the leader for a partition
[KAFKA-843] - Re-add the release-zip sbt target

Bug

[KAFKA-15] - SBT release-zip target doesn't include bin and config directories anymore
[KAFKA-42] - Support rebalancing the partitions with replication
[KAFKA-43] - Rebalance to preferred broke with intra-cluster replication support
[KAFKA-46] - Commit thread, ReplicaFetcherThread for intra-cluster replication
[KAFKA-49] - Add acknowledgement to the produce request.
[KAFKA-81] - wrong path in bin/kafka-run-class.sh
[KAFKA-97] - SocketServer.scala refers to Handler-specific variables
[KAFKA-171] - Kafka producer should do a single write to send message sets
[KAFKA-192] - CompressionUtilTest does not run and fails when it does
[KAFKA-215] - Improve system tests for the mirroring code
[KAFKA-229] - SimpleConsumer is not logging exceptions correctly so detailed stack trace is not coming in the logs
[KAFKA-259] - Give better error message when trying to run shell scripts without having built/downloaded the jars yet
[KAFKA-295] - Bug in async producer DefaultEventHandler retry logic
[KAFKA-305] - SyncProducer does not correctly timeout
[KAFKA-306] - broker failure system test broken on replication branch
[KAFKA-351] - Refactor some new components introduced for replication
[KAFKA-352] - Throw exception to client if it makes a produce/consume request to a Kafka broker for a topic that hasn't been created
[KAFKA-367] - StringEncoder/StringDecoder use platform default character set
[KAFKA-370] - Exception "java.util.NoSuchElementException: None.get" appears inconsistently in Mirror Maker log.
[KAFKA-371] - Creating topic of empty string puts broker in a bad state
[KAFKA-376] - expose different data to fetch requests from the follower replicas and consumer clients
[KAFKA-379] - TopicCount.constructTopicCount isn't thread-safe
[KAFKA-382] - Write ordering guarantee violated
[KAFKA-385] - RequestPurgatory enhancements - expire/checkSatisfy issue; add jmx beans
[KAFKA-386] - Race condition in accessing ISR
[KAFKA-391] - Producer request and response classes should use maps
[KAFKA-396] - Mirroring system test fails on 0.8
[KAFKA-412] - deal with empty TopicData list in producer and fetch request
[KAFKA-413] - single_host_multi_brokers system test fails on laptop
[KAFKA-415] - Controller throws NoSuchElementException while marking a broker failed
[KAFKA-416] - Controller tests throw several zookeeper errors
[KAFKA-418] - NullPointerException in ConsumerFetcherManager
[KAFKA-420] - maintain HW correctly with only 1 replica in ISR
[KAFKA-422] - LazyInitProducerTest has transient test failure
[KAFKA-424] - Remove invalid mirroring arguments from kafka-server-start.sh
[KAFKA-425] - Wrong class name in performance test scripts
[KAFKA-427] - LogRecoverTest.testHWCheckpointWithFailuresSingleLogSegment has transient failure
[KAFKA-428] - need to update leaderAndISR path in ZK conditionally in ReplicaManager
[KAFKA-431] - LogCorruptionTest.testMessageSizeTooLarge fails occasionally
[KAFKA-433] - ensurePartitionLeaderOnThisBroker should not access ZK
[KAFKA-434] - IllegalMonitorStateException in ReplicaManager.makerFollower
[KAFKA-452] - follower replica may need to backoff the fetching if leader is not ready yet
[KAFKA-453] - follower replica may need to backoff the fetching if leader is not ready yet
[KAFKA-456] - ProducerSendThread calls ListBuffer.size a whole bunch. That is a O(n) operation
[KAFKA-459] - KafkaController.RequestSendThread can throw exception on broker socket
[KAFKA-460] - ControllerChannelManager needs synchronization btw shutdown and add/remove broker
[KAFKA-461] - remove support for format for magic byte 0 in 0.8
[KAFKA-463] - log.truncateTo needs to handle targetOffset smaller than the lowest offset in the log
[KAFKA-464] - KafkaController NPE in SessionExpireListener
[KAFKA-466] - Controller tests throw IllegalStateException
[KAFKA-467] - Controller based leader election failed ERROR messages in LazyInitProducerTest
[KAFKA-468] - String#getBytes is platform dependent
[KAFKA-470] - transient unit test failure in RequestPurgatoryTest
[KAFKA-471] - Transient failure in ProducerTest
[KAFKA-473] - Use getMetadata Api in ZookeeperConsumerConnector
[KAFKA-474] - support changing host/port of a broker
[KAFKA-481] - Require values in Utils.getTopic* methods to be positive
[KAFKA-490] - Check max message size on server instead of producer
[KAFKA-491] - KafkaRequestHandler needs to handle exceptions
[KAFKA-495] - Handle topic names with "/" on Kafka server
[KAFKA-497] - recover consumer during unclean leadership change
[KAFKA-499] - Refactor controller state machine
[KAFKA-500] - javaapi support for getTopoicMetaData
[KAFKA-501] - getOfffset Api needs to return different latest offset to regular and follower consumers
[KAFKA-506] - Store logical offset in log
[KAFKA-508] - split out partiondata from fetchresponse and producerrequest
[KAFKA-509] - server should shut down on encountering invalid highwatermark file
[KAFKA-510] - broker needs to know the replication factor per partition
[KAFKA-511] - offset returned in Producer response may not be correct
[KAFKA-512] - Remove checksum from ByteBufferMessageSet.iterator
[KAFKA-514] - Replication with Leader Failure Test: Log segment files checksum mismatch
[KAFKA-516] - Consider catching all exceptions in ShutdownableThread
[KAFKA-525] - newly created partitions are not added to ReplicaStateMachine
[KAFKA-528] - IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread
[KAFKA-531] - kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException
[KAFKA-537] - expose clientId and correlationId in ConsumerConfig
[KAFKA-539] - Replica.hw should be initialized to the smaller of checkedpointed HW and log end offset
[KAFKA-540] - log.append() should halt on IOException
[KAFKA-553] - confusing reference to zk.connect in config/producer.properties
[KAFKA-556] - Change MessageSet.sizeInBytes to Int
[KAFKA-557] - Replica fetch thread doesn't need to recompute message id
[KAFKA-562] - Non-failure System Test Log Segment File Checksums mismatched
[KAFKA-563] - KafkaScheduler shutdown in ZookeeperConsumerConnector should check for config.autocommit
[KAFKA-567] - Replication Data Loss in Mirror Maker Bouncing testcase
[KAFKA-573] - System Test : Leader Failure Log Segment Checksum Mismatched When request-num-acks is 1
[KAFKA-575] - Partition.makeFollower() reads broker info from ZK
[KAFKA-576] - SimpleConsumer throws UnsupportedOperationException: empty.head
[KAFKA-577] - extend DumpLogSegments to verify consistency btw data and index
[KAFKA-578] - Leader finder thread in ConsumerFetcherManager needs to handle exceptions
[KAFKA-579] - remove connection timeout in SyncProducer
[KAFKA-580] - system test testcase_0122 under replication fails due to large # of data loss
[KAFKA-584] - produce/fetch remote time metric not set correctly when num.acks = 1
[KAFKA-586] - system test configs are broken
[KAFKA-591] - Add test cases to test log size retention and more
[KAFKA-592] - Register metrics beans at kafka server startup
[KAFKA-596] - LogSegment.firstAppendTime not reset after truncate to
[KAFKA-604] - Add missing metrics in 0.8
[KAFKA-608] - getTopicMetadata does not respect producer config settings
[KAFKA-612] - move shutting down of fetcher thread out of critical path
[KAFKA-613] - MigrationTool should disable shallow iteration in the 0.7 consumer
[KAFKA-614] - DumpLogSegment offset verification is incorrect for compressed messages
[KAFKA-618] - Deadlock between leader-finder-thread and consumer-fetcher-thread during broker failure
[KAFKA-622] - Create mbeans per client
[KAFKA-625] - Improve MessageAndMetadata to expose the partition
[KAFKA-634] - ConsoleProducer compresses messages and ignores the --compress flag
[KAFKA-646] - Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level
[KAFKA-648] - Use uniform convention for naming properties keys
[KAFKA-664] - Kafka server threads die due to OOME during long running test
[KAFKA-668] - Controlled shutdown admin tool should not require controller JMX url/port to be supplied
[KAFKA-669] - Irrecoverable error on leader while rolling to a new segment
[KAFKA-673] - Broker recovery check logic is reversed
[KAFKA-680] - ApiUtils#writeShortString uses String length instead of byte length
[KAFKA-681] - Unclean shutdown testing - truncateAndStartWithNewOffset is not invoked when it is expected to
[KAFKA-684] - ConsoleProducer does not have the queue-size option
[KAFKA-690] - TopicMetadataRequest throws exception when no topics are specified
[KAFKA-691] - Fault tolerance broken with replication factor 1
[KAFKA-692] - ConsoleConsumer outputs diagnostic message to stdout instead of stderr
[KAFKA-693] - Consumer rebalance fails if no leader available for a partition and stops all fetchers
[KAFKA-695] - Broker shuts down due to attempt to read a closed index file
[KAFKA-698] - broker may expose uncommitted data to a consumer
[KAFKA-701] - ConsoleProducer does not exit correctly and fix some config properties following KAFKA-648
[KAFKA-702] - Deadlock between request handler/processor threads
[KAFKA-708] - ISR becomes empty while marking a partition offline
[KAFKA-710] - Some arguments are always set to default in ProducerPerformance
[KAFKA-713] - Update Hadoop producer for Kafka 0.8 changes
[KAFKA-726] - Add ReplicaFetcherThread name to mbean names
[KAFKA-732] - MirrorMaker with shallow.iterator.enable=true produces unreadble messages
[KAFKA-738] - correlationId is not set in FetchRequest in AbstractFetcherThread
[KAFKA-743] - PreferredReplicaLeaderElectionCommand has command line error
[KAFKA-748] - Append to index fails due to invalid offset
[KAFKA-750] - inconsistent index offset during broker startup
[KAFKA-751] - Fix windows build script - kafka-run-class.bat
[KAFKA-753] - Kafka broker shuts down while loading segments
[KAFKA-755] - standardizing json values stored in ZK
[KAFKA-756] - Processor thread blocks due to infinite loop during fetch response send
[KAFKA-757] - System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started
[KAFKA-758] - startHighWaterMarksCheckPointThread is never called
[KAFKA-767] - Message Size check should be done after assigning the offsets
[KAFKA-768] - broker should exit if hitting exceptions durin startup
[KAFKA-769] - On startup, a brokers highwatermark for every topic partition gets reset to zero
[KAFKA-770] - KafkaConfig properties should be verified in the constructor
[KAFKA-772] - System Test Transient Failure on testcase_0122
[KAFKA-776] - Changing ZK format breaks some tools
[KAFKA-779] - Standardize Zk data structures for Re-assign partitions and Preferred replication election
[KAFKA-785] - Resolve bugs in PreferredReplicaLeaderElection admin tool
[KAFKA-786] - Use "withRequiredArg" while parsing jopt options in all tools
[KAFKA-793] - Include controllerId in all requests sent by controller
[KAFKA-798] - Use biased histograms instead of uniform histograms in KafkaMetricsGroup
[KAFKA-800] - inSyncReplica in Partition needs some tweaks
[KAFKA-801] - Fix MessagesInPerSec mbean to count uncompressed message rate
[KAFKA-804] - Incorrect index in the log of a follower
[KAFKA-807] - LineMessageReader doesn't correctly parse the key separator
[KAFKA-809] - Dependency on zkclient 0.1 (redundant) prevents building in IntelliJ
[KAFKA-811] - Fix clientId in migration tool
[KAFKA-813] - Minor cleanup in Controller
[KAFKA-825] - KafkaController.isActive() needs to be synchronized
[KAFKA-826] - Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
[KAFKA-827] - improve list topic output format
[KAFKA-828] - Preferred Replica Election does not delete the admin path on controller failover
[KAFKA-830] - partition replica assignment map in the controller should be a Set
[KAFKA-832] - 0.8 Consumer prevents rebalance if consumer thread is blocked or slow
[KAFKA-840] - Controller tries to perform preferred replica election on failover before state machines have started up
[KAFKA-846] - AbstractFetcherThread should do shallow instead of deep iteration
[KAFKA-856] - Correlation id for OffsetFetch request (#2) always responds with 0
[KAFKA-861] - IndexOutOfBoundsException while fetching data from leader
[KAFKA-866] - Recover segment does shallow iteration to fix index causing inconsistencies
[KAFKA-871] - Rename ZkConfig properties
[KAFKA-872] - Socket server does not set send/recv buffer sizes
[KAFKA-880] - NoLeaderPartitionSet should be cleared before leader finder thread is started up
[KAFKA-884] - Get java.lang.NoSuchMethodError: com.yammer.metrics.core.TimerContext.stop()J when stopping kafka brokers
[KAFKA-895] - Protocol documentation is not clear about requiredAcks = 0.
[KAFKA-899] - LeaderNotAvailableException the first time a new message for a partition is processed.
[KAFKA-900] - ClosedByInterruptException when high-level consumer shutdown normally
[KAFKA-903] - [0.8.0 - windows] FATAL - [highwatermark-checkpoint-thread1] (Logging.scala:109) - Attempt to swap the new high watermark file with the old one failed
[KAFKA-905] - Logs can have same offsets causing recovery failure
[KAFKA-907] - controller needs to close socket channel to brokers on exception
[KAFKA-914] - Deadlock between initial rebalance and watcher-triggered rebalances
[KAFKA-916] - Deadlock between fetcher shutdown and handling partitions with error
[KAFKA-919] - Disabling of auto commit is ignored during consumer group rebalancing
[KAFKA-920] - zkclient jar 0.2.0 is not compatible with 0.1.0
[KAFKA-921] - Expose max lag mbean for consumers and replica fetchers
[KAFKA-927] - Integrate controlled shutdown into kafka shutdown hook
[KAFKA-937] - ConsumerFetcherThread can deadlock
[KAFKA-938] - High CPU usage when more or less idle
[KAFKA-940] - Scala match error in javaapi.Implicits
[KAFKA-941] - Add Apache 2.0 license to missing code source files
[KAFKA-942] - the version of the jar should be 0.8.0-beta1 not 0.8.0-SNAPSHOT
[KAFKA-944] - the pom output from publish and publish-local is not accurate

Improvement

[KAFKA-77] - Implement "group commit" for kafka logs
[KAFKA-100] - ProducerShell should use high-level producer instead of SyncProducer
[KAFKA-133] - Publish kafka jar to a public maven repository
[KAFKA-134] - Upgrade Kafka to sbt 0.11.3
[KAFKA-139] - cross-compile multiple Scala versions and upgrade to SBT 0.12.1
[KAFKA-155] - Support graceful Decommissioning of Broker
[KAFKA-165] - Add helper script for zkCli.sh
[KAFKA-181] - Log errors for unrecognized config options
[KAFKA-187] - Add Snappy Compression as a Codec and refactor CompressionUtil and option on startup to select what the default codec
[KAFKA-193] - use by name parameter helper for logging and trait to include lazy logging and refactor code to use the new LogHelper
[KAFKA-246] - log configuration values used
[KAFKA-253] - Refactor the async producer to have only one queue instead of one queue per broker in a Kafka cluster
[KAFKA-258] - Remove broker.id from the broker.list config in the Producer
[KAFKA-267] - Enhance ProducerPerformance to generate unique random Long value for payload
[KAFKA-271] - Modify new FetchResponse object to remove the initial offset field
[KAFKA-281] - support multiple root log directories
[KAFKA-285] - Increase maximum value of log.retention.size
[KAFKA-296] - Update Go Client to new version of Go
[KAFKA-311] - Allow KafkaLog4jAppender to take in a configurable producer.type
[KAFKA-314] - Go Client Multi-produce
[KAFKA-323] - Add the ability to use the async producer in the Log4j appender
[KAFKA-324] - enforce broker.id to be a non-negative integer
[KAFKA-325] - revisit broker config in 0.8
[KAFKA-349] - Create individual "Response" types for each kind of request and wrap them with "BoundedByteBufferSend", remove "xxResponseSend" types for all requests except "FetchRequest"
[KAFKA-365] - change copyright in NOTICE to 2012
[KAFKA-366] - add jmx beans in broker to track # bytes in consumer
[KAFKA-368] - use the pig core jar from maven instead of distributing it
[KAFKA-393] - Add constructor for message which takes both byte array offset and length
[KAFKA-408] - ProducerPerformance does not work with all producer config options
[KAFKA-437] - Unused var statement in ZookeeperConsumerConnectorTest
[KAFKA-439] - @returns was used in scala doc when it should have been @return
[KAFKA-505] - Remove errorcode from TopicMetaDataResponse
[KAFKA-548] - remove partition from ProducerRequestPartitionData and FetchResponsePartitionData
[KAFKA-581] - provides windows batch script for starting Kafka/Zookeeper
[KAFKA-632] - ProducerRequest should take ByteBufferMessageSet instead of MessageSet
[KAFKA-638] - remove ProducerShell
[KAFKA-667] - Rename .highwatermark file
[KAFKA-675] - Only bind to the interface declared in the 'hostname' config property
[KAFKA-699] - Disallow clients to set replicaId in FetchRequest
[KAFKA-733] - Fat jar option for build, or override for ivy cache location
[KAFKA-762] - Improve second replica assignment
[KAFKA-763] - Add an option to replica from the largest offset during unclean leader election
[KAFKA-812] - Support deep iteration in DumpLogSegments tool
[KAFKA-850] - add an option to show under replicated partitions in list topic command
[KAFKA-931] - make zookeeper.connect a required property

New Feature

[KAFKA-50] - kafka intra-cluster replication support
[KAFKA-188] - Support multiple data directories
[KAFKA-202] - Make the request processing in kafka asynchonous
[KAFKA-203] - Improve Kafka internal metrics
[KAFKA-235] - Add a 'log.file.age' configuration parameter to force rotation of log files after they've reached a certain age
[KAFKA-429] - Expose JMX operation to set logger level dynamically
[KAFKA-475] - Time based log segment rollout
[KAFKA-545] - Add a Performance Suite for the Log subsystem
[KAFKA-546] - Fix commit() in zk consumer for compressed messages

Task

[KAFKA-93] - Change code header to follow standard ASF source header
[KAFKA-317] - Add support for new wire protocol to Go client
[KAFKA-341] - Create a new single host system test to validate all replicas on 0.8 branch
[KAFKA-348] - rebase 0.8 branch from trunk
[KAFKA-380] - Enhance single_host_multi_brokers test with failure to trigger leader re-election in replication
[KAFKA-440] - Create a regression test framework for distributed environment testing
[KAFKA-526] - System Test should remove the top level data log directory
[KAFKA-594] - Update System Test due to new argument "--sync" in ProducerPerformance
[KAFKA-605] - System Test - Log Retention Cases should wait longer before getting the common starting offset in replica log segments
[KAFKA-688] - System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention
[KAFKA-737] - System Test : Disable shallow.iterator in Mirror Maker test cases to make compression work correctly
[KAFKA-791] - Fix validation bugs in System Test
[KAFKA-792] - Update multiple attributes in testcase_xxxx_properties.json
[KAFKA-819] - System Test : Add validation of log segment index to offset