Release Notes for Storm 1.2.0

JIRA issues addressed in the 1.2.0 release of Storm. Documentation for this release is available at the Apache Storm project site.

New Feature

• [STORM-2383] - [storm-hbase] Support HBase as state backend
• [STORM-2484] - Flux: support bolt+spout memory configuration
• [STORM-2648] - Kafka spout can't show acks/fails and complete latency when auto commit is enabled
• [STORM-2694] - Create a listener to handle tuple state changes of the KafkaSpout

Improvement

• [STORM-2153] - New Metrics Reporting API
• [STORM-2160] - Expose interface to MetricRegistry via TopologyContext
• [STORM-2164] - Create simple generic plugin system to register codahale reporters
• [STORM-2369] - [storm-redis] Use binary type for State management
• [STORM-2379] - [storm-elasticsearch] switch ES client to Java REST API
• [STORM-2421] - Support lists of childopts beyond just worker
• [STORM-2448] - Support running workers using older JVMs/storm versions
• [STORM-2481] - Upgrade Aether version to resolve Aether bug BUG-451566
• [STORM-2482] - Refactor the Storm auto credential plugins to be more usable
• [STORM-2491] - Missing various configuration parameters to configure the Cassandra client used by the Cassandra Bolts
• [STORM-2501] - Implement auto credential plugin for Hive
• [STORM-2512] - Change KafkaSpoutConfig in storm-kafka-client to make it work with flux
• [STORM-2519] - AbstractAutoCreds should look for configKeys in both nimbus and topology configs
• [STORM-2524] - Set Kafka client.id with storm-kafka
• [STORM-2527] - Initialize java.sql.DriverManager earlier to avoid deadlock between DriverManager static initializer and driver static initializer
• [STORM-2528] - Bump log4j version to 2.8.2
• [STORM-2548] - Simplify KafkaSpoutConfig
• [STORM-2551] - Thrift client socket timeout
• [STORM-2553] - JedisCluster does not support password
• [STORM-2598] - Add proxy server option for dependency resolver
• [STORM-2601] - the method of getting the nimbus cilent doenot accept timeout parameter
• [STORM-2616] - Document the built in metrics (just in time to replace them???)
• [STORM-2618] - Add TridentKafkaStateUpdater for storm-kafka-client
• [STORM-2650] - Add test for non-string property substitution in Flux tests
• [STORM-2657] - Update SECURITY.MD
• [STORM-2663] - Backport STORM-2558 and deprecate storm.cmd on 1.x-branch
• [STORM-2712] - accept arbitrary number of rows per tuple in storm-cassandra
• [STORM-2775] - Improve KafkaPartition Metric Names
• [STORM-2781] - Refactor storm-kafka-client KafkaSpout Processing Guarantees
• [STORM-2791] - Add support for multiple output fields to FixedTupleSpout
• [STORM-2796] - Flux: Provide means for invoking static factory methods and improve non-primitive number handling
• [STORM-2807] - Integration test should shut down topologies immediately after the test
• [STORM-2854] - Expose IEventLogger to make event logging pluggable
• [STORM-2860] - Add Kerberos support to Solr bolt
• [STORM-2862] - More flexible logging in multilang (Python, Ruby, JS)
• [STORM-2864] - Minor optimisation about trident kafka state
• [STORM-2867] - Add Consumer lag metrics to Kafka Spout
• [STORM-2877] - Introduce an option to configure pagination in Storm UI
• [STORM-2901] - Reuse ZK connection for getKeySequenceNumber
• [STORM-2914] - Remove enable.auto.commit support from storm-kafka-client
• [STORM-2917] - Check the config(nimbus.host) before using it to connect

Bug

• [STORM-1114] - Racing condition in trident zookeeper zk-node create/delete
• [STORM-2194] - ReportErrorAndDie doesn't always die
• [STORM-2231] - NULL in DisruptorQueue while multi-threaded ack
• [STORM-2315] - New kafka spout can't commit offset when ack is disabled.
• [STORM-2343] - New Kafka spout can stop emitting tuples if more than maxUncommittedOffsets tuples fail at once
• [STORM-2357] - add At-Most-Once guarantee in KafkaSpout
• [STORM-2413] - New Kafka spout is ignoring retry limit
• [STORM-2426] - First tuples fail after worker is respawn
• [STORM-2429] - non-string values in supervisor.scheduler.meta cause crash
• [STORM-2431] - the default blobstore.dir is storm.local.dir/blobs which is different from distcache-blobstore.md
• [STORM-2432] - Storm-Kafka-Client Trident Spout Seeks Incorrect Offset With UNCOMMITTED_LATEST Strategy
• [STORM-2435] - Logging in storm.js inconsistent to console.log and does not support log levels
• [STORM-2440] - Kafka outage can lead to lockup of topology
• [STORM-2449] - Iterator of Redis State may return same key multiple time, with different values
• [STORM-2450] - supervisor v2 broke ShellBolt/Spout in local mode from storm jar
• [STORM-2467] - Encoding issues in Kafka consumer
• [STORM-2478] - BlobStoreTest.testDeleteAfterFailedCreate fails on Windows
• [STORM-2486] - bin/storm launcher script can be broken if CDPATH is exported from environment
• [STORM-2488] - The UI user Must be HTTP
• [STORM-2489] - Overlap and data loss on WindowedBolt based on Duration
• [STORM-2494] - KafkaSpout does not handle CommitFailedException
• [STORM-2496] - Dependency artifacts should be uploaded to blobstore with READ permission for all
• [STORM-2498] - Download Full File link broken in 1.x branch
• [STORM-2500] - waitUntilReady in PacemakerClient cannot be invoked
• [STORM-2503] - lgtm.com alerts: bugs in equality and comparison operations
• [STORM-2505] - Kafka Spout doesn't support voids in the topic (topic compaction not supported)
• [STORM-2511] - Submitting a topology with name containing unicode getting failed.
• [STORM-2516] - WindowedBoltExecutorTest.testExecuteWithLateTupleStream is flaky
• [STORM-2517] - storm-hdfs writers can't be subclassed
• [STORM-2518] - NPE during uploading dependency artifacts with secured cluster
• [STORM-2520] - AutoHDFS should prefer cluster-wise hdfs kerberos principal to global hdfs kerberos principal
• [STORM-2521] - "storm sql" fails since '--jars' can't handle wildcard
• [STORM-2525] - Fix flaky integration tests
• [STORM-2535] - test-reset-timeout is flaky. Replace with a more reliable test.
• [STORM-2536] - storm-autocreds adds jersey 1.x to worker classpath
• [STORM-2541] - Manual partition assignment doesn't work
• [STORM-2544] - Bugs in the Kafka Spout retry logic when using manual commit
• [STORM-2546] - Kafka spout can stall / get stuck due to edge case with failing tuples
• [STORM-2549] - The fix for STORM-2343 is incomplete, and the spout can still get stuck on failed tuples
• [STORM-2552] - KafkaSpoutMessageId should be serializable
• [STORM-2555] - storm-autocreds for HBase doesn't handle impersonation
• [STORM-2557] - A bug in DisruptorQueue causing severe underestimation of queue arrival rates
• [STORM-2562] - Use stronger key size for blow fish key generator and get rid of stack trace
• [STORM-2563] - Remove the workaround to handle missing UGI.loginUserFromSubject
• [STORM-2564] - We should provide a template for storm-cluster-auth.yaml
• [STORM-2568] - 'api/vi/topology/:id/lag' returns empty json {}
• [STORM-2597] - EXT_CLASSPATH strips out directories
• [STORM-2599] - BasicContainer.getWildcardDir tries to resolve the wildcard character with Paths.get, which prevents workers from booting on Windows
• [STORM-2602] - "storm.zookeeper.topology.auth.payload" doesn't work even you set it
• [STORM-2607] - [kafka-client] Consumer group every time with lag 1
• [STORM-2608] - Out Of Range Offsets Should Be Removed From Pending Queue
• [STORM-2621] - STORM-2557 broke sojourn time estimation
• [STORM-2627] - The annotation of "storm.zookeeper.topology.auth.scheme" in Config.java is wrong
• [STORM-2639] - Kafka Spout incorrectly computes numCommittedOffsets due to voids in the topic (topic compaction)
• [STORM-2642] - Storm-kafka-client spout cannot be serialized when using manual partition assignment
• [STORM-2645] - update storm.py to be python3 compatible
• [STORM-2652] - Exception thrown in JmsSpout open method
• [STORM-2660] - The Nimbus storm-local directory is relative to the working directory of the shell executing "storm nimbus"
• [STORM-2666] - Storm-kafka-client spout can sometimes emit messages that were already committed.
• [STORM-2674] - NoNodeException when ZooKeeper tries to delete nodes
• [STORM-2675] - KafkaTridentSpoutOpaque not committing offsets to Kafka
• [STORM-2677] - consider all sampled tuples which took greater than 0 ms processing time
• [STORM-2682] - Supervisor crashes with NullPointerException
• [STORM-2690] - resurrect invocation of ISupervisor.assigned() & make Supervisor.launchDaemon() accessible
• [STORM-2692] - Load only configs specific to the topology in populateCredentials
• [STORM-2695] - BlobStore uncompress argument should be Boolean
• [STORM-2705] - DRPCSpout sleeps twice when idle
• [STORM-2706] - Nimbus stuck in exception and does not fail fast
• [STORM-2722] - JMSSpout test fails way too often
• [STORM-2724] - ExecutorService in WaterMarkEventGenerator never shutdown
• [STORM-2736] - o.a.s.b.BlobStoreUtils [ERROR] Could not update the blob with key
• [STORM-2751] - Remove AsyncLoggingContext from Supervisor
• [STORM-2756] - STORM-2548 on 1.x-branch broke setting key/value deserializers with the now deprecated setKey/setValue methods
• [STORM-2764] - HDFSBlobStore leaks file system objects
• [STORM-2769] - Fast-fail if output stream Id is null
• [STORM-2771] - Some tests are being run twice
• [STORM-2779] - NPE on shutting down WindowedBoltExecutor
• [STORM-2784] - storm-kafka-client KafkaTupleListener method onPartitionsReassigned() should be called after initialization is complete
• [STORM-2786] - Ackers leak tracking info on failure and lots of other cases.
• [STORM-2787] - storm-kafka-client KafkaSpout should set 'initialized' flag independently of processing guarantees
• [STORM-2810] - Storm-hdfs tests are leaking resources
• [STORM-2811] - Nimbus may throw NPE if the same topology is killed multiple times, and the integration test kills the same topology multiple times
• [STORM-2814] - Logviewer HTTP server should return 403 instead of 200 if the user is unauthorized
• [STORM-2815] - UI HTTP server should return 403 if the user is unauthorized
• [STORM-2825] - storm-kafka-client configuration fails with a ClassCastException if "enable.auto.commit" is present in the consumer config map, and the value is a string
• [STORM-2826] - KafkaSpoutConfig.builder doesn't set key/value deserializer properties in storm-kafka-client
• [STORM-2833] - Cached Netty Connections can have different keys for the same thing.
• [STORM-2835] - storm-kafka-client KafkaSpout can fail to remove all tuples from waitingToEmit
• [STORM-2843] - Flux: properties file not found when loading resources from classpath
• [STORM-2844] - KafkaSpout Throws IllegalStateException After Committing to Kafka When First Poll Strategy Set to EARLIEST
• [STORM-2847] - Exception thrown after rebalance IllegalArgumentException
• [STORM-2850] - ManualPartitionSubscription assigns new partitions before calling onPartitionsRevoked
• [STORM-2851] - org.apache.storm.kafka.spout.KafkaSpout.doSeekRetriableTopicPartitions sometimes throws ConcurrentModificationException
• [STORM-2853] - Deactivated topologies cause high cpu utilization
• [STORM-2855] - Travis build doesn't work after update of Ubuntu image
• [STORM-2856] - Make Storm build work on post 2017Q4 Travis Trusty image
• [STORM-2868] - Address handling activate/deactivate in multilang module files
• [STORM-2869] - KafkaSpout discards all pending records when adjusting the consumer position after a commit
• [STORM-2870] - FileBasedEventLogger leaks non-daemon ExecutorService which prevents process to be finished
• [STORM-2873] - Backpressure implentation delete ephemeral too frequently
• [STORM-2876] - Some storm-hdfs tests fail with out of memory periodically
• [STORM-2879] - Supervisor collapse continuously when there is a expired assignment for overdue storm
• [STORM-2881] - Storm-druid topologies fail with NoSuchMethodError
• [STORM-2892] - Flux test fails to parse valid PATH environment variable
• [STORM-2894] - fix some random typos in tests
• [STORM-2900] - Subject is not populated and NPE is thrown while populating credentials in nimbus.
• [STORM-2903] - Fix possible NullPointerException in AbstractAutoCreds
• [STORM-2906] - HDFS and HBase bolt on the same worker fails with GSS no valid credentials exception
• [STORM-2907] - In a secure cluster with storm-autocreds enabled storm-druid can fail with NoSuchMethodError
• [STORM-2912] - Tick tuple is being shared without resetting start time and incur side-effect to break metrics
• [STORM-2913] - STORM-2844 made autocommit and at-most-once storm-kafka-client spouts log warnings on every emit, because those modes don't commit the right metadata to Kafka
• [STORM-2918] - Upgrade Netty version
• [STORM-2942] - Remove javadoc and source jars from toollib directory in binary distribution

Documentation

• [STORM-2620] - Update the docs to better indicate the versions of java tested

Task

• [STORM-2191] - shorten classpaths in worker and LogWriter commands
• [STORM-2506] - Make Kafka Spout log its assigned partition
• [STORM-2874] - Minor style improvements to backpressure code
• [STORM-2904] - Document Metrics V2

Sub-task

• [STORM-2161] - Stop shading the codahale metrics library so that it is available to users
• [STORM-2640] - Deprecate KafkaConsumer.subscribe APIs on 1.x, and make KafkaConsumer.assign the default
• [STORM-2858] - Fix worker-launcher build