forked from apache/logging-flume
-
Notifications
You must be signed in to change notification settings - Fork 0
/
CHANGELOG
1225 lines (1135 loc) · 79.4 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release Notes - Flume - Version v1.8.0
** New Feature
* [FLUME-2171] - Add Interceptor to remove headers from event
* [FLUME-2917] - Provide netcat UDP source as alternative to TCP
* [FLUME-2993] - Support environment variables in configuration files
** Improvement
* [FLUME-1520] - Timestamp interceptor should support custom headers
* [FLUME-2945] - Bump java target version to 1.8
* [FLUME-3020] - Improve HDFSEventSink Escape Ingestion by more then 10x by not getting InetAddress on every record
* [FLUME-3025] - Expose FileChannel.open on JMX
* [FLUME-3072] - Add IP address to headers in flume log4j appender
* [FLUME-3092] - Extend the FileChannel's monitoring metrics
* [FLUME-3100] - Support arbitrary header substitution for topic of Kafka
* [FLUME-3144] - Improve Log4jAppender's performance by allowing logging collection of messages
** Bug
* [FLUME-2620] - File channel throws NullPointerException if a header value is null
* [FLUME-2752] - Flume AvroSource will leak the memory and the OOM will be happened.
* [FLUME-2812] - Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.Error: Maximum permit count exceeded
* [FLUME-2857] - Kafka Source/Channel/Sink does not restore default values when live update config
* [FLUME-2905] - NetcatSource - Socket not closed when an exception is encountered during start() leading to file descriptor leaks
* [FLUME-2991] - ExecSource command execution starts before starting the sourceCounter
* [FLUME-3027] - Kafka Channel should clear offsets map after commit
* [FLUME-3031] - sequence source should reset its counter for event body on channel exception
* [FLUME-3043] - KafkaSink SinkCallback throws NullPointerException when Log4J level is debug
* [FLUME-3046] - Kafka Sink and Source Configuration Improvements
* [FLUME-3049] - Wrapping the exception into SecurityException in UGIExecutor.execute hides the original one
* [FLUME-3057] - Build fails due to unsupported snappy-java version on ppc64le
* [FLUME-3080] - Close failure in HDFS Sink might cause data loss
* [FLUME-3083] - Taildir source can miss events if file updated in same second as file close
* [FLUME-3085] - HDFS Sink can skip flushing some BucketWriters, might lead to data loss
* [FLUME-3112] - Upgrade jackson-core library dependency
* [FLUME-3127] - Upgrade libfb303 library dependency
* [FLUME-3131] - Upgrade spring framework library dependencies
* [FLUME-3132] - Upgrade tomcat jasper library dependencies
* [FLUME-3135] - property logger in org.apache.flume.interceptor.RegexFilteringInterceptor confused
* [FLUME-3141] - Small typo found in RegexHbaseEventSerializer.java
* [FLUME-3152] - Add Flume Metric for Backup Checkpoint Errors
* [FLUME-3155] - Use batch mode in mvn to fix Travis CI error
* [FLUME-3157] - Refactor TestHDFSEventSinkOnMiniCluster to not use LeaseManager private API
* [FLUME-3173] - Upgrade joda-time
* [FLUME-3174] - HdfsSink AWS S3A authentication does not work on JDK 8
* [FLUME-3175] - Javadoc generation fails due to Java8's strict doclint
** Documentation
* [FLUME-2175] - Update Developer Guide with notes on how to upgrade Protocol Buffer version
* [FLUME-2817] - Sink for multi-agent flow example in user guide is set up incorrectly
** Wish
* [FLUME-2579] - JMS source support durable subscriptions and message listening
** Question
* [FLUME-2427] - java.lang.NoSuchMethodException and warning on HDFS (S3) sink
** Task
* [FLUME-3093] - Groundwork for version changes in root pom
* [FLUME-3154] - Add HBase client version check to AsyncHBaseSink and HBaseSink
** Test
* [FLUME-2997] - Fix flaky junit test in SpillableMemoryChannel
* [FLUME-3002] - Some tests in TestBucketWriter are flaky
Release Notes - Flume - Version v1.7.0
** New Feature
* [FLUME-2498] - Implement Taildir Source
** Improvement
* [FLUME-1899] - Make SpoolDir work with Sub-Directories
* [FLUME-2526] - Build flume by jdk 7 in default
* [FLUME-2628] - Add an optional parameter to specify the expected input text encoding for the netcat sourcef the netcat source
* [FLUME-2704] - Configurable poll delay for spooling directory source
* [FLUME-2718] - HTTP Source to support generic Stream Handler
* [FLUME-2729] - Allow pollableSource backoff times to be configurable
* [FLUME-2755] - Kafka Source reading multiple topics
* [FLUME-2781] - A Kafka Channel defined as parseAsFlumeEvent=false cannot be correctly used by a Flume source
* [FLUME-2799] - Kafka Source - Message Offset and Partition add to headers
* [FLUME-2801] - Performance improvement on TailDir source
* [FLUME-2810] - Add static Schema URL to AvroEventSerializer configuration
* [FLUME-2820] - Support New Kafka APIs
* [FLUME-2852] - Kafka Source/Sink should optionally read/write Flume records
* [FLUME-2868] - Kafka Channel partition topic by key
* [FLUME-2872] - Kafka Sink should be able to select which header as the key
* [FLUME-2875] - Allow RollingFileSink to specify a file prefix and a file extension.
* [FLUME-2909] - Bump Rat version
* [FLUME-2910] - AsyncHBaseSink - Failure callbacks should log the exception that caused them
* [FLUME-2911] - Add includePattern option in SpoolDirectorySource configuration
* [FLUME-2918] - TaildirSource is underperforming with huge parent directories
* [FLUME-2937] - Integrate checkstyle for non-test classes
* [FLUME-2941] - Integrate checkstyle for test classes
* [FLUME-2954] - make raw data appearing in log messages explicit
* [FLUME-2955] - Add file path to the header in TaildirSource
* [FLUME-2959] - Fix issues with flume-checkstyle module
* [FLUME-2982] - Add localhost escape sequence to HDFS sink
* [FLUME-2999] - Kafka channel and sink should enable statically assigned partition per event via header
* [FLUME-2821] - Flume-Kafka Source with new Consumer
* [FLUME-2822] - Flume-Kafka-Sink with new Producer
* [FLUME-2823] - Flume-Kafka-Channel with new APIs
** Bug
* [FLUME-1668] - Hdfs Sink File Rollover
* [FLUME-2132] - Exception while syncing from Flume to HDFS
* [FLUME-2143] - Flume build occasionally fails with OutOfMemoryError on Windows.
* [FLUME-2215] - ResettableFileInputStream can't support ucs-4 character
* [FLUME-2318] - SpoolingDirectory is unable to handle empty files
* [FLUME-2448] - Building flume from trunk failing with dependency error
* [FLUME-2484] - NullPointerException in Kafka Sink test
* [FLUME-2485] - Thrift Source tests fail on Oracle JDK 8
* [FLUME-2514] - Some TestFileChannelRestart tests are extremely slow
* [FLUME-2567] - Remove unneeded repository declarations in pom.xml
* [FLUME-2573] - flume-ng --conf parameter is not used when starting a flume agent
* [FLUME-2593] - ResettableFileInputStream returns negate values from read() method
* [FLUME-2619] - Spooldir source does not log channel exceptions
* [FLUME-2632] - High CPU on KafkaSink
* [FLUME-2652] - Documented transaction handling semantics incorrect
* [FLUME-2660] - Add documentation for EventValidator
* [FLUME-2672] - NPE in KafkaSourceCounter
* [FLUME-2712] - Optional channel errors slows down the Source to Main channel event rate
* [FLUME-2725] - HDFS Sink does not use configured timezone for rounding
* [FLUME-2732] - Make maximum tolerated failures before shutting down and recreating client in AsyncHbaseSink configurable
* [FLUME-2734] - Kafka Channel timeout property is overridden by default value
* [FLUME-2738] - Async HBase sink FD leak on client shutdown
* [FLUME-2746] - How to include this Flume Patch in Flume 1.5.2 ?
* [FLUME-2749] - Kerberos configuration error when using short names in multiple HDFS Sinks
* [FLUME-2751] - Upgrade Derby version to 10.11.1.1
* [FLUME-2753] - Error when specifying empty replace string in Search and Replace Interceptor
* [FLUME-2754] - Hive Sink skipping first transaction in each Batch of Hive Transactions
* [FLUME-2761] - Move Hive sink out of preview mode
* [FLUME-2763] - flume_env script should handle jvm parameters like -javaagent -agentpath -agentlib
* [FLUME-2773] - TailDirSource throws FileNotFound Exception if ~/.flume directory is not created already
* [FLUME-2797] - SyslogTcpSource uses Deprecated Class + Deprecate SyslogTcpSource
* [FLUME-2798] - Malformed Syslog messages can lead to OutOfMemoryException
* [FLUME-2804] - Hive sink - abort remaining transactions on shutdown
* [FLUME-2806] - flume-ng.ps1 Error running script to start an agent on Windows
* [FLUME-2835] - Hive Sink tests need to create table with transactional property set
* [FLUME-2841] - Upgrade commons-collections to 3.2.2
* [FLUME-2844] - ChannelCounter of SpillableMemoryChannel doesn't register actually.
* [FLUME-2881] - Windows Launch Script fails in plugins dir code
* [FLUME-2886] - Optional Channels can cause OOMs
* [FLUME-2889] - Fixes to DateTime computations
* [FLUME-2891] - Revert FLUME-2712 and FLUME-2886
* [FLUME-2897] - AsyncHBase sink NPE when Channel.getTransaction() fails
* [FLUME-2901] - Document Kerberos setup for Kafka channel
* [FLUME-2908] - NetcatSource - SocketChannel not closed when session is broken
* [FLUME-2913] - Flume classpath too long
* [FLUME-2915] - The kafka channel using new APIs will be stuck when the sink is avro sink
* [FLUME-2920] - Kafka Channel Should Not Commit Offsets When Stopping
* [FLUME-2922] - HDFSSequenceFile Should Sync Writer
* [FLUME-2923] - Bump AsyncHBase version
* [FLUME-2936] - KafkaSource tests arbitrarily fail
* [FLUME-2939] - Upgrade recursive SpoolDir to use Java7 features
* [FLUME-2948] - Docs: Fixed parameters on Replicating Channel Selector documentation example
* [FLUME-2949] - Flume fails to build on Windows
* [FLUME-2950] - ReliableSpoolingFileEventReader.rollCurrentFile is broken
* [FLUME-2952] - SyslogAgent possible NPE on stop()
* [FLUME-2972] - Handle offset migration in the new Kafka Channel
* [FLUME-2974] - Some tests are broken in TestReliableSpoolingFileEventReader and TestSpoolingFileLineReader
* [FLUME-2983] - Handle offset migration in the new Kafka Source
** Documentation
* [FLUME-2575] - FLUME-2548 brings SSLv2Hello back for Avro Sink, but UG says it is one of the protocols to exclude
* [FLUME-2713] - Document Fault Tolerant Config parameters in FlumeUserGuide
* [FLUME-2737] - Documentation for Pollable Source config parameters introduced in FLUME-2729
* [FLUME-2783] - Update Website Team page with new Committer's
* [FLUME-2890] - Typo in Twitter source warning
* [FLUME-2934] - Document new cachePatternMatching option for TaildirSource
* [FLUME-2963] - FlumeUserGuide - error in Kafka Source properties table
* [FLUME-2971] - Document secure Kafka Sink/Source/Channel setup
* [FLUME-2975] - Minor mistake in NetCat Source example in documentation
* [FLUME-2998] - Add missing configuration parameter to SequenceSource docs
** Task
* [FLUME-2935] - Bump java target version to 1.7
** Test
* [FLUME-3003] - testSourceCounter in TestSyslogUdpSource is flaky
Release Notes - Flume - Version v1.6.0
** Sub-task
* [FLUME-2250] - Add support for Kafka Source
* [FLUME-2251] - Add support for Kafka Sink
* [FLUME-2677] - Update versions in 1.6.0 branch
* [FLUME-2686] - Update KEYS file for 1.6 release
** Bug
* [FLUME-1793] - Unit test TestElasticSearchLogStashEventSerializer fails with IBM JDK
* [FLUME-1934] - Spoolingdir source exception when reading multiple zero size files
* [FLUME-2024] - Add ExecSource flush timeout to Flume User Guide
* [FLUME-2126] - Problem in elasticsearch sink when the event body is a complex field
* [FLUME-2148] - Windows : Add flume-env.ps1
* [FLUME-2214] - FileChannelIntegrityTool needs documentation in user guide
* [FLUME-2245] - HDFS files with errors unable to close
* [FLUME-2337] - export JAVA_HOME in flume-env.sh.template and increase heap size
* [FLUME-2346] - idLogFileMap in Log can lose track of file ids
* [FLUME-2404] - Default maxReadBufferBytes might cause OOM and cause scribe source exit
* [FLUME-2408] - Remove FLUME-1899 from Flume 1.5.0 change log.
* [FLUME-2416] - Use CodecPool in compressed stream to prevent leak of direct buffers
* [FLUME-2420] - HDFSEventSink#stop might throw ConcurrentModificationException
* [FLUME-2425] - FileChannel should trim data and checkpoint directories
* [FLUME-2432] - Update Kite dependency to 0.15.0
* [FLUME-2438] - Make Syslog source message body configurable
* [FLUME-2450] - Improve replay index insertion speed.
* [FLUME-2466] - Embedded agent name cannot contain spaces
* [FLUME-2470] - Kafka Sink and Source must use camel case for all configs.
* [FLUME-2472] - SyslogUtils fails when system locale is not English
* [FLUME-2475] - toLowerCase/toUpperCase used without Locale parameter to manipulate Enum
* [FLUME-2479] - Kafka property auto.commit.enable is incorrect for KafkaSource
* [FLUME-2480] - Typo in file channel exception: "Usable space exhaused"
* [FLUME-2481] - TestFileChannelEncryption fails with System locale other than en_US
* [FLUME-2482] - Race condition in File Channels' Log.removeOldLogs
* [FLUME-2486] - TestExecSource fails on some environments
* [FLUME-2487] - SyslogParser has rounding errors in timestamp parsing
* [FLUME-2488] - TestElasticSearchRestClient fails on Oracle JDK 8
* [FLUME-2489] - Upgrade Apache Curator to 2.6.0
* [FLUME-2492] - Flume's Kafka Source doesn't account time correctly
* [FLUME-2495] - Kafka Source may miss events when channel is not available
* [FLUME-2497] - TCP and UDP syslog sources parsing the timestamp incorrectly
* [FLUME-2500] - Add a channel that uses Kafka
* [FLUME-2505] - Test added in FLUME-2502 is flaky
* [FLUME-2517] - Performance issue: SimpleDateFormat constructor takes 30% of HDFSEventSink.process()
* [FLUME-2525] - flume should handle a zero byte .flumespool-main.meta file for the spooldir source
* [FLUME-2538] - TestResettableFileInputStream fails on JDK 8
* [FLUME-2556] - TestBucketPath.testDateFormatHours fails intermittently
* [FLUME-2557] - DatasetSink doesn't work with Kerberos
* [FLUME-2560] - Kafka channel entries missing for output jar generation
* [FLUME-2568] - Additional fix for TestReliableSpoolingFileEventReader
* [FLUME-2578] - Kafka source throws NPE if Kafka record has null key
* [FLUME-2586] - HDFS Sink should have an option to try rename even if close fails
* [FLUME-2592] - Specify main manifest attribute in flume tools jar
* [FLUME-2594] - Close Async HBase Client if there are large number of consecutive timeouts
* [FLUME-2595] - Add option to checkpoint on file channel shutdown
* [FLUME-2624] - Improve Hive Sink performance
* [FLUME-2626] - Remove trustmanager-type from Thrift RPC client
* [FLUME-2633] - Update Kite dependency to 1.0.0
* [FLUME-2639] - SecureThriftRpcClient client privileged calls throw IllegalAccessError
* [FLUME-2645] - ipFilter.rules property name is wrong
* [FLUME-2654] - Memory channel error message related to byteCapacity needs refining
* [FLUME-2657] - Upgrade to Hive 1.0
* [FLUME-2658] - Upgrade to Hbase 1.0
* [FLUME-2664] - Site profile fails due to dependency issues
* [FLUME-2665] - Update documentation for hdfs.closeTries based on FLUME-2586
* [FLUME-2670] - Modify dependencies to not pull hadoop/hive related jars
* [FLUME-2673] - Remove unused import in TestLog
* [FLUME-2678] - Upgrade xalan to 2.7.2 to take care of CVE-2014-0107 vulnerability
* [FLUME-2679] - Make hbase-1 the default profile
** Dependency upgrade
* [FLUME-2443] - org.apache.hadoop.fs.FSDataOutputStream.sync() is deprecated in hadoop 2.4
** Documentation
* [FLUME-1594] - Document JMX options
* [FLUME-2509] - Add description for properties in Failover Sink Processor
* [FLUME-2630] - Update documentation for Thrift Src/Sink SSL support and Kerberos support
* [FLUME-2635] - Documentation for zookeeper dynamic configuration in flume
* [FLUME-2668] - Document SecureThriftRpcClient/SecureRpcClientFactory in Flume Developer Guide
** Improvement
* [FLUME-1334] - Write an startscript for flume agents on Windows
* [FLUME-1491] - Dynamic configuration from Zookeeper watcher
* [FLUME-1521] - Document the StressSource
* [FLUME-1710] - JSONEvent.getBody should not return null
* [FLUME-1920] - Test case TestFileChannel fails when flume is built from paths containing the string "hadoop"
* [FLUME-2237] - Example config in User Guide to use same notation for Agent name
* [FLUME-2246] - event body data size can make it configurable for logger sinker
* [FLUME-2273] - ElasticSearchSink: Add handling for header substitution in indexName
* [FLUME-2373] - Enable configuration to switch thrift source serialization protocol
* [FLUME-2385] - Flume spans log file with "Spooling Directory Source runner has shutdown" messages at INFO level
* [FLUME-2401] - Optionally Compress Backup Checkpoint
* [FLUME-2439] - Update DatasetSink for Kite 0.15.0
* [FLUME-2462] - Remove use of deprecated methods in DatasetSink
* [FLUME-2463] - Add support for Hive and HBase datasets to DatasetSink
* [FLUME-2469] - DatasetSink should load dataset when needed, not at startup
* [FLUME-2499] - Include Kafka Message Key in Event Header, Updated Comments
* [FLUME-2502] - Spool source's directory listing is inefficient
* [FLUME-2558] - Update javadoc for StressSource
* [FLUME-2562] - Metrics for Flafka components
* [FLUME-2591] - Kite DatasetSink 2.0
* [FLUME-2613] - Tool/script for deleting individual message from queue
* [FLUME-2642] - Limit kerberos relogin attempt
** New Feature
* [FLUME-1734] - Create a Hive Sink based on the new Hive Streaming support
* [FLUME-2242] - Flume Sink and Source for Apache Kafka
* [FLUME-2426] - Support interceptors in the Embedded Agent
* [FLUME-2431] - Add simple regex search-and-replace interceptor
* [FLUME-2570] - Add option to not pad date fields
* [FLUME-2574] - SSL Support for Thrift Rpc
* [FLUME-2631] - End to End authentication in Flume
** Task
* [FLUME-2365] - Please create a DOAP file for your TLP
* [FLUME-2454] - Support batchSize to allow multiple events per transaction to the Kafka Sink
* [FLUME-2455] - Documentation update for Kafka Sink
* [FLUME-2523] - Document Kafka channel
* [FLUME-2612] - Update kite to 0.17.1
** Test
* [FLUME-1501] - Flume Scribe Source needs unit tests.
Release Notes - Flume - Version v1.5.2
** Bug
* [FLUME-2547] - Removing SSLv2Hello causes Java 6 clients to break
** Sub-task
* [FLUME-2548] - Enable SSLv2Hello for Avro Source and NettyAvroRpcClient
* [FLUME-2549] - Enable SSLv2Hello for HttpSource
** Improvement
* [FLUME-2551] - Add dev-support to src tarball
Release Notes - Flume - Version v1.5.1
** Bug
* [FLUME-2441] - Unit test TestHTTPSource.java failed with IBM JDK 1.7
* [FLUME-2520] - HTTP Source should be able to block a prefixed set of protocols.
* [FLUME-2533] - HTTPS tests fail on Java 6
** Improvement
* [FLUME-2511] - Allow configuration of enabled protocols in Avro source and Rpc client
Release Notes - Flume - Version v1.5.0.1
** Bug
* [FLUME-2389] - Spillable Memory Channel Example is incorrect
* [FLUME-2397] - HBase-98 compatibility
* [FLUME-2400] - Dataset Sink is not built in hbase-98 profile
Release Notes - Flume - Version v1.5.0
** New Feature
* [FLUME-1227] - Introduce some sort of SpillableChannel
* [FLUME-2056] - Allow SpoolDir to pass just the filename that is the source of an event
* [FLUME-2071] - Flume Context doesn't support float or double configuration values.
* [FLUME-2185] - Upgrade morphlines to 0.7.0
* [FLUME-2188] - flume-ng-log4jappender Support user supplied headers
* [FLUME-2225] - Elasticsearch Sink for ES HTTP API
* [FLUME-2294] - Add a sink for Kite Datasets
* [FLUME-2309] - Spooling directory should not always consume the oldest file first.
** Improvement
* [FLUME-1542] - Flume User Guide sample configurations need fixing
* [FLUME-2052] - Spooling directory source should be able to replace or ignore malformed characters
* [FLUME-2130] - Handle larger payloads via SyslogUDPSource
* [FLUME-2139] - upgrade morphline library dependency to a 0.5.0 release
* [FLUME-2154] - Reducing duplicate events caused by reset-connection-interval
* [FLUME-2155] - Improve replay time
* [FLUME-2181] - Optionally disable File Channel fsyncs
* [FLUME-2202] - AsyncHBaseSink should coalesce increments to reduce RPC roundtrips
* [FLUME-2206] - ElasticSearchSink ttl field modification to mimic Elasticsearch way of specifying TTL
* [FLUME-2207] - HDFS file suffix style index suffix in ElasticSearchSink
* [FLUME-2212] - upgrade to Morphlines-0.8.0
* [FLUME-2213] - MorphlineInterceptor should share metric registry across threads for better (aggregate) reporting
* [FLUME-2217] - Preserve priority, timestamp and hostname fields in MultiportSyslogTcp and Udp sources
* [FLUME-2231] - Add details in Flume Ganglia config in User Guide
* [FLUME-2243] - AvroSource to use TransceiverThreadFactory for Thread naming while initializing NioServerSocketChannelFactory
* [FLUME-2267] - Increase default transactionCapacity for FileChannel to 10000 from 1000
* [FLUME-2275] - Improve scalability of MorphlineInterceptor under contention
* [FLUME-2292] - Upgrade mapdb to 0.9.8
* [FLUME-2316] - Upgrade MorphlineSolrSink to kite-0.12.0
* [FLUME-2340] - Refactor to make room for Morphlines Elasticsearch Sink
* [FLUME-2343] - Add user impersonation to DatasetSink
* [FLUME-2351] - Ability to override any parameter from the configuration file
* [FLUME-2352] - HDFSCompressedDataStream should support appendBatch
** Bug
* [FLUME-1666] - Syslog source strips timestamp and hostname from log message body
* [FLUME-1679] - Add dependency on Guava to flume-ng-elasticsearch-sink POM
* [FLUME-1892] - IRC Sink NPE
* [FLUME-1951] - Remove unused future from FlumeEventQueue
* [FLUME-2007] - HDFS Sink should check if file is closed and retry if it is not.
* [FLUME-2088] - Minor typo in Flume User Guide JSON Reporting section
* [FLUME-2109] - HTTPS support in HTTP Source
* [FLUME-2110] - Scribe Source must check if category is null before inserting the headers
* [FLUME-2119] - duplicate files cause flume to enter irrecoverable state
* [FLUME-2121] - Upgrade Flume to log4j 1.2.17 to be in sync with HDFS/HBase
* [FLUME-2122] - Minor cleanups of User guide
* [FLUME-2123] - Morphline Solr sink missing short type name
* [FLUME-2124] - Upgrade Morphline Solr Sink to CDK 0.4.1
* [FLUME-2127] - JMX shutdown command for Flume
* [FLUME-2134] - AsyncHbase Sink bugfix plus tests errors on Windows
* [FLUME-2135] - Add zip to the build distribution for Windows support
* [FLUME-2136] - Windows - Fix intermitent test failure in TestMonitoredCounterGroup.java
* [FLUME-2137] - Fix StagedInstall.java to invoke the correct startup script on Windows
* [FLUME-2142] - HTTPS tests for http source
* [FLUME-2145] - TestCheckpointRebuilder.testFastReplay fails on Windows due to checkpoint file being memory mapped
* [FLUME-2151] - Windows: Update TestExecSource to use native commands on Windows
* [FLUME-2152] - Flume user guide says "Ganglia support" where it should say "JSON support"
* [FLUME-2156] - Unregister then re-register MonitoredCounterGroup JMX MBeans on reconfigure
* [FLUME-2157] - Spool directory source does not shut down correctly when Flume is reconfigured
* [FLUME-2159] - Sporadic failures in TestNettyAvroRpcClient.spinThreadsCrazily()
* [FLUME-2161] - Flume does not support spaces in -X java-opt command line args
* [FLUME-2172] - Update protocol buffer from 2.4.1 to 2.5.0
* [FLUME-2176] - SpoolDir Source, get 'File has changed' exception but actually there is no change on the file
* [FLUME-2182] - Spooling Directory Source will not ingest data completely when a wide character appears at the edge of a buffer
* [FLUME-2184] - flume-ng-morphline-solr-sink Build failing due to incorrect hadoop-common dependency declaration
* [FLUME-2191] - HDFS Minicluster tests failing after protobuf upgrade.
* [FLUME-2192] - AbstractSinkProcessor stop incorrectly calls start
* [FLUME-2198] - Avro Source should disable itself if ipFilterRules contains invalid rules
* [FLUME-2199] - Flume builds with new version require mvn install before site can be generated
* [FLUME-2200] - HTTP Source should be able to use "port" parameter if SSL is enabled
* [FLUME-2208] - Jetty's default SocketSelector leaks File descriptors
* [FLUME-2209] - AsyncHBaseSink will never recover if the column family does not exists for the first start
* [FLUME-2210] - UnresolvedAddressException when using multiple hostNames in Elasticsearch sink configuration
* [FLUME-2220] - ElasticSearch sink - duplicate fields in indexed document
* [FLUME-2229] - Backoff period gets reset too often in OrderSelector
* [FLUME-2233] - MemoryChannel lock contention on every put due to bytesRemaining Semaphore
* [FLUME-2235] - idleFuture should be cancelled at the start of append
* [FLUME-2238] - Provide option to configure worker threads in NettyAvroRpcClient
* [FLUME-2239] - Clarify File Channel's dataDirs setting in User Guide
* [FLUME-2252] - Add null check before closing table in HbaseSink
* [FLUME-2253] - Please delete old releases from mirroring system
* [FLUME-2255] - Spooling Directory Source cannot handle channel exceptions
* [FLUME-2259] - transaction closure not happening for all the scenario in hbasesink
* [FLUME-2262] - Log4j Appender should use timeStamp field not getTimestamp
* [FLUME-2263] - Bump Hadoop 2 version to 2.3.0
* [FLUME-2264] - Log4j Appender + Avro Reflection on string results in an invalid avro schema
* [FLUME-2265] - Closed bucket writers should be removed from sfwriters map
* [FLUME-2266] - Update Morphline Sink to kite-0.10.0
* [FLUME-2270] - Twitter Source Documentation Does not load properly
* [FLUME-2272] - Getting start page returning 503 error
* [FLUME-2283] - Spool Dir source must check interrupt flag before writing to channel
* [FLUME-2289] - Disable maxUnderReplication test which is extremely flakey
* [FLUME-2301] - Update HBaseSink tests to reflect sink returning backoff only on empty batches
* [FLUME-2302] - TestHDFS Sink fails with Can't get Kerberos realm
* [FLUME-2303] - HBaseSink tests can fail based on order of execution
* [FLUME-2304] - DatasetSink test fails unexpectedly
* [FLUME-2305] - BucketWriter#close must cancel idleFuture
* [FLUME-2307] - Remove Log writetimeout
* [FLUME-2311] - Use standard way of finding queue/topic
* [FLUME-2312] - Add utility for adorning HTTP contexts in Jetty
* [FLUME-2314] - Upgrade to Mapdb 0.9.9
* [FLUME-2320] - Deadlock in DatasetSink
* [FLUME-2323] - Morphline sink must increment eventDrainAttemptCount when it takes event from channel
* [FLUME-2324] - Support writing to multiple HBase clusters using HBaseSink
* [FLUME-2325] - BucketWriter might throw BucketClosedException incorrectly
* [FLUME-2328] - FileChannel Dual Checkpoint Backup Thread not released on Application stop
* [FLUME-2329] - Add an alias for the Morphline Solr Sink
* [FLUME-2330] - Remove the MorphlineHandlerImpl configuration option from MorphlineSink
* [FLUME-2334] - Upgrade Asynchbase to 1.5.0 as this include hbase 0.96.x support
* [FLUME-2335] - TestHBaseSink#testWithoutConfigurationObject() must delete the table at the end of the test
* [FLUME-2336] - HBase tests that pass in ZK configs must use a new context object
* [FLUME-2338] - Support coalescing increments in HBaseSink
* [FLUME-2345] - Update to Kite 0.12.0 dependency
* [FLUME-2347] - Add FLUME_JAVA_OPTS which allows users to inject java properties from cmd line
* [FLUME-2350] - Consume Order tests need to space out file creation
* [FLUME-2357] - HDFS sink should retry closing files that previously had close errors
* [FLUME-2381] - Upgrade Hadoop version in Hadoop 2 profile to 2.4.0
* [FLUME-2379] - Flume's pom files are invalid and not parseable by non-maven build tools
** Documentation
* [FLUME-1223] - Userguide improvement rolling file sink
* [FLUME-1678] - Incorrect documentation for HBase sink
* [FLUME-1851] - User Guide grammar mistake
* [FLUME-2064] - Typo/Grammar in flume main user doc under Scribe
* [FLUME-2065] - Regex Extractor Interceptor config agent name inconsistent with rest of docs
* [FLUME-2183] - Add "Other Resources" page to wiki
* [FLUME-2278] - Incorrect documentation for write-timeout of File Channel
* [FLUME-2319] - Incorrect property name of Ganglia Reporting documentation
Release Notes - Flume - Version v1.4.0
** New Feature
* [FLUME-924] - Implement a JMS source for Flume NG
* [FLUME-997] - Support secure transport mechanism
* [FLUME-1170] - Create a framework for transaction listeners
* [FLUME-1502] - Support for running simple configurations embedded in host process
* [FLUME-1516] - FileChannel Write Dual Checkpoints to avoid replays
* [FLUME-1632] - Persist progress on each file in file spooling client/source
* [FLUME-1735] - Add support for a plugins.d directory
* [FLUME-1894] - Implement Thrift RPC
* [FLUME-1917] - FileChannel group commit (coalesce fsync)
* [FLUME-2004] - Need to capture metrics on the Flume exec source such as events received, rejected, etc.
* [FLUME-2010] - Support Avro records in Log4jAppender and the HDFS Sink
* [FLUME-2048] - Avro container file deserializer
* [FLUME-2070] - Add a Flume Morphline Solr Sink
** Improvement
* [FLUME-1076] - Sink batch sizes vary wildy
* [FLUME-1100] - HDFSWriterFactory and HDFSFormatterFactory should allow extension
* [FLUME-1571] - Channels should check for positive capacity and transaction capacity values
* [FLUME-1586] - File Channel should support verifying integrity of individual events.
* [FLUME-1652] - Logutils.getLogs could NPE
* [FLUME-1661] - ExecSource cannot execute complex *nix commands
* [FLUME-1677] - Add File-channel dependency to flume-ng-node's pom.xml
* [FLUME-1699] - Make the rename of the meta file platform neutral
* [FLUME-1702] - HDFSEventSink should write to a hidden file as opposed to a .tmp file
* [FLUME-1740] - Remove contrib/ directory from Flume NG
* [FLUME-1745] - FlumeConfiguration Eats Exceptions
* [FLUME-1756] - Avro client should be able to use load balancing RPC
* [FLUME-1757] - Improve configuration of hbase serializers
* [FLUME-1762] - File Channel should recover automatically if the checkpoint is incomplete or bad by deleting the contents of the checkpoint directory
* [FLUME-1768] - Multiplexing channel selector should allow optional-only channels
* [FLUME-1769] - Replicating channel selector should support optional channels
* [FLUME-1770] - Flume should have serializer which supports serializer the headers to a simple string
* [FLUME-1777] - AbstractSource does not provide enough implementation for sub-classes
* [FLUME-1790] - Commands in EncryptionTestUtils comments require high encryption pack to be installed
* [FLUME-1794] - FileChannel check for full disks in the background
* [FLUME-1800] - Docs for spooling source durability changes
* [FLUME-1808] - ElasticSearchSink is missing log4.properties
* [FLUME-1821] - Support configuration of hbase instances to be used in AsyncHBaseSink from flume config
* [FLUME-1847] - NPE in SourceConfiguration
* [FLUME-1848] - HDFSDataStream logger is actually for a sequence file
* [FLUME-1855] - Sequence gen source should be able to stop after a fixed number of events
* [FLUME-1864] - Allow hdfs idle callback to clean up closed bucket writers
* [FLUME-1874] - Ship with log4j.properties file that has a reliable time based rolling policy
* [FLUME-1876] - Document hadoop dependency of FileChannel when used with EmbeddedAgent
* [FLUME-1878] - FileChannel replay should print status every 10000 events
* [FLUME-1886] - Add a JMS enum type to SourceType so that users don't need to enter FQCN for JMSSource
* [FLUME-1889] - Add HBASE and ASYNC_HBASE enum types to SinkType so that users don't need to enter FQCNs
* [FLUME-1906] - Ability to disable WAL for put operation in HBaseSink
* [FLUME-1915] - Enhance NettyAvroRpcClient and the use of NettyServer to optionally use compression
* [FLUME-1926] - Optionally timeout Avro Sink Rpc Clients to avoid stickiness
* [FLUME-1940] - Log a snapshot of Flume metrics on shutdown
* [FLUME-1945] - HBase Serializer allow key from regular expression group
* [FLUME-1976] - JMS Source document should provide instruction on JMS implementation jars
* [FLUME-1977] - JMS Source connectionFactory property is not documented
* [FLUME-1992] - ElasticSearch dependency is marked optional
* [FLUME-1994] - Add ELASTICSEARCH enum type to SinkType to eliminate need for FQCN in agent configuration files
* [FLUME-2005] - Minor improvements to Flume assembly config
* [FLUME-2008] - it would be very convenient to have a fat jar of flume-ng-log4jappender
* [FLUME-2009] - Flume project throws error when imported into Eclipse IDE (Juno)
* [FLUME-2013] - Parametrize java source and target version in the main pom file
* [FLUME-2015] - ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
* [FLUME-2046] - Typo in HBaseSink java doc
* [FLUME-2049] - Compile ElasticSearchSink with elasticsearch 0.90
* [FLUME-2062] - make it possible for HBase sink to deposit event headers into corresponding column qualifiers
* [FLUME-2063] - Add Configurable charset to RegexHbaseEventSerializer
* [FLUME-2076] - JMX metrics support for HTTP Source
* [FLUME-2093] - binary tarball that is created by flume's assembly shouldn't contain sources
* [FLUME-2100] - Increase default batchSize of Morphline Solr Sink
* [FLUME-2105] - Add docs for MorphlineSolrSink
** Bug
* [FLUME-1110] - HDFS Sink throws IllegalStateException when flume-daemon shuts down
* [FLUME-1153] - flume-ng script is missing some agent options in help output
* [FLUME-1175] - RollingFileSink complains of Bad File Descriptor upon a reconfig event
* [FLUME-1262] - Move doc generation to a different profile
* [FLUME-1285] - FileChannel has a dependency on Hadoop IO classes
* [FLUME-1296] - Lifecycle supervisor should check if the monitor service is still running before supervising
* [FLUME-1511] - Scribe-source doesn't handle zero message request correctly.
* [FLUME-1676] - ExecSource should provide a configurable charset
* [FLUME-1688] - Bump AsyncHBase version to 1.4.1
* [FLUME-1709] - HDFS CompressedDataStream doesn't support serializer parameter
* [FLUME-1720] - LICENSE file contain entry for protobuf-<version>.jar, however proper artifact name is protobuf-java-<version>.jar
* [FLUME-1731] - SpoolableDirectorySource should have configurable support for deleting files it has already completed instead of renaming
* [FLUME-1741] - ElasticSearch tests leave directory data/elasticsearch/nodes/ lying around
* [FLUME-1748] - HDFS Sink should check if the thread is interrupted before performing any HDFS operations
* [FLUME-1755] - Load balancing RPC client has issues with downed hosts
* [FLUME-1766] - AvroSource throws confusing exception when configured without a port
* [FLUME-1772] - AbstractConfigurationProvider should remove component which throws exception from configure method.
* [FLUME-1773] - File Channel worker thread should not be daemon
* [FLUME-1774] - EventBackingStoreFactory error message asks user to delete checkpoint which is now done automatically
* [FLUME-1775] - FileChannel Log Background worker should catch Throwable
* [FLUME-1776] - Several modules require commons-lang but do not declare this in the pom
* [FLUME-1778] - Upgrade Flume to use Avro 1.7.3
* [FLUME-1784] - JMSource fix minor documentation problem and parameter name
* [FLUME-1788] - Flume Thrift source can fail intermittently because of a race condition in Thrift server implementation on some Linux systems
* [FLUME-1789] - Unit tests TestJCEFileKeyProvider and TestFileChannelEncryption fail with IBM JDK and flume-1.3.0
* [FLUME-1795] - Flume thrift legacy source does not have proper logging configured
* [FLUME-1797] - TestFlumeConfiguration is in com.apache.flume.conf namespace.
* [FLUME-1799] - Generated source tarball is missing flume-ng-embedded-agent
* [FLUME-1802] - Missing parameter --conf in example of the Flume User Guide
* [FLUME-1803] - Generated dist tarball is missing flume-ng-embedded-agent
* [FLUME-1804] - JMS source not included in binary dist
* [FLUME-1805] - Embedded agent deps should be specified in dependencyManagement section of pom
* [FLUME-1818] - Support various layouts in log4jappender
* [FLUME-1819] - ExecSource don't flush the cache if there is no input entries
* [FLUME-1820] - Should not be possible for RPC client to block indefinitely on close()
* [FLUME-1822] - Update javadoc for FlumeConfiguration
* [FLUME-1823] - LoadBalancingRpcClient method must throw exception if it is called after close is called.
* [FLUME-1824] - Inflights can complete successfully even if checkpoint fails
* [FLUME-1828] - ResettableInputStream should support seek()
* [FLUME-1834] - Userguide on trunk is missing some memory channel props
* [FLUME-1835] - Flume User Guide has wrong prop in Load Balancing Sink Selector
* [FLUME-1844] - HDFSEventSink should have option to use RawLocalFileSystem
* [FLUME-1845] - Document plugin.d directory structure
* [FLUME-1849] - Embedded Agent doesn't shutdown supervisor
* [FLUME-1852] - Issues with EmbeddedAgentConfiguration
* [FLUME-1854] - Application class can deadlock if stopped immediately after start
* [FLUME-1863] - EmbeddedAgent pom must pull in file channel
* [FLUME-1865] - Rename the Sequence File formatters to Serializer to be consistent with the rest of Flume
* [FLUME-1866] - ChannelProcessor is not logging ChannelExceptions.
* [FLUME-1867] - There's no option to set hostname for HTTPSource
* [FLUME-1868] - FlumeUserGuide mentions wrong FQCN for JSONHandler
* [FLUME-1869] - Request to add "HTTP" source type to SourceType.java
* [FLUME-1870] - Flume sends non-numeric values with type as float to Ganglia causing ganglia to crash
* [FLUME-1872] - SpoolingDirectorySource doesn't delete tracker file when deletePolicy is "immediate"
* [FLUME-1879] - Secure HBase documentation
* [FLUME-1880] - Double-logging of created HDFS files
* [FLUME-1882] - Allow case-insensitive deserializer value for SpoolDirectorySource
* [FLUME-1890] - Flume should set the hbase keytab and principal in HBase conf object.
* [FLUME-1891] - Fast replay runs even when checkpoint exists.
* [FLUME-1893] - File Channel could miss possible checkpoint corruption
* [FLUME-1911] - Add deprecation back to the legacy thrift code
* [FLUME-1916] - HDFS sink should poll for # of active replicas. If less than required, roll the file.
* [FLUME-1918] - File Channel cannot handle capacity of more than 500 Million events
* [FLUME-1922] - HDFS Sink should optionally insert the timestamp at the sink
* [FLUME-1924] - Bug in serializer context parsing in RollingFileSink
* [FLUME-1925] - HDFS timeouts should not starve other threads
* [FLUME-1929] - CheckpointRebuilder main method does not work
* [FLUME-1930] - Inflights should clean up executors on close.
* [FLUME-1931] - HDFS Sink has a commons-lang dependency which is missing in pom
* [FLUME-1932] - no-reload-conf command line param does not work
* [FLUME-1937] - Issue with maxUnderReplication in HDFS sink
* [FLUME-1939] - FlumeEventQueue must check if file is open before setting the length of the file
* [FLUME-1943] - ExecSource tests failing on Jenkins
* [FLUME-1948] - plugins.d directory(ies) should be separately overridable, independent of FLUME_HOME
* [FLUME-1949] - Documentation for sink processor lists incorrect default
* [FLUME-1955] - fileSuffix does not work with compressed streams
* [FLUME-1958] - Remove attlasian-ide-plugin.xml from the repo
* [FLUME-1964] - hdfs sink depends on commons-io but does not specify it in the pom
* [FLUME-1965] - Thrift sink alias doesn't exist
* [FLUME-1969] - Update user Guide to explain the purpose of minimumRequiredSpace setting for FileChannel
* [FLUME-1974] - Thrift compatibility issue with hbase-0.92
* [FLUME-1975] - Use TThreadedSelectServer in ThriftSource if it is available
* [FLUME-1980] - Log4jAppender should optionally drop events if append fails
* [FLUME-1981] - Rpc client expiration can be done in a more thread-safe way
* [FLUME-1986] - doTestInflightCorrupts should not commit transactions
* [FLUME-1993] - On Windows, when using the spooling directory source, there is a file sharing violation when trying to delete tracker file
* [FLUME-2002] - Flume RPC Client creates 2 threads per each log attempt if the remote flume agent goes down
* [FLUME-2011] - "mvn test" fails
* [FLUME-2012] - Two tests fail on Mac OS (saying they fail to load native library) with Java 7
* [FLUME-2014] - Race condition when using local timestamp with BucketPath
* [FLUME-2023] - Flume must login to secure HBase before creating the HTable instance
* [FLUME-2025] - ThriftSource throws NPE in stop() if start() failed because socket open failed or if thrift server instance creation threw.
* [FLUME-2026] - TestHTTPSource should use any available port rather than a hardcoded port number
* [FLUME-2027] - Check for default replication fails on federated cluster in hdfs sink
* [FLUME-2032] - HDFSEventSink doesn't work in Windows
* [FLUME-2036] - Make hostname optional for HTTPSource
* [FLUME-2042] - log4jappender timeout should be configurable
* [FLUME-2043] - JMS Source removed on failure to create configuration
* [FLUME-2044] - HDFS Sink impersonation fails after the first file
* [FLUME-2051] - Surefire 2.12 cannot run a single test on Windows. Upgrade to 2.12.3
* [FLUME-2054] - Support Version Info on Windows and fix failure of TestVersionInfo
* [FLUME-2057] - Failures in FileChannel's TestEventQueueBackingStoreFactory on Windows
* [FLUME-2060] - Failure in TestLog.testReplaySucceedsWithUnusedEmptyLogMetaDataFastReplay test on Windows
* [FLUME-2072] - JMX metrics support for HBase Sink
* [FLUME-2081] - JMX metrics support for SpoolDir
* [FLUME-2082] - JMX support for Seq Generator Source
* [FLUME-2083] - Avro Source should not start if SSL is enabled and keystore cannot be opened
* [FLUME-2098] - Make Solr sink depend on the CDK version of morphlines
** Documentation
* [FLUME-1621] - Document new MemoryChannel parameters in Flume User Guide
* [FLUME-1910] - Add thrift RPC documentation
* [FLUME-1953] - Fix dev guide error that says sink can read from multiple channels
* [FLUME-1962] - Document proper specification of lzo codec as lzop in Flume User Guide
* [FLUME-1979] - Wrong propname for connection reset interval in avro sink
* [FLUME-2030] - Documentation of Configuration Changes JMSSource, HBaseSink, AsyncHBaseSink and ElasticSearchSink
** Task
* [FLUME-1686] - Exclude target directories & Eclipse files from rat checks
* [FLUME-2094] - Remove the deprecated - Recoverable Memory Channel
** Sub-task
* [FLUME-1626] - Support Hbase security in Hbase sink
* [FLUME-1630] - Flume configuration code could be improved
* [FLUME-1674] - Documentation / Wiki
* [FLUME-1896] - Implement Thrift RpcClient
* [FLUME-1897] - Implement Thrift Sink
* [FLUME-1898] - Implement Thrift Source
* [FLUME-2102] - Update LICENSE file for Flume 1.4.0
----
Release Notes - Flume - Version v1.3.0
** New Feature
* [FLUME-1199] - Add HTTP Post Source
* [FLUME-1371] - ElasticSearch Sink
* [FLUME-1382] - Flume adopt message from existing local Scribe
* [FLUME-1385] - Add a multiport syslog source
* [FLUME-1424] - File Channel should support encryption
* [FLUME-1425] - Create a SpoolDirectory Source and Client
* [FLUME-1488] - Load Balancing RPC client should support exponential backoff of failed nodes
* [FLUME-1537] - Dump RollingFileSink's counter status when agent stops
* [FLUME-1657] - Regex Extractor Interceptor
** Improvement
* [FLUME-946] - Allow multiplexing channel selector to specify optional channels.
* [FLUME-1337] - Add IDEA files to .gitignore
* [FLUME-1358] - Add a regex-based filtering interceptor
* [FLUME-1383] - Improve various log messages in FileChannel and HDFSSink
* [FLUME-1408] - ScheduledExecutorService does not log uncaught Throwables, we should log them
* [FLUME-1418] - Improvement for Log4j configuration
* [FLUME-1419] - Using system time if 'timestamp' property is absent in event header
* [FLUME-1434] - Distinguish background worker with channel name
* [FLUME-1480] - Replace object descriptor with detailed component type plus name
* [FLUME-1487] - FileChannel format needs to be extensible
* [FLUME-1490] - Option to limit number of events sent in Stress source
* [FLUME-1496] - TestFileChannel is bloated
* [FLUME-1507] - Have "Topology Design Considerations" in User Guide
* [FLUME-1509] - Flume HDFS sink should allow for the use of different timezones when resolving sink paths
* [FLUME-1519] - LifecycleController prints tons of DEBUG messages
* [FLUME-1523] - Allow -X java opts to be passed to the agent on the command line
* [FLUME-1526] - LogFile log message is scary when no harm has been done
* [FLUME-1531] - Flume User Guide should provide more details on configuring the timestamp interceptor
* [FLUME-1535] - Ability to specify the capacity of MemoryChannel in bytes
* [FLUME-1536] - Support for batch size in StressSource
* [FLUME-1538] - Channels should expose channel fill ratio through JMX
* [FLUME-1543] - TestFileChannel should be factored into many tests
* [FLUME-1546] - File channel encryption: trim() passwords and warn user if he doesn't have JCE policy file
* [FLUME-1548] - Build dies due to older protocol buffers compiler
* [FLUME-1550] - Use maven-antrun-plugin to save version
* [FLUME-1554] - FileChannel fails to build on machines with old protocol buffer compiler
* [FLUME-1556] - It would be nice if NullSink logged the number of event processed every 10K or so
* [FLUME-1560] - TestFileChannel* tests which fill up the channel should use larger batch size than 1
* [FLUME-1563] - FileChannel Encryption KeyProvider configuration properties should be more consistent
* [FLUME-1564] - FileChannel log file creation could be clarified and tested
* [FLUME-1569] - MemoryChannel uses an Integer as a lock
* [FLUME-1575] - FIleChannel Encryption should disallow a null key
* [FLUME-1603] - FileChannel capacity reached message is unclear
* [FLUME-1607] - FileChannel We should use a regex as opposed to simple filename filter when finding logs
* [FLUME-1609] - FileChannel detecting when the underlying file systems are full could provide cleaner error recovery
* [FLUME-1631] - Retire hdfs.txnEventMax in HDFS sink
* [FLUME-1645] - add hdfs.fileSuffix property to HDFSEventSink
* [FLUME-1660] - Close "idle" hdfs handles
* [FLUME-1675] - Ignore netbeans config files in rat & git
* [FLUME-1681] - Disable empty-file unit test for Spooling File Reader
* [FLUME-1684] - Re-enable empty file unit test
* [FLUME-1689] - BodyTextSerializer should allow an option not to add a newline to each serialized event
* [FLUME-1692] - MultiportSyslogTCPSource user documentation and nickname
* [FLUME-1707] - Update FlumeDevGuide
* [FLUME-1706] - Website for 1.3 fails to build
* [FLUME-1698] - Update RELEASE-NOTES
* [FLUME-1711] - Update Flume User Guide
* [FLUME-1713] - Netcat source should allow for *not* returning "OK" upon receipt of each message.
* [FLUME-1740] - Remove contrib/ directory from Flume NG
* [FLUME-1750] - File spooling client uses -D as command line option
* [FLUME-1751] - User Guide Examples for File Channel encryption are broken in 1.3 rc5
* [FLUME-1749] - .gitignore and elipse related files should not be included in source tarball
* [FLUME-1752] - Update CHANGELOG for flume 1.3.0 rc6 to include latest changes
** Bug
* [FLUME-1208] - Hbase sink should be built only in Hadoop-1.0 profile
* [FLUME-1256] - OutofMemory erros in Flume build
* [FLUME-1259] - Flume throws OutOfMemory error when sending data from netcat to avro source (negative test case)
* [FLUME-1276] - Create a static header interceptor
* [FLUME-1277] - Error parsing Syslog rfc 3164 messages with null values
* [FLUME-1310] - Make Asynch hbase sink test work with other versions of Hbase
* [FLUME-1354] - Update docs to show that recoverable memory channel is deprecated
* [FLUME-1362] - Port retrying in TestThriftLegacySource not working
* [FLUME-1363] - flume-ng-node - TestNetcatSource doesn't try multiple ports
* [FLUME-1364] - Document the necessity of the timestamp header when using time-related escapes for hdfs sink paths
* [FLUME-1373] - Remove hardcoded file separator in HDFSEventSink
* [FLUME-1374] - Support ganglia reporting
* [FLUME-1376] - StaticInterceptor doc update
* [FLUME-1377] - ChannelProcessor clobbers exception with NPE if Channel.getTransaction() throws
* [FLUME-1389] - Flume gives opaque error if interceptor type not specified
* [FLUME-1391] - Use sync() instead of syncFs() in HDFS Sink to be compatible with hadoop 0.20.2
* [FLUME-1392] - Inactive channels get added to source channels list causing NPE
* [FLUME-1398] - Improve concurrency for async hbase sink
* [FLUME-1412] - Commons collections is used in file channel - even though it is not in pom.xml
* [FLUME-1414] - VersionInfo should not create a log instance
* [FLUME-1416] - Version Info should have hardcoded git repo address
* [FLUME-1417] - File Channel checkpoint can be bad leading to the channel being unable to start.
* [FLUME-1420] - Exception should be thrown if we cannot instaniate an EventSerializer
* [FLUME-1421] - PollableSourceRunner does not name it's thread
* [FLUME-1422] - Fix "BarSource" Class Signature in Flume Developer Guide
* [FLUME-1426] - FileChannel Replay could be faster
* [FLUME-1428] - File Channel should not consider a file as inactive until all takes are committed.
* [FLUME-1432] - FileChannel should replay logs in the order they were written
* [FLUME-1437] - Checkpoint can miss pending takes.
* [FLUME-1470] - Syslog source does not parse facility correctly
* [FLUME-1479] - Multiple Sinks can connect to single Channel
* [FLUME-1482] - Flume should support exposing metrics via HTTP in JSON/some other web service format.
* [FLUME-1498] - File channel - Log updates and queue updates should be atomic
* [FLUME-1500] - Upgrade flume to use latest version of Avro - v1.7
* [FLUME-1504] - Test file channel times out randomly
* [FLUME-1506] - Child poms pull in specific versions of packages not in top level pom
* [FLUME-1512] - File Channel should not stop during a checkpoint.
* [FLUME-1513] - File Channel log close() method should not be synchronized
* [FLUME-1514] - Log4jAppender should also have flume-ng-configuration in the pom
* [FLUME-1515] - Fix flume-1.3.0 branch test failures on ASF Jenkins.
* [FLUME-1524] - TestMonitoredCounterGroup is flaky
* [FLUME-1525] - On some (slow) machines TestFileChannel can fail
* [FLUME-1534] - CheckpointRebuilder$ComparableFlumeEventPointer#equals does not work correctly.
* [FLUME-1540] - CheckpointBuilder needs to open logfiles in inactive mode
* [FLUME-1541] - Implement a SinkSelector for LoadBalancingSinkProcessor that includes failover mechanics
* [FLUME-1544] - Update dev guide to reflect the protoc requirement
* [FLUME-1545] - File channel missing implicit dependency on commons-lang
* [FLUME-1552] - TestFileChannelEncryption fails without a high encryption policy file
* [FLUME-1553] - TestFileChannelEncryption should be refactored to use TestFileChannelBase
* [FLUME-1555] - StressSource outputs bad log messages that reference (Sequence generator)
* [FLUME-1557] - It would be nice if SequenceGeneratorSource could do batching
* [FLUME-1562] - TestLoadBalancingSinkProcessor.testRoundRobinBackoffFailureRecovery is flaky, fails every once in a while...
* [FLUME-1565] - FileChannel Decryption in RandomReader is not thread safe
* [FLUME-1567] - Avro source should expose the number of active connections through JMX
* [FLUME-1570] - StressSource batching does not work unless maxTotalEvents is specified
* [FLUME-1572] - Add batching to FILE_ROLL sink
* [FLUME-1576] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
* [FLUME-1577] - CHECKPOINT_INCOMPLETE should be synced to disk before starting the checkpoint.
* [FLUME-1578] - Proposal to modify file channel encryption config
* [FLUME-1582] - flume-ng script should set LD_LIBRARY_PATH
* [FLUME-1583] - FileChannel fast full replay will always be used if enabled
* [FLUME-1595] - HDFS SequenceFile implementation is not durable due to not using syncFs()
* [FLUME-1606] - Rollbacks of Put transactions does not clear the transaction from inflight puts.
* [FLUME-1610] - HDFSEventSink and bucket writer have a race condition
* [FLUME-1611] - LogUtils regex can be precompiled
* [FLUME-1613] - All of the sink examples in the user guide are broken
* [FLUME-1616] - FileChannel will lose data in when rollback fails with IOException
* [FLUME-1620] - Update flume user guide for LoadBalancingSinkProcessor with the backoff changes.
* [FLUME-1622] - MemoryChannel throws NPE if the event has no body
* [FLUME-1638] - LoadBalancingRpcClient depends on com.google.common.collect.Maps
* [FLUME-1639] - Client SDK should not have dependency on Guava
* [FLUME-1650] - Fix flume-ng-hbase-sink compilation against Hadoop 2.X
* [FLUME-1651] - in the hadoop-0.23 profile HBase version needs to be at least 0.94.2
* [FLUME-1653] - Update Hadoop-23 profile to point to hadoop-2 alpha artifacts
* [FLUME-1655] - Doc update needed for Regex Filtering Interceptor
* [FLUME-1656] - flume-ng script disregards stderr from hadoop command when finding hadoop jars
* [FLUME-1659] - JSON Handler should return simple events, not JSONEvents.
* [FLUME-1662] - Convert null body in events into zero length arrays.
* [FLUME-1664] - Logutils.getLogs remove unneeded directory check
* [FLUME-1671] - Add support for custom components to MonitoredCounterGroup
* [FLUME-1673] - MonitoredCounterGroup publishes this reference to platform MBean server in constructor
* [FLUME-1683] - Fix Time Granularity Bug in SpoolingFileLineReader
* [FLUME-1690] - Elastic Search Sink doesn't run it's unit tests
* [FLUME-1712] - Regex Extractor Interceptor tests have timezone issues
* [FLUME-1705] - SpoolDirectory short name points at the wrong class
* [FLUME-1719] - Example export command in README do not properly close the string
* [FLUME-1723] - AsyncHBase and Avro bring in different versions of Netty
* [FLUME-1726] - SpoolingFileLineReader must close the reader before renaming
* [FLUME-1743] - Multiport syslog tcp source does not load (v1.3 rc5)
** Test
* [FLUME-1492] - Create integration test for file channel
** Task
* [FLUME-1359] - Update main pom.xml file with regards to Flume TLP promotion
** Sub-task
* [FLUME-897] - Implement write ahead log library
* [FLUME-1629] - Add Audience/Stability annotations
* [FLUME-1694] - Fix LICENSE file for binary artifacts
* [FLUME-1695] - Fix tarball names and directories
* [FLUME-1696] - Update build instructions as Flume build requires more memory
* [FLUME-1697] - Update CHANGELOG after 1.3.0 RC0
* [FLUME-1727] - Update CHANGELOG for rc4
----
Release Notes - Flume - Version v1.2.0
** New Feature
* [FLUME-896] - Implement a Durable Memory Channel
* [FLUME-971] - Create developer guide for Flume NG
* [FLUME-988] - Client SDK
* [FLUME-1085] - Implement a durable FileChannel
* [FLUME-1157] - Implement Interceptors (previously known as Decorators) for Flume 1.x
* [FLUME-1183] - Implement an HBase Sink which supports table level access
* [FLUME-1215] - Implement Timestamp Interceptor
* [FLUME-1252] - Asynchronous HBase Sink
** Improvement
* [FLUME-828] - LoggerSink representation of the event's body isn't too useful
* [FLUME-881] - Would be nice if HDFS Sink would automatically choose best writableFormat based on fileType
* [FLUME-979] - ExecSource should optionally restart the command when it exits
* [FLUME-985] - All HDFS Operations in HDFSEventSink should have a timeout
* [FLUME-1001] - Support custom processors
* [FLUME-1011] - AvroSource should have a configurable max thread count
* [FLUME-1020] - Implement Kerberos security for HDFS Sink
* [FLUME-1030] - Retry logic for failover sink processor to handle downstream exceptions in a predictable manner.
* [FLUME-1032] - Fix / clean up Flume NG build
* [FLUME-1043] - SDK should mark slf4j deps as optional
* [FLUME-1048] - speed up mvn package: stop building .zip packages
* [FLUME-1049] - Use hadoop-1.0.0 as basis for default Flume build instead of 0.20.205
* [FLUME-1078] - flume-ng script has no way to add, not replace, classpath
* [FLUME-1090] - JDBC Channel: Minimize logging under nominal conditions
* [FLUME-1117] - Support output to files in Avro container format
* [FLUME-1122] - Flume documentation layout should be refactored
* [FLUME-1126] - Support RFC 3164 and 5424 syslog format timestamp parsing
* [FLUME-1127] - Add configuration support to AbstractAvroEventSerializer for Avro sync interval and compression support
* [FLUME-1132] - HDFSEventSink has spurious and verbose log message
* [FLUME-1140] - Adding Xms value in flume-env.sh
* [FLUME-1160] - ComponentConfigurationFactory catches NullPointerException
* [FLUME-1196] - Allow different HDFS Sinks within the same agent to write to HDFS as different users
* [FLUME-1198] - Implement a load-balancing sink processor
* [FLUME-1212] - Flume should pick HBase jars from HBASE_HOME
* [FLUME-1238] - Support active rolling of files created by HDFS Event Sink
* [FLUME-1242] - Make flume user & dev guides easily editable
* [FLUME-1275] - Add Regex Serializer for HBaseSink
* [FLUME-1287] - Add Standalone Example to Docs
* [FLUME-1330] - Avro Source should not use Fixed thread pool for boss threads when pool size is specified
* [FLUME-1338] - Produce helpful error message in case that timestamp header is missing when time based bucketing is in use
* [FLUME-1343] - Improve user guide
* [FLUME-1345] - Use apache-flume for the artifact instead of flume-ng-dist
* [FLUME-1351] - Add release version to Flume documentation
** Bug
* [FLUME-862] - AvroSource breaks when config properties changes different service
* [FLUME-1002] - FailoverSinkProcessor replaces sinks with same priority
* [FLUME-1017] - syslog classes missing
* [FLUME-1026] - Document Thread Safety Guarantees
* [FLUME-1027] - Missing log4j library in Flume distribution
* [FLUME-1031] - Deprecate code generated by Thrift and Avro OG sources that is under com.cloudera package
* [FLUME-1035] - slf4j error in flume sdk unit tests
* [FLUME-1036] - Reconfiguration of AVRO or NETCAT source causes port bind exception
* [FLUME-1037] - NETCAT handler theads terminate under stress test
* [FLUME-1040] - Release-Notes says Apache Ivy instead of Apache Flume
* [FLUME-1041] - maven warns of duplicate dependencies
* [FLUME-1046] - invoking flume-ng script from bin directory fails
* [FLUME-1047] - Client SDK has dependency on apache commons
* [FLUME-1070] - Fix javadoc for configuring hdfsSink
* [FLUME-1074] - AvroSink if any non-caught exception is thrown, an exception is thrown in finally clause
* [FLUME-1075] - HDFSEventSink begin is called when transaction opened due to other error
* [FLUME-1079] - Flume agent reconfiguration enters permanent bad state
* [FLUME-1080] - Issue with HDFSEventSink for append support
* [FLUME-1083] - Why does flume binary archive produces the following empty directories: bin/{ia64,amd64} ?
* [FLUME-1087] - Restore Client API compat with v1.1.0
* [FLUME-1088] - TestWAL.testThreadedAppend fails on jenkins build server
* [FLUME-1094] - hadoop.profile=23 build is broken by slf4j-jcl dependencies
* [FLUME-1096] - Add support to pass headers through AvroCLIClient
* [FLUME-1098] - Hadoop jars from compilation step included in assembly build
* [FLUME-1099] - copy-paste issue with RecoverableMemoryChannel
* [FLUME-1102] - HDFSEventSink rollInterval is broken
* [FLUME-1104] - HDFS rolls the first file incorrectly
* [FLUME-1108] - FILE_ROLL sink doesn't accept value 0 for unlimited wait time before rolling file
* [FLUME-1109] - Syslog sources need to be refactored
* [FLUME-1112] - HDFSCompressedDataStream append does not work
* [FLUME-1114] - Syslog Sources does not implement maxsize
* [FLUME-1116] - Extra event created for max payload size of 2500 bytes in Flume syslogtcp source
* [FLUME-1119] - Remove default ports for syslog sources
* [FLUME-1121] - Recoverable Memory Channel cannot recover data
* [FLUME-1124] - Lifecycle supervisor can cause thread contention, sometimes causing components to not startup.
* [FLUME-1125] - flume-ng script allows flume-env.sh to clobber some command-line arguments
* [FLUME-1128] - Conf poller should use schedule with fixed delay
* [FLUME-1129] - change foo to agent in sample config
* [FLUME-1130] - flume-ng script bad ordering on FLUME_HOME var
* [FLUME-1135] - flume-docs exclude is not sufficient for rat
* [FLUME-1136] - Remove from executor service does not always remove the runnables from the queue
* [FLUME-1142] - Seq source fails with multiplexing channel selector
* [FLUME-1148] - Refactor logging
* [FLUME-1149] - All sources get same channel list even if configuration is different.
* [FLUME-1154] - flume-ng script should first try finding java from PATH and then try using bigtop, instead of vice-versa
* [FLUME-1156] - If config file has empty sources, then throws NPE
* [FLUME-1163] - HDFSEventSink leaves .tmp files in place when Flume is stopped
* [FLUME-1164] - Configure should be called after stopping all events.
* [FLUME-1177] - Maven deps on flume-ng-configuration module are brought in transitively instead of directly
* [FLUME-1180] - ChannelSelectorFactory creates incorrect selector for multiplexing selector type
* [FLUME-1181] - Context must enforce dot-separated prefix for sub-properties.
* [FLUME-1182] - Syslog source cannot read format correctly from configuration
* [FLUME-1184] - TestFileChannel.testThreaded fails sometimes
* [FLUME-1188] - TestRecoverableMemoryChannel.testThreaded can fail sometimes
* [FLUME-1190] - DurableFileChannel requires FILE enum definition in ChannelConfigurationType
* [FLUME-1194] - RecoverableMemoryChannel prop misspelled -- "rentention" should be "retention"
* [FLUME-1200] - HDFSEventSink causes *.snappy file to be created in HDFS even when snappy isn't used (due to missing lib)
* [FLUME-1202] - Too many approved licenses
* [FLUME-1204] - Add more unit tests for hbase sink
* [FLUME-1205] - NPE related to checkpointing when using FileChannel
* [FLUME-1213] - HDFS sink should allow bucketpath rounding down.
* [FLUME-1216] - Need useful error message when keytab does not exist
* [FLUME-1217] - HDFS Event Sink generates warnings due to recent change
* [FLUME-1219] - Race conditions in BucketWriter / HDFSEventSink
* [FLUME-1220] - Load balancing channel selector needs to be in the configuration type
* [FLUME-1221] - ThriftLegacySource doesn't handle fields -> headers conversions for bytebuffers
* [FLUME-1226] - FailoverRpcClient should check for NULL batch-size property
* [FLUME-1229] - System.nanoTime incorrectly used in filename for HDFS file rolling
* [FLUME-1230] - Sink gets initialized even when not active
* [FLUME-1231] - Deadlock between BucketWriter and LeaseChecker on shutdown
* [FLUME-1232] - Cannot start agent a 3rd time when using FileChannel
* [FLUME-1234] - Can't use %P escape sequence for bucket path of HDFS sink
* [FLUME-1236] - File channel has a race condition between start and create transaction method
* [FLUME-1240] - Add version info to Flume NG
* [FLUME-1241] - Flume dist should include the flume-ng-doc directory
* [FLUME-1244] - Implement a load-balancing RpcClient with round/robin and random distribution capabilties.
* [FLUME-1245] - HDFSCompressedDataStream calls finish() on sync instead of flush()
* [FLUME-1246] - FileChannel hangs silently when Hadoop libs not found
* [FLUME-1248] - flume-ng script gets broken when it tried to load hbase classpath
* [FLUME-1253] - Support for running integration tests
* [FLUME-1254] - RpcClient can hang when communication is broken with the source.
* [FLUME-1270] - Incorrect default hdfs.callTimeout and hdfs.fileType of HDFSEventSink in FlumeUserGuide.rst
* [FLUME-1271] - Incorrect configuration causes NPE
* [FLUME-1280] - Make all config properties of Hbase sinks public constants
* [FLUME-1284] - Need host interceptor for hdfs bucket path escape sequence
* [FLUME-1288] - Async hbase sink should throw exception when hbase reports failure and check hbase table correctness
* [FLUME-1290] - HDFS Sink should accept fileType parameters of arbitrary case
* [FLUME-1297] - Flume tests should wait for a few seconds for agent to start.
* [FLUME-1301] - HDFSCompressedDataStream can lose data
* [FLUME-1303] - java.library.path value is being truncated at first 'n' char
* [FLUME-1304] - Allow for faster allocation of checkpoint file.
* [FLUME-1306] - LoadBalancingRpcClient should catch exception for invalid RpcClient and failover to valid one
* [FLUME-1309] - Integration tests not included in assembly build artifacts
* [FLUME-1312] - Host interceptor should support custom headers
* [FLUME-1314] - File channel log file can grow beyond max size which causes startup failure
* [FLUME-1315] - Null sink should support batching
* [FLUME-1316] - AvroSink should be configurable for connect-timeout and request-timeout
* [FLUME-1317] - Assembly build pulls in target folder from flume-ng-tests
* [FLUME-1319] - File Channel optimize replay of logs when a checkpoint is present
* [FLUME-1320] - Add safeguard for checkpoint corruption detection
* [FLUME-1322] - ChannelProcessor should catch Throwable to work around close() clobbering uncaught Exceptions
* [FLUME-1323] - Remove shutdown hook from FileChannel