v2.11.0

Features

• Added "json" action to decode field and copy resulting fields into root of event (fba6388)
• Combine similiar event index failures, reporting only first message and total count (4a148b3)
• Grok action will now only remove the field if grok parsing was successful (676225e)
• Implement @metadata[receiver][name] and others against all events to allow processors to process based on connection details (this replaced agent[source] which is removed) (9ba6ba7)
• Log Carver now appends direct source of a received event to agent[source] (68b88c7)
• The @timestamp field can now be set from grok and json actions if in the correct format expected (RFC3339 with optional nanoseconds: 2006-01-02T15:04:05.999Z) (36bc467)
• TLS handshake errors are now prefixed to show they originate from a handshake (1d8a57f)

Bug Fixes

• Fix a rare race in publisher for immediately failing reconnections that can cause high CPU and pipeline freeze (e3aacc8)
• Fix crash if reconnection reusing port occurs (d5c1cc2)
• Fix potential log-carver deadlock when using lc-admin -carver and a new connection appears during the status poll (d702ce1)
• Fix potential panic in receiver if a connection is lost (3615aa3)
• Fix receiver pool not cleaning up connections in some instances and preventing shutdown, and add additional logs (f072865)
• Fix receiver shutting down and reset messages displaying incorrectly (4d7c873)
• Fix scheduling issues that can cause unexpected timeouts on new connections (b0980cc)
• Fix send congestion or send errors not causing connection shutdown and leaking connections (02c8ede)
• Fix successful ES bulk request failing with cluster block message due to cluster block handling fix (20e5751)
• Fix timeout on courier displaying two error messages, one for unexpected end (e453486)
• Float grok values and conversion to float expressions now serialize correctly to float in the Elasticsearch output (bb0f4a9)
• Receiver can cause panic during reload in Log Carver (c43f935), closes #400
• Receiver pool can sometimes crash when a disconnect occurs unexpectedly (3553db5)
• Resolve a formatting issue in Handshake failure output (2f6d77d)
• Resolve connection leakage that can cause too many open files, especially with stream input (39f87a9)
• Resolve crash introduced with cleanup code for force failed connections (a17595b)
• Resolve ES template not indexing terms for messages longer than 256 characters (99bf67f)
• Resolve force closed connections on receiver not being cleaned up fully, preventing shutdown (b6689de)
• Resolve linereader hanging on streams due to recent fix on missing lines (bdc463c)
• Resolve missing lines at end of streams due to harvester early failure (710fc3c)
• Resolve potential crash in receiver cleanup if shutdown of a connection occurs whilst draining (fcd54d6)
• Resolve potential crash when no endpoints are available to publish to (463ed63)
• Workaround some EOF errors on connections that closed gracefully (7d20247)

v2.10.0

Features

• As of 2.10.0 SHA1 signed certificates will be no longer supported, but can be temporarily enabled by setting the GODEBUG environment variable to x509sha1=1 (d7659a3)
• Implement support for /**/ matching in file paths, and report IO errors on first scan (95daa0d), closes #327 #285
• Implement TCP streaming receiver (b3a3720)
• Improved failover of connections when using SRV records (05fcd48)
• Improved log output for transports to display more meaningful connection details in some instances (1a14888)
• lc-admin file list is now sorted to prevent display jumps on highly active instances (17e9014), closes #396
• Sort receivers and transports within lc-admin (74775cc)
• SRV record servers now expand after lookup as if the looked up hosts were listed servers, enabling failover and load balancing support (3a2fecf), closes #354

Bug Fixes

• Fix :127.0.0.1:1234 not working as specified in documentation for admin connect string (ef11492), closes #395
• Fix es-https transport reporting ssl ca required when it was (1741ba7)
• Fix for cache of ES transport clients that can result in too many open files (bf45f4e)
• Fix loadbalance not balancing effectively when under pressure and queueing more payloads than it should on single endpoints (54aef03)
• Fix logstash-input-courier not shutting down with Logstash pipeline (ea7a63c), closes #397
• Fix rare crash due to concurrent encoding activity in transport by preventing encoding making writes to the event (ab6e5f7)
• Fix tls not working correctly for es-https, imporve tls config management, fix verify peers, make cert/key required for carver receiver (c840355)

v2.9.1

Bug Fixes

• Fix installation issue of gems due to missing version.rb file (a2dc5e4), closes #393
• Fix recursive lock in harvester blocking admin socket (dc03b40), closes #394

v2.9.0

26th March 2022

Log Carver

• Updated CEL Go library to 0.9.0
• Implemented CEL Go string extensions and base64 encoder
• Implemented JSON encoder (json_encode and json_decode CEL Go functions)
• Fixed lc-admin status for transports not updating in some cases

v2.8.1

23rd November 2021

Log Courier / Log Carver

• Fixed crash during configuration reload for tcp receiver and transport
• Fixed reload of configuration not correctly updating endpoints
• Fixed reload of configuration sometimes causing a deadlock with hugh CPU

v2.8.0

22nd November 2021

Log Courier / Log Carver

• Fixed transports sometimes not restarting when configuration is changed
• Fix regression caused by fix for #367 that can cause Log Courier to crash on startup

Log Carver

• Added es-https transport to use encrypted HTTPS communication with Elasticsearch
• Added min tls version and max tls version to es-https transport
• Added ssl ca for es-https transport to allow certificate pinning
• Added username and password for es and es-https transports to allow Basic authentication scheme
• Improved Elasticsearch bulk response validation
• Fixed missing RuntimeDirectory in Log Carver systemd configuration which is necessary for default admin socket listen location

lc-admin

• Added -carver option to connect to automatically detect the administration socket from Log Carver's configuration file or from known defaults

v2.7.4

26th October 2021

Logstash Input Plugin

• Fix regression caused by removal of peer_recv_queue - some code remained that was still attempting to access it

v2.7.3

26th October 2021

Log Courier

• Add new add timezone name field configuration that adds the timezone name such as UTC or Europe/London for use with the Logstash Date Filter. The existing timezone / event.timezone (ECS) fields were in a format the filter could not use (#345)
• Fix race that might cause a file to use a configuration other than the first configuration it matches (#367)

Logstash Input Plugin

• Removed peer_recv_queue configuration as it is unused. Only a single payload is received and processed at any one time by the plugin.

Logstash Output Plugin

• Now maintained again and updated to use latest log-courier ruby implementation which includes protocol handshake support
• Added support for tcp only output

v2.7.2

21st October 2021

Logstash Input Plugin

• Fix compatibility issue between Ruby Log Courier and Go Log Courier caused by a discrepency in the HELO/VERS protocol message that form the handshake

v2.7.1

21st October 2021

Logstash Input Plugin

• Fix missing protocol.rb file causing import error