Exporter not working in Logstash 6.3

[matejzero]

Hello,

I'm running this collector on Logstash 6.3.2 and it's not exposing any metrics. It worked on Logstash 5.6.

According to the tests, it should work, but for some reason it doesn't and I'm not enough dev to look into it.

Output of exporter:

# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.0557e-05
go_gc_duration_seconds{quantile="0.25"} 6.958e-05
go_gc_duration_seconds{quantile="0.5"} 0.000106215
go_gc_duration_seconds{quantile="0.75"} 0.000336537
go_gc_duration_seconds{quantile="1"} 0.005423857
go_gc_duration_seconds_sum 7.33736376
go_gc_duration_seconds_count 17538
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 17
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 1.955104e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.8217981696e+10
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.552267e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.02241432e+08
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 507904
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 1.955104e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.038656e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.646016e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 7488
# HELP go_memstats_heap_released_bytes_total Total number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes_total counter
go_memstats_heap_released_bytes_total 1.10592e+06
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.684672e+06
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.534244197725826e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 370671
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.0224892e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 4800
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 30704
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 49152
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 4.194304e+06
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.124717e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 655360
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 655360
# HELP go_memstats_sys_bytes Number of bytes obtained by system. Sum of all system allocations.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.0590456e+07
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 9675.681
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 29112.124
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 76503.452
http_request_duration_microseconds_sum{handler="prometheus"} 5.661427601970019e+08
http_request_duration_microseconds_count{handler="prometheus"} 44665
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 175
http_request_size_bytes{handler="prometheus",quantile="0.9"} 175
http_request_size_bytes{handler="prometheus",quantile="0.99"} 471
http_request_size_bytes_sum{handler="prometheus"} 7.817436e+06
http_request_size_bytes_count{handler="prometheus"} 44665
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 44665
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 2329
http_response_size_bytes{handler="prometheus",quantile="0.9"} 2333
http_response_size_bytes{handler="prometheus",quantile="0.99"} 2339
http_response_size_bytes_sum{handler="prometheus"} 1.03254351e+08
http_response_size_bytes_count{handler="prometheus"} 44665
# HELP logstash_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which logstash_exporter was built.
# TYPE logstash_exporter_build_info gauge
logstash_exporter_build_info{branch="",goversion="go1.8.3",revision="",version=""} 1
# HELP logstash_exporter_scrape_duration_seconds logstash_exporter: Duration of a scrape job.
# TYPE logstash_exporter_scrape_duration_seconds summary
logstash_exporter_scrape_duration_seconds{collector="node",result="success",quantile="0.5"} 0.008065311
logstash_exporter_scrape_duration_seconds{collector="node",result="success",quantile="0.9"} 0.02810888
logstash_exporter_scrape_duration_seconds{collector="node",result="success",quantile="0.99"} 0.075507031
logstash_exporter_scrape_duration_seconds_sum{collector="node",result="success"} 497.8846417519984
logstash_exporter_scrape_duration_seconds_count{collector="node",result="success"} 44666
# HELP logstash_node_gc_collection_duration_seconds_total gc_collection_duration_seconds_total
# TYPE logstash_node_gc_collection_duration_seconds_total counter
logstash_node_gc_collection_duration_seconds_total{collector="old"} 2.1408032e+07
logstash_node_gc_collection_duration_seconds_total{collector="young"} 1.509914e+06
# HELP logstash_node_gc_collection_total gc_collection_total
# TYPE logstash_node_gc_collection_total gauge
logstash_node_gc_collection_total{collector="old"} 166593
logstash_node_gc_collection_total{collector="young"} 229027
# HELP logstash_node_jvm_threads_count jvm_threads_count
# TYPE logstash_node_jvm_threads_count gauge
logstash_node_jvm_threads_count 174
# HELP logstash_node_jvm_threads_peak_count jvm_threads_peak_count
# TYPE logstash_node_jvm_threads_peak_count gauge
logstash_node_jvm_threads_peak_count 176
# HELP logstash_node_mem_heap_committed_bytes mem_heap_committed_bytes
# TYPE logstash_node_mem_heap_committed_bytes gauge
logstash_node_mem_heap_committed_bytes 1.038876672e+09
# HELP logstash_node_mem_heap_max_bytes mem_heap_max_bytes
# TYPE logstash_node_mem_heap_max_bytes gauge
logstash_node_mem_heap_max_bytes 1.038876672e+09
# HELP logstash_node_mem_heap_used_bytes mem_heap_used_bytes
# TYPE logstash_node_mem_heap_used_bytes gauge
logstash_node_mem_heap_used_bytes 6.618756e+08
# HELP logstash_node_mem_nonheap_committed_bytes mem_nonheap_committed_bytes
# TYPE logstash_node_mem_nonheap_committed_bytes gauge
logstash_node_mem_nonheap_committed_bytes 2.90275328e+08
# HELP logstash_node_mem_nonheap_used_bytes mem_nonheap_used_bytes
# TYPE logstash_node_mem_nonheap_used_bytes gauge
logstash_node_mem_nonheap_used_bytes 2.49728912e+08
# HELP logstash_node_mem_pool_committed_bytes mem_pool_committed_bytes
# TYPE logstash_node_mem_pool_committed_bytes gauge
logstash_node_mem_pool_committed_bytes{pool="old"} 7.2482816e+08
logstash_node_mem_pool_committed_bytes{pool="survivor"} 3.4865152e+07
logstash_node_mem_pool_committed_bytes{pool="young"} 2.7918336e+08
# HELP logstash_node_mem_pool_max_bytes mem_pool_max_bytes
# TYPE logstash_node_mem_pool_max_bytes gauge
logstash_node_mem_pool_max_bytes{pool="old"} 7.2482816e+08
logstash_node_mem_pool_max_bytes{pool="survivor"} 3.4865152e+07
logstash_node_mem_pool_max_bytes{pool="young"} 2.7918336e+08
# HELP logstash_node_mem_pool_peak_max_bytes mem_pool_peak_max_bytes
# TYPE logstash_node_mem_pool_peak_max_bytes gauge
logstash_node_mem_pool_peak_max_bytes{pool="old"} 7.2482816e+08
logstash_node_mem_pool_peak_max_bytes{pool="survivor"} 7.2482816e+08
logstash_node_mem_pool_peak_max_bytes{pool="young"} 7.2482816e+08
# HELP logstash_node_mem_pool_peak_used_bytes mem_pool_peak_used_bytes
# TYPE logstash_node_mem_pool_peak_used_bytes gauge
logstash_node_mem_pool_peak_used_bytes{pool="old"} 6.05408008e+08
logstash_node_mem_pool_peak_used_bytes{pool="survivor"} 6.05408008e+08
logstash_node_mem_pool_peak_used_bytes{pool="young"} 6.05408008e+08
# HELP logstash_node_mem_pool_used_bytes mem_pool_used_bytes
# TYPE logstash_node_mem_pool_used_bytes gauge
logstash_node_mem_pool_used_bytes{pool="old"} 5.944496e+08
logstash_node_mem_pool_used_bytes{pool="survivor"} 8.000088e+06
logstash_node_mem_pool_used_bytes{pool="young"} 5.9425912e+07
# HELP logstash_node_pipeline_duration_seconds_total pipeline_duration_seconds_total
# TYPE logstash_node_pipeline_duration_seconds_total counter
logstash_node_pipeline_duration_seconds_total 0
# HELP logstash_node_pipeline_events_filtered_total pipeline_events_filtered_total
# TYPE logstash_node_pipeline_events_filtered_total counter
logstash_node_pipeline_events_filtered_total 0
# HELP logstash_node_pipeline_events_in_total pipeline_events_in_total
# TYPE logstash_node_pipeline_events_in_total counter
logstash_node_pipeline_events_in_total 0
# HELP logstash_node_pipeline_events_out_total pipeline_events_out_total
# TYPE logstash_node_pipeline_events_out_total counter
logstash_node_pipeline_events_out_total 0
# HELP logstash_node_process_cpu_total_seconds_total process_cpu_total_seconds_total
# TYPE logstash_node_process_cpu_total_seconds_total counter
logstash_node_process_cpu_total_seconds_total 673483
# HELP logstash_node_process_max_filedescriptors process_max_filedescriptors
# TYPE logstash_node_process_max_filedescriptors gauge
logstash_node_process_max_filedescriptors 16384
# HELP logstash_node_process_mem_total_virtual_bytes process_mem_total_virtual_bytes
# TYPE logstash_node_process_mem_total_virtual_bytes gauge
logstash_node_process_mem_total_virtual_bytes 5.824548864e+09
# HELP logstash_node_process_open_filedescriptors process_open_filedescriptors
# TYPE logstash_node_process_open_filedescriptors gauge
logstash_node_process_open_filedescriptors 302
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 205.06
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 8.822784e+06
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.53379762791e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 4.45329408e+08

[thiagoit]

Also looking for support for Logstash metrics on version 6.3. Anyone working on it?

[mpucholblasco]

I'm woring with logstash 6.4.0 and it works properly. Which are exactly your problems @matejzero & @thiagoit ?

[matejzero]

In my case, there was no metrics for pipelines, filters and such. I guess I’ll have to upgrade to 6.4

[mpucholblasco]

@matejzero , I just tested on version 6.3 and it worked correctly. I have added instructions to test it via docker on https://github.com/sequra/logstash_exporter/blob/master/integration-tests/README.md (if you want to test it with another version, you can do it in this way).