Help to configure Grafana Agent (OSS)

[rfratto]

Discussed in #6032

^{Originally posted by Shivam327 January 2, 2024}
I have tried to implenemnet Agent on Cloud it worked successfully, now when i try to implemenet it in OSS
It gives below error

ts=2024-01-02T11:39:02.5403937Z caller=reporter.go:103 level=info msg="running usage stats reporter"
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x545c8b5]

goroutine 554 [running]:
github.com/prometheus-community/windows_exporter/pkg/collector/process.(*collector).Collect(0xc00358c340, 0xc006d89be0, 0x0?)
        /go/pkg/mod/github.com/prometheus-community/[email protected]/pkg/collector/process/process.go:309 +0xd5
github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).execute(0xc0028222c0, {0x78dd53b, 0x7}, {0x89aca20, 0xc00358c340}, 0x0?, 0x0?)
        /go/pkg/mod/github.com/prometheus-community/[email protected]/pkg/collector/prometheus.go:176 +0x8f
github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).Collect.func2({0x78dd53b, 0x7}, {0x89aca20?, 0xc00358c340?})
        /go/pkg/mod/github.com/prometheus-community/[email protected]/pkg/collector/prometheus.go:117 +0xa5
created by github.com/prometheus-community/windows_exporter/pkg/collector.(*Prometheus).Collect in goroutine 449
        /go/pkg/mod/github.com/prometheus-community/[email protected]/pkg/collector/prometheus.go:115 +0x43b

Below is my Agent Config

integrations:
  windows_exporter:
    enabled: true
    instance: "shivam-local" # must match instance used in logs
    # enable default collectors and time collector:
    enabled_collectors: cache,cpu,cpu_info,iis,logon,memory,mscluster_cluster,mscluster_network,netframework_clrmemory,tcp,time,mssql,cs,net,service,os,cpu_info,memory,mssql,process,time
    metric_relabel_configs:
      # drop disk volumes named HarddiskVolume.*
      - action: drop
        regex: HarddiskVolume.*
        source_labels: [volume]
    relabel_configs:
      - target_label: job
        replacement: "integrations/windows_exporter" # must match job used in logs

  prometheus_remote_write:
    - url: http://localhost:9090/api/v1/write
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
      - action: replace
        target_label: job
        replacement: "integrations/agent-check"
    metric_relabel_configs:
      - action: keep
        regex: (prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|agent_build.*|agent_wal_samples_appended_total|process_start_time_seconds)
        source_labels:
          - __name__
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
logs:
  configs:
    - clients:
        - url: http://localhost:3100/loki/api/v1/push
      name: integrations
      positions:
        filename: /tmp/positions.yaml
      scrape_configs:
        - job_name: integrations/windows-exporter-application
          windows_events:
            use_incoming_timestamp: true
            bookmark_path: "./bookmarks-app.xml"
            eventlog_name: "Application"
            xpath_query: "*"
            locale: 1033
            # - 1033 to force English language
            # -  0 to use default Windows locale
            labels:
              job: integrations/windows_exporter
              instance: "shivam-local" # must match instance used in windows_exporter
          relabel_configs:
            - source_labels: ["computer"]
              target_label: "agent_hostname"
          pipeline_stages:
            - json:
                expressions:
                  source: source
                  level: levelText
            - labels:
                source:
                level:
        - job_name: integrations/windows-exporter-system
          windows_events:
            use_incoming_timestamp: true
            bookmark_path: "./bookmarks-sys.xml"
            eventlog_name: "System"
            xpath_query: "*"
            locale: 1033
            # - 1033 to force English language
            # -  0 to use default Windows locale
            labels:
              job: integrations/windows_exporter
              instance: "shivam-local" # must match instance used in windows_exporter
          relabel_configs:
            - source_labels: ["computer"]
              target_label: "agent_hostname"
          pipeline_stages:
            - json:
                expressions:
                  source: source
                  level: levelText
            - labels:
                source:
                level:

          # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
          # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
metrics:
  configs:
    - name: integrations
      remote_write:
        - url: http://localhost:9090/api/v1/write
      scrape_configs:
        # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
        # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal

Earlier it was working, but now this error is comming,
Please help me out to solve this porblem

[brngates98]

take a look at my config for refrence
#6036

i am having a timeout issue currently on some metrics but it is sending data

[avcercel]

Hello folks

We're having the same problem as @rfratto .

@brngates98 , your config is not really relevant in this case, as per my understanding.
You are not using the process collector, which is what's causing the relevant error.

Your collectors:
enabled_collectors: cpu,cs,logical_disk,net,os,service,system,time,diskdrive <- no "process"

@rfratto collectors:
enabled_collectors: cache,cpu,cpu_info,iis,logon,memory,mscluster_cluster,mscluster_network,netframework_clrmemory,tcp,time,mssql,cs,net,service,os,cpu_info,memory,mssql,process,time <- second one from the end

I've been trying for two days to enable the process monitoring for our Windows fleet and whatever the config I'm trying and I'm including here not using any config block at all because it should fall back on the defaults, I'm getting this nil pointer exception.

I've seen some big PRs on the community windows_exporter codebase which I think caused some defaults to not be passed on correctly?

[brngates98]

Ahhh my example did not include the process collector but I do use it on a few windows nodes, I can share that snippet if you’d like

…

On Thu, Jan 4, 2024 at 11:27 AM Alex Cercel ***@***.***> wrote: Hello folks We're having the same problem as @rfratto <https://github.com/rfratto> . @brngates98 <https://github.com/brngates98> , your config is not really relevant in this case, as per my understanding. You are not using the process collector, which is what's causing the relevant error. Your collectors: enabled_collectors: cpu,cs,logical_disk,net,os,service,system,time,diskdrive <- no "process" @rfratto <https://github.com/rfratto> collectors: enabled_collectors: cache,cpu,cpu_info,iis,logon,memory,mscluster_cluster,mscluster_network,netframework_clrmemory,tcp,time,mssql,cs,net,service,os,cpu_info,memory,mssql,process,time <- second one from the end I've been trying for two days to enable the process monitoring for our Windows fleet and whatever the config I'm trying and I'm including here not using any config block at all because it should fall back on the defaults, I'm getting this nil pointer exception. I've seen some big PRs on the community windows_exporter codebase which I think caused some defaults to not be passed on correctly? — Reply to this email directly, view it on GitHub <#6034 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOFQ2CPZQR3UQV2UQIIQQ6TYM3J63AVCNFSM6AAAAABBKBBIT6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZXGM4TEMZZGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[brngates98]

sum by(process) (rate(windows_process_cpu_time_total{agent_hostname="$server"}[5m]))

query for processes ill get you the actual Grafana Agent config that includes the process collector and collects metrics from all the processes on the windows system

[avcercel]

The actual config would be really helpful. Thank you.

[tpaschalis]

cc @mattdurham

This looks related to the fix we got out for #5866, right?

[mattdurham]

The fix is in the RC if you want to see if that resolves the problem.

[brngates98]

@avcercel here is the config that is on one of our Cash register(PALM POS, Windows 10LTSB, we use it to monitor the processes as palm.exe and pharoh.exe like to have memory leaks and

This is the windows exporter specific piece

  windows_exporter:
    enabled: true
    instance: PALM
    enabled_collectors: "cpu,cs,logical_disk,net,os,service,system,textfile,process"
    relabel_configs:
    - target_label: job
      replacement: 'integrations/windows_exporter' # must match job used in logs
    metric_relabel_configs:
    - action: keep
      source_labels:
      - __name__
    process:
      whitelist: ".+"

integrations:
  prometheus_remote_write:
  - url: https://mimir.thetmg.com/api/v1/push
  # Add here any snippet that belongs to the `integrations` section.
  # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.  
  windows_exporter:
    enabled: true
    instance: PALM
    enabled_collectors: "cpu,cs,logical_disk,net,os,service,system,textfile,process"
    relabel_configs:
    - target_label: job
      replacement: 'integrations/windows_exporter' # must match job used in logs
    metric_relabel_configs:
    - action: keep
      source_labels:
      - __name__
    process:
      whitelist: ".+"
  agent:
    enabled: true
    relabel_configs:
      - action: replace
        source_labels:
          - agent_hostname
        target_label: instance
logs:
  configs:
  - clients:
    - url: https://logs.thetmg.com/loki/api/v1/push
    name: integrations
    positions:
      filename: palmpositions.yaml
    scrape_configs:
      # Add here any snippet that belongs to the `logs.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
    - job_name: integrations/windows-exporter-application
      windows_events:
        use_incoming_timestamp: false
        bookmark_path: "C:\\Program Files\\Grafana Agent\\bookmarks-app.xml"
        eventlog_name: "Application"
        labels:
          job: integrations/windows_exporter
          instance: PALM # must match instance used in windows_exporter
      relabel_configs:
        - source_labels: ['computer']
          target_label: 'agent_hostname'
      pipeline_stages:
        - json:
            expressions:
              source: source
        - labels:
            source:
    - job_name: integrations/windows-exporter-application
      windows_events:
        use_incoming_timestamp: false
        bookmark_path: "C:\\Program Files\\Grafana Agent\\bookmarks-security.xml"
        eventlog_name: "Security"
        labels:
          job: integrations/windows_exporter
          instance: PALM # must match instance used in windows_exporter
      relabel_configs:
        - source_labels: ['computer']
          target_label: 'agent_hostname'
      pipeline_stages:
        - json:
            expressions:
              source: source
        - labels:
            source:
    - job_name: integrations/windows-exporter-system
      windows_events:
        use_incoming_timestamp: false
        bookmark_path: "C:\\Program Files\\Grafana Agent\\bookmarks-system.xml"
        eventlog_name: "System"
        labels:
          job: integrations/windows_exporter
          instance: PALM # must match instance used in windows_exporter
      relabel_configs:
        - source_labels: ['computer']
          target_label: 'agent_hostname'
      pipeline_stages:
        - json:
            expressions:
              source: source
        - labels:
            source:
    - job_name: integrations/agent
      windows_events:
        use_incoming_timestamp: true
        bookmark_path: "C:\\Program Files\\Grafana Agent\\bookmarks-agent.xml"
        eventlog_name: "Application"
        xpath_query: "*[System[Provider[@Name='Grafana Agent']]]"
        labels:
          job: integrations/agent
      relabel_configs:
        - source_labels: ['computer']
          target_label: 'instance'
        - replacement: grafana-agent.service
          target_label: unit
      pipeline_stages:
      - json:
          expressions:
            message: message
      - regex:
          expression: "(level=(?P<log_level>[\\s]*debug|warn|info|error))"
      - labels:
          level: log_level
      - output:
          source: message
metrics:
  configs:
  - name: integrations
    remote_write:
    - url: https://mimir.thetmg.com/api/v1/push
    scrape_configs:
      # Add here any snippet that belongs to the `metrics.configs.scrape_configs` section.
      # For a correct indentation, paste snippets copied from Grafana Cloud at the beginning of the line.
  global:
    scrape_interval: 60s
  wal_directory: /tmp/grafana-agent-wal