Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nothing in the Splunk index #707

Open
alithhero opened this issue Feb 27, 2023 · 5 comments
Open

Nothing in the Splunk index #707

alithhero opened this issue Feb 27, 2023 · 5 comments

Comments

@alithhero
Copy link

alithhero commented Feb 27, 2023 •
edited
Loading

Can somebody help me with this problem? I see nothing significant in the em_logs index that can explain me why I see nothing in the netops index ... Here is my value.yaml file :
All XXXXX are properly completed in my configuration.

splunk:
  enabled: true
  protocol: http
  host: XXXXXX
  token: XXXXXXXX
  insecureSSL: "true"
  port: "8088"
  eventIndex: "myindex_netops"
  metricsIndex: "myindex_netmetrics"

image:
  pullPolicy: "Always"
traps:
  communities:
    2c:
      - public
      - homelab
  usernameSecrets:
    - XXXXXX
  #  - sc4snmp-hlab-sha-des

  #loadBalancerIP: The IP address in the metallb pool
  loadBalancerIP: 10.128.149.27
worker:
  # There are 3 types of workers 
  trap:
    # replicaCount: number of trap-worker pods which consumes trap tasks
    replicaCount: 2
    #autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  poller:
    # replicaCount: number of poller-worker pods which consumes polling tasks
    replicaCount: 2
    #autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  sender:
    # replicaCount: number of sender-worker pods which consumes sending tasks
    replicaCount: 1
    # autoscaling: use it instead of replicaCount in order to make pods scalable by itself
    #autoscaling:
    #  enabled: true
    #  minReplicas: 2
    #  maxReplicas: 10
    #  targetCPUUtilizationPercentage: 80
  # udpConnectionTimeout: timeout in seconds for SNMP operations
  #udpConnectionTimeout: 5
  logLevel: "INFO"
scheduler:
  logLevel: "INFO"
  profiles: |
    generic_switch:
      frequency: 300
      varBinds:
        - ['SNMPv2-MIB', 'sysDescr']
        - ['SNMPv2-MIB', 'sysName', 0]
        - ['TCP-MIB', 'tcpActiveOpens']
        - ['TCP-MIB', 'tcpAttemptFails']
        - ['IF-MIB']
  groups: |
    ciusss_group_commutateur:
      - address: 10.112.104.5
      - address: 10.130.44.1
      - address: 10.130.32.9
      - address: 10.130.32.8
      - address: 10.130.32.1
      - address: 10.130.32.6
      - address: 10.130.32.7
      - address: 10.130.40.9
      - address: 10.130.40.1
      - address: 10.130.40.6
      - address: 10.130.40.10
      - address: 10.130.40.8
      - address: 10.130.40.7
      - address: 10.132.92.1
      - address: 10.132.92.5
      - address: 10.132.92.6
      - address: 10.48.180.1
      - address: 10.133.0.136
      - address: 10.133.0.134
      - address: 10.133.0.1
      - address: 10.133.0.139
      - address: 10.133.0.131
      - address: 10.133.0.132
      - address: 10.133.0.143
      - address: 10.133.0.144
      - address: 10.133.0.145
      - address: 10.133.0.133
      - address: 10.133.0.140
      - address: 10.133.0.148
      - address: 10.133.0.146
      - address: 10.133.0.135
      - address: 10.133.0.138
      - address: 10.133.4.1
      - address: 10.133.8.1
      - address: 10.133.16.1
      - address: 10.133.24.1
      - address: 10.133.20.1
      - address: 10.112.88.1
      - address: 10.49.168.5
      - address: 10.49.168.6
      - address: 10.112.136.1
      - address: 10.130.192.5
      - address: 10.154.69.5
      - address: 10.132.136.5
      - address: 10.132.136.6
      - address: 10.130.164.6
      - address: 10.132.144.1
      - address: 10.133.64.15
      - address: 10.133.64.5
      - address: 10.133.64.1
      - address: 10.133.64.6
      - address: 10.133.64.8
      - address: 10.133.64.13
      - address: 10.132.156.5
      - address: 10.132.184.5
      - address: 10.130.176.5
      - address: 10.130.132.1
      - address: 10.132.140.6
      - address: 10.133.140.5
      - address: 10.131.60.8
      - address: 10.131.60.1
      - address: 10.131.60.9
      - address: 10.130.208.1
      - address: 10.130.180.7
      - address: 10.130.180.6
      - address: 10.133.76.1
      - address: 10.133.76.11
      - address: 10.133.76.12
      - address: 10.133.88.1
      - address: 10.133.84.1
      - address: 10.132.128.8
      - address: 10.132.128.1
      - address: 10.132.128.7
      - address: 10.132.128.9
      - address: 10.133.96.1
      - address: 10.133.128.5
      - address: 10.130.168.1
      - address: 10.133.100.1
      - address: 10.130.160.5
      - address: 10.131.4.6
      - address: 10.130.204.5
      - address: 10.130.240.5
      - address: 10.133.92.1
      - address: 10.131.52.5
      - address: 10.133.104.1
      - address: 10.130.128.5
      - address: 10.133.116.1
      - address: 10.131.20.5
      - address: 10.140.64.97
      - address: 10.140.64.102
      - address: 10.140.64.106
      - address: 10.140.64.123
      - address: 10.140.64.114
      - address: 10.140.64.98
      - address: 10.140.64.99
      - address: 10.140.68.34
      - address: 10.140.68.33
      - address: 10.140.68.35
      - address: 10.140.68.36
      - address: 10.49.100.1
      - address: 10.134.128.1
      - address: 10.134.128.6
      - address: 10.53.166.150
      - address: 10.53.166.66
      - address: 10.53.166.68
      - address: 10.53.166.70
      - address: 10.53.166.72
      - address: 10.53.166.74
      - address: 10.53.166.76
      - address: 10.53.166.78
      - address: 10.53.166.56
      - address: 10.53.166.58
      - address: 10.53.166.62
      - address: 10.53.166.64
      - address: 10.53.166.54
      - address: 10.53.166.52
      - address: 10.53.166.46
      - address: 10.53.166.48
      - address: 10.53.166.50
      - address: 10.53.166.80
      - address: 10.53.166.104
      - address: 10.53.166.106
      - address: 10.53.166.90
      - address: 10.53.166.92
      - address: 10.53.166.94
      - address: 10.53.166.96
      - address: 10.53.166.98
      - address: 10.53.166.100
      - address: 10.53.166.102
      - address: 10.53.166.82
      - address: 10.53.166.83
      - address: 10.53.166.84
      - address: 10.53.166.86
      - address: 10.53.166.81
      - address: 10.53.166.114
      - address: 10.53.166.118
      - address: 10.53.166.120
      - address: 10.53.166.122
      - address: 10.53.166.110
      - address: 10.53.166.124
      - address: 10.53.166.126
      - address: 10.53.166.128
      - address: 10.53.166.130
      - address: 10.53.166.132
      - address: 10.53.166.108
      - address: 10.53.166.151
      - address: 10.53.166.152
      - address: 10.53.166.103
      - address: 10.53.166.112
      - address: 10.53.166.109
      - address: 10.53.166.42
      - address: 10.53.166.40
      - address: 10.53.166.138
      - address: 10.53.166.140
      - address: 10.53.166.142
      - address: 10.53.166.144
      - address: 10.53.166.146
      - address: 10.53.166.136
      - address: 10.53.166.134
      - address: 10.134.224.19
      - address: 10.134.224.17
      - address: 10.134.224.16
      - address: 10.134.224.5
      - address: 10.134.224.14
      - address: 10.134.224.15
      - address: 10.134.224.11
      - address: 10.134.224.12
      - address: 10.134.224.18
      - address: 10.134.224.20
      - address: 10.134.224.7
      - address: 10.134.224.6
      - address: 10.134.224.4
      - address: 10.134.224.21
      - address: 10.134.224.22
      - address: 10.134.224.23
      - address: 10.133.12.1
      - address: 10.140.160.8
      - address: 10.140.160.7
      - address: 10.140.160.6
      - address: 10.140.160.24
      - address: 10.140.160.22
      - address: 10.140.160.23
      - address: 10.140.160.21
      - address: 10.140.160.17
      - address: 10.140.160.9
      - address: 10.140.160.11
      - address: 10.140.160.12
      - address: 10.140.160.13
      - address: 10.140.160.15
      - address: 10.140.160.18
      - address: 10.140.160.14
      - address: 10.140.160.16
      - address: 10.140.160.10
      - address: 10.140.160.31
      - address: 10.140.160.29
      - address: 10.140.160.28
      - address: 10.140.160.30
      - address: 10.140.160.27
      - address: 10.140.160.25
      - address: 10.140.160.26
      - address: 10.140.160.19
      - address: 10.140.160.20
      - address: 10.140.160.33
      - address: 10.140.160.34
      - address: 10.51.0.7
      - address: 10.51.0.8
      - address: 10.51.0.6
      - address: 10.51.0.9
      - address: 10.51.0.1
      - address: 10.51.0.10
      - address: 10.51.0.21
      - address: 10.51.0.20
      - address: 10.133.36.1
      - address: 10.133.36.7
      - address: 10.133.36.6
      - address: 10.139.4.1
      - address: 10.139.4.6
      - address: 10.139.4.8
      - address: 10.139.4.9
      - address: 10.139.4.7
      - address: 10.133.32.1
      - address: 10.133.32.7
      - address: 10.133.32.6
      - address: 10.136.100.1
      - address: 10.141.32.1
      - address: 10.141.32.6
      - address: 10.141.32.7
      - address: 10.141.32.8
      - address: 10.141.36.1
      - address: 10.139.0.1
      - address: 10.139.0.6
      - address: 10.139.0.7
      - address: 10.139.0.8
      - address: 10.51.12.5
      - address: 10.51.12.1
      - address: 10.51.12.6
      - address: 10.51.12.7
      - address: 10.141.40.1
      - address: 10.141.40.3
      - address: 10.141.40.4
      - address: 10.141.40.6
      - address: 10.51.160.6
      - address: 10.51.160.1
      - address: 10.51.160.7
      - address: 10.51.160.8
      - address: 10.51.160.9
      - address: 10.51.164.1
      - address: 10.143.128.3
      - address: 10.143.128.5
      - address: 10.143.128.7
      - address: 10.143.128.6
      - address: 10.143.128.4
      - address: 10.143.128.8
      - address: 10.143.128.1
      - address: 10.51.4.1
      - address: 10.48.127.5
      - address: 10.51.16.1
      - address: 10.48.156.44
      - address: 10.53.20.1
      - address: 10.112.64.10
      - address: 10.112.64.15
      - address: 10.112.64.7
      - address: 10.112.118.4
      - address: 10.112.64.6
      - address: 10.112.64.8
      - address: 10.112.64.1
      - address: 10.112.64.12
      - address: 10.112.65.12
      - address: 10.112.65.14
      - address: 10.112.66.124
      - address: 10.112.66.14
      - address: 10.112.118.7
      - address: 10.112.118.5
      - address: 10.112.240.1
      - address: 10.112.156.5
      - address: 10.112.76.8
      - address: 10.112.76.5
      - address: 10.112.76.6
      - address: 10.112.76.7
      - address: 10.112.148.1
      - address: 10.112.72.9
      - address: 10.112.72.6
      - address: 10.112.72.7
      - address: 10.112.72.8
      - address: 10.112.72.1
      - address: 10.112.68.9
      - address: 10.112.68.8
      - address: 10.112.68.7
      - address: 10.112.68.1
      - address: 10.112.68.6
      - address: 10.112.68.10
      - address: 10.112.68.11
      - address: 10.112.68.12
      - address: 10.112.68.5
      - address: 10.112.152.1
      - address: 10.112.108.1
      - address: 10.113.0.1
      - address: 10.113.0.6
      - address: 10.113.0.7
      - address: 10.112.84.10
      - address: 10.112.84.8
      - address: 10.112.84.7
      - address: 10.112.84.9
      - address: 10.112.84.1
      - address: 10.112.84.6
      - address: 10.112.96.10
      - address: 10.112.96.9
      - address: 10.112.96.6
      - address: 10.112.96.11
      - address: 10.112.96.8
      - address: 10.112.96.7
      - address: 10.112.96.1
      - address: 10.112.96.12
      - address: 10.112.81.7
      - address: 10.112.80.9
      - address: 10.112.81.5
      - address: 10.112.82.14
      - address: 10.112.81.6
      - address: 10.112.80.1
      - address: 10.112.82.15
      - address: 10.112.82.10
      - address: 10.112.82.11
      - address: 10.112.82.12
      - address: 10.112.82.13
      - address: 10.112.82.6
      - address: 10.112.82.7
      - address: 10.112.82.8
      - address: 10.112.82.9
      - address: 10.112.80.20
      - address: 10.112.80.7
      - address: 10.112.80.11
      - address: 10.112.92.7
      - address: 10.112.92.11
      - address: 10.112.92.9
      - address: 10.112.92.10
      - address: 10.112.92.8
      - address: 10.112.92.6
      - address: 10.112.92.1
      - address: 10.112.132.1
      - address: 10.112.112.5
      - address: 10.112.112.1
      - address: 10.112.196.1
      - address: 10.112.184.1
      - address: 10.112.208.1
      - address: 10.112.248.1
      - address: 10.112.216.1
      - address: 10.112.220.1
      - address: 10.112.164.5
      - address: 10.112.192.1
      - address: 10.112.236.1
      - address: 10.112.204.1
      - address: 10.112.224.1
      - address: 10.112.168.1
      - address: 10.112.160.1
      - address: 10.112.172.1
      - address: 10.112.228.1
      - address: 10.112.212.1
      - address: 10.112.176.1
      - address: 10.112.140.1
      - address: 10.112.128.1
      - address: 10.112.100.7
      - address: 10.112.100.6
      - address: 10.112.100.1
      - address: 10.130.68.4
      - address: 10.130.68.1
      - address: 10.137.164.5
      - address: 10.137.164.1
      - address: 10.137.164.12
      - address: 10.130.64.1
      - address: 10.130.64.5
      - address: 10.130.64.13
      - address: 10.137.160.1
      - address: 10.137.160.6
      - address: 10.139.232.1
      - address: 10.139.228.1
      - address: 10.139.228.5
      - address: 10.139.224.1
      - address: 10.139.224.130
      - address: 10.48.196.1
      - address: 10.50.224.1
      - address: 10.20.224.6
      - address: 10.50.224.6
      - address: 10.51.64.2
      - address: 10.51.64.4
      - address: 10.51.64.6
      - address: 10.51.64.5
      - address: 10.51.64.3
      - address: 10.51.64.1
      - address: 10.51.64.8
      - address: 10.51.64.9
      - address: 10.51.68.1
      - address: 10.51.68.6
      - address: 10.130.104.1
      - address: 10.131.97.191
      - address: 10.131.97.192
      - address: 10.131.97.181
      - address: 10.131.97.180
      - address: 10.131.120.5

poller:
  usernameSecrets:
    - XXXXXX
 #   - sc4snmp-hlab-sha-des
  inventory: |
    address,port,version,community,secret,security_engine,walk_interval,profiles,smart_profiles,delete
    ciusss_group_commutateur,,3,,XXXXX,,1800,,,
sim:
  # sim must be enabled if you want to use signalFx
  enabled: false
#  signalfxToken: BCwaJ_Ands4Xh7Nrg
#  signalfxRealm: us0
mongodb:
  pdb:
    create: true
  persistence:
    storageClass: "microk8s-hostpath"
  volumePermissions:
    enabled: true
@omrozowicz-splunk
Copy link
Contributor

Ok, so the problem is that you have no data in Splunk at all? Both myindex_netops and myindex_netmetrics?
Did it work with smaller number of devices or you've never tried it?
Also, I know you mentioned that logs look fine, but do you have any logs: An error of SNMP isWalk in em_logs?

The only thing that is coming to my mind without knowing any context is that the devices are very big and SC4SNMP cannot walk it before it reaches the timeout. You can configure walk profile to limit the number of data -> https://splunk.github.io/splunk-connect-for-snmp/main/configuration/configuring-profiles/#walk-profile. In such a case you would see many Task splunk_connect_for_snmp.splunk.tasks.walk in logs, but nothing more.

This could be also a problem with snmpv3 secret, but I think If the secret is badly configured, you would see some Error in logs.

Last thing - when you run a helm upgrade command and then microk8s kubectl logs -f job/snmp-splunk-connect-for-snmp-inventory -n sc4snmp, what does it say? You should see many New record or Unchanged record.

@alithhero
Copy link
Author

First of all thanks for your help.
I just tried with a smaller number of device (7). I got data for 5/7 devices in my personal index, so it left two devices who didn't send data. All those devices are in snmpv3, so I can say that my secret configuration is correct.
I see nothing with An error of SNMP isWalk in the em_logs index ...

When you say "configure walk profile to limit the number of data" you mean, increase the walk interval, be more specific in the varBinds section or configure a small_walk ?

Yes, I can see a Unchanged record for my 7 devices in the inventory logs.

@omrozowicz-splunk
Copy link
Contributor

I meant be more specific in the varBinds section or configure a small_walk, if there's no An error of SNMP isWalk that means walk have a hard time to finish itself and that might mean the scope of walk is too big.
Overall, it is good idea to limit walk scope as you usually don't need to poll all the data. Try to create such a profile and add it to inventory, and let me know if anything has changed.

@alithhero
Copy link
Author

I think it is a part of the the problem, when I only use those two varBinds it works for my 300 appliances :
['SNMPv2-MIB', 'sysName'] ['SNMPv2-MIB', 'sysUpTime',0]

But data returned are not enough for my client so after analysed the MIBS, I would like to pull those varBinds :
['SNMPv2-MIB', 'sysName'] ['SNMPv2-MIB', 'sysUpTime',0] ['SNMPv2-MIB', 'sysLocation'] ['SNMPv2-MIB', 'sysDescr'] ['IF-MIB', 'ifIndex'] ['IF-MIB', 'ifDescr'] ['IF-MIB', 'ifType'] ['IF-MIB', 'ifSpeed'] ['IF-MIB', 'ifAdminStatus'] ['IF-MIB', 'ifInOctets'] ['IF-MIB', 'ifOutOctets']
And again, nothing comes to Splunk Index and nothing significant in the em_log index ...

@wojtekzyla
Copy link
Collaborator

Could you please check if you have any data in the myindex_netmetrics index, provided that you used varBinds which you mentioned? Some of those varBinds are sent to the Metric index https://splunk.github.io/splunk-connect-for-snmp/main/configuration/snmp-data-format/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@omrozowicz-splunk @wojtekzyla @alithhero