Filtering events doesnt work as expected

[saiharshitachava]

We Configured otel and pods are runnign fine in clusters

But when we check for logs in splunk for some reason queries are not working as expected

index=test "k8s.cluster.name"="poc2" "k8s.namespace.name"="kube-system" ---returns events

but when searched with below

index=test "k8s.namespace.name"="kube-system" --doesn't return anything

same goes with any other fields like k8s.* for filtering like pod name etc

it only works fine with soure or sourcetype as in like below

index=test source="/var/log/pods/kube-system_cluster-autoscaler-797ff5dd***/cluster-autoscaler/0.log"

Unfortunately with source it returns lot more events than any othe rworking query like with clustername and namesapce name so I assume thats indexing the data fine but something is wrong with the fields and filtering the data

[saiharshitachava]

I think fields selection is just doing a string match and not matching the fields

As in index=test "k8s.namespace.name"="kube-system" returns something if events has string called kube-system in it

[matthewmodestino]

Hi @saiharshitachava you will either need to configure fields.conf for the indexed fields as described here or here, or like we do in Splunk Cloud, set a feature called always_include_indexedfield_lispy

always_include_indexedfield_lispy = <boolean>
* Whether or not search always looks for a field that does not have
  "INDEXED = true" set in fields.conf using both the indexed and non-
  indexed forms.
* If set to "true", when searching for <field>=<value>, the lexicon is
  searched for both "<field>::<value>" and "<value>".
* If set to "false", when searching for <field>=<val>, the lexicon is
  searched only for "<value>".
* Set to "true" if you have fields that are sometimes indexed and
  sometimes not indexed.
* For field names that are always indexed, it is much better
  for performance to set "INDEXED = true" in fields.conf for
  that field instead.
* Default: false

You can also search with index field syntax k8s.cluster.name::foo to test and confirm.