No data exported. Fails to load data. No dashboards in the list.

[ashokkm04]

Its a standalone deployed not included with viya deployment. I confirmed there is sufficient resource in the cluster,

Opensearch content is not exported. There is no data connection. No dashboards available in the dashboard list.

INFO Helm client version: 3.14.4
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
INFO Kubernetes client version: v1.27.12
INFO Kubernetes server version: v1.27.9

Deploying logging components to the [logging] namespace [Fri Jun 7 16:48:51 CEST 2024]

secret/internal-user-kibanaserver labeled
secret/v4m-root-ca-tls-secret created
secret/v4m-root-ca-tls-secret annotate
secret/v4m-root-ca-tls-secret labeled
secret/kibana-tls-secret created
secret/kibana-tls-secret annotate
secret/kibana-tls-secret labeled

"opensearch" has been added to your repositories
secret/v4m-osd-tls-enabled created
INFO Deploying OpenSearch Dashboards
Release "v4m-osd" does not exist. Installing it now.
NAME: v4m-osd
LAST DEPLOYED: Fri Jun 7 16:51:00 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:

1. Get the application URL by running these commands:
   https://myexample.com/dashboards
   INFO OpenSearch Dashboards has been deployed
   serviceaccount/v4m-osd-dashboards patched

secret/internal-user-admin labeled
secret/internal-user-logcollector labeled
secret/internal-user-metricgetter labeled
secret/es-transport-tls-secret created
secret/es-transport-tls-secret annotate
secret/es-transport-tls-secret labeled
secret/es-rest-tls-secret created
secret/es-rest-tls-secret annotate
secret/es-rest-tls-secret labeled
secret/es-admin-tls-secret created
secret/es-admin-tls-secret annotate
secret/es-admin-tls-secret labeled
secret/opensearch-cert-subjects created
secret/opensearch-cert-subjects labeled
configmap/run-securityadmin.sh created
configmap/run-securityadmin.sh labeled
"opensearch" has been added to your repositories

INFO Copying OpenSearch Security Configuration files from [/home/ashokm/viya4-monitoring-kubernetes/logging/opensearch/securityconfig]
secret/opensearch-securityconfig created
secret/opensearch-securityconfig labeled
INFO Deploying OpenSearch

Release "opensearch" does not exist. Installing it now.
NAME: opensearch
LAST DEPLOYED: Fri Jun 7 16:56:13 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Watch all cluster members come up.
$ kubectl get pods --namespace=logging -l app.kubernetes.io/component=v4m-search -w
INFO Waiting on OpenSearch pods to be Ready
pod/v4m-search-0 condition met
Looping: 0 Fri Jun 7 14:59:01 UTC 2024
RC: 0
| run_securityadmin.sh script starting [Fri Jun 7 14:59:01 UTC 2024]
| **************************************************************************
| ** This tool will be deprecated in the next major release of OpenSearch **
| ** opensearch-project/security#1755 **
| **************************************************************************
| Security Admin v7
| Will connect to localhost:9200 ... done
| Connected as "CN=es-admin,O=v4m"
| OpenSearch Version: 2.12.0
| Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
| Clustername: opensearch-cluster
| Clusterstate: GREEN
| Number of nodes: 3
| Number of data nodes: 3
| .opendistro_security index does not exists, attempt to create it ... done (0-all replicas)
| Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
| Will update '/config' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml
| SUCC: Configuration for 'config' created or updated
| Will update '/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml
| SUCC: Configuration for 'roles' created or updated
| Will update '/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml
| SUCC: Configuration for 'rolesmapping' created or updated
| Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml
| SUCC: Configuration for 'internalusers' created or updated
| Will update '/actiongroups' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml
| SUCC: Configuration for 'actiongroups' created or updated
| Will update '/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml
| SUCC: Configuration for 'tenants' created or updated
| Will update '/nodesdn' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml
| SUCC: Configuration for 'nodesdn' created or updated
| Will update '/whitelist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml
| SUCC: Configuration for 'whitelist' created or updated
| Will update '/allowlist' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/allowlist.yml
| SUCC: Configuration for 'allowlist' created or updated
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","internalusers","actiongroups","config"]) due to: null
| Done with success
INFO OpenSearch has been deployed

INFO Deploying Elasticsearch metric exporter ...

"prometheus-community" has been added to your repositories

Release "es-exporter" does not exist. Installing it now.
NAME: es-exporter
LAST DEPLOYED: Fri Jun 7 17:00:34 2024
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:

1. Get the application URL by running these commands:
   export POD_NAME=$(kubectl get pods --namespace logging -l "app=v4m-es-exporter" -o jsonpath="{.items[0].metadata.name}")
   echo "Visit http://127.0.0.1:9108/metrics to use your application"
   kubectl port-forward $POD_NAME 9108:9108 --namespace logging
   INFO Elasticsearch metric exporter has been deployed

INFO Loading Content into OpenSearch
ERROR Unable to identify the temporary port used for port-forwarding [v4m-search]; exiting script.
ERROR Exiting script [deploy_logging.sh] due to an error executing the command [logging/bin/deploy_opensearch_content.sh].

Resources deployed in logging namespace:

NAME READY STATUS RESTARTS AGE
pod/v4m-es-exporter-6854fd79f7-gl72g 1/1 Running 0 31m
pod/v4m-fb-459n2 0/1 Pending 0 11m
pod/v4m-fb-4n57z 1/1 Running 0 11m
pod/v4m-fb-9ztnh 1/1 Running 0 12m
pod/v4m-fb-kvrnl 1/1 Running 0 12m
pod/v4m-fb-lm79d 1/1 Running 0 12m
pod/v4m-fb-ncb6h 1/1 Running 0 12m
pod/v4m-fb-pkcl4 1/1 Running 0 12m
pod/v4m-fb-wqw25 1/1 Running 0 12m
pod/v4m-fb-zrg9p 1/1 Running 0 12m
pod/v4m-osd-78f8ddfc57-j98k5 1/1 Running 0 40m
pod/v4m-search-0 1/1 Running 0 35m
pod/v4m-search-1 1/1 Running 0 35m
pod/v4m-search-2 1/1 Running 0 35m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/v4m-es-exporter ClusterIP 10.0.3.144 9108/TCP 31m
service/v4m-fb ClusterIP 10.0.180.50 2020/TCP 12m
service/v4m-osd ClusterIP 10.0.253.147 5601/TCP 40m
service/v4m-search ClusterIP 10.0.121.64 9200/TCP,9300/TCP 35m
service/v4m-search-headless ClusterIP None 9200/TCP,9300/TCP,9600/TCP 35m

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/v4m-fb 9 9 8 2 8 12m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/v4m-es-exporter 1/1 1 1 31m
deployment.apps/v4m-osd 1/1 1 1 40m

NAME DESIRED CURRENT READY AGE
replicaset.apps/v4m-es-exporter-6854fd79f7 1 1 1 31m
replicaset.apps/v4m-osd-78f8ddfc57 1 1 1 40m

NAME READY AGE
statefulset.apps/v4m-search 3/3 35m

I also tried executing deploy_osd_conten.sh that fails with same response:

INFO Helm client version: 3.14.4
INFO Kubernetes client version: v1.27.12
INFO Kubernetes server version: v1.27.9
INFO Loading user environment file: /home/ashokom/viya4-monitoring-kubernetes/logging/user.env

INFO Configuring OpenSearch Dashboards...this may take a few minutes
INFO Waiting for OpenSearch Dashboards pods to be ready (Fri Jun 7 17:29:02 CEST 2024 - timeout 10m)
pod/v4m-osd-78f8ddfc57-j98k5 condition met
INFO Waiting (up to more 8 minutes) for OpenSearch Dashboards API endpoint to be ready
ERROR Unable to identify the temporary port used for port-forwarding [v4m-osd]; exiting script.
ERROR Unable to obtain the URL for the OpenSearch Dashboards API Endpoint

[ashokkm04]

[ashokkm04]

FYI this specifically happens only in WSL environment.

[gsmith-sas]

@ahsokm07 We are sorry you are running into this problem. Thank you for including the entire set of log messages generated by the script, that is very helpful.

As a next step, we will have you run the deploy_opensearch_content.sh script (NOT the deploy_osd_content.sh script) in the logging/bin directory of the project repo. But, please set some environment variables to enable some additional debugging. Please set the environment variable LOG_DEBUG_ENABLE to 'true'. This will generate some DEBUG-level messages that might be helpful. In addition, please set the environment variable KEEP_TMP_DIR to 'true'. This will keep some temporary files generated during the deployment process around so we can review them after the script completes execution. You can set these environment variables and run the script in a single command using the following syntax: LOG_DEBUG_ENABLE=true KEEP_TMP_DIR=true ./logging/bin/deploy_opensearch_content.sh Please share the output generated (including all messages) when the script executes.

In addition, one of the last messages displayed will be something like this: INFO TMP_DIR [/tmp/sas.mon.jq0ztxIp] was not removed. Please share the contents of the output.txt file found in the temporary directory identified in the message.

Finally, the error messages are reporting a problem when attempting to use Kubernetes port-forwarding capability. Please try the following command to see if any helpful error messages are generated when you attempt to use port-forwarding manually:
kubectl -n logging port-forward --address localhost svc/v4m-search :9200
If the command executes successfully, you will see a couple of messages like the following:

$ kubectl -n logging port-forward --address localhost svc/v4m-search :9200
Forwarding from 127.0.0.1:37813 -> 9200
Forwarding from [::1]:37813 -> 9200

You can terminate the port-forwarding by pressing Ctrl-C which should return you to the command line. Please let us know if that command succeeded or failed and share any error messages generated.

I regularly use WSL with an Ubuntu image in my work, so I don't think WSL is the issue. Please let me know if you are using WSL with a different image.

Regards,
Greg

[gsmith-sas]

@ashokkm04 We're you able to resolve this issue or is it still a challenge? If you're still facing it, see my last note for details of information that would help us figure things out. If things are resolved, let me know and I'll close out this issue.