Skip to content

Latest commit

 

History

History
62 lines (40 loc) · 3.6 KB

monitoring.md

File metadata and controls

62 lines (40 loc) · 3.6 KB
Raw

Monitoring

Sidecar container resource usage

Cloud Storage FUSE instances run inside sidecar containers and mount Cloud Storage buckets for your workload. To ensure the Cloud Storage FUSE instances run properly, it is important to monitor the sidecar container resource consumption. To learn more about how to configure the sidecar container resource allocation, see GKE documentation Configure resources for the sidecar container.

You can use GCP Metrics Explorer to check the sidecar container resource usage. Use the following filters:

Memory usage

Insufficient memory will cause Cloud Storage FUSE out-of-memory errors and crash the workload application. Ensure the sidecar container memory limit is large enough, or leave the memory limit unset to allow the Cloud Storage FUSE to consume all the available resources on a node.

  • Metric: Kubernetes Container - Memory usage (kubernetes.io/container/memory/used_bytes)

  • Filter:

    • memory_type = non-evictable
    • container_name = gke-gcsfuse-sidecar
    • pod_name = your-pod-name

  • For example: example of memory usage

CPU usage time

Insufficient CPU will cause Cloud Storage FUSE throttling and lead to unsatisfying performance. Ensure the sidecar container CPU limit is large enough, or leave the CPU limit unset to allow the Cloud Storage FUSE to consume all the available resources on a node.

  • Metric: Kubernetes Container - CPU usage time (kubernetes.io/container/cpu/core_usage_time)
  • Filter:
    • container_name = gke-gcsfuse-sidecar
    • pod_name = your-pod-name
  • For example: example of CPU usage

Cloud Storage bucket observability

To check metrics of Cloud Storage buckets, go to the bucket page, and click the OBSERVABILITY tab. For example: example of bucket metrics

Total read/list/get request count

This chart shows total requests issued by the Cloud Storage FUSE for Read, List, Get. If the GetObjectMetadata request is observed throughout your workload, consider enabling the Cloud Storage FUSE metadata cache and increasing the cache capacity. For more information, refer to the troubleshooting guide.

Data egress rate over the network

This chart presents an approximate representation of the object download speed from Cloud Storage FUSE. If the throughput is inadequate, you can refer to the performance troubleshooting steps for guidance on tuning Cloud Storage FUSE to improve its performance.

Cloud Storage FUSE metrics

Cloud Storage FUSE supports exporting custom metrics to Google cloud monitoring. Currently, these metrics are not available on GKE. GKE is working on integrating these metrics with the CSI driver.