How to monitor disk usage of persistent volumes?

Question

I want to monitor disk usages of persistent volumes in the cluster. I am using CoreOS Kube Prometheus. A dashboard is trying to query with a metric called kubelet_volume_stats_capacity_bytes which is not available anymore with Kubernetes versions starting from v1.12.

I am using Kubernetes version v1.13.4 and hostpath-provisioner to provision volumes based on persistent volume claim. I want to access current disk usage metrics for each persistent volume.

• kube_persistentvolumeclaim_resource_requests_storage_bytes is available but it shows only the persistent claim request in bytes

• container_fs_usage_bytes is not fully covers my problem.

Solution 1

Yes, in newest version of Kubernetes you cannot monitor metric such as kubelet_volume_stats_capacity_bytes, but there are some workarounds. Unfortunately this is a bit fragmented in Kubernetes today. PVCs may have capacity and usage metrics, depending on the volume provider, and it seems that any CSI based volume doesn't have these at all. We can do this on a best effort basis butit is simple to quickly hit cases where these metrics are not available.

First, just simply write your own script which will be every time values of metric like container_fs_usage_bytes are gathered will be count difference between capacity before measurement and container usage in bytes (metric will container_fs_usage_bytes be helpful).

Prometheus is quite popular solution but to monitor capacity especially disk usage you can use Heapster, now he is about to "retire", but just for this special case you can use it, but you will have to implement script too. Take look on repository: heapster-memory

"res.Containers = append(res.Containers, metrics.ContainerMetrics{Name: c.Name, Usage: usage})"

I hope it helps.

Solution 2

I have a job like the following in my prom config:

  - job_name: 'kubernetes-nodes'

    kubernetes_sd_configs:
    - role: node

    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

    relabel_configs:
    - action: labelmap
      regex: __meta_kubernetes_node_label_(.+)
    - target_label: __address__
      replacement: kubernetes.default.svc:443
    - source_labels: [__meta_kubernetes_node_name]
      regex: (.+)
      target_label: __metrics_path__
      replacement: /api/v1/nodes/${1}/proxy/metrics

With this job in place I see the following metrics available in Prometheus:

kubelet_volume_stats_available_bytes
kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_inodes
kubelet_volume_stats_inodes_free
kubelet_volume_stats_inodes_used
kubelet_volume_stats_used_bytes

More here: https://github.com/google/cadvisor/issues/1702

Solution 3

Per-PVC disk space usage in percentage can be determined with the following query:

100 * sum(kubelet_volume_stats_used_bytes) by (persistentvolumeclaim)
  /
sum(kubelet_volume_stats_capacity_bytes) by (persistentvolumeclaim)

The kubelet_volume_stats_used_bytes metric shows per-PVC disk space usage in bytes.

The kubelet_volume_stats_capacity_bytes metric shows per-PVC disk size in bytes.

Comments

[iridian]

I was looking at the container_fs_usage_bytes metric and while this is showing what's available on the container the issue is that a PVC can come from outside the container and outside the host system. For example, we're using NFS and the PV's from NFS aren't showing up in container_fs_usage_bytes.