How to configure alerts in Prometheus for diskspace
Solution 1
assuming you are using https://github.com/martinlindhe/wmi_exporter/blob/master/docs/collector.logical_disk.md you could use something along these lines for > 90 % use
- alert: DiskSpaceUsage
expr: 100.0 - 100 * (wmi_logical_disk_free_bytes / wmi_logical_disk_size_bytes) > 90
for: 10m
labels:
severity: high
annotations:
summary: "Disk Space Usage (instance {{ $labels.instance }})"
description: "Disk Space on Drive is used more than 90%\n VALUE = {{ $value }}\n LABELS: {{ $labels }}"
there are other examples on wmi_exporter repo for default node_exporter metrics ( not sure if available with windows ) it should be
- alert: DiskSpace10%Free
expr: 100 - (100 * node_filesystem_avail_bytes / node_filesystem_size_bytes) > 90
labels:
severity: moderate
annotations:
summary: "Instance {{ $labels.instance }} is low on disk space"
description: "diskspace on {{ $labels.instance }} is used over {{ $value }}% ."
´´´
Solution 2
To send email notification based on alert you need to setup alertmanager with prometheus. Here is the guide how to do that: https://github.com/prometheus/alertmanager
Also you can configure the alert rules. I am using node exporter to fetch node metrics and using the following rule
- alert: DiskSpace10%Free
expr: node_exporter:node_filesystem_free:fs_used_percents >= 90
labels:
severity: moderate
annotations:
summary: "Instance {{ $labels.instance }} is low on disk space"
description: "{{ $labels.instance }} has only {{ $value }}% free."
You can configure the above rule according to WMI exporter and you will be good to go. Hope this helps.
Solution 3
You might want to alert based on if it's going to fill up, not based on how full it is:
- name: node.rules
rules:
- alert: DiskWillFillIn4Hours
expr: predict_linear(node_filesystem_free{job="node"}[1h], 4 * 3600) < 0
for: 5m
labels:
severity: page
https://www.robustperception.io/reduce-noise-from-disk-space-alerts
Abraham Dhanyaraj Arumbaka
Updated on June 09, 2022Comments
-
Abraham Dhanyaraj Arumbaka almost 2 years
We have prometheus running on Win Server box, and WMI exporter on a separate box(client). Able to read client metrics in Prometheus. Now the requirement is the moment Diskspace =>90 % , send an email alert, so that we can run a job to clean up space using an automated job / manual job.
Could you please help on how to configure alert for diskspace >90
-
David over 4 yearsPresumably you are using some sort of custom metric here
node_exporter:node_filesystem_free:fs_used_percents
is not a core metric. You give no explanation about that. Perhaps you could? -
Brom558 almost 4 yearsUse of
for: 5m
good practice as this should avoid false positives where disk usage spikes but levels off i.e. backup clear down / creation