How do you delete a counter in graphite whisper?
Solution 1
The reason so many people are struggling with this issue (me included) is because statsd (depending on its settings) continues to write data to old stats (whisper files) even after you delete the whisper file (.wsp).
Check your statsd settings at /etc/statsd/config.js (on Ubuntu). Example settings at https://github.com/etsy/statsd/blob/master/exampleConfig.js. I would recommend setting deleteIdleStats to true so that statsd does not write stats when they are not received.
The downfall of deleteIdleStats is that Graphite may now have null values for certain stats. This can be handled on the visualization layer (I recommend Grafana).
You have two options to stop regeneration of deleted whisper files:
- simply restart statsd with
service statsd restart
or - update statsd settings to not send stats to backend if the stats are not received AND then
service statsd restart
To delete wsp files that you don't need anymore:
- List the files to be deleted with
find pathToUnwantedStatistic -print
- if that is printing only the files you want deleted, run the following line to delete the unwanted files
find pathToUnwantedStatistic -delete
Solution 2
I had a similar problem where entries still existed in Graphite although no data was available. I am using statsd and it didn't have metrics in there either. It turns out I had to delete these folders in addition to the wsp files.
My exact case is, I used the machine name as prefix for statsd. Carbon created folder for each of the machines in e.g /opt/graphite/storage/whisper/stats/ip-0A215A08, where it stored the wsp for my metrics. Because of that, once I deleted the wsp files, I still had a bunch of "folders" in Graphite w/o any data that still showed up in the UI.
Checks on Carbon/Graphite
- .wsp files
- directories for the stats you got rid of
On statsd (if you are using it), verify there are no more gauges, timers or counters for the metrics you want to get rid of. For this, you may also telnet on your instance (default port 8126) and check for metrics with proper commands of the Admin interface instead of killing the process.
Related videos on Youtube
hafichuk
Updated on September 18, 2022Comments
-
hafichuk over 1 year
I have a counter at
stats.message.foo
and want to move it tostats.messages.foo
.I've updated my code to populate the new counter, however the old one still exists.
I've read all I need to do to remove a stat from graphite is to delete the appropriate whisper file on disk, however it seems that within a few seconds of removing the
wsp
it gets regenerated (without data).This is an annoyance if I want to rename the key that the data is stored under as I need to remember which key is the correct one.
Does anyone know how to permanently remove the old counter?
-
David Elner over 10 yearsDid you ever find an answer for this? Suffering from the same problem.
-
hafichuk over 10 years@dave Unfortunately no. It's been a while and we've move to a new graphite server so the issue has "solved" itself.
-