prometheus sum one NaN value result into NaN. How to avoid it?

Question

I have the following metrics in prometheus: it counts memcached command by sec:

sum (irate (memcached_commands_total{instance="memcached-instance"}[5m])) by (command)

Result:

{command="delete"}  0
{command="flush"}   0
{command="get"} 62.733333333333334
{command="incr"}    0
{command="set"} 93.43333333333334
{command="touch"}   NaN
{command="cas"} 0
{command="decr"}    0

I want to count commands by sec (without separate rate for different commands). I have tried the following formula:

sum (irate (memcached_commands_total{instance="memcached-instance"}[5m]))

But the result is:

{}  NaN

I expect about 155, but it is NaN. I suppose it is command="touch" the culprit. It is possible to exclude NaN from the sum?

Solution

I have figured it out:

sum (irate (memcached_commands_total{instance="memcached-instance"}[5m]) > 0)

returns the correct number. >0 does the trick.

Comments

[valyala]

Side notes: 1) it is better to use rate() instead of irate() - see this article; 2) VictoriaMetrics ignores NaN values in sum(), so it returns the expected result.

[Giovanni]

NaN arrives from memcached exporter. The following lines are copied from the exporter output: memcached_commands_total{command="touch",status="hit"} NaN memcached_commands_total{command="touch",status="miss"} NaN

[Giovanni]

I am monitoring a old memcached version (1.4.4) that doesn't support touch command, so memcached exporter report me NaN about touch statistics. Maybe this it is asked a million of times: why prometheus sum doesn't exclude NaN values aka sum in sql excludes null values? Thank you

[brian-brazil]

This is a bug in the exporter. NaN is not the same as null, a missing value is the equivalent in Prometheus.

[brian-brazil]

github.com/prometheus/memcached_exporter/issues/38 filed for this.

[Joel]

This is nice for counter ... but what if it was a gauge, with possible negative values? I'd like to do sum(my_metric{...} != NaN) but there's nothing as such as far as I know ...