Relabeling in Prometheus

Question

Setup

Prometheus node exporter is registered as a service with consul agent with various tags. Example service definition provided to consul agent:

{
  "service":{
      "id": "server-stats",
      "name": "server-stats",
      "tags": [
        "a=1_meow",
        "b=2_woof",
        "c=3_moo",
        "monkey"
      ],
      "port": 9100,
      "checks": [
        {
          "name": "Process #1",
          "script": "/path/to/healthcheck/script.sh",
          "interval": "5s"
        }
      ]
    }
}

Prometheus is set to look for this server-stats service and use the configuration (host address and port) provided by Consul to scrape stats from servers. The above tags are available as a comma separated list in __meta_consul_tags that can be used for relabeling.

Prometheus relabeling configuration:

relabel_configs:
- source_labels: [__meta_consul_tags]
  separator:     ','
  #regex:         '(.+)=(.+)'
  regex:         '([a-z_]+)=([a-z_]+|\d+)'
  target_label:  ${1}
  replacement:   ${2}

Issue

I am trying to expose tags to Prometheus so that we can get stats and graphs based on labels. Keeping the above service configuration in mind, I would like each metric to have following labels in addition to whatever Prometheus does internally: a=1_meow, b=2_woof, c=3_moo and ignore monkey because it is just a string. I can remove monkey from my list of tags if there is a solution that requires = to be there. The relabel configuration written above is not leading to exposing any tag at all and seems to be getting ignored. Running Prometheus with log level set to debug is also not yielding anything.

Relevant docs

• https://prometheus.io/docs/operating/configuration/#%3Crelabel_config%3E
• https://www.robustperception.io/extracting-full-labels-from-consul-tags/

Solution 1

Incorrect understanding

I think there was a mistake in my understanding of how labeling in prometheus works. My incorrect understanding was:

1. before applying regex, string would be first split on separator (otherwise what is its purpose?),
2. each substring has regex evaluated against it,
3. if match groups are declared and found, they will be available as indexed values available to use in target_label and replacement fields.
4. if regex does not match, then that substring will be ignored.
5. because regex is expected to be applied to each substring after the split, it will lead to multiple labels from multiple substrings.

Correct understanding

However, from brian-brazil's post linked in his answer and Prometheus's documentation, it seems the following happening:

1. All __meta tags are combined into one long separator separated line.
2. regex is applied on that line only once.
3. If regex matches and includes groups, they are indexed beginning from 1 and available for use in target_label and replacement.
4. separator seems to be getting ignored in this section even if you mention it.

Config from corrected understanding

From this idea and following from example in the question, I was able to make the following config that works

relabel_configs:
- source_labels: [__meta_consul_tags]
  regex:         '.*,a=([a-z0-9_]+),.+'
  target_label:  'a'
  replacement:   ${1}

- source_labels: [__meta_consul_tags]
  regex:         '.*,b=([a-z0-9_]+),.+'
  target_label:  'b'
  replacement:   ${1}

- source_labels: [__meta_consul_tags]
  regex:         '.*,c=([a-z0-9_]+),.+'
  target_label:  'c'
  replacement:   ${1}

- source_labels: [__meta_consul_tags]
  regex:         '.*,d=([a-z0-9_]+),.+'
  target_label:  'd'
  replacement:   ${1}

Caveats

I believe both approaches (the approach brian-brazil wrote in his blogpost, and what I am using above) have caveats - we either need to know all the labels we want beforehand, or have a set number of them. This means if a developer wants to associate different, or more labels with his/her service, s/he would need to work with ops as general flow will not be able to handle it. I think it is a minor caveat that should be addressed.

Solution 2

https://www.robustperception.io/extracting-full-labels-from-consul-tags/ shows how to do this, in particular the last example.

Comments

[Admin]

That link is already included in relevant docs in my original question. It didn't get what i was hoping for.

[brian-brazil]

The link covers exactly what you want to do, you need to use it as written - not the altered version you posted.

[Admin]

yes... i see now why that needs to be done. I have included my understanding and what i was working off of in my answer. Please correct me if I am wrong somewhere in my answer.