I created a label from our annotations on K8s Deployments named criticality but having a hard time in having the label only on 2 specific metrics. Here is the query I have on the PodMonitor:

      - action: replace
        sourceLabels: [__meta_kubernetes_pod_annotation_fire_layout_criticality]
        targetLabel: criticality
      - action: drop
        separator: ;
        sourceLabels: [__name__, criticality]
        regex: "(istio_request_duration_milliseconds_bucket$|istio_requests_total$).*"

The following query doesn't work. I currently only want the tag on istio_request_duration_milliseconds_bucket and istio_requests_total. Would appreciate any help on this.

I created a label from our annotations on K8s Deployments named criticality but having a hard time in having the label only on 2 specific metrics. Here is the query I have on the PodMonitor:

      - action: replace
        sourceLabels: [__meta_kubernetes_pod_annotation_fire_layout_criticality]
        targetLabel: criticality
      - action: drop
        separator: ;
        sourceLabels: [__name__, criticality]
        regex: "(istio_request_duration_milliseconds_bucket$|istio_requests_total$).*"

The following query doesn't work. I currently only want the tag on istio_request_duration_milliseconds_bucket and istio_requests_total. Would appreciate any help on this.

• prometheus

1 Answer 1

From your config, the replace rule is adding the criticality label to all metrics, and the drop rule is trying to remove it from the ones you don’t want. Instead of applying the label everywhere and then filtering, it’s better to add it only to the specific metrics you’re concerned about.

I’ll suggest you remove the drop rule and update the replace rule like this:

- action: replace
  sourceLabels: [__meta_kubernetes_pod_annotation_fire_layout_criticality, __name__]
  regex: "(istio_request_duration_milliseconds_bucket|istio_requests_total)"
  targetLabel: criticality

This way, the criticality label will only be added to istio_request_duration_milliseconds_bucket and istio_requests_total, without affecting other metrics.