In Grafana, I have an alert that sends notifications to Opsgenie. Each Opsgenie notification has a priority level, P1-P5, with P1 being the most important. To set this priority level for Opsgenie in Grafana you use the label og_priority.
Currently in Grafana I have a metric that I am monitoring that returns Unix Timestamps (Epoch time). These metrics are coming from Prometheus. An alert should send a P5 priority notification to opsgenie if the alert in Grafana triggers for a timestamp older than 3 minutes and P4 if the alert in Grafana triggers for a timestamp older than 5 minuntes. This should be done using the og_priority label.
Here is my Prometheus query:
(time() - date_event{role="worker"}) > 180
or
(time() - date_event{role="worker"}) > 300
Here are my expressions.
Here some things I have tried to make a dynamic label for og_priority.
Example 1:
{{ if gt $values.B.Value (time() - 5*60) }}P4{{ else if gt $values.B.Value (time() - 3*60) }}P5{{ end }}
Example 2:
{{ if gt $values.B (time() - 5*60) }}P4{{ else if gt $values.B (time() - 3*60) }}P5{{ end }}
Example 3:
{{ if gt (time() - date_event{role="worker"}) 300 }}P4{{ else if gt (time() - date_event{role="worker"}) 180 }}P5{{ end }}
Is it possible to create dynamic custom labels in Grafana? If so, how would I go about doing this?
If dynamic labelling is not possible, must I create separate alerts to trigger for timestamps older than 3 minutes and timestamps older than 5 minutes separately?