Grafana alert template language
I have talked about Prometheus and Grafana before. I currently have onse prom scraping two servers and one grafana where I have a few dashboards and many panels. Now I started using alers and it's working great: my biggest issues are too much wind (yes, really), too hot (also really) and disk space (always short).
So I'm getting a few alerts per day, currently mostly about weather. But by default, alerts look like this:
**Firing** Value: A=29.6, C=1 Labels: - alertname = Temperature - grafana_folder = Critical - instance = server:9100 - job = node - source = WG_IMARSE62 Annotations: Source: http://localhost:3000/alerting/grafana/fenz0y2eyv0u8b/view?orgId=1 Silence: http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dfenz0y2eyv0u8b&matcher=instance%3Dserver%3A9100&matcher=job%3Dnode&matcher=source%3DWG_IMARSE62&orgId=1 Dashboard: http://localhost:3000/d/be93c7dhu0lq8f?from=1750071590000&orgId=1&to=1750089925094 Panel: http://localhost:3000/d/be93c7dhu0lq8f?from=1750071590000&orgId=1&to=1750089925094&viewPanel=1 Value: A=29.8, C=1 Labels: - alertname = Temperature - grafana_folder = Critical - instance = server:9100 - job = node - source = yr_no Annotations: Source: http://localhost:3000/alerting/grafana/fenz0y2eyv0u8b/view?orgId=1 Silence: http://localhost:3000/alerting/silence/new?alertmanager=grafana&matcher=__alert_rule_uid__%3Dfenz0y2eyv0u8b&matcher=instance%3Dserver%3A9100&matcher=job%3Dnode&matcher=source%3Dyr_no&orgId=1 Dashboard: http://localhost:3000/d/be93c7dhu0lq8f?from=1750069190000&orgId=1&to=1750089925094 Panel: http://localhost:3000/d/be93c7dhu0lq8f?from=1750069190000&orgId=1&to=1750089925094&viewPanel=1
Many things are ugly here:
- Alert name and current value are not first on the text, so I have to open the contact point app instead of just reading it from the notification, which is faster.
- Links do not have the right domain name (this should be simple configuration error).
- It's too generic and includes more info than I actually need to action on these1, which could be confusing if you're awaken at 3AM after the baby has cried all night long and/or you probably hasd one or two beers more than you should have4.
So of course we have to customize the alerts2, but let's see the docs: Template notifications3. It describes a few can's and can't's7 and then BAM: an example. It shows what a rendered alert would look like, but utterly fails to explain the template language. That's I will try to do here.
The first thing you have to know about these templates is that they use the template language that comes in golang's stdlib5. You should start reading that, which is a lot already.
Second, you should be aware of the default templates that grafana provides, which comes from Grafana's alertmanager fork6, and that every Contact Point Integration type has its own template. See also the ["Built-in Templates" in the "Notification Templates" tab on the "Contact points" section in your Grafana]3(https://example.com:3000/alerting/notifications?search=&tab=templates) (change the server name and possibly port).
Third, you should know what data objects are available to those templates.
-
You might ask yourself, what do I do about weather conditions? I open or close windows and shutters as needed. This way I can ride a 30C+ day without any fans, much less and AC. ↩
-
Grafana calls them notifications. I think it's because it's trying to separate an alert of the messages it generates. ↩
-
I have to say, it's confusing that the page's title is "Template notifications", as if using "template" as a verb, I guess, but starts with "You can use notification templates to ...". ↩↩
-
This is how my boss describes you should write both code and documentation. I find it very illustrative. ↩
-
It could be (also) html/template, but both have the same basic laguage. ↩
-
No wonder the last time I tried to find where the templates are rendered in Grafana failed. It uses Prometheus'
alertmanager
underneath, which I already knew but forgot. ↩ -
How do you even write that... ↩