Grafana alerts to your iPhone with your own rules

Prerequisites

• A PagerDeck account with at least one paired device, so alerts have somewhere to land.
• A source in the dashboard with its credentials, see the first step below.
• Grafana Alerting with a webhook contact point.

How to set it up

1. 1

   Create a source

   In the dashboard, create a source with the slug grafana and copy the ingest key. You need the slug later in the rule.

2. 2

   Create the contact point

   In Grafana, open Alerting, then Contact points, and create a new one. Choose Webhook as the integration and enter the URL, method and Authorization header. Leave Optional Webhook settings and Custom Payload empty.

   Integration:                         Webhook
   URL:                                 https://api.pagerdeck.com/v1/integrations/alertmanager
   HTTP Method:                         POST
   Authorization Header - Scheme:       Bearer
   Authorization Header - Credentials:  src_live_REPLACE_ME

3. 3

   Test and set the severity

   Use Test in the contact point to send a sample alert. For the urgency to be right, give your alert rules a label severity with info, warning, error or critical. Without the label, an alert arrives as warning.

Payload

PagerDeck reads these fields on POST /v1/push. Only title is required, unknown fields are ignored.

{
  "title": "Disk almost full on web-01",
  "body": "/ is 94% used",
  "severity": "error",
  "tags": [
    "prod",
    "disk"
  ],
  "dedup_key": "web-01:disk",
  "url": "https://status.example.com/web-01",
  "ttl": "30m"
}

title

Required, at most 250 bytes.

body

Body text, at most 8 KiB.

severity

info, warning, error or critical. Without it the source default applies.

tags

At most 20, 64 bytes each.

dedup_key

Keeps an incident together until it is resolved.

url

A link for the app, http and https only.

ttl

Delivery deadline and automatic resolve, for example 30m.

Test it

Send a test alert straight to the API after setting PAGERDECK_KEY to your key. If it shows up in the app under your source, key and source are right and any problem sits in the sender configuration.

curl -sS -X POST https://api.pagerdeck.com/v1/push \
  -H "Authorization: Bearer $PAGERDECK_KEY" \
  -H "Content-Type: application/json" \
  -d '{"title":"PagerDeck test","body":"If you read this, the key works.","severity":"warning"}'

What the rules do with it

The sender proposes an urgency, your rule chain decides the alert level. This rule raises errors from the source grafana to critical and repeats until you acknowledge. Use the slug of your own source.

{
  "enabled": true,
  "stop": false,
  "when": {
    "source": [
      "grafana"
    ],
    "severity_at_least": "error"
  },
  "then": {
    "level": "critical",
    "repeat": {
      "every": "5m",
      "until": "ack",
      "max": 6
    }
  }
}

Set the volume

By source, title pattern, tags and time of day you can set a level from silent to critical.

Repeat until acknowledged

A rule repeats an alert at a fixed interval until you acknowledge it.

Escalate

If nobody reacts, PagerDeck notifies another address after a deadline.

What you should know

• The urgency comes from the severity label of the alert rule. Common values such as page, high, crit, minor or none are mapped to info, warning, error or critical, and unknown values become warning. What happens next is decided by your rules.
• Every alert is its own incident with the key am:fingerprint. If Grafana repeats the same group, it stays one incident. You control how often Grafana repeats in the notification policy, with Group interval and Repeat interval.
• A request takes at most 100 alerts. For large groups, set Max Alerts in the contact point to 100 or less.
• Android devices do not get push today.

Troubleshooting

When the API answers with a status code, it usually names the cause already.

202

Accepted and stored. This is not an error, delivery follows.

400

The input is invalid, for example a title over 250 bytes or malformed JSON. The message field gives the reason.

401

The key is missing, unknown or expired. Check the Authorization: Bearer header and that the key was copied completely.

402

The account may not send. Nothing is wrong with the connection, check the plan and billing in the dashboard.

413

The request is larger than 64 KiB. Limit the amount per request.

429

Sent too fast or quota used up. The Retry-After header says when to continue.

Frequently asked questions

Do I need a template or a custom payload?

No. Grafana's default webhook is enough. PagerDeck understands the same format as Prometheus Alertmanager and ignores fields it does not need.

Are resolved alerts closed?

Yes. When Grafana reports an alert as resolved, PagerDeck closes the incident with the same key. You no longer need a rule that silences resolved messages.

Why do I get a 400 response?

The request contains no alerts, more than 100 alerts, or no valid JSON. Set Max Alerts in the contact point to 100 or less.

Do I need one contact point per team?

No, one is enough. If you want to handle alerts from different teams separately, create one source per team with its own key and route the alerts with notification policies.

Does this work with Grafana Cloud?

Yes, as long as Grafana can reach api.pagerdeck.com over HTTPS. The alerts are then stored with PagerDeck on servers in Nuremberg.

Read next

• Connect Prometheus Alertmanager to PagerDeck

  Alertmanager sends its own JSON, and PagerDeck understands it directly. You only enter a URL and the ingest key.

• PagerDeck for IT operations

  Ten connected systems, one phone, and a rule chain that decides which of them is allowed to ring at night.

• PagerDeck as a PagerDuty alternative

  PagerDuty is incident management for organizations. PagerDeck is push alerting with rules for individuals and small teams.

• Push notifications via webhook and curl

  One HTTP request with JSON. No SDK, no library, and the all-clear does not cost you an alert.