Connect Axiom’s monitors to your favorite services

Axiom’s monitors allow you to build alerting rules on top of your event data. With our ready-made notifiers, connecting these alerts to your favorite services is easy. With a few clicks, you can trigger an incident on PagerDuty when your error rate spikes, or send a message to your sales channel in Slack when a new prospect signs up.

Slack and PagerDuty are examples of services where Axiom already has pre-built integrations. This means we handle translating our monitor alert output into a format compatible with each service’s API. But every day there are new products being released that we’d love to integrate with, and we know that Axiom users are at the cutting edge when it comes to testing out and integrating new services into their stacks.

Which is why we’ve released the custom webhook notifier type. When you link a custom webhook notifier to a monitor, a POST request will be sent to a URL of your choosing each time the monitor triggers. What’s contained in that POST request is up to you. This means the message body can be fully customized to match the format expected at its destination. You can use custom headers to handle authentication.

With custom webhooks, you can also leverage the expressiveness of the Go template syntax to conditionally transform the message body depending on the properties of the trigger event. Let’s look in detail at two examples of what’s now possible.

Using threshold monitors to create alerts in incident.io

We’re big fans of incident.io at Axiom, not least for their super-simple API. Here’s the two-step process for integrating with incident.io using custom notifiers.

1. In incident.io, create a new HTTP alert source. This provides you with a URL similar to https://api.incident.io/v2/alert_events/http/{unique-identifier} and an API token.

2. In Axiom, create a custom webhook notifier. Use the URL provided by incident.io as the webhook URL. Create a webhook header with the name Authorization and the value Bearer API_TOKEN where API_TOKEN is the token generated above.

Between the URL and the header details, you'll see Axiom’s default webhook body shown as an initial template to work with:

{
  "action": "{{.Action}}",
  "event": {
    "monitorID": "{{.MonitorID}}",
    "body": "{{.Body}}",
    "description": "{{.Description}}",
    "queryEndTime": "{{.QueryEndTime}}",
    "queryStartTime": "{{.QueryStartTime}}",
    "timestamp": "{{.Timestamp}}",
    "title": "{{.Title}}",
    "value": {{.Value}},
    "matchedEvent": {{.MatchedEvent}}
  }
}

If you’re not familiar with the Go template syntax, you might be wondering what all these extra brackets are about. All we’re doing here is including the various properties of the notification event as variables that will be specified when the monitor triggers. For example, {{.MonitorID}} will have the value of the corresponding monitor ID. The variables shown in the initial template are the complete set you have to work with. For more information on the possible values, see the custom webhook documentation.

Our goal is to reformat the webhook message to match what’s expected by incident.io. If we take a quick peek at their alerts documentation, we can see a sample request as follows:

{
  "deduplication_key": "4293868629",
  "description": "We've detected a number of timeouts on hello.world.com",
  "metadata": {
    "service": "hello.world.com",
    "team": []
  },
  "source_url": "https://www.my-alerting-platform.com/alerts/my-alert-123",
  "status": "firing",
  "title": "*errors.withMessage: PG::Error failed to connect"
}

The two interesting fields here are "deduplication_key" and "status". The first allows you to tell incident.io what alert you’re referring if you send a follow-up message, and the second is how you declare whether you’re triggering an alert or resolving it. Since each Axiom monitor comes with a unique identifier, and since we can access this with the variable {{.MonitorID}}, this is a great candidate for "deduplication_key". As Axiom distinguishes between trigger and resolution events using the {{.Action}}, we'll want to include this when defining the value of "status". However, the value of {{.Action}} is either "Open" or "Closed" depending on whether the monitor is triggering or resolving, whereas incident.io expects "status" to have the values "firing" or "resolved". Luckily, through the magic of Go templates, you can easily condition on the value of {{.Action}} to send the correct message to incident.io.

Pulling all this together, and setting some sensible entries for the other fields, we get the following body for our Axiom notifier:

{
  "title": "{{.Title}}",
  "description": "{{.Body}}",
  "deduplication_key": "{{.MonitorID}}",
  "status": "{{ if eq .Action "Open" }}firing{{ else }}resolved{{ end }}",
  "metadata": {
    "description": "{{.Description}}",
    "value": {{.Value}},
    "url": "https://app.axiom.co/{your-org-id-here}/monitors/{{.MonitorID}}"
  }
}

Using match monitors to create a Linear issue when a certain event occurs

Linear might seem like a strange place to send your monitor output, but it’s been a commonly requested notifier type ever since we launched match monitors. Whether you’re matching on an error message that needs rapid triage, or a sales opportunity that requires outreach, integrating with project management software like Linear can ensure you keep track of which events have been handled and by whom.

1. In Linear, create a new API token and make a note of it. For more information, see the Linear documentation.

2. Use Linear’s API to find the unique identifier for the team you want to create an issue for. For more information, see the Linear documentation.

3. In Axiom, create a new custom webhook notifier. Use https://api.linear.app/graphql as the webhook URL. Create a webhook header with the name Authorization and the value given by the token generated above.

For the webhook body, the only field expected by Linear is "query". Within the corresponding string value, you can define the various properties of the issue to be created, and just as in the example of incident.io, you can use the Go template syntax to pull in values from the notification to further customize the issue. In particular, the {{.MatchedEvent}} variable will allow you to pull in the matching event itself. You'll also need to include the team ID obtained above to ensure Linear is able to associate the issue with the correct team.

An example webhook body for creating Linear issues is shown below:

{
    "query": "mutation { issueCreate(input: { title: \"{{ .MatchedEvent.name }} form capture\", description: \"Prospect completed interest form with message: {{.MatchedEvent.message}}. Follow up per runbook https://linear.app/axiom/project/inbound-leads \", teamId: \"{your-team-id}\" }) { success issue { id title description } } }"
}

Get started with custom webhooks

We encourage you to explore custom webhooks today.

To try out custom webhooks, create an Axiom account and check out the documentation. Note that custom webhooks are only available on Axiom's Team and Enterprise plans.

Feel like you have unlocked an exciting new use case with your custom webhook? Why not share it over at our Discord community!