# Receive Alert Notifications via Pagerduty

> Setup Pagerduty integration and receive alert notifications from Last9.

Source: https://last9.io/docs/pagerduty-integration/

## Getting started

Last9 can send alert notifications and resolutions to Pagerduty. This document
lists the step by step instructions on how to setup Pagerduty integration with Last9
and start receiving alert notifications.

:::tip
Last9 integrates with PagerDuty [Events V2 API](https://developer.pagerduty.com/docs/events-api-v2/overview/) to send alert notifications to PagerDuty.
:::

The Pagerduty Events V2 API is a highly reliable, highly available asynchronous API that
ingests machine events from monitoring tools and other systems like Last9.
Events sent to this API are ultimately routed
to a PagerDuty service and processed.

## Setting up an Events API V2 integration in Pagerduty

[Create an integration on any PagerDuty service](https://support.pagerduty.com/docs/services-and-integrations/#section-events-api-v2)

![Add new Pagerduty Integration](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/pagerduty-add-integration.png)

Select **Events API V2** as the Integration Type.

![Select the Events API Integration](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/pagerduty-events-api.png)

Copy the integration key and keep it handy as we will use it while creating a notification channel in Last9.

![Copy Events API Key](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/pagerduty-events-key.jpg)

## Setting up a notification channel in Last9

1.  In [Notification Channels](https://app.last9.io/channels), add a new PagerDuty channel

![Create Notification Channel](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/add-notification-channel.png)

2.  Add the API key copied from Pagerduty integration in the `Integration Key` field

![Create Pagerduty Channel](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/create-pagerduty-channel.png)

3.  By default it is not assigned to any alert group. You can assign it to the
    alert group either via IaC flow or manually once alert rules are added
    to an alert group.

![Notification Channels](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/notification-channels.png)

## Assigning a notification channel to an alert group

First navigate to the alert group and click on Pagerduty icon.

![Set notification channel](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/set-notification-channel-for-an-alert-group.png)

Select from the dropdown which Notification channel you want to select from the dropdown.

![Pagerduty channel](../../../../../assets/content/docs/alerting/guides/pagerduty-integration/pagerduty-channel-for-alert-group.png)

:::tip
You can set different Pagerduty channels for `threat` and `breach` notifications depending on the severity of the alerts.
:::

---

## Notification Payload

Use these JSON fields for custom incident details, automation, alert enrichment, or integrating with other tools.

| PagerDuty field        | Type             | Description                                                              |
| ---------------------- | ---------------- | ------------------------------------------------------------------------ |
| payload                | object           |                                                                          |
| payload.summary        | string           | Title for the incident                                                   |
| payload.timestamp      | timestamp        | The ending time of this alert, in ISO 8601 format                        |
| payload.severity       | string           | critical / warning for alerts marked as breach/threat in alert rule      |
| payload.source         | string           | Dedup key for the incident                                               |
| payload.component      | string           | Empty                                                                    |
| payload.group          | string           | Dedup key for the incident                                               |
| payload.class          | string           | Alert Rule Type                                                          |
| payload.custom_details | object           | Described below                                                          |
| routing_key            | string           | PagerDuty integration key                                                |
| event_action           | string           | 'trigger' for active notifications, 'resolve' for resolved notifications |
| dedup_key              | string           | Dedup key for the incident                                               |
| client                 | string           | "Last9 Dashboard"                                                        |
| client_url             | string           | Link to health dashboard for the alert in Last9                          |
| links                  | array of objects | Empty array                                                              |
| images                 | array of objects | Empty array                                                              |

### Custom Details

- `alert_condition` - Condition set on alert. Static alerts, it is of the
  format.`expr > 10` based on the threshold configured. For pattern-based
  alerts, it is of the format `algo_type(tunable, expr)`. For example, for a
  high spike alert set with tunable 3, this would be `high_spike(3, expr)`
- `algo_type` - Type of alert (`static_threshold`, `increasing_changepoint`
  etc)
- `client_url` - Link to the health dashboard for this alert on Last9
- `description` - Description of the alert. If a description is provided while
  configuring the rule, it appears here. Otherwise, a default description based
  on the algorithm, indicator, and entity is shown
- `start` - Starting time of this alert, in ISO 8601 format
- `end`- Ending time of this alert, in ISO 8601 format
- `expression` - Name of the indicator
- `entity_name` - Entity name
- `entity_type` - Entity type
- `entity_team` - Entity team. Is `None` if not assigned
- `entity_tier` - Entity tier. Is `None` if not assigned
- `entity_workspace` - Entity workspace. Is `None` if not assigned
- `entity_namespace` - Entity namespace. Is `None` if not assigned
- `severity` - Severity of the alert (`breach`/ `threat`)
- `notification_call` - Whether this alert is sent for the first time or
  repeated (`first`/ `repeat`)
- `runbook` - Link to the runbook for this alert (has to be configured while
  setting up alert). This key is omitted if the runbook isn’t configured
- If the entity under alert has `tags` associated with it, they are included in
  custom details as `tag_<tag_name>` = `true`
- `time_in_alert` - Duration for which this alert was observed. E.g., 8 in 10
  minutes.

---

## Troubleshooting

Please get in touch with us on [Discord](https://discord.com/invite/Q3p2EEucx9) or [Email](mailto:support@last9.io) if you have any questions.
