# PagerDuty

> Tell reviewers when a pull request touches a service with an open PagerDuty incident, page on critical findings, and post suspect pull requests to chat when an incident starts.

MergeHarbor works with PagerDuty in three ways. Each one works on its own:

| Feature                                                                 | What it does                                                                                                                           | You need                                          |
| ----------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- |
| [Incident context](#incident-context)                                   | Reviews and pull request pages show open incidents on the services a repository runs as, and incidents that started soon after a merge | A read-only PagerDuty API key                     |
| [Alerts to PagerDuty](#alerts-to-pagerduty)                             | Critical findings or failures, such as indexing errors, trigger a PagerDuty incident                                                   | An Events API v2 integration key                  |
| [Suspect pull requests](#suspect-pull-requests-when-an-incident-starts) | When an incident starts, the pull requests recently merged to the affected repositories are posted to chat, riskiest first             | Incident context, a PagerDuty webhook, a chat app |

None of these uses a model or credits.

## Before you begin

- You need the **`settings.manage`** permission in MergeHarbor.
- In PagerDuty, creating an account API key needs the **Admin** or **Account Owner** role. Adding an integration to a service needs permission to manage that service.
- Know your PagerDuty region: **US** (`app.pagerduty.com`) or **EU** (`app.eu.pagerduty.com`).

## Incident context

### Connect

1. In PagerDuty, open **Integrations → API Access Keys** and choose **Create New API Key**. Name it MergeHarbor, check **Read-only API Key**, and copy the key. A read-only user API token (**My Profile → User Settings → Create API User Token**) also works, but MergeHarbor then sees only what that user sees.
2. In the console, open **Integrations** and choose **Connect** on **PagerDuty**, under **Observability**.
3. Paste the key, choose your region, and connect. MergeHarbor checks the key by listing one of your services.

### Map repositories to services

Open **Settings** on **PagerDuty** and map each repository to the PagerDuty services it runs as (up to 20 per repository). Only mapped repositories get incident context. A service ID looks like `PABC123`; it is the last part of the service's URL in PagerDuty.

### How it works

- **In reviews.** A review of a mapped repository is told about open (triggered or acknowledged) incidents on its services, up to two, highest priority first. The summary notes them, for example: "This change touches service `checkout-api`, which has an open PagerDuty incident #812 P1: Payment errors above 5%." No extra model call is made.
- **On pull request pages.** A PagerDuty panel lists incidents on the repository's services: ones that started within 24 hours after the pull request merged, marked as possibly related, and ones still open. Up to 20 are shown.
- **Freshness.** MergeHarbor keeps a snapshot of your services and the incidents of the last 7 days, refreshed in the background at most every 15 minutes when a review or pull request page needs it. Reviews never wait for PagerDuty.

### What is read and stored

| Read                                                                                              | Not read                                             |
| ------------------------------------------------------------------------------------------------- | ---------------------------------------------------- |
| Services: ID, name, link                                                                          | Incident notes, timeline and log entries             |
| Incidents of the last 7 days: number, title, status, urgency, priority, service, start time, link | Responders, on-call schedules, users, alert payloads |

Incident titles are redacted of secrets. The snapshot is replaced at each refresh. The API key is encrypted at rest and never shown again. **Test** on the PagerDuty card checks that the key can still list services.

## Alerts to PagerDuty

1. In PagerDuty, open the service that should be paged, then **Integrations → Add an integration → Events API V2**. Copy the **Integration Key**.
2. In the console, open **Settings → Alerts & webhooks** and add a destination of type **PagerDuty**. Paste the integration key and choose your region.
3. Add rules that choose what pages people, for example **Findings posted** with minimum severity **critical**, and **Indexing failed**.
4. Turn the destination on. PagerDuty destinations start turned off, so nobody is paged while you set up the rules.

Add one destination for each PagerDuty service you want to page.

| Event                                                          | PagerDuty severity |
| -------------------------------------------------------------- | ------------------ |
| Findings posted, with a critical finding (including security)  | Critical           |
| Findings posted, other findings                                | Warning            |
| Review failed, indexing failed, code host connection unhealthy | Error              |
| Reviews paused, webhooks stopped arriving                      | Warning            |
| Weekly digest, test                                            | Info               |

Each MergeHarbor event opens at most one incident: retries reuse the same deduplication key. Each **Send test** opens a new incident, so resolve it in PagerDuty afterwards. The incident links back to the review in MergeHarbor.

## Suspect pull requests when an incident starts

When this is on and PagerDuty triggers an incident on a mapped service, MergeHarbor posts a short list to the chat channels mapped to that service's repositories in the [Slack](https://mergeharbor.com/docs/slack), [Microsoft Teams](https://mergeharbor.com/docs/microsoft-teams), [Google Chat](https://mergeharbor.com/docs/google-chat) and [Discord](https://mergeharbor.com/docs/discord) apps: up to 10 pull requests merged to those repositories in the last 24 hours (you can choose 1 to 168 hours), with their [change risk](https://mergeharbor.com/docs/change-risk) scores, highest risk first.

PagerDuty asks for the webhook URL before it shows the signing secret, so you create the URL in MergeHarbor first.

1. Connect PagerDuty and map repositories to services, as above.
2. In the console, open **Settings** on **PagerDuty** and create the webhook URL. Copy it.
3. In PagerDuty, open **Integrations → Generic Webhooks (v3)** and choose **New Webhook**. Paste the URL, choose the account or a mapped service as the scope, and subscribe to at least **incident.triggered**. Save, then copy the **signing secret**. PagerDuty shows it only once.
4. Back in MergeHarbor, paste the signing secret.
5. Turn on **suspect pull requests** and choose the time window.
6. Send a test event from the webhook in PagerDuty. The PagerDuty settings show when the last event arrived and its type.

Example of what is posted:

```text
PagerDuty incident #812 (P1) on checkout-api: Payment errors above 5%
3 pull requests merged to this service's repositories in the last 24 hours, highest risk first. Listed from merge times and stored risk scores only (no model was used).
Risk 82 (high)    acme/checkout #1290 Retry card authorisation on timeout — merged 2 h ago
Risk 41 (medium)  acme/checkout #1287 Update currency rounding — merged 9 h ago
Risk not scored   acme/payments #455 Bump client library — merged 20 h ago
[Open incident]
```

Good to know:

- Each incident is posted once, even if PagerDuty sends the event again.
- Posts go to channels mapped to the repository by channel ID, up to 20 channels per incident. Wildcard mappings such as "all public channels" are not used.
- If nothing was merged in the window, the post says so.
- MergeHarbor checks PagerDuty's signature on every request and rejects anything unsigned or signed with another secret.
- To rotate the secret, regenerate it in PagerDuty and paste the new one. The URL stays the same.

## Disconnect

Open **PagerDuty** in **Integrations** and choose **Disconnect**. MergeHarbor deletes the API key, the service mapping, the snapshot and the webhook secret. It cannot revoke a PagerDuty key, so it links you to PagerDuty's API key page: delete the key there, and delete the webhook subscription. Alert destinations are separate: remove them under **Settings → Alerts & webhooks**.

## Troubleshooting

| Problem                                    | What to do                                                                                                                            |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| PagerDuty is not listed under Integrations | On self-hosted installations, set `PAGERDUTY_ENABLED=true`                                                                            |
| The key is refused                         | Use a REST API key (not an Events integration key), check it was not deleted, and check the region                                    |
| Reviews never mention incidents            | Map the repository to its services. Only triggered or acknowledged incidents are mentioned                                            |
| The alert destination never pages          | Turn it on, check the region, and check a rule sends events to it                                                                     |
| PagerDuty shows webhook deliveries failing | 404: the URL was not created in MergeHarbor, or the signing secret was not pasted yet. 401: the secret does not match; paste it again |
| No suspect pull requests are posted        | Turn the feature on, map the incident's service to a repository, and map a chat channel to that repository by channel ID              |

## Self-hosted installations

PagerDuty needs no OAuth application: set `PAGERDUTY_ENABLED=true` on the API and workers to offer it in the console. `PAGERDUTY_SYNC_INTERVAL` (default `15m`) sets how often the snapshot may refresh. Alerts to PagerDuty work without either.
