Alert notifications

Send copies of your alerts to Telegram, Slack, email or your own HTTPS webhook, choose which alerts go where, and check every delivery.

1. What notifications do

Every alert is shown in the dashboard and the API (see Monitoring and alerts). Notifications send a copy of chosen alerts to channels outside PerkBridge:

• Telegram, through a bot you create;
• Slack, through an incoming webhook;
• Email, through your own SMTP server;
• HTTPS webhooks, signed so your service can check they came from PerkBridge. Paging services can be reached this way.

A separate process on the control plane host, nile-notifier, makes every call to a provider. It has no database access and no keys of its own. The control plane decides what to send, stores each delivery and writes every attempt to the audit trail.

Notifications never touch trading. If the notifier stops, the control plane keeps serving, your servers keep trading, and deliveries wait in the database for up to 6 hours. The dashboard then shows the notifier as “not connected”.

2. Before you start

The installer sets notifications up for you. On its first run it creates the notification secrets key, which seals the channel secrets you enter, and it runs the notifier as part of the Docker Compose stack. An installation made before this feature gets the key the next time you run install with a newer bundle.

• Webhooks are off until you list the hosts they may post to. In /var/lib/nile/compose/.env, set for example NILE_NOTIFIER_WEBHOOK_HOSTS=hooks.example.com,alerts.example.org (at most 100 hosts), then restart the stack.
• Messages link to the alert on your dashboard. The link uses the dashboard address you gave the installer.
• The control plane host needs outbound access to the providers you use (section 3).

Reading channels, rules and the delivery log needs notification:read, which is in the OPERATOR bundle. Every change needs notification:manage. It is in no bundle and is granted by name, for organization administrators. See Users and single sign-on.

3. Add a channel

A channel is one place alerts can go. Add one for each place you want copies sent. You need notification:manage for every change on these pages.

1. In the sidebar, under NOTIFICATIONS, choose Channels (1).
2. Choose Add channel (2). A form opens above the list.
3. In Type, choose Telegram, Slack, Webhook or Email, then follow the matching part below.

Every channel form has a Name and a box Include server and deployment names and the alert description (they leave Nile). It is off by default. Leave it off if you do not want server names and alert text to reach the provider. The password, token or URL you type is stored encrypted and is never shown again.

Telegram

You need a Telegram bot and a chat the bot can post to. Create the bot with Telegram’s BotFather, which gives you a bot token, add the bot to your group or channel, and note the chat ID.

1. Choose Telegram in Type (1).
2. Type a Name (2), for example “Ops on-call”.
3. Type the Chat ID (3): a number such as -1001234567890, or an @channel name.
4. Paste the Bot token (4). It looks like 123456789:AA….
5. Choose Add channel (5).

Slack

Create an incoming webhook for the Slack channel you want, in Slack’s own app settings, and copy its URL.

1. Choose Slack in Type (1).
2. Type a Name (2).
3. Paste the Incoming webhook URL (3). It must start with https://hooks.slack.com/services/.
4. Choose Add channel (4).

Email

You need an SMTP server on the public internet, with a valid TLS certificate, and an account on it that may send mail.

1. Choose Email in Type (1) and type a Name (2).
2. Type the SMTP server name (3) and choose the Port (4): 587 (STARTTLS) or 465 (TLS).
3. Type the SMTP user name (5), the From address (6) and the Recipients (7): up to 10 addresses, separated by commas.
4. Type the SMTP password (8), then choose Add channel (9).

Webhook

Your receiving service needs an https:// address on a host you listed in section 2. The form shows the allowed hosts under the URL box.

1. Choose Webhook in Type (1) and type a Name (2).
2. Type the URL (3). Never put a secret in the URL.
3. Choose Add channel (4). PerkBridge then shows the signing secret once. Copy it, store it where your service can read it, and choose I saved it. You cannot see it again. Section 6 shows how your service uses it.

After you add a channel

Each channel row has four buttons.

1. Choose Send test (1) for the new channel. Then open Delivery log (section 5): a “Test” row should reach DELIVERED. Do this before you rely on the channel.
2. Disable (2) pauses a channel and Enable resumes it. Replace secret (3) lets you type a new token, password or URL. For a webhook, the key button rotates the signing secret and shows the new one once. Remove (4) deletes the channel and cancels what is still waiting for it.

The state column shows Enabled, Failing or off, and the last result of the channel.

The notifier refuses private, loopback, link-local and other non-public addresses, follows no redirects and never skips TLS checks. That keeps a channel from being used to reach your internal network.

4. Choose which alerts go where

A channel receives nothing until a rule points to it. A rule sends matching alerts to one or more channels.

1. Under NOTIFICATIONS, choose Rules, then Add rule.
2. Type a Name (1).
3. Choose the Minimum severity (2): Info and above, Warning and above, or Critical only.
4. Under Channels (up to 10) (3), tick every channel that should receive these alerts. You must tick at least one.
5. Under Alerts (none chosen: all) (4), tick the categories you want: server, deployment, credentials, notifications or license. Tick none to match all alerts.
6. Under Send when an alert is (5), tick opened, escalated to a higher severity and resolved as you need. Tick at least one.
7. Set the Quiet period after a reopening (minutes) (6). An alert that reopens within that time after a sent opening is not sent again. 0 sends every reopening.
8. Choose Save rule (7).

Rules can also be limited to exact alert types, servers, deployments or trading accounts through the API. The form keeps those limits when you edit a rule.

In the rules list, use the pencil to edit a rule, the pause button to turn it off and on, and the bin to remove it.

Only alerts raised after the first rule is saved are sent. Older alerts stay in the dashboard. A “resolved” message is sent only after the “opened” message for the same alert was attempted on that channel.

5. Check deliveries

Under NOTIFICATIONS, choose Delivery log. It lists each notification and its last attempt. Messages and secrets are not kept, only the outcome.

1. Use the State filter (1) to show only one state, for example FAILED.
2. Read the State column (2). Under a failed or cancelled row, a short reason is shown.
3. Read Last result (3) for the provider’s answer, for example http_502. The Alert link in the “What” column opens the alert that caused the message.

Delivery is at least once. If a provider may have received a message but its answer was lost, the attempt is marked “outcome unknown” and a duplicate is possible. Failed attempts are retried with growing waits, at most 15 times within 6 hours. Each organization can create at most 120 new deliveries a minute; anything over that is counted and shown as not sent.

Two alerts watch the notifications themselves: NOTIFICATION_CHANNEL_FAILING when a channel fails permanently or three times in a row, and NOTIFIER_UNAVAILABLE when the notifier has not been connected for 2 minutes while channels are on.

6. Verify a webhook

Each webhook request carries three headers: X-Nile-Timestamp, X-Nile-Delivery-Id and X-Nile-Signature. The signature is v1= followed by the hex HMAC-SHA256 of the timestamp, a dot and the raw body, keyed with the channel’s signing secret. Check it before you trust the request, and use the delivery ID to drop duplicates.

7. Backups, restores and the key

• A backup holds channel secrets only in sealed form, and never the notification secrets key. Keep a copy of the key apart from your backups if secrets should survive a move to a new host. Anyone who holds both the key and a backup can read every channel secret.
• After a restore, notifications are held. Unfinished deliveries are cancelled and nothing is sent until an administrator reviews the channels and rules and confirms “I reviewed them; resume”.
• A channel whose secret was sealed with another key, for example after a restore on a new host, is turned off as “Needs a new secret”. Enter the secret again on the Channels page.
• If the key file is lost or unusable, the control plane keeps running with notifications off. Copy the saved key back and restart the control plane; nothing is lost.

See Backup and restore.

8. If something is wrong

External notifications are offered with the paid plans. See Pricing.