Backup and restore
This guide shows you how to back up the control plane, keep the archive safe, and restore it with the recovery checks that follow.
1. What a backup covers
A backup covers the control plane host: the database, its keys and its configuration. Your execution servers are not in it. Their terminals are ephemeral and are never backed up.
| IN THE ARCHIVE | NOT IN THE ARCHIVE |
|---|---|
| The database dump: servers, deployments, users, grants, trading accounts, risk policies, history | Database passwords. New ones are created on restore. |
| The control plane state: agent certificate authority key, command signing key, gateway certificate | The security journal. It stays outside the restorable data on purpose (section 5). |
| Caddy’s certificate storage | The control plane’s vault key for remembered credentials, and its half of each remembered password |
| The .env configuration | Broker passwords in transit. A backup never captures them (section 2). |
| With single sign-on: the provider’s database and the dashboard client’s secret | Anything on your execution servers |
| Notification channel secrets, in sealed form only | The notification secrets key. Keep a copy of it apart from your backups. |
2. Take a backup
On the control plane host, run:
The archive is written to /var/lib/nile/backups/nile-backup-<id>.tar, readable by root only.
While it runs, the backup:
- pauses new starts that carry broker credentials. The dashboard answers 503 maintenance_backup, “try again shortly”;
- waits until no encrypted credential is still waiting to be delivered, so none ends up in the archive;
- dumps exactly the state it checked;
- lifts the pause.
Nothing else pauses. Other starts, stops, monitoring and agent traffic carry on. If a credential delivery does not finish in time, the backup is refused, names the open operations and lifts the pause. Run it again shortly.
3. Keep the archive and keys safe
The archive contains the keys that let the control plane sign commands to your agents and issue agent certificates. Anyone who holds it could sign commands as your control plane.
- Keep archives on encrypted storage that only administrators can read.
- Never attach an archive to a support request or a ticket.
- Delete old archives you no longer need.
Remembered credentials are split between each server’s vault and the control plane. The control plane’s vault key is kept in its own volume and is never in a backup. After a restore, every remembered password is lost and must be entered again.
4. Restore
A restore needs an installation of the same PerkBridge version and the same host name as the archive. On a new server, install that version with the same host name first. See Install the control plane.
- Copy the archive to the control plane host.
- Run:
sudo /var/lib/nile/nile-install restore –archive /var/lib/nile/backups/nile-backup-<id>.tar
- Wait for it to finish. A record of the restore is written to /var/lib/nile/restores/<time>/restore-record.json.
Before it changes anything, the restore checks the archive, the version, the host name and PostgreSQL 16. It keeps the old database under a new name, nile_pre_restore_<UTC time>, closed and read-only. It is never dropped automatically.
The restored control plane then starts in a recovery phase:
- every dashboard session, single sign-on attempt and unused enrollment token is cancelled;
- unfinished operations are quarantined and never sent again;
- access removed after the backup (revoked agents, disabled users, revoked grants) is applied again from the security journal;
- nobody can sign in, and no command that changes anything is sent.
Workloads on your execution servers keep running throughout.
5. Finish the recovery
Check the recovery state:
If the security journal proves nothing was missed, open sign-in:
If continuity cannot be proven (the journal is missing or broken, or you restored on another host), use admin recovery accept-without-journal –actor WHO. This puts every agent on hold, suspends every grant and keeps every user in review. Confirm each one with admin recovery release-agent –id, confirm-grants –user and confirm-user –id, or disable them, then run admin recovery complete.
With bundled single sign-on
Every single sign-on user and provider administrator is disabled until you review them. The provider stays unreachable until the review ends:
If a step fails, fix the cause and run sudo /var/lib/nile/nile-install recovery resume-provider. While users are in review, sign in with a password account confirmed with admin recovery confirm-user.
6. After a restore
- Grants, enabled users and enrollments made after the backup are lost. Make them again.
- An agent enrolled after the backup must enrol again.
- Remembered credentials are lost. Each deployment that used them asks for its password again.
- Review each server’s Gateway access page. Addresses removed after the backup are back.
- Risk policies continue above the revision each server already holds, so a restore never sends an older policy.
- Notifications are held until an administrator reviews the channels and rules and resumes them. A channel sealed under another key asks for its secret again. See Alert notifications.
- A failover switch after a restore stays blocked until the old server’s orders are settled, because the restored database may not know orders released after the backup. See Failover.
A restore always needs the same version as the archive. If a restore stops, see Troubleshooting.