Outages and reboots

This guide explains what keeps running and what stops when the control plane, an agent or a server goes down, and what to do afterwards.

1. The basic rule

Your servers do not depend on the control plane to keep running. The agent on each server connects out to the control plane, and every workload runs on the server itself.

• A lost connection never stops a terminal, deletes a container or closes a position.
• Risk rules run on the server, next to the terminal, so they keep holding.
• Nothing restarts on its own unless you chose it. By default PerkBridge shows problems and raises alerts, and a person decides what to do. With remembered credentials you can let the control plane start a new session after a reboot or a failure, or switch to a standby (sections 5 and 6).

2. If the control plane is down

While the control plane or your connection to it is down:

• Running deployments keep running, and broker logins stay up. The one exception: a deployment with a failover standby closes its terminal after 5 minutes without the control plane, so it cannot trade unseen. Its positions stay at the broker.
• The dashboard keeps the last data it loaded, labelled as possibly out of date, and shows Cannot reach the Control Plane.
• No new requests can be sent: no starts, stops or removals.
• The Trading API runs on the control plane, so no new orders, and no closes, can be sent through it. If you must close a position during an outage, close it at your broker.
• On a risk-mode deployment, the server keeps enforcing the risk policy it has applied. A latch clear you asked for stays pending until the server can be reached.

Each agent reconnects by itself with growing waits between attempts, from 1 second up to 5 minutes, and never gives up. When it is back, it reports the state of every runtime and unfinished command before it accepts new ones.

To check an agent’s connection on its server:

The state field shows CONNECTED, RECONNECTING, REVOKED and so on, with the last heartbeat and last error.

3. If an agent or the Watchdog stops

MT5 runtimes are separate containers, so they keep running when the Fleet Agent stops or crashes. When the agent starts again, it re-attaches to them and reconciles its records first. A start sent during that time is refused with not_reconciled; send it again on purpose once the agent is ready.

The dashboard raises AGENT_DISCONNECTED after 60 seconds without an agent session, and SERVER_OFFLINE after 300 seconds. Both clear when the agent reconnects.

The Watchdog is an optional health monitor on each server, installed separately from the agent. It raises alerts, saves diagnostic snapshots and asks the agent to correct its records. With its default settings it never restarts, stops or deletes a terminal, and never closes positions. It keeps working when the control plane is unreachable.

4. If the risk engine stops

On a risk-mode deployment, the risk engine runs inside the deployment’s container. If it is not running or not ready:

• every new opening order is refused with risk_engine_unavailable;
• closing an existing position still works, through a separate close-only channel.

The engine is restarted within seconds. It first recovers its saved state, including latches, and only then accepts orders again. A request that was in flight is settled from the broker’s history and never resent.

5. If a server reboots

An MT5 session does not survive a reboot and is never resumed. After the reboot the agent and the Watchdog start again, no MT5 container is restarted, temporary credential files are gone, and the deployment is recorded as stopped by the reboot. The agent also creates a new credential encryption key.

What happens next depends on the deployment’s Start again after a server reboot setting, on its Failover tab:

• On, with remembered credentials and automatic use allowed. When the server is back and healthy, the control plane starts a new session on it with the remembered password. The session starts halted, is checked against the broker, and then resumes. At most 3 automatic starts per hour. If a failover standby is ready, the switch goes first.
• Otherwise. The deployment stays stopped and the STOPPED_BY_REBOOT alert is raised. A person starts it.

The setting is on by default for deployments created from now on, and works with failover on or off. Older deployments need it ticked. It is included in every plan.

If nothing started the deployment again:

1. Check your positions at the broker.
2. Open Agents and check that the server’s agent is connected again.
3. Start the deployment with the remembered password, or stop and remove it and create a new deployment. See Removing servers and deployments and the deployment guide.

On the control plane host, the containers keep their data in named volumes. The gateway firewall service applies its rules again after Docker at every boot. Single sign-on can take a few seconds longer than the other services to be ready.

6. Failover to a standby server

Without failover, an outage of a server means its deployments stop trading until the server is back or you act. With failover on, you choose a standby server for a deployment, and PerkBridge switches to it by hand or automatically when the server, container or terminal fails. A switch never sends an order twice: the account stays halted for new positions until it is checked against the broker.

Failover is included. See Failover to a standby server. Alerts for an outage can also be sent to Telegram, Slack, email or a webhook; see Alert notifications.