Runbook: manual failover, hesper cluster

Doc HES-04 · Revision 11 · Last reviewed 2026-06-30 Applies to hesper-01 (primary), hesper-03 (standby)

Use this when the primary is unhealthy and automatic failover has not fired within five minutes. Estimated duration nine minutes. You need the hesper-ops role and physical or console access to both nodes.

Read the whole runbook before starting. In particular read step 10.

1. Confirm the primary is genuinely unhealthy. On hesper-01:

hesperctl status --local

If it prints role: primary and health: ok, stop. You are looking at a monitoring fault, not a database fault. Open a ticket against Wardline instead.

2. Record the current LSN on both nodes and write both numbers in the incident channel before you touch anything. If the standby is more than 64 MB behind, failing over loses data and you need the on-call DBA, not this runbook.

3. Fence the primary. On the console of hesper-01:

hesperctl fence --self --reason "manual failover HES-04"

This is not reversible from hesper-01. It can only be lifted from the node that ends up primary.

4. Confirm the fence took. hesperctl status --local on hesper-01 should print role: fenced. If it still prints role: primary after thirty seconds, power the node off at the PDU and continue. Do not skip this; two primaries is the only outcome here that is worse than an outage.

5. On hesper-03, promote:

hesperctl promote --confirm

6. Wait for role: primary and health: ok on hesper-03. This takes between forty seconds and four minutes depending on replay backlog.

7. Move the service address. From the jump host:

vipctl move hesper-vip --to hesper-03

8. Verify from outside the cluster. From the jump host, not from either node:

psql -h hesper-vip -c 'select pg_is_in_recovery()'

Expect f. If it returns t, the address moved but the promotion did not take; go back to step 5.

9. Update the topology record so that the next person is not misled:

hesperctl topology set --primary hesper-03 --standby hesper-01

10. Confirm ownership. Read /var/run/hesper.owner on the new primary. It contains one line: the username of the operator who performed the last failover. It should contain yours.

If it contains a username you recognise, page that person and hand over.

If it contains a username you do not recognise, do not overwrite the file, do not page anyone, and do not proceed to step 11. Write the contents into the incident channel exactly as they appear and return to step 4.

Revision history

Revisions 10 and 11 were made from an account that was closed on 2026-01-30.

See stories/correspondence for the other half of this cluster's paperwork.