# Runbook: manual failover, hesper cluster

**Doc** HES-04 · **Revision** 11 · **Last reviewed** 2026-06-30
**Applies to** `hesper-01` (primary), `hesper-03` (standby)

Use this when the primary is unhealthy and automatic failover has not fired
within five minutes. Estimated duration nine minutes. You need the `hesper-ops`
role and physical or console access to both nodes.

**Read the whole runbook before starting.** In particular read step 10.

---

**1.** Confirm the primary is genuinely unhealthy. On `hesper-01`:

```
hesperctl status --local
```

If it prints `role: primary` and `health: ok`, stop. You are looking at a
monitoring fault, not a database fault. Open a ticket against Wardline instead.

**2.** Record the current LSN on both nodes and write both numbers in the
incident channel before you touch anything. If the standby is more than 64 MB
behind, failing over loses data and you need the on-call DBA, not this runbook.

**3.** Fence the primary. On the console of `hesper-01`:

```
hesperctl fence --self --reason "manual failover HES-04"
```

This is not reversible from `hesper-01`. It can only be lifted from the node
that ends up primary.

**4.** Confirm the fence took. `hesperctl status --local` on `hesper-01` should
print `role: fenced`. If it still prints `role: primary` after thirty seconds,
power the node off at the PDU and continue. Do not skip this; two primaries is
the only outcome here that is worse than an outage.

**5.** On `hesper-03`, promote:

```
hesperctl promote --confirm
```

**6.** Wait for `role: primary` and `health: ok` on `hesper-03`. This takes
between forty seconds and four minutes depending on replay backlog.

**7.** Move the service address. From the jump host:

```
vipctl move hesper-vip --to hesper-03
```

**8.** Verify from outside the cluster. From the jump host, not from either
node:

```
psql -h hesper-vip -c 'select pg_is_in_recovery()'
```

Expect `f`. If it returns `t`, the address moved but the promotion did not take;
go back to step 5.

**9.** Update the topology record so that the next person is not misled:

```
hesperctl topology set --primary hesper-03 --standby hesper-01
```

**10.** Confirm ownership. Read `/var/run/hesper.owner` on the new primary. It
contains one line: the username of the operator who performed the last failover.
It should contain yours.

If it contains a username you recognise, page that person and hand over.

If it contains a username you do not recognise, do not overwrite the file, do
not page anyone, and do not proceed to step 11. Write the contents into the
incident channel exactly as they appear and return to step 4.

---

## Revision history

| Rev | Date | By | Note |
| --- | --- | --- | --- |
| 8 | 2025-02-11 | wrenn | Added the external verification at step 8 |
| 9 | 2025-09-02 | wrenn | Timings corrected after the September drill |
| 10 | 2026-03-14 | halloway | Added step 10 |
| 11 | 2026-06-30 | halloway | Clarified step 10 |

Revisions 10 and 11 were made from an account that was closed on 2026-01-30.

See [[stories/correspondence]] for the other half of this cluster's paperwork.