# PIR-0918: quorum held

**Cluster** `corvid` (config store) · **Incident** 2026-08-11 04:02–04:29
**Impact** none · **Author** platform on-call · **Status** review complete,
one action item outstanding

## What happened

A cooling failure in row 4 tripped the thermal cut-out on two of `corvid`'s
three members, `corvid-a` and `corvid-b`, within eleven seconds of each other.
Both nodes powered off hard at 04:02:17 and 04:02:28.

A three-member cluster loses quorum when it loses two members. `corvid` did not
lose quorum. Reads and writes continued through the event with no elevated
latency, the config store served 2.1 M requests during the window, and the
first anybody knew of the outage was the facilities alarm.

## The cluster as drawn in the review

```
              row 4                             row 9
      ┌──────────────────┐             ┌──────────────────┐
      │  corvid-a        │             │  corvid-c        │
      │  192.0.2.71      │             │  192.0.2.73      │
      │  U14, rack 4-2   │             │                  │
      │  ── OFF 04:02:17 │             │  ── term 1       │
      └────────┬─────────┘             └────────┬─────────┘
               │                                │
      ┌────────┴─────────┐                      │
      │  corvid-b        │                      │
      │  192.0.2.72      │                      │
      │  U15, rack 4-2   │                      │
      │  ── OFF 04:02:28 │                      │
      └────────┬─────────┘                      │
               │                                │
               └────────────────┬───────────────┘
                                │
                        ┌───────┴────────┐
                        │  quorum 2 of 3 │
                        │  HELD 04:02:28 │
                        └────────────────┘
```

The box for `corvid-c` is drawn without a rack position because the reviewer
could not find one. It is not in the asset register. Facilities walked row 9
during the review with the address in hand and found the U-positions either side
of where it should be occupied by a patch panel and a blank.

## Findings

**F1.** `corvid-c` has been a voting member since the cluster was built on
2021-04-06. The bootstrap log names three members. The procurement record names
two servers.

**F2.** `corvid-c` has never been the leader. It has voted in all 1,904
elections and has never stood.

**F3.** Its term counter is 1. Every other member's term counter tracks the
cluster's, which is currently 1,904. A member at term 1 should be rejected as
stale by the other members. It is not, and the code path that would reject it is
covered by a test that passes.

**F4.** Round-trip time from `corvid-a` to `corvid-c`, sampled every 15 seconds
since 2021, has a standard deviation of 0.00 ms. It is 0.31 ms and it has been
0.31 ms 10,614,880 times.

**F5.** During the window in which `corvid-a` and `corvid-b` were powered off,
the cluster committed 41 entries. The commit index moved from 8,441,902 to
8,441,943. All 41 entries replicated cleanly to `corvid-a` and `corvid-b` when
they came back at 04:29.

## Action items

| # | Action | Owner | Status |
| --- | --- | --- | --- |
| A1 | Restore cooling redundancy in row 4 | facilities | done 2026-08-12 |
| A2 | Add `corvid-c` to the asset register | platform | see below |

A2 is outstanding. The register requires a serial number.

Related: [[stories/eleven-year-check]], [[stories/the-fourth-nameserver]].