# The check that has never failed

*Wardline monitoring, pre-decommission review. Filed against WLN-2214.*

Wardline is being retired in favour of the new collector, and the migration
plan requires that every check be either ported or justified as dead. There are
4,109 checks. 4,108 of them ported cleanly.

## chk-0041

```
$ wardline show chk-0041
id           chk-0041
name         gate-02 icmp
target       192.0.2.41
interval     60s
timeout      2000ms
owner        (unset)
created      2015-03-02T09:14:00Z
created_by   (unset)
last_change  2015-03-02T09:14:00Z
notify       (none)

$ wardline stats chk-0041 --since created
samples      5,785,092
ok           5,785,092
fail         0
availability 100.0000000%
rtt_min      0.31 ms
rtt_med      0.42 ms
rtt_max      0.94 ms
rtt_p999     0.71 ms
```

Zero failures in eleven years and six months. No maintenance windows, no
suppressions, no gaps in the sample count — the interval is 60 seconds and the
sample count is exactly what 60-second polling produces over that span, to the
sample.

Every other check in Wardline has failed at least once. The next-best record
belongs to `chk-2277`, which watches a power distribution unit in the same
building and has failed 41 times, all of them during the two site-wide outages
in 2018 and in 2022. During both of those outages `chk-0041` continued to
report, and its median response time did not change.

## What the target is

Nobody knows. `gate-02` does not appear in the inventory, in DNS, in the DHCP
reservations, in any Terraform state, or in the two rack diagrams from the
period. The address `192.0.2.41` is inside the range that was allocated to the
old management VLAN, which was flattened in 2017.

There is no route to it.

```
$ ip route get 192.0.2.41
RTNETLINK answers: Network is unreachable
$ ping -c 3 192.0.2.41
connect: Network is unreachable
$ traceroute 192.0.2.41
traceroute: bind: Network is unreachable
```

That is from `wardline-01`, the host the checks run on, at 14:52 today. Six
minutes later, at 14:58, `chk-0041` recorded a successful poll with a round-trip
time of 0.39 ms.

## Distances

The nearest of our own equipment to that address, physically, is in the second
building. Light takes 0.20 ms to make the round trip over the fibre between the
two buildings, before any switching. The median for `chk-0041` is 0.42 ms, which
is consistent with a device roughly seventeen metres from the poller — the far
side of the same room.

The far side of that room is a wall, and beyond it the stairwell.

## Recommendation

I was asked to either port the check or justify it as dead. I cannot do either.
Porting it means giving the new collector a target it will not be able to reach,
and it will alert on that, and someone will be woken up. Justifying it as dead
means writing down that a thing which answered eleven minutes ago is dead.

I have set `notify` to a channel nobody reads and left it running on Wardline,
and I have asked that Wardline not be powered off until this is resolved.

See also [[stories/quorum]], which is the same shape from the other direction,
and [[stories/the-fourth-nameserver]].