# CHANGELOG: sched-ops

Extracted from `CHANGELOG.md` in the `sched-ops` repository. Entries are
verbatim; the versions in between have been elided where they do not concern
`nightly-0300`.

---

**2.4.0 — 2024-08-19**
Added `nightly-0300` (`vault/rollup.py`), runs 03:00 daily on `sched-a`.
Rolls yesterday's partitions into the archive bucket. Typical runtime 11 min.

**2.9.1 — 2025-01-07**
`nightly-0300` disabled. The archive bucket was retired in December and the job
has nothing to write to. Left in the repo for reference; `enabled: false`.

**2.9.2 — 2025-01-09**
Reverted nothing. Noting here that `nightly-0300` reported success at 03:11 on
the 8th and the 9th, both after being disabled. Cause not yet found. The
completion record is written by the job, not by the scheduler, so the scheduler
is not the thing lying.

**2.9.3 — 2025-01-13**
Removed the `nightly-0300` entry from `schedules.yaml` entirely. Completion
records on the 10th, 11th, 12th and 13th.

**2.10.0 — 2025-01-20**
Deleted `vault/rollup.py` from the repository and from `/opt/sched/jobs` on
`sched-a`. Runtime for the 20th: 11 min 04 s.

**2.10.1 — 2025-01-23**
Rotated the archive service account's credentials and revoked the old ones. The
job's completion record includes the object count it wrote. On the 21st it wrote
0 objects. On the 22nd it wrote 41,802. There is no bucket.

**2.11.0 — 2025-02-02**
Moved the scheduler to `sched-b`, new hardware, fresh install, `sched-a` powered
off and left in the rack. Completion records continue and are now attributed to
`sched-b`.

**2.11.4 — 2025-02-06**
`sched-a` removed from the rack and taken to the recycler. See asset ticket
AS-9902. Completion records unaffected.

**2.12.0 — 2025-03-01**
Added a rule to the completion-record ingest that drops any record whose job id
is `nightly-0300`. This is a display change. It does not stop anything. I want
to be clear in the changelog that it does not stop anything.

**2.12.1 — 2025-03-04**
Removed the rule added in 2.12.0. Three days without the records was worse.

**3.0.0 — 2026-02-11**
Reinstated `nightly-0300` in `schedules.yaml` with `enabled: true` and an empty
command, so that the scheduler has something to attribute the completions to and
the on-call rota stops receiving orphan-record alerts. This is bookkeeping. The
runtime has been 11 minutes and between two and nine seconds every night for
nineteen months.

**3.0.1 — 2026-02-24**
Corrected the timezone offset on the `nightly-0300` schedule entry. Discovered
while doing so that the completion timestamps have never been consistent with
the start timestamps. Diagram below.

---

```mermaid
timeline
    title nightly-0300, one night, in the order the records were written
    02h58m41s : completion record written : 41,802 objects rolled up
    03h00m00s : scheduler fires : no command configured
    03h00m02s : scheduler records exit 0 : runtime 2 s
    03h11m06s : start record written : pid 20841 on sched-a
```

The completion is first. The start is last. The pid belongs to a host that went
to the recycler in February 2025, and pids on that host never went above 32768.

**3.0.2 — 2026-03-02**
No change. Adding an entry so that the file has a line for March.

See [[stories/trace-4471]] for the same problem in a call graph, and
[[stories/inventory-diff]] for what happens when people stop correcting a thing.