CHANGELOG: sched-ops

Extracted from CHANGELOG.md in the sched-ops repository. Entries are verbatim; the versions in between have been elided where they do not concern nightly-0300.

2.4.0 — 2024-08-19 Added nightly-0300 (vault/rollup.py), runs 03:00 daily on sched-a. Rolls yesterday's partitions into the archive bucket. Typical runtime 11 min.

2.9.1 — 2025-01-07 nightly-0300 disabled. The archive bucket was retired in December and the job has nothing to write to. Left in the repo for reference; enabled: false.

2.9.2 — 2025-01-09 Reverted nothing. Noting here that nightly-0300 reported success at 03:11 on the 8th and the 9th, both after being disabled. Cause not yet found. The completion record is written by the job, not by the scheduler, so the scheduler is not the thing lying.

2.9.3 — 2025-01-13 Removed the nightly-0300 entry from schedules.yaml entirely. Completion records on the 10th, 11th, 12th and 13th.

2.10.0 — 2025-01-20 Deleted vault/rollup.py from the repository and from /opt/sched/jobs on sched-a. Runtime for the 20th: 11 min 04 s.

2.10.1 — 2025-01-23 Rotated the archive service account's credentials and revoked the old ones. The job's completion record includes the object count it wrote. On the 21st it wrote 0 objects. On the 22nd it wrote 41,802. There is no bucket.

2.11.0 — 2025-02-02 Moved the scheduler to sched-b, new hardware, fresh install, sched-a powered off and left in the rack. Completion records continue and are now attributed to sched-b.

2.11.4 — 2025-02-06 sched-a removed from the rack and taken to the recycler. See asset ticket AS-9902. Completion records unaffected.

2.12.0 — 2025-03-01 Added a rule to the completion-record ingest that drops any record whose job id is nightly-0300. This is a display change. It does not stop anything. I want to be clear in the changelog that it does not stop anything.

2.12.1 — 2025-03-04 Removed the rule added in 2.12.0. Three days without the records was worse.

3.0.0 — 2026-02-11 Reinstated nightly-0300 in schedules.yaml with enabled: true and an empty command, so that the scheduler has something to attribute the completions to and the on-call rota stops receiving orphan-record alerts. This is bookkeeping. The runtime has been 11 minutes and between two and nine seconds every night for nineteen months.

3.0.1 — 2026-02-24 Corrected the timezone offset on the nightly-0300 schedule entry. Discovered while doing so that the completion timestamps have never been consistent with the start timestamps. Diagram below.

timeline
    title nightly-0300, one night, in the order the records were written
    02h58m41s : completion record written : 41,802 objects rolled up
    03h00m00s : scheduler fires : no command configured
    03h00m02s : scheduler records exit 0 : runtime 2 s
    03h11m06s : start record written : pid 20841 on sched-a

The completion is first. The start is last. The pid belongs to a host that went to the recycler in February 2025, and pids on that host never went above 32768.

3.0.2 — 2026-03-02 No change. Adding an entry so that the file has a line for March.

See stories/trace-4471 for the same problem in a call graph, and stories/inventory-diff for what happens when people stop correcting a thing.