A local model turns outage notices in an operations mailbox into structured events: which service, announced or cancelled or rescheduled, what window, how severe. The extraction is the straightforward half. The half worth reading is the eval, which reports recall per class and emits no single accuracy figure.
A real operations mailbox is overwhelmingly not outage notices. Over that distribution a classifier that answers "not an alert" to everything scores in the nineties — and its score rises as the mailbox gets noisier, improving on paper while doing strictly less. Any single figure is dominated by the majority class, so the classes you care about can fail freely without moving it.
The four classes also carry different costs. Missing an announcement is an outage nobody prepared for. Missing a cancellation is a crew standing by at 1 AM for work that was called off. An average over those is not a summary; it is a number you cannot act on.
== PREFILTER (does it reach the model?) ==
class support recall
not_an_alert 6 83.3%
wrongly assigned: ['neg-03']
announces 6 100.0%
cancels 3 100.0%
reschedules 3 100.0%
gate precision: 92.3% (1 non-alert(s) let through: ['neg-03'])
Recall is the number that matters here: a false positive costs one model call,
a false negative is an outage notice nothing downstream can recover.
(extraction not scored: --backend prefilter-only)
Recall with the ids of the failing cases, so the report names what to go look at. In the repository that block is not pasted: a test runs the harness and compares its stdout to the lines quoted in the README, so a page that drifts from the code fails CI.
A false positive costs one model call, after which the model returns no event and the message is dropped. A false negative is a notice nothing downstream can recover, because nothing downstream sees it. The two are not comparable, so the gate is not tuned to balance them.
Scored on the decision the downstream system consumes. Detecting "something is happening" and calling a cancellation an announcement are different outcomes for whoever is on call, so they do not score the same.
A small model reading a maintenance notice fails in ways that are specific, repeatable and well-formed — which is why prompting does not fix them. The output is not malformed, so nothing in the parsing layer objects.
de 17:00 a 21:00 occasionally comes back as 07:00 — a valid timestamp in a valid object. What rejects it is that a window starting before the message announcing it, or running past 48 h, is implausible however it was produced. The result is a third state: a silently dropped window leaves an event that looks complete, and a kept one leaves an event that is confidently wrong.
Matching a cancellation to its announcement means comparing two free-text names written at different times. Substring matching is the obvious approach and it is wrong: a name normalising to ia matches inside via satelital, and the cancellation deletes an unrelated event. Word boundaries plus a length floor.
The extraction runs at temperature=0 with format=json. That constrains the shape of the output; it does not make it reproducible. The guards exist because it is not.
The 18 cases are invented notices in Spanish and English, written in the same commit as the code they exercise. Each carries a why field naming the rule it encodes. The point of the set is that those rules stay true — not that the percentages say anything about a real mailbox, which has a class balance you did not choose, forwarding chains, quoted replies containing last month's notice, and the vocabulary of whoever writes the notices where you work.
Recording exists because an eval whose result depends on a model being reachable goes red when the host is busy and green when it is idle. It separates did the model change? from did my code change?. A case with no recorded answer raises rather than scoring as silence.