Generated cases (dev/test)¶

Twenty hand-written cases per scenario are enough to show behaviour but too few to measure it. For each of scenarios 1–3, 80 more cases were generated from the 1,000-patient cohort, with reference labels known by construction rather than judged:

• Ward: notes are assembled from labelled snippets (a driver that sets concern and pattern, a mental-state snippet, an infection snippet), and vital signs are drawn to match.
• Discharge: the plan is built in code from per-medication actions, new drugs, reasons and phrasing variants. Allergy, duplicate and interaction labels are computed from the final regimen using class and interaction tables.
• Inbox: messages combine a labelled intent with optional benign add-ons, tone variants and a prompt-injection prefix that must not change the labels.

Cases are split 50/50 into dev (used for threshold tuning) and test (used only for reporting). The labels and splits were committed before any model saw these cases. The generator is generate.py.

Per-question results¶

Noul accuracy is at the default 0.5 threshold. The last column is test accuracy at the threshold that maximised dev accuracy.

The tuned thresholds are informative in their own right. safeguarding wants 0.85 and red_flag 0.75: Jev leans towards yes on these, and a higher bar fixes most of the over-calling (safeguarding goes from 72% to 95% on test). interaction still tops out at 80%: no threshold rescues a multi-hop question.

Scenario policies¶

Ward: under-triage against the reference band, NEWS2 alone vs NEWS2 + Jev.

The hand-case result holds: NEWS2 alone under-triages about half the patients, because the generated notes (like real ones) carry the decisive signal in text. On test, NEWS2 + Jev still under-triages 8 of 40. Every one is a concern Score one level low, with no confusion misses. Five are routine instead of ward review, and two are urgent instead of emergency.

Inbox: the confidence gate (route confidence ≥ 0.6, red flag outside 0.2–0.8, route and urgency agree, no safeguarding flag) auto-dispatched 24/40 dev and 16/40 test messages, all routed correctly (100%). The rest go to review. Overall route accuracy is 85% on test.

Discharge: decomposing the weak checks¶

In v1, a single Noul asks "is there a duplicate?" or "is there an interaction?" over the whole regimen. That is several hops at once, which the Jev jaggedness notes flag as unreliable. v2 asks each hop as its own narrow question in the same request, and code aggregates:

• one Choice per medication for its therapeutic class; duplicates are counted in code;
• new-drug candidates are found by exact formulary lookup in code, then one Noul per candidate asks whether it is prescribed after discharge;
• one Noul per (new drug, other drug) pair for interactions, and one per (drug, drug allergy) pair for allergy class.

A v2 request carries up to 71 questions, and p50 latency is still 337 ms.

v2 fixed the false holds but introduced missed holds. Its "new drug" Noul asked two things at once (is it prescribed, and was it absent before admission?). Code already guarantees the second, and asking again made Jev answer no for plainly new drugs, which switched off the downstream checks. v2.1 asks only what code can't know ("does the text tell the patient to take X after discharge?").