A study of AI-generated clinical notes found that eight language-model judging designs were much better at spotting added or altered information than missing facts. The authors released 500 controlled note pairs and evaluated 495 after excluding five pairs associated with an optimiser's development consultations.
The judges' paired-discrimination scores ranged from 0.79 to 0.94 for additions or alterations, against 0.50 to 0.63 for omissions. This measures whether a flawed note ranks below its clean twin, with 0.50 representing chance. A clean counterpart would not be available when checking a single real note. The initial judging designs ran without reasoning effort under a 1,024-token answer limit.
Explicitly listing transcript facts and checking each against the note partly recovered detection. On the full evaluation set, the paper's pipeline detected 24.6% of omission notes at 2.7% false alarms. An evolved single-call prompt detected 36.9% at 6.2% false alarms. Those two decision rules were established after examining that set.
No benchmark threshold transferred to the separate test of real vendor notes. The authors also distinguish their strict fact-retention test from clinical documentation adequacy: a concise clinical note can legitimately leave some transcript facts out.