LLMジャッジは「存在」を検証し、「不在」を検証しない：AI臨床記録における省略盲とそれを回復するもの
LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes (arxiv.org)
要約
本研究では、AIによる臨床記録作成における主要なエラーである「情報の省略」をLLM（大規模言語モデル）ジャッジが検出できるかを検証しました。標準的なLLMジャッジは、記録された情報（存在）の検証には優れていますが、記録されなかった情報（不在）の検出は苦手であることが明らかになりました。この問題を解決するため、記録された事実をリストアップし、それぞれを検証する新しいタスク構造を提案し、その有効性を示しました。
全文翻訳
Computer Science > Computation and Language
arXiv:2608.31016 (cs)
[Submitted on 31 Aug 2026]
Title: LLMジャッジは存在を確認し、不在ではない：AI臨床記録における欠落盲視とその回復策
Authors: Sebastian Fox, Luke Markham, Ryan Lail, Michael Karotsieris
View a PDF of the paper titled LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It, by Sebastian Fox and 3 other authors
Abstract: Ambient AI scribes draft clinical notes, and published audits find their dominant error is omission: information the encounter established that the note fails to record. The standard check is an LLM judge: a second model reads the note against the transcript and flags problems. We ask whether judges detect omissions. Public corpora cannot supply the answer key: their clinician reference notes and transcripts are materially discrepant. Our benchmark has 500 single-error note pairs from audited fact sheets, 298 with a named fact certainly absent and 202 added-or-altered controls. Across eight judge designs, paired discrimination (the flawed note below its clean twin, 0.5 a coin flip) reads 0.79-0.94 on added or altered content and 0.50-0.63 on omissions. On single notes, no design flags omissions reliably more often than perfect notes. Wording changes, voting and GEPA prompt optimisation move the operating point without creating usable detection. Restructuring the task recovers it: list the facts the transcript establishes, then check the note for each. Two methods reach it independently and trade off: a per-fact pipeline, and a GEPA-evolved prompt doing the same in one call. The pipeline's flags name the missing fact and its severity at 2.7% false alarms. The single call detects more (36.9% against 24.6%, p=0.002) at 6.2% false alarms and a tenth of the cost per note. A physician author validated 70 items and, where the two routes disagree, sided with the pipeline on 10 of 10 (p=0.002). A second clinician, not an author, graded the severity rubric blind and agrees to within a grade. On real vendor notes from a companion census no benchmark threshold transfers, but the re-calibrated single call detects more than the best of the eight at half its false-alarm rate. Omissions whose fact is restated elsewhere defeat both routes. We release the benchmark, prompts and judgements.
| Comments: | 97 pages, 8 figures. Dataset: this https URL . Code: this https URL . Companion paper: "One note in three: a verified census of three deployed AI scribes, and the instrument that counted it" |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2608.31016 [cs.CL] |
| (or arXiv:2608.31016v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.31016 arXiv-issued DOI via DataCite |
Submission history
From: Sebastian Fox view email
[v1] Mon, 31 Aug 2026 15:59:54 UTC (337 KB)
Full-text links:
Access Paper:
View a PDF of the paper titled LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It, by Sebastian Fox and 3 other authors
Additional Features
Current browse context:
cs.CL
Change to browse by:
References & Citations
Loading...
BibTeX formatted citation
loading...
Data provided by:
Bookmark
Bibliographic Tools
Bibliographic and Citation Tools
Bibliographic Explorer Toggle
Bibliographic Explorer (What is the Explorer?)
Connected Papers Toggle
Connected Papers (What is Connected Papers?)
Litmaps Toggle
Litmaps (What is Litmaps?)
scite.ai Toggle
scite Smart Citations (What are Smart Citations?)
Code, Data, Media
Code, Data and Media Associated with this Article
alphaXiv Toggle
alphaXiv (What is alphaXiv?)
Links to Code Toggle
CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub Toggle
DagsHub (What is DagsHub?)
GotitPub Toggle
Gotit.pub (What is GotitPub?)
Huggingface Toggle
Hugging Face (What is Huggingface?)
ScienceCast Toggle
ScienceCast (What is ScienceCast?)
Demos
Demos
Replicate Toggle
Replicate (What is Replicate?)
Spaces Toggle
Hugging Face Spaces (What is Spaces?)
Spaces Toggle
TXYZ.AI (What is TXYZ.AI?)
Related Papers
Recommenders and Search Tools
Link to Influence Flower
Influence Flower (What are Influence Flowers?)
Core recommender toggle
CORE Recommender (What is CORE?)
- Author
- Venue
- Institution
- Topic
About arXivLabs
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.
Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)