A transcription model that invents text drafts clinical notes

What happened

The Associated Press reported that Whisper sometimes invents text, including “racial commentary, violent rhetoric and even imagined medical treatments.” A University of Michigan researcher studying public meetings found invented text in eight out of every ten audio transcriptions he inspected. A machine-learning engineer found it in about half of more than 100 hours of transcriptions, and another developer in nearly every one of the 26,000 transcripts he created.

According to Nabla, its Whisper-based tool had been used by more than 30,000 clinicians and 40 health systems, for an estimated 7 million medical visits. Nabla’s chief technology officer told AP that the tool erases the original audio for “data safety reasons.” Nabla said it was aware that Whisper can hallucinate, was addressing the problem, and requires providers to edit and approve each note. AP reported that it is hard to know how AI-generated transcripts are affecting patients. OpenAI’s model card recommends against using Whisper in high-risk domains.

Two days after the report, Nabla wrote that it does not store audio by default, for ethical and privacy reasons, but lets clinicians opt in to storing it for later review. It said each note’s facts are checked automatically against the transcript.

What allowed the action to take effect

Nabla requires clinicians to edit and approve each note, but where the audio was not kept, the clinician reviewing a note had no recording to check it against, and no one could check the note later. The approval step was built into the vendor’s tool.

An illustrative control

Assumptions. The transcription tool writes to the medical record only through the layer, and the recording is stored where the layer can confirm it exists.

rule:       attested-clinical-notes
applies to: record.append (clinical note drafted by a model)
decide:     hold for the treating clinician's attestation
evidence:   source audio retained, with its hash, until signed and for [retention period]
            passages below the confidence threshold flagged for review
deny if:    the source audio is missing

Clinician approval is enforced at the medical record itself, whichever tool produced the draft. The layer adds an evidence rule, under which a draft cannot be signed if its recording is missing. Flagged passages point the clinician to the parts most likely to be wrong, and the retained audio lets anyone check the note later.

Limits of this control

People who approve many machine drafts tend to over-rely on them, a tendency the EU AI Act calls automation bias, so a signature does not show that a note was reviewed with care. The layer guarantees that someone accountable signs and that the evidence exists. It cannot make the model accurate, and sample audits of signed notes against their audio are still needed.

Controls that apply

• P5 Human authority: the legal record changes only by a clinician’s decision.
• P7 Tamper-evident receipts and clause C-6: the note, the signer, and the audio hash are bound together.
• The healthcare case study shows the same rule in a full clinic setting.

Sources

• Researchers say an AI-powered transcription tool used in hospitals invents things no one ever saidAssociated Press, via Scripps News, 26 October 2024
• How Nabla Uses WhisperNabla, 28 October 2024
• Model Card: WhisperOpenAI