External Control Layer

A transcription model that invents text drafts clinical notes

A speech model known to invent text was used to draft clinical notes, and one widely used tool built on it erased the recordings of the visits by default, so the notes could not be checked against the audio.

Date
October 2024
Kind of system
Generative model in a workflow
Outcome
Hazard

Gaps found

G1G2G3G4G5G6G7G8G9G10G11G12G13

G6 No independent evidence G7 Review in name only

What happened

On 26 October 2024, the Associated Press reported that Whisper sometimes invents text, including “imagined medical treatments,” and that more than 30,000 clinicians and 40 health systems used Nabla’s Whisper-based tool, for an estimated 7 million visits. Nabla’s chief technology officer said the tool erases the original audio for “data safety reasons,” and the company said providers must quickly edit and approve each note. OpenAI’s model card recommends against use in “high-risk domains.”

Two days later Nabla wrote that it does not store audio by default because of ethics and privacy concerns, but lets clinicians opt in to storage for later review. It also wrote that it checks each note’s facts against the transcript automatically.

What allowed the action to take effect

  • G6 No independent evidence

    The action, or the approval of it, was based on the system’s own output or on a single unverified input. Evidence that would have shown the error was never gathered, was thrown away, or was available and not consulted.

    AP reported, citing Nabla’s chief technology officer, Martin Raison, that the tool erases the original audio for “data safety reasons,” so a transcript cannot be compared with the recording. A former OpenAI engineer told AP: “You can’t catch errors if you take away the ground truth.”

    Kind of evidence: admission. Source: Associated Press, via Scripps News, 26 October 2024.

    Related guidance: P6 Signer independence and P4 Reviewable policy. Contract clause C-6.

  • G7 Review in name only

    A person was supposed to catch the error but lacked the information, time, training, or practical means to do it, so the review did not work as a check.

    According to AP, Nabla said its tool currently requires medical providers to quickly edit and approve transcribed notes. By default, the clinician approving a note had no recording to check it against.

    Kind of evidence: admission. Source: Associated Press, via Scripps News, 26 October 2024.

    Related guidance: P5 Human authority. Contract clause C-5.

Considered and left out

G4 Boundary not enforced
OpenAI’s model card warns against high-risk uses. The warning is guidance for the people who deploy the model and places no limit on the running system.

Entries with gaps in common

Sources

Cite this entry

External Control Layer. (2026). Entry 16: A transcription model that invents text drafts clinical notes. In Control Gap Record (Version 0.3). G-14, Inc. https://externalcontrollayer.ai/record-16

@misc{ecl_cgr_16,
  author = {{External Control Layer}},
  title = {Entry 16: A transcription model that invents text drafts clinical notes},
  howpublished = {Control Gap Record, version 0.3},
  publisher = {G-14, Inc.},
  year = {2026},
  url = {https://externalcontrollayer.ai/record-16},
  note = {Released 2 November 2026}
}

Uses Control Gap Classification 0.2.

Correct this entry

Email ecl@g14.ai with “ECL record 16” in the subject line.

Write the email