External Control Layer

An eating disorder helpline’s chatbot gives dieting advice

A chatbot built to give only pre-programmed answers was changed by its vendor to generate its own, and it then gave dieting advice. NEDA said it was never told of the change, which the vendor said was part of NEDA’s contract.

Date
May 2023
Kind of system
Conversational assistant
Outcome
Hazard

Gaps found

G1G2G3G4G5G6G7G8G9G10G11G12G13

G3 Unchecked statement G12 Unreviewed change

What happened

A research team that Ellen Fitzsimmons-Craft helped lead built Tessa as a rule-based program in which “all of the responses were pre-programmed.” The reason, she said, was that “A.I. isn’t ready for this population.” In 2022 its vendor, Cass, added an “enhanced question and answer feature” that used generative AI, as part of what its chief executive called a systems upgrade.

In May 2023 an eating disorder consultant tested Tessa, found it giving weight-loss guidance, and shared screenshots. NEDA took Tessa down on 30 May, within 24 hours of receiving them. Cass’s chief executive told NPR the change was part of NEDA’s contract, and NEDA’s chief executive said the association “was never advised of these changes and did not and would not have approved them.”

What allowed the action to take effect

  • G3 Unchecked statement

    Words the system produced reached customers or the public in the organization’s name, on matters people rely on such as prices, policy, law, or health, and nothing checked them first.

    After the switch to generated answers, the sources describe no check of those answers against approved content, and the chatbot gave weight-loss advice. The original design had avoided this because “all of the responses were pre-programmed.”

    Kind of evidence: reporting. Source: NPR, 8 June 2023.

    Related guidance: P9 Declared scope and P4 Reviewable policy. Contract clause C-1.

  • G12 Unreviewed change

    The change that introduced the harmful behavior went live without independent review. The change can be code, a prompt, a model, a package, a configuration, or a procedure.

    The vendor changed the chatbot from pre-programmed responses to generated ones as a systems upgrade. NEDA said it “was never advised of these changes and did not and would not have approved them.” The vendor said the change was part of NEDA’s contract.

    Kind of evidence: admission. Source: NPR, 8 June 2023.

    Related guidance: P4 Reviewable policy and P2 Independence. Contract clauses C-10 and C-4.

Considered and left out

G9 Warning with no authority
The problem surfaced when an eating disorder consultant posted screenshots, and NEDA acted within a day. The sources show no earlier warning.

Entries with gaps in common

Sources

Cite this entry

External Control Layer. (2026). Entry 12: An eating disorder helpline’s chatbot gives dieting advice. In Control Gap Record (Version 0.3). G-14, Inc. https://externalcontrollayer.ai/record-12

@misc{ecl_cgr_12,
  author = {{External Control Layer}},
  title = {Entry 12: An eating disorder helpline's chatbot gives dieting advice},
  howpublished = {Control Gap Record, version 0.3},
  publisher = {G-14, Inc.},
  year = {2026},
  url = {https://externalcontrollayer.ai/record-12},
  note = {Released 2 November 2026}
}

Uses Control Gap Classification 0.2.

Correct this entry

Email ecl@g14.ai with “ECL record 12” in the subject line.

Write the email