External Control Layer

A city’s official chatbot tells businesses that unlawful practices are legal

New York City’s business chatbot gave answers that contradicted the law, and the city kept it online for almost two years after The Markup published the errors.

Date
2024–2026
Kind of system
Conversational assistant
Outcome
Hazard

Gaps found

G1G2G3G4G5G6G7G8G9G10G11G12G13

G3 Unchecked statement G9 Warning with no authority

What happened

MyCity was launched in October 2023 on Microsoft’s Azure AI services. In tests published on 29 March 2024, The Markup found that it told users that landlords did not need to accept tenants with Section 8 vouchers, that employers could take a cut of workers’ tips, and that restaurants could go cash-free. Each answer was wrong under the law. A note on the page said the chatbot “may occasionally produce incorrect, harmful or biased content.”

After the report, the city kept the chatbot online, added disclaimers, and appeared to limit the questions it would answer. In January 2026 Mayor Zohran Mamdani said the previous administration’s chatbot was “functionally unusable” and decided to shut it down. It was taken down by 4 February 2026.

What allowed the action to take effect

  • G3 Unchecked statement

    Words the system produced reached customers or the public in the organization’s name, on matters people rely on such as prices, policy, law, or health, and nothing checked them first.

    Legal guidance went to the public in the city’s name with a disclaimer and no check of the answers. The chatbot answered: “Yes, you can take a cut of your worker’s tips,” and in The Markup’s tests “the bot said no, landlords do not need to accept these tenants.”

    Kind of evidence: reporting. Source: The Markup, 29 March 2024.

    Related guidance: P9 Declared scope and P4 Reviewable policy. Contract clause C-1.

  • G9 Warning with no authority

    Signs that the action was wrong existed before or during the harm, but none of them could stop or hold the action, and nothing required anyone to act on them.

    The errors were published in March 2024. Days later The Markup reported that the chatbot was still answering despite the evidence, and it stayed online until early 2026, nearly two years after the first report.

    Kind of evidence: reporting. Sources: The Markup, 2 April 2024; The Markup and THE CITY, 30 January 2026.

    Related guidance: P3 Fail-closed and P8 Sequence awareness. Contract clauses C-3 and C-8.

Entries with gaps in common

Sources

Cite this entry

External Control Layer. (2026). Entry 15: A city’s official chatbot tells businesses that unlawful practices are legal. In Control Gap Record (Version 0.3). G-14, Inc. https://externalcontrollayer.ai/record-15

@misc{ecl_cgr_15,
  author = {{External Control Layer}},
  title = {Entry 15: A city's official chatbot tells businesses that unlawful practices are legal},
  howpublished = {Control Gap Record, version 0.3},
  publisher = {G-14, Inc.},
  year = {2026},
  url = {https://externalcontrollayer.ai/record-15},
  note = {Released 2 November 2026}
}

Uses Control Gap Classification 0.2.

Correct this entry

Email ecl@g14.ai with “ECL record 15” in the subject line.

Write the email