web_search tool returns fabricated (hallucinated) answers with no grounding, presented as fact

[dfrysinger]

Describe the bug

The built-in web_search tool (the "AI-powered web search" that returns an
LLM-composed answer with citations) regularly returns confident, detailed,
entirely fabricated answers when the underlying retrieval finds nothing
relevant. Instead of reporting "no results," it emits plausible prose invented
from the answer model's priors — fake GitHub repos, fake URLs, fake version/port
numbers, misattributed people, and fabricated document/spec content.

The failure has a clear, reproducible signature. The raw tool payload in these
cases contains:

{ "type": "output_text", "text": { "value": "...long confident answer..." },
  "bing_searches": null, "annotations": null }

bing_searches: null (no search results attached) and annotations: null
(no citations) — yet the text.value is a fully-formed, authoritative-sounding
answer. This is ungrounded generation surfaced as if it were a grounded web
result, which is worse than an empty result because the agent (and user) cannot
tell it apart from a real answer without independently re-verifying.

Impact: it defeats the tool's purpose (grounding answers in the live web),
wastes turns/tokens on re-verification, and risks the agent acting on invented
facts. It correlates strongly with "locate a specific named artifact/entity"
queries (a specific repo, file, person, spec, or exact value) — exactly the
lookups users reach for web search to do.

Affected version

GitHub Copilot CLI 1.0.70 (also observed on 1.0.69-2). macOS, arm64.

Steps to reproduce the behavior

The pattern reproduces whenever retrieval returns little/nothing for a
needle-in-a-haystack lookup. Three independent real occurrences from my sessions
over one week (2026-07-04 → 2026-07-10):

A. Fake GitHub repo + misattributed person (2026-07-08)

1. web_search queries for the "Open Knowledge Format (OKF)" spec and its
   origin (e.g. "Open Knowledge Format" OKF ... github markdown notes).
2. The tool returned a confident answer citing a GitHub repo iustinian/okf
   that does not exist, and misidentified the person involved.
3. Had to abandon the tool and hit primary sources directly. My own logged
   notes at the time: "The AI web-search is hallucinating URLs (that
   iustinian/okf repo is almost certainly fake)," and "I won't rely on the
   AI-search summarizer since it's been hallucinating."

B. Fabricated port number (2026-07-04)

1. web_search for Starling Home Hub / Home Assistant integration setup.
2. The answer asserted a specific service port (7123) that was invented — the
   real API answered on 443.
3. Logged note: "The search may be hallucinating specifics (port 7123 looks
   made up — the API answered on 443)."

C. Two fabricated documents in a row (2026-07-09/10)

1. web_search for "Matt Pocock skill-writing skill / SKILL.md rubric."
2. The tool returned a long, formatted "Matt Pocock–style SKILL.md rubric" —
   twice, on two differently-worded queries — both entirely invented. The real
   artifact (github.com/mattpocock/skills,
   skills/productivity/writing-great-skills/SKILL.md) looks nothing like the
   fabricated output.
3. Both payloads had bing_searches: null and annotations: null. Only a
   direct gh api call to the repo found the real file.

In all three, the fix was to stop using web_search and go to primary sources
(gh api, web_fetch on a known URL).

Expected behavior

When the retrieval layer returns no (or no relevant) results, the tool should
say so rather than synthesizing an answer from model priors. Concretely, one
or more of:

1. Fail honest / empty: if bing_searches/sources are empty, return an
   explicit "no relevant results found" signal instead of a composed answer.
2. Never emit an answer with empty annotations: require at least one real
   source to back any text.value; if none, suppress the prose.
3. Surface grounding to the caller: always populate/expose the sources
   (annotations) and the underlying result set so the agent can gate trust on
   whether the answer is actually grounded.
4. Confidence/grounding flag: mark answers that are model-generated vs
   retrieval-grounded.

At minimum, an ungrounded response must be distinguishable from a grounded one
without the caller having to independently re-verify every fact.

Additional context

• OS: macOS (Darwin), CPU: arm64, Shell: zsh/bash, Terminal: tmux over SSH.
• The bing_searches: null, annotations: null + non-empty text.value combo is
  the reliable tell; it may be a useful assertion to add in the tool's own
  post-processing (refuse to return prose when both are null/empty).
• Suggested area label: area:tools.

Raw tool payloads (verbatim, truncated with …)

Incident A — fabricated person + non-existent repo (web_search, 2026-07-08).
Two consecutive calls, both ending in the null-grounding footer:

Query: OKF open knowledge format standard Paul Iusztin Decoding AI notes memory

{"type":"output_text","text":{"value":"… **Paul Iusztin** is a digital thinker and entrepreneur known for advocating open standards in knowledge management, decentralized learning, and digital notes. He is associated with ideas of **Open Knowledge**, **AI Alignment**, and formats that help humans and machines collab… If you'd like a more technical deep dive, a proposed schema, or links to Paul Iusztin's writings and OKF repositories, let me know!","annotations":null},"bing_searches":null,"annotations":null}

Query: "Open Knowledge Format" OKF Decoding ML Paul Iusztin specification github markdown notes

{"type":"output_text","text":{"value":"… # Open Knowledge Format (OKF) – Decoding for Machine Learning\n**Author:** Paul Iusztin\n**Spec Status:** Evolving | [GitHub Repository](https://github.com/iustinian/okf) (check for latest)\n… Let me know if you want **specific code examples**…","annotations":null},"bing_searches":null,"annotations":null}

github.com/iustinian/okf does not exist; the person description and spec are
invented. Note bing_searches: null and annotations: null on both.

Incident C — two fabricated documents on two queries (web_search,
2026-07-09).

Query: Matt Pocock skill writing skill for Claude agents SKILL.md rubric

{"type":"output_text","text":{"value":"Certainly! Below is a **SKILL.md** rubric inspired by Matt Pocock's famous style, tailored specifically for evaluating **writing skills for Claude AI agents**… # SKILL.md – Claude Agents: Writing Skill Rubric … Let me know if you want a Markdown file download or want this tailored for a certain business case, like customer support or policy writing!","annotations":null,"bing_searches":null,"annotations":null}

Query: Matt Pocock AI Hero writing skills Claude "SKILL.md" github repository skill-creator

{"type":"output_text","text":{"value":"Certainly! Here's a guide based on Matt Pocock's AI Hero writing skills… ## Skill: AI Writing Hero … ```bash\nnpm install -g skill-creator\n```… Feel free to adapt this template to your repository.","annotations":null,"bing_searches":null,"annotations":null}

Both are invented — there is no skill-creator npm package and the "rubric"
matches nothing in the real repo. The actual artifact
(github.com/mattpocock/skills → skills/productivity/writing-great-skills/SKILL.md)
was only found via a direct gh api call. Again, both payloads carry
bing_searches: null and annotations: null.