Generate llms.txt, llms-full.txt and Markdown twins of every docs page (OPTIP-478)

[devin-ai-integration]

Summary

Makes the built site consumable by AI tools (ChatGPT, Claude, Cursor, Copilot, …), which today only get rendered HTML and sitemap.xml. The build now also emits:

Output	Content
/docs/llms.txt	Two-level root index: one section per product linking to its llms.txt / llms-full.txt (~5 KB)
/docs/llms-full.txt	Every page as Markdown (6.8 MB)
/docs/<product>/llms.txt	Flat page list with title + description, grouped by sub-directory
/docs/<product>/llms-full.txt	All pages of that product (theoplayer 4.4 MB, millicast 1.5, theolive 0.47, ads 0.28, open-video-ui 0.12, ad-engine 0.04)
/docs/<route>.md	Markdown twin of every page, e.g. /docs/theoplayer/how-to-guides/web/ads/google-ima.md

Products: theoplayer, open-video-ui, theolive, millicast, ads, ad-engine. Excluded: Player v4–v9, /contributing, /search, 404. Kept: Player v10, Ads v1, THEOlive v1.

How

Markdown is generated from the rendered HTML (not the MDX source) by @signalwire/docusaurus-plugin-llms-txt, so shared partials, usePlatforms() card lists, generated OpenAPI pages and versioned docs are all handled uniformly. Two local pieces around it:

• src/plugin/llmsTxt.ts wraps the SignalWire plugin and, in the same postBuild, re-groups its output into the per-product files above (Docusaurus runs plugin postBuild hooks in parallel, so a separate plugin could not rely on the .md files existing yet). It fails the build if a top-level route has no products entry configured. Descriptions come from each page's <meta name="description">; heading/markup fragments and title repeats are dropped, long ones are cut at ~200 chars.
• src/plugin/llmsTxtMarkdown.ts cleans up Docusaurus HTML before conversion: Prism token spans → fenced code with language, admonitions → blockquote with bold title, tabs (incl. OpenAPI response tabs) → label + panel in order, strips # hash-links and comments; and fixes the plugin's /docs/docs/ duplicated base URL and /route/.md link shape.

Preview builds use the preview origin

All links in llms.txt / .md are absolute (siteConfig.url + baseUrl). Preview builds only changed baseUrl to /pr-<n>/, so every absolute URL the build emits (canonical, og:url, and now the llms.txt links) pointed at optiview.dolby.com/pr-<n>/…, which does not exist. The pull-request workflow now passes the preview CloudFront origin to the build:

# pull-request.yml → _build.yml
docusaurus_url: ${{ vars.PREVIEW_CLOUDFRONT_URL }}   # new optional input → DOCUSAURUS_URL

// docusaurus.config.ts
url: process.env.DOCUSAURUS_URL || 'https://optiview.dolby.com/',

main.yml does not set it, so production output is unchanged (https://optiview.dolby.com/docs/…).

Verification

• npm run build (prod): 1,714 pages, all internal .md links resolve, no v4–v9 / contributing / search / pr-N / announcement-bar text in any output; build time unchanged (3m53–4m27 vs 4m24 baseline).
• DOCUSAURUS_URL=https://docs-preview.optiview.dolby.com DOCUSAURUS_BASE_URL=/pr-838/ DOCUSAURUS_PR_NUMBER=838 npm run build: all 8,007 generated absolute links use docs-preview.optiview.dolby.com/pr-838/, canonical too; preview banner absent from .md.
• node serve.js: llms*.txt → text/plain, .md → text/markdown, HTML untouched. Deploy is the existing aws s3 sync (content type inferred from extension) — no deploy changes.
• npm run lint, npm run typecheck, npm run check-format pass; actionlint reports nothing new on the touched workflow lines.

Known limitation

OpenAPI reference pages are client-rendered, so their .md contains only the summary and response codes (parameters/schemas are not in the static HTML). Follow-up tickets are raised.

Link to Devin session: https://dolby.devinenterprise.com/sessions/ca62555ae6ad43d9b7be6ecdc256e68e
Open in Devin Desktop: https://dolby.devinenterprise.com/desktop/session/ca62555ae6ad43d9b7be6ecdc256e68e?variant=devin

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[theoplayer-bot]

Documentation preview
🚀 View preview at https://docs-preview.optiview.dolby.com/pr-838/
Deployed from commit e6c3e29 at 2026-09-03 22:30 UTC.