From 7d6ca71d8c6c571c1be706096386e28cf24d17f5 Mon Sep 17 00:00:00 2001 From: Rachael Rose Renk <91027132+rachaelrenk@users.noreply.github.com> Date: Mon, 14 Sep 2026 15:08:29 -0600 Subject: [PATCH] AFDocs: add Web Documentation Delivery Spec v0.6 checks Co-Authored-By: Warp --- .agents/skills/afdocs-audit/SKILL.md | 64 +- .../afdocs-audit/scripts/afdocs_audit.mjs | 679 ++++++++++++++++-- .../scripts/afdocs_audit.test.mjs | 81 +++ .agents/skills/afdocs-fix/SKILL.md | 34 + 4 files changed, 790 insertions(+), 68 deletions(-) create mode 100644 .agents/skills/afdocs-audit/scripts/afdocs_audit.test.mjs diff --git a/.agents/skills/afdocs-audit/SKILL.md b/.agents/skills/afdocs-audit/SKILL.md index 35b3dfccc..1d87a80e9 100644 --- a/.agents/skills/afdocs-audit/SKILL.md +++ b/.agents/skills/afdocs-audit/SKILL.md @@ -10,7 +10,7 @@ description: >- # AFDocs Audit -Run the [AFDocs scorecard](https://agentdocsspec.com/spec/) against docs.warp.dev and report results. +Run the [Web Documentation Delivery Spec v0.6](https://agentdocsspec.com/spec/web/) against docs.warp.dev and report results. ## Running the audit @@ -21,7 +21,10 @@ node .agents/skills/afdocs-audit/scripts/afdocs_audit.mjs \ --output /tmp/afdocs-report.json ``` -The script runs `npx afdocs check https://docs.warp.dev --format json`, parses the output, and writes a structured report. +The script runs `npx afdocs check https://docs.warp.dev --format json`, then supplements +the current 23-check CLI with bounded first-party probes for the five checks added in +spec v0.6. Once `afdocs` natively implements any of those checks, the wrapper defers to +the CLI result instead of duplicating it. ### Options @@ -31,12 +34,17 @@ The script runs `npx afdocs check https://docs.warp.dev --format json`, parses t ## Reading the report The JSON report contains: +- `spec_version` — The Web Documentation Delivery Spec version represented by the report - `score` — Overall score out of 100 - `grade` — Letter grade (A+ through F) -- `total_checks` — Number of checks run +- `score_method` — Whether the score came from the native CLI or the compatibility estimate +- `legacy_cli_score` — The 23-check CLI score when compatibility probes were needed; never compare it to the v0.6 score +- `total_checks` — Number of checks run (28 for a full v0.6 compatibility audit) - `summary` — Counts by status (`pass`, `fail`, `warn`, `skip`) - `categories` — Per-category scores and grades - `issues` — Array of failing and warning checks with details and fix guidance +- `scan_reliability` — `complete` or `partial`; partial scans must not establish a regression baseline +- `interaction_effects` — Non-scored combinations of findings that can degrade agent access more than an individual check suggests Each issue includes: - `id` — Check identifier (e.g., `llms-txt-directive-html`) @@ -45,6 +53,29 @@ Each issue includes: - `message` — Human-readable description - `fix` — Suggested fix from the AFDocs spec +### v0.6 compatibility checks and reliability + +Until the CLI is updated, the wrapper runs the following checks against up to eight +sitemap-discovered documentation pages. It fetches sequentially with a 10-second timeout +per request so the scan is bounded and does not create unnecessary load: + +- `bot-protection-interference` — detects challenge pages, tarpits, and volume-correlated blocking during sustained automated requests +- `page-size-transfer` — measures decoded HTML response bytes (`<1 MB` pass, `1–10 MB` warn, `>10 MB` fail) +- `single-fetch-completeness` — verifies that detected markdown pagination has a working, absolute continuation near the top of the response +- `markdown-link-portability` — requires absolute served-markdown links and verifies sampled `.md` links return a markdown representation +- `embedded-data-serialization` — attributes size-risk pages to large tables, data blocks, or base64 payloads that dominate converted content + +The native probes are intentionally a compatibility bridge, not a replacement for the +future CLI implementation. Its compatibility score uses equal weighting and counts +warnings as half-passes. Establish a new regression baseline when a report changes +`score_method` or `spec_version`. + +When `bot-protection-interference` warns or fails, treat every multi-page result as a +partial-sample observation. The `bot-protection-degrading-scan-reliability` interaction +effect records this condition. The `dynamic-content-rendered-statically` interaction +effect is recorded when multiple size, serialization, parity, or pagination signals +co-occur; it is diagnostic and does not add another score penalty. + ### Known exceptions Before reporting, cross-reference every issue against the known exceptions in `references/known-exceptions.md`. Classify each issue into exactly one bucket: @@ -84,15 +115,17 @@ remediation steps. After running the audit, ALWAYS report the results to the user before taking any action. Include: -1. **Score**: Overall score and grade +1. **Score provenance**: Report `spec_version`, score, grade, and `score_method`. If present, show `legacy_cli_score` separately and never compare it to the v0.6 compatibility score. 2. **Failures first**: List every fail-severity check with its message and fix guidance. These are the most impactful. 3. **Warnings**: List warning-severity checks with context. 4. **Allowlisted**: Briefly note any known exceptions that were flagged. -5. **If all checks pass**: Explicitly tell the user everything looks clean. +5. **Interaction effects**: Report each observed interaction effect after individual findings; explain that it is diagnostic, not an additional scored failure. +6. **If all checks pass**: Explicitly tell the user everything looks clean. Example report format: ``` -AFDocs audit complete: 23 checks run, score 82/100 (B). +AFDocs v0.6 compatibility audit complete: 28 checks run, score 82/100 (B). +Score method: unweighted compatibility estimate; legacy 23-check CLI score: 84/100. **Failures (5):** - llms-txt-directive-html: No llms.txt directive in HTML pages @@ -106,6 +139,9 @@ AFDocs audit complete: 23 checks run, score 82/100 (B). **Allowlisted (2):** - page-size-markdown: 1 page over 50K (changelog — intentionally long) - markdown-content-parity: 7 pages with minor diffs (Turndown escaping, not real content gaps) + +**Interaction effects (1):** +- dynamic-content-rendered-statically: transfer-size, serialized-data, and parity findings co-occur; review both rendering paths together. ``` After reporting, ask the user which issues they want to address. @@ -131,7 +167,8 @@ If any git step fails, write the entry to the run output and continue. ### Run log format ```markdown -## YYYY-MM-DD — [valid | blocked] +## YYYY-MM-DD — [valid | partial | blocked] +- **Spec**: v — - **Score**: N/100 (grade) - **Checks**: N total — N pass, N fail, N warn - **Failing check ids**: comma-separated list, or "none" @@ -140,11 +177,11 @@ If any git step fails, write the entry to the run output and continue. - **Notes**: [anything unusual] ``` -For a firewall-blocked run, record `blocked`, omit the score entirely rather than logging the meaningless one, and note the mitigation status. +For a bot-protection-interfered run, record `partial`, omit the score from regression comparisons, and note the observed enforcement mode. For a firewall-blocked run, record `blocked`, omit the score entirely rather than logging the meaningless one, and note the mitigation status. ## Regression detection -Compare this run against the most recent **valid** entry in the run log — never against a `blocked` entry, whose score is an artifact of the firewall challenge rather than a real measurement. If there is no prior valid entry, this run establishes the baseline: log it and post nothing. +Compare this run against the most recent **valid** entry with the same `spec_version` and `score_method` — never against a `blocked` entry, whose score is an artifact of a firewall challenge rather than a real measurement. A v0.6 compatibility report establishes a new baseline; do not compare it to legacy 23-check scores. If there is no comparable prior entry, log it and post nothing. A run is a regression when either: - The score dropped versus the last valid entry. @@ -167,6 +204,7 @@ A first-ever run with no baseline posts nothing. ``` *AFDocs Audit — * — regression +Spec: v | Score method: Score: /100 (), down from /100 on checks | pass, fail, warn @@ -217,10 +255,10 @@ The AFDocs scorecard evaluates these categories: **Content Discoverability** — llms.txt existence, validity, size, link resolution, markdown links, and in-page directives **Markdown Availability** — .md URL support and Accept: text/markdown content negotiation -**Page Size and Truncation Risk** — rendering strategy, page sizes (markdown and HTML), and content start position -**Content Structure** — tabbed content serialization, section header quality, code fence validity +**Page Size and Truncation Risk** — rendering strategy, markdown/HTML/transfer size, content start position, and single-fetch completeness +**Content Structure** — tabbed content serialization, section header quality, code fence validity, markdown link portability, and embedded-data serialization **URL Stability and Redirects** — HTTP status codes and redirect behavior **Observability and Content Health** — llms.txt coverage, markdown/HTML parity, cache headers -**Authentication and Access** — auth gate detection and alternative access paths +**Authentication and Access** — auth gate detection, alternative access paths, and bot-protection interference -Full spec: https://agentdocsspec.com/spec/ +Full spec: https://agentdocsspec.com/spec/web/ diff --git a/.agents/skills/afdocs-audit/scripts/afdocs_audit.mjs b/.agents/skills/afdocs-audit/scripts/afdocs_audit.mjs index c5dd9d761..cf94635f5 100755 --- a/.agents/skills/afdocs-audit/scripts/afdocs_audit.mjs +++ b/.agents/skills/afdocs-audit/scripts/afdocs_audit.mjs @@ -11,9 +11,21 @@ * node .agents/skills/afdocs-audit/scripts/afdocs_audit.mjs --url https://preview.docs.warp.dev */ -import { execSync } from 'node:child_process'; +import { execFileSync } from 'node:child_process'; import { writeFileSync } from 'node:fs'; import { resolve } from 'node:path'; +import { pathToFileURL } from 'node:url'; + +const SPEC_VERSION = '0.6.0'; +const COMPATIBILITY_SAMPLE_LIMIT = 8; +const REQUEST_TIMEOUT_MS = 10_000; +const V06_CHECK_IDS = new Set([ + 'bot-protection-interference', + 'page-size-transfer', + 'single-fetch-completeness', + 'markdown-link-portability', + 'embedded-data-serialization', +]); const GRADE_THRESHOLDS = [ [97, 'A+'], @@ -113,7 +125,7 @@ function runAfdocsCheck(url) { } try { - const stdout = execSync(`npx afdocs check ${url} --format json`, { + const stdout = execFileSync('npx', ['--yes', 'afdocs', 'check', url, '--format', 'json'], { encoding: 'utf8', maxBuffer: 10 * 1024 * 1024, // 10 MB — the JSON output can be large timeout: 300_000, // 5 minutes @@ -134,9 +146,540 @@ function runAfdocsCheck(url) { } } -function buildReport(raw) { - const { summary, results } = raw; - const score = raw.summary?.score ?? estimateScore(results); +function createResult(id, category, status, message, fix = null, details = null) { + return { id, category, status, message, fix, details }; +} + +function statusSummary(results) { + return results.reduce( + (summary, result) => { + summary[result.status] = (summary[result.status] || 0) + 1; + return summary; + }, + { pass: 0, fail: 0, warn: 0, skip: 0 } + ); +} + +function decodeHtmlEntities(value) { + return value + .replaceAll('&', '&') + .replaceAll('<', '<') + .replaceAll('>', '>') + .replaceAll('"', '"') + .replaceAll(''', "'"); +} + +function htmlToText(html) { + return decodeHtmlEntities( + html + .replace(/]*>[\s\S]*?<\/script>/gi, ' ') + .replace(/]*>[\s\S]*?<\/style>/gi, ' ') + .replace(/<[^>]+>/g, ' ') + .replace(/\s+/g, ' ') + .trim() + ); +} + +function isSoft404(response) { + if (!response || response.status !== 200) return false; + const text = htmlToText(response.body || '').slice(0, 4_000); + return /\b(404|page not found|not found|does not exist)\b/i.test(text); +} + +function isChallenge(response) { + if (!response) return false; + const mitigated = response.headers['x-vercel-mitigated']; + const hasChallengeHeader = + mitigated === 'challenge' || + response.headers['x-vercel-challenge-token'] != null || + response.headers['cf-mitigated'] === 'challenge'; + const hasChallengeBody = + /(verifying (you are )?(human|browser)|just a moment|security checkpoint|captcha|attention required)/i.test( + response.body || '' + ); + return hasChallengeHeader || hasChallengeBody; +} + +async function fetchResource(url, headers = {}) { + const controller = new AbortController(); + const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS); + const startedAt = Date.now(); + + try { + const response = await fetch(url, { + redirect: 'follow', + headers: { + 'user-agent': 'afdocs-v06-compatibility-audit/1.0', + ...headers, + }, + signal: controller.signal, + }); + const bytes = new Uint8Array(await response.arrayBuffer()); + return { + url: response.url, + status: response.status, + headers: Object.fromEntries(response.headers.entries()), + body: new TextDecoder().decode(bytes), + bytes: bytes.byteLength, + duration_ms: Date.now() - startedAt, + timeout: false, + }; + } catch (error) { + return { + url, + status: null, + headers: {}, + body: '', + bytes: 0, + duration_ms: Date.now() - startedAt, + timeout: error.name === 'AbortError', + error: error.message, + }; + } finally { + clearTimeout(timeout); + } +} + +function extractMarkdownLinks(markdown) { + const links = []; + const pattern = /!?\[[^\]]*]\(\s*(?:<([^>]+)>|([^\s)]+))/g; + let match; + while ((match = pattern.exec(markdown)) !== null) { + const href = (match[1] || match[2]).trim(); + if (href && !href.startsWith('#')) links.push(href); + } + return links; +} + +function toMarkdownUrl(rawUrl) { + const url = new URL(rawUrl); + if (url.pathname.endsWith('.md') || url.pathname === '/') return null; + url.pathname = `${url.pathname.replace(/\/$/, '')}.md`; + return url.toString(); +} + +function toHtmlUrl(markdownUrl) { + const url = new URL(markdownUrl); + url.pathname = url.pathname.replace(/\.md$/, '/'); + return url.toString(); +} + +function markdownTargets(baseUrl, llmsResponse) { + const origin = new URL(baseUrl).origin; + const candidates = extractMarkdownLinks(llmsResponse?.body || '') + .map((href) => { + try { + return new URL(href, llmsResponse.url).toString(); + } catch { + return null; + } + }) + .filter((url) => url && new URL(url).origin === origin) + .map(toMarkdownUrl) + .filter(Boolean); + + return [...new Set(candidates)].slice(0, COMPATIBILITY_SAMPLE_LIMIT); +} + +function urlsFromSitemap(xml) { + return [...xml.matchAll(/([^<]+)<\/loc>/gi)].map((match) => decodeHtmlEntities(match[1].trim())); +} + +async function sitemapMarkdownTargets(baseUrl) { + const origin = new URL(baseUrl).origin; + const index = await fetchResource(new URL('/sitemap-index.xml', baseUrl).toString(), { + accept: 'application/xml, text/xml;q=0.9, */*;q=0.1', + }); + if (index.status !== 200) return []; + + const sitemapUrls = urlsFromSitemap(index.body); + const leafSitemaps = sitemapUrls.filter((url) => url.endsWith('.xml')); + const pageUrls = + leafSitemaps.length === 0 + ? sitemapUrls + : ( + await Promise.all( + leafSitemaps.slice(0, 4).map((url) => + fetchResource(url, { accept: 'application/xml, text/xml;q=0.9, */*;q=0.1' }) + ) + ) + ).flatMap((response) => (response.status === 200 ? urlsFromSitemap(response.body) : [])); + + return [...new Set(pageUrls)] + .filter((url) => { + try { + return new URL(url).origin === origin; + } catch { + return false; + } + }) + .map(toMarkdownUrl) + .filter(Boolean) + .slice(0, COMPATIBILITY_SAMPLE_LIMIT); +} + +async function collectSamples(baseUrl) { + const llmsResponse = await fetchResource(new URL('/llms.txt', baseUrl).toString(), { + accept: 'text/plain, text/markdown;q=0.9, */*;q=0.1', + }); + const sitemapTargets = await sitemapMarkdownTargets(baseUrl); + const fallbackTargets = markdownTargets(baseUrl, llmsResponse); + const targets = [...new Set([...sitemapTargets, ...fallbackTargets])].slice( + 0, + COMPATIBILITY_SAMPLE_LIMIT + ); + const samples = []; + + for (const markdownUrl of targets) { + const htmlUrl = toHtmlUrl(markdownUrl); + const html = await fetchResource(htmlUrl, { accept: 'text/html, */*;q=0.1' }); + const markdown = await fetchResource(markdownUrl, { + accept: 'text/markdown, text/plain;q=0.9, */*;q=0.1', + }); + samples.push({ html, markdown }); + } + + return { llmsResponse, samples }; +} + +function assessBotProtection(samples) { + const observations = samples.flatMap((sample) => [sample.html, sample.markdown]); + if (observations.length === 0) { + return createResult( + 'bot-protection-interference', + 'authentication', + 'skip', + 'No first-party documentation samples were available to test sustained automated fetching' + ); + } + + const blocked = observations.filter( + (response) => + response.timeout || + isChallenge(response) || + [403, 429, 503].includes(response.status) + ); + if (blocked.length === 0) { + return createResult( + 'bot-protection-interference', + 'authentication', + 'pass', + `No bot-protection interference observed across ${observations.length} sustained automated fetches` + ); + } + + const status = blocked.length * 2 >= observations.length ? 'fail' : 'warn'; + const modes = [...new Set(blocked.map((response) => (response.timeout ? 'tarpit timeout' : isChallenge(response) ? 'challenge page' : `HTTP ${response.status}`)))]; + return createResult( + 'bot-protection-interference', + 'authentication', + status, + `${blocked.length}/${observations.length} sustained automated fetches showed bot-protection interference (${modes.join(', ')})`, + 'Exempt public documentation routes from behavioral bot enforcement. When limits are necessary, return an explicit 429 with Retry-After instead of challenge pages or stalled responses.', + { blocked_fetches: blocked.length, total_fetches: observations.length, modes } + ); +} + +function assessTransferSize(samples) { + const htmlSamples = samples.map((sample) => sample.html).filter((response) => response.status === 200); + if (htmlSamples.length === 0) { + return createResult( + 'page-size-transfer', + 'page-size', + 'skip', + 'No successful HTML responses were available to measure transfer size' + ); + } + + const maxBytes = Math.max(...htmlSamples.map((response) => response.bytes)); + const status = maxBytes > 10 * 1024 * 1024 ? 'fail' : maxBytes >= 1024 * 1024 ? 'warn' : 'pass'; + return createResult( + 'page-size-transfer', + 'page-size', + status, + `Largest decoded HTML response was ${Math.round(maxBytes / 1024)} KB across ${htmlSamples.length} sampled pages`, + status === 'pass' + ? null + : 'Reduce inline framework serialization, embedded data, and duplicate content. HTML-path agents pay this transfer cost before conversion.' + ); +} + +function extractContinuation(markdown, headers) { + const signals = []; + const linkHeader = headers.link || ''; + const headerMatch = linkHeader.match(/<([^>]+)>;\s*rel="?next"?/i); + if (headerMatch) signals.push({ href: headerMatch[1], position: markdown.length, source: 'Link header' }); + + const links = extractMarkdownLinks(markdown); + for (const href of links) { + if (/(?:[?&](?:page|offset|cursor)=|next)/i.test(href)) { + signals.push({ href, position: markdown.indexOf(href), source: 'markdown link' }); + } + } + if (/\b\d+\s+of\s+\d+\b/i.test(markdown) && signals.length === 0) { + signals.push({ href: null, position: markdown.search(/\b\d+\s+of\s+\d+\b/i), source: 'count marker' }); + } + return signals; +} + +async function assessSingleFetchCompleteness(samples) { + const markdownSamples = samples + .map((sample) => sample.markdown) + .filter((response) => response.status === 200 && response.body.length > 0); + if (markdownSamples.length === 0) { + return createResult( + 'single-fetch-completeness', + 'page-size', + 'skip', + 'No successful markdown responses were available to inspect for pagination' + ); + } + + const findings = []; + for (const response of markdownSamples) { + for (const continuation of extractContinuation(response.body, response.headers)) { + if (!continuation.href) { + findings.push({ status: 'fail', response, continuation, reason: 'pagination is declared without a continuation URL' }); + continue; + } + let target; + try { + target = new URL(continuation.href, response.url); + } catch { + findings.push({ status: 'fail', response, continuation, reason: 'continuation URL cannot be resolved' }); + continue; + } + const next = await fetchResource(target.toString(), { + accept: 'text/markdown, text/plain;q=0.9, */*;q=0.1', + }); + const absolute = /^https?:\/\//i.test(continuation.href); + const nearTop = continuation.position >= 0 && continuation.position / response.body.length <= 0.1; + const working = + next.status >= 200 && + next.status < 300 && + next.body.trim().length > 0 && + !isChallenge(next) && + !isSoft404(next); + findings.push({ + status: !working ? 'fail' : absolute && nearTop ? 'pass' : 'warn', + response, + continuation, + reason: !working ? 'continuation does not return substantive content' : !absolute ? 'continuation URL is relative' : 'continuation is declared after the first 10% of content', + }); + } + } + + if (findings.length === 0) { + return createResult( + 'single-fetch-completeness', + 'page-size', + 'pass', + `No pagination signals detected across ${markdownSamples.length} sampled markdown responses` + ); + } + const status = findings.some((finding) => finding.status === 'fail') + ? 'fail' + : findings.some((finding) => finding.status === 'warn') + ? 'warn' + : 'pass'; + return createResult( + 'single-fetch-completeness', + 'page-size', + status, + `${findings.length} pagination signal(s) found; ${findings.filter((finding) => finding.status === 'fail').length} broken and ${findings.filter((finding) => finding.status === 'warn').length} fragile`, + status === 'pass' + ? null + : 'Serve complete markdown in one response where possible. Otherwise put an absolute, working continuation link near the top of the response.', + { findings: findings.map(({ status: findingStatus, reason, response }) => ({ status: findingStatus, reason, url: response.url })) } + ); +} + +async function assessMarkdownLinkPortability(samples) { + const markdownSamples = samples + .map((sample) => sample.markdown) + .filter((response) => response.status === 200 && response.body.length > 0); + if (markdownSamples.length === 0) { + return createResult( + 'markdown-link-portability', + 'content-structure', + 'skip', + 'No successful markdown responses were available to inspect links' + ); + } + + const links = markdownSamples.flatMap((response) => + extractMarkdownLinks(response.body) + .filter((href) => !/^(mailto:|tel:|data:)/i.test(href)) + .map((href) => ({ href, response })) + ); + const pathRelative = links.filter(({ href }) => !/^(https?:\/\/|\/)/i.test(href)); + const rootRelative = links.filter(({ href }) => href.startsWith('/')); + const markdownLinks = links.filter(({ href }) => /\.md(?:[?#]|$)/i.test(href)).slice(0, COMPATIBILITY_SAMPLE_LIMIT); + const broken = []; + + for (const { href, response } of markdownLinks) { + const target = new URL(href, response.url).toString(); + const resolved = await fetchResource(target, { + accept: 'text/markdown, text/plain;q=0.9, */*;q=0.1', + }); + const contentType = resolved.headers['content-type'] || ''; + if ( + resolved.status < 200 || + resolved.status >= 300 || + isSoft404(resolved) || + isChallenge(resolved) || + !/(text\/markdown|text\/plain)/i.test(contentType) + ) { + broken.push({ href, url: target, status: resolved.status, content_type: contentType || null }); + } + } + + const status = pathRelative.length > 0 || broken.length > 0 ? 'fail' : rootRelative.length > 0 ? 'warn' : 'pass'; + return createResult( + 'markdown-link-portability', + 'content-structure', + status, + `${links.length} markdown link(s): ${links.length - pathRelative.length - rootRelative.length} absolute, ${rootRelative.length} root-relative, ${pathRelative.length} path-relative; ${broken.length}/${markdownLinks.length} sampled .md link(s) failed representation verification`, + status === 'pass' + ? null + : 'Generate absolute URLs in served markdown. Fetch a representative sample in CI and verify both successful status and the promised markdown representation.', + { absolute: links.length - pathRelative.length - rootRelative.length, root_relative: rootRelative.length, path_relative: pathRelative.length, broken_markdown_links: broken } + ); +} + +function bulkElements(html) { + const content = htmlToText(html); + const elements = []; + for (const table of html.match(/]*>[\s\S]*?<\/table>/gi) || []) { + const rows = (table.match(/= 25) elements.push({ type: 'table', rows, characters: text.length }); + } + for (const block of content.match(/```(?:json|ya?ml|csv)?[\s\S]{1024,}?```/gi) || []) { + elements.push({ type: 'data block', characters: block.length }); + } + for (const base64 of content.match(/[A-Za-z0-9+/]{1024,}={0,2}/g) || []) { + elements.push({ type: 'base64 payload', characters: base64.length }); + } + return { contentCharacters: content.length, elements }; +} + +function assessEmbeddedDataSerialization(samples) { + const analyses = samples + .map((sample) => ({ url: sample.html.url, ...bulkElements(sample.html.body) })) + .filter((analysis) => analysis.contentCharacters > 0); + if (analyses.length === 0) { + return createResult( + 'embedded-data-serialization', + 'content-structure', + 'skip', + 'No successful HTML responses were available to inspect for embedded bulk data' + ); + } + + const dominant = analyses + .map((analysis) => ({ + ...analysis, + bulkCharacters: analysis.elements.reduce((total, element) => total + element.characters, 0), + })) + .filter((analysis) => analysis.bulkCharacters / analysis.contentCharacters > 0.5); + const failing = dominant.filter((analysis) => analysis.contentCharacters > 100_000); + const warning = dominant.filter((analysis) => analysis.contentCharacters >= 50_000 && analysis.contentCharacters <= 100_000); + const status = failing.length > 0 ? 'fail' : warning.length > 0 ? 'warn' : 'pass'; + const maxBulkShare = Math.max( + ...analyses.map((analysis) => analysis.elements.reduce((total, element) => total + element.characters, 0) / analysis.contentCharacters) + ); + return createResult( + 'embedded-data-serialization', + 'content-structure', + status, + `${dominant.length}/${analyses.length} sampled pages have bulk data as the dominant converted-content contributor (largest share ${Math.round(maxBulkShare * 100)}%)`, + status === 'pass' + ? null + : 'Split large generated tables into self-contained pages, load bulk payloads on demand, and place prose before bulk data. Do not replace complete pages with pagination windows.', + { dominant_pages: dominant.map(({ url, contentCharacters, bulkCharacters, elements }) => ({ url, content_characters: contentCharacters, bulk_characters: bulkCharacters, elements })) } + ); +} + +function interactionEffects(results) { + const resultFor = (id) => results.find((result) => result.id === id); + const interactions = []; + const botProtection = resultFor('bot-protection-interference'); + if (botProtection && ['warn', 'fail'].includes(botProtection.status)) { + interactions.push({ + id: 'bot-protection-degrading-scan-reliability', + status: 'observed', + affected_checks: results + .filter((result) => result.category !== 'authentication' && result.status !== 'skip') + .map((result) => result.id), + message: 'Bot protection interfered with sustained fetching, so multi-page findings may reflect a partial sample rather than the complete site.', + }); + } + + const dynamicCheckIds = [ + 'page-size-transfer', + 'single-fetch-completeness', + 'embedded-data-serialization', + 'markdown-content-parity', + 'page-size-markdown', + 'page-size-html', + ]; + const dynamicSignals = dynamicCheckIds.filter((id) => { + const result = resultFor(id); + return result && ['warn', 'fail'].includes(result.status); + }); + if (dynamicSignals.length >= 2) { + interactions.push({ + id: 'dynamic-content-rendered-statically', + status: 'observed', + affected_checks: dynamicSignals, + message: 'Multiple dynamic-content failure signals co-occur. Review HTML and markdown rendering paths together: serialization payloads, static bulk-data output, parity, and pagination can fail independently.', + }); + } + return interactions; +} + +function scanReliability(results) { + const botProtection = results.find((result) => result.id === 'bot-protection-interference'); + if (botProtection && ['warn', 'fail'].includes(botProtection.status)) { + return { + status: 'partial', + reason: 'bot-protection-interference', + message: 'Sustained automated fetching was interrupted, so multi-page findings may not represent the complete site.', + }; + } + return { + status: 'complete', + reason: null, + message: 'No bot-protection interference was observed during this scan.', + }; +} + +export async function runV06CompatibilityChecks(baseUrl, existingResults = []) { + const existingIds = new Set(existingResults.map((result) => result.id)); + const missingCheckIds = new Set( + [...V06_CHECK_IDS].filter((checkId) => !existingIds.has(checkId)) + ); + if (missingCheckIds.size === 0) return { results: [], samples: [] }; + const { samples } = await collectSamples(baseUrl); + const candidates = [ + assessBotProtection(samples), + assessTransferSize(samples), + await assessSingleFetchCompleteness(samples), + await assessMarkdownLinkPortability(samples), + assessEmbeddedDataSerialization(samples), + ]; + return { + results: candidates.filter((result) => missingCheckIds.has(result.id)), + samples, + }; +} + +export function buildReport(raw, supplementalResults = []) { + const results = [...raw.results, ...supplementalResults]; + const summary = statusSummary(results); + const nativeCliScore = raw.summary?.score ?? raw.score ?? estimateScore(raw.results); + const score = supplementalResults.length > 0 ? estimateScore(results) : nativeCliScore ?? estimateScore(results); const grade = scoreToGrade(score); // Group results by category @@ -163,15 +706,16 @@ function buildReport(raw) { return { url: raw.url, timestamp: raw.timestamp || new Date().toISOString(), + spec_version: SPEC_VERSION, score, grade, - total_checks: summary.total, - summary: { - pass: summary.pass, - fail: summary.fail, - warn: summary.warn, - skip: summary.skip, - }, + score_method: + supplementalResults.length > 0 + ? 'unweighted compatibility estimate; warnings count as half-passes' + : 'native afdocs CLI score', + legacy_cli_score: supplementalResults.length > 0 ? nativeCliScore : null, + total_checks: results.length, + summary, categories: Object.fromEntries( Object.entries(categories).map(([name, cat]) => [ name, @@ -180,6 +724,8 @@ function buildReport(raw) { ), issues, all_results: results.map((r) => ({ id: r.id, category: r.category, status: r.status, message: r.message })), + scan_reliability: scanReliability(results), + interaction_effects: interactionEffects(results), }; } @@ -198,10 +744,17 @@ function estimateScore(results) { function printSummary(report) { console.log(`\nAFDocs Audit — ${report.url}`); + console.log(`Spec: v${report.spec_version} compatibility audit`); console.log(`Score: ${report.score}/100 (${report.grade})`); + if (report.legacy_cli_score != null) { + console.log(`Legacy 23-check CLI score: ${report.legacy_cli_score}/100 (not comparable to the v0.6 score)`); + } console.log( `Checks: ${report.total_checks} total | ${report.summary.pass} pass, ${report.summary.fail} fail, ${report.summary.warn} warn, ${report.summary.skip} skip` ); + if (report.scan_reliability.status !== 'complete') { + console.log(`Reliability: ${report.scan_reliability.status} — ${report.scan_reliability.message}`); + } if (report.issues.length === 0) { console.log('\n✅ All checks passed!'); @@ -225,58 +778,74 @@ function printSummary(report) { console.log(` ⚠ ${w.id}: ${w.message}`); } } + + if (report.interaction_effects.length > 0) { + console.log('\nInteraction effects:'); + for (const effect of report.interaction_effects) { + console.log(` ⚠ ${effect.id}: ${effect.message}`); + } + } } -// Main -const args = parseArgs(process.argv.slice(2)); +async function main() { + const args = parseArgs(process.argv.slice(2)); -// Preflight: if the site is behind a Vercel Firewall bot challenge, the -// afdocs crawler can't reach any real content and every check becomes a false -// positive. Bail out with a clear "invalid" status so we never publish a -// misleading score. -const challenge = await detectVercelChallenge(args.url); -if (challenge) { - const invalidReport = { - url: args.url, - timestamp: new Date().toISOString(), - status: 'invalid', - reason: 'vercel-firewall-challenge', - detail: challenge, - }; + // Preflight: if the site is behind a Vercel Firewall bot challenge, the + // afdocs crawler can't reach any real content and every check becomes a false + // positive. Bail out with a clear "invalid" status so we never publish a + // misleading score. + const challenge = await detectVercelChallenge(args.url); + if (challenge) { + const invalidReport = { + url: args.url, + timestamp: new Date().toISOString(), + spec_version: SPEC_VERSION, + status: 'invalid', + reason: 'vercel-firewall-challenge', + detail: challenge, + }; - console.error( - `\n⛔ AFDocs audit INVALID — ${args.url} is behind a Vercel Firewall bot challenge.` - ); - console.error( - ` Probe ${challenge.probeUrl} returned HTTP ${challenge.status}` + - (challenge.mitigated ? ` (x-vercel-mitigated: ${challenge.mitigated})` : '') + - '.' - ); - console.error( - ' The afdocs crawler cannot solve the JavaScript challenge, so a score cannot be computed.' - ); - console.error( - ' Fix: see .agents/skills/afdocs-audit/references/vercel-firewall-challenge.md' - ); + console.error( + `\n⛔ AFDocs audit INVALID — ${args.url} is behind a Vercel Firewall bot challenge.` + ); + console.error( + ` Probe ${challenge.probeUrl} returned HTTP ${challenge.status}` + + (challenge.mitigated ? ` (x-vercel-mitigated: ${challenge.mitigated})` : '') + + '.' + ); + console.error( + ' The afdocs crawler cannot solve the JavaScript challenge, so a score cannot be computed.' + ); + console.error( + ' Fix: see .agents/skills/afdocs-audit/references/vercel-firewall-challenge.md' + ); - if (args.output) { - const outputPath = resolve(args.output); - writeFileSync(outputPath, JSON.stringify(invalidReport, null, 2)); - console.error(`\nInvalid-audit report written to ${outputPath}`); + if (args.output) { + const outputPath = resolve(args.output); + writeFileSync(outputPath, JSON.stringify(invalidReport, null, 2)); + console.error(`\nInvalid-audit report written to ${outputPath}`); + } + + process.exitCode = 2; + return; } - process.exit(2); -} + console.log(`Running AFDocs check on ${args.url}...`); -console.log(`Running AFDocs check on ${args.url}...`); + const raw = runAfdocsCheck(args.url); + console.log(`Running ${SPEC_VERSION} compatibility checks on a bounded markdown sample...`); + const { results: supplementalResults } = await runV06CompatibilityChecks(args.url, raw.results); + const report = buildReport(raw, supplementalResults); -const raw = runAfdocsCheck(args.url); -const report = buildReport(raw); + printSummary(report); -printSummary(report); + if (args.output) { + const outputPath = resolve(args.output); + writeFileSync(outputPath, JSON.stringify(report, null, 2)); + console.log(`\nReport written to ${outputPath}`); + } +} -if (args.output) { - const outputPath = resolve(args.output); - writeFileSync(outputPath, JSON.stringify(report, null, 2)); - console.log(`\nReport written to ${outputPath}`); +if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href) { + await main(); } diff --git a/.agents/skills/afdocs-audit/scripts/afdocs_audit.test.mjs b/.agents/skills/afdocs-audit/scripts/afdocs_audit.test.mjs new file mode 100644 index 000000000..659b8112e --- /dev/null +++ b/.agents/skills/afdocs-audit/scripts/afdocs_audit.test.mjs @@ -0,0 +1,81 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { buildReport } from './afdocs_audit.mjs'; + +const legacyResult = { + id: 'page-size-html', + category: 'page-size', + status: 'fail', + message: 'Converted content is too large', +}; + +test('creates a transparent v0.6 compatibility report when the CLI has 23 checks', () => { + const report = buildReport( + { + url: 'https://docs.example.com', + score: 80, + summary: {}, + results: [ + legacyResult, + { + id: 'page-size-markdown', + category: 'page-size', + status: 'fail', + message: 'Markdown is too large', + }, + { + id: 'markdown-content-parity', + category: 'observability', + status: 'fail', + message: 'Markdown differs from HTML', + }, + ], + }, + [ + { + id: 'page-size-transfer', + category: 'page-size', + status: 'warn', + message: 'Large transfer', + }, + { + id: 'embedded-data-serialization', + category: 'content-structure', + status: 'fail', + message: 'Bulk table dominates page content', + }, + { + id: 'bot-protection-interference', + category: 'authentication', + status: 'warn', + message: 'Sustained fetches are throttled', + }, + ] + ); + + assert.equal(report.spec_version, '0.6.0'); + assert.equal(report.total_checks, 6); + assert.equal(report.legacy_cli_score, 80); + assert.match(report.score_method, /compatibility estimate/); + assert.equal(report.summary.fail, 4); + assert.equal(report.summary.warn, 2); + assert.equal(report.scan_reliability.status, 'partial'); + assert.deepEqual( + report.interaction_effects.map((effect) => effect.id), + ['bot-protection-degrading-scan-reliability', 'dynamic-content-rendered-statically'] + ); +}); + +test('uses the native CLI score after afdocs supplies every v0.6 check', () => { + const report = buildReport({ + url: 'https://docs.example.com', + score: 91, + summary: {}, + results: [legacyResult], + }); + + assert.equal(report.score, 91); + assert.equal(report.legacy_cli_score, null); + assert.equal(report.score_method, 'native afdocs CLI score'); +}); diff --git a/.agents/skills/afdocs-fix/SKILL.md b/.agents/skills/afdocs-fix/SKILL.md index d0bfde8c7..cc81bdffe 100644 --- a/.agents/skills/afdocs-fix/SKILL.md +++ b/.agents/skills/afdocs-fix/SKILL.md @@ -125,6 +125,38 @@ grep -A1 "label:" astro.config.mjs | grep "paths:" **Files**: `src/integrations/docs-markdown-integration.js`, `src/pages/[...slug].md.ts` +### markdown-link-portability (Content Structure) + +**What's wrong**: Generated markdown contains root-relative or path-relative links, or `.md` links resolve to HTML/error content. + +**Fix**: Update `src/integrations/docs-markdown-integration.js` to rewrite internal links to absolute `https://docs.warp.dev/...` URLs after Turndown conversion. Preserve same-document fragment links (`#section`) unchanged. Add a test to `src/integrations/docs-markdown-integration.test.js` that verifies an internal link is absolute, then build and fetch a representative `.md` page to verify `Content-Type: text/markdown`. + +**Files**: `src/integrations/docs-markdown-integration.js`, `src/integrations/docs-markdown-integration.test.js` + +### single-fetch-completeness (Page Size and Truncation Risk) + +**What's wrong**: A served markdown response is paginated without a working, absolute continuation declared near the top of the content. + +**Fix**: Do not copy HTML-UI pagination into markdown when the complete resource fits within the page-size limits. Otherwise add an absolute continuation link before the main content and test that it returns substantive markdown rather than an error page. Audit the markdown generator and any data-driven route that produces paginated output. + +**Files**: `src/integrations/docs-markdown-integration.js`, `src/pages/[...slug].md.ts`, affected data-driven page or endpoint + +### page-size-transfer / embedded-data-serialization (Page Size and Content Structure) + +**What's wrong**: HTML responses are large before conversion, or static bulk data (tables, JSON, base64) dominates content that agents receive. + +**Fix**: Treat this as a design or rendering-pipeline change, not an automatic content split. Attribute the bytes to inline framework payloads, generated tables, or data blocks first. Move large payloads behind on-demand requests, split generated tables into self-contained topic pages, and keep prose before bulk content. Do not replace a complete markdown page with pagination windows. + +**Files**: Varies by the reported page and its rendering/data source. + +### bot-protection-interference (Authentication and Access) + +**What's wrong**: Sustained automated requests receive challenge pages, time out, or are blocked after volume thresholds are reached. + +**Fix**: This requires CDN/WAF configuration, not a docs-repository code change. Exempt public docs paths from behavioral enforcement or scope enforcement to interactive product surfaces. Prefer explicit `429` responses with `Retry-After` over challenge interstitials or stalled responses. A partial scan must not be compared with prior audit scores. + +**Files**: Vercel Firewall or the active CDN/WAF configuration; see `afdocs-audit/references/vercel-firewall-challenge.md`. + ### http-status-codes (URL Stability) **What's wrong**: The site returns 200 for non-existent pages (soft 404). @@ -157,6 +189,8 @@ These checks require infrastructure or design changes that can't be automated: - **content-start-position** — Inherent to Starlight's layout. Mitigated by content negotiation and llms.txt directives. See `known-exceptions.md`. - **page-size-markdown / page-size-html** — Requires editorial decision to split long pages. Flag in the report but do not auto-fix. +- **page-size-transfer / embedded-data-serialization** — Requires rendering-pipeline or information-architecture decisions after attributing the bulk bytes. +- **bot-protection-interference** — Requires CDN/WAF configuration and turns the scan into a partial observation rather than a deployable docs change. - **section-header-quality** — Content-level change requiring human judgment. ## Applying fixes