[test] Add an assertions mode to the axe regression harness

[michelengelen]

Adds an assertions mode to the axe regression harness so a fixture can assert
every axe rule it exercises, not only the CSS-dependent visual ones. This is the
foundation the per-component WCAG conformance reports build on.

Also stops excluding the progress slug from the demo bundle so axe can reach
the LinearProgress demos; screenshots stay disabled slug-wide for it, since the
animated bars are flaky.

Original work by @mj12albert, extracted from #48708 so the harness change reviews
separately from the Button report.

Part of the WCAG conformance effort (see #14187).

Important

How to review this PR

Layer 1 of 14 in a stacked series (#48915 → #48926). Its branch is the base of the series. GitHub cannot chain PR bases across a fork, so every PR in the series targets master instead.

Its diff is exactly its own change — a0b8a93.

Part of the WCAG conformance effort (#14187).

[code-infra-dashboard]

Deploy preview

https://deploy-preview-48915--material-ui.netlify.app/

Bundle size

Bundle	Parsed size	Gzip size
@mui/material	0B(0.00%)	0B(0.00%)
@mui/lab	0B(0.00%)	0B(0.00%)
@mui/private-theming	0B(0.00%)	0B(0.00%)
@mui/system	0B(0.00%)	0B(0.00%)
@mui/utils	0B(0.00%)	0B(0.00%)

Details of bundle changes

---

Check out the code infra dashboard for more information about this PR.

[michelengelen]

Follow-up: #48936 adds the Playwright suite for the CSS-dependent criteria (1.4.10, 1.4.4, 1.4.12) on top of this harness change, and #48935 wires pnpm a11y:scorecard:check into test_static.

[michelengelen]

Correction to my earlier comment: the scorecard CI wiring is in #48925, not #48935 (that number does not exist). Follow-ups on top of this harness change are #48936 (Playwright suite for 1.4.10 / 1.4.4 / 1.4.12) and #48925 (pnpm a11y:scorecard:check in test_static).

[michelengelen]

Correcting myself again, properly this time: #48935 is [release] v9.3.1 — it exists, it is just unrelated to this series. The scorecard CI wiring is in #48925. Apologies for the noise.

[siriwatknp]

👍 Thanks for splitting this out of #48708. The assertions refactor reads well — shouldAssert keeps the visual default exactly as before, and a11yRule cannot be undefined at the new call site.

A few things I found.

[michelengelen]

Thanks — this was a good catch on all counts. Everything is addressed; the branch is force-pushed.

#	Change
2	demoMeta.test.ts is back to master exactly. You were right that all three still pass here — this layer changes no tests, so it should not touch the file at all.
3	Added test/regressions/a11y/axe.test.ts with your three cases plus three more: recording happens in both modes, visual still asserts the CSS-dependent rules, and skipAssertions keeps the result while dropping the assertion.
4	Moved the Avatar 1.4.12 check to #48917, where avatars is actually enrolled.
5	Restored the original sentence. The forward-looking wording now lives in the layer that adds the first accessibility.md.
7	Set assertions: 'all' on the real buttons rule, as you suggested. That makes both docs accurate and gets the new branch exercised in CI, which covers 3 from the other direction too.
8	Intended, yes — an unresolved rule is not evidence of conformance. Documented it on the assertions JSDoc, including skipAssertions as the escape hatch for a known-noisy incomplete.
9	Took the suggestion verbatim.

Two further problems surfaced while fixing these, both mine:

• A stray unformatted buttons.a11y.json had been swept into this layer by a git add -A. Prettier rejects it, so it would have failed test_static. Removed.
• index.test.js was being clobbered layer to layer — four layers modify it and each was copying its own older version, which is how the Avatar block ended up here in the first place. It is now composed cumulatively and formatted before commit.

The root cause of 2, 4, 5 and 7 is the same: I split the series mechanically and validated that each layer's tests pass and that the top of the stack was correct. Neither check catches a layer that deletes a passing test or carries a test for a component it does not touch. Reading each layer as a PR in its own right, which is what you did, is the check that was missing.

Points 1 and 6 are not in the API response for this review — the numbering starts at 2. Did they get dropped before submitting?

[michelengelen]

PR review

The assertions mode is implemented correctly: the default stays visual, so existing enrolments keep their behavior, and the all mode asserts the incomplete bucket deliberately and documents why. Docs (AGENTS.md, README) move with the code. Nothing is merge-blocking; one convention nit.

Tests (1)

1. 🟡 New test file imports expect from vitest instead of chai

Location: test/regressions/a11y/axe.test.ts:1

import { describe, it, expect } from 'vitest';

The sibling files (demoMeta.test.ts, axe.ts) import expect from chai. Vitest's chai-compatible expect makes the .to.deep.equal / .to.throw chains work, but the file breaks the local convention and the two expects format failures differently.

Failure scenario: contributors copy the inconsistent pattern; a plugin matcher or expect.fail behaves differently from the sibling suites.

Fix: import { expect } from 'chai'; and keep describe/it from vitest, as demoMeta.test.ts does.

Verdict

Approve after nits - the logic and the new-mode test coverage are sound; only a convention nit remains.

---

🤖 Review generated with Claude Code

[siriwatknp]

👍

[silviuaavram]

mention that 'visual' is default with @default