test(functional): upgrade Playwright and fix Sync tests for keys_optional by vbudhram · Pull Request #20791 · mozilla/fxa

vbudhram · 2026-06-26T15:11:32Z

Because

The functional suite was pinned to Playwright 1.44.1; upgrading to 1.61.1 (Firefox 151) keeps the browser engine current and unblocks newer test APIs.
Firefox 151 / decoupled Sync (keys_optional) changed signin routing, breaking several Sync and OAuth functional tests.

This pull request

Upgrades @playwright/test 1.44.1 → 1.61.1 (packages/functional-tests/package.json, yarn.lock).
Bumps the fxa-circleci CI images v9 → v10 (.circleci/config.yml) so the functional-test-runner ships the matching browsers.
Updates keys_optional mocks to be capability-based (Sync not decoupled) rather than browser-version-based in vpnIntegration.spec.ts, passkeyPasswordFallback.spec.ts, and the syncV3/* specs.
Adds waitForOauthPage() in relier.ts so email-first/AAL2 nav waits for the settled /oauth page, not the intermediate /authorization redirect.
Adds respondToWebChannelMessageAlways() in layout.ts for flows that re-dispatch fxa_status.
Adds gotoSyncSession() in lib/sync-helpers.ts to abstract /pair navigation across Sync tests.

Issue that this pull request solves

Closes: https://mozilla-hub.atlassian.net/browse/FXA-13647

Checklist

My commit is GPG signed.
If applicable, I have modified or added tests which pass locally.
I have added necessary documentation (if appropriate).
I have verified that my changes render correctly in RTL (if appropriate).
I have manually reviewed all AI generated code.

Other information

Requires the -v10 CI image (ci-functional-test-runner-v10) to be published before functional tests pass — built via the update-ci-image branch / force-deploy-fxa-ci-images. Verify by re-running the "Firefox Functional Tests - Playwright (PR)" job once -v10 is in the registry.

Copilot

Pull request overview

This PR upgrades the functional test suite to Playwright 1.61.1 and adjusts Sync/OAuth functional tests to account for Firefox’s keys_optional/decoupled-Sync routing changes, including more robust navigation + WebChannel mocking for browser-driven Sync handshakes.

Changes:

Upgrades @playwright/test (and Playwright deps) and bumps CircleCI runner images to -v10 to ship matching browsers.
Refactors Sync-entry navigation across many specs to use /pair via a new gotoSyncSession() helper, and updates several tests for new signin routing.
Improves test stability with new/updated helpers (e.g., settled /oauth wait, repeated WebChannel responder, email-first fill retry, QR decode retry).

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
yarn.lock	Updates Playwright-related dependency resolutions/checksums.
packages/functional-tests/package.json	Bumps `@playwright/test` to 1.61.1.
packages/functional-tests/playwright.config.ts	Makes retries always-on (1) to absorb rare flakes.
packages/functional-tests/lib/sync-helpers.ts	Adds `gotoSyncSession()` helper to centralize Sync session entry via `/pair`.
packages/functional-tests/pages/signin.ts	Makes email-first form fill more resilient via `expect().toPass()` retry.
packages/functional-tests/pages/settings/totp.ts	Retries QR screenshot+decode to avoid blank-canvas race.
packages/functional-tests/pages/relier.ts	Adds a stricter wait for the settled `/oauth` page after redirects.
packages/functional-tests/pages/layout.ts	Adds `respondToWebChannelMessageAlways()` and refactors responder install logic.
packages/functional-tests/tests/syncV3/signIn.spec.ts	Uses `gotoSyncSession()` instead of hardcoded Sync URLs.
packages/functional-tests/tests/syncV3/settings.spec.ts	Uses `gotoSyncSession()` and minor fixture formatting adjustments.
packages/functional-tests/tests/syncV3/fxDesktopV3ForceAuth.spec.ts	Updates force-auth flow to handle chained OAuth re-auth/token steps.
packages/functional-tests/tests/settings/recoveryPhone.spec.ts	Starts Sync handshake via `/pair` before recovery-phone assertions.
packages/functional-tests/tests/react-conversion/signup.spec.ts	Initiates Sync v3 signup flow via `/pair`.
packages/functional-tests/tests/react-conversion/oauthSignup.spec.ts	Initiates Sync OAuth flows via `/pair` and updates send-tab flow assertions.
packages/functional-tests/tests/passkeyAuth/passkeySetPassword.spec.ts	Uses `/pair`-initiated Sync handshake for passkey set-password flows.
packages/functional-tests/tests/passkeyAuth/passkeyPasswordFallback.spec.ts	Refactors setup + fallback helpers and uses capability-based `FxAStatus` mock.
packages/functional-tests/tests/pairing/pairChoice.spec.ts	Adds explicit sign-in steps since `/pair` redirects for fresh browsers.
packages/functional-tests/tests/oauth/syncSignIn.spec.ts	Uses `gotoSyncSession()` and updates expectations for cached-signin behavior.
packages/functional-tests/tests/misc/vpnIntegration.spec.ts	Switches to capability-based “Sync not decoupled” mock and always-respond WebChannel handler.
packages/functional-tests/tests/misc/recoveryKeyPromoInline.spec.ts	Uses `gotoSyncSession()` and adjusts expectations when session persists.
packages/functional-tests/tests/key-stretching-v2/signInTokenCode.spec.ts	Uses `gotoSyncSession()` while preserving stretch param behavior.
packages/functional-tests/tests/cms/cms.spec.ts	Re-enters Sync CMS entrypoint after `/pair` handshake; updates flow steps.
packages/functional-tests/tests/cms/cms-2fa.spec.ts	Enrolls TOTP out-of-band and reworks Sync CMS 2FA test to fit new routing.
.circleci/README.md	Documents image rebuild/tag bump process for Playwright upgrades.
.circleci/config.yml	Bumps CI images `-v9` → `-v10` across executors/build/push steps.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        await expect(connectAnotherDevice.fxaConnected).toBeVisible();
+        await connectAnotherDevice.clickNotNowPair();


vbudhram · 2026-07-01T19:58:24Z

+import { BaseTarget } from './targets/base';
+
+/**
+ * Initiate a desktop Sync sign-in/up session in the test browser.


I dont really like this, but it is the "cleanest" way to get the Firefox desktop browser to initate a login session. We can't fake the OAuth params because Firefox checks these and will not log a user in completely because of that. I've pull this into a helper so that we can change it later.

TLDR; goto /pair to Sign into Sync. The FxA sends webchannel messages and browser returns a valid login url that we direct to.

vbudhram · 2026-07-01T19:58:53Z

+    await expect(this.step1QRCode).toBeVisible();
+
+    // Retry the screenshot+decode: in the change-2FA flow the canvas can repaint
+    // with the new secret a beat after the heading appears, so a single early


Unrelated, but ran into this error a few times locally, figured I would just fix it.

vbudhram · 2026-07-01T20:02:09Z

-      syncDesktopOAuthQueryParams.set('entrypoint', ENTRYPOINT_SYNC);
-      await signin.goto('/authorization', syncDesktopOAuthQueryParams);
-
-      await assertCmsCustomization(page, {


This was already asserted below, removed duplicate.

vbudhram · 2026-07-01T20:05:17Z

    const { email, password } = await testAccountTracker.signUpPasswordless();
+    // Android (Fenix) has no keys_optional (Sync not decoupled), so signin needs
+    // a password and routes to /set_password; fxa_status repeats, so answer all.
+    const syncNotDecoupledStatus: FxAStatusResponse = {


@LZoog Mind a santify check on this test case?

LZoog · 2026-07-03T22:00:49Z

@vbudhram just want to note that in #20817 I had to temporarily add a couple of lines to fill out the "repeat password" form since we can't test the keys_optional capability yet. Depending on what's merged first one of us might need to update that (at least update the comment since it references the ticket).

…th/CMS tests Because: * Playwright 1.44 -> 1.61.1 (Firefox 151) and decoupled Sync (keys_optional) broke several Sync/OAuth/CMS functional tests. * Hardcoded fake OAuth params (keys_jwk) no longer complete key derivation on FF151, causing password re-prompt loops in Sync flows. This commit: * Upgrades Playwright to 1.61.1 and bumps the CI images. * Routes Sync sign-in/up tests through the real /pair handshake (gotoSyncSession) instead of hardcoded fake OAuth params. * Restructures the CMS 2FA TOTP-Sync test to enroll TOTP out-of-band so a single sign-in exercises the branded signin_totp_code flow (no unusable second /pair). * Reworks the SmartWindow-after-Sync test into a two-phase flow that asserts the real fxa_oauth_login WebChannel scope instead of only reaching a signed-in view. * Converts the Sync passkey password-fallback test to the real Firefox Sync browser (syncOAuthBrowserPages + /pair), dropping the unnecessary no-keys_optional mock since the passkey simply cannot derive Sync keys. * Tightens passkey assertions: wrong-password now asserts the Incorrect password alert (not any 'password' text), asserts the passkey button on prompt=login, and corrects a misleading 'passkey-only account' test name. * Bounds email-first waits (waitForURL(action=email), fillOutEmailFirstForm toPass) so broken flows fail fast instead of hitting the 60s test timeout. * Retries once locally and on CI to absorb rare parallel-worker flakes. * Reloads the avatar settings page with waitUntil: 'commit' so the heading assertion gates readiness, avoiding a hang when a background request stalls the load event. * Adds an end-to-end AMO add-on submission test asserting a passwordless passkey developer is diverted to inline TOTP setup instead of looping on cached sign-in (stage target only).

nshirley

Two kind of nit things, and then I tried to approve the tests but looks like they failed. Is it expected they should run on PR here? Or will there need to be additional changes to the deployed image after merged?

nshirley · 2026-07-14T18:37:45Z

 circleci config process .circleci/config.yml > .circleci/local.yml
 circleci local execute -c .circleci/local.yml --job test-many
 ```
+


Thanks for including these steps!

nshirley · 2026-07-14T19:56:58Z

+            return;
+          }
+
+          if (once) {


Smart! IIRC, a registered event listeners doesn't prevent other listeners on the same event, right? So we're not at risk of having this consume an event before something else needs it?

nshirley · 2026-07-14T21:17:17Z

+    await expect(async () => {
+      await this.emailTextbox.fill(email);
+      await expect(this.emailFirstSubmitButton).toBeEnabled();
+    }).toPass({ timeout: 15000 });


This feels like a really long timeout to allow this. Also, should the .fill() be outside this and we're just waiting for the expected enabled state, that way we give as much of this timeout to the enabled state?

nshirley · 2026-07-14T21:26:05Z

+      //    account-2FA gap.
+      await settings.signOut();
+      await page.goto(AMO_DEVELOPER_HUB);
+      await page


I do appreciate the reasoning behind this kind of test, but it makes me nervous reaching into another page like this that we don't control. Obviously, it's only a few steps, but if they change this then our test will fail. Is this not something we can push into an integration or unit test within auth-server? Candidly, I'm not familiar with how/where the check happens to redirect the user to the inline TOTP setup, but that feels like the right place to test this, not with a functional test. What do you think?

vbudhram force-pushed the fxa-13647 branch 3 times, most recently from a3887da to 648cca0 Compare June 30, 2026 17:29

vbudhram marked this pull request as ready for review June 30, 2026 17:50

vbudhram requested a review from a team as a code owner June 30, 2026 17:50

vbudhram force-pushed the fxa-13647 branch 4 times, most recently from 26a1a8c to 743877d Compare June 30, 2026 20:43

vbudhram self-assigned this Jul 1, 2026

vbudhram force-pushed the fxa-13647 branch from 743877d to 6e13692 Compare July 1, 2026 16:06

Copilot AI review requested due to automatic review settings July 1, 2026 16:06

Copilot started reviewing on behalf of vbudhram July 1, 2026 16:07 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread packages/functional-tests/tests/cms/cms.spec.ts

Comment on lines +373 to +374

await expect(connectAnotherDevice.fxaConnected).toBeVisible();

await connectAnotherDevice.clickNotNowPair();

vbudhram force-pushed the fxa-13647 branch from 6e13692 to 7d86937 Compare July 1, 2026 17:55

vbudhram commented Jul 1, 2026

View reviewed changes

vbudhram force-pushed the fxa-13647 branch 3 times, most recently from 0d6ccd5 to 176b525 Compare July 2, 2026 18:10

vbudhram force-pushed the fxa-13647 branch from 176b525 to a1113e4 Compare July 7, 2026 18:05

nshirley self-assigned this Jul 14, 2026

vbudhram force-pushed the fxa-13647 branch from a1113e4 to ed61cad Compare July 14, 2026 18:31

nshirley reviewed Jul 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(functional): upgrade Playwright and fix Sync tests for keys_optional#20791

test(functional): upgrade Playwright and fix Sync tests for keys_optional#20791
vbudhram wants to merge 1 commit into
mainfrom
fxa-13647

vbudhram commented Jun 26, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

vbudhram Jul 1, 2026

Uh oh!

vbudhram Jul 1, 2026

Uh oh!

vbudhram Jul 1, 2026

Uh oh!

vbudhram Jul 1, 2026

Uh oh!

LZoog commented Jul 3, 2026

Uh oh!

nshirley left a comment

Uh oh!

nshirley Jul 14, 2026

Uh oh!

nshirley Jul 14, 2026

Uh oh!

nshirley Jul 14, 2026

Uh oh!

nshirley Jul 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		await expect(connectAnotherDevice.fxaConnected).toBeVisible();
		await connectAnotherDevice.clickNotNowPair();

Uh oh!

Conversation

vbudhram commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Because

This pull request

Issue that this pull request solves

Checklist

Other information

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LZoog commented Jul 3, 2026

Uh oh!

nshirley left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vbudhram commented Jun 26, 2026 •

edited

Loading