[AIGTWY-4759] Managed integration tests: green lanes + coverage - #699
Merged
Merged
Conversation
david-siqi-liu
added
the
quicktree-assisted
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
from
September 17, 2026 00:11
c24cf0d to
86622f6
Compare
david-siqi-liu
changed the title
david-siqi-liu
marked this pull request as ready for review
September 17, 2026 00:11
david-siqi-liu
requested review from
AarushiShah-db,
lilly-luo and
rohita5l
as code owners
September 17, 2026 00:11
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
2 times, most recently
from
September 17, 2026 01:14
c6c82ba to
fbc97d8
Compare
david-siqi-liu
changed the title
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
from
September 17, 2026 01:27
fbc97d8 to
3c4cab0
Compare
david-siqi-liu
changed the title
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
2 times, most recently
from
September 17, 2026 01:51
e642107 to
bd11153
Compare
david-siqi-liu
changed the title
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
from
September 17, 2026 02:34
648c998 to
de07883
Compare
The managed integration lanes were failing (the managed workspace IP-blocked CI runners), so they were non-blocking and thin. This makes them work and adds real coverage. - Install both agent CLIs in each managed lane: a managed config enables both agents and `ug configure` applies it to every enabled agent, so a lane with only one agent failed configuring the other. - Live `managed` tests against ca-central's published config: per-agent apply and an idempotent re-configure. - New `managed_fixture` layer: inject the admin CodingAgentConfig via the built-in UCODE_MANAGED_CONFIG_STUB hook so the real `ug configure` path is exercised across shapes the live workspace does not publish (enabled-agents allowlist per agent, config-driven model list, tracing OTEL env). Each case differs from the published config in what it asserts, so a pass proves the injected config drove configure. Auth, config writers, and binaries stay real; the fetch/wire contract stays covered by the un-stubbed `managed` tests. - Sanction the input-only stub in tests/AGENTS.md and tests/CLAUDE.md; run both markers in the existing managed lanes. Lanes stay non-blocking until proven stable. Co-authored-by: Isaac <no-reply@databricks.com>
david-siqi-liu
force-pushed
the
david/aigtwy-4759-managed-diagnostics
branch
from
September 17, 2026 03:11
de07883 to
8a9ee0d
Compare
david-siqi-liu
changed the title
lilly-luo
approved these changes
Sep 17, 2026
Replace the config-file assertions with tests that launch the real agent and drive /model and /mcp, so they verify what the agent actually presents, not the generated config (asserting config is unit tests' job). Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
# Conflicts: # tests/integration/README.md
lilly-luo
enabled auto-merge (squash)
September 17, 2026 13:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The managed integration lanes were failing (the managed workspace IP-blocked CI runners) and thin. This makes them work and adds end-to-end coverage that drives the agent TUI.
What
ug configureapplies it to every enabled agent, so a lane with only one agent failed configuring the other.managedtests against ca-central's published config: per-agent apply and an idempotent re-configure.managed_fixtureTUI suites: inject the admin CodingAgentConfig via the built-inUCODE_MANAGED_CONFIG_STUBhook (only the config INPUT is stubbed; auth, the config writers, and the agent binary stay real), then launch the real agent and drive the TUI, asserting what the agent presents, not the generated config files (asserting the files is unit tests' job). Shared builders + stub injection live intests/integration/utils/managed.py; each CUJ area is its own suite (_models,_mcp), so skills and spend_tiers drop in as new suites reusing it.tests/AGENTS.md/tests/CLAUDE.md. Lanes stay non-blocking until proven stable.CUJs covered
ug configureis stable for both agents/modellists the injected models, including one the live workspace does not publish, and omits a live-only model/model/mcplists the injected managed MCP server/mcpBugs this surfaced (repro in the handoff notes, for filing)
default_sonnet_modelignored; configure wrote a model outside the allow-list).ug configureapply path does not runvalidate_manifest(client applies malformed configs unvalidated).configureunionsavailable_tools(de-enabling an agent may not remove a previously-configured one).UCODE_MANAGED_CONFIG_STUBis ungated in prod (admin-policy override risk).Follow-ups
utils/managed.py./mcpassertion to a picker-specific pattern (needs the rendered screen from evidence).This pull request and its description were written by Isaac.