Skip to content

test(e2e): run compute e2e on the shared test-infra federation foundation - #181

Draft
scotwells wants to merge 5 commits into
feat/e2e-in-clusterfrom
feat/e2e-test-infra-federation
Draft

test(e2e): run compute e2e on the shared test-infra federation foundation#181
scotwells wants to merge 5 commits into
feat/e2e-in-clusterfrom
feat/e2e-test-infra-federation

Conversation

@scotwells

@scotwells scotwells commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What this does

Compute's end-to-end tests now run on the shared test-infra federation foundation (datum-cloud/test-infra#32) instead of a bespoke, compute-only Kind + Karmada harness. Standing up three Kind clusters, installing and tuning Karmada, and joining the POP cells is no longer compute's code to maintain — it's a reusable foundation any Datum service can consume the same way. Compute keeps only what is genuinely service-specific: its Karmada federation config, CRD installs, the management/cell federation credentials, the operator image build, and the deploy overlays. Local task e2e:up on a laptop and the CI job run the identical path, because both pull the foundation in as a remote Taskfile include.

This builds directly on the in-cluster e2e harness from #179 (the deployability work for #149): the operators, overlays, federation credentials, and all nine Chainsaw suites are unchanged — only the cluster/Karmada provisioning underneath them moves to the shared foundation.

Net result: ~620 lines of Karmada/Kind plumbing and four hack/e2e scripts deleted from this repo, the same nine Chainsaw suites still exercising the full federated topology.

What moved where

  • Taskfile.yaml — adds a remote Taskfile include of datum-cloud/test-infra (pinned via TEST_INFRA_REF) and delegates cluster/Karmada lifecycle to it. e2e:env:up becomes e2e:toolsinfra:federation-upe2e:karmada:configuree2e:crds:install; e2e:downinfra:federation-down; e2e:image:loadinfra:federation-load-image. The cell-credential step reads the hub's docker-reachable address from the foundation-provided karmada-internal.yaml rather than re-deriving the hub container IP. Deleted: the karmadactl/helm-repo tool installs, cluster-create/kubeconfig-export, and the Karmada install/tune/join tasks.
  • hack/e2e/ — deleted (kind-control-plane.yaml, kind-pop.yaml, make-internal-kubeconfig.sh, patch-cluster-secret.sh); all absorbed by the foundation.
  • Federation kubeconfigs — now written by the foundation under .test-infra/kubeconfigs/federation/ as compute-control-plane.yaml, karmada.yaml, compute-pop-dfw.yaml, compute-pop-ord.yaml (replacing tmp/e2e/kubeconfigs/* and the downstream.yaml copy). The chainsaw config's named clusters and the referenced-data README are repointed accordingly. The hardcoded --kubeconfig paths inside the Chainsaw suite steps are a path-substitution-only change isolated in its own commit (test(e2e): repoint suite kubeconfig paths…) — no assertion, workload spec, or step logic changes; verify with git diff --word-diff. The relative depth (../../../ from test/e2e/<suite>/) is unchanged because .test-infra/ sits at the repo root just like tmp/ did.
  • .github/workflows/test-e2e.yml — opts into TASK_X_REMOTE_TASKFILES=1 at job level, runs task --yes …, drops the manual kind install (the foundation's ensure-tools provides it) and the PyYAML shim (no harness Python parses YAML any more). Diagnostics repointed at the foundation kubeconfig directory/filenames.

Deviations inherited from the foundation

These are intentional consequences of moving onto the shared foundation, each validated by the compute e2e CI acceptance gate:

Area Before (compute harness) After (foundation) Why it's safe
Node image kind's default node image pinned kindest/node:v1.32.0 (foundation's K8S_VERSION) reproducibility improvement; exercised by CI
kind version manual install of v0.32.0 foundation ensure-tools installs v0.30.0 (accepts any >= v0.30.0) v0.30.0 supports the v1.32.0 node image
Hub kind config render sed substitution of the NodePort envsubst of ${KARMADA_API_NODEPORT} same rendered config; matches the foundation's existing pattern
Karmada helm install helm repo add then install helm install --repo … avoids mutating the developer's global helm repo list
Kubeconfig rewrite python3 + PyYAML kubectl config (unset-CA-then-set) drops the PyYAML dependency; PP round-trip in the foundation's dogfood CI proves it

Var-naming note: the e2e tool-bin var is E2E_BIN, not the conventional LOCALBIN. A remote Taskfile include's root var clobbers a same-named var in the consumer, and the foundation already defines LOCALBIN — sharing the name rendered compute's value as a nil path. KARMADA_API_NODEPORT is also shared, but it is passed through the include by design and resolves to the same value (local override included, verified 32443→32643).

Verification

Full e2e CI is green on this branch with the foundation pinned to feat/federation-topology: E2E Tests run 29112915347 — foundation-provisioned Kind + Karmada environment, operator image build and deploy, and all nine Chainsaw suites pass. This run doubles as the cross-repo validation for datum-cloud/test-infra#32.

Pinning chain — DO NOT MERGE until re-pinned

TEST_INFRA_REF is currently pinned to the test-infra branch feat/federation-topology so CI can exercise the foundation before it merges. This must be flipped to the tagged foundation release before this PR merges:

  1. Add a Karmada federation topology foundation for service e2e test-infra#32 merges first → tag v0.7.0.
  2. Flip TEST_INFRA_REF to v0.7.0 here (one var moves both the include URL and the foundation's self-clone ref), re-run compute CI green.
  3. Only then is this PR mergeable — and it merges after test(e2e): validate operator deployability with an in-cluster federation e2e #179 (see below).

Stacked on #179

Base branch is feat/e2e-in-cluster (PR #179, which closes #149). This PR is stacked on it and should be reviewed/merged after #179 lands.

Replace compute's bespoke Kind + Karmada standup with a remote Taskfile
include of the shared test-infra federation foundation, so a laptop and CI
run the identical path. The foundation now owns creating the three Kind
clusters, installing and tuning Karmada, and joining the POP cells; compute
keeps only its service-specific layer (Karmada federation config, CRD
installs, the management/cell federation credentials, image build, and the
deploy overlays).

Drop the absorbed plumbing: the karmadactl/helm-repo tool installs, the
cluster-create and kubeconfig-export tasks, the Karmada install/tune/join
tasks, and the four hack/e2e config and kubeconfig-rewrite scripts. Read the
federation kubeconfigs the foundation writes under
.test-infra/kubeconfigs/federation, and source the cell operators' hub
address from the foundation-provided karmada-internal.yaml instead of
re-deriving the hub container IP. Gitignore the foundation self-clone and
Task's remote cache.
Mechanical path substitution only — the shared foundation writes the cluster
kubeconfigs under .test-infra/kubeconfigs/federation with new names, so the
hardcoded --kubeconfig paths inside the Chainsaw suite steps move:

  tmp/e2e/kubeconfigs/control-plane.yaml -> .test-infra/kubeconfigs/federation/compute-control-plane.yaml
  tmp/e2e/kubeconfigs/downstream.yaml    -> .test-infra/kubeconfigs/federation/karmada.yaml
  tmp/e2e/kubeconfigs/pop-dfw.yaml       -> .test-infra/kubeconfigs/federation/compute-pop-dfw.yaml
  tmp/e2e/kubeconfigs/pop-ord.yaml       -> .test-infra/kubeconfigs/federation/compute-pop-ord.yaml

No test assertions, workload specs, or step logic change; verify with
`git diff --word-diff`. The relative depth (../../../ from test/e2e/<suite>/)
is unchanged because .test-infra sits at the repo root just like tmp did.
…onfigs

Update the chainsaw configuration's named clusters (downstream, pop-dfw,
pop-ord) to the foundation-written kubeconfigs under
.test-infra/kubeconfigs/federation, and refresh the referenced-data README to
describe the new layout: the foundation writes the hub kubeconfig as
karmada.yaml and there is no longer a downstream.yaml copy or a
build-kubeconfig task.
Drive the e2e job through the remote federation include: opt into Task's
remote-taskfile support at the job level and add --yes to each task step for
the one-time trust prompt. Drop the manual kind install (the foundation's
ensure-tools provides it) and the PyYAML shim (no harness python parses YAML
any more). Point diagnostics collection at the foundation kubeconfig
directory and cluster filenames.
A remote Taskfile include's root var clobbers a same-named var in the
consumer Taskfile. Compute and the test-infra foundation both defined
LOCALBIN, so compute's {{.ROOT_DIR}}/bin was overridden by the foundation's
REPO_DIR-based definition; REPO_DIR is undefined in the compute scope, so the
value rendered as a literal "%!s(<nil>)/bin" — chainsaw would have installed
to a bogus directory and e2e:test could not have found it. Rename the var to
E2E_BIN, which does not collide. KARMADA_API_NODEPORT also shares a name with
a foundation var, but it is passed through the include by design and resolves
to the same value (override included), so it is unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant