test(e2e): validate operator deployability with an in-cluster federation e2e - #179
Open
scotwells wants to merge 23 commits into
Open
test(e2e): validate operator deployability with an in-cluster federation e2e#179scotwells wants to merge 23 commits into
scotwells wants to merge 23 commits into
Conversation
Restore the multi-cluster e2e building blocks from the archived local-run branch so the federation layer can be exercised end to end again: - hack/e2e/kind-control-plane.yaml exposes the Karmada API server on a host NodePort for developer kubectl and Chainsaw access. - hack/e2e/make-internal-kubeconfig.sh and patch-cluster-secret.sh rewrite member-cluster kubeconfigs (and the Cluster apiEndpoint) to the Docker-bridge IP so the in-Docker Karmada controller can reach the POP cell API servers. - test/e2e/chainsaw-config.yaml declares the downstream/pop-dfw/pop-ord clusters the suites reference. These are environment scaffolding only. Unlike the retired harness, the operators are deployed in-cluster via the real kustomize overlays rather than run on the host; that path lands in the following commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the retired host-run operator harness with an in-cluster deploy path so
e2e exercises the same kustomize overlays, RBAC, and pod wiring that run in
production. The Taskfile now builds a dev image, side-loads it into every Kind
cluster, wires the management operator's access to the Karmada hub, and applies
the real overlays to the control-plane and both POP cells, waiting on rollout.
Deploy layer (test/e2e/deploy/{management,cell}) references the production
overlays verbatim and layers only the deviations a local Kind environment
forces, each annotated inline so drift from production stays visible:
- management: discovery.mode milo -> single (no Milo control plane in Kind);
admission webhook disabled and its cert-manager CSI serving-cert volume
removed (no cert-manager/CSI driver in Kind; the suites create
WorkloadDeployments, never Workloads, so the Workload webhook is never
exercised); dev image with IfNotPresent pull.
- cell: discovery.quotaKubeconfigPath dropped so the operator takes its
documented quota opt-out branch instead of crash-looping on the absent edge
credential; dev image with IfNotPresent pull. FEDERATION_KUBECONFIG stays
empty exactly as in production, so cell controllers run against the local
cluster where Karmada propagates WorkloadDeployments.
Federation access for the management operator mints a Karmada-native
ServiceAccount token bound to the real compute-manager hub ClusterRole
(config/base/downstream-rbac), so the operator authenticates as a non-admin
identity and any missing grant surfaces as a forbidden error rather than being
masked by cluster-admin.
The Karmada bring-up is hardened for busy hosts: a 10m Helm wait that resumes a
timed-out install, a wait for the cluster.karmada.io aggregated API before
joining members, retries around karmadactl join while the member token Secret
populates, and leader election disabled on the single-replica Karmada
controllers to avoid a lease-renewal crashloop under etcd write-latency spikes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make e2e:karmada:install robust to a Karmada control plane whose deployments were scaled to zero (e.g. quiesced between runs on a busy host, or left partial by an earlier failed install). Before waiting for the deployments to become Available, scale any that sit at zero replicas back to one, so an idempotent re-run resumes the environment instead of blocking forever on a deployment that has no desired replicas. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore the eight chainsaw e2e suites that exercise the compute federation and referenced-data delivery paths, bringing them back onto the in-cluster harness branch: - full-federation, workload-deployment-federation, instance-projection, instance-writeback, propagation-policy-lifecycle, deletion-cascade (from split/federation-e2e) - referenced-data-mounts, referenced-data-delete-cascade (from archive/e2e-local-deferred) Restored verbatim; adaptation to the in-cluster deploy path follows in a separate commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Delete seven loose assert-*.yaml files that no chainsaw-test.yaml referenced; two still asserted the pre-merge PropagationPolicy name workload-deployments-dfw (the federator now emits city-<cityCode>). The suites assert these resources inline, so nothing is lost. - Enable featureFlags.enableReferencedDataGate on the cell deploy layer. Its only consumer is the cell WorkloadDeploymentReconciler, which stamps the ReferencedData scheduling gate; without it the referenced-data suites observe an Instance that is never gated. The management layer sets the same flag for parity (inert there — that overlay runs management controllers only). - Document the upstream-cluster-name label value (cluster-single). Both operators run discovery.mode=single with the default cluster name single; the management federator stamps EncodeClusterName(single)=cluster-single on the hub namespace and the write-back/projection copy it, so the existing assertions hold. Corrected the instance-projection and instance-writeback descriptions, which named a stale provider symbol and an incorrect cluster-<cell> example. - Refresh referenced-data prose (suite descriptions + README) that referred to the retired host-run operator-start tasks; the feature flag now ships through the deploy layer under task e2e:up. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both POP cells now run operators, but every restored suite exercises only the dfw cell. Add full-federation-ord: an ord-placed WorkloadDeployment must produce PropagationPolicy city-ord, propagate to pop-ord, and create its Instance there, then write back and project like the dfw path. This is the coverage that catches city-code routing regressions and a mis-registered second cell — the dfw suites would stay green through both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Server-side apply already validates on the API server, so the client-side
OpenAPI schema download is redundant — and on a resource-constrained API server
that download times out ("failed to download openapi"), failing the apply. Pass
--validate=false (and --server-side where it was missing) on the CRD, hub-RBAC,
federation-component, and operator applies so bring-up survives a slow API
server.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bake the control-plane stability fixes into cluster creation so bring-up no longer needs manual runtime patching on a resource-constrained host, where a base Kind cluster co-located with the full Karmada control plane otherwise crash-loops on etcd write-latency spikes. - kind configs (control-plane + new kind-pop for the cells) disable etcd fsync and turn off leader election on the single-instance kube-scheduler and kube-controller-manager via kubeadmConfigPatches. kind renders its kubeadm config as v1beta3, so extraArgs use the map form. - the Karmada install disables fsync on the Karmada control plane's own etcd StatefulSet, alongside the leader-election disable already applied to its controllers. Data durability does not matter for these throwaway e2e clusters, and fsync contention between the co-located etcds was the root cause of the latency spikes. Measured effect: hub writes drop from multi-second timeouts to ~170ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The management WorkloadReconciler watches networking.datumapis.com Networks, but
the compute ClusterRole only granted locations, networkcontexts, and subnets —
so the Network informer was denied ("networks ... is forbidden") and the manager
never reconciled WorkloadDeployments cleanly. Add networks to the read-only
networking rule. Surfaced by the in-cluster federation e2e: with this grant the
manager runs with zero RBAC denials and federates WorkloadDeployments to the
hub.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… policy) Two corrections to the e2e management overlay found by deploying it for real: - The webhook-server-tls volume is now backed by an emptyDir rather than deleting the volume + mount. A strategic-merge "$patch: delete" on a single volumeMount dropped the container's entire volumeMounts list, which unmounted the federation kubeconfig — the manager then crash-looped on a missing downstream-kubeconfig.yaml. Overriding only the volume source keeps every mount intact; the webhook server is disabled, so an empty cert dir is fine. - Drop the ResourceMetricsPolicy: its CRD (resourcemetrics.miloapis.com) is owned by a separate operator absent from Kind, so the apply failed with "no matches for kind ResourceMetricsPolicy". It is orthogonal to the federation path under test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…der load The chainsaw suites put sustained load on the co-hosted Karmada control plane. Under that load a transient etcd/DNS blip made the API server's default health probes fail and the kubelet SIGKILLed it (exit 137), cascading into a crashloop that failed every suite. Relax the karmada-apiserver liveness/readiness probes so it rides out blips instead of being killed, and widen the chainsaw timeouts (assert 120s, delete 120s) so Karmada propagation and finalizer-driven cascade deletes have room on the constrained node. Both are constrained-host/CI tuning, not production changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the placeholder E2E workflow (a single throwaway Kind cluster running a commented-out make target) with one that stands up the full harness the Taskfile builds and runs the Chainsaw suites against it: three Kind clusters (management/control-plane hosting Karmada, plus the dfw and ord POP cells), the real production kustomize overlays, and the real hub RBAC. This validates the operators as deployed pods authenticating to Karmada as a non-admin identity rather than as an in-process test binary. The job splits provisioning, deploy, and test into separate steps so a failure lands on the phase that broke, and always collects per-cluster operator logs, Karmada component logs, events, and a full kind log export as an artifact when any step fails. Runs on the free ubuntu-latest runner: the harness is engineered for a constrained host (single-replica Karmada, leader election disabled, generous component waits), so a larger paid runner is not needed unless real runs show resource pressure. Triggers only on pull_request and pushes to main rather than every branch push, given the job's cost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Running all suites concurrently floods the single federation manager: many WorkloadDeployments federate at once and individual suites' downstream assertions time out with "resource not found" even though the federation path itself works (the first suite to reach the manager passes). Pin --parallel 1 so the federation path stays uncontended; a faster local run can still override with `task e2e:test -- --parallel N`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first CI run failed workload-deployment-federation on the city-dfw PropagationPolicy assertion with "lengths of slices don't match": the federator emits three resourceSelectors (WorkloadDeployment plus the always-on ConfigMap and Secret referenced-data selectors that co-propagate companions), but the suite asserted only the WorkloadDeployment. Chainsaw matches list length exactly, so assert all three. Apply the same correction to full-federation-ord's city-ord assertion, which carried the same stale single-selector expectation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ired labels Both suites synthesize objects that the real cell path would stamp, and the first CI run showed the operators correctly rejecting the incomplete fixtures: - instance-projection: the projector logged "missing the workload-deployment-name label; cannot resolve its WorkloadDeployment" and never created the projection. The injected hub Instance now carries workload-deployment-name, and upstream-namespace points at the project namespace (where the WD lives and the projection lands), not the hub ns-<uid>. Also restores the owner-reference assertion the projector sets. - instance-writeback: writeBackToUpstream derives identity from the hub namespace labels and requires the full linking-label set on the Instance (instance_controller.go). Seed both so the write-back fires instead of erroring on missing identity. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
referenced.CompanionName returns the source object's own name when it is a valid DNS subdomain within the length budget (the kind argument is ignored), so the companions materialize as app-config and app-secret, not configmap.app-config and secret.app-secret. The first CI run confirmed this: the referenced-data-delete-cascade suite (source names gc-test-config / gc-test-secret) found its un-prefixed companions, while referenced-data-mounts timed out looking for the prefixed names. Correct the assertions and the naming-convention note in the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Right after the Karmada install, applies made over the API server's host
NodePort can hit a transient connection drop ("EOF" / API discovery failure)
while the control plane settles, which failed provisioning (karmada:configure)
on an otherwise-healthy environment. Retry the federation-component apply and
the hub-RBAC apply a few times so a momentary blip no longer breaks the run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…#144 Scenario 2 asserted a level-triggered hub-side companion GC reclaiming a stranded companion, but that reconciler is not in this branch — it ships in PR #144 (refdata-hub-gc), which was still unmerged when the branch was cut, so the scenario could never pass. Keep the happy-path cascade (Scenario 1), which exercises the delete path that IS implemented, and restore the backstop scenario once #144 lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The serial diagnostic run showed two assertions that need more than 30s on a constrained CI runner (the object does appear, just later): - full-federation assert-instance-on-pop-dfw waits on the whole project WD → federate → Karmada propagate → cell reconcile → Instance chain. - instance-projection wait-for-downstream-namespace waits on federation of a freshly-created namespace. Raise both to 120s. This is a timing allowance, not a masked stall. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
full-federation-ord, instance-writeback, and referenced-data-delete-cascade asserted federated/companion resources with 30-60s per-step timeouts, too tight for the federate->propagate chain on a constrained CI runner (the same class of delay already given 120s in full-federation and instance-projection). Bump the failing steps to 120s. Noted as provably-timing: the resources appear, just after the old deadline. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The federator resolves WorkloadDeployment source namespaces through an informer, which needs watch in addition to get/list. With watch forbidden, the cache only refreshes on periodic re-lists whose backoff grows as the denied watch keeps retrying, so namespaces created after manager startup stay invisible for minutes and their WorkloadDeployments never federate. Second production RBAC gap surfaced by running the operators in-cluster under their real ServiceAccount (after the networks grant).
The federated Instance settles Ready=False, not Unknown: it is blocked before scheduling completes (a scheduling gate, or no matching Location in the e2e environment), which is a definite not-ready state rather than an indeterminate one. The old Unknown expectation predates the scheduling-gate semantics. Assert status only — the exact blocking reason is environment-dependent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The e2e cells ran with FEDERATION_KUBECONFIG empty, which disabled the cell InstanceReconciler's write-back to the hub — so cell-created Instances never reached the Karmada hub or the management InstanceProjector, failing instance-writeback and the write-back/projection steps of full-federation and full-federation-ord. This was a harness bug, not production behavior. Verified against datum-cloud/ infra: every cell that runs compute gets a hub credential via apps/compute-system/edge/manager-patch.yaml (FEDERATION_KUBECONFIG + a mounted client-cert kubeconfig to the Karmada hub). The cell overlay ships the value empty on purpose because infra, not the overlay, supplies it per cell. Mirror that here: federation:setup mints the same compute-manager Karmada SA token and publishes it on both cells as compute-cell-federation-kubeconfig, reaching the kind hub over the control-plane node's docker-bridge IP + NodePort (cell pods live in other kind clusters, so the in-cluster Service the management pod uses is unreachable). federation_patch.yaml mounts it the way infra does. The corrected overlay comment now states what production actually does. Only the plain write-back client is built for cells, so this does not gate readiness. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
scotwells
marked this pull request as ready for review
July 10, 2026 03:39
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Our e2e until now exercised the operators as an in-process test binary, which never touched the way they actually ship: as deployed Instances authenticating to Karmada through real RBAC, wired by the real kustomize overlays. This PR makes e2e validate deployability — RBAC, manifests, image, service-account identity, leader election — not just control logic. That difference is not academic: on its first real runs this harness caught three genuine gaps invisible to in-process tests (two production RBAC grants and a missing cell hub credential — see Verification status). That class of gap is exactly what this exists to catch.
What's in the PR
Taskfile.yaml,hack/e2e/): stands up three Kind clusters — a management/control-plane cluster hosting Karmada plus thedfwandordPOP cells — installs Karmada, registers the cells over the Docker-bridge network, and installs the compute/NSO/quota CRDs. Idempotent and re-runnable.test/e2e/deploy/{management,cell}): references the real production overlays (config/overlays/{management-plane,cell}) and layers only the deviations a local/CI Kind environment forces, each annotated inline with why (discoverymilo→single, webhook + cert-manager CSI not present, cell quota credential absent, dev image). Federation identity is a Karmada-native ServiceAccount token bound to the realcompute-managerhub ClusterRole, so the operators authenticate as a non-admin identity and any missing grant surfaces as a forbidden error. The cell operators also get a hub credential mirroring how infra wires real cells (see Verification status), so the Instance write-back path is exercised.unsafe-no-fsyncand leader-election disabled on the single-instance infrastructure control-plane components (base Kind + Karmada), relaxed Karmada API server probes, retried Karmada-NodePort applies, serial suite execution, and widened Chainsaw timeouts. These are throwaway-cluster/CI tuning only, never production, and the compute operators keep leader election on so it stays exercised..github/workflows/test-e2e.yml): runs the whole harness on GitHub Actions in phased provision/deploy/test steps, and on failure collects per-cluster operator logs, Karmada component logs, events, and a full Kind log export as an artifact.Verification status
All 9 Chainsaw suites green on CI (run 29066748470): provision → in-cluster deploy of the real overlays to all three clusters → the full 9-suite federation run all pass. Management + both cell operators come up Ready with zero RBAC denials; the federation flow is validated end to end (WorkloadDeployment → Karmada hub companion +
city-dfw/city-ordPropagationPolicy → propagation to the POP cell → Instance created → Instance written back to the hub → projected).Three real gaps this harness caught and fixed along the way — the point of #149, all invisible to an in-process test:
networks— the management WorkloadReconciler watchesnetworking.datumapis.com/networks, but thecomputeClusterRole granted only locations/networkcontexts/subnets, so the Network watch was denied and the manager could not reconcile WorkloadDeployments cleanly. Addednetworksget/list/watch.namespaceswatch — the role granted namespacesget,listbut notwatch, so the federator's namespace→UID informer could never establish its watch; its cache refreshed only via backed-off re-LISTs, so namespaces created after startup stayed invisible for minutes and their WorkloadDeployments never federated. Addedwatch. (Post-fix diagnostics sweep: zero compute-manager forbidden across all four planes.)FEDERATION_KUBECONFIGempty, which silently no-ops the cell InstanceReconciler's write-back, so cell-created Instances never reached the hub or the management InstanceProjector (Karmada never propagates Instances, so status aggregation can't surface them — write-back is the sole path). Verified againstdatum-cloud/infra: every cell that runs compute DOES get a hub credential viaapps/compute-system/edge/manager-patch.yaml; the overlay's "empty is production-exact" comment was wrong and is corrected. The harness now mints a cell Karmada credential mirroring infra, exercising the full cell→hub→projector path.Follow-up (not blocking)
delete-cascadeScenario 2 (the stranded-companion GC backstop) is temporarily removed — the hub-side companion GC it asserts ships in PR Reliable cleanup of delivered config and secrets on cells (no orphan leak) #144 (refdata-hub-gc), which is unmerged. Scenario 1 (happy-path cascade) stays green. Restore Scenario 2 once Reliable cleanup of delivered config and secrets on cells (no orphan leak) #144 lands.Closes #149.