Add a Karmada federation topology foundation for service e2e - #32
Draft
scotwells wants to merge 10 commits into
Draft
Add a Karmada federation topology foundation for service e2e#32scotwells wants to merge 10 commits into
scotwells wants to merge 10 commits into
Conversation
Federation installs Karmada with Helm, so add an idempotent install_helm() alongside the existing kind/kubectl/kustomize/flux installers (get-helm-3 on darwin/linux, choco/winget on windows).
Hub and member kind configs for throughput-tuned throwaway clusters (etcd unsafe-no-fsync, leader-election off) that co-locate a Karmada control plane on a busy host. The hub exposes the Karmada apiserver NodePort, rendered from KARMADA_API_NODEPORT via envsubst. Chart values pin apiServer.serviceType to NodePort; all runtime tuning stays as post-install patches in the Taskfile.
make-internal-kubeconfig.sh rewrites a kubeconfig's server to a kind node's docker-bridge IP so the in-Docker Karmada controller can reach it; an optional port arg lets the same script build both member internals (6443) and the hub's NodePort variant. patch-cluster-secret.sh points a joined member's stored secret and spec.apiEndpoint at that docker IP (the member never goes Ready otherwise). Both use kubectl config for the rewrite, dropping the PyYAML dependency.
federation-up stands up a Karmada hub plus N member kind clusters, joins and labels them, and writes a kubeconfig bundle any Datum service e2e can consume; federation-down/-status/-load-image round out the lifecycle. Members are declared via FEDERATION_MEMBERS='name[=label] ...'. Karmada is installed imperatively with Helm (federation clusters run no Flux) and tuned with the proven post-install patches (leader-election off, etcd fsync off, relaxed apiserver probes) so it settles on a busy host. create-kind is refactored into a thin wrapper over a new internal create-kind-raw (config + kubeconfig parameterized, no Flux) that the hub/member creation reuses; its single-cluster behavior is unchanged. Requires Task >= 3.44 for the member for-loops. Also corrects the stale gateway port in the cluster-up echoes (8443 -> 30443).
Stands up a federation topology and emits the kubeconfig bundle paths as step outputs so any consumer workflow gets hub + members in one action. Unlike kind-bootstrap it deliberately does not set KUBECONFIG in the environment (no single sensible default among the hub/karmada/member configs). The inner test-infra checkout follows the action ref so the Taskfile matches the pin.
Exercises federation-bootstrap on a hub plus two members, asserts both members reach Ready with the expected city-code labels, and round-trips a ConfigMap via a PropagationPolicy targeting one member only -- the assertion that proves the patched member secret and apiEndpoint actually carry traffic. Collects kind and Karmada diagnostics on failure and always tears down.
Add a Federation topology section covering the member-spec syntax, variables, kubeconfig bundle, and the external-consumer include example, and note the Task >= 3.44 requirement. Also correct the Flux installer comment to match the pinned v2.7.5.
An out-of-range KARMADA_API_NODEPORT otherwise fails deep inside the helm install with a cryptic Service validation error; precondition it in federation-up so the footgun surfaces immediately with a clear message.
Task treats a CLI-provided empty variable as authoritative and skips the Taskfile's `| default`, so passing KARMADA_VERSION="" (the action's empty input default) zeroed out both the chart version and the derived karmadactl version, producing a bad download URL. Only pass the var when the input is set.
A remote Taskfile include shares one root variable namespace, so the foundation's root vars clobber a consumer's same-named vars (compute's own LOCALBIN rendered as a nil path until renamed). Document the reserved names and that passing them as include vars is the intended way to steer the foundation.
This was referenced Jul 13, 2026
Draft
test(e2e): validate operator deployability with an in-cluster federation e2e
datum-cloud/compute#179
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Datum services increasingly run as multi-cluster federations — a Karmada hub that places workloads onto member cells. Until now, every service that wanted to e2e-test that shape had to hand-roll the whole topology: create the kind clusters, install and tune Karmada, join and label the members, and stitch together the cross-cluster kubeconfigs. That harness lived inside one service's repo, so the next service copied it.
This PR moves that capability into the shared test-infra so any Datum service e2e gets a Karmada hub + N members in one task (
task federation-up) or one composite action (federation-bootstrap). Compute is the first consumer (datum-cloud/compute#181); NSO-style multi-cluster labs are the natural next one. It sits alongside the existing single-clustercluster-upwithout changing it.What you get
FEDERATION_HUB_CLUSTER,FEDERATION_MEMBERS="<name>[=<label>] ...",KARMADA_API_NODEPORT, etc. Fits Task's var/override style and composite-action string inputs without adding a YAML-parser dependency.kubeconfigs/federation/: host-side configs for the hub and each member, docker-bridge-IP variants for in-cluster reachers,karmada.yaml(localhost:NodePort) +karmada-internal.yaml(hub-docker-ip:NodePort), and a shell-sourceablefederation.env.federation-bootstrap) that stands the topology up in CI and emits the kubeconfig paths as step outputs.Design notes
--unsafe-no-fsync, relaxed apiserver probes) because the chart does not expose those knobs as values. The create-kind task was refactored into a thin wrapper over a new internalcreate-kind-rawso the federation path can reuse cluster creation without Flux; the public single-cluster behavior is unchanged.kubectl config unset …certificate-authority-datathenset-cluster --server … --insecure-skip-tls-verify.hack/federation/*.sh— a docker--formatGo-template collides with Task's own{{ }}templating.for:-over-task:member loops. Existing single-cluster consumers are unaffected; CI pins Task 3.x.Verification
.github/workflows/federation-test.yml, this PR): hub + two members, asserts both reach Ready with the expected city-code labels, and round-trips a ConfigMap through a PropagationPolicy that targets one member only — the assertion that proves the patched member secret + apiEndpoint actually carry traffic.bootstrap-test.ymlcontinues to run as the single-cluster regression gate.KARMADA_API_NODEPORTto exercise override propagation): converged in ~5 min; the override flowed through the Service nodePort,karmada.yaml,karmada-internal.yaml, andfederation.envconsistently; the member reached Ready with a docker-IP apiEndpoint; a ConfigMap propagated to and was withdrawn from the member in a few seconds;federation-downcleaned up completely. The refactoredcreate-kindwas confirmed to still create a cluster and install Flux.Rollout / pinning chain
feat/federation-topology) merges first.TEST_INFRA_REFfrom this branch tov0.7.0, its e2e CI re-runs green, and the compute PR becomes mergeable.Also folds in two small doc/drift fixes touched along the way: the Flux installer comment now matches the pinned v2.7.5, and the
cluster-upgateway echo now prints the correct30443.