Skip to content

Add a Karmada federation topology foundation for service e2e - #32

Draft
scotwells wants to merge 10 commits into
mainfrom
feat/federation-topology
Draft

Add a Karmada federation topology foundation for service e2e#32
scotwells wants to merge 10 commits into
mainfrom
feat/federation-topology

Conversation

@scotwells

@scotwells scotwells commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Why

Datum services increasingly run as multi-cluster federations — a Karmada hub that places workloads onto member cells. Until now, every service that wanted to e2e-test that shape had to hand-roll the whole topology: create the kind clusters, install and tune Karmada, join and label the members, and stitch together the cross-cluster kubeconfigs. That harness lived inside one service's repo, so the next service copied it.

This PR moves that capability into the shared test-infra so any Datum service e2e gets a Karmada hub + N members in one task (task federation-up) or one composite action (federation-bootstrap). Compute is the first consumer (datum-cloud/compute#181); NSO-style multi-cluster labs are the natural next one. It sits alongside the existing single-cluster cluster-up without changing it.

What you get

task federation-up FEDERATION_MEMBERS="pop-dfw=dfw pop-ord=ord"
# → hub cluster running Karmada + two member cells, joined and labeled,
#   with a ready-to-consume kubeconfig bundle.
task federation-status
task federation-down FEDERATION_MEMBERS="pop-dfw=dfw pop-ord=ord"
  • Topology as vars, not a fileFEDERATION_HUB_CLUSTER, FEDERATION_MEMBERS="<name>[=<label>] ...", KARMADA_API_NODEPORT, etc. Fits Task's var/override style and composite-action string inputs without adding a YAML-parser dependency.
  • A stable kubeconfig contract under kubeconfigs/federation/: host-side configs for the hub and each member, docker-bridge-IP variants for in-cluster reachers, karmada.yaml (localhost:NodePort) + karmada-internal.yaml (hub-docker-ip:NodePort), and a shell-sourceable federation.env.
  • A composite action (federation-bootstrap) that stands the topology up in CI and emits the kubeconfig paths as step outputs.

Design notes

  • Imperative Karmada install, no Flux on federation clusters. These are throwaway, throughput-tuned kind clusters; Karmada goes on via Helm (matching the kyverno-style direct install), and the hard-won runtime tuning stays as post-install patches (leader-election off, etcd --unsafe-no-fsync, relaxed apiserver probes) because the chart does not expose those knobs as values. The create-kind task was refactored into a thin wrapper over a new internal create-kind-raw so the federation path can reuse cluster creation without Flux; the public single-cluster behavior is unchanged.
  • kubectl-based kubeconfig rewrite (drops the previous PyYAML dependency): kubectl config unset …certificate-authority-data then set-cluster --server … --insecure-skip-tls-verify.
  • docker-IP derivation lives only in hack/federation/*.sh — a docker --format Go-template collides with Task's own {{ }} templating.
  • Requires Task ≥ 3.44 for the for:-over-task: member loops. Existing single-cluster consumers are unaffected; CI pins Task 3.x.

Verification

  • Dogfood CI (.github/workflows/federation-test.yml, this PR): hub + two members, asserts both reach Ready with the expected city-code labels, and round-trips a ConfigMap through a PropagationPolicy that targets one member only — the assertion that proves the patched member secret + apiEndpoint actually carry traffic. bootstrap-test.yml continues to run as the single-cluster regression gate.
  • Cross-repo validation (test(e2e): run compute e2e on the shared test-infra federation foundation compute#181): compute's full e2e harness runs on this foundation — foundation-provisioned hub + two POP cells, real operator deploy, and all nine Chainsaw federation suites green on CI while pinned to this branch: compute E2E Tests run 29112915347.
  • Local smoke (hub + one member, with a non-default KARMADA_API_NODEPORT to exercise override propagation): converged in ~5 min; the override flowed through the Service nodePort, karmada.yaml, karmada-internal.yaml, and federation.env consistently; the member reached Ready with a docker-IP apiEndpoint; a ConfigMap propagated to and was withdrawn from the member in a few seconds; federation-down cleaned up completely. The refactored create-kind was confirmed to still create a cluster and install Flux.

Rollout / pinning chain

  1. This branch (feat/federation-topology) merges first.
  2. Tag v0.7.0 (latest is v0.6.4; new feature ⇒ minor bump).
  3. Compute (test(e2e): run compute e2e on the shared test-infra federation foundation compute#181) then re-pins its TEST_INFRA_REF from this branch to v0.7.0, its e2e CI re-runs green, and the compute PR becomes mergeable.

Also folds in two small doc/drift fixes touched along the way: the Flux installer comment now matches the pinned v2.7.5, and the cluster-up gateway echo now prints the correct 30443.

scotwells added 10 commits July 10, 2026 12:28
Federation installs Karmada with Helm, so add an idempotent install_helm()
alongside the existing kind/kubectl/kustomize/flux installers (get-helm-3 on
darwin/linux, choco/winget on windows).
Hub and member kind configs for throughput-tuned throwaway clusters (etcd
unsafe-no-fsync, leader-election off) that co-locate a Karmada control plane on
a busy host. The hub exposes the Karmada apiserver NodePort, rendered from
KARMADA_API_NODEPORT via envsubst. Chart values pin apiServer.serviceType to
NodePort; all runtime tuning stays as post-install patches in the Taskfile.
make-internal-kubeconfig.sh rewrites a kubeconfig's server to a kind node's
docker-bridge IP so the in-Docker Karmada controller can reach it; an optional
port arg lets the same script build both member internals (6443) and the hub's
NodePort variant. patch-cluster-secret.sh points a joined member's stored secret
and spec.apiEndpoint at that docker IP (the member never goes Ready otherwise).
Both use kubectl config for the rewrite, dropping the PyYAML dependency.
federation-up stands up a Karmada hub plus N member kind clusters, joins and
labels them, and writes a kubeconfig bundle any Datum service e2e can consume;
federation-down/-status/-load-image round out the lifecycle. Members are declared
via FEDERATION_MEMBERS='name[=label] ...'. Karmada is installed imperatively with
Helm (federation clusters run no Flux) and tuned with the proven post-install
patches (leader-election off, etcd fsync off, relaxed apiserver probes) so it
settles on a busy host. create-kind is refactored into a thin wrapper over a new
internal create-kind-raw (config + kubeconfig parameterized, no Flux) that the
hub/member creation reuses; its single-cluster behavior is unchanged. Requires
Task >= 3.44 for the member for-loops. Also corrects the stale gateway port in
the cluster-up echoes (8443 -> 30443).
Stands up a federation topology and emits the kubeconfig bundle paths as step
outputs so any consumer workflow gets hub + members in one action. Unlike
kind-bootstrap it deliberately does not set KUBECONFIG in the environment (no
single sensible default among the hub/karmada/member configs). The inner
test-infra checkout follows the action ref so the Taskfile matches the pin.
Exercises federation-bootstrap on a hub plus two members, asserts both members
reach Ready with the expected city-code labels, and round-trips a ConfigMap via a
PropagationPolicy targeting one member only -- the assertion that proves the
patched member secret and apiEndpoint actually carry traffic. Collects kind and
Karmada diagnostics on failure and always tears down.
Add a Federation topology section covering the member-spec syntax, variables,
kubeconfig bundle, and the external-consumer include example, and note the
Task >= 3.44 requirement. Also correct the Flux installer comment to match the
pinned v2.7.5.
An out-of-range KARMADA_API_NODEPORT otherwise fails deep inside the helm install
with a cryptic Service validation error; precondition it in federation-up so the
footgun surfaces immediately with a clear message.
Task treats a CLI-provided empty variable as authoritative and skips the
Taskfile's `| default`, so passing KARMADA_VERSION="" (the action's empty input
default) zeroed out both the chart version and the derived karmadactl version,
producing a bad download URL. Only pass the var when the input is set.
A remote Taskfile include shares one root variable namespace, so the foundation's
root vars clobber a consumer's same-named vars (compute's own LOCALBIN rendered as
a nil path until renamed). Document the reserved names and that passing them as
include vars is the intended way to steer the foundation.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant