E2E gating test red: turn trafficprotectionpolicy-enforce-serves-traffic green

[ecv]

Summary

The E2E gating test chainsaw/trafficprotectionpolicy-enforce-serves-traffic (added in #245) went red on main — failing on every open PR, so the whole E2E Tests workflow was red across the board. This ticket tracks turning it green.

Update 2026-07-13 (post-#252). The E2E Tests run on the #252 merge commit (411d117c, run 29267425732) shows the gating test is still red, and #252 is not the unblock. The e2e runs in-cluster from the merged code with a pinned coraza image, so that run already contained #252 — yet:

• trafficprotectionpolicy-paranoia-validation → PASS (the new fix: reject inverted TrafficProtectionPolicy paranoia levels at admission #251/fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 inverted-paranoia guard works)
• trafficprotectionpolicy-enforce-serves-traffic → FAIL — Gateway waf-gw never programs the Enforce WAF listener (status: {}, Programmed=Unknown), 30s assert timeout.

The remaining blocker is #242 (core Enforce filter defect), now marked as a blocked_by dependency. #252 still matters for prod (stops tenant inverted-paranoia policies 500ing), but it is orthogonal to greening this e2e.

Failure signature

The downstream Gateway never reaches Programmed=True; Envoy rejects the Coraza WAF Enforce listener:

status.(conditions[?type == 'Programmed'])[0].status: Invalid value: "Unknown": Expected value: "True"
--- expected
-  - status: "True"
+status: {}

Two independent TPP-Enforce paths produced the same "500 / broken listener" family that reddens this test:

1. Coraza filter defect (TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242) — the WAF Enforce filter breaks the listener, so the Gateway never programs (the current, remaining blocker).
2. Inverted paranoia (TrafficProtectionPolicy: blocking paranoia level > detection triggers CRS 901500 → 100% of traffic gets HTTP 500 #250) — blocking paranoia level > detection level trips CRS rule 901500, fail-closed 500 at phase 1. Fixed and now guarded (below).

What's landed

NSO:

• fix: make error-page body Envoy-safe via operator allowlist + startup validation (#243) #244 — %-escape the branded error page (operator allowlist + startup validation) — fixes extension-server: branded error page injected as Envoy SubstitutionFormatString → unescaped % rejects the listener (breaks TPP Enforce) #243 (merged 2026-07-10)
• fix: clone RouteBaseDirectives before per-policy append (#246) #249 — clone RouteBaseDirectives before per-policy append — fixes Latent: getCorazaDirectivesForTrafficProtectionPolicy aliases the shared RouteBaseDirectives slice (corrupts directives across policies if EPP emission is enabled) #246 (merged 2026-07-10)
• fix: reject inverted TrafficProtectionPolicy paranoia levels at admission #251 — reject inverted paranoia at admission (CEL, fail-closed) — fixes TrafficProtectionPolicy: blocking paranoia level > detection triggers CRS 901500 → 100% of traffic gets HTTP 500 #250 (merged 2026-07-11)
• fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252 — flag inverted paranoia via status condition (Accepted=False/Invalid, skip attachment) — merged 2026-07-13 16:41. Verified green by the new trafficprotectionpolicy-paranoia-validation e2e.

Edge coraza-waf (infra repo):

• datum-cloud/infra#3319 — pin coraza-waf to v1.3.0-alpha8 — MERGED (e2e + staging now on v1.3.0-alpha10)
• datum-cloud/infra#3324 — response-phase SendLocalReply(empty body) dropped by Envoy 1.37.x — CLOSED
• datum-cloud/infra#3329 — nil-pointer panic in EncodeData (shadowed err) on response-phase block — CLOSED

e2e config:

• coraza-waf pin in config/dev/downstream_resources/downstream-gateway.yaml:66 bumped v1.1.1-dev2 → v1.3.0-alpha10.

Remaining to turn green

Blocked by #242. Even with all of the above landed, run 411d117c proves the Enforce WAF listener still fails to program (status: {}). The coraza v1.3.0-alpha10 runtime fixes (#3324/#3329) addressed response-phase 500s but not this listener-programming failure.

1. Resolve TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242 — the Gateway must reach Programmed=True with a WAF policy in Enforce.
2. Re-run E2E Tests on main and confirm trafficprotectionpolicy-enforce-serves-traffic passes.

Acceptance criteria

• E2E Tests workflow green on main
• chainsaw/trafficprotectionpolicy-enforce-serves-traffic passes with the WAF in Enforce mode (Programmed=True on the downstream Gateway)
• coraza-waf image pin in e2e config no longer v1.1.1-dev2 (now v1.3.0-alpha10)
• inverted-paranoia guard verified in e2e (trafficprotectionpolicy-paranoia-validation PASS)

Related

• Blocker: TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242 (Enforce filter defect)
• Root bugs: TrafficProtectionPolicy Enforce mode: 500 with truncated/empty body on all traffic #242 (filter), TrafficProtectionPolicy: blocking paranoia level > detection triggers CRS 901500 → 100% of traffic gets HTTP 500 #250 (inverted paranoia, CLOSED)
• NSO fixes: extension-server: branded error page injected as Envoy SubstitutionFormatString → unescaped % rejects the listener (breaks TPP Enforce) #243, fix: make error-page body Envoy-safe via operator allowlist + startup validation (#243) #244, Latent: getCorazaDirectivesForTrafficProtectionPolicy aliases the shared RouteBaseDirectives slice (corrupts directives across policies if EPP emission is enabled) #246, fix: clone RouteBaseDirectives before per-policy append (#246) #249, fix: reject inverted TrafficProtectionPolicy paranoia levels at admission #251, fix: flag inverted TrafficProtectionPolicy paranoia levels via status condition #252
• Test that gates: test: e2e for TrafficProtectionPolicy Enforce serving traffic (#243) #245
• Parity / coverage follow-ups: datum-cloud/infra#3341, datum-cloud/infra#3408, datum-cloud/infra#3402

[ecv]

This is the companion to #242 (TPP Enforce / WAF data-plane path)

Clarifying what actually fails, since the E2E Tests red on main was briefly misattributed.

• gateway-invalid-certificate passes on both recent main commits (b7f5d29 and 411d117 / v0.23.12). The good hostname never returned 200 / hostname-less listener was admitted strings in --log-failed are echoed chainsaw script bodies, not fired asserts — the Gateway: hostname-less tenant listeners rejected with misleading 'must be unique' (default-listener injection collision) #219 guard holds.
• The deterministic failure is this test — trafficprotectionpolicy-enforce-serves-traffic — at step "Route through the WAF gateway with an Enforce policy", ASSERT ERROR on gateway.networking.k8s.io/v1/Gateway @ <ns>/waf-gw.
• fix: pin e2e coraza-waf to a pullable fixed image #264's coraza-waf image pin merged and rolled out but did not turn this green.

Because this exercises the TPP-Enforce → WAF gateway data-plane path, it is the runtime behavioral coverage that #242 (Box 4) needs, and its failure is the same class of problem. Treating #263 and #242 as companions.

Root-cause of the waf-gw assert (Gateway not reaching the asserted state) is under active investigation — will post the expected-vs-actual condition diff and a fix here.

Cross-ref: #242, #264.

[ecv]

Root cause: CI never installs the downstream WAF data plane (and #264 was a no-op for CI)

Failing assert

Step "Route through the WAF gateway with an Enforce policy" (test/e2e/trafficprotectionpolicy-enforce/chainsaw-test.yaml:133-134), 30s AssertTimeout on gateway.networking.k8s.io/v1/Gateway @ <ns>/waf-gw:

status.(conditions[?type == 'Programmed'])[0].status: Invalid value: "Unknown": Expected value: "True"
--- expected              +++ actual
-status:                  +status: {}
-  Programmed: "True"

waf-gw requests gatewayClassName: datum-downstream-gateway (chainsaw-test.yaml:83) and never reaches Programmed=True — it ends with empty status.

Why

The CI e2e workflow never stands up the downstream WAF data plane on the infra cluster, so nothing programs waf-gw.

• Makefile:156 prepare-infra-cluster: cert-manager envoy-gateway external-dns downstream-crds — installs only stock Envoy Gateway (default controllerName) + 3 NSO CRDs.
• It does NOT apply any of: the datum-downstream-gateway GatewayClass + coraza EnvoyProxy (config/dev/downstream_resources/downstream-gateway.yaml), the extension-server (config/extension-server/), or the envoy-gateway-downstream EG instance carrying the extensionManager + Coraza filter (config/tools/envoy-gateway-downstream/). The test's own header comment says exactly this.

Why #264 didn't move it

#264 changed only the coraza initContainer image in config/dev/downstream_resources/downstream-gateway.yaml:64. That file is consumed solely by config/dev via make install (local prepare-dev/Taskfile.dev) — it is never built or applied by the CI e2e workflow (.github/workflows/test-e2e.yml runs prepare-e2e = config/e2e + prepare-infra-cluster). So the pin landed in a manifest CI doesn't apply → zero effect. The image was never the blocker; the missing data plane is.

Secondary mismatch

Even once a data plane is installed, the class name won't match: config/e2e/config.yaml:10 sets downstreamGatewayClassName: datum-downstream-gateway-e2e, but the test hardcodes datum-downstream-gateway (the dev-only name, config/dev/config.yaml:15).

Fix (to actually turn this green)

Wire the WAF data plane into the CI infra cluster — extend prepare-infra-cluster (or a new dep target) to apply on kind-nso-infra:

1. config/tools/envoy-gateway-downstream (EG instance, controllerName gateway.envoyproxy.io/datum-downstream-gateway, extensionManager + Coraza).
2. config/extension-server (resolve its placeholder-ca / client-TLS cert-manager wiring — noted at config/tools/envoy-gateway-downstream/kustomization.yaml:48-50).
3. config/dev/downstream_resources/downstream-gateway.yaml (the GatewayClass + coraza EnvoyProxy).
4. Reconcile the class name so the test's datum-downstream-gateway matches what the data plane registers (drop the -e2e suffix in config/e2e/config.yaml:10, or point the test at the provisioned class).

Alternative — gate the test behind a skip/precondition if the full data plane can't run in kind CI — but that contradicts this issue's goal, so option 1 is preferred.

Reproduces identically on current main HEAD 1b06e9c (prepare-infra-cluster unchanged). This also unblocks #242 Box 4 (runtime TPP-Enforce behavioral coverage).

[ecv]

Green — waf-gw programs, trafficprotectionpolicy-enforce-serves-traffic passes

E2E Tests is now passing on the PR #268 branch across two consecutive commits:

• 1722a30 — run 29279171688 ✅
• 09610bf — run 29279595137 ✅

waf-gw reaches Programmed=True and the enforce serves-traffic assert passes. PR #268 is marked ready for review.

What actually blocked it (two prod-divergences, not the image pin)

1. The datum-downstream-gateway namespace was created unlabelled, so the downstream Envoy Gateway (which watches meta.datumapis.com/upstream-cluster-name-labelled namespaces) never saw the EnvoyProxy and left the GatewayClass unaccepted → empty status. Prod labels this namespace (0-datum-internal-anchor).
2. The coraza init container copied /app/coraza-waf.so, but coraza-waf:v1.3.0-alpha10 ships the library at /coraza-waf.so → cp exit 1 → init CrashLoopBackOff → Envoy pod Pending → Programmed=False reason NoResources. Prod copies /coraza-waf.so.

Plus: downstream EG chart pinned to v1.7.4 (prod), extensionManager.maxMessageSize 256Mi (prod), a 3m Programmed assert timeout (cold image-pull), and failure-dump diagnostics. #264's coraza image pin was both a CI no-op and pointed at the stale .so path; both .so-path copies (config/e2e-downstream and the orphaned config/dev) are now corrected.

No production WAF defect surfaced — every fix was to make the e2e environment match prod.