feat: add exporter source for local and Docker use

[delgadof]

Adds a self-contained OpenShell Event Exporter Go module under projects/openshell-exporter/, so users can build and run the exporter locally or build their own Docker image from public source.

The package includes the exporter components and their tests, a local quickstart, Dockerfile, full configuration reference, HTTPS CloudEvents example, license/notices, and project.yaml with kind: tool. Module/import paths target Research. Other deployments and advanced-feature inquiries are directed to delgadof@nvidia.com. All additions stay in the project directory: 80 files and 15,569 lines including tests, comments, configuration, and documentation.

Delivery and privacy behavior

• The gateway receiver preserves pristine evidence across downstream mutation and retries. Draft/status checkpoint hashes are calculated before delivery and saved only after acceptance.
• CloudEvent source components are redacted before URL encoding, after identity calculation.
• Trace privacy checks cover resources, spans, scopes, events, and links; allow mode clears link trace state and drops input-supplied correlation status/missing/invalid markers outside spans. Only markers regenerated by span normalization retain their exemption; validated canonical correlation identifiers are preserved.
• A permanent rejection of one split HTTP batch does not prevent later batches from being attempted. A later transient failure still retries the queued input and can duplicate previously accepted records.
• The edge-to-central reference selects the internal authenticator and explicitly identifies the extensions to enable.

config.yaml normalizes and redacts records into local OpenTelemetry JSON output. http.config.yaml forwards raw OCSF inside CloudEvents data, bypassing normalization/redaction for receivers expecting that shape. Normalized HTTP delivery remains configurable and places the redacted source record in data.original.

Validation

Latest revision (b5be45f):

• All 24 cases of TestAllowModeRejectsInputCorrelationMarkers failed against 0e5258e and pass with the fix. The cases cover all three internal marker keys in each resource/scope/event/link container, inspect complete serialized output for the injected value, and verify both complete and partial span markers plus valid canonical identifiers.
• From the project directory: go mod verify, go vet ./..., go test -race ./..., and go build ./cmd/... passed. Builds used normal VCS stamping after allowing access to the worktree metadata.
• Both starter configurations passed validation and live local-process checks. The local starter produced valid normalized OCSF, served health, and resumed without replay after restart. The HTTPS starter verified the server certificate, sent the configured bearer token and raw OCSF, and retried identical batch bytes after HTTP 503. These checks used a temporary local receiver and synthetic credentials.

Additional qualification completed on the preceding revision (0e5258e); Docker, cross-platform builds, live gateway, internal mTLS, and documentation checks were not repeated for this trace-filter-only change:

• go mod verify, go vet ./..., go test -race ./..., and builds of both commands passed.
• New regression groups failed against the original implementation and passed after the fixes. Tests use the actual normalizer to check retry body/ID stability, rejection without checkpoint advancement, and unchanged reconciliation across restart. Serialized-output tests cover source-derived credentials and every trace metadata container. Split-batch tests cover permanent-then-success and permanent-then-transient responses.
• A read-only live gateway run collected lifecycle, log, and policy evidence from an existing sandbox. The first failed batch was identically retried, and no unchanged policy events were emitted after restarting against the persistent checkpoint store. No gateway workload or policy was changed.
• Two running local collectors verified the authentication correction over mTLS with distinct internal and destination tokens: the old pairing was rejected and the corrected pairing delivered.
• Native Linux AMD64 and the locally built Docker image passed the local and HTTPS quickstarts, including valid OCSF output, source-attribute redaction, health, bearer authentication, and stable retry after HTTP 503.
• Linux ARM64 and macOS ARM64 cross-builds of both commands passed.
• Research documentation renderer tests and strict site build passed.

Scope and release limits

The injector, separate proxy binary, Helm chart, deployment wrappers, and qualification tooling stay outside this package. Shared Go CI/license/header automation remains a maintenance follow-up.

Actual Apple Silicon execution and external message posting have not been validated. The quickstart is in the project README; documentation-site publication is a follow-up. Remote versioned go install needs a public project-prefixed module tag after merge. This PR publishes source, not binaries or container images.

[github-actions]

New project review

Revision: b5be45f3f6838e8d44524d58a3136a510a74d482 · Workflow and result artifacts

Review findings are advisory. Required checks remain separate merge gates.

Execution: success

Reviewer and guidelines revision: 21e28c92caafed3ed69d4252c76c10c9fcf79df7

Reviews

Project	Verdict	Guidelines	Findings
projects/openshell-exporter	✅ Pass	✅ Pass	2

projects/openshell-exporter

The PR adds `projects/openshell-exporter`, a self-contained Go module that builds a custom OpenTelemetry Collector distribution (`openshell-event-exporter`) plus a `healthcheck` helper for collecting OpenShell/OCSF evidence, normalizing and redacting it, and delivering it as CloudEvents or OTLP. It ships six first-party components (watchsandbox receiver; openshell, relay and evidencecontract processors; cloudevents exporter; storagehealth extension), a distroless multi-stage Dockerfile, a runnable local `config.yaml`, an HTTPS CloudEvents `http.config.yaml`, a 654-line `template.config.yaml` configuration reference, Apache-2.0 LICENSE/NOTICE, `project.yaml` (`kind: tool`), and ~139 Go tests. The README gives a credible zero-dependency first run (synthetic OCSF record → normalized OTel JSON in `output/events.json`, health probe on 127.0.0.1:13133) and a hardened Docker equivalent, and discloses key limitations (experimental, no published image or module tag, raw HTTP mode bypasses redaction, retries may duplicate, health/local output does not prove remote delivery, Apple Silicon execution unvalidated). Sampled validation and security defaults (HTTPS required unless explicitly opted out, TLS verification cannot be disabled, no credentials over plain HTTP, redirects refused, bounded batch/response sizes, identity computed before redaction) agree with the documented behavior. The only concrete inconsistency found is a Windows service entry point that cannot compile because the storagehealth extension uses `syscall.Statfs` unconditionally; the README does not claim Windows support, so this is non-blocking dead scope.

Project guidelines: ✅ Pass

Declared kind: `project.yaml` contains `kind: tool`. Purpose and scope: README states what the tool does, who to contact for other deployments, how it uses OpenShell (watchsandbox gateway receiver, sandbox OCSF logs, OpenShell Go SDK dependency), and marks itself experimental research software; plans (published tags, images, docs-site publication) are separated from working behavior. Self-contained layout: all 80 files live under the project directory with its own go.mod/go.sum; no empty docs scaffolding, and the configuration reference lives in `template.config.yaml` rather than a stub `docs/` tree. Reproducible first run: Go 1.26.8 or Docker prerequisites, exact build/run commands, synthetic input, and explicit expected output (`quickstart-1` in OTel JSON, not bare OCSF) plus a health check; credentials/CA needs for the HTTPS path are called out, secrets are gitignored, and required env vars (`EXPORTER_HEALTH_ENDPOINT`, `EXPORTER_DESTINATION_ENDPOINT`) are documented inline — no `.env` loading exists, so `.env.example` is inapplicable. Evidence: focused unit tests cover advertised critical behaviors (batch splitting, HTTP disposition/retryability, TLS/bearer lifecycle, redaction of serialized logs without changing identity, checkpoint-only-on-accept), and unverified areas are stated. Dependencies/licensing: go.mod plus committed go.sum (the Go-native lockfile; the `uv`/pyproject requirement is Python-only and inapplicable), Apache-2.0 LICENSE, NOTICE, SPDX headers, and go-licenses collection in the image. Proportionate engineering: direct component code with absolute module import paths, validation at real trust boundaries, no speculative frameworks; the one small excess (an uncompilable Windows service path) is recorded as a low finding. Tool/library kind expectations are met: installable CLI path, documented inputs/outputs/failures, focused verification of critical behavior.

Findings

• low: Windows service entry point cannot build because the storagehealth extension is POSIX-only — projects/openshell-exporter/cmd/openshell-event-exporter/main_windows.go:4
  • Evidence: `main_windows.go` is guarded by `//go:build windows` and runs `svc.Run(...)`, but `extension/storagehealthextension/extension.go:208` calls `syscall.Statfs` and declares `type statfsFunc func(string, *syscall.Statfs_t) error` with no build tags; `syscall.Statfs`/`Statfs_t` do not exist on Windows, so `GOOS=windows go build ./cmd/...` cannot compile the registered component set.
  • Recommendation: Either drop `main_windows.go` (folding `run` into a single non-Windows file) or state in the README that Windows is unsupported; if Windows is wanted later, split the statfs sampling into per-GOOS files.
  • Impact: A reader sees a Windows service path and may assume Windows is supported; in fact the build fails there. The file is unreachable dead scope for the documented Linux/macOS targets.
• low: Shipped configs depend on contrib component names that the project cannot demonstrate offline — projects/openshell-exporter/config.yaml:15
  • Evidence: `config.yaml:15` uses `file_log/ocsf` and `template.config.yaml:556` uses `load_balancing/internal`, while the registered factories come from upstream `filelogreceiver`/`loadbalancingexporter` at v0.160.0; `components.go:36-48` adds `DeprecatedAlias` handling, indicating in-flight upstream renaming. No test pins the documented configurations against the built factory set.
  • Recommendation: Add one small test (or a `validate`-based check) that loads `config.yaml` and `http.config.yaml` through the built factory set, pinning the documented configs against upstream component-name changes.
  • Impact: If either upstream name differs from the value used in the quickstart config, `validate --config config.yaml` fails at the very first documented step, and the failure would only surface at run time.

Strengths

• Two runnable, clearly differentiated configurations (`config.yaml` normalized/redacted local recovery; `http.config.yaml` raw OCSF inside CloudEvents `data`) with the privacy trade-off stated in both the file header and README.
• Security-conscious defaults verified in sampled code: TLS verification cannot be disabled, HTTPS required for credentials, redirects refused, response/batch sizes bounded, health endpoint bound to localhost by default.
• Regression tests target the exact behaviors the description claims were fixed (retry body/ID stability, checkpoint advance only after acceptance, permanent-then-later-batch delivery, redaction after identity).
• Honest limitation disclosure: no published image or module tag, duplicates possible on retry, local health/output does not prove remote delivery, Apple Silicon execution unvalidated.
• Reproducible packaging: digest-pinned multi-stage distroless Dockerfile with `go mod verify`, third-party license collection, and LICENSE/NOTICE plus SPDX headers throughout.

Limitations

• No Go toolchain, module cache, or network in the review sandbox: `go mod verify`, `go vet`, `go test -race`, `go build`, and the Docker quickstart were not executed, and upstream component type names could not be resolved.
• Operator-reported validations in the PR description (live read-only gateway run, two-collector mTLS check, native and container quickstarts, cross-builds) are untestable here and leave no artifact in the project.
• Implementation sampling only: read `cmd/*`, both shipped configs, part of `template.config.yaml`, cloudevents exporter config/exporter/factory, openshellprocessor config/factory/validation, evidencecontract config, storagehealth config/extension/factory, watchsandbox config and client head, and four representative tests. `envelope.go`, `redaction.go`, `relayprocessor`, `reconciler.go`, `convert.go`, and most test bodies were not reviewed.
• Snapshot omitted two symlinks (`CLAUDE.md -> AGENTS.md` at repo root and in `projects/openshell-agent-runner/`); neither is inside this project, so no in-scope content was affected, but repository-level agent-doc routing could not be confirmed.
• The claimed upstream source snapshot `6a440f05...` and the referenced OpenShell Go SDK pseudo-version could not be fetched or diffed.

[johnnygreco]

Update after reviewing 0e5258e33b82bcad15dd073b5e8d72d7f344a6e2: All five original concrete failure cases are fixed, all six original reproductions pass, and the completed checklist items below are checked. The native/HTTPS checks were independently rerun; live gateway and deployed distinct-token mTLS qualification are documented by the author. The original inline conversations are being resolved. One additional pre-existing internal-marker privacy bypass remains; see the current merge-readiness instructions for the focused fix, regression tests, and final validation needed before approval. The assessment below is the original review of 0358725, retained for context.

Requesting changes on commit 0358725569d20f5fa5eb50894887858a79f11444. The extraction generally fits the repository, but there are reproducible correctness/privacy defects and an authentication mismatch in the configuration reference. Five inline findings describe the triggers and targeted fixes.

Required next steps before re-review:

• Preserve pristine gateway receiver data across downstream delivery and retry. Calculate draft/status checkpoint hashes before downstream mutation, and save them only after successful acceptance. Add tests using the actual openshell normalizer that verify stable retry bodies/IDs and no output from a second unchanged reconciliation.
• Sanitize CloudEvent source components before URL encoding, after calculating identity. Add a serialized-output test proving that redacted values cannot survive in derived attributes.
• Apply trace privacy rules to scope, event, and link attributes as well as resources/spans; clear link trace state in allow mode. Cover every affected container in regression tests.
• Isolate a permanent HTTP rejection to its split batch so later valid batches are still attempted. Add a mixed rejection/success test, preserving the documented retry behavior.
• Use the internal authenticator for edge-to-central export and verify that the matching extension is enabled. Test with distinct internal and destination tokens.
• Rerun go mod verify, go vet ./..., go test -race ./..., go build ./cmd/..., and the local/HTTPS configuration checks. Include the regression-test results and qualify the changed receiver/durability paths as required by the project's AGENTS.md; then request re-review.

Maintainability and necessity: The OpenTelemetry component/factory organization and reuse of upstream storage, queues, receivers, and exporters are appropriate. Keep the focused tests; the failures below show where component-interaction coverage is missing. Kubernetes collection, edge/central contracts, and load balancing are unnecessary for the two quickstarts, but their retention is explicit PR scope and the reference demonstrates their intended use. I would clarify the intended supported surface before requiring their removal. Two optional simplifications are removing storagehealth from starter configurations with metrics.level: none, and removing or documenting the retained healthcheck --writable mode, which has no caller in this package.

Onboarding and repository fit: The self-contained project layout, kind: tool, go.mod/go.sum, LICENSE/NOTICE, and nested Go-specific instructions fit the repo. Direct Go commands are appropriate for its first Go project. The README explains installation, expected output, persistence, and raw-versus-redacted delivery. The native and HTTPS starter paths worked independently. However, the synthetic quickstart record omits metadata.version and is marked invalid; add that field so the first run demonstrates valid evidence. Explicit clone/cd instructions, a Go installation link, and a short tested gateway recipe would improve the path from the synthetic example to real OpenShell collection.

Recommended maintenance follow-ups, separate from the five inline defects: Add a path-filtered Go workflow for the documented checks and runnable configurations. No current workflow builds/tests this project; shared dependency-license scanning also overlooks go.mod/go.sum, and source-header checking overlooks .go. Extend that shared automation deliberately. Published binaries/images, documentation-site publication, and remote tagged installation were explicitly deferred by this PR and are not objections to this source extraction.

Independent validation: On Linux AMD64 with Go 1.26.8, go mod verify, go vet ./..., the existing go test -race ./... suite, and both command builds passed. The isolated archive build used -buildvcs=false because it lacked checkout metadata. Local configuration validation, normalized output, health, clean shutdown, and restart without replay passed. The HTTPS configuration passed CA verification, bearer-authenticated raw CloudEvents delivery, and retry after HTTP 503 with an unchanged batch. Six temporary regression-test functions failed on this head, confirming the behavioral defects described inline, including all three protected-attribute-container cases. The temporary tests used a Go overlay; PR source was unchanged.

Docker execution, Apple Silicon execution, live gateway collection, external applications, and a deployed edge/central pair were not independently exercised. Docker/configuration files were reviewed statically. The three independent review lenses covered gateway correctness/durability/TLS, processors/privacy/delivery, and adoption/complexity/repository fit; findings were verified before inclusion.

Reviewed diff: 3e0318e54020c440723f496dcf67b6bea6d00c40..0358725569d20f5fa5eb50894887858a79f11444, restricted to projects/openshell-exporter.

[johnnygreco]

Completed on b5be45f: The remaining privacy fix and all verification steps below are complete. The final conversation is resolved and an approving review has been submitted. See the final assessment. The original instructions are retained below for context.

The update in 0e5258e33b82bcad15dd073b5e8d72d7f344a6e2 addresses all five previously reported concrete failure cases. The six original independent reproduction tests now pass, and the fixes are focused and supported by meaningful regression coverage. I am resolving those original conversations and marking their checklist items complete.

One additional, pre-existing privacy bypass remains. It is tracked in the new inline comment on this review. That is the remaining code change required for my approval.

To make this ready to merge:

• In the relay processor's allow mode, remove input-supplied openshell.correlation.status, openshell.correlation.invalid, and openshell.correlation.missing from resources, scopes, events, and links, or otherwise ensure only values regenerated by the processor receive the protected-field exemption. Preserve the valid status/missing/invalid markers produced by span normalization and the already validated canonical correlation identifiers.
• Add a regression covering each of those three keys in each non-span container. Inspect complete serialized output and assert that the injected value is absent. Include a positive assertion that generated span markers remain correct. Confirm the regression fails on 0e5258e and passes with the fix.
• From projects/openshell-exporter, run go mod verify, go vet ./..., go test -race ./..., and go build ./cmd/...; check the local and HTTPS starter configurations and include the results in the PR update.
• Push the focused fix and request re-review. Once the new privacy conversation is resolved, my review can move to approval, subject to the repository's required checks and approvals.

The new reproduction supplies an event attribute such as openshell.correlation.status = "Bearer review-private-token". The current default allow mode retains the value in serialized output. The same reproduction fails for .invalid and .missing. These keys are absent from allowed_attributes, unlike deliberately permitted metadata such as service.name; their exemption exists because the processor generates them on spans, not because arbitrary input should be accepted under those names.

Verified on this revision: dependency verification, vet, the complete race-test suite, both builds, all six original independent reproductions, local startup/health/normalized output/restart without replay, and HTTPS CA verification/bearer authentication/stable retry after HTTP 503 passed. The archive build used -buildvcs=false solely because the review snapshot lacked checkout metadata. The revised synthetic record produces no previous invalid-source warning. Live gateway, Docker, Apple Silicon, and deployed internal mTLS execution were not independently repeated; the PR describes the author's additional qualification.

Go CI/license/header automation, a shorter gateway recipe, and optional removal of unused storage-health polling remain maintenance follow-ups, not additional blockers for this revision. The README setup, sample-record, and writable-healthcheck documentation improvements are addressed.

[johnnygreco]

Reviewed b5be45f3f6838e8d44524d58a3136a510a74d482: all findings from my review are now addressed, and I am approving this revision.

The final trace-filter change correctly removes input-supplied correlation status/missing/invalid markers from resources, scopes, events, and links in allow mode, while preserving markers regenerated by span normalization and valid canonical identifiers. The fix is focused and maintainable; independent review found no remaining issue.

Independent verification on Linux AMD64 with Go 1.26.8:

• All 24 new regression cases fail against 0e5258e and pass with this fix.
• All original independent review reproductions, including the reserved-marker privacy case, pass.
• go mod verify, go vet ./..., the full go test -race ./... suite, and builds of both commands pass. The isolated archive build used -buildvcs=false solely because it lacked checkout metadata.
• Local configuration validation, health, normalized output, clean shutdown, and restart without replay pass.
• The HTTPS starter passes certificate verification, bearer authentication, raw OCSF delivery, and identical batch retry after HTTP 503.

No further code changes are required by my review. I am resolving the final conversation. Go CI/license/header automation and the shorter gateway recipe remain nonblocking maintenance follow-ups. Approval does not replace the repository's required checks or other approvals. Docker, live gateway, deployed internal mTLS, and cross-platform execution were not independently repeated for this trace-filter-only update.

[johnnygreco]

All findings from my review are addressed in b5be45f. The focused fix, regression coverage, full Go checks, original reproductions, and native/HTTPS smoke checks have been independently verified. No unresolved code-review findings remain. Detailed assessment and validation: #70 (comment)