Add Node dependency matrix sample and Compose setup#93
Add Node dependency matrix sample and Compose setup#93officialasishkumar wants to merge 19 commits intomainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new node-dependency-matrix TypeScript sample intended to exercise a broad set of inbound (HTTP/gRPC) and outbound dependency protocols to validate Keploy cloud record/replay, with local Docker Compose and Kind/Kubernetes deployment tooling.
Changes:
- Introduces a new Node/TypeScript app + fixture services covering HTTP/HTTP2/gRPC, MySQL/Postgres/Mongo/Redis, Kafka, SQS, and generic TLS scenarios.
- Adds Dockerfile + entrypoint CA-bundling, plus a Compose stack to run the full dependency set locally.
- Adds a Kind/Kubernetes deployment bundle (manifests, deploy script, docs) and a machine-readable expectations contract for verification.
Reviewed changes
Copilot reviewed 34 out of 39 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| node-dependency-matrix/tsconfig.json | TypeScript build configuration for the sample. |
| node-dependency-matrix/src/lib/proto.ts | gRPC proto loading helpers and client typings. |
| node-dependency-matrix/src/lib/log.ts | Structured JSON logging helper used across the sample. |
| node-dependency-matrix/src/lib/dependencies.ts | Scenario runners that exercise each outbound dependency/protocol. |
| node-dependency-matrix/src/lib/config.ts | Env-driven config loader and expectations file reader. |
| node-dependency-matrix/src/global.d.ts | Global typing shim for crypto.randomUUID(). |
| node-dependency-matrix/src/bin/tlsProxy.ts | TLS proxy process for wrapping plaintext dependencies with TLS. |
| node-dependency-matrix/src/bin/dependencyFixture.ts | Fixture service that provides HTTPS/HTTP2/gRPC/generic TLS endpoints. |
| node-dependency-matrix/src/bin/app.ts | Main HTTP + gRPC app exposing scenario endpoints and orchestration. |
| node-dependency-matrix/scripts/send_grpc_traffic.sh | Script to generate inbound gRPC traffic against the sample. |
| node-dependency-matrix/scripts/record_traffic.sh | Script to generate inbound HTTP traffic across all scenarios. |
| node-dependency-matrix/scripts/localstack-init/01-create-queue.sh | LocalStack init script to create the SQS queue. |
| node-dependency-matrix/scripts/generate_certs.sh | Helper to generate a CA + leaf certs for the TLS flows. |
| node-dependency-matrix/scripts/compose_up.sh | Convenience wrapper to generate certs and start Compose stack. |
| node-dependency-matrix/scripts/compose_down.sh | Convenience wrapper to stop Compose stack and clean volumes. |
| node-dependency-matrix/proto/dependency_matrix.proto | Proto definition for fixture + matrix gRPC services. |
| node-dependency-matrix/package.json | Node package manifest and runtime/dev dependencies. |
| node-dependency-matrix/k8s/port-forward.sh | Port-forward helper for local/Playwright-friendly access. |
| node-dependency-matrix/k8s/manifests/08-app.yaml | K8s Service/Deployment for the main matrix app. |
| node-dependency-matrix/k8s/manifests/07-tls-proxies.yaml | K8s services/deployments for TLS proxy pods around dependencies. |
| node-dependency-matrix/k8s/manifests/06-fixture-service.yaml | K8s Service/Deployment for the fixture-service. |
| node-dependency-matrix/k8s/manifests/05-redpanda.yaml | K8s Service/Deployment for Redpanda (Kafka). |
| node-dependency-matrix/k8s/manifests/04-redis.yaml | K8s Service/Deployment for Redis. |
| node-dependency-matrix/k8s/manifests/03-mongo.yaml | K8s Service/Deployment for MongoDB. |
| node-dependency-matrix/k8s/manifests/02-postgres.yaml | K8s Service/Deployment for Postgres. |
| node-dependency-matrix/k8s/manifests/01-mysql.yaml | K8s Service/Deployment for MySQL. |
| node-dependency-matrix/k8s/manifests/00-localstack.yaml | K8s ConfigMap/Service/Deployment for LocalStack (SQS). |
| node-dependency-matrix/k8s/kind-config.yaml | Kind cluster config with NodePort host mappings. |
| node-dependency-matrix/k8s/deploy-kind.sh | End-to-end Kind bootstrap script (cluster, certs, image load, apply). |
| node-dependency-matrix/k8s/README.md | Detailed Kind + Keploy UI/Helm flow documentation. |
| node-dependency-matrix/fixtures/expected-values.json | Machine-readable contract for coverage and replay expectations. |
| node-dependency-matrix/entrypoint.sh | Container entrypoint to build combined CA bundle for TLS clients. |
| node-dependency-matrix/compose.yaml | Compose orchestration for app + all dependency services/proxies. |
| node-dependency-matrix/README.md | Sample-level usage docs (Kind + Compose quickstart). |
| node-dependency-matrix/Dockerfile | Multi-stage build and runtime image packaging. |
| node-dependency-matrix/.gitignore | Ignores build artifacts and generated certs for the sample dir. |
| node-dependency-matrix/.dockerignore | Excludes non-runtime content from the image build context. |
| README.md | Adds the new sample to the repository’s sample index. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…n the node dependency matrix. Update documentation to reflect new features, including async workflows and enhanced deduplication for GET and POST requests. Adjust traffic recording scripts and improve error handling in the application logic.
…for better usability. Modify README and STAGING_RUNBOOK to clarify port-forwarding steps and add hints for using the hosted UI. Improve error handling in traffic recording scripts to guide users when using NodePorts.
…nd response structure. Update Kafka scenario to include cluster and topic details, and enhance generic scenario with better socket management and timeout handling.
…s. Introduced a new KIND_NODE_IMAGE variable to allow users to specify a custom image when creating the cluster, improving flexibility in deployment configurations.
- expectedTestcasesExact: 23 → 18 (post-dedup count, UI shows unique) - expectedAdditionalTestcasesExact: 2 → 0 (gRPC not counted in UI total) - Dedup entries: use path-only format without query params to match how the Static dedup section groups entries in the UI - /dedup/catalog duplicateCount: 4 (all alpha+beta grouped) - /dedup/order duplicateCount: 3 (all orders grouped) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The enterprise agent (v3.2.47) reliably captures Http, MySQL, PostgresV2, and Redis mock kinds. Add PostgresV2 and Redis to requiredKinds so the kube-regression tests assert all currently supported parsers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Route ERROR logs to stderr via console.error for proper log routing - Cache CA bundle read to avoid blocking the event loop on repeated calls - Ensure entrypoint.sh always creates the CA bundle file with a warning if no certs found - Align @types/express version with express v4 runtime dependency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 36 out of 41 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| const catalogSyncJobs = new Map<string, CatalogSyncJobState>(); | ||
|
|
There was a problem hiding this comment.
catalogSyncJobs is an unbounded in-memory Map and completed/failed jobs are never removed. Repeated calls to start jobs with unique IDs will grow memory over time; consider deleting jobs after completion (or implementing a TTL/size cap) so the sample doesn’t leak state in long-running runs.
| tls: { | ||
| ca: readCaBundle(config), | ||
| rejectUnauthorized: true, | ||
| checkServerIdentity: () => undefined | ||
| }, |
There was a problem hiding this comment.
checkServerIdentity: () => undefined disables TLS hostname verification. Even in a sample, this is a sharp security footgun and can mask misconfiguration. Consider gating this behind an explicit env flag (defaulting to secure behavior) and/or adding an inline comment explaining why hostname verification is intentionally skipped here.
| host: config.fixtureGenericHost, | ||
| port: config.fixtureGenericPort, | ||
| ca: readCaBundle(config), | ||
| rejectUnauthorized: true, | ||
| checkServerIdentity: () => undefined | ||
| }, |
There was a problem hiding this comment.
checkServerIdentity: () => undefined disables TLS hostname verification for the generic TLS socket. This reduces security and can hide real cert/SAN issues; consider guarding it behind an explicit env flag (or at least documenting the rationale inline) so the insecure behavior is clearly intentional and easy to disable.
| error('dependency scenario failed', { | ||
| scenario: runner.name, | ||
| error: err instanceof Error ? err.message : String(err) | ||
| }); |
There was a problem hiding this comment.
This error log doesn’t give a clear next step for someone running the sample (e.g., which dependency to check, whether CA bundle/config is missing, or how to reproduce). Consider including actionable remediation hints in the log fields/message (for example: which config URL/host was targeted and a short suggestion like verifying Compose/K8s dependencies are up and the CA bundle path is populated).
| "devDependencies": { | ||
| "@types/express": "^4.17.21", | ||
| "@types/node": "^24.7.2", | ||
| "@types/pg": "^8.15.5", | ||
| "tsx": "^4.20.6", | ||
| "typescript": "^5.9.3" |
There was a problem hiding this comment.
npm ci in the Dockerfile will use package-lock.json, but the lock currently resolves @types/express to v5.x while package.json pins v4.x. This can reintroduce the Express v4/v5 typing mismatch and potentially break the TypeScript build. Regenerate/update package-lock.json so it matches the @types/express@^4.17.21 constraint in package.json (and re-run npm ci to confirm).
| : > "${COMBINED_CA_CERT_PATH}" | ||
| >&2 echo "Warning: No CA certificates were found to create the combined CA bundle at '${COMBINED_CA_CERT_PATH}'. To fix this, either set SAMPLE_CA_CERT_PATH to a valid CA bundle file or mount /tmp/keploy-tls/ca.crt before starting this container so fixture endpoints can establish TLS correctly." |
There was a problem hiding this comment.
This startup message is labeled as "Warning" and the container continues with an empty CA bundle, which will later cause TLS-dependent scenarios to fail in less obvious ways. Consider failing fast with a clear error (and remediation steps) unless an explicit env flag allows running without CA certs; also avoid adding warning-level messaging if the project’s logging policy is to not emit warnings.
| : > "${COMBINED_CA_CERT_PATH}" | |
| >&2 echo "Warning: No CA certificates were found to create the combined CA bundle at '${COMBINED_CA_CERT_PATH}'. To fix this, either set SAMPLE_CA_CERT_PATH to a valid CA bundle file or mount /tmp/keploy-tls/ca.crt before starting this container so fixture endpoints can establish TLS correctly." | |
| if [ "${ALLOW_EMPTY_CA_BUNDLE:-}" = "true" ]; then | |
| : > "${COMBINED_CA_CERT_PATH}" | |
| >&2 echo "Running without any CA certificates at '${COMBINED_CA_CERT_PATH}' because ALLOW_EMPTY_CA_BUNDLE=true. TLS connections that rely on system CAs may fail. To fix this, set SAMPLE_CA_CERT_PATH to a valid CA bundle file or mount /tmp/keploy-tls/ca.crt before starting this container." | |
| else | |
| >&2 echo "Error: No CA certificates were found to create the combined CA bundle at '${COMBINED_CA_CERT_PATH}'." | |
| >&2 echo "Remediation: Set SAMPLE_CA_CERT_PATH to a valid CA bundle file or mount /tmp/keploy-tls/ca.crt before starting this container so fixture endpoints can establish TLS correctly. To bypass this check (not recommended), set ALLOW_EMPTY_CA_BUNDLE=true." | |
| exit 1 | |
| fi |
Mock kinds captured by the sidecar are non-deterministic across runs (depends on pod timing and sidecar readiness). Only Http is guaranteed in every run. Other kinds are logged as informational via preferredKinds. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
npm ci fails in CI because @types/express and related packages were updated in package.json without regenerating the lock file. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Docker Hub 429 rate limits cause Kind cluster deployment failures in CI. Switch mysql, postgres, mongo, and redis images to public.ecr.aws/docker/library/ mirrors which have no rate limits. This matches the pattern used by api-server and enterprise-ui CI pipelines (playwright-setup.sh). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
crictl pull goes directly to the registry (no cache), hitting rate limits. docker pull uses the Docker daemon cache (restored from MinIO by CI). kind load transfers cached images into the Kind node's containerd without any registry access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Both Docker Hub and ECR public are rate-limited. Configure Kind's containerd to use mirror.gcr.io (Google's Docker Hub mirror) which has no rate limits. Reverted ECR image references back to docker.io since the GCR mirror handles them transparently inside Kind. This matches the approach used by k8s-proxy CI (setup-cluster.sh). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All Keploy CI repos use a local Docker registry mirror at 192.168.116.165:5000 to avoid rate limits. Updated Kind containerd config to use this mirror (primary) with mirror.gcr.io as fallback. Matches the pattern in k8s-proxy/scripts/ci/setup-cluster.sh. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
kind load docker-image with multiple images fails entirely if one image has issues. Load each image individually so failures are isolated. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This pull request introduces a new sample application,
node-dependency-matrix, to the repository. The sample is designed to validate Keploy cloud record/replay across a wide range of dependencies, providing comprehensive coverage for various protocols, databases, and messaging systems. The changes include the application's source, configuration files for Docker and Docker Compose, Kubernetes deployment scripts and documentation, and supporting scripts for setup and traffic generation.Key highlights of the changes:
Addition of the Node Dependency Matrix Sample
node-dependency-matrixsample application, which exercises incoming HTTP/gRPC and outgoing connections to HTTPS, HTTP/2, gRPC, MySQL, Postgres, Mongo, Redis, Kafka, SQS, and generic TLS endpoints, as well as noisy and replay-failure scenarios. Includes endpoints and fixtures for comprehensive dependency matrix testing. [1] [2]Containerization and Local Development
Dockerfile,.dockerignore, and.gitignorefor building and running the application in a containerized environment, supporting both local and production deployments. [1] [2] [3]compose.yamlfor Docker Compose, allowing local orchestration of the app and all required services (databases, message brokers, TLS proxies, etc.) for end-to-end testing.entrypoint.shto handle CA certificate bundling and environment setup within containers.Kubernetes Support and Documentation
k8s/deploy-kind.sh) and detailed documentation (k8s/README.md) for deploying the sample and its dependencies on a Kind cluster, including guidance for both local and VM-based environments, ingress configuration, and troubleshooting. [1] [2]Integration with Keploy
fixtures/expected-values.json), and scripts for recording and replaying traffic, ensuring the sample can be used for regression testing and Playwright-based verification with Keploy. [1] [2]These changes significantly expand the repository's coverage for dependency matrix testing and provide robust tooling and documentation for both local and Kubernetes-based workflows.