Skip to content

Add is_first_party label to CLI command metrics#7758

Open
SimonSiju wants to merge 2 commits into
mainfrom
simon/1p-org-migration-cli-metrics-22933
Open

Add is_first_party label to CLI command metrics#7758
SimonSiju wants to merge 2 commits into
mainfrom
simon/1p-org-migration-cli-metrics-22933

Conversation

@SimonSiju

Copy link
Copy Markdown

WHY are these changes introduced?

closes: https://github.com/shop/issues-develop/issues/22933

Part of the 1P org migration observability work (GSD #50881). "1P CI/CD pipelines function without breakage after cutover" is a named guardrail on the [Dev Platform] 1P Orgs Migration dashboard (Observe si2kjvt), but today it can't be measured: cli_commands_total (and the companion duration/elapsed metrics) carry only cli_version, exit, and job labels — no org / 1P dimension — so the CLI guardrail panels blend 1P + 3P + all external developers.

WHAT is this pull request doing?

Adds an is_first_party label to the CLI command metrics emitted from cli-kit:

  • cli_commands_total
  • cli_commands_duration_ms
  • cli_commands_wall_clock_elapsed_ms

The label is keyed off the 1P dev path via firstPartyDev() (the SHOPIFY_CLI_1P_DEV env var), which is the same signal that sets the X-Shopify-Cli-Employee header. Values are the strings "true" / "false", keeping cardinality low.

This lets the dashboard's CLI guardrail panels filter to 1P, so we get a 1P-scoped success/failure signal for commands like app dev, app dev clean, and generate extension after cutover.

Files changed:

  • packages/cli-kit/src/private/node/otel-metrics.ts — added is_first_party to the Labels type and populated it from firstPartyDev() in recordMetrics; imported firstPartyDev from context/local.
  • packages/cli-kit/src/private/node/otel-metrics.test.ts — added a test asserting all recorded metrics carry is_first_party: "true" when SHOPIFY_CLI_1P_DEV is set; added afterEach env cleanup.
  • packages/cli-kit/src/private/node/__snapshots__/otel-metrics.test.ts.snap — regenerated to include the new label ("false" when the 1P dev path is off).
  • .changeset/first-party-cli-metrics-label.md@shopify/cli-kit patch changeset.

How to test your changes?

  1. From the repo root, run the unit tests for the metrics module:
    pnpm --filter @shopify/cli-kit vitest run src/private/node/otel-metrics.test.ts
    
    All three tests should pass, including labels metrics as first-party when the 1P dev path is enabled.
  2. Manually verify the label is emitted. With metrics debug output enabled, run a CLI command as a third-party developer and confirm cli_commands_total records is_first_party: "false".
  3. Re-run the same command with the 1P dev path enabled (SHOPIFY_CLI_1P_DEV=1 shopify app dev) and confirm the metric records is_first_party: "true".

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

Adds an is_first_party label to cli_commands_total (and the duration /
wall-clock-elapsed metrics) so first-party (1P) Shopify developer usage
can be distinguished from third-party usage in observability dashboards.

The label is keyed off the 1P dev path (SHOPIFY_CLI_1P_DEV) via
firstPartyDev(), the same signal that sets the X-Shopify-Cli-Employee
header. This unblocks the 1P CLI guardrail panels on the [Dev Platform]
1P Orgs Migration dashboard, which currently blend 1P + 3P traffic.

Closes: shop/issues-develop#22933
@SimonSiju SimonSiju requested review from a team as code owners June 8, 2026 19:20
@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Jun 8, 2026
@github-actions github-actions Bot added no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users. and removed Area: @shopify/cli @shopify/cli package issues labels Jun 8, 2026

@murray-andrew-r murray-andrew-r left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think CLI is the right place to inject these metrics. We don't have guarantees that CLI will get updated in time, and I'm generally leery of trusting the client in general.

Let's look for another solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog This PR doesn't include a changeset entry. Is an internal only change not relevant to end users.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants