Skip to content

Instrument the auth client's network calls in Datadog - #3140

Merged
StylianosGakis merged 6 commits into
developfrom
fix/auth-datadog-instrumentation
Sep 10, 2026
Merged

Instrument the auth client's network calls in Datadog#3140
StylianosGakis merged 6 commits into
developfrom
fix/auth-datadog-instrumentation

Conversation

@StylianosGakis

@StylianosGakis StylianosGakis commented Sep 7, 2026

Copy link
Copy Markdown
Member

Pass the okhttp engine to the ktor client used for auth.
This allows for it to be hooked up to the Datadog setup so we get tracing properly there too.

More context in https://claude.ai/code/artifact/11bf9662-fb53-4507-bda7-7be7e0dea6a3

🤖 AI description:

:authlib builds its own Ktor client and has never carried a Datadog dependency, so every login, OTP and token-exchange call is invisible to RUM. Over 7 days in prod, auth.prod.hedvigit.com showed 634 resource events from the shared client's /member-authorization-codes and 4 from :authlib, all of them from 12.11.6. Auth failures reached us only as @error.source:logger prose with no status code and no duration, which is why five 504s on /member-login needed an investigation rather than a glance.

This hands :authlib an engine that is already instrumented, and leaves it Datadog-free. :datadog-android builds it, restoring the three pieces 7d14888e81 deleted. The auth host also joins setFirstPartyHosts and the engine's traced hosts.

Instrumenting at the OkHttp layer rather than adding the Datadog Ktor plugin is deliberate, and it is also what brings back the per-request timing breakdown. The Ktor plugin keys each RUM resource with its own uuid4() stored in a Ktor attribute, which DatadogEventListener cannot see, so timings can never attach to it. :network-clients is untouched and keeps the plugin.

Worth a reviewer's eye

The constructor shape in NetworkAuthRepository. Kotlin/Native does not carry default parameter values into the Obj-C API, so engine: HttpClientEngine? = null on the existing constructor would export initWithEnvironment:additionalHttpHeadersProvider:engine: and break Ugglan's AuthenticationClientAuthLib.swift at the next umbrella bump. Hence an internal primary constructor plus a public secondary one that keeps the original two parameters, and a JVM-only networkAuthRepositoryWithEngine so HttpClientEngine never reaches the shared surface.

Verification

Static:

  • Affected-module unit tests and the full :app:test pass. :app:compileDebugKotlin, :authlib:compileKotlinIosSimulatorArm64 and ktlintCheck pass, including the newly merged NamespaceImportRule.
  • NetworkAuthRepositoryEngineTest asserts the caller-supplied engine actually receives the request, and that the environment reaches the right host. Mutation-checked: removing the engine branch from buildKtorClient makes it fail, restoring it makes it pass.
  • HedvigShared.framework was relinked on a clean tree and with the change, and the generated Obj-C header is byte-identical. NetworkAuthRepository keeps its single init(environment:additionalHttpHeadersProvider:), and neither networkAuthRepositoryWithEngine nor baseUrl appears at all. This branch does not touch the iOS API surface, so Ugglan is unaffected by construction rather than by inspection.

On device, debug build 14.4.7 on a Pixel 6 Pro, full logout and BankID login against staging:

  • Logout, POST /member-login, status polling and /oauth/token all succeeded. Zero RUM errors across the episode.
  • All four auth calls now produce RUM resources, where the same endpoints produced none before: /member-login, /member-login/{id} twice, and /oauth/token, all status 200.
  • They carry dns, download and size, and provider.type: first party, confirming both the event listener and the setFirstPartyHosts change. connect, ssl and first_byte are absent on these, consistent with a pooled connection.
  • Control from the same session: apollo-router.dev.hedvigit.com resources, which still go through the Ktor plugin, have dns, size and download all null. Same app, same minute, which is a clean A/B on the timing claim.

The Datadog side is already done

Handled outside this PR on 9 September, so nothing else is needed once this ships. Both login metrics are now event_type: resource counting POST /member-login on the auth host for real users, failures are those same events filtered to [500 TO 599], and the SLO window moved from 7 to 30 days. The SLO keeps its id and monitor 93408872 keeps its Slack routing.

Two Datadog behaviours found on the way, worth knowing: event_type cannot be updated on a generated metric (the PATCH returns 200 and silently discards it), and deleting a generated metric does not purge its computed timeseries.

Why a planning doc is in this diff

docs/plans/2026-08-27-datadog-android-metric-recovery.md loses 143 lines, which is unrelated to the
Kotlin change and worth explaining rather than skimming past.

That doc was actively dangerous once the Datadog side landed. It listed android.login.network.count
and android.login.network.error among its pending edits with ready-to-paste @view.name target
filters and instructions to apply them. Following it would have overwritten the new auth-host filters
and put the login denominator back to counting apollo-router traffic. The snippets are removed rather
than annotated, so there is nothing left to paste.

While in there, three things that were carrying no weight came out. Its rollback appendix recorded ten
pre-change filter definitions on the stated grounds that Datadog keeps no history of a generated
metric's definition, which was true when written and stopped being true when 9e5525fc27 landed; it
is now a pointer to that commit. The app-side narrative is in #3104 and is reduced to a summary. And
eight metrics that had read zero since March, held only because nobody knew whether deleting a
generated metric purges its timeseries, are deleted; it does not, and their definitions are in the
message of the commit that deleted them.

No behaviour anywhere depends on that file. Skipping it costs a reviewer nothing.

`:authlib` builds its own Ktor client and carries no Datadog dependency, so every
login, OTP and token-exchange call was invisible to RUM. Measured over 7 days in
prod: 4 resource events on auth.prod.hedvigit.com came from `:authlib`, all of them
from 12.11.6, against 634 from the shared client's `/member-authorization-codes`.
Auth failures surfaced only as `@error.source:logger` prose with no status code and
no duration, which is why five 504s on `/member-login` took an investigation to find.

`:authlib` now accepts a caller-supplied engine, and `:datadog-android` supplies one
carrying `DatadogInterceptor`, `TracingInterceptor` and `DatadogEventListener`,
restoring what 7d14888 deleted. Instrumenting at the OkHttp layer rather than
through the Datadog Ktor plugin also brings back the per-request timing breakdown:
the plugin keys each RUM resource with its own `uuid4()` held in a Ktor attribute,
which `DatadogEventListener` cannot see, so timings can never attach to it.

The auth host joins `setFirstPartyHosts` and the engine's traced hosts, so its
resources stop being classified as third party.

`NetworkAuthRepository`'s primary constructor is `internal`, with a `public`
secondary constructor keeping the original two parameters. Kotlin/Native does not
carry default values into the Obj-C API, so a defaulted parameter would have
exported `initWithEnvironment:additionalHttpHeadersProvider:engine:` and broken
Ugglan's `AuthenticationClientAuthLib.swift` at the next umbrella bump. Diffed
against a clean relink of `HedvigShared.framework`, the exported API gains only a
read-only `AuthEnvironment.baseUrl` property.
@StylianosGakis
StylianosGakis force-pushed the fix/auth-datadog-instrumentation branch from 54c68cc to 521655d Compare September 7, 2026 15:05
@StylianosGakis
StylianosGakis removed this pull request from stack #3142 September 9, 2026 08:15
…doing it

Two gaps a review of this branch turned up.

Nothing asserted that the caller-supplied engine is actually used, which is the
whole behavioural change here. Simplifying `buildKtorClient` back to a single
`HttpClient(httpClientEngineFactory())` call left every test, ktlint and the app
itself passing while auth silently stopped producing RUM resources again.
Verified by mutation: with the wiring removed the new test fails, with it restored
it passes. A second test covers the environment reaching the right host.

`docs/plans/2026-08-27-datadog-android-metric-recovery.md` listed
`android.login.network.count` and `android.login.network.error` among its pending
edits, with copy-pasteable `@view.name` target filters. Those two were rebuilt on
auth resource events on 2026-09-09, so following that doc would have reverted them
and put the denominator back to counting apollo-router traffic that merely
coincided with the login screen. The targets are removed rather than annotated, so
there is nothing left to paste, and the rollback section is marked historical.

Also records that `event_type` cannot be changed via the RUM metrics PATCH, which
returns 200 and silently discards it, for whoever does the remaining six.
Review follow-ups, all of which shrink the change.

`authHost()` was re-deriving the auth environment from `isProduction` and reading
`AuthEnvironment.baseUrl`, which meant a second place deciding which backend the
build talks to and a public API widening on a module the umbrella exports. Neither
was necessary: `HedvigBuildConstants.urlAuthService` already exists, already yields
the same hostnames, and is driven by the full flavor rather than a boolean.

So the host now comes from that constant, the duplicated environment decision is
gone, and `AuthEnvironment.baseUrl` goes back to `internal`. The exported Obj-C
header is now byte-identical to develop: `NetworkAuthRepository` keeps its single
`init(environment:additionalHttpHeadersProvider:)` and `baseUrl` does not appear at
all, so this branch no longer touches the iOS API surface in any way.

`:auth-core-public` also declares `api(libs.ktor.client.core)` rather than
`implementation`, since `AuthMetroProviders` exposes `HttpClientEngine` in a public
signature. Previously `:app` only compiled because authlib's jvmMain `api` leaked
it two modules away, so tidying that line would have broken the build with an
unresolved reference inside generated Metro code.

The provider KDoc no longer names the alternative it was chosen over, per the
comment rules in CLAUDE.md.
The eight metrics below targeted screens deleted in March 2026, or 2023 for
`android.auth.failure`, and had read zero ever since. They were held back only
because nobody knew whether deleting a generated metric also purges its
already-computed timeseries.

It does not. Verified 2026-09-09 when `android.login.network.error` was deleted and
recreated under the same name and kept all 24 of its points. These eight had no
data to lose either way. Confirmed before deleting that no dashboard, notebook,
monitor or SLO referenced any of them: 37 dashboards and 21 notebooks swept, plus
every monitor and SLO.

The doc also loses 143 lines that were carrying no weight. The rollback appendix
recorded ten pre-change filter definitions on the stated grounds that Datadog keeps
no history of a generated metric's definition, which was true when written and
stopped being true when commit 9e5525f landed. It is replaced by a pointer to
that commit. The app-side narrative is in PR #3104 and is reduced to a summary. The
claim-failure section keeps its live interpretation caveat and its open follow-up
and drops the rest.

Definitions of the deleted metrics, recorded here because they exist nowhere else:

      `android.claim.singleitempayout`

    `event_type: view`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:com.hedvig.android.data.claimflow.ClaimFlowDestination.SingleItemPayout

      `android.claim.submitclaim`

    `event_type: resource`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:"com/hedvig/android/data/claimflow/ClaimFlowDestination/Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}"

      `android.claimsummary.network.count`

    `event_type: resource`, `count`, grouped by ``

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"com.hedvig.android.data.claimflow.ClaimFlowDestination.Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}"

      `android.claimsummary.network.error`

    `event_type: error`, `count`, grouped by ``

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"com.hedvig.android.data.claimflow.ClaimFlowDestination.Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}" -@error.stack:java.net.SocketTimeoutException* -@error.stack:java.net.ConnectException*

      `android.resource.claimflow`

    `event_type: resource`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:com/hedvig/android/data/claimflow/ClaimFlowDestination/*

      `android.claimflow.errors`

    `event_type: error`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:com/hedvig/android/data/claimflow/ClaimFlowDestination/* @connectivity.status:connected -@error.stack:java.net.ConnectException* -@error.stack:java.net.SocketException* -@error.stack:java.net.SocketTimeoutException* -@error.stack:java.net.UnknownHostException* -@error.stack:java.util.concurrent.CancellationException*

      `android.auth.failure`

    `event_type: view`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"BankId Error"

      `android.claim.failure`

    `event_type: view`, `count`, grouped by `env`

    @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:com.hedvig.android.data.claimflow.ClaimFlowDestination.Failure
It still claimed one open follow-up and a 2026-08-28 date. There are three open
items, and the OR-branch trigger has a measured value now: versions at or below
14.3.2 were still about 13% of prod view traffic on 2026-09-10.
@StylianosGakis
StylianosGakis force-pushed the fix/auth-datadog-instrumentation branch from 759a72d to 5abf4f1 Compare September 10, 2026 09:51
@StylianosGakis
StylianosGakis marked this pull request as ready for review September 10, 2026 11:38
@StylianosGakis
StylianosGakis requested a review from a team as a code owner September 10, 2026 11:38
@StylianosGakis
StylianosGakis merged commit f97978b into develop Sep 10, 2026
4 checks passed
@StylianosGakis
StylianosGakis deleted the fix/auth-datadog-instrumentation branch September 10, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants