Instrument the auth client's network calls in Datadog - #3140
Merged
Conversation
`:authlib` builds its own Ktor client and carries no Datadog dependency, so every login, OTP and token-exchange call was invisible to RUM. Measured over 7 days in prod: 4 resource events on auth.prod.hedvigit.com came from `:authlib`, all of them from 12.11.6, against 634 from the shared client's `/member-authorization-codes`. Auth failures surfaced only as `@error.source:logger` prose with no status code and no duration, which is why five 504s on `/member-login` took an investigation to find. `:authlib` now accepts a caller-supplied engine, and `:datadog-android` supplies one carrying `DatadogInterceptor`, `TracingInterceptor` and `DatadogEventListener`, restoring what 7d14888 deleted. Instrumenting at the OkHttp layer rather than through the Datadog Ktor plugin also brings back the per-request timing breakdown: the plugin keys each RUM resource with its own `uuid4()` held in a Ktor attribute, which `DatadogEventListener` cannot see, so timings can never attach to it. The auth host joins `setFirstPartyHosts` and the engine's traced hosts, so its resources stop being classified as third party. `NetworkAuthRepository`'s primary constructor is `internal`, with a `public` secondary constructor keeping the original two parameters. Kotlin/Native does not carry default values into the Obj-C API, so a defaulted parameter would have exported `initWithEnvironment:additionalHttpHeadersProvider:engine:` and broken Ugglan's `AuthenticationClientAuthLib.swift` at the next umbrella bump. Diffed against a clean relink of `HedvigShared.framework`, the exported API gains only a read-only `AuthEnvironment.baseUrl` property.
StylianosGakis
force-pushed
the
fix/auth-datadog-instrumentation
branch
from
September 7, 2026 15:05
54c68cc to
521655d
Compare
StylianosGakis
removed this pull request from stack #3142
September 9, 2026 08:15
…doing it Two gaps a review of this branch turned up. Nothing asserted that the caller-supplied engine is actually used, which is the whole behavioural change here. Simplifying `buildKtorClient` back to a single `HttpClient(httpClientEngineFactory())` call left every test, ktlint and the app itself passing while auth silently stopped producing RUM resources again. Verified by mutation: with the wiring removed the new test fails, with it restored it passes. A second test covers the environment reaching the right host. `docs/plans/2026-08-27-datadog-android-metric-recovery.md` listed `android.login.network.count` and `android.login.network.error` among its pending edits, with copy-pasteable `@view.name` target filters. Those two were rebuilt on auth resource events on 2026-09-09, so following that doc would have reverted them and put the denominator back to counting apollo-router traffic that merely coincided with the login screen. The targets are removed rather than annotated, so there is nothing left to paste, and the rollback section is marked historical. Also records that `event_type` cannot be changed via the RUM metrics PATCH, which returns 200 and silently discards it, for whoever does the remaining six.
Review follow-ups, all of which shrink the change. `authHost()` was re-deriving the auth environment from `isProduction` and reading `AuthEnvironment.baseUrl`, which meant a second place deciding which backend the build talks to and a public API widening on a module the umbrella exports. Neither was necessary: `HedvigBuildConstants.urlAuthService` already exists, already yields the same hostnames, and is driven by the full flavor rather than a boolean. So the host now comes from that constant, the duplicated environment decision is gone, and `AuthEnvironment.baseUrl` goes back to `internal`. The exported Obj-C header is now byte-identical to develop: `NetworkAuthRepository` keeps its single `init(environment:additionalHttpHeadersProvider:)` and `baseUrl` does not appear at all, so this branch no longer touches the iOS API surface in any way. `:auth-core-public` also declares `api(libs.ktor.client.core)` rather than `implementation`, since `AuthMetroProviders` exposes `HttpClientEngine` in a public signature. Previously `:app` only compiled because authlib's jvmMain `api` leaked it two modules away, so tidying that line would have broken the build with an unresolved reference inside generated Metro code. The provider KDoc no longer names the alternative it was chosen over, per the comment rules in CLAUDE.md.
The eight metrics below targeted screens deleted in March 2026, or 2023 for `android.auth.failure`, and had read zero ever since. They were held back only because nobody knew whether deleting a generated metric also purges its already-computed timeseries. It does not. Verified 2026-09-09 when `android.login.network.error` was deleted and recreated under the same name and kept all 24 of its points. These eight had no data to lose either way. Confirmed before deleting that no dashboard, notebook, monitor or SLO referenced any of them: 37 dashboards and 21 notebooks swept, plus every monitor and SLO. The doc also loses 143 lines that were carrying no weight. The rollback appendix recorded ten pre-change filter definitions on the stated grounds that Datadog keeps no history of a generated metric's definition, which was true when written and stopped being true when commit 9e5525f landed. It is replaced by a pointer to that commit. The app-side narrative is in PR #3104 and is reduced to a summary. The claim-failure section keeps its live interpretation caveat and its open follow-up and drops the rest. Definitions of the deleted metrics, recorded here because they exist nowhere else: `android.claim.singleitempayout` `event_type: view`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:com.hedvig.android.data.claimflow.ClaimFlowDestination.SingleItemPayout `android.claim.submitclaim` `event_type: resource`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:"com/hedvig/android/data/claimflow/ClaimFlowDestination/Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}" `android.claimsummary.network.count` `event_type: resource`, `count`, grouped by `` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"com.hedvig.android.data.claimflow.ClaimFlowDestination.Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}" `android.claimsummary.network.error` `event_type: error`, `count`, grouped by `` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"com.hedvig.android.data.claimflow.ClaimFlowDestination.Summary/{locationOptions}?claimTypeTitle={claimTypeTitle}&selectedLocation={selectedLocation}&dateOfOccurrence={dateOfOccurrence}&maxDate={maxDate}&preferredCurrency={preferredCurrency}&purchaseDate={purchaseDate}&purchasePrice={purchasePrice}&availableItemBrands={availableItemBrands}&selectedItemBrand={selectedItemBrand}&availableItemModels={availableItemModels}&selectedItemModel={selectedItemModel}&availableItemProblems={availableItemProblems}&selectedItemProblems={selectedItemProblems}" -@error.stack:java.net.SocketTimeoutException* -@error.stack:java.net.ConnectException* `android.resource.claimflow` `event_type: resource`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:com/hedvig/android/data/claimflow/ClaimFlowDestination/* `android.claimflow.errors` `event_type: error`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.url:com/hedvig/android/data/claimflow/ClaimFlowDestination/* @connectivity.status:connected -@error.stack:java.net.ConnectException* -@error.stack:java.net.SocketException* -@error.stack:java.net.SocketTimeoutException* -@error.stack:java.net.UnknownHostException* -@error.stack:java.util.concurrent.CancellationException* `android.auth.failure` `event_type: view`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:"BankId Error" `android.claim.failure` `event_type: view`, `count`, grouped by `env` @application.id:4d7b8355-396d-406e-b543-30a073050e8f @view.name:com.hedvig.android.data.claimflow.ClaimFlowDestination.Failure
It still claimed one open follow-up and a 2026-08-28 date. There are three open items, and the OR-branch trigger has a measured value now: versions at or below 14.3.2 were still about 13% of prod view traffic on 2026-09-10.
StylianosGakis
force-pushed
the
fix/auth-datadog-instrumentation
branch
from
September 10, 2026 09:51
759a72d to
5abf4f1
Compare
StylianosGakis
marked this pull request as ready for review
September 10, 2026 11:38
panasetskaya
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pass the okhttp engine to the ktor client used for auth.
This allows for it to be hooked up to the Datadog setup so we get tracing properly there too.
More context in https://claude.ai/code/artifact/11bf9662-fb53-4507-bda7-7be7e0dea6a3
🤖 AI description:
:authlibbuilds its own Ktor client and has never carried a Datadog dependency, so every login, OTP and token-exchange call is invisible to RUM. Over 7 days in prod,auth.prod.hedvigit.comshowed 634 resource events from the shared client's/member-authorization-codesand 4 from:authlib, all of them from 12.11.6. Auth failures reached us only as@error.source:loggerprose with no status code and no duration, which is why five 504s on/member-loginneeded an investigation rather than a glance.This hands
:authliban engine that is already instrumented, and leaves it Datadog-free.:datadog-androidbuilds it, restoring the three pieces7d14888e81deleted. The auth host also joinssetFirstPartyHostsand the engine's traced hosts.Instrumenting at the OkHttp layer rather than adding the Datadog Ktor plugin is deliberate, and it is also what brings back the per-request timing breakdown. The Ktor plugin keys each RUM resource with its own
uuid4()stored in a Ktor attribute, whichDatadogEventListenercannot see, so timings can never attach to it.:network-clientsis untouched and keeps the plugin.Worth a reviewer's eye
The constructor shape in
NetworkAuthRepository. Kotlin/Native does not carry default parameter values into the Obj-C API, soengine: HttpClientEngine? = nullon the existing constructor would exportinitWithEnvironment:additionalHttpHeadersProvider:engine:and break Ugglan'sAuthenticationClientAuthLib.swiftat the next umbrella bump. Hence aninternalprimary constructor plus apublicsecondary one that keeps the original two parameters, and a JVM-onlynetworkAuthRepositoryWithEnginesoHttpClientEnginenever reaches the shared surface.Verification
Static:
:app:testpass.:app:compileDebugKotlin,:authlib:compileKotlinIosSimulatorArm64and ktlintCheck pass, including the newly mergedNamespaceImportRule.NetworkAuthRepositoryEngineTestasserts the caller-supplied engine actually receives the request, and that the environment reaches the right host. Mutation-checked: removing the engine branch frombuildKtorClientmakes it fail, restoring it makes it pass.HedvigShared.frameworkwas relinked on a clean tree and with the change, and the generated Obj-C header is byte-identical.NetworkAuthRepositorykeeps its singleinit(environment:additionalHttpHeadersProvider:), and neithernetworkAuthRepositoryWithEnginenorbaseUrlappears at all. This branch does not touch the iOS API surface, so Ugglan is unaffected by construction rather than by inspection.On device, debug build 14.4.7 on a Pixel 6 Pro, full logout and BankID login against staging:
POST /member-login, status polling and/oauth/tokenall succeeded. Zero RUM errors across the episode./member-login,/member-login/{id}twice, and/oauth/token, all status 200.dns,downloadandsize, andprovider.type: first party, confirming both the event listener and thesetFirstPartyHostschange.connect,sslandfirst_byteare absent on these, consistent with a pooled connection.apollo-router.dev.hedvigit.comresources, which still go through the Ktor plugin, havedns,sizeanddownloadall null. Same app, same minute, which is a clean A/B on the timing claim.The Datadog side is already done
Handled outside this PR on 9 September, so nothing else is needed once this ships. Both login metrics are now
event_type: resourcecountingPOST /member-loginon the auth host for real users, failures are those same events filtered to[500 TO 599], and the SLO window moved from 7 to 30 days. The SLO keeps its id and monitor93408872keeps its Slack routing.Two Datadog behaviours found on the way, worth knowing:
event_typecannot be updated on a generated metric (the PATCH returns 200 and silently discards it), and deleting a generated metric does not purge its computed timeseries.Why a planning doc is in this diff
docs/plans/2026-08-27-datadog-android-metric-recovery.mdloses 143 lines, which is unrelated to theKotlin change and worth explaining rather than skimming past.
That doc was actively dangerous once the Datadog side landed. It listed
android.login.network.countand
android.login.network.erroramong its pending edits with ready-to-paste@view.nametargetfilters and instructions to apply them. Following it would have overwritten the new auth-host filters
and put the login denominator back to counting apollo-router traffic. The snippets are removed rather
than annotated, so there is nothing left to paste.
While in there, three things that were carrying no weight came out. Its rollback appendix recorded ten
pre-change filter definitions on the stated grounds that Datadog keeps no history of a generated
metric's definition, which was true when written and stopped being true when
9e5525fc27landed; itis now a pointer to that commit. The app-side narrative is in #3104 and is reduced to a summary. And
eight metrics that had read zero since March, held only because nobody knew whether deleting a
generated metric purges its timeseries, are deleted; it does not, and their definitions are in the
message of the commit that deleted them.
No behaviour anywhere depends on that file. Skipping it costs a reviewer nothing.