refactor: use crustdata enrichment source only for new member profiles (CM-924)#3830
Merged
refactor: use crustdata enrichment source only for new member profiles (CM-924)#3830
Conversation
services/apps/members_enrichment_worker/src/sources/crustdata/service.ts
Outdated
Show resolved
Hide resolved
services/apps/members_enrichment_worker/src/sources/crustdata/service.ts
Show resolved
Hide resolved
…ource skips a member
services/apps/members_enrichment_worker/src/sources/crustdata/service.ts
Show resolved
Hide resolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
isEnrichableBySource, we check if a Crustdata cache row already exists for the member (regardless of whether data is null or not). If it does, the member has already been through Crustdata and we skip it.enrichMember, when a source returns null but the cache already holds valid data, we now refresh the cache timestamp instead of overwriting with null. This prevents Crustdata data from being wiped when its cache becomes obsolete after 90 days and the source intentionally skips the member.Note
Medium Risk
Changes enrichment eligibility and cache update behavior, which can affect which sources run and whether cached profiles are retained; risk is moderated by being localized to the enrichment worker and primarily gatekeeping/DB-write logic.
Overview
Crustdata enrichment is now one-shot per member:
isEnrichableBySourceloads cache rows even whendataisnulland skips Crustdata entirely if any Crustdata cache row already exists for that member.The enrichment workflow now avoids wiping valid cached data: when a source returns
nullbut the cache has non-null data, it only updatescache.updatedAtinstead of overwriting the cache withnull(supporting sources that intentionally skip members).Written by Cursor Bugbot for commit 5c17650. This will update automatically on new commits. Configure here.