Skip to content

update ingestion-helper status and version endpoints to support lists of multiple imports#563

Merged
dwnoble merged 3 commits into
datacommonsorg:masterfrom
dwnoble:ingestion-helper-multi-import
Jun 23, 2026
Merged

update ingestion-helper status and version endpoints to support lists of multiple imports#563
dwnoble merged 3 commits into
datacommonsorg:masterfrom
dwnoble:ingestion-helper-multi-import

Conversation

@dwnoble

@dwnoble dwnoble commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

This enables the DCP ingestion workflow to update multiple imports at once

Submit with: datacommonsorg/datacommons#134

  • API Payload Schema Change: Updated UpdateImportStatusRequest and UpdateImportVersionRequest models to accept lists of imports under an imports parameter instead of single-import fields.
  • Service Endpoint Processing: Refactored the /imports/status and /imports/version route handlers to iterate and update GCS file version/provenance logs and Spanner status records for multiple imports in a single call.
  • Workflow & Helper Updates: Adjusted caller definitions in import-automation-workflow.yaml and import_helper.py to match the new list-based HTTP request structures.
  • Test Coverage: Added new unit test scenarios in app_test.py mock-validating the updated endpoints with lists of multiple imports.
  • Documentation: Updated the API overview table in the ingestion-helper README.md to document the list support.

@dwnoble dwnoble requested review from gmechali and vish-cs June 19, 2026 00:37
@codacy-production

codacy-production Bot commented Jun 19, 2026

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 1 minor

Alerts:
⚠ 1 issue (≤ 0 issues of at least minor severity)

Results:
1 new issue

Category Results
CodeStyle 1 minor

View in Codacy

🟢 Metrics 5 complexity

Metric Results
Complexity 5

View in Codacy

NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the import automation workflow and ingestion helper to support batch processing of multiple imports. The API endpoints and schemas for updating import status and version have been modified to accept lists of imports, and corresponding unit tests and documentation have been updated. The review feedback suggests two key improvements: replacing the 'Z' suffix in next_refresh with '+00:00' to maintain compatibility with Python versions prior to 3.11, and extracting the caller identity retrieval outside of the loop in update_import_version to avoid redundant JWT decoding and verification.

Comment thread pipeline/workflow/ingestion-helper/routes/imports.py
Comment thread pipeline/workflow/ingestion-helper/routes/imports.py

@vish-cs vish-cs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, it has a single status for all the imports. Would it make sense to make status as part of the import item so each import can have its own status?

@vish-cs

vish-cs commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

@vish-cs

vish-cs commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Currently, it has a single status for all the imports. Would it make sense to make status as part of the import item so each import can have its own status?

I am approving the PR to unblock DCP work but it might be helpful to update this.

@gmechali gmechali left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Dan!

@dwnoble

dwnoble commented Jun 22, 2026

Copy link
Copy Markdown
Contributor Author

Currently, it has a single status for all the imports. Would it make sense to make status as part of the import item so each import can have its own status?

Updated!

@dwnoble dwnoble enabled auto-merge (squash) June 23, 2026 08:16
@dwnoble dwnoble merged commit 5ef370e into datacommonsorg:master Jun 23, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants