Skip to content

fix(circleci): skip HTTP 500 errors and apply TimeAfter window on full sync#8956

Merged
klesh merged 1 commit into
apache:mainfrom
jbsmith7741:fix-circleci-8948
Jun 25, 2026
Merged

fix(circleci): skip HTTP 500 errors and apply TimeAfter window on full sync#8956
klesh merged 1 commit into
apache:mainfrom
jbsmith7741:fix-circleci-8948

Conversation

@jbsmith7741

Copy link
Copy Markdown
Contributor

Summary

Two bug fixes for the CircleCI workflow and job collectors:

1. Skip HTTP 500 responses from corrupt CircleCI Server records

CircleCI Server occasionally returns 500 Internal Server Error for pipelines whose internal records are corrupt or stuck (e.g. a pipeline exists in the DB but its /workflow endpoint errors). Previously, this caused ignoreDeletedBuilds to fall through and abort the entire collector subtask. Both the workflow and job collectors now use a renamed ignoreDeletedOrBrokenBuilds hook that treats 500 the same as 404 (data-retention deletion) — a non-fatal, per-item skip so collection continues past broken entries.

Before                                     After
─────────────────────────────────────────  ─────────────────────────────────────────
HTTP 404  →  ErrIgnoreAndContinue (skip)   HTTP 404  →  ErrIgnoreAndContinue (skip)
HTTP 500  →  error propagated, run aborts  HTTP 500  →  ErrIgnoreAndContinue (skip)

2. Apply SyncPolicy.TimeAfter window on full sync

The DB iterator clause that limits which pipelines/workflows trigger downstream API calls previously only activated when isIncremental == true. This meant a full sync with a TimeAfter policy still iterated every historical tool-layer row and issued an API request for each one. The condition is now if createdAfter != nil, so both incremental and windowed full syncs skip rows outside the time window.

The boundary condition also changed from > to >= so the exact bookmark timestamp is not silently dropped.

Before                                     After
─────────────────────────────────────────  ─────────────────────────────────────────
Full sync + TimeAfter  →  all rows iterated,    Full sync + TimeAfter  →  rows before
  API call per row regardless of age              window skipped, boundary inclusive
Incremental            →  rows after bookmark   Incremental            →  unchanged
  (exclusive)

Does this close any open issues?

Closes #8948

Screenshots

N/A — no UI changes.

Other Information

… time range

  * Skip 500 (corrupt CircleCI Server records) alongside 404 in AfterResponse hook so a single bad pipeline does not abort the entire subtask
  * Apply SyncPolicy.TimeAfter to workflow and job DB iterators on full sync to avoid calling the API for every historical tool-layer row

Signed-off-by: Joshua Smith <jbsmith7741@gmail.com>

@klesh klesh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
Thanks for your contribution.

BTW. I would like to know if you are interested in becoming an Apache Committer of the project? I would like to nominate you, please send me email: klesh@apache.org if you are interested. https://community.apache.org/contributors/becomingacommitter.html

@klesh klesh merged commit bcfdfb6 into apache:main Jun 25, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug][CircleCI] collectWorkflows aborts entire sync when pipeline workflow API returns HTTP 500

2 participants