multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC by ibm-adarsh · Pull Request #84699 · openshift/release

ibm-adarsh · 2026-09-07T11:59:54Z

OCPBUGS-105501:

Why shifting to 00:30 fixes it

By 00:30 UTC the image-pruner has finished. ci-operator imports base images into a
clean namespace with no concurrent pruner activity → MTO reads the manifest cleanly
Ex: → sets amd64 nodeAffinity → pod always runs on an amd64 node → pull succeeds.

PR #83574 applied this same fix to the ppc64le Power jobs and clean runs were
observed on the next scheduled execution. This PR applies the identical fix to all
s390x / IBM Z libvirt jobs that remained at midnight UTC.

Confirmed Failing Jobs (from OCPBUGS-105501)

Job	OCP	Arch	Recent pass rate
`ocp-e2e-ovn-remote-libvirt-s390x`	4.14	s390x	10% (1/10) — fixed here
`ocp-e2e-ovn-remote-libvirt-s390x`	4.20	s390x	0% — fixed here
`ocp-e2e-ovn-remote-libvirt-multi-z-z`	4.20	s390x	0% — fixed here
+ 32 additional s390x/Z jobs across 4.14–5.1 at midnight UTC		s390x	at risk — fixed here

For context, the equivalent ppc64le jobs fixed by #83574:

Job	OCP	Arch	Pass rate before fix
`ocp-e2e-ovn-remote-s2s-libvirt-ppc64le`	4.20	ppc64le	27.8% (5/18)
`ocp-e2e-ovn-remote-s2s-libvirt-multi-p-p`	4.20	ppc64le	11.1% (2/18)
`ocp-e2e-ovn-remote-s2s-libvirt-ppc64le`	4.14	ppc64le	25.0% (3/12)

What Changed

All changes are cron-only: 0 0 … → 30 0 … UTC.
No job configuration, workflow, cluster profile, or step-registry entry was modified.
Zero ppc64le / Power jobs were touched.

Config file	s390x/Z jobs shifted
`nightly-4.14`	3 — e2e (weekly Fri), image-ecosystem (Jan 12), serial (Jan 14)
`nightly-4.15`	3 — heterogeneous (Jan 9), image-ecosystem (Jan 20), serial (Jan 22)
`nightly-4.16`	3 — heterogeneous (Jan 25), image-ecosystem (Jan 28), serial (Jan 30)
`nightly-4.17`	4 — multi-z-z e2e/agent/serial/image-ecosystem (Feb)
`nightly-4.18`	5 — multi-z-z e2e/agent/serial/image-ecosystem/z-xz (Feb)
`nightly-4.20`	5 — s390x Mon+Wed, multi-z-z Tue+Thu/Sat/Sun, z-xz Tue+Thu
`nightly-4.21`	1 — z-xz Tue+Thu
`nightly-4.22`	1 — z-xz Tue+Thu
`nightly-4.23`	8 — s390x, multi-z-z, z-xz, agent, serial, fips, image-ecosystem, techpreview
`nightly-4.23-upgrade-from-nightly-4.22`	1 — multi-z-z (Mar 24)
`nightly-5.0`	1 — z-xz Tue+Thu
`nightly-5.1`	1 — z-xz Tue+Thu
`openshift-multiarch-main-periodics.yaml`	36 generated entries updated

Total: 37 cron entries shifted across 13 files.

Summary by CodeRabbit

Shifted 37 IBM Z s390x/libvirt cron jobs from 00:00 to 00:30 UTC across OCP 4.14–4.23 and 5.0–5.1 CI configurations.
This prevents overlap with the image-pruner and reduces failures during base-image imports and manifest processing.
No job settings, workflows, cluster profiles, step registries, or ppc64le/Power jobs changed.

The OCP image-pruner CronJob runs at 00:00 UTC (default schedule:"") on all build farm clusters. Any CI job that also starts at 00:00 UTC races with the pruner: partially-written blob layers in the ephemeral ci-op-* namespace can be deleted mid-import, leaving MTO with a broken manifest ("manifest unknown"). MTO then cannot set a nodeAffinity, the pod is scheduled randomly across amd64/aarch64 nodes, and an arm64-node landing produces ImagePullBackOff for the amd64-only image. PR openshift#83574 already shifted the ppc64le Power jobs from 00:00 -> 00:30. This PR applies the same fix to ALL s390x (IBM Z) libvirt jobs that were still at 00:00 UTC across OCP 4.14-4.23 and 5.0-5.1. Jobs changed: 4.14: ocp-e2e-ovn-remote-libvirt-s390x (weekly) ocp-image-ecosystem-ovn-remote-libvirt-s390x (Jan 12) ocp-e2e-serial-ovn-remote-libvirt-s390x (Jan 14) 4.15: ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous (Jan 9) ocp-image-ecosystem-ovn-remote-libvirt-s390x (Jan 20) ocp-e2e-serial-ovn-remote-libvirt-s390x (Jan 22) 4.16: ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous (Jan 25) ocp-image-ecosystem-ovn-remote-libvirt-s390x (Jan 28) ocp-e2e-serial-ovn-remote-libvirt-s390x (Jan 30) 4.17: ocp-e2e-ovn-remote-libvirt-multi-z-z (Feb 5) ocp-e2e-ovn-agent-remote-libvirt-multi-z-z (Feb 13) ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z (Feb 11) ocp-e2e-serial-ovn-remote-libvirt-multi-z-z (Feb 8) 4.18: ocp-e2e-ovn-remote-libvirt-multi-z-z (Feb 16) ocp-e2e-ovn-agent-remote-libvirt-multi-z-z (Feb 24) ocp-e2e-ovn-remote-libvirt-multi-z-xz (Feb 27) ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z (Feb 22) ocp-e2e-serial-ovn-remote-libvirt-multi-z-z (Feb 19) 4.20: ocp-e2e-ovn-remote-libvirt-s390x (Mon+Wed) ocp-e2e-ovn-remote-libvirt-multi-z-z (Tue+Thu) ocp-e2e-ovn-remote-libvirt-multi-z-xz (Tue+Thu) ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z (Sun) ocp-e2e-serial-ovn-remote-libvirt-multi-z-z (Sat) 4.21: ocp-e2e-ovn-remote-libvirt-multi-z-xz (Tue+Thu) 4.22: ocp-e2e-ovn-remote-libvirt-multi-z-xz (Tue+Thu) 4.23: ocp-e2e-ovn-remote-libvirt-s390x, multi-z-z, z-xz, agent, serial, fips, image-ecosystem, techpreview upgrade-from-nightly-4.22 multi-z-z (Mar 24) 5.0: ocp-e2e-ovn-remote-libvirt-multi-z-xz (Tue+Thu) 5.1: ocp-e2e-ovn-remote-libvirt-multi-z-xz (Tue+Thu) No ppc64le/Power jobs were modified. Fixes: OCPBUGS-105501 Signed-off-by: Bob

openshift-ci · 2026-09-07T12:00:16Z

Hi @ibm-adarsh. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

coderabbitai · 2026-09-07T12:00:28Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Team

Run ID: 79d6c159-0e57-4dae-ba5a-0431a1bd8697

📥 Commits

Reviewing files that changed from the base of the PR and between 7212f75 and 951cbdf.

⛔ Files ignored due to path filters (1)

ci-operator/jobs/openshift/multiarch/openshift-multiarch-main-periodics.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (12)

ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.14.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.15.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.16.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.17.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.18.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.20.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.21.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.22.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23-upgrade-from-nightly-4.22.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.23.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.0.yaml
ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-5.1.yaml

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.

Walkthrough

The change updates 36 cron schedules across 12 multiarch CI configuration files. Each affected libvirt test job moves from minute 0 to minute 30 while retaining its existing hour, day, and weekday fields.

Changes

Multiarch CI schedule updates

Layer / File(s)	Summary
Shift libvirt job schedules to minute 30 `ci-operator/config/openshift/multiarch/openshift-multiarch-main__nightly-4.14.yaml`, `...nightly-4.15.yaml`, `...nightly-4.16.yaml`, `...nightly-4.17.yaml`, `...nightly-4.18.yaml`, `...nightly-4.20.yaml`, `...nightly-4.21.yaml`, `...nightly-4.22.yaml`, `...nightly-4.23-upgrade-from-nightly-4.22.yaml`, `...nightly-4.23.yaml`, `...nightly-5.0.yaml`, `...nightly-5.1.yaml`	The affected s390x and multi-architecture libvirt jobs now run 30 minutes later. Their remaining cron schedule fields are unchanged.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 951cb

The affected s390x/libvirt CI jobs will start 30 minutes later while retaining their existing schedules otherwise. The configuration-only change is ready to merge with no identified current-head risk.

Suggested reviewers: neha-dot-yadav

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PASS. The commit changes only `cron:` values from `0 0 ...` to `30 0 ...` in 12 YAML files and the generated periodics file. The complete diff has 72 added and 72 removed lines, and every changed line…
Test Structure And Quality	✅ Passed	PASS: The pull request changes only YAML cron values in 13 CI configuration/generated-job files. The actual diff contains no Ginkgo test source, test blocks, resource setup, waits, or assertions. Ther…
Microshift Test Compatibility	✅ Passed	PASS: The pull request adds no Ginkgo tests and introduces no MicroShift API or feature usage. The commit changes only 13 CI YAML files. The complete `HEAD^..HEAD` diff contains 72 removals and 72 add…
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PASS: The pull request adds no Ginkgo e2e tests. The commit changes only `cron:` values in 12 multiarch config files and one generated periodic-jobs YAML file, from `0 0 ...` to `30 0 ...`. No Go test…
Topology-Aware Scheduling Compatibility	✅ Passed	PASS. The pull request changes only 13 YAML files under ci-operator configuration and generated Prow periodic jobs. The complete diff contains 72 added and 72 deleted lines, and every changed line is …
Ote Binary Stdout Contract	✅ Passed	PASS: The pull request changes only 13 YAML files. The complete parent-to-HEAD diff contains 72 cron-line removals and 72 cron-line additions, with no non-cron changes. No Go or OTE binary source, sui…
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS: The pull request adds no Ginkgo tests and introduces no test code. The parent-to-HEAD diff contains only 13 YAML files, with cron values changed from `0 0 ...` to `30 0 ...`; the generated perio…
No-Weak-Crypto	✅ Passed	PASS. The pull-request diff changes only `cron` values from `0 0 ...` to `30 0 ...` in 12 YAML config files and the generated periodics file. The exact added and removed lines contain no MD5, SHA1, DE…
Container-Privileges	✅ Passed	PASS — The pull request changes only `cron` values from `0 0 ...` to `30 0 ...` in 12 CI configuration files and the generated periodics file. The diff adds no `privileged`, `hostPID`, `hostNetwork`, …
No-Sensitive-Data-In-Logs	✅ Passed	PASS: The pull request changes only cron-minute values from `0` to `30` in 12 configuration files and 36 generated periodic entries. The committed diff adds no logging statements or sensitive values s…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: shifting IBM Z s390x libvirt job schedules from midnight to 00:30 UTC.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci · 2026-09-07T12:00:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ibm-adarsh
Once this PR has been reviewed and has the lgtm label, please assign rvanderp3 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-09-07T12:03:01Z

[REHEARSALNOTIFIER]
@ibm-adarsh: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
periodic-ci-openshift-multiarch-main-nightly-4.18-ocp-e2e-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-5.1-ocp-e2e-ovn-remote-libvirt-multi-z-xz	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.17-ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-upgrade-from-nightly-4.22-ocp-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.17-ocp-e2e-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.16-ocp-e2e-serial-ovn-remote-libvirt-s390x	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.18-ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-fips-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.18-ocp-e2e-serial-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-e2e-ovn-remote-libvirt-s390x	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.16-ocp-image-ecosystem-ovn-remote-libvirt-s390x	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-e2e-ovn-remote-libvirt-multi-z-xz	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-e2e-ovn-agent-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.18-ocp-e2e-ovn-remote-libvirt-multi-z-xz	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.21-ocp-e2e-ovn-remote-libvirt-multi-z-xz	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.15-ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.16-ocp-e2e-ovn-remote-libvirt-s390x-heterogeneous	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.18-ocp-e2e-ovn-agent-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.17-ocp-e2e-serial-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.14-ocp-image-ecosystem-ovn-remote-libvirt-s390x	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.14-ocp-e2e-serial-ovn-remote-libvirt-s390x	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.20-ocp-image-ecosystem-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.20-ocp-e2e-ovn-remote-libvirt-multi-z-xz	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-fips-ovn-techpreview-multi-z-z	N/A	periodic	Ci-operator config changed
periodic-ci-openshift-multiarch-main-nightly-4.23-ocp-e2e-ovn-remote-libvirt-multi-z-z	N/A	periodic	Ci-operator config changed

A total of 36 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here
Prior to this PR being merged, you will need to either run and acknowledge or opt to skip these rehearsals.

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

ibm-adarsh changed the title ~~multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC~~ [WIP] multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC Sep 7, 2026

openshift-ci Bot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 7, 2026

openshift-ci Bot requested review from hamzy and stbenjam September 7, 2026 12:01

ibm-adarsh changed the title ~~[WIP] multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC~~ multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC Sep 7, 2026

openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 7, 2026

Navigation Menu

Uh oh!

multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC - #84699

multiarch: shift IBM Z s390x libvirt midnight jobs to 00:30 UTC#84699
ibm-adarsh wants to merge 1 commit into
openshift:mainfrom
ibm-adarsh:fix-s390x-libvirt-midnight-cron

ibm-adarsh commented Sep 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

openshift-ci Bot commented Sep 7, 2026

Uh oh!

coderabbitai Bot commented Sep 7, 2026 •

edited

Loading

Uh oh!

openshift-ci Bot commented Sep 7, 2026

Uh oh!

openshift-merge-bot Bot commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

ibm-adarsh commented Sep 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why shifting to 00:30 fixes it

Confirmed Failing Jobs (from OCPBUGS-105501)

What Changed

Summary by CodeRabbit

Uh oh!

openshift-ci Bot commented Sep 7, 2026

Uh oh!

coderabbitai Bot commented Sep 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

openshift-ci Bot commented Sep 7, 2026

Uh oh!

openshift-merge-bot Bot commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ibm-adarsh commented Sep 7, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Sep 7, 2026 •

edited

Loading