USHIFT-6973 Add CIS Level2 Smoke Testing#7041
Conversation
Signed-off-by: Jonathan H. Cope <jcope@redhat.com>
Introduce a periodic CI scenario that installs MicroShift on a CIS Level 2 hardened system and validates it functions correctly. The test provisions a VM from the source image-installer ISO, registers it with subscription-manager, installs OpenSCAP and Ansible, applies CIS Level 2 hardening via ansible-role-rhel9-cis, then runs four validation checks: OpenSCAP scan produces a report, CIS failure count stays within threshold, all pods are running, and a smoke test route is accessible. New files: - test/scenarios/periodics/el98-src@cis-lvl2.sh - test/suites/cis/validate-cis-lvl2.robot - test/assets/cis/cis-harden.yml - test/assets/cis/cis-requirements.yml - test/image-blueprints/layer2-presubmit/group1/rhel98-source.image-installer The .image-installer marker enables build_images.sh to produce rhel-9.8-microshift-source.iso, which is required for the liveimg kickstart pattern (mutable RPM-based system needed for post-boot CIS.
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: copejon The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository YAML (base), Central YAML (inherited) Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds a CIS Level 2 test scenario that provisions a VM, applies the pinned RHEL 9 CIS Ansible role, runs OpenSCAP validation, checks MicroShift readiness and networking, and cleans up test resources and artifacts. ChangesCIS Level 2 compliance
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Scenario
participant VM
participant Ansible
participant OpenSCAP
participant MicroShift
Scenario->>VM: Provision and configure
Scenario->>Ansible: Run CIS hardening
Ansible-->>VM: Complete hardening
Scenario->>OpenSCAP: Run Level 2 scan
OpenSCAP-->>Scenario: Report and results
Scenario->>MicroShift: Check pods and route access
MicroShift-->>Scenario: Return validation status
Suggested reviewers: 🚥 Pre-merge checks | ✅ 13 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (13 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/suites/cis/validate-cis-lvl2.robot`:
- Around line 83-89: Update the grep command in Get CIS Failure Count to run
with sudo=True so the root-owned OSCAP_RESULTS_FILE is readable, and replace “||
echo 0” with “|| true” so zero matches preserve grep’s single 0 output without
causing conversion errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 8a8b0b74-4e7f-4bf5-8ca5-67d5fd5b23cf
📒 Files selected for processing (5)
test/assets/cis/cis-harden.ymltest/assets/cis/cis-requirements.ymltest/image-blueprints/layer2-presubmit/group1/rhel98-source.image-installertest/scenarios/periodics/el98-src@cis-lvl2.shtest/suites/cis/validate-cis-lvl2.robot
| Get CIS Failure Count | ||
| [Documentation] Parse the OpenSCAP results XML and count failures | ||
| ${stdout} ${stderr} ${rc}= Execute Command | ||
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0 | ||
| ... sudo=False return_rc=True return_stdout=True return_stderr=True | ||
| ${fail_count}= Convert To Integer ${stdout.strip()} | ||
| RETURN ${fail_count} |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
Get CIS Failure Count can produce false negatives and errors on zero failures.
Two issues in the grep command:
-
sudo=Falseon a root-owned file:Run CIS Level 2 Scancreates the results file withsudo=True, so it's owned by root. On a CIS-hardened system (restrictive umask), the SSH user likely can't read it. When grep fails,|| echo 0outputs"0"— the test passes with zero failures even if the file was unreadable. TheVerify Remote File Exists With Sudokeyword in the prior test case confirms these files require sudo. -
|| echo 0duplicates output on zero matches:grep -calready prints"0"and exits 1 when there are no matches.|| echo 0then appends another"0", producing stdout"0\n0".Convert To Integerfails on this, erroring the test in the best-case scenario (zero failures).
🐛 Proposed fix
Get CIS Failure Count
[Documentation] Parse the OpenSCAP results XML and count failures
${stdout} ${stderr} ${rc}= Execute Command
- ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0
- ... sudo=False return_rc=True return_stdout=True return_stderr=True
+ ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true
+ ... sudo=True return_rc=True return_stdout=True return_stderr=True
${fail_count}= Convert To Integer ${stdout.strip()}
RETURN ${fail_count}sudo=True ensures the root-owned file is readable. || true suppresses grep's non-zero exit on zero matches without adding duplicate output — grep -c already prints the count.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Get CIS Failure Count | |
| [Documentation] Parse the OpenSCAP results XML and count failures | |
| ${stdout} ${stderr} ${rc}= Execute Command | |
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0 | |
| ... sudo=False return_rc=True return_stdout=True return_stderr=True | |
| ${fail_count}= Convert To Integer ${stdout.strip()} | |
| RETURN ${fail_count} | |
| Get CIS Failure Count | |
| [Documentation] Parse the OpenSCAP results XML and count failures | |
| ${stdout} ${stderr} ${rc}= Execute Command | |
| ... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true | |
| ... sudo=True return_rc=True return_stdout=True return_stderr=True | |
| ${fail_count}= Convert To Integer ${stdout.strip()} | |
| RETURN ${fail_count} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@test/suites/cis/validate-cis-lvl2.robot` around lines 83 - 89, Update the
grep command in Get CIS Failure Count to run with sudo=True so the root-owned
OSCAP_RESULTS_FILE is readable, and replace “|| echo 0” with “|| true” so zero
matches preserve grep’s single 0 output without causing conversion errors.
There was a problem hiding this comment.
We should avoid creation of another ISO just for this test.
If we need RPM host, please follow the existing pattern of RPM scenarios
| run_command_on_vm host1 "sudo reboot" || true | ||
| sleep 10 | ||
| local -r ip=$(get_vm_property host1 ip) | ||
| wait_for_ssh "${ip}" |
There was a problem hiding this comment.
This is not reliable because we do not guarantee the host / ssh would go down in 10s.
Consider doing reboot in RF code where we have reliable sequences checking boot id.
Alternatively, check for some other scenarios for reboots.
| @@ -0,0 +1,7 @@ | |||
| roles: | |||
| - name: ansible-role-rhel9-cis | |||
| src: https://github.com/RedHatOfficial/ansible-role-rhel9-cis | |||
There was a problem hiding this comment.
Please, create another set of tests for RHEL 10.
https://github.com/RedHatOfficial/ansible-role-rhel10-cis
There was a problem hiding this comment.
+1 note that this is going to require an additional iso. CIS hardening can't effectively be deployed against a bootc image. The blocker is that the playbook is written under the assumption of a mutable system and expects changes to persist across reboots. It wouldn't make sense to mutate the playbook (I tried) to fit the tests because it's likely to change over time. Another alternative I tried was to split the hardening process across build and runtime environs, but again that requires customizing the playbook, including reimplementing portions of it as in the scenario script. It's brittle, complex, and not maintainable.
Switch the CIS scenario to use the existing rhel-9.8-microshift-source-isolated ISO instead of building a dedicated rhel-9.8-microshift-source ISO. The isolated ISO is already built for other periodic scenarios and is a functional superset. Run the CIS hardening playbook detached (nohup) because update-crypto-policies restarts sshd mid-run, killing the SSH session. Poll for PLAY RECAP in the log to detect completion and check for failures. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@copejon: The following test failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
2 participants
Introduce a periodic CI scenario that installs MicroShift on a
CIS Level 2 hardened system and validates it functions correctly.
The test provisions a VM from the source image-installer ISO,
registers it with subscription-manager, installs OpenSCAP and
Ansible, applies CIS Level 2 hardening via ansible-role-rhel9-cis,
then runs four validation checks: OpenSCAP scan produces a report,
CIS failure count stays within threshold, all pods are running,
and a smoke test route is accessible.
New files:
The .image-installer marker enables build_images.sh to produce
rhel-9.8-microshift-source.iso, which is required for the liveimg
kickstart pattern (mutable RPM-based system needed for post-boot
CIS.
Summary by CodeRabbit
New Features
Tests