Skip to content

USHIFT-6973 Add CIS Level2 Smoke Testing #7041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
copejon wants to merge 3 commits into
base: main
Choose a base branch
from
Open

USHIFT-6973 Add CIS Level2 Smoke Testing #7041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c4f29ca
init cis smoketest
copejon Jun 26, 2026
5196251
Add CIS Level 2 smoke test for MicroShift on hardened RHEL 9.8
copejon Jul 13, 2026
6690cdc
Reuse isolated ISO for CIS test and fix ansible SSH disconnect
copejon Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions test/assets/cis/cis-harden.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
---
- name: Apply CIS Level 2 hardening
hosts: localhost
connection: local
become: true
roles:
- role: ansible-role-rhel9-cis
vars:
# MicroShift OVN networking requires IP forwarding
sysctl_net_ipv4_ip_forward: false
sysctl_net_ipv6_conf_all_forwarding: false
# CI requires passwordless sudo
sudo_remove_nopasswd: false
7 changes: 7 additions & 0 deletions test/assets/cis/cis-requirements.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
roles:
- name: ansible-role-rhel9-cis
src: https://github.com/RedHatOfficial/ansible-role-rhel9-cis

@ggiguash ggiguash Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, create another set of tests for RHEL 10.
https://github.com/RedHatOfficial/ansible-role-rhel10-cis

@copejon copejon Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 note that this is going to require an additional iso. CIS hardening can't effectively be deployed against a bootc image. The blocker is that the playbook is written under the assumption of a mutable system and expects changes to persist across reboots. It wouldn't make sense to mutate the playbook (I tried) to fit the tests because it's likely to change over time. Another alternative I tried was to split the hardening process across build and runtime environs, but again that requires customizing the playbook, including reimplementing portions of it as in the scenario script. It's brittle, complex, and not maintainable.

version: "0.1.80"
collections:
- community.general
- ansible.posix
51 changes: 51 additions & 0 deletions test/scenarios/periodics/el98-src@cis-lvl2.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/bin/bash

# Sourced from scenario.sh and uses functions defined there.

scenario_create_vms() {
prepare_kickstart host1 kickstart-liveimg.ks.template ""
launch_vm rhel-9.8-microshift-source-isolated

subscription_manager_register host1

# Install packages needed for CIS hardening and OpenSCAP scanning
run_command_on_vm host1 "sudo dnf install -y openscap-scanner scap-security-guide ansible-core"
run_command_on_vm host1 "sudo ansible-galaxy collection install community.general ansible.posix"

# Copy ansible assets and install the CIS hardening role
copy_file_to_vm host1 "${ROOTDIR}/test/assets/cis/cis-requirements.yml" "/tmp/cis-requirements.yml"
copy_file_to_vm host1 "${ROOTDIR}/test/assets/cis/cis-harden.yml" "/tmp/cis-harden.yml"
run_command_on_vm host1 "sudo ansible-galaxy role install -r /tmp/cis-requirements.yml"

# Apply CIS Level 2 hardening. The playbook is run detached because
# update-crypto-policies restarts sshd mid-run, killing the SSH session.
# The playbook itself uses -c local and survives the sshd restart.
run_command_on_vm host1 "nohup sudo ansible-playbook -c local /tmp/cis-harden.yml &>/tmp/cis-harden.log &"
local -r ip=$(get_vm_property host1 ip)
wait_for_ssh "${ip}"
local attempts=0
while ! run_command_on_vm host1 "grep -q 'PLAY RECAP' /tmp/cis-harden.log" 2>/dev/null ; do
(( attempts++ )) && (( attempts > 180 )) && { echo "Timed out waiting for CIS hardening"; return 1; }
sleep 10
done
if run_command_on_vm host1 "grep -qE 'unreachable=[1-9]|failed=[1-9]' /tmp/cis-harden.log" 2>/dev/null ; then
run_command_on_vm host1 "cat /tmp/cis-harden.log" || true
echo "CIS hardening playbook failed"
return 1
fi

# Reboot to apply hardening changes and wait for the VM to come back
run_command_on_vm host1 "sudo reboot" || true
sleep 10
wait_for_ssh "${ip}"
Comment on lines +38 to +40

@ggiguash ggiguash Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not reliable because we do not guarantee the host / ssh would go down in 10s.
Consider doing reboot in RF code where we have reliable sequences checking boot id.
Alternatively, check for some other scenarios for reboots.


configure_vm_firewall host1
}

scenario_remove_vms() {
remove_vm host1
}

scenario_run_tests() {
run_tests host1 suites/cis/
}
103 changes: 103 additions & 0 deletions test/suites/cis/validate-cis-lvl2.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
*** Settings ***
Documentation Tests related to CIS Level 2 validation on a hardened RHEL system

Resource ../../resources/common.resource
Resource ../../resources/oc.resource
Resource ../../resources/ostree-health.resource
Resource ../../resources/microshift-host.resource
Resource ../../resources/microshift-process.resource
Resource ../../resources/microshift-network.resource
Library Collections
Library String

Suite Setup Setup
Suite Teardown Teardown

Test Tags slow


*** Variables ***
${USHIFT_HOST} ${EMPTY}
${USHIFT_USER} ${EMPTY}
${OSCAP_REPORT_FILE} /tmp/cis-lvl2-report.html
${OSCAP_RESULTS_FILE} /tmp/cis-lvl2-results.xml
${OSCAP_PROFILE} xccdf_org.ssgproject.content_profile_cis
${SCAP_DS_FILE} /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
${CIS_MAX_FAILURES} 16


*** Test Cases ***
OpenSCAP Scan Runs And Produces Report
[Documentation] Run OpenSCAP scan with CIS Level 2 profile and verify it produces output files
Run CIS Level 2 Scan
Verify Remote File Exists With Sudo ${OSCAP_REPORT_FILE}
Verify Remote File Exists With Sudo ${OSCAP_RESULTS_FILE}

CIS Failure Count Within Threshold
[Documentation] Verify the number of CIS failures does not exceed the expected threshold
${fail_count}= Get CIS Failure Count
Log CIS Level 2 failures: ${fail_count} (threshold: ${CIS_MAX_FAILURES})
Should Be True ${fail_count} <= ${CIS_MAX_FAILURES}
... CIS failure count ${fail_count} exceeds threshold ${CIS_MAX_FAILURES}

All Pods Are Running After CIS Hardening
[Documentation] Verify all MicroShift pods are running on the hardened system
All Pods Should Be Running

Smoke Test With Route
[Documentation] Deploy hello-microshift and expose via route to verify networking on hardened system
[Setup] Setup Smoke Test
Wait Until Keyword Succeeds 10x 6s
... Access Hello MicroShift Success ${HTTP_PORT}
[Teardown] Teardown Smoke Test


*** Keywords ***
Setup
[Documentation] Test suite setup
Check Required Env Variables
Login MicroShift Host
Setup Kubeconfig
Start MicroShift
Wait Until Greenboot Health Check Exited

Teardown
[Documentation] Test suite teardown — archive scan artifacts
Run Keyword And Ignore Error
... SSHLibrary.Get File ${OSCAP_REPORT_FILE} ${OUTPUTDIR}/cis-lvl2-report.html
Run Keyword And Ignore Error
... SSHLibrary.Get File ${OSCAP_RESULTS_FILE} ${OUTPUTDIR}/cis-lvl2-results.xml
Logout MicroShift Host

Run CIS Level 2 Scan
[Documentation] Execute the OpenSCAP scan with CIS Level 2 profile
${stdout} ${stderr} ${rc}= Execute Command
... oscap xccdf eval --profile ${OSCAP_PROFILE} --results ${OSCAP_RESULTS_FILE} --report ${OSCAP_REPORT_FILE} ${SCAP_DS_FILE}
... sudo=True
... return_rc=True
... return_stdout=True
... return_stderr=True
Should Be True ${rc} == 0 or ${rc} == 2
... OpenSCAP scan failed with unexpected return code ${rc}: ${stderr}

Get CIS Failure Count
[Documentation] Parse the OpenSCAP results XML and count failures
${stdout} ${stderr} ${rc}= Execute Command
... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0
... sudo=False return_rc=True return_stdout=True return_stderr=True
${fail_count}= Convert To Integer ${stdout.strip()}
RETURN ${fail_count}
Comment on lines +83 to +89

@coderabbitai coderabbitai Bot Jul 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift

Get CIS Failure Count can produce false negatives and errors on zero failures.

Two issues in the grep command:

  1. sudo=False on a root-owned file: Run CIS Level 2 Scan creates the results file with sudo=True, so it's owned by root. On a CIS-hardened system (restrictive umask), the SSH user likely can't read it. When grep fails, || echo 0 outputs "0" — the test passes with zero failures even if the file was unreadable. The Verify Remote File Exists With Sudo keyword in the prior test case confirms these files require sudo.

  2. || echo 0 duplicates output on zero matches: grep -c already prints "0" and exits 1 when there are no matches. || echo 0 then appends another "0", producing stdout "0\n0". Convert To Integer fails on this, erroring the test in the best-case scenario (zero failures).

🐛 Proposed fix 
 Get CIS Failure Count
     [Documentation]    Parse the OpenSCAP results XML and count failures
     ${stdout}    ${stderr}    ${rc}=    Execute Command
-    ...    grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0
-    ...    sudo=False    return_rc=True    return_stdout=True    return_stderr=True
+    ...    grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true
+    ...    sudo=True    return_rc=True    return_stdout=True    return_stderr=True
     ${fail_count}=    Convert To Integer    ${stdout.strip()}
     RETURN    ${fail_count}

sudo=True ensures the root-owned file is readable. || true suppresses grep's non-zero exit on zero matches without adding duplicate output — grep -c already prints the count.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Get CIS Failure Count
[Documentation] Parse the OpenSCAP results XML and count failures
${stdout} ${stderr} ${rc}= Execute Command
... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || echo 0
... sudo=False return_rc=True return_stdout=True return_stderr=True
${fail_count}= Convert To Integer ${stdout.strip()}
RETURN ${fail_count}
Get CIS Failure Count
[Documentation] Parse the OpenSCAP results XML and count failures
${stdout} ${stderr} ${rc}= Execute Command
... grep -c '<result>fail</result>' ${OSCAP_RESULTS_FILE} || true
... sudo=True return_rc=True return_stdout=True return_stderr=True
${fail_count}= Convert To Integer ${stdout.strip()}
RETURN ${fail_count}
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/suites/cis/validate-cis-lvl2.robot` around lines 83 - 89, Update the
grep command in Get CIS Failure Count to run with sudo=True so the root-owned
OSCAP_RESULTS_FILE is readable, and replace “|| echo 0” with “|| true” so zero
matches preserve grep’s single 0 output without causing conversion errors.


Setup Smoke Test
[Documentation] Create hello-microshift pod and expose via route
${ns}= Create Unique Namespace
VAR ${NAMESPACE}= ${ns} scope=TEST
Create Hello MicroShift Pod
Expose Hello MicroShift
Oc Expose svc hello-microshift --hostname hello-microshift.cluster.local -n ${NAMESPACE}

Teardown Smoke Test
[Documentation] Clean up smoke test resources
Run Keyword And Ignore Error Oc Delete route/hello-microshift -n ${NAMESPACE}
Run Keyword And Ignore Error Delete Hello MicroShift Pod And Service
Run Keyword And Ignore Error Remove Namespace ${NAMESPACE}