Skip to content

WIP: Testing some new stuff#30871

Closed
dgoodwin wants to merge 3 commits intoopenshift:mainfrom
dgoodwin:coderabbit-test
Closed

WIP: Testing some new stuff#30871
dgoodwin wants to merge 3 commits intoopenshift:mainfrom
dgoodwin:coderabbit-test

Conversation

@dgoodwin
Copy link
Contributor

@dgoodwin dgoodwin commented Mar 12, 2026

This should flag heavily.

Summary by CodeRabbit

  • Chores

    • Added an automated review configuration enabling early-access review features, high-level summaries, auto-replies, and extensive pre-merge checks enforcing deterministic test names, test quality guidelines, and compatibility with MicroShift, single-node OpenShift, IPv6, and disconnected environments.
  • Tests

    • Added a new CLI test suite validating client version output, BuildConfig listing, local HTTP endpoint connectivity, and a minimum node count check.

@coderabbitai
Copy link

coderabbitai bot commented Mar 12, 2026

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

Adds a new automated review policy file .coderabbit.yaml and a new Ginkgo-based CLI test suite test/extended/cli/fauxtests.go with a private randomString() helper and four test cases covering oc version, BuildConfig listing, an HTTP GET, and node count. (50 words)

Changes

Cohort / File(s) Summary
Configuration
.coderabbit.yaml
Adds a comprehensive automated review policy: early access, review_profile: chill, request_changes_workflow: false, enables high_level_summary and auto_review, chat:auto_reply: true, extensive custom_checks enforcing deterministic test names, test quality/structure, MicroShift/SNO/IPv6/disconnected compatibility guidance, and path_filters to exclude vendor/generated files.
Test Suite
test/extended/cli/fauxtests.go
Adds a new Ginkgo test suite [sig-cli] oc fauxtests and a private randomString() helper. Introduces tests: assert oc version contains "Client"; list BuildConfigs in default is non-empty; HTTP GET http://127.0.0.1:8080/foo returns 200; verify oc get nodes -o name yields ≥3 nodes.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes


Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (5 warnings, 2 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality ⚠️ Warning Test file lacks meaningful assertion error messages, timeouts on cluster operations, proper setup/cleanup, and makes environment-specific assumptions (localhost HTTP server, 3+ node cluster) that conflict with SNO compatibility goals. Add meaningful assertion messages to all Expect() calls, implement timeouts on Eventually/Consistently calls and kubectl operations, add BeforeEach/AfterEach blocks for setup/cleanup, and remove or make environment-specific assumptions conditional.
Microshift Test Compatibility ⚠️ Warning Test at lines 50-55 makes explicit multi-node assumption by expecting at least 3 nodes, which violates MicroShift compatibility as a single-node distribution. Add [Skipped:MicroShift] label to the three-node test or guard it with exutil.IsMicroShiftCluster() to skip on MicroShift deployments.
Single Node Openshift (Sno) Test Compatibility ⚠️ Warning The test file contains a node count assertion requiring at least three nodes, which violates Single Node OpenShift (SNO) compatibility since SNO has exactly one node. Modify the node count test to detect SNO clusters using exutil.IsSingleNode() or infrastructure.Status.ControlPlaneTopology and skip the assertion conditionally, or remove the hard requirement entirely.
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning Test contains hardcoded IPv4 localhost address (127.0.0.1) without IPv6 support, failing in IPv6-only environments. Update test to dynamically detect IP family and use appropriate localhost address (127.0.0.1 for IPv4, ::1 for IPv6), or skip on IPv6-only clusters.
Title check ❓ Inconclusive The pull request title 'WIP: Testing some new stuff' is vague and generic, using non-descriptive language that fails to convey meaningful information about the actual changes (CodeRabbit configuration and new test suite). Replace the vague title with a descriptive summary of the main changes, such as 'Add CodeRabbit configuration and CLI test suite' or similar that reflects the actual content.
Stable And Deterministic Test Names ❓ Inconclusive Test files referenced in PR summary are not accessible in current repository state, preventing direct verification of Ginkgo test name stability and determinism. Provide access to test/extended/cli/fauxtests.go with actual Ginkgo test declarations to verify test names contain no dynamic elements.
✅ Passed checks (1 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
📝 Coding Plan
  • Generate coding plan for human review comments

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.11.3)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 12, 2026
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/cli/fauxtests.go`:
- Around line 40-47: The test in g.It ("should be able to contact the foo
endpoint") calls http.Get against a hardcoded 127.0.0.1:8080 which is never
started and breaks IPv6 policy; replace the external call with an in-test HTTP
fixture: create an httptest.NewServer with a handler for "/foo" that returns
http.StatusOK, use server.URL (not a hardcoded host/port) for the request, call
http.Get(server.URL + "/foo"), and defer server.Close(); update the assertions
to use the response from that server instead of the unreachable address.
- Around line 34-37: This test must not rely on cluster-provided BuildConfigs;
before calling oc.Run("get").Args("buildconfigs", "-n", "default").Output()
inside the g.It block, either create a minimal BuildConfig fixture in the test
namespace using oc.Run("create"/"apply") (and ensure cleanup after the test) or
detect that the server supports the buildconfigs API and call framework.Skipf
(or equivalent) to skip the test on unsupported platforms; update the g.It
anonymous test to perform the create-or-skip logic using oc.Run and the same
namespace used in the test rather than assuming objects exist in "default".
- Around line 50-55: The test "should have at least three nodes" hardcodes a
3-node expectation and fails on single-node clusters; change it to first detect
cluster topology or single-replica mode and only assert >=3 nodes when
appropriate. Use the existing oc helper (oc.Run("get")...) to fetch the
Infrastructure resource (e.g.,
oc.Run("get").Args("infrastructures.config.openshift.io", "cluster", "-o",
"jsonpath={.status.controlPlaneTopology}") or similar) or check node count and
skip/adjust the assertion: if topology indicates SingleReplica or node count <3,
either skip the test or assert the expected single-node behavior, otherwise keep
the existing len(lines) >= 3 check; update the g.It block and related variables
(out, lines) accordingly so the test gates on cluster topology instead of
hardcoding 3 nodes.
- Around line 17-20: The test helper randomString() produces non-deterministic
It(...) titles and violates the deterministic-name check; remove the
randomString() function and replace any uses in test descriptions (the It(...)
calls that reference randomString()) with stable, deterministic literal strings
(e.g., a fixed descriptive name) so test names remain constant across runs;
update or simplify any helper references to use the new fixed names and delete
the randomString() function (and its import usage) from
test/extended/cli/fauxtests.go.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d423c97e-9cee-46b4-bed0-b5604eb28272

📥 Commits

Reviewing files that changed from the base of the PR and between 7fd854e and 8f8e013.

📒 Files selected for processing (2)
  • .coderabbit.yaml
  • test/extended/cli/fauxtests.go

Comment on lines +34 to +37
g.It(fmt.Sprintf("should list BuildConfigs in default namespace %s", randomString()), func() {
out, err := oc.Run("get").Args("buildconfigs", "-n", "default").Output()
o.Expect(err).NotTo(o.HaveOccurred())
o.Expect(out).NotTo(o.BeEmpty())
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t depend on pre-existing BuildConfigs in default.

This spec assumes two things that are not stable: that the cluster exposes the OpenShift buildconfigs API, and that default already contains at least one object. Both assumptions are false on some supported jobs, including MicroShift. Create the fixture in the test namespace or gate/skip the test on unsupported platforms instead.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 34 - 37, This test must not rely
on cluster-provided BuildConfigs; before calling
oc.Run("get").Args("buildconfigs", "-n", "default").Output() inside the g.It
block, either create a minimal BuildConfig fixture in the test namespace using
oc.Run("create"/"apply") (and ensure cleanup after the test) or detect that the
server supports the buildconfigs API and call framework.Skipf (or equivalent) to
skip the test on unsupported platforms; update the g.It anonymous test to
perform the create-or-skip logic using oc.Run and the same namespace used in the
test rather than assuming objects exist in "default".

Comment on lines +40 to +47
g.It("should be able to contact the foo endpoint", func() {
host := "127.0.0.1"
port := 8080
url := fmt.Sprintf("http://%s:%d/foo", host, port)
resp, err := http.Get(url)
o.Expect(err).NotTo(o.HaveOccurred())
defer resp.Body.Close()
o.Expect(resp.StatusCode).To(o.Equal(http.StatusOK))
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

This test calls an endpoint that the spec never provisions.

Nothing in this file starts a server on 127.0.0.1:8080, so this will normally fail with connection refused. It also hardcodes IPv4 localhost, which conflicts with the IPv6-compatibility policy added in this PR. If the goal is a local-process check, use an in-test fixture; if the goal is cluster networking, create and probe a cluster-backed endpoint instead.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 40 - 47, The test in g.It
("should be able to contact the foo endpoint") calls http.Get against a
hardcoded 127.0.0.1:8080 which is never started and breaks IPv6 policy; replace
the external call with an in-test HTTP fixture: create an httptest.NewServer
with a handler for "/foo" that returns http.StatusOK, use server.URL (not a
hardcoded host/port) for the request, call http.Get(server.URL + "/foo"), and
defer server.Close(); update the assertions to use the response from that server
instead of the unreachable address.

Comment on lines +50 to +55
g.It("should have at least three nodes", func() {
out, err := oc.Run("get").Args("nodes", "-o", "name").Output()
o.Expect(err).NotTo(o.HaveOccurred())
lines := strings.Split(strings.TrimSpace(out), "\n")
o.Expect(len(lines)).To(o.BeNumerically(">=", 3), "expected at least 3 nodes, got %d", len(lines))
})
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Avoid hardcoding a 3-node requirement.

This spec bakes in a multi-node cluster assumption and will fail on Single Node OpenShift and MicroShift, both called out in the new pre-merge checks. Gate it on topology or change the assertion to something that is valid regardless of node count.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 50 - 55, The test "should have
at least three nodes" hardcodes a 3-node expectation and fails on single-node
clusters; change it to first detect cluster topology or single-replica mode and
only assert >=3 nodes when appropriate. Use the existing oc helper
(oc.Run("get")...) to fetch the Infrastructure resource (e.g.,
oc.Run("get").Args("infrastructures.config.openshift.io", "cluster", "-o",
"jsonpath={.status.controlPlaneTopology}") or similar) or check node count and
skip/adjust the assertion: if topology indicates SingleReplica or node count <3,
either skip the test or assert the expected single-node behavior, otherwise keep
the existing len(lines) >= 3 check; update the g.It block and related variables
(out, lines) accordingly so the test gates on cluster topology instead of
hardcoding 3 nodes.

@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci bot requested review from p0lyn0mial and sjenning March 12, 2026 16:54
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 12, 2026

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 12, 2026
@dgoodwin
Copy link
Contributor Author

@coderabbitai full review

@coderabbitai
Copy link

coderabbitai bot commented Mar 13, 2026

✅ Actions performed

Full review triggered.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (4)
test/extended/cli/fauxtests.go (4)

17-28: ⚠️ Potential issue | 🟠 Major

Remove randomized spec naming to keep test IDs deterministic.

Line 28 builds the test name with randomString(), making the spec name different on every run.

Proposed fix
-import (
-	"crypto/rand"
-	"encoding/hex"
-	"fmt"
+import (
+	"fmt"
 	"net/http"
 	"strings"
@@
-func randomString() string {
-	b := make([]byte, 8)
-	_, _ = rand.Read(b)
-	return hex.EncodeToString(b)
-}
-
 var _ = g.Describe("[sig-cli] oc fauxtests", func() {
@@
-	g.It(fmt.Sprintf("should pass the %s faux test", randomString()), func() {
+	g.It("should report client version", func() {
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 17 - 28, The test spec name is
non-deterministic because g.It uses randomString() to build the description;
remove the randomized suffix and make the spec name deterministic by replacing
the fmt.Sprintf("should pass the %s faux test", randomString()) call with a
constant or stable string (e.g., "should pass the faux test"); also remove or
keep randomString() only if used elsewhere, and update references in this file
(randomString, g.It) so test IDs remain stable across runs.

40-47: ⚠️ Potential issue | 🔴 Critical

Replace hardcoded localhost call with an in-test HTTP fixture.

Lines 41-44 call 127.0.0.1:8080 without creating a server, so this spec is expected to fail in CI.

Proposed fix
 import (
 	"fmt"
 	"net/http"
+	"net/http/httptest"
 	"strings"
@@
 	g.It("should be able to contact the foo endpoint", func() {
-		host := "127.0.0.1"
-		port := 8080
-		url := fmt.Sprintf("http://%s:%d/foo", host, port)
-		resp, err := http.Get(url)
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.URL.Path != "/foo" {
+				w.WriteHeader(http.StatusNotFound)
+				return
+			}
+			w.WriteHeader(http.StatusOK)
+		}))
+		defer srv.Close()
+
+		resp, err := http.Get(srv.URL + "/foo")
 		o.Expect(err).NotTo(o.HaveOccurred())
 		defer resp.Body.Close()
 		o.Expect(resp.StatusCode).To(o.Equal(http.StatusOK))
 	})
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 40 - 47, The test currently
calls a hardcoded host/port (127.0.0.1:8080) without starting a server; replace
that with an in-test HTTP fixture by creating an httptest.Server that handles
the "/foo" path and returns http.StatusOK, use server.URL+"/foo" as the request
URL, call http.Get against that URL (keeping resp and err checks), and ensure
you defer server.Close() and resp.Body.Close(); update imports to include
net/http/httptest as needed.

34-37: ⚠️ Potential issue | 🟠 Major

Don’t rely on pre-existing BuildConfig objects in default.

Lines 35-37 assume both API availability and existing resources in default, which is not stable across supported jobs.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 34 - 37, The test block g.It
currently assumes BuildConfig API and resources exist by calling
oc.Run("get").Args("buildconfigs", "-n", "default").Output(); instead, make the
test create its own BuildConfig in an isolated namespace (or create a temporary
namespace) before calling oc.Run("get") and then delete it as cleanup.
Concretely, update the g.It test to: create/ensure a namespace, use
oc.Run("create").Args(...) or oc.Run("apply").Args(...) to create a minimal
BuildConfig resource, call oc.Run("get").Args("buildconfigs", "-n",
<testNS>).Output() to assert presence, and remove the resource/namespace in
AfterEach to avoid relying on pre-existing state.

50-55: ⚠️ Potential issue | 🟠 Major

Avoid hardcoding a 3-node requirement.

Lines 54-55 enforce >= 3 nodes unconditionally, which fails on valid single-node topologies.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/cli/fauxtests.go` around lines 50 - 55, The test "should have
at least three nodes" hardcodes a minimum of 3 and should be made configurable;
modify the test in the It block (the function containing
oc.Run("get").Args("nodes", "-o", "name").Output(), variables out/err, lines and
the o.Expect(len(lines)).To(...)) to compute the required minimum from a
configurable source (for example read an env var like TEST_MIN_NODES with a
sensible default of 1) and parse it to an int (handle parse errors by failing
the test), then replace the literal 3 with that parsed minNodes value in the
o.Expect assertion so single-node topologies pass by default but tests can
require more nodes via configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@test/extended/cli/fauxtests.go`:
- Around line 17-28: The test spec name is non-deterministic because g.It uses
randomString() to build the description; remove the randomized suffix and make
the spec name deterministic by replacing the fmt.Sprintf("should pass the %s
faux test", randomString()) call with a constant or stable string (e.g., "should
pass the faux test"); also remove or keep randomString() only if used elsewhere,
and update references in this file (randomString, g.It) so test IDs remain
stable across runs.
- Around line 40-47: The test currently calls a hardcoded host/port
(127.0.0.1:8080) without starting a server; replace that with an in-test HTTP
fixture by creating an httptest.Server that handles the "/foo" path and returns
http.StatusOK, use server.URL+"/foo" as the request URL, call http.Get against
that URL (keeping resp and err checks), and ensure you defer server.Close() and
resp.Body.Close(); update imports to include net/http/httptest as needed.
- Around line 34-37: The test block g.It currently assumes BuildConfig API and
resources exist by calling oc.Run("get").Args("buildconfigs", "-n",
"default").Output(); instead, make the test create its own BuildConfig in an
isolated namespace (or create a temporary namespace) before calling
oc.Run("get") and then delete it as cleanup. Concretely, update the g.It test
to: create/ensure a namespace, use oc.Run("create").Args(...) or
oc.Run("apply").Args(...) to create a minimal BuildConfig resource, call
oc.Run("get").Args("buildconfigs", "-n", <testNS>).Output() to assert presence,
and remove the resource/namespace in AfterEach to avoid relying on pre-existing
state.
- Around line 50-55: The test "should have at least three nodes" hardcodes a
minimum of 3 and should be made configurable; modify the test in the It block
(the function containing oc.Run("get").Args("nodes", "-o", "name").Output(),
variables out/err, lines and the o.Expect(len(lines)).To(...)) to compute the
required minimum from a configurable source (for example read an env var like
TEST_MIN_NODES with a sensible default of 1) and parse it to an int (handle
parse errors by failing the test), then replace the literal 3 with that parsed
minNodes value in the o.Expect assertion so single-node topologies pass by
default but tests can require more nodes via configuration.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5c61a4c7-fabb-427a-a363-84323443c18d

📥 Commits

Reviewing files that changed from the base of the PR and between 8f8e013 and da2a0be.

📒 Files selected for processing (1)
  • test/extended/cli/fauxtests.go

@dgoodwin
Copy link
Contributor Author

/close

@openshift-ci openshift-ci bot closed this Mar 13, 2026
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 13, 2026

@dgoodwin: Closed this PR.

Details

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 13, 2026

@dgoodwin: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/lint da2a0be link true /test lint

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants