reduce storagenode reconcile noise during bootstrap

[boddumanohar]

reduce storagenode reconcile noise during bootstrap

2026-06-23T10:05:28Z    INFO    Storage node API not reachable yet, requeueing  {"controller": "storagenode", "controllerGroup": "storage.simplyblock.io", "controllerKind": "StorageNode", "StorageNode": {"name":"simplyblock-node","namespace":"simplyblock"}, "namespace": "simplyblock", "name": "simplyblock-node", "reconcileID": "dc85f71a-6a1d-4511-820c-f7e3c62d0514", "node": "ocp-installer-r6pzk-worker-a-lbhkz", "ip": "10.0.128.28", "error": "node info endpoint not reachable: Get \"https://ocp-installer-r6pzk-worker-a-lbhkz.simplyblock-storage-node-api.simplyblock.svc.cluster.local:5000/snode/info\": dial tcp 10.0.128.28:5000: connect: connection refused"}
2026-06-23T10:05:28Z    INFO    Storage node API not reachable yet, requeueing  {"controller": "storagenode", "controllerGroup": "storage.simplyblock.io", "controllerKind": "StorageNode", "StorageNode": {"name":"simplyblock-node","namespace":"simplyblock"}, "namespace": "simplyblock", "name": "simplyblock-node", "reconcileID": "dc85f71a-6a1d-4511-820c-f7e3c62d0514", "node": "ocp-installer-r6pzk-worker-a-kdrdt", "ip": "10.0.128.27", "error": "node info endpoint not reachable: Get \"https://ocp-installer-r6pzk-worker-a-kdrdt.simplyblock-storage-node-api.simplyblock.svc.cluster.local:5000/snode/info\": dial tcp 10.0.128.27:5000: connect: connection refused"}
2026-06-23T10:05:28Z    INFO    Storage node API not reachable yet, requeueing  {"controller": "storagenode", "controllerGroup": "storage.simplyblock.io", "controllerKind": "StorageNode", "StorageNode": {"name":"simplyblock-node","namespace":"simplyblock"}, "namespace": "simplyblock", "name": "simplyblock-node", "reconcileID": "83dc28e9-fdff-425c-ab8e-67a9c40eed0f", "node": "ocp-installer-r6pzk-worker-a-xrzkz", "ip": "10.0.128.26", "error": "node info endpoint not reachable: Get \"https://ocp-installer-r6pzk-worker-a-xrzkz.simplyblock-storage-node-api.simplyblock.svc.cluster.local:5000/snode/info\": dial tcp 10.0.128.26:5000: connect: connection refused"}

During initial cluster bring-up, "Storage node API not reachable yet"
logs flooded at INFO level from multiple concurrent reconcile loops
triggered by rapid SPDK proxy pod state transitions.

Two fixes:

• Demote both "not reachable yet" log sites to V(1) — these are
  expected transient conditions, not actionable at default verbosity.
• Add a pod phase-change predicate to the SPDK proxy pod watch so
  Update events only trigger a reconcile when the pod phase actually
  changes, not on every annotation/status/probe update. This eliminates
  the multiple-reconcileID pile-up seen within the same second.

---

PR description:

Problem

During initial cluster bootstrap, the storagenode controller emitted
"Storage node API not reachable yet" at INFO level at very high
frequency. Two separate issues compounded each other:

1. The log level was INFO, so every requeue attempt was visible even
   though an unreachable node API during bootstrap is fully expected.

2. More critically: SPDK proxy pods fire many Update events during
   startup (Pending → ContainerCreating → Running, plus readiness probe
   updates, annotation patches, etc.). Each event triggered a full
   reconcile, producing multiple concurrent reconcile loops — visible
   as several different reconcileIDs all logging at the same second —
   even before the 10s RequeueAfter fired.

Fix

• Demote both "not reachable yet" log sites (postStorageNode and
  waitForNodeInfoReachable) from log.Info to log.V(1).Info.
  They remain available at -v=1 for debugging but are suppressed at
  default operator verbosity.

• Add a predicate.Funcs{UpdateFunc: ...} to the SPDK proxy pod watch
  that only passes Update events through when the pod's .Status.Phase
  changes. Create and Delete events are unaffected. This eliminates the
  spurious reconcile storm during bootstrap without missing any
  meaningful pod lifecycle transition.

No behaviour change

The reconcile logic, requeue interval, and error handling are
unchanged. Nodes that genuinely fail to come up are still detected and
requeued on the same schedule.