k8s: detect OOM/preemption as structured failure reasons (REMOTE-2111)#100
Draft
seemeroland wants to merge 1 commit into
Draft
k8s: detect OOM/preemption as structured failure reasons (REMOTE-2111)#100seemeroland wants to merge 1 commit into
seemeroland wants to merge 1 commit into
Conversation
- Add TaskFailureReasonPodPreempted metric constant + priming entry. - inspectPodFailure: detect pod preemption via Kubernetes event reason "Preempting" and via pod Status.Reason=="Evicted" with a message containing "preempt" (covers different k8s versions / eviction plugins). - TaskFailedMessage: add FailureReason field (json:"failure_reason") so the server can identify OOM/preemption without parsing the human-readable Message string. - executeTask/sendTaskFailed: propagate the structured failure reason from backendFailureError into the wire message. - userFacingTaskError: return distinct, actionable messages for container_oom and pod_preempted failure reasons instead of the raw error string. Current behavior for context: when a k8s worker pod is OOM-killed or preempted, the cloud agent run fails with no state preservation and no resumption mechanism. These changes surface the specific failure cause in metrics, the user-facing error message, and the server-visible FailureReason field for better observability and timeline display. Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Addresses REMOTE-2111: enterprise customer (IMC) asked how k8s worker pods handle OOM kills and pod preemptions.
Current behavior: when a k8s worker pod is OOM-killed or preempted, the cloud agent run fails with no state preservation, no snapshotting, and no resumption. This PR does not change that (checkpointing is a larger investment), but improves observability so users and operators know why a run failed.
Changes
Preemption detection
inspectPodFailurenow checks for pod preemption via two signals:Reason == "Preempting"(emitted by the scheduler before eviction)Pod.Status.Phase == FailedwithStatus.Reason == "Evicted"and a message containing "preempt" (covers different k8s versions and eviction plugins)pod_preemptedfailure reason (previously fell through as genericjob_failedorcontainer_exit)OOM classification already existed but is now surfaced better
classifyTerminatedReason("OOMKilled")→container_oomwas already tracked in metrics; now also carried through in user-facing messages and the wire protocolStructured
FailureReasoninTaskFailedMessageFailureReason stringfield (JSON:failure_reason) so the server can identify OOM/preemption without parsing the human-readable message stringBetter user-facing error messages
container_oom: "The task container was terminated because it ran out of memory (OOM killed). Consider increasing the runner memory limit or reducing the task's memory usage."pod_preempted: "The task pod was preempted by the Kubernetes scheduler (evicted to make room for higher-priority workloads). The run was not checkpointed — please retry. Consider using higher-priority pods or nodes with sufficient capacity."Testing
go test ./internal/worker/...passesgo build ./...passesConversation: https://staging.warp.dev/conversation/c5421a3c-e5ac-459e-bf33-9c988a87de8a
Run: https://oz.staging.warp.dev/runs/019f4843-caaf-719a-861a-dd6cd9042bba
This PR was generated with Oz.