Skip to content

No per-job read surface for sealed manual jobs: add ob job history #166

Description

@vishr

The problem

The two job-execution paths have inverted read surfaces: the lower-risk one is
well instrumented and the higher-risk one is not.

A scheduled job has a full per-job read surface:

  • ob schedule history <job> — the newest run records, newest first
  • ob schedule logs <job> [run] — the output of one exact activation
  • ob schedule list — timer state, next elapse, last trigger

A sealed manual job — the path that exists precisely because the job
declares data_effect: migration or destructive — has neither. ob job
carries exactly two subcommands, plan and run (cmd/ob/job.go:57). After
running one, the only way to see what happened is ob audit, which is
journal-wide and unfiltered: one row per invocation across every operation
kind, with -n as the sole control (cmd/ob/commands.go:219). There is no way
to ask "what has catalog-refresh done, and when."

So the job whose failure matters most is the one an operator can least easily
review after the fact.

It is worse than a clean split between the two paths, because a single job can
land on both. Only deploy_lock: pinned forces data_effect: none
(internal/app/validate.go:450); a scheduled job on the default exclusive
lock may declare migration or destructive, and its timer fires it on cron
like any other. What scheduleRun refuses is the operator-initiated run of
such a job (internal/engine/schedule_run.go:57), redirecting it to
ob job plan and ob job run. So that job's unattended firings are visible in
ob schedule history, its hand-triggered runs are not visible anywhere
job-scoped, and nothing shows both in one place — for exactly the jobs where
the interleaving matters most.

Proposed change

Add ob job history <job> — and, if it falls out cheaply, ob job logs <job>
— mirroring the ob schedule equivalents.

This is mostly a read over data that already exists, but not entirely — see
"What is missing" below.
Every job run is already journaled through the canonical writer, and
JobResultEvidence (internal/journal/job_result.go:8) already records the
normalized, redaction-safe result per invocation:

type JobResultEvidence struct {
    SchemaVersion   string
    Changed         bool
    Provider        string
    BeforeRevisions []string
    AfterRevisions  []string
    Digest          string
}

It hangs off the journal record as job_result
(internal/journal/journal.go:68), and the start record already carries the
operator, timestamp, git SHA, approval class, approved-by, and the migration
backup mode with its override operator and reason
(internal/journal/journal.go:152-178).

What is missing

Three fields a review row wants are not journaled today, so this needs a
small write change before the read command:

  • Plan digest. Neither JobPlan.PlanDigest
    (internal/onebox/job_plan.go:45) nor Operation.PlanDigest
    (internal/onebox/operation_types.go:194) ever reaches a Record. The
    journal carries only ApprovalDigest — the grant's digest over a struct that
    contains the plan digest, so it is not recoverable from it. This matters
    most for the interactive ob job run <id> path, which writes neither a plan
    nor a grant to disk (cmd/ob/job.go:223-244): the journal is that run's only
    durable record.
  • Release. Present only as free text, Detail: "release=" + current
    (internal/engine/job.go:85). Parseable, but fragile.
  • Data effect at run time. Not journaled. A break-glass review wants it.

Suggested: add ReleaseID, PlanDigest and DataEffect to journal.Record,
set them in the job start record, and thread PlanDigest through
JobRunRequest. Keep the existing Detail: "release=" for older readers.

Note also that journal.Summarize will not do the reduction — Started,
Operator, StartedAt and Finished are all gated on Phase == "deploy"
(internal/journal/journal.go:485-506) — so this needs a small job-specific
reducer rather than reuse.

Retention

PruneCandidates (internal/journal/journal.go:315-351) puts job_run
journals in a single "auxiliary" window shared with exec, schedule-run,
schedule-pause and service-apply, sized RetainReleases*2 (default 5, so
10 — internal/engine/deploy.go:572, internal/app/defaults.go:25-26), pruned
on every deploy. So a burst of ob exec can evict the break-glass job run
somebody wanted to review, and ob job history would silently have nothing to
show. Either the command's help text states this plainly, or PruneCandidates
grows per-OperationKind windows. Worth deciding with the command rather than
after it. (ob schedule history is unaffected — its depth is journald's.)

New command, not an ob audit --job filter

Settling the alternative this issue originally left open: a new sibling
command, for four reasons.

  1. auditRows deliberately flattens to eleven generic fields
    (internal/engine/audit.go:104-116). Service, JobResult,
    ApprovalClass, MigrationBackup and the release do not survive it. A
    --job filter means adding those to AuditRecord — empty on every non-job
    row — plus a job-specific branch. That is a job reader living inside
    audit's name.
  2. ob audit is documented and cross-referenced as the who-did-what
    invocation table, and is the next command for operation_failed and
    cancelled. Widening its shape has blast radius; a sibling has none.
  3. ob audit reads N+1 round trips, one cat per journal
    (internal/engine/audit.go:66-81). A job reader can use journal.Journals
    in one round trip and skip every id without the -job_run- infix that
    newOperationID embeds (internal/onebox/service.go:60), never parsing a
    deploy journal at all.
  4. Symmetry with ob schedule history (cmd/ob/schedule.go:86-126), same
    finite_envelope output class.

Row shape

Suggested columns, mirroring ob schedule history where the fields allow it:
STARTED OUTCOME DURATION OPERATOR RELEASE EFFECT APPROVAL BACKUP CHANGED OPERATION. Outcome is succeeded | failed | incomplete — a start with no
finish is a crashed run and should say so rather than vanish. The help text
should also say that timer firings of a job which also declares a schedule
live in ob schedule history, not here.

One naming note: ob schedule history's help calls an ob schedule run a
"manual run" (cmd/ob/schedule.go:89), while the code and docs use "manual
job" for when: manual. ob job history should avoid the phrase.

Current workaround

ob audit -n <large> and read past every unrelated record, or
ob audit --output json and filter the journal client-side. Neither is a
per-job view, and both require the operator to know the journal's shape.

Worse: ob audit does not currently render sealed job runs correctly — it
reports them as action job, outcome deployed, with no job name — so the
documented fallback does not actually tell you which job ran. Filed separately;
that bug stands regardless of this issue.

Scope and safety

No change to the one-application, one-host scope. Read-only: no lock, no
fence, nothing written, in line with ob status, ob audit and the
ob schedule read commands. Redaction is already handled upstream —
JobResultEvidence is normalized and redaction-safe by construction, so this
surfaces existing evidence rather than widening what is recorded.

Related: #164 covers the interactive approval prompt on the same ob job path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions