Stop treating optional signals as universal readiness requirements

[placerda]

Problem

Cockpit, Doctor, and release evidence currently treat optional evaluation and observability capabilities as requirements for every workspace.

The affected signals include:

• multi-turn evaluation coverage, even for single-turn datasets;
• rubric evaluators, even when no rubric was requested;
• trace sampling, based either on observability.trace_sampling.enabled: true or an unverified text field copied from an imported trace;
• trace replay, based either on a manually pasted observability.trace_replay_url or an unverified URL copied from an imported trace.

This creates misleading readiness gaps. More seriously, trace sampling and replay can be reported as ready without AgentOps checking the Foundry configuration, confirming that intelligent sampling ran, or verifying that a replay link is valid.

Optional capabilities that were not requested should be not applicable, not missing. User-authored declarations and source-provided labels or URLs are intent or provenance, not proof.

Expected behavior

Multi-turn

• Treat multi-turn as a property of the configured evaluation dataset, not the project or agent.
• dataset_kind: single-turn means not applicable.
• dataset_kind: multi-turn means applicable.
• For dataset_kind: auto, infer multi-turn only from non-empty messages conversation rows.
• Do not infer a requirement before initialization or without a readable dataset.
• Only applicable multi-turn datasets receive coverage checks and remediation.

Rubrics

• Apply rubric readiness only when agentops.yaml declares rubrics or the active Foundry/azd evaluation recipe declares a rubric evaluator.
• Do not show a rubric item or evidence warning when no rubric intent exists.
• A rubric is gated only when its evaluator emits metrics that are bound to configured thresholds.

Trace sampling

• Remove observability.trace_sampling from the agentops.yaml schema, templates, and documentation.
• Remove the Cockpit sampling readiness card and the Doctor observability.trace_sampling_missing finding.
• Do not list trace sampling as missing or ready in release evidence.
• Preserve agentops eval promote-traces and its trace-to-dataset behavior.
• Source fields such as sampling_policy may remain as unverified lineage metadata, but must not count as readiness evidence.
• Do not add a replacement Foundry query in this issue. Sampling can return to readiness only after AgentOps has a supported, verifiable integration.

Trace replay

• Remove observability.trace_replay_url from the agentops.yaml schema, templates, and documentation.
• Remove the Cockpit Trace replay linked to evidence card and the Doctor observability.trace_replay_missing finding.
• Do not list trace replay as missing or ready in release evidence.
• Replay URLs found in promoted traces may remain as optional provenance links, but must not count as readiness evidence and do not need to be copied into configuration.
• Do not add replacement replay discovery or validation in this issue.

Scheduled evaluations

• Remove the Cockpit Scheduled eval (drift watch) readiness item and its Complete readiness action when no scheduled workflow exists.
• Do not recommend a duplicate CI cron for Foundry agents; Foundry already supports scheduled and continuous recurring evaluations with live traffic or datasets.
• The absence of a scheduled evaluation must not count as incomplete release readiness.
• Existing scheduled CI workflows may still be detected and displayed as informational automation when present.
• Scheduled CI evaluation for external agents or repository-controlled datasets may remain documented as an optional pattern, not a universal requirement.

Apply these changes directly; no compatibility behavior or migration path is required.

Acceptance criteria

• Uninitialized workspaces do not show these optional readiness items.
• Single-turn datasets produce no missing multi-turn item, Doctor finding, or evidence warning.
• Auto datasets are classified from their actual row shape.
• Workspaces without rubric intent produce no rubric readiness item or evidence warning.
• Declared rubrics are ready only when emitted metrics are threshold-gated.
• observability.trace_sampling and observability.trace_replay_url are no longer accepted or generated.
• Cockpit, Doctor, and release evidence make no sampling or replay readiness claim.
• Imported sampling labels and replay URLs are never interpreted as proof.
• Missing scheduled evaluation does not create a readiness item, warning, or action.
• Foundry agents are not instructed to duplicate native recurring evaluations with a CI cron.
• Trace promotion remains available and retains its review-first behavior and optional provenance metadata.
• Tests and documentation reflect the simplified semantics.

Next Actions

• Generate actions only for concrete, actionable readiness gaps.
• Do not create Complete readiness actions for checks that are hidden, not applicable, informational, or muted.
• A cannot verify state must never be presented as proof that the capability is missing. If AgentOps knows a concrete remediation for restoring verification, it may emit a distinctly worded Enable verification action; otherwise it remains informational.
• Before workspace initialization, do not expand absent observability context into one action per readiness check. At most, present a single initialization/onboarding action.
• Preserve actionable Doctor finding remediation independently from readiness actions.
• Use action labels that match the state: reserve Complete readiness for an actual missing requirement, and never use it for uncertainty or optional capabilities.

Additional acceptance criteria:

• _build_next_actions() no longer converts every status other than ok into a mandatory readiness action.
• Uninitialized workspaces do not produce derived observability-readiness actions.
• Hidden, not-applicable, informational, and muted checks produce no Next Action.
• Cannot verify is clearly distinguished from not configured in both readiness and Next Actions.
• Tests cover action generation for required, optional, informational, muted, uninitialized, and cannot-verify states.

[placerda]

Implemented in 818ba35 - scope observability readiness to verifiable signals.

Merged into develop via #460 (merge commit 88a89aa). This issue stays open until develop is promoted to main, where the Closes keyword will fire.