[repository-quality] Repository Quality: Error Message Consistency & Actionability (2026-04-07)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — Actionable Error UX

Analysis Date: 2026-04-07
Focus Area: Error Message Consistency & Actionability
Strategy Type: Custom (first run — no prior history)
Custom Area: Yes — Identified by analyzing the inconsistency between 1,790 error-producing locations and only 13 production uses of FormatErrorWithSuggestions, despite the infrastructure existing to provide structured, actionable error guidance.

---

Executive Summary

This analysis examined error message quality across the gh-aw codebase, focusing on whether users who hit errors during workflow compilation or execution receive actionable guidance to resolve the problem. With 1,790 total error-producing locations spanning pkg/workflow (517), pkg/cli (978), and pkg/parser (253), only ~0.7% use the dedicated FormatErrorWithSuggestions helper that provides structured bullet-point suggestions.

The codebase has a clear inconsistency problem in error message approach. Some modules are excellent (e.g., pkg/workflow/mcp_config_validation.go embeds inline YAML examples and docs URLs in every error). Other modules like pkg/parser/schema_validation.go produce terse, one-liner messages with no guidance. The FormatErrorWithSuggestions helper is available and used by 13 production paths, but the majority of user-facing compilation errors do not use it.

This is a high-impact area for developer experience: when gh aw compile fails, users need to know exactly what to change and where. Inconsistent error UX erodes trust and increases support load.

Full Analysis Report

Focus Area: Error Message Consistency & Actionability

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total error-producing locations	1,790	—
FormatErrorWithSuggestions production uses	13	⚠️ Very low (0.7%)
Errors with %w wrapping	962	✅ Good chaining
Modules with excellent inline examples	1 (mcp_config_validation.go)	⚠️ Isolated
Modules with terse messages only	Several (schema_validation.go, schema_triggers.go)	❌
TODO/FIXME debt	7	✅ Low
Packages without any tests	2 (pkg/tty, pkg/types)	⚠️ Minor

Findings

Strengths

• pkg/workflow/mcp_config_validation.go is exemplary: every error includes an inline YAML snippet, an explanation of the rule, and a docs URL. This is the gold standard for this codebase.
• fmt.Errorf with %w wrapping is widely used (962 uses), ensuring errors chain correctly for debugging.
• FormatErrorWithSuggestions helper exists in pkg/console/render.go and is already used in 13 key user-facing paths (compile-not-found, run errors, log download).
• One validation file (tools_validation.go) embeds actionable descriptions inline, e.g., "Use 'bash: true' (enable all commands), 'bash: false' (disable), or 'bash: [\"cmd1\", \"cmd2\"]' (specific commands). Run 'gh aw fix' to automatically migrate".

Areas for Improvement

• pkg/parser/schema_validation.go: Two terse errors like "field '%s' cannot be used in shared workflows (only allowed in main workflows with 'on' trigger)" — no example showing correct usage. Severity: High

• pkg/parser/schema_triggers.go: Long inline explanation text included directly in fmt.Errorf strings instead of using FormatErrorWithSuggestions. The messages are good but hard to format consistently. Severity: Medium

• pkg/parser/remote_fetch.go: Errors like "invalid workflowspec: must be owner/repo/path[@ref]" — valid format is shown but no example, no docs link. Severity: Medium

• pkg/workflow/compiler_jobs.go: Internal errors like "failed to build main job: %w" bubble up to users without contextual guidance. Severity: Medium

• Inconsistency: Three different patterns used across the codebase:

  1. FormatErrorWithSuggestions (structured bullet points)
  2. Inline example YAML in fmt.Errorf strings (mcp_config_validation style)
  3. Terse error strings with no guidance

  No single standard is enforced.

Detailed Analysis

The core issue is the lack of a project-wide standard for "when and how to add actionable guidance to errors". The mcp_config_validation.go approach (inline YAML + docs URL) is excellent for configuration errors where the fix is a specific YAML change. The FormatErrorWithSuggestions approach is better for runtime/workflow resolution errors where the fix is a CLI command. Terse errors are only appropriate for internal/programming errors not shown to end users.

The impact multiplier is high: gh-aw targets developers writing agentic workflows, a relatively new skill domain. When compilation fails, users need immediate, specific guidance to proceed. Each additional second spent searching docs represents friction that reduces adoption.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Please split these into individual work items for Copilot to process sequentially.

---

Task 1: Add Actionable Suggestions to Parser Shared-Workflow Validation Errors

Priority: High
Estimated Effort: Small
Focus Area: Error Message Actionability

Description:
pkg/parser/schema_validation.go contains two terse validation errors for shared workflows that lack any guidance on how to fix the issue. These are user-facing compilation errors shown when someone incorrectly adds trigger-specific fields to a shared workflow file.

Acceptance Criteria:

• Both errors in schema_validation.go use FormatErrorWithSuggestions or include inline suggestions
• Suggestions include: what shared workflows are, what fields are forbidden, and how to split into separate files if needed
• Error messages reference docs or example of correct shared workflow structure
• Existing tests pass (make test-unit)

Code Region: pkg/parser/schema_validation.go (lines ~39–41)

In `pkg/parser/schema_validation.go`, the two validation errors for shared workflows (around lines 39–41) are terse:

```go
return fmt.Errorf("field '%s' cannot be used in shared workflows (only allowed in main workflows with 'on' trigger)", forbiddenFound[0])
```

Improve these errors to use `console.FormatErrorWithSuggestions` with helpful suggestions such as:
- "Shared workflows cannot define triggers. Remove the '%s' field from this file."
- "If you need a trigger, create a main workflow file with an 'on:' section that imports this shared workflow."
- "Shared workflows are reusable fragments imported via 'imports:' in a main workflow."

Import `github.com/github/gh-aw/pkg/console` if not already imported. Run `make fmt && make test-unit` after making changes.

---

Task 2: Convert Inline Trigger Conflict Errors to Structured Suggestions

Priority: High
Estimated Effort: Small
Focus Area: Error Message Consistency

Description:
pkg/parser/schema_triggers.go contains two verbose fmt.Errorf strings that embed long explanation text inline (lines 63–65). The intent is good but the format is inconsistent with the FormatErrorWithSuggestions pattern used elsewhere. These should be refactored to use structured suggestions.

Acceptance Criteria:

• Both trigger-conflict errors in schema_triggers.go use FormatErrorWithSuggestions
• The main error message is concise (one sentence stating what's wrong)
• Suggestions include actionable steps: remove the conflicting event, or use separate workflows
• Error message clarity is preserved or improved
• Run make fmt && make test-unit after making changes

Code Region: pkg/parser/schema_triggers.go (lines ~63–65)

In `pkg/parser/schema_triggers.go`, refactor the two `fmt.Errorf` calls around lines 63–65 from verbose inline strings to use `console.FormatErrorWithSuggestions`:

Current pattern:
```go
return fmt.Errorf("command trigger cannot be used with '%s' event in the same workflow. Command triggers are designed to respond to slash commands in comments and should not be combined with event-based triggers for issues or pull requests", foundConflicts[0])
```

New pattern:
```go
return errors.New(console.FormatErrorWithSuggestions(
    fmt.Sprintf("command trigger cannot be combined with '%s' event trigger", foundConflicts[0]),
    []string{
        "Command triggers respond to slash commands (e.g., '/agent help') in comments.",
        fmt.Sprintf("Remove the '%s' trigger from this workflow, or move it to a separate workflow file.", foundConflicts[0]),
        "See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows",
    },
))
```

Import `errors`, `github.com/github/gh-aw/pkg/console` as needed. Run `make fmt && make test-unit` after changes.

---

Task 3: Add Docs URL References to Remote Fetch Validation Errors

Priority: Medium
Estimated Effort: Small
Focus Area: Error Message Discoverability

Description:
pkg/parser/remote_fetch.go and pkg/parser/remote_fetch_wasm.go contain several errors for invalid workflow import paths (e.g., "invalid workflowspec: must be owner/repo/path[@ref]") that show the format but no example and no docs URL. These appear when users write incorrect import syntax.

Acceptance Criteria:

• The "invalid workflowspec" error includes a concrete import example like imports: github/gh-aw/.github/aw/shared.md@main
• Security path errors include guidance about where files must be located
• Errors are consistent between remote_fetch.go and remote_fetch_wasm.go
• Run make fmt && make test-unit after changes

Code Region: pkg/parser/remote_fetch.go (line ~270), pkg/parser/remote_fetch_wasm.go

In `pkg/parser/remote_fetch.go` around line 270, improve the import path validation error:

Current:
```go
return "", errors.New("invalid workflowspec: must be owner/repo/path[`@ref`]")
```

Improved (using console.FormatErrorWithSuggestions):
```go
return "", errors.New(console.FormatErrorWithSuggestions(
    "invalid workflow import path: must follow format 'owner/repo/path[`@ref`]'",
    []string{
        "Example: 'github/gh-aw/.github/aw/shared.md@main'",
        "The ref (`@main`, `@v1`.0.0) is optional and defaults to the default branch.",
        "Check the 'imports:' field in your workflow frontmatter.",
    },
))
```

Apply the same improvement to the equivalent error in `remote_fetch_wasm.go`. Import `github.com/github/gh-aw/pkg/console` in both files. Run `make fmt && make test-unit` after changes.

---

Task 4: Add Error Message Style Guide to Contributing Documentation

Priority: Medium
Estimated Effort: Medium
Focus Area: Long-term Consistency

Description:
The codebase has three different styles for adding actionable guidance to errors, and no documented standard. Adding a short style guide to CONTRIBUTING.md or DEVGUIDE.md will ensure future contributors use consistent patterns. The guide should document when to use each pattern with before/after examples.

Acceptance Criteria:

• CONTRIBUTING.md or DEVGUIDE.md has a new section "Error Message Guidelines"
• Section documents 3 patterns with when to use each:
  1. FormatErrorWithSuggestions — for user-facing CLI/workflow errors with bullet suggestions
  2. Inline YAML examples in fmt.Errorf — for configuration validation errors (mcp_config_validation style)
  3. Terse fmt.Errorf — only for internal/programming errors not shown to users
• Each pattern includes a Go code example
• Section references pkg/console/render.go for the helper function
• Documentation is clear and under 60 lines

Code Region: CONTRIBUTING.md or DEVGUIDE.md

Add a new section "## Error Message Guidelines" to `DEVGUIDE.md` documenting the three error patterns used in gh-aw, with before/after examples. Use the content from `pkg/workflow/mcp_config_validation.go` as the model for inline YAML examples, and `pkg/workflow/resolve.go` as the model for `FormatErrorWithSuggestions` usage.

Key guidance to include:
- User-facing errors MUST include either bullet suggestions or inline examples
- Internal errors (wrapping framework failures) may be terse
- Always include a docs URL when one exists in `pkg/constants`
- Use `FormatErrorWithSuggestions` for CLI-level errors (compile, run, log commands)
- Use inline YAML+docs URL pattern for YAML configuration validation errors

No code changes needed — documentation only.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-04-07	actionable-error-ux	First run	Yes	Established baseline; identified 3 error styles, inconsistency across 1,790 error locations

---

🎯 Recommendations

Immediate Actions (This Week)

1. Task 1 — Add suggestions to schema_validation.go shared-workflow errors — Priority: High (30 min)
2. Task 2 — Refactor trigger conflict errors to FormatErrorWithSuggestions — Priority: High (30 min)

Short-term Actions (This Month)

1. Task 3 — Improve remote fetch import path errors — Priority: Medium (1 hr)
2. Task 4 — Document error message style guide in DEVGUIDE.md — Priority: Medium (1 hr)

Long-term Actions (This Quarter)

1. Audit all 1,790 error locations and classify by user-facing vs internal, then add suggestions to all user-facing errors missing them.
2. Consider a linter rule that flags user-facing error strings without a suggestion pattern in key compilation paths.

---

📈 Success Metrics

Track these metrics to measure improvement in Error Message Actionability:

• FormatErrorWithSuggestions coverage: 13 uses → 20+ uses
• Parser validation errors with guidance: ~30% → 80%
• pkg/workflow compilation errors with guidance: ~20% → 60%
• Support questions about "what does this error mean?": qualitative reduction

---

Next Steps

1. Review and prioritize the 4 tasks above
2. Assign Tasks 1 & 2 to Copilot coding agent (high priority, small effort)
3. Track improvements via grep -r FormatErrorWithSuggestions --include="*.go" pkg/ | wc -l
4. Re-evaluate this focus area in 30 days — target 20+ FormatErrorWithSuggestions uses

---

References:

• §24083261096 — Workflow run that generated this report

Generated by Repository Quality Improvement Agent · ● 956.6K · ◷

• expires on Apr 8, 2026, 1:22 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #25307.