Skip to content

CANARY_POLICY.md

Latest commit

 

History

History
657 lines (442 loc) · 16.1 KB

CANARY_POLICY.md

File metadata and controls

657 lines (442 loc) · 16.1 KB

CANARY Policy (Project-Wide)

Version: 2.0 Effective Date: 2025-10-18 Scope: All code in this repository

Purpose

Make every feature claim searchable, verifiable, and traceable by linking requirements → code → tests → docs through embedded CANARY tokens.

Policy Statement

All implemented features MUST be tracked with CANARY tokens. This enables:

  • Automated verification of requirement completion
  • Traceability from specification to implementation
  • Prevention of overclaiming in progress reports
  • Evidence-based status tracking (TESTED vs. claimed-tested)

Token Placement Requirements

Where to Place Tokens

  1. Primary implementation file - At the point where feature is implemented
  2. Test files - Where tests for the feature are defined
  3. Specification files - In the spec.md for planning (STATUS=STUB)

Where NOT to Place Tokens

  • Build configuration files (Makefile, go.mod, etc.)
  • Autogenerated code
  • Third-party dependencies
  • Example/demo code (unless it's a documented feature)

Token Format (Required)

CANARY: REQ=<req-id>; FEATURE="<name>"; ASPECT=<aspect>; STATUS=<status>; [OPTIONAL]; UPDATED=<yyyy-mm-dd>

Required Fields

  • REQ - Requirement ID (format: CBIN-###)
  • FEATURE - Feature name (CamelCase, quoted)
  • ASPECT - Architecture layer
  • STATUS - Implementation state
  • UPDATED - Last modification date

Optional Fields (Strongly Recommended)

  • TEST - Test function names (required for STATUS=TESTED)
  • BENCH - Benchmark function names (required for STATUS=BENCHED)
  • OWNER - Team or person responsible
  • DOC - Documentation reference (type:path)
  • DOC_HASH - Documentation hash for currency tracking

Valid Field Values

ASPECT (Architectural Layer)

Core Layers:

  • API - Public interfaces, exported functions
  • CLI - Command-line interfaces, terminal UI
  • Engine - Core algorithms, business logic
  • Storage - Database, persistence, repositories

Specialized Layers:

  • Security - Authentication, authorization, encryption
  • Wire - Serialization, protocols, networking
  • Planner - Planning and scheduling algorithms
  • Docs - Documentation files

Data Handling:

  • Decode - Deserialization, unmarshaling
  • Encode - Serialization, marshaling
  • RoundTrip - Full encode/decode cycles

Supporting:

  • Bench - Performance benchmarks
  • FrontEnd - User interface components
  • Dist - Distribution, deployment, packaging

STATUS (Implementation State)

Progression Path:

MISSING → STUB → IMPL → TESTED → BENCHED → (REMOVED)

  1. MISSING - Planned but no code exists

    • Use in spec.md before implementation
    • Indicates requirement exists but not started

  2. STUB - Placeholder code exists

    • Structure is present but not functional
    • Returns placeholder values or errors
    • No tests required yet

  3. IMPL - Implementation exists but untested

    • Code is functional
    • No TEST= field present
    • WARNING: Cannot satisfy dependencies
    • WARNING: Cannot be claimed in GAP_ANALYSIS.md

  4. TESTED - Fully tested with passing tests

    • Requires TEST= field with test names
    • Tests must exist and pass
    • REQUIRED to satisfy dependencies
    • REQUIRED to claim in GAP_ANALYSIS.md

  5. BENCHED - Tested and performance benchmarked

    • Requires both TEST= and BENCH= fields
    • Benchmarks must exist and run
    • Indicates production-ready quality

  6. REMOVED - Feature deprecated or removed

    • Keep token for historical tracking
    • Mark old features that no longer exist

Requirement ID Format

Format: CBIN-###

Rules:

  • Prefix must be CBIN- (uppercase)
  • Number must be 3 digits, zero-padded
  • Sequential allocation recommended

Examples:

  • CBIN-001 - First requirement
  • CBIN-105 - 105th requirement
  • CBIN-147 - Specification Dependencies

Legacy Tokens & Normalization

To preserve historical data, the scanner accepts certain legacy token variants and normalizes them to the canonical form.

Accepted Legacy Forms

  1. ID-Only Segments inside a CANARY line – Older tokens sometimes included a bare requirement identifier (e.g. REQ-1 or CBIN-5) alongside modern key/value pairs.
  2. Non‑padded numeric IDs – Forms like CBIN-5, CBIN-42, REQ-7, REQ-12, or namespaced forms such as REQ-GQL-4.

Normalization Rules (Automated)

The scanner normalizes legacy forms before reporting:

Input Normalized
CBIN-5 CBIN-005
CBIN-42 CBIN-042
REQ-7 REQ-007
REQ-12 REQ-012
REQ-GQL-4 REQ-GQL-004

All numeric suffixes are zero‑padded to three digits. Namespaced legacy IDs (REQ-<NAMESPACE>-<n>) preserve their namespace and only pad the final numeric segment.

Precedence When Both Appear

If a CANARY line contains both a legacy ID‑only segment and a canonical REQ= key/value pair, the value of REQ= takes precedence for the requirement identifier.

Rationale

This approach allows incremental migration without breaking historical references, while ensuring reporting, verification, and status summaries remain deterministic and sortable.

Recommendation

Prefer the canonical form going forward:

// Example canonical token after normalization
// (avoid adding bare legacy IDs in new code)
CANARY: REQ=CBIN-005; FEATURE="Parser"; ASPECT=Engine; STATUS=IMPL; UPDATED=2025-10-18

Legacy bare IDs should not be added to new tokens; they are supported only for backward compatibility.

Feature Naming Conventions

Format: CamelCase, quoted, descriptive

Good Examples:

FEATURE="PasswordHasher"
FEATURE="DependencyParser"
FEATURE="FuzzySearch"
FEATURE="CircularDetection"

Bad Examples:

FEATURE="Utils"           # Too generic
FEATURE="Helper1"         # Non-descriptive
FEATURE="foo_bar"         # Use CamelCase, not snake_case
FEATURE="password-hash"   # Use CamelCase, not kebab-case

Guidelines:

  • Describe WHAT the feature does, not HOW
  • Keep under 40 characters
  • Use domain language
  • Be specific enough to search for

Test Naming Convention

Format: Test[CANARY_CBIN_<###>_]<DescriptiveName>

Examples:

// Full CANARY naming (recommended for primary tests)
func TestCANARY_CBIN_147_Engine_ParseFullDependency(t *testing.T)
func TestCANARY_CBIN_147_Engine_ParsePartialFeatures(t *testing.T)

// Short naming (acceptable for sub-features)
func TestParseDependencies_FullDependency(t *testing.T)
func TestParseDependencies_PartialFeatures(t *testing.T)

In Token:

TEST=TestCANARY_CBIN_147_Engine_ParseFullDependency
TEST=TestParseDependencies_FullDependency,TestParseDependencies_PartialFeatures

Benchmark Naming Convention

Format: Benchmark[CANARY_CBIN_<###>_]<DescriptiveName>

Examples:

// Full CANARY naming (recommended)
func BenchmarkCANARY_CBIN_147_Engine_CircularDetection(b *testing.B)

// Short naming (acceptable)
func BenchmarkValidateDependencies(b *testing.B)

Token Placement Examples

Implementation File

// internal/specs/parser_dependency.go

package specs

import "io"

// CANARY: REQ=CBIN-147; FEATURE="DependencyParser"; ASPECT=Engine; STATUS=TESTED; TEST=TestParseDependencies_FullDependency,TestParseDependencies_PartialFeatures; OWNER=specs; UPDATED=2025-10-18
func ParseDependencies(sourceReqID string, reader io.Reader) ([]Dependency, error) {
    // Implementation
}

Test File

// internal/specs/parser_dependency_test.go

package specs

import "testing"

// CANARY: REQ=CBIN-147; FEATURE="DependencyParser"; ASPECT=Engine; STATUS=TESTED; TEST=TestParseDependencies_FullDependency; UPDATED=2025-10-18
func TestParseDependencies_FullDependency(t *testing.T) {
    // Test implementation
}

// CANARY: REQ=CBIN-147; FEATURE="DependencyParser"; ASPECT=Engine; STATUS=TESTED; TEST=TestParseDependencies_PartialFeatures; UPDATED=2025-10-18
func TestParseDependencies_PartialFeatures(t *testing.T) {
    // Test implementation
}

Benchmark File

// internal/specs/benchmark_test.go

package specs

import "testing"

// CANARY: REQ=CBIN-147; FEATURE="DependencyValidator"; ASPECT=Engine; STATUS=BENCHED; TEST=TestValidateDependencies_ValidGraph; BENCH=BenchmarkCANARY_CBIN_147_Engine_CircularDetection; UPDATED=2025-10-18
func BenchmarkCANARY_CBIN_147_Engine_CircularDetection(b *testing.B) {
    // Benchmark implementation
}

Specification File (Planning)

<!-- .canary/specs/CBIN-147-specification-dependencies/spec.md -->

<!-- CANARY: REQ=CBIN-147; FEATURE="DependencyParser"; ASPECT=Engine; STATUS=STUB; UPDATED=2025-10-18 -->

**Feature 1: Dependency Parser**

- [ ] Parse full dependencies (CBIN-XXX format)
- [ ] Parse partial feature dependencies (CBIN-XXX:Feature1,Feature2)
- [ ] Parse partial aspect dependencies (CBIN-XXX:AspectName)

Verification Commands

Finding Tokens

Find all CANARY tokens:

rg -n "CANARY:\s*REQ=" src internal cmd tools

Find tokens for specific requirement:

rg -n "REQ=CBIN-147" .

Find test functions:

rg -n "TestCANARY_CBIN_" .

Find benchmark functions:

rg -n "BenchmarkCANARY_CBIN_" .

Using CANARY CLI

# Show all tokens for a requirement
canary show CBIN-147

# List implementation files
canary files CBIN-147

# Check progress
canary status CBIN-147

# Search by pattern
canary grep "DependencyParser"

# Verify claims
canary scan --verify GAP_ANALYSIS.md --strict

Update Requirements

When to Update UPDATED Field

Always update when:

  • Modifying implementation code
  • Changing STATUS value
  • Adding/removing TEST or BENCH fields
  • Refactoring that affects behavior

Example:

// Before change (2025-10-01)
// CANARY: REQ=CBIN-147; FEATURE="Parser"; ASPECT=Engine; STATUS=TESTED; TEST=TestParser; UPDATED=2025-10-01

// After modification (2025-10-18)
// CANARY: REQ=CBIN-147; FEATURE="Parser"; ASPECT=Engine; STATUS=TESTED; TEST=TestParser; UPDATED=2025-10-18

Staleness Threshold

Rule: TESTED and BENCHED tokens must have UPDATED within 30 days

Automated Update:

canary scan --update-stale

Effect:

  • Finds all TESTED/BENCHED tokens with UPDATED >30 days ago
  • Rewrites tokens in-place with current date
  • Reports count of updated tokens

Claiming Requirements

GAP_ANALYSIS.md Format

Claim Requirements:

# Requirements Gap Analysis

## Claimed Requirements

✅ CBIN-101 - Scanner Core
✅ CBIN-102 - Verify Gate
✅ CBIN-147 - Specification Dependencies

Claim Requirements

  1. At least one token must be TESTED or BENCHED

    • IMPL is insufficient
    • STUB cannot be claimed

  2. Run verification before committing:

    canary scan --verify GAP_ANALYSIS.md --strict

  3. Exit code 2 indicates failure:

    CANARY_VERIFY_FAIL REQ=CBIN-XXX reason=claimed_but_not_TESTED_OR_BENCHED

Dependency Management

Declaration (in spec.md)

## Dependencies

### Full Dependencies (entire requirement must be complete)

- CBIN-146 (Multi-Project Support - required for token namespacing)

### Partial Dependencies (specific features/aspects required)

- CBIN-140:GapRepository,GapService (only gap storage features needed)
- CBIN-133:Engine (only Engine aspect required)

Satisfaction Rules

Dependency is satisfied when:

  1. Full: ALL features in target requirement are TESTED or BENCHED
  2. Partial Features: Specified features are TESTED or BENCHED
  3. Partial Aspect: All features with specified aspect are TESTED or BENCHED

IMPL is insufficient to satisfy dependencies!

Commands

# Check if dependencies satisfied
canary deps check CBIN-147

# Show dependency tree
canary deps graph CBIN-147 --status

# What depends on this?
canary deps reverse CBIN-146

# Detect circular dependencies
canary deps validate

Documentation Tracking

DOC Field Usage

Format: DOC=<type>:<path>

Types:

  • user - User-facing documentation
  • api - API reference
  • arch - Architecture docs
  • dev - Developer docs

Example:

// CANARY: REQ=CBIN-147; FEATURE="DependencyParser"; ASPECT=Engine; STATUS=TESTED;
// TEST=TestParseDependencies; DOC=user:docs/user/dependency-guide.md;
// DOC_HASH=a3f5b8c2e1d4a6f9; UPDATED=2025-10-18

Hash Calculation

# Calculate hash
sha256sum docs/user/dependency-guide.md | cut -c1-16

# Or use CANARY
canary doc update --req CBIN-147 --feature DependencyParser

Currency Checking

# Check specific feature
canary doc status CBIN-147 DependencyParser
# Output: DOC_CURRENT or DOC_STALE

# Check all features
canary doc report --show-undocumented

Compliance Checklist

Before Committing Code

  • All implemented features have CANARY tokens
  • STATUS accurately reflects implementation state
  • TESTED features have TEST= field with actual test names
  • BENCHED features have both TEST= and BENCH= fields
  • UPDATED field shows current date (if code was modified)
  • Tokens use correct ASPECT for their location
  • Feature names are descriptive (not generic like "Utils")

Before Claiming in GAP_ANALYSIS.md

  • At least one token for requirement is TESTED or BENCHED
  • All tests pass: go test ./...
  • Verification succeeds: canary scan --verify GAP_ANALYSIS.md --strict
  • Dependencies are satisfied: canary deps check <req-id>

For TESTED Status

  • TEST= field contains actual test function names
  • Tests exist in codebase
  • Tests pass: go test ./...
  • Tests cover primary functionality
  • Test names follow naming convention

For BENCHED Status

  • Both TEST= and BENCH= fields present
  • Benchmarks exist in codebase
  • Benchmarks run: go test -bench=. ./...
  • Benchmark names follow naming convention

Enforcement

Automated Checks

Pre-commit hook:

#!/bin/bash
canary scan --verify GAP_ANALYSIS.md --strict
if [ $? -ne 0 ]; then
    echo "CANARY verification failed!"
    exit 1
fi

CI Pipeline:

- name: Verify CANARY claims
  run: |
    canary scan --verify GAP_ANALYSIS.md --strict
    canary deps validate

Manual Review

Code reviewers should verify:

  1. New features have CANARY tokens
  2. STATUS matches implementation reality
  3. Tests exist for TESTED status
  4. UPDATED field is current
  5. GAP_ANALYSIS.md claims are backed by tokens

Exemptions

The following code does NOT require CANARY tokens:

  1. Third-party dependencies - Code in vendor/ or node_modules/
  2. Generated code - Output from code generators
  3. Build configuration - Makefile, go.mod, package.json, etc.
  4. Documentation - README.md, docs/ (unless it's a tracked feature)
  5. Test fixtures - Test data files
  6. Examples - Example/demo code (unless documented as a feature)

Policy Updates

This policy may be updated to:

  • Add new ASPECT values for project-specific needs
  • Adjust staleness threshold (currently 30 days)
  • Add project-specific requirements

Updates must be:

  1. Documented in this file
  2. Communicated to team
  3. Applied retroactively with canary scan --update-stale

Quick Reference

Minimal Token

CANARY: REQ=CBIN-001; FEATURE="FeatureName"; ASPECT=API; STATUS=STUB; UPDATED=2025-10-18

Tested Token

CANARY: REQ=CBIN-001; FEATURE="FeatureName"; ASPECT=API; STATUS=TESTED; TEST=TestFeatureName; UPDATED=2025-10-18

Benched Token

CANARY: REQ=CBIN-001; FEATURE="FeatureName"; ASPECT=API; STATUS=BENCHED; TEST=TestFeatureName; BENCH=BenchmarkFeatureName; UPDATED=2025-10-18

Documented Token

CANARY: REQ=CBIN-001; FEATURE="FeatureName"; ASPECT=API; STATUS=TESTED; TEST=TestFeatureName; DOC=user:docs/user/guide.md; DOC_HASH=a3f5b8c2; UPDATED=2025-10-18

Related Documentation

Policy Owner: Project Team Last Reviewed: 2025-10-18 Next Review: 2025-11-18