Upgrade version to 1.12.0

[jdetter]

Description of Changes

Version upgrade to v1.12.0.

API and ABI breaking changes

None

Expected complexity level and risk

1 - this is just a version upgrade

Testing

The testsuite failures are fixed by #4120

• License has been properly updated including version number and date
• CI passes

[jdetter]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	26/27	91.7% ⬆️ +17.4%
Rust	rustdoc_json	schema	22/34	62.0% ⬇️ -13.3%
Rust	rustdoc_json	total	48/61	78.2% ⬆️ +3.4%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	8/34	20.5% ⬆️ +8.0%
Rust	docs	total	13/61	15.4% ⬆️ +3.6%
C#	docs	basics	27/27	100.0% ⬆️ +8.3%
C#	docs	schema	22/34	63.7%
C#	docs	total	49/61	83.5% ⬆️ +4.5%

Compared against master branch baseline

_{Generated at: 2026-01-25T02:39:25.598Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 35

---

SpacetimeDB Benchmark Test Failure Analysis

Summary of Analysis

This document organizes the test failures by language (Rust, C#, etc.) and provides detailed analysis of common issues. Each failure will include comparisons between the LLM-generated and expected code, identify the root causes, and recommend actionable changes to the documentation.

---

Rust / rustdoc_json Failures (8 total)

Common Issues

1. Faulty Function Signatures and Types: Many failures stem from incorrect function signatures and types in declared reducers and tables.

2. Missing Expected Traits/Attributes: LLM-generated code often lacks the required public visibility (e.g., pub), causing structure and attribute mismatches.

3. Incorrect Use of Scheduling: The scheduling function setup was incorrect in multiple instances, failing to align with the expected API-style.

---

1. Compile/Publish Errors

a. t_002_scheduled_table

LLM Output:

use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};

#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    if ctx.db.tick_timer().count() == 0 {
        ctx.db.tick_timer().insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::RepeatMicros(50_000),
        });
    }
}

Expected (golden):

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    pub scheduled_id: u64,
    pub scheduled_at: ScheduleAt,
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) -> Result<(), String> {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
    });
    Ok(())
}

Error: publish_error: spacetime publish failed (exit=1)

Difference:

• Missing pub visibility on struct fields.
• Usage of ScheduleAt::RepeatMicros instead of ScheduleAt::Interval(Duration::from_millis(...).into()).

Root Cause:
The documentation lacks clear examples of field visibility and correct usage of scheduling parameters.

Recommendation:
Update the documentation to include visibility modifiers and examples utilizing ScheduleAt::Interval.

---

b. t_017_scheduled_columns

Similar to t_002_scheduled_table.

• Action Required: Same recommendations as above.

---

2. Other Failures

a. t_012_spacetime_product_type

LLM Output:

use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType)]
struct Score {
    left: i32,
    right: i32,
}

#[table(name = result)]
struct ResultRow {
    #[primary_key]
    id: i32,
    value: Score,
}

#[reducer]
fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
    ctx.db.result().insert(ResultRow {
        id,
        value: Score { left, right },
    });
}

Expected (golden):

use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Score {
    pub left: i32,
    pub right: i32,
}

#[table(name = result)]
pub struct ResultRow {
    #[primary_key]
    pub id: i32,
    pub value: Score,
}

#[reducer]
pub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
    ctx.db.result().insert(ResultRow { id, value: Score { left, right } });
}

Error: no such table: "result"

Difference:

• Missing pub declaration on the struct and fields.

Root Cause:
The need for public access levels on types and fields is not sufficiently covered in the documentation.

Recommendation:
Add clear examples demonstrating the use of pub in type definitions.

---

Additional Failures

• The patterns and problems arising in these similar failures (t_013, t_015, t_016, t_018) align closely with public visibility issues or missing API attributes.
• Action Required: Follow similar recommendations emphasizing public access in documented type structures.

---

Rust / docs Failures (22 total)

The issues observed in the Rust documentation failures mirror the previous section, particularly around building, struct declarations, and public visibility.

Example: t_000_empty_reducers

The LLM-generated code lacks return types on various reducers, while the golden example properly uses Result<(), String>.

Actionable Insight: Clearly document the importance of declaring function return types on reducers.

---

C# / docs Failures (5 total)

Common Issues:

1. Visibility Modifiers: Many C# failures stem from missing or wrong visibility modifiers.
2. Attributes and Enums: Incorrect use or omission of expected attributes (like [Type]) was frequent.

---

Example: t_013_spacetime_sum_type

LLM Output:

using SpacetimeDB;
using SpacetimeDB.Types;

public static partial class Module
{
    [SpacetimeDB.Type]
    public partial struct Circle { public int Radius; }
    
    [SpacetimeDB.Table(Name = "Result", Public = true)]
    public partial struct Result
    {
        [SpacetimeDB.PrimaryKey] public int Id;
        public Shape Value;
    }
}

Expected (golden):

using SpacetimeDB;

public static partial class Module
{
    [Type] public partial struct Circle { public int Radius; }
    
    [Table(Name = "Result")]
    public partial struct Result
    {
        [PrimaryKey] public int Id;
        public Shape Value;
    }
}

Error: publish_error: spacetime build (csharp) failed

Difference: Incorrectly specified Public = true attribute in the table declaration; missing the [Type] attribute.

Root Cause: Lack of explicit examples and standards regarding visibility and attribute application in the documentation.

Recommendation: Incorporate detailed examples that differentiate between public and default access levels, especially in C# attributes.

---

Conclusion

The analysis reveals consistent issues to be addressed across various tests. The primary points of failure involve:

• Visibility modifiers.
• Correct API usage of structure and attributes.
• Suggest clarifying the conditions and requirements for successful compilation in the documentation to enhance developer understanding and reduce confusion.

[jdetter]

Merge after #4121

[jdetter]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	26/27	91.7% ⬆️ +15.3%
Rust	rustdoc_json	schema	23/34	65.3% ⬇️ -10.0%
Rust	rustdoc_json	total	49/61	79.7% ⬆️ +3.8%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	4/30	12.5% ⬇️ -8.0%
Rust	docs	total	9/57	11.7% ⬇️ -3.6%
C#	docs	basics	24/27	91.7% ⬇️ -8.3%
C#	docs	schema	22/34	63.7%
C#	docs	total	46/61	78.9% ⬇️ -4.5%

Compared against master branch baseline

_{Generated at: 2026-01-27T05:19:18.787Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 35

---

Analysis of SpacetimeDB Benchmark Failures

Rust / rustdoc_json Failures

Compile/Publish Errors (2 failures)

t_002_scheduled_table

Generated Code:

use spacetimedb::{table, reducer, ReducerContext, Table, UniqueColumn, ScheduleAt};

#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    let tbl = ctx.db.tick_timer();

    let mut keep_id: Option<u64> = None;
    for row in tbl.iter() {
        if keep_id.is_none() {
            let mut r = row;
            r.scheduled_at = ScheduleAt::IntervalMicros(50_000);
            let updated = tbl.scheduled_id().update(r);
            keep_id = Some(updated.scheduled_id);
        } else {
            tbl.scheduled_id().delete(&row.scheduled_id);
        }
    }

    if keep_id.is_none() {
        let _ = tbl.insert(TickTimer {
            scheduled_id: 0,
            scheduled_at: ScheduleAt::IntervalMicros(50_000),
        });
    }
}

#[reducer]
pub fn tick(_ctx: &ReducerContext) {}

Expected (Golden Code):

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
    Ok(())
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) -> Result<(), String> {
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
    });
    Ok(())
}

Error: publish_error: spacetime publish failed (exit=1)

Differences:

• The LLM used schedule(reducer = tick, column = scheduled_at) instead of the expected scheduled(tick).
• The tick reducer function lacked a return type.
• The init function's scheduled_at field was incorrectly assigned using IntervalMicros(50_000) instead of Interval(Duration::from_millis(50).into()).

Root Cause: Misunderstanding of how to set up scheduling and return types for reducer functions in Rust.

Recommendation: Update the documentation to clarify:

• How to define scheduling within the table attribute.
• The need for specifying return types in reducer functions.
• Correct usage examples for setting up intervals for scheduling.

---

t_017_scheduled_columns

Generated Code:

use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};

#[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    scheduled_id: u64,
    scheduled_at: ScheduleAt,
}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    if ctx.db.tick_timer().count() == 0 {
        ctx.db
            .tick_timer()
            .insert(TickTimer {
                scheduled_id: 0,
                scheduled_at: ScheduleAt::repeat_micros(50_000),
            });
    }
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}

Expected (Golden Code):

use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
use std::time::Duration;

#[table(name = tick_timer, scheduled(tick))]
pub struct TickTimer {
    #[primary_key]
    #[auto_inc]
    pub scheduled_id: u64,
    pub scheduled_at: ScheduleAt,
}

#[reducer]
pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {}

#[reducer(init)]
pub fn init(ctx: &ReducerContext) {
    let every_50ms: ScheduleAt = Duration::from_millis(50).into();
    ctx.db.tick_timer().insert(TickTimer {
        scheduled_id: 0,
        scheduled_at: every_50ms,
    });
}

Error: publish_error: spacetime publish failed (exit=1)

Differences:

• The LLM again used the incorrect syntax for scheduling.
• Used ScheduleAt::repeat_micros(50_000) while the expected syntax is using Duration::from_millis(...).

Root Cause: Similar to the first failure, a misunderstanding of the correct APIs for table scheduling and the desired syntax.

Recommendation: Amend documentation or the code generation model to ensure the correct formatting and usage patterns are demonstrated, especially around scheduling and types.

---

Other Failures (5 failures)

t_013_spacetime_sum_type (2/3 tests passed)

Generated Code:

use spacetimedb::{ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType)]
pub struct Rect {
    pub width: i32,
    pub height: i32,
}

#[derive(SpacetimeType)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[spacetimedb::table(name = result)]
pub struct ResultRow {
    #[primary_key]
    pub id: i32,
    pub value: Shape,
}

#[spacetimedb::reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
    ctx.db.result().insert(ResultRow {
        id,
        value: Shape::Circle(radius),
    });
}

Expected (Golden Code):

use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Rect {
    pub width: i32,
    pub height: i32,
}

#[derive(SpacetimeType, Clone, Debug)]
pub enum Shape {
    Circle(i32),
    Rectangle(Rect),
}

#[table(name = result)]
pub struct ResultRow {
    #[primary_key]
    pub id: i32,
    pub value: Shape,
}

#[reducer]
pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
    ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });
}

Error: no such table: result.

Differences:

• The LLM missed implementing the Clone and Debug traits for Rect and Shape.
• Notably, this could have contributed to the inability to handle implications regarding mutability and state during tests.

Root Cause: The omission of Clone and Debug traits in derived types may confuse the necessary behavior for interacting with the database and running appropriate tests.

Recommendation: Include compliance with trait derivations in the documentation and examples to reflect necessary traits for database interaction.

---

t_015_product_type_columns (2/3 tests passed)

Generated Code:

use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};

#[derive(SpacetimeType)]
pub struct Address {
    street: String,
    zip: i32,
}

#[derive(SpacetimeType)]
pub struct Position {
    x: i32,
    y: i32,
}

#[table(name = profile)]
pub struct Profile {
    #[primary_key]
    id: i32,
    home: Address,
    work: Address,
    pos: Position,
}

#[reducer]
pub fn seed(ctx: &ReducerContext) {
    ctx.db.profile().insert(Profile {
        id: 1,
        home: Address { street: "1 Main".to_string(), zip: 11111 },
        work: Address { street: "2 Broad".to_string(), zip: 22222 },
        pos: Position { x: 7, y: 9 },
    });
}

Expected (Golden Code):

use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};

#[derive(SpacetimeType, Clone, Debug)]
pub struct Address {
    pub street: String,
    pub zip: i32,
}

#[derive(SpacetimeType, Clone, Debug)]
pub struct Position {
    pub x: i32,
    pub y: i32,
}

#[table(name = profile)]
pub struct Profile {
    #[primary_key]
    pub id: i32,
    pub home: Address,
    pub work: Address,
    pub pos: Position,
}

#[reducer]
pub fn seed(ctx: &ReducerContext) {
    ctx.db.profile().insert(Profile {
        id: 1,
        home: Address { street: "1 Main".into(), zip: 11111 },
        work: Address { street: "2 Broad".into(), zip: 22222 },
        pos: Position { x: 7, y: 9 },
    });
}

Error: no such table: profile.

Differences:

• Missing the pub visibility modifier on the struct fields resulted in inadequate exposure to the database layer for runtime reflection or generation.

Root Cause: The LLM lacks clarity in visibility context leading to struct field inaccessibility.

Recommendation: Update documentation examples to emphasize the importance of pub modifiers for public API, particularly in database-related types.

---

Recommendation Summary

The fundamental issues in the test failures stem from:

• Incorrect syntax (particularly in schedule attributes).
• Missing visibility modifiers (pub).
• Omitted trait implementations necessary for database operations.

Enhancing the clarity, correctness, and completeness of documentation is essential. Specific examples that match the expected outputs must illustrate successful patterns for users. This would significantly lower the occurrence of erroneous submissions, leading to successful builds and test runs.

[bfops]

/update-llm-benchmark

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	25/27	83.3% ⬆️ +6.9%
Rust	rustdoc_json	schema	23/34	65.3% ⬇️ -10.0%
Rust	rustdoc_json	total	48/61	75.2% ⬇️ -0.8%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	8/34	20.5%
Rust	docs	total	13/61	15.4%
C#	docs	basics	27/27	100.0%
C#	docs	schema	23/34	68.3% ⬆️ +4.7%
C#	docs	total	50/61	85.6% ⬆️ +2.1%

Compared against master branch baseline

_{Generated at: 2026-01-27T06:59:15.063Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 34

---

Analysis of SpacetimeDB Benchmark Test Failures

Rust / rustdoc_json Failures

Compile/Publish Errors

t_002_scheduled_table

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
   
   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       let _ = ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: ScheduleAt::repeat(50_000),
       });
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext) {
       log::debug!("tick fired");
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) -> Result<(), String> {
       Ok(())
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) -> Result<(), String> {
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: ScheduleAt::Interval(Duration::from_millis(50).into()),
       });
       Ok(())
   }

3. The error:

   publish_error: spacetime publish failed (exit=1)
   

4. Explain the difference:

   • The generated code used an incorrect ScheduleAt value and omitted returning a Result<()> in both the init and tick reducers. The golden example specifies the schedule attribute properly and uses Duration.

5. Root cause:

   • The documentation is unclear about the proper usage of ScheduleAt and return types in reducer functions.

6. Recommendation:

   • Clarify the types for ScheduleAt in the documentation, emphasizing the need to handle the function results, especially for reducers.

---

t_017_scheduled_columns

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
   
   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(scheduled)]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {}
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: ScheduleAt::RepeatMicros(50_000),
       });
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {}
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       let every_50ms: ScheduleAt = Duration::from_millis(50).into();
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: every_50ms,
       });
   }

3. The error:

   publish_error: spacetime publish failed (exit=1)
   

4. Explain the difference:

   • The generated code incorrectly uses ScheduleAt::RepeatMicros. The golden example uses Duration properly, and the reducer method signatures are defined accurately.

5. Root cause:

   • The documentation lacks clear usage examples for handling scheduled columns correctly.

6. Recommendation:

   • Update the documentation to emphasize accurate definitions and expected types in reducer functions and their parameters.

---

Other Failures

t_001_basic_tables

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       id: i32,
       name: String,
       age: i32,
       active: bool,
   }
   
   #[table(name = product)]
   pub struct Product {
       #[primary_key]
       id: i32,
       title: String,
       price: f32,
       in_stock: bool,
   }
   
   #[table(name = note)]
   pub struct Note {
       #[primary_key]
       id: i32,
       body: String,
       rating: i64,
       pinned: bool,
   }
   
   #[reducer(init)]
   pub fn init(_ctx: &ReducerContext) {}

2. The golden example:

   use spacetimedb::table;
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       pub id: i32,
       pub name: String,
       pub age: i32,
       pub active: bool,
   }
   
   #[table(name = product)]
   pub struct Product {
       #[primary_key]
       pub id: i32,
       pub title: String,
       pub price: f32,
       pub in_stock: bool,
   }
   
   #[table(name = note)]
   pub struct Note {
       #[primary_key]
       pub id: i32,
       pub body: String,
       pub rating: i64,
       pub pinned: bool,
   }

3. The error:

   schema_parity: reducers differ - expected [], got ["init()"]
   

4. Explain the difference:

   • The primary key fields don't have pub visibility in the generated code which results in schema mismatch.

5. Root cause:

   • The documentation does not specify visibility modifiers adequately.

6. Recommendation:

   • Explicitly mention the need for public visibility on fields in table definitions in the documentation.

---

t_012_spacetime_product_type (similar to t_013 and t_014)

1. The generated code:

   use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
   
   #[derive(SpacetimeType, Clone)]
   struct Score {
       left: i32,
       right: i32,
   }
   
   #[table(name = result)]
   struct ResultRow {
       #[primary_key]
       id: i32,
       value: Score,
   }
   
   #[reducer]
   fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
       ctx.db.result().insert(ResultRow {
           id,
           value: Score { left, right },
       });
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Score {
       pub left: i32,
       pub right: i32,
   }
   
   #[table(name = result)]
   pub struct ResultRow {
       #[primary_key]
       pub id: i32,
       pub value: Score,
   }
   
   #[reducer]
   pub fn set_score(ctx: &ReducerContext, id: i32, left: i32, right: i32) {
       ctx.db.result().insert(ResultRow { id, value: Score { left, right } });
   }

3. The error:

   product_type_row_parity: spacetime sql failed: ... no such table: result
   

4. Explain the difference:

   • Missing pub visibility on struct fields in the generated code prevents successful table creation and querying.

5. Root cause:

   • The importance of public visibility for struct fields in SpacetimeDB is not emphasized.

6. Recommendation:

   • Include specific examples in the documentation highlighting struct field visibility requirements.

---

C# / docs Failures

Other Failures

t_014_elementary_columns

1. The generated code:

   using SpacetimeDB;
   
   public static partial class Module
   {
       [SpacetimeDB.Table(Name = "Primitive")]
       public partial struct Primitive
       {
           [SpacetimeDB.PrimaryKey]
           public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [SpacetimeDB.Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive
           {
               Id = 1,
               Count = 2,
               Total = 3000000000L,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice"
           });
       }
   }

2. The golden example:

   using SpacetimeDB;
   
   public static partial class Module
   {
       [Table(Name = "Primitive")]
       public partial struct Primitive
       {
           [PrimaryKey] public int Id;
           public int Count;
           public long Total;
           public float Price;
           public double Ratio;
           public bool Active;
           public string Name;
       }
   
       [Reducer]
       public static void Seed(ReducerContext ctx)
       {
           ctx.Db.Primitive.Insert(new Primitive {
               Id = 1,
               Count = 2,
               Total = 3000000000,
               Price = 1.5f,
               Ratio = 2.25,
               Active = true,
               Name = "Alice"
           });
       }
   }

3. The error:

   no such table: primitive
   

4. Explain the difference:

   • The usage of attributes in the generated code was inconsistent (e.g., missing public for the Id field), causing schema definition issues.

5. Root cause:

   • Insufficient clarity on the correct usage of access modifiers and attributes in class definitions.

6. Recommendation:

   • Provide a detailed section in the documentation on defining tables and reducers with clear examples of access modifiers and parameter attributes.

---

This analysis highlights several key areas in the documentation that need improvement, especially regarding visibility modifiers, function returns, and proper formatting to assist users in avoiding common pitfalls during coding.

[jdetter]

/update-llm-benchmarks

[clockwork-labs-bot]

LLM Benchmark Results (ci-quickfix)

Language	Mode	Category	Tests Passed	Task Pass %
Rust	rustdoc_json	basics	25/27	83.3% ⬆️ +6.9%
Rust	rustdoc_json	schema	18/34	55.3% ⬇️ -20.0%
Rust	rustdoc_json	total	43/61	70.6% ⬇️ -5.3%
Rust	docs	basics	5/27	11.1%
Rust	docs	schema	4/30	12.5% ⬇️ -8.0%
Rust	docs	total	9/57	11.7% ⬇️ -3.6%
C#	docs	basics	27/27	100.0%
C#	docs	schema	25/34	73.7% ⬆️ +10.0%
C#	docs	total	52/61	88.0% ⬆️ +4.5%

Compared against master branch baseline

_{Generated at: 2026-01-27T16:47:52.904Z}

Failure Analysis (click to expand)

Benchmark Failure Analysis

Generated from: /__w/SpacetimeDB/SpacetimeDB/tools/xtask-llm-benchmark/../../docs/llms/docs-benchmark-details.json

Summary

• Total failures analyzed: 35

---

Analysis of SpacetimeDB Benchmark Failures

Rust / rustdoc_json Failures (9 total)

Compile/Publish Errors

t_002_scheduled_table & t_017_scheduled_columns

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, ScheduleAt};
   
   #[table(name = tick_timer, schedule(reducer = tick, column = scheduled_at))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       scheduled_id: u64,
       scheduled_at: ScheduleAt,
   }
   
   #[reducer(scheduled)]
   pub fn tick(_ctx: &ReducerContext, _row: TickTimer) {
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       if ctx.db.tick_timer().count() == 0 {
           ctx.db.tick_timer().insert(TickTimer {
               scheduled_id: 0,
               scheduled_at: ScheduleAt::repeat_micros(50_000),
           });
       }
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, ScheduleAt, Table};
   use std::time::Duration;
   
   #[table(name = tick_timer, scheduled(tick))]
   pub struct TickTimer {
       #[primary_key]
       #[auto_inc]
       pub scheduled_id: u64,
       pub scheduled_at: ScheduleAt,
   }
   
   #[reducer]
   pub fn tick(_ctx: &ReducerContext, _schedule: TickTimer) {
   }
   
   #[reducer(init)]
   pub fn init(ctx: &ReducerContext) {
       let every_50ms: ScheduleAt = Duration::from_millis(50).into();
       ctx.db.tick_timer().insert(TickTimer {
           scheduled_id: 0,
           scheduled_at: every_50ms,
       });
   }

3. The error:

   publish_error: spacetime publish failed (exit=1)
   

4. Explain the difference:

   • The LLM used incorrect syntax for reducer attributes and data type for scheduled_at. The expected code uses scheduled as an attribute and specifies data converts.

5. Root cause:

   • The documentation lacks clarity on the correct use of the scheduled attribute and how to handle the ScheduleAt type properly.

6. Recommendation:

   • Update documentation to clearly explain the usage and requirement of reducer attributes and how to define the ScheduleAt type. Show examples covering both attribute definitions and data types.

---

t_019_many_to_many

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       user_id: i32,
       name: String,
   }
   
   #[table(name = group)]
   pub struct Group {
       #[primary_key]
       group_id: i32,
       title: String,
   }
   
   #[table(
       name = membership,
       index(name = by_user, btree(columns = [user_id])),
       index(name = by_group, btree(columns = [group_id]))
   )]
   pub struct Membership {
       #[primary_key]
       id: i32,
       user_id: i32,
       group_id: i32,
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, Table};
   
   #[table(name = user)]
   pub struct User {
       #[primary_key]
       pub user_id: i32,
       pub name: String,
   }
   
   #[table(name = group)]
   pub struct Group {
       #[primary_key]
       pub group_id: i32,
       pub title: String,
   }
   
   #[table(
       name = membership,
       index(name = by_user, btree(columns = [user_id])),
       index(name = by_group, btree(columns = [group_id]))
   )]
   pub struct Membership {
       #[primary_key]
       pub id: i32,
       pub user_id: i32,
       pub group_id: i32,
   }

3. The error:

   publish_error: spacetime publish failed (exit=1)
   

4. Explain the difference:

   • The generated code does not mark primary key fields as public, whereas the expected code does, which can cause access issues.

5. Root cause:

   • Lack of clarity in the documentation regarding the access level of struct fields in SpacetimeDB.

6. Recommendation:

   • Add notes in the documentation stating that public access is required for fields in database schemas.

---

Other Failures

t_003_struct_in_table

1. The generated code:

   use spacetimedb::ReducerContext;
   use spacetimedb::SpacetimeType;
   
   #[derive(SpacetimeType)]
   pub struct Position {
       pub x: i32,
       pub y: i32,
   }
   
   #[spacetimedb::table(name = entity)]
   pub struct Entity {
       #[primary_key]
       pub id: i32,
       pub pos: Position,
   }
   
   #[spacetimedb::reducer(init)]
   pub fn init(_ctx: &ReducerContext) {}

2. The golden example:

   use spacetimedb::{table, SpacetimeType};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Position {
       pub x: i32,
       pub y: i32,
   }
   
   #[table(name = entity)]
   pub struct Entity {
       #[primary_key]
       pub id: i32,
       pub pos: Position,
   }

3. The error:

   schema_parity: reducers differ - expected [], got ["init()"]
   

4. Explain the difference:

   • The generated function init is not necessary, and it's leading to additional unwanted complexity that results in a failure.

5. Root cause:

   • The documentation does not specify when reducers are required, which should be clarified.

6. Recommendation:

   • Clarify in the docs that reducers are only needed when initializing data or other specific purposes.

---

t_013_spacetime_sum_type

1. The generated code:

   use spacetimedb::{table, reducer, ReducerContext, Table, SpacetimeType};
   
   #[derive(SpacetimeType)]
   pub struct Rect {
       width: i32,
       height: i32,
   }
   
   #[derive(SpacetimeType)]
   pub enum Shape {
       Circle(i32),
       Rectangle(Rect),
   }
   
   #[table(name = result)]
   pub struct ResultRow {
       #[primary_key]
       id: i32,
       value: Shape,
   }
   
   #[reducer]
   fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
       ctx.db.result().insert(ResultRow {
           id,
           value: Shape::Circle(radius),
       });
   }

2. The golden example:

   use spacetimedb::{reducer, table, ReducerContext, SpacetimeType, Table};
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub struct Rect {
       pub width: i32,
       pub height: i32,
   }
   
   #[derive(SpacetimeType, Clone, Debug)]
   pub enum Shape {
       Circle(i32),
       Rectangle(Rect),
   }
   
   #[table(name = result)]
   pub struct ResultRow {
       #[primary_key]
       pub id: i32,
       pub value: Shape,
   }
   
   #[reducer]
   pub fn set_circle(ctx: &ReducerContext, id: i32, radius: i32) {
       ctx.db.result().insert(ResultRow { id, value: Shape::Circle(radius) });
   }

3. The error:

   sum_type_row_parity: spacetime sql failed: no such table: `result`.
   

4. Explain the difference:

   • The LLM did not make the necessary fields public, which can lead to visibility issues with database inserts.

5. Root cause:

   • The documentation may not be clear about the visibility of struct fields.

6. Recommendation:

   • Update the documentation to stress the importance of public access for all fields that interact with the database.

---

Summary Recommendations

• Visibility: The documentation should consistently emphasize that public access is necessary for struct fields used in SpacetimeDB.
• Reducer Necessity: Clarify when reducers should be defined in accordance with specific use cases (e.g., initial data setup).
• Attribute Usage: Provide specific examples of attribute usage, including trait bounds for types like ScheduleAt and when/how to annotate functions within data models.

By addressing the above gaps in the documentation, future errors can be mitigated, making it easier for developers to utilize SpacetimeDB effectively.