Skip to content

Commit 04797d3

Browse files
kasvithkasvith
committed
feat: resumable parallel jobs
1 parent c783042 commit 04797d3

17 files changed

Lines changed: 3211 additions & 51 deletions

File tree

‎agents/.archived-topics.json‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
{
2+
"version": "1.0",
3+
"lastUpdated": "2026-01-03",
4+
"topics": [
5+
{
6+
"slug": "embeddable-library-transformation",
7+
"title": "Embeddable Library Transformation",
8+
"description": "Transform Durable from standalone application to Oban-style embeddable library",
9+
"created": "2026-01-02",
10+
"lastUpdated": "2026-01-02",
11+
"sessionCount": 1,
12+
"status": "completed",
13+
"tags": [
14+
"embedding",
15+
"oban-pattern",
16+
"configuration",
17+
"supervision-tree",
18+
"ecto",
19+
"migrations"
20+
],
21+
"keyFiles": [
22+
"lib/durable/config.ex",
23+
"lib/durable/migration.ex",
24+
"lib/durable/supervisor.ex"
25+
],
26+
"sessions": [
27+
{
28+
"number": 1,
29+
"date": "2026-01-02",
30+
"file": "sessions/2026-01-02-session-01.md",
31+
"focus": "Full implementation of embeddable pattern",
32+
"status": "completed"
33+
}
34+
]
35+
},
36+
{
37+
"slug": "parallel-durability-implementation",
38+
"title": "Parallel Durability Implementation",
39+
"description": "Make parallel execution truly durable and resumable in Durable workflow engine",
40+
"created": "2026-01-03",
41+
"lastUpdated": "2026-01-03",
42+
"sessionCount": 1,
43+
"status": "completed",
44+
"tags": [
45+
"parallel",
46+
"durability",
47+
"resume",
48+
"context",
49+
"executor",
50+
"integration-tests"
51+
],
52+
"keyFiles": [
53+
"lib/durable/executor.ex",
54+
"lib/durable/executor/step_runner.ex",
55+
"test/durable/integration_test.exs",
56+
"test/durable/parallel_test.exs"
57+
],
58+
"sessions": [
59+
{
60+
"number": 1,
61+
"date": "2026-01-03",
62+
"file": "sessions/2026-01-03-session-01.md",
63+
"focus": "Full parallel durability implementation",
64+
"status": "completed"
65+
}
66+
]
67+
}
68+
]
69+
}

‎agents/WORKPLAN.md‎

Lines changed: 64 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22

33
## Executive Summary
44

5-
**Completed:** Phase 0 (Foundation) + Phase 1 (Core MVP) + Phase 2 Log Capture (2.1-2.2) + Phase 3 Wait Primitives (3.1-3.3) + Phase 5 Query API
6-
**Remaining:** Phase 2 (Graph: 2.3-2.5), Phase 3 (Control Flow: 3.4-3.10), Phase 4 (Scalability), Phase 5 (Mix Tasks, Docs)
5+
**Completed:** Phase 0 (Foundation) + Phase 1 (Core MVP) + Phase 2 Log Capture (2.1-2.2) + Phase 3 Wait Primitives (3.1-3.3) + Phase 3 Control Flow (3.4, 3.6, 3.7) + Phase 5 Query API
6+
**Remaining:** Phase 2 (Graph: 2.3-2.5), Phase 3 (Control Flow: 3.5, 3.8-3.10), Phase 4 (Scalability), Phase 5 (Mix Tasks, Docs)
77

8-
**Current State:** 25 modules, 57 passing tests, core DSL, executor, wait primitives, log capture, and query API working
8+
**Current State:** 28 modules, 104 passing tests, core DSL, executor, wait primitives, log capture, query API, branch, parallel, and foreach working
99

10-
**Overall Progress:** ~40% complete
10+
**Overall Progress:** ~55% complete
1111

1212
---
1313

@@ -651,12 +651,12 @@ Durable.Wait.list_pending_inputs(
651651

652652
---
653653

654-
### 3.4 Branch - Conditional Flow ✅ REDESIGNED
654+
### 3.4 Branch - Conditional Flow ✅ COMPLETE
655655

656656
**Priority:** High
657657
**Complexity:** Medium
658658
**Dependencies:** Core DSL
659-
**Status:** REDESIGNED - Replaces original `decision` + `on_decision` pattern
659+
**Status:** IMPLEMENTED - Replaces original `decision` + `on_decision` pattern
660660

661661
**Goal:** Implement intuitive `branch` macro for conditional workflow execution that reads top-to-bottom like normal code.
662662

@@ -847,11 +847,12 @@ end
847847

848848
---
849849

850-
### 3.6 Parallel Execution
850+
### 3.6 Parallel Execution ✅ COMPLETE
851851

852852
**Priority:** High
853853
**Complexity:** High
854854
**Dependencies:** Core executor
855+
**Status:** IMPLEMENTED
855856

856857
**Goal:** Implement `parallel` block for concurrent step execution.
857858

@@ -924,47 +925,71 @@ end
924925
| `:complete_all` | Wait for all, collect errors |
925926

926927
**Acceptance Criteria:**
927-
- [ ] `parallel` macro works
928-
- [ ] Steps execute concurrently
929-
- [ ] All steps complete before continuing
930-
- [ ] Context merging works
931-
- [ ] Error handling configurable
932-
- [ ] Graph shows parallel branches
928+
- [x] `parallel` macro works
929+
- [x] Steps execute concurrently (verified with timing test)
930+
- [x] All steps complete before continuing
931+
- [x] Context merging works (deep_merge, last_wins, collect strategies)
932+
- [x] Error handling configurable (fail_fast, complete_all)
933+
- [ ] Graph shows parallel branches (needs graph implementation)
934+
935+
**Implementation Notes:**
936+
- Uses `Task.Supervisor.async/3` for proper supervision
937+
- Task.Supervisor added to `Durable.Supervisor` children
938+
- Step naming: `parallel_<id>__<step_name>`
939+
- Files: `lib/durable/dsl/step.ex`, `lib/durable/executor.ex`
940+
- Tests: `test/durable/parallel_test.exs` (11 tests)
933941

934942
---
935943

936-
### 3.7 ForEach
944+
### 3.7 ForEach ✅ COMPLETE
937945

938946
**Priority:** Medium
939947
**Complexity:** Medium
940948
**Dependencies:** 3.6 (Parallel)
949+
**Status:** IMPLEMENTED
941950

942951
**Goal:** Implement `foreach` for processing collections.
943952

944953
**DSL:**
945954

946955
```elixir
947-
foreach :process_items,
948-
items: fn -> get_context(:items) end,
949-
concurrency: 5 do |item|
950-
956+
# Using context key for items
957+
foreach :process_items, items: :items do
951958
step :process_item do
959+
item = current_item()
960+
index = current_index()
952961
result = ItemProcessor.process(item)
953962
append_context(:results, result)
954963
end
955964
end
965+
966+
# Concurrent with collect_as
967+
foreach :process_items, items: :items, concurrency: 5, collect_as: :results do
968+
step :process_item do
969+
put_context(:result, process(current_item()))
970+
end
971+
end
956972
```
957973

958974
**Implementation:**
959-
- Sequential: Process items one at a time
960-
- Concurrent: Process up to N items in parallel
975+
- Sequential: Process items one at a time (default)
976+
- Concurrent: Process up to N items in parallel using Task.Supervisor
977+
- `current_item()` and `current_index()` available in Context
978+
- `collect_as` option for gathering results in concurrent mode
979+
- Error strategies: `:fail_fast` (default) and `:continue`
961980

962981
**Acceptance Criteria:**
963-
- [ ] `foreach` iterates over collection
964-
- [ ] Item available in step context
965-
- [ ] Sequential mode works
966-
- [ ] Concurrent mode with limit works
967-
- [ ] Results collected correctly
982+
- [x] `foreach` iterates over collection
983+
- [x] Item available via `current_item()`, index via `current_index()`
984+
- [x] Sequential mode works
985+
- [x] Concurrent mode with limit works
986+
- [x] Results collected correctly via `collect_as`
987+
988+
**Implementation Notes:**
989+
- Macro: `lib/durable/dsl/step.ex` lines 476-607
990+
- Executor: `lib/durable/executor.ex` lines 603-894
991+
- Context: Added `current_item/0`, `current_index/0`, `set_foreach_item/2`, `clear_foreach_item/0`
992+
- Tests: `test/durable/foreach_test.exs` (13 tests)
968993

969994
---
970995

@@ -1470,11 +1495,11 @@ end
14701495
| Sleep Primitives | High | Medium | None | ✅ DONE |
14711496
| Wait for Events | High | Medium | Sleep | ✅ DONE |
14721497
| Wait for Input | High | Medium | Events | ✅ DONE |
1473-
| Decision Steps | High | Medium | None | TODO |
1474-
| Loops | Medium | Medium | Decisions | TODO |
1475-
| Parallel | High | High | None | TODO |
1476-
| ForEach | Medium | Medium | Parallel | TODO |
1477-
| Switch/Case | Medium | Low | Decisions | TODO |
1498+
| Branch (3.4) | High | Medium | None | ✅ DONE |
1499+
| Loops | Medium | Medium | Branch | SKIPPED (use retries/each) |
1500+
| Parallel (3.6) | High | High | None | ✅ DONE |
1501+
| ForEach (3.7) | Medium | Medium | Parallel | ✅ DONE |
1502+
| Switch/Case | Medium | Low | Branch | TODO |
14781503
| Workflow Orchestration | High | Medium | Core | TODO |
14791504
| Pipe-based API | Medium | Medium | Core | TODO |
14801505
| Compensation | High | High | None | TODO |
@@ -1502,13 +1527,13 @@ end
15021527
4. Execution State (2.5)
15031528

15041529
### Sprint 3: Control Flow (1.5 weeks)
1505-
1. Decision Steps (3.4)
1506-
2. Switch/Case (3.8)
1507-
3. Loops (3.5)
1508-
4. Parallel Execution (3.6)
1530+
1. Branch (3.4) ✅ DONE
1531+
2. Parallel Execution (3.6) ✅ DONE
1532+
3. Loops (3.5) - SKIPPED (use step retries or ForEach instead)
1533+
4. Switch/Case (3.8) - Can use branch instead
15091534

15101535
### Sprint 4: Advanced Patterns (1 week)
1511-
1. ForEach (3.7)
1536+
1. ForEach (3.7) ✅ DONE
15121537
2. Compensation/Saga (3.9)
15131538
3. Cron Scheduling (3.10)
15141539

@@ -1534,7 +1559,9 @@ end
15341559

15351560
### Phase 3 Complete When:
15361561
- [x] All wait primitives work (sleep, event, input) ✅
1537-
- [ ] All control flow works (decision, loop, parallel, foreach)
1562+
- [x] Branch/conditional flow works ✅
1563+
- [x] Parallel execution works ✅
1564+
- [x] ForEach works ✅
15381565
- [ ] Compensation pattern works
15391566
- [ ] Cron scheduling works
15401567

@@ -1556,7 +1583,7 @@ end
15561583
| Risk | Mitigation |
15571584
|------|------------|
15581585
| Macro complexity | Extensive testing, incremental development |
1559-
| Parallel execution bugs | Use Task.Supervisor, proper error handling |
1586+
| Parallel execution bugs | ✅ Using Task.Supervisor, proper error handling |
15601587
| Context merge conflicts | Well-defined merge strategies, documentation |
15611588
| Performance issues | Benchmark critical paths, use database indexes |
15621589
| Breaking changes | Semantic versioning, deprecation warnings |

‎agents/context-index.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Durable Development Context Index
2+
3+
This index provides quick access to archived development discussions and implementation plans for the Durable workflow engine.
4+
5+
## Active Topics
6+
7+
| Topic | Last Updated | Sessions | Status |
8+
|-------|--------------|----------|--------|
9+
| [Embeddable Library Transformation](./conversations/embeddable-library-transformation/) | 2026-01-02 | 1 | Completed |
10+
| [Parallel Durability Implementation](./conversations/parallel-durability-implementation/) | 2026-01-03 | 1 | Completed |
11+
12+
## Completed Topics
13+
14+
| Topic | Completed | Description |
15+
|-------|-----------|-------------|
16+
| [Embeddable Library Transformation](./conversations/embeddable-library-transformation/) | 2026-01-02 | Transformed Durable into an Oban-style embeddable library |
17+
| [Parallel Durability Implementation](./conversations/parallel-durability-implementation/) | 2026-01-03 | Made parallel execution truly durable and resumable |
18+
19+
## Topic Quick Reference
20+
21+
### Embeddable Library Transformation
22+
**Path**: `agents/conversations/embeddable-library-transformation/`
23+
24+
Covers the transformation of Durable from a standalone application to an embeddable library pattern. Key outcomes:
25+
- Dynamic repo via `Durable.Config.repo()`
26+
- Programmatic migrations via `Durable.Migration.up()/down()`
27+
- Supervisor-based integration into host application
28+
- PostgreSQL schema isolation with `durable` prefix
29+
30+
**Key Files Created**:
31+
- `lib/durable/config.ex`
32+
- `lib/durable/migration.ex`
33+
- `lib/durable/supervisor.ex`
34+
35+
### Parallel Durability Implementation
36+
**Path**: `agents/conversations/parallel-durability-implementation/`
37+
38+
Covers making parallel steps truly durable so completed steps are NOT re-executed on resume. Key outcomes:
39+
- Context snapshot storage in `__context__` key for parallel step outputs
40+
- Resume logic checks for completed parallel steps before execution
41+
- Stored contexts merged when resuming workflows
42+
- 11 integration tests covering complex workflow combinations
43+
- Bug fix: Decision/goto converging step pattern
44+
45+
**Key Files Modified**:
46+
- `lib/durable/executor.ex`
47+
- `lib/durable/executor/step_runner.ex`
48+
- `test/durable/integration_test.exs`
49+
- `test/durable/parallel_test.exs`
50+
51+
---
52+
53+
## How to Use This Index
54+
55+
1. **Finding Context**: Search this index when starting work on a feature that may have prior discussion
56+
2. **Adding New Topics**: Use the conversation-archiver agent to add new topics
57+
3. **Updating Existing Topics**: Add new sessions to existing topic folders
58+
59+
## Archive Structure
60+
61+
```
62+
agents/
63+
├── conversations/ # Archived discussion topics
64+
│ └── {topic-slug}/
65+
│ ├── README.md # Topic overview
66+
│ ├── sessions/ # Individual session records
67+
│ └── implementation-plan.md
68+
├── context-index.md # This file
69+
├── .archived-topics.json # Machine-readable metadata
70+
├── arch.md # Architecture notes
71+
└── WORKPLAN.md # Work planning
72+
```
73+
74+
---
75+
*Maintained by conversation-archiver agent*

0 commit comments

Comments
 (0)