feat(mothership): Chat-scoped outputs/ folder in VFS + Fork/Duplicate Chat

[j15z]

Summary

Adds two related capabilities on the chat/VFS surface, plus the field fixes found while testing them. (1) A chat-scoped outputs/ namespace for one-off generated files (images, audio, video, ffmpeg results) — separate from workspace files/, living with the chat, saveable to the workspace on request; agent-side steering is behind the mothership chat-scoped-outputs flag, so it dark-launches. (2) Fork and Duplicate — one engine, two gestures. The Fork button on each assistant reply copies the conversation up to and including that message into a fresh chat (Fork | <name>); the right-click Duplicate action is the same fork route with no cut point (upToMessageId is now optional) — a whole-chat copy titled <name> (Copy) that keeps every message and copies agent outputs too. In both modes, copied files get fresh row ids and storage keys, bytes are physically copied and quota-charged, and every in-transcript file reference is re-pointed at the copies so the copy survives deletion of the original. Duplicates ask the mothership fork endpoint for its whole-chat clone mode (compacted working memory preserved verbatim); branch forks keep the truncate-and-rebuild path. Schema: two additive migrations — 0254 adds nullable workspace_files.message_id, the provenance column that lets a branch fork copy only the chat-owned files (uploads AND outputs) born at-or-before the cut, and 0255 adds a partial unique index on output display names (see review-response changes below).

Field fixes included (found via live testing of the above):

• Ghost resources: a branch fork copied the chat's resources jsonb wholesale, keeping chips for files it doesn't have; file resources whose chat-owned file wasn't copied are now dropped.
• Fork on a live message: a just-streamed reply carries a synthetic live-assistant:<streamId> id and forking it 400'd; the Fork button now waits for the persisted id.
• Media-tool reference inputs: generate_image/video/audio and ffmpeg resolved inputs.files workspace-only, so uploads//outputs/ references never loaded — and generate_image silently generated from the prompt alone. New shared resolveToolInputFile covers all three VFS namespaces, and unresolvable explicit references now fail the call instead of being silently skipped.
• Phantom-stream reconnect wedge: an outputs/ link click on the home surface minted an empty-id resource that 400'd the next send and wedged the UI in "running" for ~3 minutes; empty-id resources are now rejected and a failed POST rolls back instead of retry-reconnecting.

Review-response changes (e43fe9c34)

Addresses all 8 findings from code review:

• Write-once in both modes: outputs/ + overwrite is now rejected in headless runs too — the files/ redirect can no longer silently replace a same-named workspace file.
• Branch forks now copy pre-cut outputs (design change from review discussion): outputs are stamped with message_id at creation and join the same timeline cut as uploads, so inline embeds are re-pointed and forks fully survive source-chat deletion. The rule for every chat-owned file: it travels with the fork iff the user message that carried/requested it is kept.
• Resolver adoption: open_resource, table import, and knowledge add-document resolve outputs//uploads/ refs and bare wf_ ids (new chat-scoped by-id fallback in resolveToolInputFile); presigning and the background CSV import honor chat-scoped storage contexts (TableImportPayload.fileContext, backward-compatible).
• create_file guard: checks the raw fileName before files/ prefixing, so outputs/notes.md can no longer create a literal files/outputs/ folder.
• Migration 0255: partial unique index on (chat_id, display_name) WHERE context='output' + a 23505 retry in uploadChatOutput, closing the concurrent same-name generation race (mirrors the existing uploads index/retry; separate index so the two namespaces stay independent).
• Reader consolidation: upload-file-reader + output-file-reader merged into one context-parameterized chat-file-reader (~200 duplicated lines removed, all public names preserved); shared isOutputsPath/leaf helpers replace 5 inline prefix checks.
• Fork blob copy: headObject replay guard (no double-copy/double-charge on retry) + bounded concurrency (4) + a single workspace_files read per fork with the branch cut applied in memory.
• Panel actions: EmbeddedFile/EmbeddedFileActions share one resolver hook (list → by-id → chat outputs by leaf name), so Download/Open work on path-referenced outputs.

Deploy note: run a one-off duplicate check for (chat_id, display_name) output rows before applying 0255 — the unique index build fails if race-produced dupes already exist (unlikely: young, flag-gated feature).

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation
• Other: ___________

Testing

• Full affected sweep green (vitest run lib/copilot lib/table lib/uploads app/api/mothership from apps/sim): 107 files, 1067 tests passing, including fork route, fork-chat-files, rewrite-file-references, effective-transcript, resolve-input-file, chat-file-reader (the merged upload/output readers), vfs, materialize-file, create-file, resources, and track-chat-upload. New coverage from the review pass: write-once rejection in both interactive and headless modes, the headless files/ redirect, the create_file raw-fileName guard, the wf_ id chat-scoped fallback, outputs joining the fork timeline cut (pre-cut copied + re-pointed, post-cut ghosts dropped), the blob-copy replay guard, and the bounded concurrency pool — alongside the earlier whole-chat duplicate, ghost-resource, and resolver cases.
• tsc --noEmit ✅ 0 errors · biome check ✅ clean · check:api-validation ✅ · check:react-query ✅. (apps/docs type-check noise is pre-existing — missing generated .source artifact; this branch touches no docs files.)
• Manual (local dev, three rounds of field-testing): uploads copy with their bytes and re-pointed references; branch forks carry pre-cut outputs with re-pointed embeds and leave post-cut files behind; duplicates open as <name> (Copy) with every message; reference-image generation actually uses the uploaded reference.
• Reviewers should focus on: the two-mode branch in fork/route.ts (isWholeChatDuplicate drives the message cut, file listing, title, Go body, and analytics — now a single workspace_files read cut in memory by filterForkableChatFiles); the ghost-resource drop rule in rewriteResourceFileRefs (chat-owned ∧ not-copied ⇒ dropped; shared workspace files pass through); the deliberate quota divergence from the workspace-fork precedent (forked bytes ARE counted — see the comment at the increment site); the write-once check ordering in resource-writer.ts (rejection must precede the headless redirect); and the two migrations (0254: additive, no backfill, NULL = birth-unknown ⇒ included in every fork; 0255: partial unique index, see deploy note).

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

Screenshots/Videos

Companion PR

Companion (mothership): https://github.com/simstudioai/mothership/pull/342

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Jul 4, 2026 8:21pm

[cursor]

PR Summary

High Risk
Changes file authorization and adds fork-time blob copying with quota accounting—both are security- and data-integrity sensitive; regressions could leak outputs across members or leave broken forks/resources.

Overview
Introduces chat-scoped output files end-to-end: list/preview APIs, resource picker and embedded-file resolution (workspace list → by-id → chat outputs by path), /api/files/view embeds for outputs, and promotion via materialize_file from outputs/ as well as uploads/.

Fork and duplicate share one route: optional upToMessageId for branch forks vs whole-chat copies. Copied uploads and outputs get new DB rows and physical blobs (quota-checked, post-transaction copy with cleanup on failure), transcript and resource references are rewritten, and ghost file chips for chat-owned files left behind on a branch cut are dropped. Sidebar Duplicate and message Fork wire through this flow; fork is hidden until the assistant message has a persisted id.

Authorization fixes a cross-member leak: output rows resolve by workspace-shaped keys but require the owning chat user, not workspace membership alone; tests lock this in for view and serve paths.

Supporting fixes: reject empty resource ids on add/send/reorder, roll back optimistic sends when the chat POST never starts a stream, treat stream resume 404 as terminal, stamp uploads with messageId for timeline cuts, and gate chat-scoped writes with interactive on execution context. Upload/output read logic is consolidated into chat-file-reader.

^{Reviewed by Cursor Bugbot for commit 4abf5d8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[greptile-apps]

Greptile Summary

This PR adds two related capabilities on the chat/VFS surface: a chat-scoped outputs/ namespace for agent-generated files (images, audio, video, ffmpeg results), and a Fork/Duplicate engine — branch forks keep messages up to a cut point while duplicates clone the whole chat. Both modes physically copy file bytes to fresh storage keys, rewrite all in-transcript and resource-chip references, and enforce storage-quota accounting. Several field bugs found during testing are also fixed (ghost resources on fork, live-message fork 400, media-tool VFS namespace misses, phantom-stream reconnect wedge).

• outputs/ namespace: uploadChatOutput persists agent-generated files as context='output' rows in workspace_files, exposed under the outputs/ VFS prefix and gated behind the chat-scoped-outputs mothership flag. Files are write-once, cascade-deleted with the chat, and require both workspace membership and owner identity for file-serve authorization — the latter enforced in both the Priority-1 DB lookup and the fallback path in authorization.ts.
• Fork/Duplicate engine: planChatFileCopies inserts fresh rows inside the transaction (so reference rewrites commit atomically), executeChatFileBlobCopies copies bytes post-commit with bounded concurrency (4) and a replay guard, and dead rows from any failed blob copy are hard-deleted rather than left listed. The two migrations are additive: 0254 adds nullable message_id and 0255 creates a CONCURRENTLY-built partial unique index preceded by an explicit COMMIT to exit any wrapping transaction.
• Resolver consolidation: resolveToolInputFile covers uploads/, outputs/, files/, and bare wf_ IDs (with a chat-scoped fallback for IDs the workspace resolver misses); the merged chat-file-reader replaces the duplicated upload-reader with a namespace-parameterized implementation. Unresolvable explicit references now fail the tool call instead of being silently skipped.

Confidence Score: 5/5

Safe to merge after the deploy note is actioned: verify no duplicate (chat_id, display_name) output rows exist before migration 0255 runs.

The fork/duplicate engine, outputs/ namespace, and authorization changes are all well-structured and internally consistent. The security-sensitive paths — output file ownership enforcement, quota gating, write-once rejection, and the chat-id FK guard for non-interactive runs — are each applied at the correct layer and covered by tests. The two migrations are additive and backward-compatible; the CONCURRENTLY index is properly sequenced with an explicit COMMIT. Test coverage is comprehensive (107 files, 1067 tests) and the PR self-documents all deliberate trade-offs.

The deploy note warrants attention before applying migration 0255 in production: a one-off query checking for existing duplicate (chat_id, display_name) output rows must run first, or the CONCURRENTLY build will leave an INVALID index.

Important Files Changed

Filename	Overview
apps/sim/app/api/mothership/chats/[chatId]/fork/route.ts	Extends fork to support whole-chat duplicate mode (upToMessageId optional); adds storage quota gate, blob copy with bounded concurrency, dead-row cleanup after blob failures, and ghost-resource drop via rewriteResourceFileRefs — all correctly sequenced around the transaction.
apps/sim/lib/copilot/chat/fork-chat-files.ts	New module implementing chat-file fork: list/filter helpers, transactional row insertion with id+key maps, and a bounded-concurrency (4) blob-copy worker pool with replay guard and per-task failure isolation. Logic and concurrency pattern are correct for single-threaded Node.
apps/sim/lib/copilot/chat/rewrite-file-references.ts	New module that rewrites all file references in a forked transcript (content text, content blocks, attachment chips, context chips) and resource list (with ghost-resource drop). Correctly delegates text rewriting to the existing fork remap grammar; pure, well-tested.
apps/sim/lib/copilot/vfs/resource-writer.ts	Adds outputs/ write path: write-once enforcement fires before the headless-redirect (correct order), interactive vs non-interactive distinction prevents FK inserts against ephemeral chat ids, and non-interactive calls fall through to a files/ write so assets still persist.
apps/sim/app/api/files/authorization.ts	Adds output file ownership enforcement: outputOwnershipSatisfied guard fires in both verifyWorkspaceFileAccess (Priority-1 path) and verifyRegularFileAccess (fallback path), so workspace membership alone cannot grant access to another user's agent outputs.
apps/sim/lib/copilot/tools/handlers/chat-file-reader.ts	New merged reader parameterized by namespace (uploads/outputs). Resolves by displayName with legacy originalName fallback for uploads, normalized canonical key for encoded names, and falls back to a full scan for visually-equivalent Unicode names.
apps/sim/lib/copilot/tools/server/files/resolve-input-file.ts	New unified resolver covering uploads/, outputs/, files/ paths, and bare wf_ ids (with chat-scoped fallback). Correctly tries workspace resolver first then falls back to chat-scoped by-id lookup for chat-owned files the workspace resolver misses.
packages/db/migrations/0255_chat_output_display_name_unique.sql	Creates a partial unique index CONCURRENTLY preceded by an explicit COMMIT to exit any wrapping transaction. Deploy note about pre-existing duplicate rows is correctly documented.
packages/db/schema.ts	Adds nullable message_id text column and partial unique index on (chat_id, display_name) for context='output'. Both additive and backward-compatible; NULL message_id has documented 'birth unknown = include in every fork' semantics.
apps/sim/app/workspace/[workspaceId]/home/hooks/use-chat.ts	Fixes: empty-id resource rejection at addResource and send-filter time; StreamNotFoundError breaks reconnect loops immediately; streamStarted flag gates post-error rollback to prevent rolling back a never-started stream.
apps/sim/lib/uploads/contexts/workspace/workspace-file-manager.ts	Adds uploadChatOutput (mirrors trackChatUpload for agent-generated files, context='output'), getPreviewableWorkspaceFile (ownership-enforced single-record fetch including outputs). Retry loop for display-name collisions correctly mirrors the uploads path.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Client
    participant ForkRoute as fork/route.ts
    participant DB as Database (tx)
    participant BlobStore as Storage
    participant GoService as Copilot Service (Go)

    Client->>ForkRoute: "POST /chats/[id]/fork {upToMessageId?}"

    Note over ForkRoute: Determine mode:<br/>upToMessageId → branch fork<br/>omitted → whole-chat duplicate

    ForkRoute->>DB: listDuplicableChatFiles (mothership + output)
    DB-->>ForkRoute: chatOwnedFiles
    ForkRoute->>ForkRoute: filterForkableChatFiles (branch) or use all (duplicate)
    ForkRoute->>ForkRoute: checkStorageQuota(totalBytes)

    ForkRoute->>DB: BEGIN TRANSACTION
    DB->>DB: INSERT copilotChats (new chat row)
    DB->>DB: planChatFileCopies → INSERT workspace_files (fresh ids + keys)
    DB->>DB: rewriteResourceFileRefs → UPDATE copilotChats.resources
    DB->>DB: appendCopilotChatMessages (rewritten transcript)
    DB->>DB: COMMIT

    ForkRoute->>BlobStore: "executeChatFileBlobCopies (concurrency=4)"
    Note over BlobStore: headObject replay guard → skip if already exists<br/>downloadFile → uploadFile → incrementStorageUsage

    alt blob copies failed
        ForkRoute->>DB: DELETE workspace_files WHERE id IN failedCopyIds
        ForkRoute->>DB: removeChatResources (drop dead chips)
    end

    ForkRoute->>GoService: "POST /api/chats/fork {sourceChatId, newChatId, upToMessageId?}"
    Note over GoService: Whole-chat: preserves compacted memory verbatim<br/>Branch: truncate-and-rebuild path

    ForkRoute-->>Client: "{success, id: newChatId, failedFileCopies?}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Client
    participant ForkRoute as fork/route.ts
    participant DB as Database (tx)
    participant BlobStore as Storage
    participant GoService as Copilot Service (Go)

    Client->>ForkRoute: "POST /chats/[id]/fork {upToMessageId?}"

    Note over ForkRoute: Determine mode:<br/>upToMessageId → branch fork<br/>omitted → whole-chat duplicate

    ForkRoute->>DB: listDuplicableChatFiles (mothership + output)
    DB-->>ForkRoute: chatOwnedFiles
    ForkRoute->>ForkRoute: filterForkableChatFiles (branch) or use all (duplicate)
    ForkRoute->>ForkRoute: checkStorageQuota(totalBytes)

    ForkRoute->>DB: BEGIN TRANSACTION
    DB->>DB: INSERT copilotChats (new chat row)
    DB->>DB: planChatFileCopies → INSERT workspace_files (fresh ids + keys)
    DB->>DB: rewriteResourceFileRefs → UPDATE copilotChats.resources
    DB->>DB: appendCopilotChatMessages (rewritten transcript)
    DB->>DB: COMMIT

    ForkRoute->>BlobStore: "executeChatFileBlobCopies (concurrency=4)"
    Note over BlobStore: headObject replay guard → skip if already exists<br/>downloadFile → uploadFile → incrementStorageUsage

    alt blob copies failed
        ForkRoute->>DB: DELETE workspace_files WHERE id IN failedCopyIds
        ForkRoute->>DB: removeChatResources (drop dead chips)
    end

    ForkRoute->>GoService: "POST /api/chats/fork {sourceChatId, newChatId, upToMessageId?}"
    Note over GoService: Whole-chat: preserves compacted memory verbatim<br/>Branch: truncate-and-rebuild path

    ForkRoute-->>Client: "{success, id: newChatId, failedFileCopies?}"

Loading

_{Reviews (5): Last reviewed commit: "fix(db): build the 0255 output-name inde..." | Re-trigger Greptile}

[j15z]

@greptile

[j15z]

@cursor review

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 9d8756c. Configure here.}

[j15z]

@greptile

[j15z]

@greptile

[j15z]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 4abf5d8. Configure here.}