Add Word document processor plugin

[mvansegbroeck]

What

Adds data-designer-docx, a processor plugin that renders each generated row as a
Microsoft Word document — headings, a front-matter metadata table, body sections
with bullets, a key-data table, page footer, and Word core properties.

The relative path of each file is written back into the dataset (docx_path by
default), so rows and documents stay joined.

Why

Data Designer produces rows. Document pipelines — enterprise RAG ingestion,
document classification, extraction evaluation, DLP tooling — consume .docx.
Today that gap is closed by a post-hoc script outside the config, which only
starts once generation finishes and is easy to forget to run.

Teams also need document corpora they are not allowed to obtain: the real policy
library lives in a customer's SharePoint. Generating them keeps the ground-truth
labels attached, because the sampler controls that produced each document are
already columns in the dataset.

Usage

import data_designer.config as dd
from data_designer_docx.config import DocxProcessorConfig
from data_designer_docx.schema import WordDocument

config_builder.add_column(
    dd.LLMStructuredColumnConfig(
        name="document",
        model_alias="doc-writer",
        output_format=WordDocument,
        prompt="Write an internal {{ doc_type }} for {{ company }} (document ID {{ doc_id }}).",
    )
)

config_builder.add_processor(
    DocxProcessorConfig(
        name="word-documents",
        document_column="document",
        filename_template="{{ doc_id }}-{{ doc_type }}.docx",
        metadata_columns={"Document ID": "{{ doc_id }}", "Company": "{{ company }}"},
        core_property_columns={"author": "{{ owner }}", "category": "{{ doc_type }}"},
        footer_template="{{ company }} · {{ doc_id }}",
        template_path="brand/corporate-template.docx",  # optional corporate styles
    )
)

Files land in <artifact_path>/<dataset>/documents/word-documents/.

How

One model, used twice. WordDocument is both the output_format of the LLM
structured column and the input contract of the renderer. Because both ends share
one definition, "the model produced something the renderer can't handle" becomes a
Pydantic validation error on the column — which Data Designer already retries —
instead of a parsing failure downstream. There is no markdown parsing anywhere in
the package: the LLM generates the document's structure, and the renderer walks it.

Post-batch, not after-generation. Documents stream out while the run is still
going, the row count stays fixed as the async engine requires at that stage, and
the dataset stays resumable. process_after_generation rewrites the final parquet
and marks the dataset terminal for resume.

Output location. Documents go to <output_subdir>/<name>/, never under
processors-files/. Data Designer reads every directory there back as a parquet
dataset, so .docx files placed there make preview() fail with "Parquet magic
bytes not found in footer". A config validator rejects the reserved names. This
was found the hard way and is covered by a regression test.

Ragged tables are repaired, not retried. Structured outputs constrain the shape
of the JSON, not the arithmetic inside it — a model asked for a three-column table
occasionally returns a row with two cells. Rows are padded or truncated to the
header width, since a retry costs a whole document generation.

Layout. schema.py holds the contract, render.py is pure python-docx with no
Data Designer imports (so layout can be iterated without spending tokens),
config.py is user-facing, impl.py is engine-side.

A row whose document fails validation is skipped with a null path and a warning;
the rest of the batch still renders.

Validation

make lint       ruff check + format --check, all passed
make test       17 passed for data-designer-docx, all plugins green
make validate   OK: docx
make check      generated metadata + SPDX headers up to date
make all        exit 0

The plugin's own suite covers the config validator, filename sanitization
(including path traversal), ragged-table normalization, rendering (headings,
tables, footers, core properties, template style inheritance), and three
integration tests that run the processor through preview() and create() using
a seeded document column — so the suite needs no API key or model access.

make plugin-docs regenerated the site docs; make codeowners regenerated
.github/CODEOWNERS.

Not registered in catalog/plugins.json — that is a first-release step, and no
release or publish is being requested here.

🤖 Generated with Claude Code

[nabinchha]

Thanks for putting this together, @mvansegbroeck — the renderer/processor split is thoughtful, and the PR does a nice job explaining why DOCX belongs in the generation pipeline.

Summary

This PR adds a data-designer-docx processor that renders structured generated rows into Word documents while retaining a dataset-relative path for each row. The implementation broadly matches the stated intent, but after ten focused review passes I found three correctness/data-safety issues and two API/rendering gaps worth addressing before merge.

Findings

Critical — Let's fix these before merge

plugins/data-designer-docx/src/data_designer_docx/impl.py:178 — Preserve structured document strings

• What: deserialize_json_values recursively processes the entire record before WordDocument validation. Data Designer's structured generators already place decoded mappings in generated rows, so legitimate string leaves that look like JSON scalars are converted: "30" becomes 30, "true" becomes True, and "null" becomes None.
• Why: WordDocument.model_validate() rejects those converted values for str and list[str] fields. I reproduced this with a generated document mapping containing key_data.rows=[["30"]]; the processor skipped the row, returned docx_path=None, and wrote no DOCX file. This is especially likely in the key-data tables the plugin is designed to produce.
• Suggestion: Could we preserve an already-decoded document mapping and only JSON-decode document_column when its top-level value is a serialized JSON string? If recursive decoding is useful for Jinja templates, we can prepare a separate template-rendering record without mutating the value passed to parse_document.

plugins/data-designer-docx/src/data_designer_docx/config.py:78 and impl.py:55 — Keep output paths inside the dataset

• What: output_dir joins two configuration-controlled strings, output_subdir and inherited processor name, while the validator only compares output_subdir.strip("/") against three literal names. Values such as ../outside, /private/tmp/outside, ./processors-files, foo/../parquet-files, tmp-partial-parquet-files, and name="../../outside" are accepted.
• Why: These values can escape base_dataset_path, bypass the managed-directory guard, or place documents in directories Data Designer reads or deletes. In particular, Data Designer removes tmp-partial-parquet-files during resume, which would leave persisted docx_path values pointing to deleted documents.
• Suggestion: Could we validate both path components, reject every Data Designer-managed directory, and resolve the final output path before writing to assert that it remains beneath base_dataset_path? Treating name as a single safe directory component would also close the second traversal route.

plugins/data-designer-docx/src/data_designer_docx/impl.py:50 — Preserve filename collisions across resume

• What: _used_filenames starts empty for every processor instance and is never seeded from files already present in output_dir. A resumed run creates a new processor instance, so new rows can reuse filenames owned by completed batches.
• Why: I reproduced this by creating one row with filename_template="same.docx" and resuming the dataset to two rows. Both rows ended up with documents/docs/same.docx, and only one file remained on disk—the resumed row overwrote the completed row's document. The current set also treats A.docx and a.docx as different even though they collide on default macOS and Windows filesystems.
• Suggestion: Could we seed collision tracking from existing output files and compare portable, case-folded collision keys? A deterministic batch/row component or exclusive atomic creation would make resume behavior even safer and avoid overwrites after partial failures.

Warnings — Worth addressing

plugins/data-designer-docx/src/data_designer_docx/config.py:51 — Make the default filename self-contained

• What: The only required data field is document_column, but the default filename template references {{ doc_id }}.
• Why: A valid minimal configuration without an unrelated doc_id column fails in post-batch template preparation with UserTemplateError, after generation work has already completed. The default configuration therefore is not usable on its own.
• Suggestion: Could we use a deterministic default that requires no additional dataset column, or make the filename source explicit and validate its references before generation starts?

plugins/data-designer-docx/src/data_designer_docx/render.py:142 — Apply the footer to generated content

• What: Generated content is appended to the template's final section, while footer_template modifies only doc.sections[0].
• Why: With a multi-section template whose final footer is not linked to the first, the generated pages never receive the configured footer. In a two-section reproduction, the footer values were ['override', 'last'], and the generated content belonged to the final section displaying last.
• Suggestion: Could we apply the configured footer to the final generated section, or to every section if the intended contract is a document-wide footer?

What Looks Good

• The separation between schema.py, the engine-independent renderer, user-facing config, and processor implementation is clean and makes the package easy to evolve.
• The PR explains the process_after_batch stage choice and the processors-files constraint clearly, including the failure mode that motivated the output layout.
• The baseline test suite covers readable DOCX output, tables, styles, core properties, invalid documents, preview/create integration, and path resolution without requiring a model or API key.
• Repository verification is green: all GitHub checks pass; locally, make lint, make test (338 tests), make validate, make check, and strict make docs passed. The wheel and sdist also built successfully and passed twine check.

Verdict

Needs changes. Before merge, I think we should address:

• preservation of string values in decoded structured mappings;
• containment and validation of output paths;
• collision-safe resume behavior;
• the undeclared doc_id dependency in the default filename; and
• footer targeting for generated content in template documents.

---

This review was generated by an AI assistant.

[mvansegbroeck]

Thanks @nabinchha — all five are real, and the first one is a good catch: I reproduced the coercion before fixing it.

rows after deserialize: [[30, True]]
validate: FAILED -> key_data.rows.0.0: Input should be a valid string [input_value=30, input_type=int]

Pushed in 1b6f24c. Test count went 17 → 39, with regression coverage for each finding.

Critical

1. Preserve structured document strings — impl.py

Took the suggested split. parse_document now reads from the raw record and only JSON-decodes when the top-level value is a str; mappings validate as-is. The recursively decoded copy is built separately and used for Jinja templates only, where the rewriting is harmless.

raw_records = data.to_dict(orient="records")
template_records = [deserialize_json_values(record) for record in raw_records]
options = self.render_options(template_records, columns)
...
document = self.parse_document(record.get(self.config.document_column))  # raw

Covered by TestStructuredStringPreservation, which drives a key-data table of ["30", "true", "null"] through preview() and asserts the rendered cells come back as those exact strings.

2. Contain output paths — config.py, impl.py

Both components are now validated: output_subdir as a relative multi-segment path, name as a single directory component. Rejects absolutes, ./.. in any segment, and reserved names per-segment — so ./processors-files, foo/../parquet-files, nested/images and name="../../outside" all fail at config time.

Added tmp-partial-parquet-files and images to the reserved set. The former was the sharpest one you flagged: Data Designer deletes it on resume, which would leave every persisted docx_path pointing at nothing.

output_dir also resolves and asserts containment beneath base_dataset_path before writing, as defence-in-depth against symlinked roots that validation can't see.

3. Resume-safe collisions — impl.py

seed_used_filenames() adopts the names already in output_dir on first use, and collisions compare casefold() keys. Your same.docx reproduction is now TestFilenameCollisions::test_seeds_from_existing_files, plus a case-insensitivity test.

I did not add a deterministic batch/row component or exclusive atomic creation. Seeding closes the overwrite you reproduced, and atomic creation is a bigger change I'd rather not fold into this PR — happy to follow up if you'd like belt-and-braces here.

Warnings

4. Self-contained default filename — config.py

Default is now document.docx, which references no dataset columns; duplicates get the numeric suffix. A minimal DocxProcessorConfig(name=..., document_column=...) now works standalone.

I looked at the other half of your suggestion — validating template references before generation — but ProcessorConfig has no required_columns hook to declare them, so there's nowhere to check before the data exists. Worth a separate conversation upstream if pre-flight validation for processors is wanted generally.

5. Footer targeting — render.py

New apply_footer() sets the footer on every section and clears is_linked_to_previous so unlinked sections actually take it. I went document-wide rather than last-section-only: footer_template reads as a document-level setting, and applying it everywhere avoids the generated pages disagreeing with the front matter. Your two-section reproduction is now TestFooterTargeting.

Validation

make lint, make test (39 for this plugin), make validate, make check, make plugin-docs, make all — all green locally. Docs updated with the new defaults and the three behavioural notes.