diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index cc75377..06fa2d8 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -7,6 +7,7 @@ /.github/ @NVIDIA-NeMo/data_designer_reviewers # Plugins +/plugins/data-designer-docx/ @NVIDIA-NeMo/data_designer_reviewers @mvansegbroeck /plugins/data-designer-github/ @NVIDIA-NeMo/data_designer_reviewers @eric-tramel /plugins/data-designer-retrieval-sdg/ @NVIDIA-NeMo/data_designer_reviewers @shan-nvidia @oliverholworthy /plugins/data-designer-template/ @NVIDIA-NeMo/data_designer_reviewers diff --git a/docs/plugins/data-designer-docx/index.md b/docs/plugins/data-designer-docx/index.md new file mode 100644 index 0000000..aa00b98 --- /dev/null +++ b/docs/plugins/data-designer-docx/index.md @@ -0,0 +1,92 @@ +# data-designer-docx + +Renders Data Designer rows as Microsoft Word documents. Registers a `docx` +processor that writes one `.docx` per row and records each file's path back into +the dataset, so rows and documents stay joined. + +## Installation + +```bash +uv add data-designer data-designer-docx +``` + +## How it works + +The plugin ships a Pydantic model, `WordDocument`, that is used twice: as the +`output_format` of an LLM structured column, and as the input contract of the +renderer. Because both ends share one definition, a model that emits something +unrenderable produces a validation error on the column — which Data Designer +already knows how to retry — instead of a parsing failure downstream. + +That is also why the package contains no markdown parsing. The LLM generates the +document's *structure*, and the renderer walks it. + +```text +llm-structured(document: WordDocument) -> processor(docx) -> documents//*.docx +``` + +## Configuration + +| Field | Required | Description | +| --- | --- | --- | +| `name` | Yes | Processor name; also the output subfolder. | +| `document_column` | Yes | Column holding a `WordDocument`-shaped value. | +| `output_subdir` | No | Folder under the dataset directory. Defaults to `documents`. Must be relative and must not name a Data Designer-managed directory. | +| `filename_template` | No | Jinja2 template for the file name. Defaults to `document.docx`, which references no dataset columns; duplicates get a numeric suffix. | +| `output_path_column` | No | Column receiving the written path. Defaults to `docx_path`. | +| `metadata_columns` | No | Label to Jinja2 template pairs, rendered as a front-matter table. | +| `core_property_columns` | No | Word core property (`author`, `category`, `subject`, `keywords`) to Jinja2 template pairs. | +| `template_path` | No | A `.docx` supplying corporate styles, header, and footer. | +| `footer_template` | No | Jinja2 template for the page footer, applied to every section. | +| `table_style` | No | Table style name; must exist in the template. Defaults to `Table Grid`. | +| `number_sections` | No | Prefix section headings with `1.`, `2.`, and so on. Defaults to `True`. | + +## Implementation notes + +**Stage choice.** The processor implements `process_after_batch` rather than +`process_after_generation`. Documents then stream out while the run is still in +progress, the row count stays fixed as the async engine requires at that stage, +and the dataset stays resumable — `process_after_generation` rewrites the final +parquet and marks the dataset terminal for resume. + +**Output location.** Documents are written to `//`, never +under `processors-files/`. Both components are validated as contained, relative, +non-reserved path segments, and the resolved directory is asserted to sit beneath +the dataset directory before anything is written. Data Designer reads every directory there back as a +parquet dataset, so `.docx` files placed there make `preview()` fail with +*"Parquet magic bytes not found in footer"*. Binary artifacts get their own +folder, the same way generated images live under `images/`. The config validator +rejects the reserved names. + +**Ragged tables.** Structured outputs constrain the shape of the JSON, not the +arithmetic inside it — a model asked for a three-column table will occasionally +return a row with two cells. Rows are padded or truncated to the header width +rather than triggering a retry, which would cost a whole document generation. + +**Package layout.** `schema.py` holds the contract, `render.py` is pure +python-docx with no Data Designer imports (so layout can be iterated without +spending tokens), `config.py` is user-facing, and `impl.py` is engine-side. + +## Templates + +`template_path` should point at a `.docx` containing styles, header, and footer +but **no body content** — python-docx appends generated content after anything +already in the file, so a template with a cover page yields a cover page on every +document. + +**Structured values are not re-decoded.** Data Designer's recursive JSON decoding +rewrites string leaves that look like scalars, turning `"30"` into `30` and +`"true"` into `True` — precisely the values a key-data table carries, and values +the schema then rejects. The processor therefore validates the document column +from the raw record and only JSON-decodes it when the top-level value is a +string. The recursively decoded copy is used for Jinja templates only. + +**Resume safety.** File name collisions are tracked with case-folded keys, since +macOS and Windows treat `A.docx` and `a.docx` as the same file, and the set is +seeded from documents already on disk. A resumed run therefore cannot overwrite a +document written by a batch that completed before the resume. + +**Footers.** Generated content is appended after any body content the template +already has, so it lands in the template's final section. `footer_template` is +applied to every section rather than just the first, which would otherwise leave +the generated pages showing the template's own footer. diff --git a/docs/plugins/data-designer-docx/usage.md b/docs/plugins/data-designer-docx/usage.md new file mode 100644 index 0000000..cbec1de --- /dev/null +++ b/docs/plugins/data-designer-docx/usage.md @@ -0,0 +1,97 @@ +# Usage + +A complete pipeline: samplers describe the corpus, one structured column writes +each document, and the processor renders them. + +```python +import data_designer.config as dd +from data_designer.interface import DataDesigner + +from data_designer_docx.config import DocxProcessorConfig +from data_designer_docx.schema import WordDocument + +MODEL_ALIAS = "doc-writer" + +config_builder = dd.DataDesignerConfigBuilder( + model_configs=[ + dd.ModelConfig( + alias=MODEL_ALIAS, + model="nvidia/nemotron-3-super-120b-a12b", + provider="nvidia", + # A whole document in one call is a long structured generation. + inference_parameters=dd.ChatCompletionInferenceParams(max_tokens=8192), + ) + ] +) + +config_builder.add_column( + dd.SamplerColumnConfig( + name="doc_id", + sampler_type=dd.SamplerType.UUID, + params=dd.UUIDSamplerParams(prefix="POL-", short_form=True, uppercase=True), + ) +) +config_builder.add_column( + dd.SamplerColumnConfig( + name="doc_type", + sampler_type=dd.SamplerType.CATEGORY, + params=dd.CategorySamplerParams(values=["Remote Work Policy", "Incident Response Runbook"]), + ) +) + +config_builder.add_column( + dd.LLMStructuredColumnConfig( + name="document", + model_alias=MODEL_ALIAS, + output_format=WordDocument, + prompt=( + "Write an internal {{ doc_type }} (document ID {{ doc_id }}). " + "Write like a real corporate policy: flat, procedural, no marketing language." + ), + ) +) + +config_builder.add_processor( + DocxProcessorConfig( + name="word-documents", + document_column="document", + filename_template="{{ doc_id }}-{{ doc_type }}.docx", + metadata_columns={"Document ID": "{{ doc_id }}"}, + footer_template="{{ doc_id }}", + ) +) + +results = DataDesigner().create(config_builder, num_records=10, dataset_name="policies") +dataset = results.load_dataset() +``` + +## Reading the output + +`docx_path` is relative to the dataset directory, which keeps the dataset +portable — move the folder and the paths still resolve. + +```python +from docx import Document + +path = results.artifact_storage.base_dataset_path / dataset["docx_path"].iloc[0] +rendered = Document(str(path)) + +print(rendered.core_properties.author) +print(rendered.sections[0].footer.paragraphs[0].text) +``` + +## Customizing the document shape + +`WordDocument` describes a title, subtitle, summary, sections, and one key-data +table. To change what gets generated, subclass or replace it and pass your model +as the column's `output_format`; the renderer only requires the fields it reads. + +The `Field(description=...)` strings on that model are not documentation. Data +Designer serializes the JSON Schema into the prompt inside `` +tags, so those descriptions are prompt text the model reads — the fastest lever +for changing output quality. + +## Rows without a valid document + +A row whose document column fails validation is skipped: its `docx_path` is +null, a warning is logged, and the rest of the batch still renders. diff --git a/docs/plugins/index.md b/docs/plugins/index.md index a3b4bcd..6473736 100644 --- a/docs/plugins/index.md +++ b/docs/plugins/index.md @@ -5,6 +5,17 @@ Browse available Data Designer plugins by what they add to your data generation workflow.
+ + + data-designer-docx + v0.1.0 + + Data Designer processor plugin that renders generated rows as Microsoft Word documents + + Entry points + docx + + data-designer-github diff --git a/plugins/data-designer-docx/CODEOWNERS b/plugins/data-designer-docx/CODEOWNERS new file mode 100644 index 0000000..da999d6 --- /dev/null +++ b/plugins/data-designer-docx/CODEOWNERS @@ -0,0 +1,3 @@ +# Owner(s) of this plugin — used to generate the root CODEOWNERS file. +# GitHub accepts @username, @org/team, or email format. +* @NVIDIA-NeMo/data_designer_reviewers @mvansegbroeck diff --git a/plugins/data-designer-docx/README.md b/plugins/data-designer-docx/README.md new file mode 100644 index 0000000..fb3d3b4 --- /dev/null +++ b/plugins/data-designer-docx/README.md @@ -0,0 +1,46 @@ +# data-designer-docx + +Render Data Designer rows as Microsoft Word documents. + +Data Designer generates rows; document pipelines consume `.docx`. This plugin +closes that gap with a processor that writes one Word document per row — with +headings, tables, a front-matter metadata table, footers, and Word core +properties — while keeping the file path joined to its row in the dataset. + +## Installation + +```bash +uv add data-designer data-designer-docx +``` + +## Usage + +```python +import data_designer.config as dd +from data_designer_docx.config import DocxProcessorConfig +from data_designer_docx.schema import WordDocument + +config_builder.add_column( + dd.LLMStructuredColumnConfig( + name="document", + model_alias="doc-writer", + output_format=WordDocument, + prompt="Write an internal {{ doc_type }} for {{ company }}.", + ) +) + +config_builder.add_processor( + DocxProcessorConfig( + name="word-documents", + document_column="document", + filename_template="{{ doc_id }}-{{ doc_type }}.docx", + metadata_columns={"Document ID": "{{ doc_id }}", "Company": "{{ company }}"}, + footer_template="{{ company }} · {{ doc_id }}", + ) +) +``` + +Files land in `//documents/word-documents/`, and the +relative path of each one is written back into the dataset as `docx_path`. + +See [`docs/`](docs/) for the full field reference and design notes. diff --git a/plugins/data-designer-docx/docs/index.md b/plugins/data-designer-docx/docs/index.md new file mode 100644 index 0000000..aa00b98 --- /dev/null +++ b/plugins/data-designer-docx/docs/index.md @@ -0,0 +1,92 @@ +# data-designer-docx + +Renders Data Designer rows as Microsoft Word documents. Registers a `docx` +processor that writes one `.docx` per row and records each file's path back into +the dataset, so rows and documents stay joined. + +## Installation + +```bash +uv add data-designer data-designer-docx +``` + +## How it works + +The plugin ships a Pydantic model, `WordDocument`, that is used twice: as the +`output_format` of an LLM structured column, and as the input contract of the +renderer. Because both ends share one definition, a model that emits something +unrenderable produces a validation error on the column — which Data Designer +already knows how to retry — instead of a parsing failure downstream. + +That is also why the package contains no markdown parsing. The LLM generates the +document's *structure*, and the renderer walks it. + +```text +llm-structured(document: WordDocument) -> processor(docx) -> documents//*.docx +``` + +## Configuration + +| Field | Required | Description | +| --- | --- | --- | +| `name` | Yes | Processor name; also the output subfolder. | +| `document_column` | Yes | Column holding a `WordDocument`-shaped value. | +| `output_subdir` | No | Folder under the dataset directory. Defaults to `documents`. Must be relative and must not name a Data Designer-managed directory. | +| `filename_template` | No | Jinja2 template for the file name. Defaults to `document.docx`, which references no dataset columns; duplicates get a numeric suffix. | +| `output_path_column` | No | Column receiving the written path. Defaults to `docx_path`. | +| `metadata_columns` | No | Label to Jinja2 template pairs, rendered as a front-matter table. | +| `core_property_columns` | No | Word core property (`author`, `category`, `subject`, `keywords`) to Jinja2 template pairs. | +| `template_path` | No | A `.docx` supplying corporate styles, header, and footer. | +| `footer_template` | No | Jinja2 template for the page footer, applied to every section. | +| `table_style` | No | Table style name; must exist in the template. Defaults to `Table Grid`. | +| `number_sections` | No | Prefix section headings with `1.`, `2.`, and so on. Defaults to `True`. | + +## Implementation notes + +**Stage choice.** The processor implements `process_after_batch` rather than +`process_after_generation`. Documents then stream out while the run is still in +progress, the row count stays fixed as the async engine requires at that stage, +and the dataset stays resumable — `process_after_generation` rewrites the final +parquet and marks the dataset terminal for resume. + +**Output location.** Documents are written to `//`, never +under `processors-files/`. Both components are validated as contained, relative, +non-reserved path segments, and the resolved directory is asserted to sit beneath +the dataset directory before anything is written. Data Designer reads every directory there back as a +parquet dataset, so `.docx` files placed there make `preview()` fail with +*"Parquet magic bytes not found in footer"*. Binary artifacts get their own +folder, the same way generated images live under `images/`. The config validator +rejects the reserved names. + +**Ragged tables.** Structured outputs constrain the shape of the JSON, not the +arithmetic inside it — a model asked for a three-column table will occasionally +return a row with two cells. Rows are padded or truncated to the header width +rather than triggering a retry, which would cost a whole document generation. + +**Package layout.** `schema.py` holds the contract, `render.py` is pure +python-docx with no Data Designer imports (so layout can be iterated without +spending tokens), `config.py` is user-facing, and `impl.py` is engine-side. + +## Templates + +`template_path` should point at a `.docx` containing styles, header, and footer +but **no body content** — python-docx appends generated content after anything +already in the file, so a template with a cover page yields a cover page on every +document. + +**Structured values are not re-decoded.** Data Designer's recursive JSON decoding +rewrites string leaves that look like scalars, turning `"30"` into `30` and +`"true"` into `True` — precisely the values a key-data table carries, and values +the schema then rejects. The processor therefore validates the document column +from the raw record and only JSON-decodes it when the top-level value is a +string. The recursively decoded copy is used for Jinja templates only. + +**Resume safety.** File name collisions are tracked with case-folded keys, since +macOS and Windows treat `A.docx` and `a.docx` as the same file, and the set is +seeded from documents already on disk. A resumed run therefore cannot overwrite a +document written by a batch that completed before the resume. + +**Footers.** Generated content is appended after any body content the template +already has, so it lands in the template's final section. `footer_template` is +applied to every section rather than just the first, which would otherwise leave +the generated pages showing the template's own footer. diff --git a/plugins/data-designer-docx/docs/usage.md b/plugins/data-designer-docx/docs/usage.md new file mode 100644 index 0000000..cbec1de --- /dev/null +++ b/plugins/data-designer-docx/docs/usage.md @@ -0,0 +1,97 @@ +# Usage + +A complete pipeline: samplers describe the corpus, one structured column writes +each document, and the processor renders them. + +```python +import data_designer.config as dd +from data_designer.interface import DataDesigner + +from data_designer_docx.config import DocxProcessorConfig +from data_designer_docx.schema import WordDocument + +MODEL_ALIAS = "doc-writer" + +config_builder = dd.DataDesignerConfigBuilder( + model_configs=[ + dd.ModelConfig( + alias=MODEL_ALIAS, + model="nvidia/nemotron-3-super-120b-a12b", + provider="nvidia", + # A whole document in one call is a long structured generation. + inference_parameters=dd.ChatCompletionInferenceParams(max_tokens=8192), + ) + ] +) + +config_builder.add_column( + dd.SamplerColumnConfig( + name="doc_id", + sampler_type=dd.SamplerType.UUID, + params=dd.UUIDSamplerParams(prefix="POL-", short_form=True, uppercase=True), + ) +) +config_builder.add_column( + dd.SamplerColumnConfig( + name="doc_type", + sampler_type=dd.SamplerType.CATEGORY, + params=dd.CategorySamplerParams(values=["Remote Work Policy", "Incident Response Runbook"]), + ) +) + +config_builder.add_column( + dd.LLMStructuredColumnConfig( + name="document", + model_alias=MODEL_ALIAS, + output_format=WordDocument, + prompt=( + "Write an internal {{ doc_type }} (document ID {{ doc_id }}). " + "Write like a real corporate policy: flat, procedural, no marketing language." + ), + ) +) + +config_builder.add_processor( + DocxProcessorConfig( + name="word-documents", + document_column="document", + filename_template="{{ doc_id }}-{{ doc_type }}.docx", + metadata_columns={"Document ID": "{{ doc_id }}"}, + footer_template="{{ doc_id }}", + ) +) + +results = DataDesigner().create(config_builder, num_records=10, dataset_name="policies") +dataset = results.load_dataset() +``` + +## Reading the output + +`docx_path` is relative to the dataset directory, which keeps the dataset +portable — move the folder and the paths still resolve. + +```python +from docx import Document + +path = results.artifact_storage.base_dataset_path / dataset["docx_path"].iloc[0] +rendered = Document(str(path)) + +print(rendered.core_properties.author) +print(rendered.sections[0].footer.paragraphs[0].text) +``` + +## Customizing the document shape + +`WordDocument` describes a title, subtitle, summary, sections, and one key-data +table. To change what gets generated, subclass or replace it and pass your model +as the column's `output_format`; the renderer only requires the fields it reads. + +The `Field(description=...)` strings on that model are not documentation. Data +Designer serializes the JSON Schema into the prompt inside `` +tags, so those descriptions are prompt text the model reads — the fastest lever +for changing output quality. + +## Rows without a valid document + +A row whose document column fails validation is skipped: its `docx_path` is +null, a warning is logged, and the rest of the batch still renders. diff --git a/plugins/data-designer-docx/pyproject.toml b/plugins/data-designer-docx/pyproject.toml new file mode 100644 index 0000000..ba6eeba --- /dev/null +++ b/plugins/data-designer-docx/pyproject.toml @@ -0,0 +1,37 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +[project] +name = "data-designer-docx" +version = "0.1.0" +description = "Data Designer processor plugin that renders generated rows as Microsoft Word documents" +requires-python = ">=3.10" +dependencies = [ + "data-designer>=0.9.1", + "python-docx>=1.1.0", +] +license = "Apache-2.0" +readme = "README.md" +authors = [ + {name = "NVIDIA Corporation"}, +] +classifiers = [ + "Development Status :: 3 - Alpha", + "Programming Language :: Python :: 3", +] + +[project.entry-points."data_designer.plugins"] +docx = "data_designer_docx.plugin:plugin" + +[project.urls] +Repository = "https://github.com/NVIDIA-NeMo/DataDesignerPlugins" + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" + +[tool.hatch.build.targets.wheel] +packages = ["src/data_designer_docx"] + +[tool.ruff] +extend = "../../pyproject.toml" diff --git a/plugins/data-designer-docx/src/data_designer_docx/__init__.py b/plugins/data-designer-docx/src/data_designer_docx/__init__.py new file mode 100644 index 0000000..52a7a9d --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/__init__.py @@ -0,0 +1,2 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 diff --git a/plugins/data-designer-docx/src/data_designer_docx/config.py b/plugins/data-designer-docx/src/data_designer_docx/config.py new file mode 100644 index 0000000..aeec538 --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/config.py @@ -0,0 +1,145 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from typing import Literal + +from data_designer.config.base import ProcessorConfig +from pydantic import Field, field_validator + +# Directories Data Designer creates, reads, or deletes inside a dataset folder. +# Hardcoded rather than imported so this module stays free of engine imports; the +# processor additionally asserts containment against the live artifact storage at +# write time. Keep in sync with data_designer.engine.storage.artifact_storage. +RESERVED_DIRECTORY_NAMES = frozenset( + { + "parquet-files", + "processors-files", + "dropped-columns-parquet-files", + "tmp-partial-parquet-files", + "images", + } +) + +DEFAULT_FILENAME_TEMPLATE = "document.docx" + + +def validate_path_component(value: str, *, field: str, allow_nested: bool) -> str: + """Reject path values that escape the dataset directory or shadow managed folders. + + Args: + value: The configured path fragment. + field: Field name, used in error messages. + allow_nested: Whether ``/`` separated segments are permitted. + + Returns: + The normalized value. + + Raises: + ValueError: If the value is absolute, contains traversal segments, or names + a Data Designer-managed directory. + """ + if not value or not value.strip(): + raise ValueError(f"{field} must not be empty.") + if value.startswith("/") or (len(value) > 1 and value[1] == ":"): + raise ValueError(f"{field}={value!r} must be relative to the dataset directory, not absolute.") + + segments = [segment for segment in value.split("/") if segment] + if not allow_nested and len(segments) > 1: + raise ValueError(f"{field}={value!r} must be a single directory name, not a nested path.") + + for segment in segments: + if segment in {".", ".."}: + raise ValueError(f"{field}={value!r} must not contain '.' or '..' path segments.") + if segment in RESERVED_DIRECTORY_NAMES: + raise ValueError( + f"{field}={value!r} uses the Data Designer-managed directory {segment!r}. " + "Data Designer reads or deletes these folders, so documents written there " + "can be misread as parquet or removed on resume." + ) + return "/".join(segments) + + +class DocxProcessorConfig(ProcessorConfig): + """Renders one Microsoft Word document per row. + + Files are written to ``////`` and + the relative path of each file is stored in ``output_path_column`` so rows and + documents stay joined. + + Documents are *not* written under ``processors-files/``. Data Designer reads + every directory there back as a parquet dataset, so binary artifacts need + their own folder, in the same way generated images live under ``images/``. + Both ``output_subdir`` and ``name`` are validated as contained, non-reserved + path components. + + Attributes: + document_column: Column holding a ``WordDocument``-shaped value, typically + produced by an ``LLMStructuredColumnConfig`` using ``WordDocument`` as + its ``output_format``. + output_subdir: Folder under the dataset directory that documents are + written to. Must be relative and must not name a managed directory. + filename_template: Jinja2 template for the file name, rendered per row and + sanitized. The default references no dataset columns; collisions are + resolved by appending ``-1``, ``-2``, and so on. + output_path_column: Name of the column that receives the written path. + metadata_columns: Label to Jinja2 template pairs rendered as a front-matter + table beneath the document title. + core_property_columns: Word core property name (``author``, ``category``, + ``comments``, ``subject``, ``keywords``) to Jinja2 template pairs. + template_path: Optional ``.docx`` supplying styles, header, and footer. The + template should contain styles only; python-docx appends generated + content after any body content already present. + footer_template: Optional Jinja2 template for the page footer, applied to + every section of the rendered document. + table_style: Table style name, which must exist in the template document. + number_sections: Whether to prefix section headings with ``1.``, ``2.``, and so on. + """ + + processor_type: Literal["docx"] = "docx" + + document_column: str = Field(description="Column containing the structured document.") + output_subdir: str = Field( + default="documents", + description="Folder under the dataset directory that .docx files are written to.", + ) + filename_template: str = Field( + default=DEFAULT_FILENAME_TEMPLATE, + description=( + "Jinja2 template for the output file name. The default references no dataset columns; " + "duplicates are de-duplicated with a numeric suffix." + ), + ) + output_path_column: str = Field( + default="docx_path", + description="Column that receives the relative path of the written file.", + ) + metadata_columns: dict[str, str] = Field( + default_factory=dict, + description="Label to Jinja2 template pairs rendered as a front-matter table.", + ) + core_property_columns: dict[str, str] = Field( + default_factory=dict, + description="Word core property name to Jinja2 template pairs.", + ) + template_path: str | None = Field( + default=None, + description="Optional .docx template supplying styles, header, and footer.", + ) + footer_template: str | None = Field( + default=None, + description="Optional Jinja2 template for the page footer.", + ) + table_style: str = Field(default="Table Grid", description="Table style name.") + number_sections: bool = Field(default=True, description="Number section headings.") + + @field_validator("output_subdir") + @classmethod + def validate_output_subdir(cls, value: str) -> str: + """Ensure the output folder stays inside the dataset and avoids managed names.""" + return validate_path_component(value, field="output_subdir", allow_nested=True) + + @field_validator("name") + @classmethod + def validate_name_is_safe_directory(cls, value: str) -> str: + """The processor name becomes a directory, so it must be a single safe component.""" + return validate_path_component(value, field="name", allow_nested=False) diff --git a/plugins/data-designer-docx/src/data_designer_docx/impl.py b/plugins/data-designer-docx/src/data_designer_docx/impl.py new file mode 100644 index 0000000..57597ec --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/impl.py @@ -0,0 +1,267 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from __future__ import annotations + +import json +import logging +from collections.abc import Mapping +from pathlib import Path +from typing import TYPE_CHECKING, Any + +from data_designer.engine.processing.ginja.environment import WithJinja2UserTemplateRendering +from data_designer.engine.processing.processors.base import Processor +from data_designer.engine.processing.utils import deserialize_json_values +from pydantic import ValidationError + +from data_designer_docx.config import DocxProcessorConfig +from data_designer_docx.render import render_document, safe_filename +from data_designer_docx.schema import WordDocument + +if TYPE_CHECKING: + import pandas as pd + +logger = logging.getLogger(__name__) + +DOCX_SUFFIX = ".docx" + + +def to_text(value: Any) -> str: + """Convert an interpolated Jinja2 value to text. + + Used as the ``record_str_fn`` finalize hook so ``None`` renders as an empty + string rather than the literal ``"None"`` in file names and metadata. + + Args: + value: The value being interpolated into a template. + + Returns: + The value as a string, or an empty string when the value is ``None``. + """ + return "" if value is None else str(value) + + +def collision_key(filename: str) -> str: + """Build a portable key for detecting file name collisions. + + macOS and Windows filesystems are case-insensitive by default, so ``A.docx`` + and ``a.docx`` are the same file even though the strings differ. + + Args: + filename: A sanitized file name. + + Returns: + A case-folded key suitable for collision comparison. + """ + return filename.casefold() + + +class DocxProcessor(WithJinja2UserTemplateRendering, Processor[DocxProcessorConfig]): + """Writes one ``.docx`` file per row as each batch completes. + + Runs at the post-batch stage rather than after generation so that documents + stream out while the run is still in progress, the row count stays fixed as + the async engine requires, and the dataset remains resumable. + """ + + def _initialize(self) -> None: + """Reset the record of file names already written.""" + self._used_filenames: set[str] = set() + self._seeded_from_disk = False + + @property + def output_dir(self) -> Path: + """Directory that documents for this processor are written to. + + Raises: + ValueError: If the configured path would resolve outside the dataset + directory. The config validators already reject traversal, so this + is a defence-in-depth check against symlinked or unusual roots. + """ + base = self.base_dataset_path.resolve() + candidate = (base / self.config.output_subdir / self.config.name).resolve() + if base != candidate and base not in candidate.parents: + raise ValueError( + f"Refusing to write documents to {candidate}, which is outside the dataset " + f"directory {base}. Check output_subdir and the processor name." + ) + return candidate + + def relative_path(self, filename: str) -> str: + """Build the dataset-relative path stored in the output column. + + Args: + filename: The file name written to disk. + + Returns: + A path relative to the dataset directory, keeping the dataset portable. + """ + return f"{self.config.output_subdir}/{self.config.name}/{filename}" + + def seed_used_filenames(self, output_dir: Path) -> None: + """Adopt file names already on disk so a resumed run cannot overwrite them. + + A resumed run builds a fresh processor instance with an empty collision set. + Without this, rows generated after the resume can reuse names owned by + batches that completed before it. + + Args: + output_dir: The directory documents are written to. + """ + if self._seeded_from_disk: + return + if output_dir.is_dir(): + existing = {collision_key(path.name) for path in output_dir.glob(f"*{DOCX_SUFFIX}")} + self._used_filenames.update(existing) + if existing: + logger.debug(f"Seeded {len(existing)} existing document name(s) from {output_dir}.") + self._seeded_from_disk = True + + def render_for_all_records(self, template: str, columns: list[str], records: list[dict]) -> list[str]: + """Render one Jinja2 template across every record in a batch. + + The renderer is prepared once per template rather than once per row, + because building the sandboxed environment is the expensive part. + + Args: + template: A user-supplied Jinja2 template. + columns: Column names allowed as template references. + records: The batch records to render against. + + Returns: + One rendered string per record, in batch order. + """ + self.prepare_jinja2_template_renderer(template, columns, record_str_fn=to_text) + return [self.render_template(record) for record in records] + + def unique_filename(self, rendered: str) -> str: + """Sanitize a rendered file name and make it unique across the dataset. + + Args: + rendered: The raw rendered file name. + + Returns: + A filesystem-safe name, suffixed with ``-1``, ``-2``, and so on if needed. + """ + filename = safe_filename(rendered) + if collision_key(filename) not in self._used_filenames: + self._used_filenames.add(collision_key(filename)) + return filename + stem = filename[: -len(DOCX_SUFFIX)] + counter = 1 + while collision_key(f"{stem}-{counter}{DOCX_SUFFIX}") in self._used_filenames: + counter += 1 + deduped = f"{stem}-{counter}{DOCX_SUFFIX}" + self._used_filenames.add(collision_key(deduped)) + return deduped + + def parse_document(self, value: Any) -> WordDocument | None: + """Parse a structured column value into a :class:`WordDocument`. + + Only a top-level JSON string is decoded. Mappings are validated as-is: the + engine's recursive JSON decoding would coerce legitimate string leaves such + as ``"30"`` or ``"true"`` into ``int`` and ``bool``, which the schema then + rejects — exactly the values that show up in key-data tables. + + Args: + value: A JSON string or already-decoded mapping from the dataset. + + Returns: + The parsed document, or ``None`` when the value is missing or invalid. + """ + if value is None: + return None + try: + if isinstance(value, str): + return WordDocument.model_validate_json(value) + if isinstance(value, Mapping): + return WordDocument.model_validate(value) + return WordDocument.model_validate(json.loads(json.dumps(value))) + except (ValidationError, ValueError, TypeError) as exc: + logger.warning(f"⚠️ Skipping row: {self.config.document_column!r} is not a valid document ({exc}).") + return None + + def render_options(self, records: list[dict], columns: list[str]) -> dict[str, Any]: + """Pre-render every configured template across the batch. + + Args: + records: The batch records, with nested JSON decoded for template access. + columns: Column names allowed as template references. + + Returns: + A mapping with rendered ``filenames``, ``metadata``, ``core_properties``, + and ``footers`` entries. + """ + return { + "filenames": self.render_for_all_records(self.config.filename_template, columns, records), + "metadata": { + label: self.render_for_all_records(template, columns, records) + for label, template in self.config.metadata_columns.items() + }, + "core_properties": { + prop: self.render_for_all_records(template, columns, records) + for prop, template in self.config.core_property_columns.items() + }, + "footers": ( + self.render_for_all_records(self.config.footer_template, columns, records) + if self.config.footer_template + else None + ), + } + + def process_after_batch(self, data: pd.DataFrame, *, current_batch_number: int | None) -> pd.DataFrame: + """Render each row of a completed batch to a ``.docx`` file. + + Args: + data: The generated batch data. + current_batch_number: The batch index, or ``None`` in preview mode. + + Returns: + The batch with the output path column added. + + Raises: + ValueError: If the configured document column is not in the dataset. + """ + if data.empty and self.config.document_column not in data.columns: + logger.warning("⚠️ Empty batch reached the docx processor; no documents written.") + return data + + if self.config.document_column not in data.columns: + raise ValueError( + f"Column {self.config.document_column!r} not found in the dataset. " + f"Available columns: {sorted(data.columns)}" + ) + + columns = data.columns.to_list() + # Documents are read from the raw records; only the template-facing copy is + # recursively JSON-decoded, since that decoding rewrites string leaves. + raw_records = data.to_dict(orient="records") + template_records = [deserialize_json_values(record) for record in raw_records] + options = self.render_options(template_records, columns) + + output_dir = self.output_dir + output_dir.mkdir(parents=True, exist_ok=True) + self.seed_used_filenames(output_dir) + + written: list[str | None] = [] + for row, record in enumerate(raw_records): + document = self.parse_document(record.get(self.config.document_column)) + if document is None: + written.append(None) + continue + filename = self.unique_filename(options["filenames"][row]) + render_document( + document, + output_dir / filename, + metadata={label: values[row] for label, values in options["metadata"].items()}, + template_path=self.config.template_path, + footer_text=options["footers"][row] if options["footers"] else None, + table_style=self.config.table_style, + number_sections=self.config.number_sections, + core_properties={prop: values[row] for prop, values in options["core_properties"].items()}, + ) + written.append(self.relative_path(filename)) + + data[self.config.output_path_column] = written + logger.info(f"📄 Wrote {sum(path is not None for path in written)} .docx file(s) to {output_dir}") + return data diff --git a/plugins/data-designer-docx/src/data_designer_docx/plugin.py b/plugins/data-designer-docx/src/data_designer_docx/plugin.py new file mode 100644 index 0000000..110a92d --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/plugin.py @@ -0,0 +1,10 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +from data_designer.plugins.plugin import Plugin, PluginType + +plugin = Plugin( + config_qualified_name="data_designer_docx.config.DocxProcessorConfig", + impl_qualified_name="data_designer_docx.impl.DocxProcessor", + plugin_type=PluginType.PROCESSOR, +) diff --git a/plugins/data-designer-docx/src/data_designer_docx/render.py b/plugins/data-designer-docx/src/data_designer_docx/render.py new file mode 100644 index 0000000..10352e3 --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/render.py @@ -0,0 +1,175 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""Pure ``.docx`` rendering. No Data Designer imports on purpose. + +Keeping the renderer free of engine imports means you can iterate on layout in a +REPL against a hand-written ``WordDocument`` and never spend a token. +""" + +from __future__ import annotations + +import re +from pathlib import Path +from typing import Any + +from docx import Document +from docx.enum.text import WD_ALIGN_PARAGRAPH +from docx.shared import Pt + +from data_designer_docx.schema import DocTable, WordDocument + +UNSAFE_FILENAME_CHARS = re.compile(r"[^A-Za-z0-9._-]+") + +# "Table Grid" ships with the python-docx default template. Corporate templates +# usually define their own; see `table_style` on the processor config. +DEFAULT_TABLE_STYLE = "Table Grid" + + +def safe_filename(name: str, *, suffix: str = ".docx", max_length: int = 120) -> str: + """Turn a rendered filename template into something safe to write to disk.""" + stem = UNSAFE_FILENAME_CHARS.sub("-", name.strip()).strip("-._") + if stem.lower().endswith(suffix.lower()): + stem = stem[: -len(suffix)] + if not stem: + stem = "document" + return stem[:max_length] + suffix + + +def normalize_rows(table: DocTable) -> list[list[str]]: + """Pad or truncate every row to the header width. + + Structured outputs constrain the *shape* of the JSON, not the arithmetic + inside it — a model asked for four columns will occasionally hand back a row + with three cells. Fixing it here is cheaper than a retry. + """ + width = len(table.columns) + normalized = [] + for row in table.rows: + cells = [str(cell) for cell in row][:width] + cells.extend([""] * (width - len(cells))) + normalized.append(cells) + return normalized + + +def add_metadata_table(doc: Any, metadata: dict[str, str], style: str) -> None: + table = doc.add_table(rows=0, cols=2) + table.style = style + for key, value in metadata.items(): + row = table.add_row().cells + row[0].text = str(key) + row[1].text = str(value) + for paragraph in row[0].paragraphs: + for run in paragraph.runs: + run.bold = True + doc.add_paragraph() + + +def add_data_table(doc: Any, table: DocTable, style: str) -> None: + doc.add_heading(table.caption, level=2) + rendered = doc.add_table(rows=1, cols=max(len(table.columns), 1)) + rendered.style = style + header_cells = rendered.rows[0].cells + for idx, column in enumerate(table.columns): + header_cells[idx].text = str(column) + for paragraph in header_cells[idx].paragraphs: + for run in paragraph.runs: + run.bold = True + for row in normalize_rows(table): + cells = rendered.add_row().cells + for idx, value in enumerate(row): + cells[idx].text = value + doc.add_paragraph() + + +def apply_footer(doc: Any, footer_text: str) -> None: + """Set the footer on every section of the document. + + Generated content is appended after whatever the template already contains, so + it lands in the template's *final* section. Writing only to ``sections[0]`` + leaves the generated pages showing the template's footer whenever the sections + are not linked. ``footer_template`` is a document-wide setting, so it is applied + to all sections. + + Args: + doc: The python-docx ``Document`` being rendered. + footer_text: The rendered footer text. + """ + for section in doc.sections: + # A section that inherits from the previous one cannot hold its own text. + section.footer.is_linked_to_previous = False + paragraph = section.footer.paragraphs[0] if section.footer.paragraphs else section.footer.add_paragraph() + paragraph.text = footer_text + paragraph.alignment = WD_ALIGN_PARAGRAPH.CENTER + + +def render_document( + document: WordDocument, + output_path: str | Path, + *, + metadata: dict[str, str] | None = None, + template_path: str | Path | None = None, + footer_text: str | None = None, + table_style: str = DEFAULT_TABLE_STYLE, + number_sections: bool = True, + core_properties: dict[str, str] | None = None, +) -> Path: + """Render a :class:`WordDocument` to a ``.docx`` file. + + Args: + document: The structured document to render. + output_path: Where to write the ``.docx``. + metadata: Optional key/value pairs rendered as a front-matter table. + template_path: Optional ``.docx`` whose styles, header, and footer are + inherited. The template should contain styles only — any body + content in it will appear above the generated content. + footer_text: Optional footer applied to the first section. + table_style: Table style name. Must exist in the (template) document. + number_sections: Prefix section headings with ``1.``, ``2.``, ... + core_properties: Optional Word core properties (author, category, ...). + + Returns: + The path that was written. + """ + output_path = Path(output_path) + output_path.parent.mkdir(parents=True, exist_ok=True) + + doc = Document(str(template_path)) if template_path else Document() + + doc.add_heading(document.title, level=0) + + subtitle = doc.add_paragraph(document.subtitle) + subtitle.alignment = WD_ALIGN_PARAGRAPH.LEFT + for run in subtitle.runs: + run.italic = True + run.font.size = Pt(12) + + if metadata: + add_metadata_table(doc, metadata, table_style) + + doc.add_heading("Summary", level=1) + doc.add_paragraph(document.summary) + + for index, section in enumerate(document.sections, start=1): + heading = f"{index}. {section.heading}" if number_sections else section.heading + doc.add_heading(heading, level=1) + for paragraph in section.paragraphs: + doc.add_paragraph(paragraph) + for bullet in section.bullets: + doc.add_paragraph(bullet, style="List Bullet") + + add_data_table(doc, document.key_data, table_style) + + if footer_text: + apply_footer(doc, footer_text) + + # Word core properties travel with the file. Downstream extraction and + # classification pipelines read them, so it is worth filling them in. + if core_properties: + props = doc.core_properties + for key, value in core_properties.items(): + if hasattr(props, key) and value is not None: + setattr(props, key, str(value)) + + doc.save(str(output_path)) + return output_path diff --git a/plugins/data-designer-docx/src/data_designer_docx/schema.py b/plugins/data-designer-docx/src/data_designer_docx/schema.py new file mode 100644 index 0000000..95e5931 --- /dev/null +++ b/plugins/data-designer-docx/src/data_designer_docx/schema.py @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +"""The document model. + +This module holds the single most important idea in this example: one Pydantic +model is used twice. + +1. As the ``output_format`` of an ``LLMStructuredColumnConfig``, so the LLM is + constrained to emit a valid document *outline* rather than a wall of prose. +2. As the input contract of the ``.docx`` renderer. + +Because both ends share the model, "the LLM produced something the renderer +can't handle" stops being a class of bug you have to defend against in the +renderer with regexes and heuristics. +""" + +from __future__ import annotations + +from pydantic import BaseModel, Field + + +class DocTable(BaseModel): + """A simple rectangular table. + + Rows are ragged in practice — LLMs drop or add a cell now and then — so the + renderer normalizes each row against ``columns`` rather than trusting it. + """ + + caption: str = Field(description="Short caption describing what the table contains.") + columns: list[str] = Field(description="Column headers, 2 to 4 of them.") + rows: list[list[str]] = Field(description="Table rows. Each row has one cell per column header.") + + +class DocSection(BaseModel): + """One numbered section of the document.""" + + heading: str = Field(description="Section heading, title case, no numbering prefix.") + paragraphs: list[str] = Field( + description="One to three body paragraphs of prose. No markdown, no bullet characters." + ) + bullets: list[str] = Field( + default_factory=list, + description=( + "Optional bulleted requirements or steps for this section. Use an empty list when the " + "section reads better as prose only." + ), + ) + + +class WordDocument(BaseModel): + """A complete business document, structured for rendering.""" + + title: str = Field(description="Document title.") + subtitle: str = Field(description="One-line subtitle, e.g. the scope or the owning function.") + summary: str = Field(description="A single paragraph executive summary, 40-80 words.") + sections: list[DocSection] = Field(description="Four to six sections that make up the body.") + key_data: DocTable = Field( + description=( + "A table carrying the document's structured facts — thresholds, review cadences, " + "roles and responsibilities, retention windows, or similar." + ) + ) diff --git a/plugins/data-designer-docx/tests/test_plugin.py b/plugins/data-designer-docx/tests/test_plugin.py new file mode 100644 index 0000000..85e80ac --- /dev/null +++ b/plugins/data-designer-docx/tests/test_plugin.py @@ -0,0 +1,341 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import json +from pathlib import Path + +import pandas as pd +import pytest +from data_designer.config import ExpressionColumnConfig +from data_designer.config.config_builder import DataDesignerConfigBuilder +from data_designer.config.seed_source_dataframe import DataFrameSeedSource +from data_designer.engine.testing.utils import assert_valid_plugin +from data_designer.interface.data_designer import DataDesigner +from docx import Document +from pydantic import ValidationError + +from data_designer_docx.config import DocxProcessorConfig +from data_designer_docx.impl import DocxProcessor +from data_designer_docx.plugin import plugin +from data_designer_docx.render import normalize_rows, render_document, safe_filename +from data_designer_docx.schema import DocSection, DocTable, WordDocument + + +def test_valid_plugin() -> None: + assert_valid_plugin(plugin) + + +def make_document(title: str = "Access Control Standard") -> WordDocument: + """Build a small but complete document for rendering tests.""" + return WordDocument( + title=title, + subtitle="Information Security", + summary="This standard defines how access is granted, reviewed, and revoked.", + sections=[ + DocSection(heading="Scope", paragraphs=["Applies to all systems."], bullets=["Production"]), + DocSection(heading="Roles", paragraphs=["The owner attests quarterly."]), + ], + key_data=DocTable( + caption="Review Cadence", + columns=["Role", "Action", "Frequency"], + rows=[["Owner", "Attest", "Quarterly"]], + ), + ) + + +class BoundDocxProcessor(DocxProcessor): + """A processor bound to an explicit dataset path. + + Lets the collision and path-containment logic be tested directly, without + standing up a ResourceProvider. + """ + + _base_dataset_path: Path + + @property + def base_dataset_path(self) -> Path: + return self._base_dataset_path + + +def build_processor(tmp_path: Path, **overrides: object) -> BoundDocxProcessor: + """Construct a BoundDocxProcessor over a temporary dataset directory.""" + dataset_path = tmp_path / "dataset" + dataset_path.mkdir(exist_ok=True) + + processor = BoundDocxProcessor.__new__(BoundDocxProcessor) + processor._base_dataset_path = dataset_path + processor._config = DocxProcessorConfig(name="docs", document_column="document", **overrides) + processor._initialize() + return processor + + +class TestDocxProcessorConfig: + def test_defaults(self) -> None: + config = DocxProcessorConfig(name="docs", document_column="document") + assert config.processor_type == "docx" + assert config.output_subdir == "documents" + assert config.output_path_column == "docx_path" + + def test_default_filename_needs_no_dataset_column(self) -> None: + """A minimal config must be renderable without inventing an unrelated id column.""" + config = DocxProcessorConfig(name="docs", document_column="document") + assert "{{" not in config.filename_template + + @pytest.mark.parametrize( + "reserved", + [ + "processors-files", + "parquet-files", + "dropped-columns-parquet-files", + # Deleted by Data Designer on resume, which would orphan every docx_path. + "tmp-partial-parquet-files", + "images", + "./processors-files", + "foo/../parquet-files", + "nested/images", + ], + ) + def test_reserved_output_subdir_is_rejected(self, reserved: str) -> None: + """Data Designer reads or deletes these folders, so documents must not go there.""" + with pytest.raises(ValidationError): + DocxProcessorConfig(name="docs", document_column="document", output_subdir=reserved) + + @pytest.mark.parametrize("escaping", ["../outside", "/private/tmp/outside", "a/../../outside", ""]) + def test_escaping_output_subdir_is_rejected(self, escaping: str) -> None: + with pytest.raises(ValidationError): + DocxProcessorConfig(name="docs", document_column="document", output_subdir=escaping) + + @pytest.mark.parametrize("bad_name", ["../../outside", "/abs", "nested/name", "..", "processors-files"]) + def test_unsafe_processor_name_is_rejected(self, bad_name: str) -> None: + """The processor name becomes a directory, so it is a traversal route too.""" + with pytest.raises(ValidationError): + DocxProcessorConfig(name=bad_name, document_column="document") + + def test_nested_output_subdir_is_allowed(self) -> None: + config = DocxProcessorConfig(name="docs", document_column="document", output_subdir="out/word") + assert config.output_subdir == "out/word" + + +class TestSafeFilename: + @pytest.mark.parametrize( + ("raw", "expected"), + [ + ("POL-1 Access Control", "POL-1-Access-Control.docx"), + ("already.docx", "already.docx"), + ("../../etc/passwd", "etc-passwd.docx"), + ("///", "document.docx"), + ], + ) + def test_sanitizes(self, raw: str, expected: str) -> None: + assert safe_filename(raw) == expected + + +class TestNormalizeRows: + def test_pads_short_rows(self) -> None: + """Structured outputs constrain JSON shape, not cell counts, so rows arrive ragged.""" + table = DocTable(caption="c", columns=["a", "b", "c"], rows=[["1", "2"]]) + assert normalize_rows(table) == [["1", "2", ""]] + + def test_truncates_long_rows(self) -> None: + table = DocTable(caption="c", columns=["a", "b"], rows=[["1", "2", "3"]]) + assert normalize_rows(table) == [["1", "2"]] + + +class TestRenderDocument: + def test_writes_readable_docx(self, tmp_path: Path) -> None: + path = render_document( + make_document(), + tmp_path / "out.docx", + metadata={"Document ID": "POL-1"}, + footer_text="Internal", + core_properties={"author": "Jane Doe", "category": "Standard"}, + ) + + rendered = Document(str(path)) + headings = [p.text for p in rendered.paragraphs if p.style.name.startswith(("Title", "Heading"))] + assert headings[0] == "Access Control Standard" + assert "1. Scope" in headings + assert rendered.core_properties.author == "Jane Doe" + assert rendered.sections[0].footer.paragraphs[0].text == "Internal" + + def test_metadata_and_key_data_tables(self, tmp_path: Path) -> None: + path = render_document(make_document(), tmp_path / "out.docx", metadata={"Owner": "Jane"}) + + rendered = Document(str(path)) + assert len(rendered.tables) == 2 + assert [cell.text for cell in rendered.tables[1].rows[0].cells] == ["Role", "Action", "Frequency"] + + def test_number_sections_disabled(self, tmp_path: Path) -> None: + path = render_document(make_document(), tmp_path / "out.docx", number_sections=False) + + rendered = Document(str(path)) + headings = [p.text for p in rendered.paragraphs if p.style.name.startswith("Heading")] + assert "Scope" in headings + assert "1. Scope" not in headings + + def test_template_styles_are_inherited(self, tmp_path: Path) -> None: + template_path = tmp_path / "template.docx" + template = Document() + template.styles["Normal"].font.name = "Georgia" + template.save(str(template_path)) + + path = render_document(make_document(), tmp_path / "out.docx", template_path=template_path) + + assert Document(str(path)).styles["Normal"].font.name == "Georgia" + + +class TestDocxProcessorPreviewIntegration: + """Run the processor through Data Designer using seed data, so no model is needed.""" + + @pytest.fixture() + def seed_df(self) -> pd.DataFrame: + return pd.DataFrame( + { + "doc_id": ["POL-1", "POL-2"], + "document": [make_document("First").model_dump_json(), make_document("Second").model_dump_json()], + } + ) + + def build(self, seed_df: pd.DataFrame, **overrides: object) -> DataDesignerConfigBuilder: + builder = DataDesignerConfigBuilder() + builder.with_seed_dataset(DataFrameSeedSource(df=seed_df)) + # Data Designer's profiler requires at least one generated column, and a + # seed-plus-processor config has none. + builder.add_column(ExpressionColumnConfig(name="doc_label", expr="{{ doc_id }}")) + builder.add_processor( + DocxProcessorConfig( + name="word-documents", + document_column="document", + filename_template="{{ doc_id }}.docx", + **overrides, + ) + ) + return builder + + def test_preview_writes_documents(self, seed_df: pd.DataFrame, tmp_path: Path) -> None: + artifact_path = tmp_path / "artifacts" + artifact_path.mkdir() + + result = DataDesigner(artifact_path=artifact_path).preview(self.build(seed_df), num_records=2) + + assert "docx_path" in result.dataset.columns + paths = sorted(artifact_path.rglob("*.docx")) + assert [path.name for path in paths] == ["POL-1.docx", "POL-2.docx"] + assert Document(str(paths[0])).paragraphs[0].text == "First" + + def test_output_path_column_resolves(self, seed_df: pd.DataFrame, tmp_path: Path) -> None: + """The stored path is dataset-relative, keeping the dataset portable. + + Uses create() rather than preview() because only DatasetCreationResults + exposes artifact_storage, which is how a caller resolves the path. + """ + artifact_path = tmp_path / "artifacts" + artifact_path.mkdir() + + results = DataDesigner(artifact_path=artifact_path).create( + self.build(seed_df), num_records=2, dataset_name="documents-test" + ) + + relative = results.load_dataset()["docx_path"].iloc[0] + assert relative.startswith("documents/word-documents/") + assert (results.artifact_storage.base_dataset_path / relative).is_file() + + def test_invalid_document_is_skipped(self, tmp_path: Path) -> None: + """A malformed row yields a null path instead of failing the whole batch.""" + artifact_path = tmp_path / "artifacts" + artifact_path.mkdir() + seed_df = pd.DataFrame( + { + "doc_id": ["POL-1", "POL-2"], + "document": [make_document("Good").model_dump_json(), json.dumps({"nope": True})], + } + ) + + result = DataDesigner(artifact_path=artifact_path).preview(self.build(seed_df), num_records=2) + + assert result.dataset["docx_path"].isna().sum() == 1 + assert len(list(artifact_path.rglob("*.docx"))) == 1 + + +class TestStructuredStringPreservation: + """Regression tests for JSON-decoding of already-decoded document mappings. + + Data Designer's recursive `deserialize_json_values` rewrites string leaves that + look like JSON scalars, so `"30"` becomes `30` and `"true"` becomes `True`. + Those values are exactly what key-data tables carry, and `WordDocument` rejects + them, which silently dropped the row. + """ + + def numeric_document(self) -> dict: + document = make_document().model_dump() + document["key_data"]["columns"] = ["Threshold", "Enabled", "Notes"] + document["key_data"]["rows"] = [["30", "true", "null"]] + return document + + def test_scalar_looking_strings_survive_as_mapping(self, tmp_path: Path) -> None: + artifact_path = tmp_path / "artifacts" + artifact_path.mkdir() + seed_df = pd.DataFrame({"doc_id": ["POL-1"], "document": [json.dumps(self.numeric_document())]}) + + builder = DataDesignerConfigBuilder() + builder.with_seed_dataset(DataFrameSeedSource(df=seed_df)) + builder.add_column(ExpressionColumnConfig(name="doc_label", expr="{{ doc_id }}")) + builder.add_processor( + DocxProcessorConfig(name="docs", document_column="document", filename_template="{{ doc_id }}.docx") + ) + + result = DataDesigner(artifact_path=artifact_path).preview(builder, num_records=1) + + assert result.dataset["docx_path"].notna().all(), "row was skipped instead of rendered" + written = sorted(artifact_path.rglob("*.docx")) + assert len(written) == 1 + cells = [cell.text for cell in Document(str(written[0])).tables[-1].rows[1].cells] + assert cells == ["30", "true", "null"], f"string leaves were coerced: {cells}" + + +class TestFilenameCollisions: + def make_processor_dir(self, tmp_path: Path) -> Path: + output_dir = tmp_path / "documents" / "docs" + output_dir.mkdir(parents=True) + return output_dir + + def test_case_insensitive_collision(self, tmp_path: Path) -> None: + """macOS and Windows treat A.docx and a.docx as the same file.""" + processor = build_processor(tmp_path, filename_template="{{ doc_id }}.docx") + processor.seed_used_filenames(self.make_processor_dir(tmp_path)) + + assert processor.unique_filename("Report") == "Report.docx" + assert processor.unique_filename("report") == "report-1.docx" + + def test_seeds_from_existing_files(self, tmp_path: Path) -> None: + """A resumed run must not overwrite documents written before the resume.""" + output_dir = self.make_processor_dir(tmp_path) + (output_dir / "same.docx").write_bytes(b"existing") + + processor = build_processor(tmp_path, filename_template="same.docx") + processor.seed_used_filenames(output_dir) + + assert processor.unique_filename("same") == "same-1.docx" + + def test_output_dir_stays_inside_dataset(self, tmp_path: Path) -> None: + processor = build_processor(tmp_path) + assert processor.output_dir.is_relative_to((tmp_path / "dataset").resolve()) + + +class TestFooterTargeting: + def test_footer_applies_to_every_section(self, tmp_path: Path) -> None: + """Generated content lands in the template's last section, not the first.""" + template_path = tmp_path / "two-section-template.docx" + template = Document() + template.add_section() + for index, section in enumerate(template.sections): + section.footer.is_linked_to_previous = False + section.footer.paragraphs[0].text = f"template-{index}" + template.save(str(template_path)) + + path = render_document( + make_document(), tmp_path / "out.docx", template_path=template_path, footer_text="override" + ) + + footers = [section.footer.paragraphs[0].text for section in Document(str(path)).sections] + assert set(footers) == {"override"}, footers diff --git a/uv.lock b/uv.lock index bd619b8..9be0c98 100644 --- a/uv.lock +++ b/uv.lock @@ -12,6 +12,7 @@ resolution-markers = [ [manifest] members = [ + "data-designer-docx", "data-designer-github", "data-designer-plugins-workspace", "data-designer-retrieval-sdg", @@ -437,6 +438,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/37/f8/a12b44092eaabb2fa431474eef069885c6d4aed576677d7c344095051733/data_designer_config-0.9.1-py3-none-any.whl", hash = "sha256:8944578a74f28980b7b06b5b12c5e866da313af3fe6b5cc5d446bedf5756744e", size = 128696, upload-time = "2026-08-11T20:24:12.669Z" }, ] +[[package]] +name = "data-designer-docx" +version = "0.1.0" +source = { editable = "plugins/data-designer-docx" } +dependencies = [ + { name = "data-designer" }, + { name = "python-docx" }, +] + +[package.metadata] +requires-dist = [ + { name = "data-designer", specifier = ">=0.9.1" }, + { name = "python-docx", specifier = ">=1.1.0" }, +] + [[package]] name = "data-designer-engine" version = "0.9.1" @@ -2153,6 +2169,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, ] +[[package]] +name = "python-docx" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a9/f7/eddfe33871520adab45aaa1a71f0402a2252050c14c7e3009446c8f4701c/python_docx-1.2.0.tar.gz", hash = "sha256:7bc9d7b7d8a69c9c02ca09216118c86552704edc23bac179283f2e38f86220ce", size = 5723256, upload-time = "2025-06-16T20:46:27.921Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d0/00/1e03a4989fa5795da308cd774f05b704ace555a70f9bf9d3be057b680bcf/python_docx-1.2.0-py3-none-any.whl", hash = "sha256:3fd478f3250fbbbfd3b94fe1e985955737c145627498896a8a6bf81f4baf66c7", size = 252987, upload-time = "2025-06-16T20:46:22.506Z" }, +] + [[package]] name = "python-dotenv" version = "1.2.2" diff --git a/zensical.toml b/zensical.toml index ac6b49c..014b79b 100644 --- a/zensical.toml +++ b/zensical.toml @@ -21,6 +21,10 @@ nav = [ {"Plugins" = [ {"Overview" = "plugins/index.md"}, # BEGIN GENERATED PLUGIN DOCS NAV + {"data-designer-docx" = [ + {"Overview" = "plugins/data-designer-docx/index.md"}, + {"Usage" = "plugins/data-designer-docx/usage.md"}, + ]}, {"data-designer-github" = [ {"Overview" = "plugins/data-designer-github/index.md"}, {"Usage" = "plugins/data-designer-github/usage.md"},