[P1] Add formatting-aware document ingestion for strikethrough and font color

[paullizer]

Summary

Add formatting-aware ingestion for uploaded documents so SimpleChat can answer questions about strikethrough, font color, highlight, and similar visual/style signals from indexed document context instead of only using flattened extracted text.

Priority: P1
Owner: Paul Lizer (@paullizer)
Size: L
Related: #1137

User Value

Users need to ask questions such as:

• "Which line items were struck through?"
• "Which report values are red, green, or blue?"
• "Which rows were marked as deleted or revised?"

Today the indexed document content generally loses these formatting signals. The assistant correctly has to say it cannot definitively identify struck-through or color-coded text from the extracted report content. Uploading an image can sometimes work through vision, but that is not reliable or scalable for long-term document search/RAG.

Required Product Behavior

• This capability requires Enhanced Citations because it depends on retaining the original source file and colocating durable sidecar metadata with that source artifact.
• When Enhanced Citations is disabled, formatting-aware sidecar extraction is disabled and status/UI messaging should make clear that formatting-aware ingestion requires Enhanced Citations.
• When Enhanced Citations is enabled, supported file types automatically get formatting-aware ingestion during upload and reprocess. Users should not need a separate per-upload toggle.
• The formatting sidecar JSON must be stored beside the preserved source file in the same enhanced-citations blob artifact folder, scoped by the document/revision GUID.
• The parent Cosmos document record stores only a small sidecar manifest, counts, status, schema version, and blob path/pointer.
• AI Search receives compact retrieval projections, and when AI Search returns a result for a document that has a sidecar, the result/context assembly hydrates bounded relevant sidecar facts from the colocated sidecar JSON.
• Users should be able to pick an existing file and rerun formatting extraction so older enhanced-citation documents can gain the sidecar and search projection without requiring a fresh upload.

Definitions

Formatting sidecar metadata

A formatting sidecar is structured JSON produced during ingestion or re-extraction. It is not a normal document text chunk, and it is not live Python execution during chat. It records sparse, meaningful formatting facts extracted from the original file.

The sidecar should not store every character run or every inherited style. It should store only signals that matter for retrieval and answers, such as struck-through spans, red text, highlighted text, deleted/revision text, and enough location context to cite or explain the finding.

Formatting fact

A formatting fact is one extracted style signal, for example:

{
  "fact_id": "fmt_00042",
  "type": "strikethrough",
  "text": "$42,000 legacy services",
  "surrounding_text": "Cloud migration costs: $42,000 legacy services $38,500 revised estimate",
  "location": {
    "page_number": null,
    "paragraph_index": 18,
    "table_index": 2,
    "row_index": 5,
    "cell_index": 3
  },
  "confidence": 0.98,
  "extractor": "docx_ooxml"
}

Retrieval projection

A retrieval projection is a compact text version of formatting facts that is indexed in AI Search so normal chat retrieval can find the facts. The full sidecar is for durable metadata and diagnostics; the retrieval projection is what makes prompts like "which line items were struck through?" work.

Example projected text:

Formatting facts:
- Struck-through text: "$42,000 legacy services" in table 2 row 5.
- Red text: "High risk" in paragraph 24.

Current Behavior

• Upload processing is dispatched through process_document_upload_background().
• DOCX, PDF, image, and PowerPoint-style files generally flow through process_di_document() and extract_content_with_azure_di(), producing page/chunk text strings for AI Search.
• DOCM and legacy DOC can flow through process_doc() / extract_word_text(), which also flatten content to text.
• Enhanced Citations stores the preserved source blob and records source blob metadata on the parent document record.
• Search chunks currently store chunk_text, page/chunk identifiers, metadata fields, embeddings, and citation context, but no normalized style metadata for formatting-specific questions.
• Parent document metadata in Cosmos stores useful document-level fields such as abstract, keywords, title, authors, and status, but it should not be used as an unbounded store for full formatting facts.

Enhanced Citations Requirement

This feature is an Enhanced Citations capability. It should only run when Enhanced Citations is enabled, because the sidecar must be saved beside the original source file and must remain available when the document is recalled for citations or reprocessing.

Target blob layout should be document-GUID scoped, for example:

<scope>/<document_id>/<original_filename>
<scope>/<document_id>/formatting_sidecar.json

For revised or archived documents, use the existing enhanced-citations revision/document folder convention and keep the sidecar next to the source file for that same document version.

If the current source-file blob layout is not already document-GUID scoped for all document states, this feature should either align with the existing revision/document folder convention or introduce a deterministic sidecar path derived from the parent document's persisted blob_path. The sidecar path must be stored in the parent manifest so retrieval and re-extraction do not guess paths.

Storage And Retrieval Architecture

Use three separate layers so the data is durable, bounded, and retrievable.

1. Cosmos parent document metadata: small manifest only

Store a small formatting manifest on the existing parent document record, similar to how abstract/keywords live with document metadata.

Example:

{
  "has_formatting_metadata": true,
  "formatting_metadata_version": "1.0",
  "formatting_metadata_status": "extracted",
  "formatting_fact_counts": {
    "strikethrough": 12,
    "font_color": 8,
    "highlight": 3
  },
  "formatting_metadata_storage": {
    "type": "blob",
    "path": "<scope>/<document_id>/formatting_sidecar.json"
  }
}

Do not store the full sidecar JSON in the parent Cosmos item. Azure Cosmos DB for NoSQL has a 2 MB maximum item size, and a heavily revised or color-coded document could exceed that if every fact is embedded into the parent metadata. The parent record should hold status, counts, schema version, timestamps, and pointers.

2. Full sidecar metadata: Blob Storage beside the source file

Store the complete detailed sidecar outside the parent Cosmos item. Preferred first implementation: Blob Storage in the same enhanced-citations artifact folder as the preserved source file, scoped by the document/revision GUID. The sidecar should live beside the source file, not in a separate global metadata area that is hard to resolve when citations recall the document.

Alternative: child Cosmos records partitioned by document id and page range if we later need server-side querying of individual facts. Even then, keep the parent item as a manifest and store detailed facts as separate child items or blob shards, with the parent metadata pointing to the colocated sidecar artifact.

Recommended sidecar shape:

{
  "schema_version": "1.0",
  "document_id": "abc123",
  "source_file_name": "report.docx",
  "source_file_type": ".docx",
  "extractors": [
    {
      "name": "docx_ooxml",
      "version": "1.0"
    }
  ],
  "facts": [
    {
      "fact_id": "fmt_00042",
      "type": "font_color",
      "value": "#FF0000",
      "text": "High risk",
      "surrounding_text": "Control status: High risk until mitigation is complete.",
      "location": {
        "paragraph_index": 24
      },
      "confidence": 0.95,
      "extractor": "docx_ooxml"
    }
  ],
  "limitations": []
}

Expected size should be sparse and bounded. Most pages will have zero or very few formatting facts. Heavily revised pages may produce a few KB of metadata. The implementation should include caps such as maximum facts per page, maximum characters per fact, maximum surrounding-text length, and sidecar shard/page-range splitting for very large documents.

3. AI Search: compact retrieval facts plus sidecar hydration

AI Search should receive compact projections needed for retrieval. Azure AI Search push indexing has an approximately 16 MB request/document payload limit, but that does not mean we should store full unbounded formatting sidecars as index fields. AI Search should get bounded text that improves retrieval and answer grounding, while the full sidecar remains beside the source file in Blob Storage.

Recommended first phase:

• Append compact formatting facts to the related source chunk text before embedding generation, or create independently embedded synthetic formatting-fact chunks.
• Keep projections short and page/chunk-local.
• Avoid pushing full JSON sidecars into AI Search.

Important retrieval note: if formatting facts are appended only after embeddings are generated, vector retrieval may still miss prompts such as "which line items were struck through?" The formatting projection must either be included in the text used for embeddings or be indexed as its own embedded synthetic chunk.

Search result hydration requirement: whenever AI Search returns a chunk for a document with has_formatting_metadata = true, the retrieval/context builder should use the parent metadata pointer to load the relevant sidecar JSON or sidecar shard from the colocated enhanced-citations blob folder. The model-facing context should include bounded relevant formatting facts, not necessarily the entire raw sidecar if it exceeds configured limits.

For aggregate questions, synthetic formatting-fact chunks may be useful:

{
  "document_id": "abc123",
  "chunk_id": "formatting_facts_0001",
  "chunk_text": "Formatting facts for report.docx, pages 1-3: Struck-through text includes ... Red text includes ...",
  "page_number": 1,
  "chunk_sequence": 1000001
}

If a new chunk_type or chunk_kind field is desired, that requires an AI Search schema change. A no-schema-change first version can use existing fields and a reserved chunk_id prefix.

Existing File Re-Extraction

Add a user/admin workflow that lets someone select an existing enhanced-citation document and rerun formatting extraction.

Required behavior:

• The action is available only when the source file is still available through Enhanced Citations.
• The action reuses the preserved source blob instead of requiring a fresh upload.
• It regenerates the formatting sidecar JSON beside the source file.
• It updates the parent Cosmos formatting manifest, counts, status, timestamps, and sidecar path.
• It updates AI Search retrieval projections for the affected document, either by rewriting affected chunks or by replacing bounded synthetic formatting-fact chunks.
• It is idempotent and version-aware: rerunning extraction for one document version must not overwrite a different revision's sidecar.
• It reports unsupported file types, missing source files, extraction errors, and low-confidence PDF cases clearly.

This allows existing DOCX/PDF documents to gain the new capability after the feature ships.

Format-Specific Approach

DOCX / DOCM: P1, high confidence

For OOXML Word documents, extract formatting from the document package during ingestion or re-extraction.

Use python-docx where it is sufficient, but inspect raw OOXML where needed for:

• w:strike and w:dstrike for strikethrough / double strikethrough
• w:color for explicit font color
• w:highlight for highlight
• w:del and tracked-change runs where available
• table coordinates, row/cell context, paragraph/run location
• style inheritance and theme color edge cases

DOCX is the best first target because the file contains semantic formatting information, not just visual pixels.

PDF: best effort with confidence scoring

PDFs do not consistently contain semantic formatting such as "this text is strikethrough." Many PDFs contain positioned glyphs, drawing commands, annotations, colors, and bounding boxes.

For born-digital PDFs:

• Use PyMuPDF text spans to extract text, bounding boxes, and font color.
• Detect real strikeout annotations when present.
• Infer possible strikethrough when a horizontal line or drawing intersects a text span bounding box.
• Optionally align with Azure Document Intelligence layout/page output so citations remain consistent with current ingestion.

For scanned/image PDFs:

• Azure Document Intelligence can OCR text and layout, but strikethrough/font color may not exist as machine-readable metadata.
• Vision-based detection may be possible later, but it should be labeled low-confidence and best-effort.
• The assistant should explicitly say formatting metadata was unavailable or low-confidence when appropriate.

PDF support should not claim DOCX-level reliability unless the source PDF actually exposes the needed evidence.

Legacy binary DOC: explicit limitation for P1

Current legacy DOC handling is text-oriented. Formatting-aware binary .doc support likely requires a safe conversion path or a deeper binary parser.

For P1, legacy .doc formatting metadata should be documented as unsupported or best-effort unless a safe server-side conversion-to-DOCX path is approved. Do not block DOCX delivery on binary DOC formatting parity.

Other file types: future expansion

Start with DOCX/DOCM and best-effort PDFs. Expand to other file types only when the source format exposes reliable formatting evidence or we have an approved extractor/conversion path. Unsupported formats should return clear status rather than implying formatting awareness.

Query-Time Behavior

When a user asks about formatting:

• Prefer retrieved formatting facts when available.
• When a returned AI Search result belongs to a document with a sidecar, hydrate bounded relevant sidecar facts from the enhanced-citations blob folder into the result/context assembly.
• Cite the related page/chunk/paragraph/table-row context.
• Say when formatting metadata was not extracted, was unsupported for the file type, or was low-confidence.
• Do not infer strikethrough or color from plain extracted text unless the formatting metadata or visual/PDF evidence supports it.

Implementation Notes

• Add a formatting extraction layer in the ingestion path, close to process_document_upload_background() / process_di_document() before chunks are saved.
• Gate formatting-aware sidecar extraction on Enhanced Citations, because the sidecar must be colocated with the retained source file.
• Add a re-extraction path for existing enhanced-citation documents that reuses the saved source blob.
• Map facts to page/chunk context before calling save_chunks() / save_chunks_batch() so retrieval projections can be embedded or separately indexed.
• Store the parent manifest in Cosmos with the document metadata record.
• Store detailed sidecar JSON in the same enhanced-citations blob folder as the source file, scoped by the document/revision GUID. Use child Cosmos records only if queryable fact records become necessary.
• Add a bounded projection builder that emits concise Formatting facts: text per source chunk or per synthetic formatting chunk.
• Hydrate relevant sidecar facts when AI Search returns documents/chunks with formatting metadata.
• Keep caps configurable so very large or heavily marked-up documents cannot bloat Cosmos, AI Search, or prompt context.
• Preserve authorization boundaries for personal, group, public workspace, and chat-upload documents.

Acceptance Criteria

• Formatting-aware sidecar extraction requires Enhanced Citations.
• When Enhanced Citations is enabled, supported file types automatically produce sidecar metadata during upload/reprocess without a separate per-upload toggle.
• DOCX uploads can answer "which line items are struck through?" from indexed document context.
• DOCX uploads can answer font-color questions such as "which line items are red?" from indexed document context.
• Formatting facts retain enough location context to support citations or clear source references.
• Parent Cosmos document metadata stores only a small manifest, counts, status, schema version, and pointer to detailed formatting metadata.
• Full detailed formatting metadata is stored outside the parent Cosmos item, in the same enhanced-citations blob folder as the preserved source file and scoped by document/revision GUID.
• AI Search receives compact retrieval projections, either appended to related chunks before embedding or stored as bounded synthetic formatting-fact chunks.
• When AI Search returns a result for a document with a sidecar, the application hydrates relevant bounded sidecar facts into the search result/context payload.
• Users can select an existing enhanced-citation document and rerun formatting extraction without reuploading the file.
• Re-extraction updates the sidecar, Cosmos manifest, and AI Search projections for the selected document version only.
• PDF formatting extraction is implemented as best-effort with confidence scoring, or explicitly documented as unsupported for cases where the source PDF lacks reliable evidence.
• Scanned/image PDF behavior does not overclaim formatting detection; answers disclose low-confidence or unavailable metadata.
• Legacy DOC behavior is explicit and does not overclaim unsupported formatting extraction.
• Existing plain-text document search, citations, metadata extraction, and normal content questions continue to work.
• Functional tests cover DOCX strikethrough, DOCX font color, DOCX unformatted negative cases, table-row context, PDF born-digital best-effort behavior, unsupported/low-confidence PDF cases, and existing-file re-extraction.
• Large or heavily formatted documents respect configured caps and do not exceed Cosmos item limits, AI Search indexing payload limits, or prompt-context budgets.

Validation Plan

• Unit-test DOCX extractor output against synthetic fixtures containing plain text, strikethrough text, colored text, highlights, table rows, and tracked-change/deleted text.
• Unit-test PDF extractor output against born-digital PDF fixtures with colored spans, strikeout annotations, drawn horizontal strikeout lines, and normal unformatted text.
• Ingestion test: verify formatting-aware sidecar extraction does not run when Enhanced Citations is disabled and does run automatically when Enhanced Citations is enabled.
• Ingestion test: verify parent Cosmos metadata contains the manifest and pointer, not the full detailed sidecar.
• Ingestion test: verify full sidecar JSON is written, bounded, and colocated in the same enhanced-citations blob folder as the preserved source file.
• Re-extraction test: verify an existing enhanced-citation document can regenerate sidecar metadata and refresh search projections without reuploading the source file.
• Search test: verify compact formatting facts are present in searchable chunk context or synthetic formatting-fact chunks.
• Search result hydration test: verify a returned document/chunk with a sidecar includes bounded relevant sidecar facts in the model-facing result/context payload.
• Chat/RAG test: verify formatting-specific prompts retrieve the facts and do not falsely report unformatted text as struck through or colored.

Notes

This should be treated as an ingestion/indexing augmentation, not as live file execution during chat. ChatGPT-style live python-docx inspection works because it can keep the original file in a secure sandbox and run code against it. SimpleChat should instead extract and persist safe formatting metadata during upload/reprocess so answers are available through the existing document search pipeline.

[paullizer]

Superseded by updated issue body

I rewrote the issue body as the source of truth for this work so the storage, retrieval, Enhanced Citations prerequisite, existing-file re-extraction flow, DOCX/PDF/DOC behavior, sizing limits, and acceptance criteria are not split across conflicting comments.

The current design is: formatting-aware sidecar extraction requires Enhanced Citations; the sidecar JSON is stored beside the preserved source file in the same document/revision GUID enhanced-citations blob folder; parent Cosmos metadata stores only a small manifest and pointer; AI Search gets compact retrieval projections and result assembly hydrates bounded sidecar facts for returned documents.