[P1] Add related metadata for chunks that span sections, tables, and document structures

[paullizer]

Summary

Add a chunk relationship model that preserves the current page-length chunking and citation strategy while linking chunks that belong to the same logical document structure, such as sections, subsections, multi-page tables, formulas, figures, or other content spans.

User Value

Users get stronger answers from documents where important context spans multiple pages or chunks, without sacrificing clear page-based citations. This is especially important for long tables, repeated section structures, appendix content, policies, financial schedules, technical specs, and other structured documents where the useful answer may require many related chunks.

Proposed Behavior

During document extraction and chunk creation, detect logical relationships between chunks and store related metadata that can be used by search, retrieval, and future graph-based analysis. The base chunk size should remain bounded so a single table or section spanning many pages does not become one huge chunk. Instead, each chunk keeps normal citation-friendly boundaries and receives relationship metadata that identifies the larger structure it belongs to.

Examples of relationships to capture:

• Chunks under the same header or section.
• Chunks under nested subsections.
• Chunks that are consecutive parts of the same table, including tables spanning many pages.
• Chunks associated with the same figure, formula, appendix, glossary entry, or named document structure.
• Parent/child relationships between sections and subsections.
• Ordered sequence relationships so retrieval can reconstruct spans when needed.

The metadata should support current search and chat retrieval flows and should also be compatible with future graph-based search or graph database analysis.

Acceptance Criteria

• Extraction/chunking stores relationship metadata without changing the existing page-length chunk boundary behavior.
• Chunks can be associated to a logical parent structure, such as a section, subsection, table, figure, formula, appendix, or other named span.
• Chunks that are part of a multi-page or long-running table can be related even when the table spans many pages.
• Relationship metadata includes enough source context to preserve page-level citations and chunk ordering.
• Retrieval can use related chunk metadata to expand or prioritize context when answering questions about spanned content.
• The approach avoids creating very large chunks for long spans and instead links normal-sized chunks together.
• Metadata is designed so it can later be projected into a graph search or graph database model.
• Behavior is configurable or gated so existing extraction/search behavior remains stable when relationship-aware retrieval is disabled.
• Functional tests cover section relationships, subsection relationships, multi-page table relationships, citation preservation, and retrieval expansion behavior.

Notes

Priority: P1
Estimated size: L

This is related to improving document extraction quality and complements multimodal augmentation work, but it should not depend on replacing the existing chunking strategy. The goal is to add structure-aware relationships on top of the current citation-friendly chunk model.