TableSpec.source_table required in databricks-bundles breaks bundle validate for SharePoint file-ingestion pipelines defined in YAML

[bjornkpu]

Describe the issue

Adding a python: section to databricks.yml makes bundle validate fail on a Lakeflow Connect SharePoint file-ingestion pipeline that is defined entirely in YAML, deploys and runs correctly without python:, and matches the shape in Microsoft's own documentation.

TypeError: TableSpec.__init__() missing 1 required keyword-only argument: 'source_table'

A SharePoint FILE object has no source table. It reads files, selected by file_filters. The documented examples omit source_table accordingly: every DAB example under Ingest data from SharePoint sets only destination_* plus connector_options.

In python/databricks/bundles/pipelines/_models/table_spec.py on main, source_table has no default and is documented as "Required. Table name in the source database", but a file-ingestion object has no source database. Note the asymmetry with its own siblings:

source_table: VariableOr[str]
"""[Public Preview] Required. Table name in the source database."""

source_catalog: VariableOrOptional[str] = None
"""[Public Preview] Source catalog name. Might be optional depending on the type of source."""

source_schema: VariableOrOptional[str] = None
"""[Public Preview] Schema name in the source database. Might be optional depending on the type of source."""

source_catalog and source_schema already acknowledge that source-side fields depend on the source type. source_table is the only one that does not, and it is the one that makes the object unconstructible.

Two things make this worse than a schema mismatch on one field:

1. No Python resources or mutators are involved. The pipeline is YAML. Merely declaring python: is enough, because the CLI round-trips every YAML resource through the dataclasses.
2. The failure happens in apply_mutators, a phase I do not use. From the debug log, load_resources completes, then PythonMutator(apply_mutators) runs anyway, with no python.mutators configured, and that is what dies. I opened Skip the apply_mutators phase when no mutators are configured #6294 to skip that phase when no mutators are declared. That is independent of the TableSpec question and does not resolve it: a bundle that does declare mutators still has to load every resource in that phase, and would hit the same field.

Net effect: one file-ingestion pipeline anywhere in a bundle prevents that bundle from using Python for DABs at all. My only workaround was to remove the SharePoint pipeline from the bundle.

Configuration

Three files. No notebook, no credentials needed to reproduce.

databricks.yml

bundle:
  name: repro

include:
  - resources/*.yml

python:
  venv_path: .venv
  resources:
    - 'resources:load_resources'

targets:
  dev:
    default: true
    workspace:
      host: https://<workspace>.azuredatabricks.net

resources/probe.yml, the shape from the docs example, no source_table:

resources:
  pipelines:
    sharepoint_connector:
      name: sharepoint_connector
      channel: PREVIEW
      catalog: main
      schema: some_schema
      ingestion_definition:
        connection_name: sharepoint_conn
        objects:
          - table:
              destination_catalog: main
              destination_schema: some_schema
              destination_table: sptest_csv
              connector_options:
                sharepoint_options:
                  entity_type: FILE
                  url: https://example.sharepoint.com/sites/site
                  file_ingestion_options:
                    format: CSV
                    schema_evolution_mode: NONE

resources/__init__.py, which deliberately adds nothing and declares no mutators:

from databricks.bundles.core import Bundle, Resources


def load_resources(bundle: Bundle) -> Resources:
    return Resources()

Steps to reproduce the behavior

1. uv venv .venv && uv pip install --python .venv databricks-bundles
2. databricks bundle validate -t dev gives the TypeError, exit 1
3. Delete resources/probe.yml and run step 2 again: Validation OK!
4. Or restore it, remove the python: section, and run step 2: also passes, with the pre-existing Warning: required field source_table is not set

Expected Behavior

source_table optional on TableSpec, so a connector_options-based file-ingestion object validates under python: exactly as it does without it.

Actual Behavior

bundle validate exits 1 with TypeError: TableSpec.__init__() missing 1 required keyword-only argument: 'source_table', and no resource in the bundle can be deployed.

Without python:, the same pipeline deploys and the connector runs. It ingests the files and writes the destination tables. So the backend accepts the definition; only the generated dataclass rejects it.

Adding a dummy source_table does make validation pass, and connector_options survives the round-trip intact, but the dummy value is then present in the emitted deploy payload. I would rather not send a meaningless source table to a Beta connector for an object that has none.

OS and CLI version

• Databricks CLI v1.11.0
• Windows 10.0.26200
• databricks-bundles 1.12.1 (current release); also reproduced on 1.11.0
• engine: direct

Is this a regression?

Not as far as I can tell. I have no CLI version where this worked. I first tried Python for DABs at v1.11.0 and hit it immediately, and it reproduces on both databricks-bundles releases I tried.

Debug Logs

Debug: Apply pid=37288 mutator=PythonMutator(load_resources)
Debug: running: .venv\Scripts\python.exe -m databricks.bundles.build --phase load_resources ...
Debug: Apply pid=37288 mutator=PythonMutator(apply_mutators)
Debug: running: .venv\Scripts\python.exe -m databricks.bundles.build --phase apply_mutators ...
Debug: python mutator process failed: ... --phase apply_mutators ...: exit status 1
  File ".venv\Lib\site-packages\databricks\bundles\build.py", line 96, in _load_resources_from_input
TypeError: TableSpec.__init__() missing 1 required keyword-only argument: 'source_table'

Note load_resources succeeds; apply_mutators then runs despite no python.mutators being configured, and that is the phase that fails.

Related

• Skip the apply_mutators phase when no mutators are configured #6294, my PR to skip apply_mutators when no mutators are configured. Removes the second point above, not this issue.
• The asset bundle does not recognise the source_table for the SharePoint connector #3227, same field, same connector, from the other direction: that reporter wanted source_table honoured for a SharePoint connector and saw it warned about and ignored. It was closed after going stale rather than on merits, and the maintainer reply says source_table should be available in DABs configuration. To be clear, this issue is not asking for the field to be removed, only for it not to be required, since the documented file-ingestion form has no source table to give it.
• [Bug]: CLI/Bundles strict validation rejects officially documented 'meta_ads_options' in Lakeflow Connect pipelines (Similar to #4883) #5253 (open), same family: strict validation rejects the documented meta_ads_options under connector_options and strips it from the payload, with the backend accepting the same JSON via raw API.
• Bundle schema does not recognize connector_options in ingestion pipeline SchemaSpec / TableSpec #4883 (closed), connector_options missing from SchemaSpec/TableSpec in the bundle JSON schema. Whatever the closure reason, that part now looks addressed: connector_options is present and optional on TableSpec in main, and it round-trips intact in the repro above. source_table being required is the remaining gap, and it is a hard error rather than a stripped field.