columnar: pin CSE FTS Spec v1 on cloud-engine

[3pointer]

What problem does this PR solve?

Issue Number: N/A

Problem Summary:

TiDB/CSE FTS Spec v1 uses full_text_index.index_fields as the canonical ordered physical field layout. CSE now owns parsing, validation, and conversion of that metadata. This PR now pins the latest fts_spec_v1 after its cloud-engine rebase (36128849380c), including match-expression query protocol support from tidbcloud/cloud-storage-engine#5801.

The production TiFlash path builds CSE through contrib/tiflash-columnar-hub, so TiFlash only needs to pin that CSE revision. TiFlash C++ does not need to parse or validate the FTS fields.

What is changed and how it works?

columnar: pin CSE FTS Spec v1 on cloud-engine

• Bump contrib/cloud-storage-engine to 36128849380c, the latest fts_spec_v1 head after the cloud-engine rebase.
• Align the columnar-hub workspace's Tantivy patch with CSE's pingcap/tantivy:cse-0.22.1 and refresh Cargo.lock. This parent-workspace patch is required because Cargo ignores a child workspace's [patch.crates-io] when CSE is built by columnar hub.
• Keep TiFlash C++ unchanged. Schema parsing, Analyzer validation, ordered/repeated column_id handling, Analyzer parameters, and enable_bm25 remain in CSE.

Metadata compatibility:

• Spec v1 canonical metadata is ordered index_fields; old filter_columns and transitional fields are not used as the physical layout.
• Legacy parser_type remains readable in CSE for rolling upgrades of existing single-field indexes.

Deployment order:

1. Land and deploy the CSE change and this TiFlash/columnar-hub pin.
2. Only then allow TiDB to emit new multi-field FTS DDL.

Check List

Tests

• Unit test
• Integration test
• Manual test (details below)
• No code

Completed locally on Apple arm64 (revalidated with CSE 36128849380c):

• cargo check --locked --manifest-path hub-runtime/Cargo.toml
• make debug, producing an arm64 libtiflash_proxy.dylib
• cargo fmt --all -- --check

Pending:

• TiFlash X64/ARM64 CI builds.
• End-to-end round-trip using TableInfo JSON emitted by a real TiDB instance.

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

[ti-chi-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fuzhe1989 for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: f22214d7-cf0f-4d39-bc11-22c915211a0c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[FORMAT CHECKER NOTIFICATION]

Notice: To remove the do-not-merge/needs-linked-issue label, please provide the linked issue number on one line in the PR body, for example: Issue Number: close #123 or Issue Number: ref #456.

_{📖 For more info, you can check the "Contribute Code" section in the development guide.}