Skip to content

[refactor](fe) Modularize external metadata cache #66382

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
924060929 wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

[refactor](fe) Modularize external metadata cache #66382

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9efbcb9
[refactor](fe) Modularize external metadata cache
924060929 Aug 3, 2026
551f7a6
[fix](fe) Preserve MetaCache publication semantics
924060929 Aug 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions fe/fe-connector/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,8 @@ endpoint properties)

| Module | Role |
|---|---|
| `fe-connector-cache` | Self-contained caching framework used by several connectors. No fe-core dependency; it is bundled into each consuming plugin, so shared third-party libraries stay at the consumers' lowest common version (see the version notes in consumer poms). |
| `fe-connector-metacache-spi` | JDK-only cache policy, entry-definition, invalidation, statistics, and lifecycle contracts. It has no fe-core, cache-library, or data-source SDK dependency. |
| `fe-connector-metacache` | Shared Caffeine-backed cache runtime used by fe-core and connector plugins. It owns entry generations, refresh, scoped invalidation, catalog entry groups, and the engine registry; it never depends on fe-core or a data-source SDK. |
| `fe-connector-hms-hive-shade` | Slim, relocated HMS metastore-client closure for connectors that speak HMS thrift. The pom comments say exactly what relocates where and why. |
| `fe-connector-paimon-hive-shade` | Paimon-private relocated HMS-thrift closure; same idea, different owner. |

Expand Down Expand Up @@ -183,10 +184,10 @@ metastore/shade/cache). For a write path, the richest example is
6. **Property ownership.** Metadata-connection properties are parsed in your
connector (or the metastore layer). Storage properties belong to
`fe-filesystem`. Do not add parsing to fe-core — rule 2 above.
7. **Caching.** Reuse `fe-connector-cache` (example:
`PaimonLatestSnapshotCache`). Bundle the caching library into your plugin
zip and keep shared third-party versions aligned with the other consumers
(see the version notes in `fe-connector-paimon/pom.xml`). Respect the
7. **Caching.** Describe reusable entries with `fe-connector-metacache-spi`
and run them through `fe-connector-metacache` (example:
`PaimonLatestSnapshotCache`). Keep shared third-party versions aligned with
the other consumers (see the version notes in `fe-connector-paimon/pom.xml`). Respect the
authorization invariant in `AGENTS.md`: a cross-query cache must never
serve metadata that would bypass per-user, load-time authorization.
8. **Shading.** If your client stack drags a conflicting closure (hive/thrift
Expand Down
345 changes: 0 additions & 345 deletions ...tor/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheEntry.java
View file
Open in desktop

This file was deleted.

Loading
Loading