Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 5 additions & 6 deletions fe/fe-connector/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,7 @@ endpoint properties)

| Module | Role |
|---|---|
| `fe-connector-metacache-spi` | JDK-only cache policy, entry-definition, invalidation, statistics, and lifecycle contracts. It has no fe-core, cache-library, or data-source SDK dependency. |
| `fe-connector-metacache` | Shared Caffeine-backed cache runtime used by fe-core and connector plugins. It owns entry generations, refresh, scoped invalidation, catalog entry groups, and the engine registry; it never depends on fe-core or a data-source SDK. |
| `fe-connector-cache` | Self-contained caching framework used by several connectors. No fe-core dependency; it is bundled into each consuming plugin, so shared third-party libraries stay at the consumers' lowest common version (see the version notes in consumer poms). |
| `fe-connector-hms-hive-shade` | Slim, relocated HMS metastore-client closure for connectors that speak HMS thrift. The pom comments say exactly what relocates where and why. |
| `fe-connector-paimon-hive-shade` | Paimon-private relocated HMS-thrift closure; same idea, different owner. |

Expand Down Expand Up @@ -184,10 +183,10 @@ metastore/shade/cache). For a write path, the richest example is
6. **Property ownership.** Metadata-connection properties are parsed in your
connector (or the metastore layer). Storage properties belong to
`fe-filesystem`. Do not add parsing to fe-core — rule 2 above.
7. **Caching.** Describe reusable entries with `fe-connector-metacache-spi`
and run them through `fe-connector-metacache` (example:
`PaimonLatestSnapshotCache`). Keep shared third-party versions aligned with
the other consumers (see the version notes in `fe-connector-paimon/pom.xml`). Respect the
7. **Caching.** Reuse `fe-connector-cache` (example:
`PaimonLatestSnapshotCache`). Bundle the caching library into your plugin
zip and keep shared third-party versions aligned with the other consumers
(see the version notes in `fe-connector-paimon/pom.xml`). Respect the
authorization invariant in `AGENTS.md`: a cross-query cache must never
serve metadata that would bypass per-user, load-time authorization.
8. **Shading.** If your client stack drags a conflicting closure (hive/thrift
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,21 +29,24 @@ under the License.
<relativePath>../pom.xml</relativePath>
</parent>

<artifactId>fe-connector-metacache</artifactId>
<artifactId>fe-connector-cache</artifactId>
<packaging>jar</packaging>
<name>Doris FE Connector MetaCache Runtime</name>
<name>Doris FE Connector Cache Framework</name>
<description>
Shared external metadata cache runtime used by fe-core and connector plugins.
Contains the Caffeine-backed entry implementation, catalog entry grouping and reusable
connector cache helpers. It depends on fe-connector-metacache-spi and never depends on fe-core.
Connector-side meta-cache framework (CacheSpec + MetaCacheEntry + CacheFactory + MetaCacheEntryStats),
an INDEPENDENT copy of fe-core's `org.apache.doris.datasource.metacache` framework re-homed under the
`org.apache.doris.connector.*` prefix so the connector plugins can reuse it (they cannot import fe-core).
fe-core keeps its own copy untouched; the two live side-by-side until every connector has migrated, then
the fe-core copy is retired. fe-core does NOT depend on this module.

This module is bundled into each connector plugin zip (child-first), so it uses the plugin's own bundled
Caffeine at runtime; Caffeine is therefore `provided` here (compiled against, never packaged by this
module). The framework's public API (MetaCacheEntry) is Caffeine-free, and fe-core and the connectors
never share a cache object across the classloader boundary, so no Caffeine type crosses and there is no
split-brain. Two knobs fe-core reads from static Config are constructor-injected here.
</description>

<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-connector-metacache-spi</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
Expand All @@ -55,19 +58,9 @@ under the License.
<artifactId>junit-jupiter</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<finalName>doris-fe-connector-metacache</finalName>
<finalName>doris-fe-connector-cache</finalName>
</build>
</project>
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.connector.metacache;
package org.apache.doris.connector.cache;

import com.github.benmanes.caffeine.cache.AsyncCacheLoader;
import com.github.benmanes.caffeine.cache.AsyncLoadingCache;
Expand All @@ -32,8 +32,11 @@
/**
* Factory to create Caffeine cache.
*
* <p>This type is internal to the shared MetaCache runtime. Its public methods return Caffeine
* types; callers outside this module use {@link MetaCacheEntry} instead.
* <p>Connector-side copy of fe-core {@code org.apache.doris.common.CacheFactory} (independent-copy meta-cache
* migration): connector plugins cannot import fe-core, so the framework is duplicated under
* {@code org.apache.doris.connector.cache}. This type is framework-internal — its public methods RETURN
* Caffeine types, which must never cross to connector (child-first) code; connectors only touch the
* Caffeine-free {@link MetaCacheEntry} API. Keep behaviourally in sync with the fe-core original.
*
* <p>This class is used to create Caffeine cache with specified parameters.
* It is used to create both sync and async cache.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.connector.metacache.spi;
package org.apache.doris.connector.cache;

import java.util.HashMap;
import java.util.Map;
Expand All @@ -25,9 +25,18 @@
/**
* Common cache specification for external metadata caches.
*
* <p>The type is part of the connector-facing MetaCache SPI and therefore depends only on JDK types.
* Property validation reports {@link IllegalArgumentException}; fe-core adapters may translate that
* exception at their own API boundary.
* <p>Connector-side copy of the meta-cache property model (independent-copy meta-cache migration). fe-core is
* NOT changed: it keeps its own {@code org.apache.doris.datasource.metacache.CacheSpec}; this is a separate
* class under {@code org.apache.doris.connector.*} used only by the connector plugins. Although that prefix is
* parent-first, fe-core does not depend on this module, so the class resolves parent → miss → CHILD and is
* child-loaded per plugin — fe-core and the plugins do NOT share one {@code Class} identity. It carries no
* third-party dependency (JDK only) and never crosses the fe-core↔connector boundary as an object (only its
* {@code IllegalArgumentException}, a JDK type, crosses), so it is safe on both classpaths.
*
* <p>The {@code check*Property} validators throw {@link IllegalArgumentException} (fe-core's
* {@code PluginDrivenExternalCatalog.checkProperties} re-wraps it into a {@code DdlException} verbatim; the
* legacy fe-core catalogs that still call these validators declare {@code throws DdlException} but no longer
* need it). The user-facing message text is identical to the legacy one ({@code "... is wrong, value is ..."}).
*
* <p>Semantics:
* <ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,7 @@
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.connector.metacache;

import org.apache.doris.connector.metacache.spi.CacheSpec;
package org.apache.doris.connector.cache;

import java.util.Collections;
import java.util.Map;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.

package org.apache.doris.connector.metacache;
package org.apache.doris.connector.cache;

import java.util.Objects;

Expand Down
Loading
Loading