Skip to content

feat(rocksdb): add optional ToplingDB integration - #3134

Open
kenssa4eedfd wants to merge 67 commits into
apache:masterfrom
hugegraph:toplingdb
Open

feat(rocksdb): add optional ToplingDB integration#3134
kenssa4eedfd wants to merge 67 commits into
apache:masterfrom
hugegraph:toplingdb

Conversation

@kenssa4eedfd

Copy link
Copy Markdown
Contributor

Purpose of the PR

This PR introduces ToplingDB as an optional RocksDB-compatible storage engine
for HugeGraph.

ToplingDB is explicitly enabled through configuration. Standard RocksDB remains
the default, so existing deployments and data paths are unaffected unless
rocksdb.provider=topling is configured.

Main Changes

  • Add explicit storage engine selection through rocksdb.provider.
  • Integrate ToplingDB with HugeGraph Server, PD, and Store.
  • Use ToplingDB Easy Migrate YAML configuration while continuing to use the
    standard RocksDB Java API.
  • Add component-specific configurations for Server, PD, and Store, including
    separate HTTP ports and JRaft column-family settings.
  • Add startup scripts that:
    • Select and preload the ToplingDB JNI library.
    • Extract required native libraries and Web UI resources.
    • Configure LD_PRELOAD, LD_LIBRARY_PATH, and
      TOPLINGDB_EASY_MIGRATE_CONF.
    • Validate the operating system and architecture.
    • Verify downloaded dependencies using SHA-256.
    • Handle jemalloc and Ubuntu 24.04 libaio compatibility.
  • Align the RocksDB JNI version across Server, PD, and Store.
  • Update CI workflows to install and test the ToplingDB runtime dependencies.
  • Update third-party dependency metadata, LICENSE information, operational
    documentation, troubleshooting guides, and security notes.

Compatibility

  • Standard RocksDB remains the default provider.
  • Existing RocksDB data and APIs remain compatible.
  • ToplingDB is enabled only when rocksdb.provider=topling is configured.
  • No public HugeGraph API changes are introduced.
  • The bundled ToplingDB runtime currently targets Linux x86_64.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • Build all modules:
      mvn clean compile -Dmaven.javadoc.skip=true                                                                                                                                                        
    • Run HugeGraph Server tests with the RocksDB backend:
      mvn test -pl hugegraph-server/hugegraph-test -am \                                                                                                                                                 
        -P core-test,rocksdb                                                                                                                                                                             
    • Run PD tests:
      mvn install -pl hugegraph-struct -am -DskipTests                                                                                                                                                   
      mvn test -pl hugegraph-pd/hg-pd-test -am                                                                                                                                                           
    • Run Store tests:
      mvn test -pl hugegraph-store/hg-store-test -am                                                                                                                                                     
    • Verify Server, PD, and Store startup in both standard RocksDB mode and
      ToplingDB mode.

Does this PR potentially affect the following parts?

  • Dependencies
  • Modify configurations
  • The public API
  • Other affects: startup scripts, native library loading, and CI workflows
  • Nope

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need

imbajin and others added 30 commits June 12, 2025 14:08
[pull] master from apache:master
[pull] master from apache:master
Co-authored-by: Hervé Boutemy <herve.boutemy@free.fr>
* refactor: Delete the package existing in java8

* chore(format): remove custom line breaks
syslucas and others added 20 commits November 4, 2025 10:24
…-detection to explicit `rocksdb.provider` configuration
# Conflicts:
#	.github/workflows/pd-store-ci.yml
#	hugegraph-server/hugegraph-dist/src/assembly/static/bin/start-hugegraph.sh
#	install-dist/scripts/dependency/known-dependencies.txt
@dosubot dosubot Bot added size:XXL This PR changes 1000+ lines, ignoring generated files. ci-cd Build or deploy dependencies Incompatible dependencies of package feature New feature labels Aug 2, 2026
# Conflicts:
#	.github/workflows/server-ci.yml
#	hugegraph-server/hugegraph-dist/src/assembly/travis/run-api-test.sh
#	install-dist/scripts/dependency/known-dependencies.txt
@codecov

codecov Bot commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 34.70%. Comparing base (431f6e6) to head (2c907fc).

Files with missing lines Patch % Lines
...a/org/apache/hugegraph/pd/store/HgKVStoreImpl.java 0.00% 1 Missing ⚠️
...raph/backend/store/rocksdb/RocksDBStdSessions.java 50.00% 1 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (431f6e6) and HEAD (2c907fc). Click for more details.

HEAD has 4 uploads less than BASE
Flag BASE (431f6e6) HEAD (2c907fc)
5 1
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3134      +/-   ##
============================================
- Coverage     41.06%   34.70%   -6.37%     
+ Complexity      519      498      -21     
============================================
  Files           771      782      +11     
  Lines         65962    67113    +1151     
  Branches       8766     8963     +197     
============================================
- Hits          27088    23292    -3796     
- Misses        36008    41216    +5208     
+ Partials       2866     2605     -261     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd Build or deploy dependencies Incompatible dependencies of package feature New feature size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants