Add OceanBase (column store and row store) - #1726
Open
alexey-milovidov wants to merge 3 commits into
Open
Conversation
OceanBase is the shared-nothing, Paxos-replicated, multi-tenant DBMS Alibaba started in 2010 and open-sourced under Mulan PSL v2 in 2021: a MySQL-compatible SQL layer — `SELECT VERSION()` answers `5.7.25-OceanBase_CE-v5.0.1.0` — over an LSM-tree storage engine. Since 4.3 that engine can store a table by column instead of by row, and the two are different enough here to justify two entries: `oceanbase` is the column store, which is what the vendor's own OLAP parameter template selects, and `oceanbase-row` is the same setup with `WITH COLUMN GROUP (all columns)`. It installs natively; no container is needed. OceanBase publishes only el7/el8 RPMs, but the observer links nothing outside glibc and a libaio that ships in the `oceanbase-ce-libs` package, so `rpm2cpio | cpio` into /opt/oceanbase runs unmodified on Ubuntu — on x86_64 and aarch64 alike, so the c8g machines are covered too. Nothing is installed system-wide. The whole one-time cluster setup lives in ./install rather than ./load, because a freshly started observer serves no tenant at all: it answers "Tenant not in this server" until ALTER SYSTEM BOOTSTRAP has created the internal sys tenant, and user data then needs a tenant of its own, which needs a resource pool, which needs a unit config. All of that has to be done before the driver's first ./check. The unit is sized from what GV$OB_SERVERS reports as unassigned rather than from arithmetic on memory_limit, because the bootstrap has already given the sys tenant a unit whose size depends on the version. Configuration is obd's, the vendor deployer's: memory_limit at 80% of RAM, system_memory from obd's step function, cpu_count at nproc-2, plus the `olap` parameter and system-variable template that ships inside the RPM itself (etc/default_parameter.json, etc/default_system_variable.json) and that obd and OCP offer as a dropdown. Two of those template entries change results and not just speed: utf8mb4_bin makes LIKE and ORDER BY byte-exact, which is what the reference ClickHouse results do, and parallel_degree_policy = AUTO is what lets a query use more than one thread without a hint in the SQL. ./load goes through LOAD DATA INFILE with the APPEND hint — OceanBase's bypass (direct) load, which sorts by primary key and writes straight into major SSTables, skipping the SQL layer, the transaction layer and the memtable, and which also collects optimizer statistics on the way so no separate ANALYZE is needed. Parallelism is the tenant's core count capped at one worker per 512 MB of tenant memory: each direct-load worker holds a sort area, a macroblock writer and a 7 MB coroutine stack, and at 90 workers against a 9 GB tenant the load dies with "No memory or reach tenant memory limit" and rolls back. On the benchmark's own machines the core count is the smaller of the two limits. ./load then forces a minor freeze of the user tenants *and* of the meta tenants and waits for it, without which the entry produces no result at all. The direct load leaves ~100 MB of redo log on the meta tenant that every user tenant carries, and that stream's base_lsn -- its checkpoint -- stays at zero, so every ./start replays the whole thing; how much replay a tenant can buffer is bounded by its memtable, and a meta tenant's memtable is about 4% of the resource unit's memory, 410 MB on a 9 GB unit. The backlog does not fit, replay stalls with "CLOG pending size in task queue exceeds limit", the observer never reaches "start success", and ./check times out on all 43 queries. Measured on the unit whose meta tenant gets exactly that 410 MB: without the freeze the server had not finished starting after ten minutes, twice; with it, 27 seconds. TENANT = all covers the sys and user tenants; the meta tenants need all_meta. For the same reason in reverse, benchmark.sh raises BENCH_CHECK_TIMEOUT to 900 s -- the observer's start is not instant even with nothing to replay, and a ./check that times out aborts the whole run rather than one query. ./data-size reports DATA_DISK_IN_USE + LOG_DISK_IN_USE from GV$OB_SERVERS rather than du of the store directory: the observer preallocates both its data file and its redo log pool at startup, so du reports the reservation and would say the same thing about an empty database as about a loaded one. On the full dataset that is 21.1 GiB -- 11.9 GiB of columns plus 9.2 GiB of redo log -- against 159 GiB of du. queries.sql differs from clickhouse/queries.sql on two lines out of 43. Q29's REGEXP_REPLACE backreference has to be '$1' — OceanBase follows MySQL 8, where '\1' is the literal character 1, and left alone the query collapses every row into a single group. Q43 groups by minute, which the mysql entry's '%H:00:00' gets wrong. Every query was compared against clickhouse-local on the same rows: 33 agree exactly, eight are LIMIT cutting through tied sort keys (verified: the sort-key multisets match, and in Q24 the ten WatchIDs are the same ten), Q4 differs because ClickHouse overflows AVG(UserID) in an Int64 while OceanBase returns the exact decimal mean, and Q6 differs by 2 in 107907 because utf8mb4_bin is a PAD SPACE collation and two phrase pairs differ only in a trailing space. The string columns are VARCHAR rather than TEXT — TEXT is a LOB type in OceanBase and the benchmark's hottest columns are URL, Title and Referer — with widths at least four times the longest value the dataset actually holds, since OceanBase caps a row at 1.5 MB of declared width and 28 VARCHAR(65535) columns exceed that fivefold. Verified end to end on the full dataset: ./load gets exactly 99,997,497 rows in, the APPEND hint's online statistics land with them, all 43 queries return a result, stop/start/check/query works, and the driver's own bench_run_query and bench_concurrent_qps were run against the loaded table. The rowstore entry is validated at 1% scale. No results yet; those need runs on the benchmark's own EC2 machines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
deployed
to
benchmark-approval
September 1, 2026 13:47 — with 
GitHub Actions
Active
GitHub Actions
Active
Member
Author
This is laughable. How not to build a database 🤣 |
Contributor
|
The run of Logs:
|
Member
Author
This is pathetic.
|
Member
Author
|
It must be a very serious enterprise database. Judging by error messages. |
The c6a.4xlarge run died at ALTER SYSTEM BOOTSTRAP with a bare "ERROR 4015 (HY000): System error" and nothing else in the log, so there was nothing to act on. Rerunning ./install locally with that machine's exact derived configuration -- memory_limit 25102M, system_memory 6144M, cpu_count 14, a 162 GB data file and a 24.5 GB log disk -- bootstraps fine on this box, so whatever it is does not follow from the numbers. Three changes, in order of usefulness: - On failure, dump the observer's own log: the lines mentioning bootstrap, CHECK_SERVER_EMPTY, WDIAG/EDIAG/ERROR, plus the tail of observer.log and of observer.log.wf. The next run will say what the server actually objected to instead of leaving us to guess. - Give the bootstrap up to three attempts. Retrying is not just a matter of re-running the statement: the observer decides whether it may be bootstrapped by looking for its own leftovers, and those are spread around $OB_HOME rather than confined to store/ -- etc2/, etc3/, wallet/, audit/ and two generated files in etc/. Leave any of them behind and the retry is refused with "Server is not empty but has never been bootstrapped ... has_data_version_file=TRUE", which is exactly what happened when a first attempt at this cleared only store/. reset_observer_state clears all of them, and re-bootstrapping from that state was verified to work. - Wait 30 s after root@sys first answers before bootstrapping. The SQL port comes up before the rest of the server has settled and the bootstrap's first act is a round of RPCs to every address in the rs_list; an instant start was reproduced here by putting the store on tmpfs, and while that did bootstrap cleanly, firing the statement into a server that answered a second ago is not worth the risk when the wait is free. Also fixes the shutdown wait in the retry path to use the pid file: the observer is launched as ./bin/observer from $OB_HOME, so pgrep on an absolute path never matches it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
deployed
to
benchmark-approval
September 1, 2026 21:55 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
|
The run of Logs:
|
Contributor
|
Results for Logs:
|
alexey-milovidov
deployed
to
benchmark-approval
September 2, 2026 06:31 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
|
The run of Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OceanBase is the shared-nothing, Paxos-replicated, multi-tenant DBMS Alibaba started in 2010 and open-sourced under Mulan PSL v2 in 2021: a MySQL-compatible SQL layer —
SELECT VERSION()answers5.7.25-OceanBase_CE-v5.0.1.0— over an LSM-tree storage engine. Since 4.3 that engine can store a table by column instead of by row, and the two are different enough here to justify two entries:oceanbaseis the column store, which is what the vendor's own OLAP parameter template selects, andoceanbase-rowis the same setup withWITH COLUMN GROUP (all columns).It installs natively; no container is needed. OceanBase publishes only el7/el8 RPMs, but the observer links nothing outside glibc and a
libaiothat ships in theoceanbase-ce-libspackage, sorpm2cpio | cpiointo/opt/oceanbaseruns unmodified on Ubuntu — on x86_64 and aarch64 alike, so thec8gmachines are covered too. Nothing is installed system-wide.What was surprising
./install. A freshly started observer serves no tenant at all — it answersTenant not in this serveruntilALTER SYSTEM BOOTSTRAPhas created the internalsystenant, and user data then needs a tenant of its own, which needs a resource pool, which needs a unit config. All of that has to finish before the driver's first./check../loadhas to force a minor freeze of the meta tenants, or the entry produces no result at all. The direct load leaves ~100 MB of redo log on the meta tenant that every user tenant carries, and that stream'sbase_lsnstays at zero, so every./startreplays the whole thing. How much replay a tenant can buffer is bounded by its memtable, and a meta tenant's memtable is about 4% of the resource unit's memory — 410 MB on a 9 GB unit. The backlog does not fit, replay stalls withCLOG pending size in task queue exceeds limit, the observer never reachesstart success, and./checktimes out on all 43 queries. Measured on the unit whose meta tenant gets exactly that 410 MB: without the freeze the server had not finished starting after ten minutes, twice; with it, 27 seconds.TENANT = allcovers the sys and user tenants — the meta tenants needall_meta.ERROR 4013: No memory or reach tenant memory limit, rolling everything back.installcaps it at one worker per 512 MB of tenant memory, which on every machine this benchmark runs on is a no-op.secure_file_privcan only be set over a Unix socket, never over TCP, soinstalluses-S /opt/oceanbase/run/sql.sockfor that one statement.duis useless for the data size. The observer preallocates its data file and its redo log pool at startup, sodureports the reservation — 159 GiB against 21.1 GiB of real content on the full dataset../data-sizereadsDATA_DISK_IN_USE + LOG_DISK_IN_USEfromGV$OB_SERVERSinstead: 11.9 GiB of columns plus 9.2 GiB of redo log.VARCHAR, notTEXT, and the widths are not free.TEXTis a LOB type in OceanBase and the benchmark's hottest columns areURL,TitleandReferer.VARCHARneeds a declared width, and OceanBase caps a row at 1.5 MB of declared width, which 28VARCHAR(65535)columns exceed fivefold. The widths are at least 4× the longest value the dataset actually holds, measured in the loaded table.Configuration
installsizes the instance the wayobd, the vendor's deployer, does —memory_limitat 80% of RAM,system_memoryfrom obd's step function,cpu_countatnproc - 2— plus theolapparameter and system-variable template that ships inside the RPM itself (etc/default_parameter.json,etc/default_system_variable.json) and that obd and OCP offer as a dropdown. Two of those entries change results and not just speed:utf8mb4_binmakesLIKEandORDER BYbyte-exact, which is what the reference ClickHouse results do, andparallel_degree_policy = AUTOis what lets a query use more than one thread without a hint in the SQL.installrefuses to run below 8 GB of RAM, the vendor's documented minimum.Query results
queries.sqldiffers fromclickhouse/queries.sqlon two lines out of 43: Q29'sREGEXP_REPLACEbackreference has to be'$1'(OceanBase follows MySQL 8, where'\1'is the literal character1, and left alone the query collapses every row into one group), and Q43 groups by minute, which themysqlentry's'%H:00:00'gets wrong.Every query was compared against
clickhouse-localon the same rows. 33 of 43 agree exactly; eight areLIMITcutting through tied sort keys (verified — the sort-key multisets match, and in Q24 the tenWatchIDs are the same ten); Q4 differs because ClickHouse overflowsAVG(UserID)in anInt64while OceanBase returns the exact decimal mean; and Q6 differs by 2 in 107907 becauseutf8mb4_binis aPAD SPACEcollation and two phrase pairs differ only in a trailing space.Verification
Run end to end on the full dataset:
./loadgets exactly 99,997,497 rows in, theAPPENDhint's online statistics land with them (num_rows = 99997497), all 43 queries return a result,./stop→./start→./check→ query works, and the driver's ownbench_run_queryandbench_concurrent_qpswere run against the loaded table. The rowstore entry is validated at 1% scale.The machine that ran it is a shared 96-core box, not a benchmark VM, so no timing here means anything — and at 96 cores against a 9 GB tenant, a tenth of the memory per core that any machine in this benchmark has,
AUTOparallelism picks a degree the tenant cannot afford for Q19, Q32 and Q33. Pinning the degree to 13 — what a 16-core machine would choose — runs all three in 22.6 s, 74.6 s and 40.9 s.No results yet; those need runs on the benchmark's own EC2 machines.