Add openGauss (row store and column store) - #1725
Open
alexey-milovidov wants to merge 3 commits into
Open
Conversation
openGauss is Huawei's open-source (Mulan PSL v2) DBMS: a PostgreSQL 9.2.4 fork
— the server still reports 9.2.4 — rewritten in C++, with a thread-per-session
model, a vectorized executor and a column store bolted onto the row engine.
Two entries, because those two storage engines behave nothing alike here:
`opengauss` is the row store a plain CREATE TABLE gives you, and
`opengauss-column` is the same setup with ORIENTATION = COLUMN, which is what
the vendor documentation prescribes for OLAP.
It installs natively; no container is needed. openGauss publishes binaries for
openEuler and CentOS only, but the 6.0.5 LTS openEuler 20.03 server tarball
(x86_64 and aarch64) is glibc 2.28 and runs unmodified on Ubuntu once two
sonames are bridged in a private directory on LD_LIBRARY_PATH: libaio.so.1,
which Ubuntu renamed to libaio.so.1t64 in a transition that is an ABI no-op on
64-bit, and libreadline.so.7, which only gsql links and only for line editing.
Everything else non-glibc is bundled in the tarball. The server refuses to run
as root, so install creates the conventional omm user and the other scripts go
through a wrapper that sources the openGauss environment; since omm cannot
traverse root's home on the benchmark VMs, create.sql and hits.tsv reach gsql
on stdin rather than by path.
DBCOMPATIBILITY = 'PG' on CREATE DATABASE is the one thing that is not
optional. openGauss defaults to 'A' (Oracle) compatibility, where the empty
string *is* NULL, and the load stops on the first row of hits.tsv with `null
value in column "referer" violates not-null constraint`. Had the schema not
been NOT NULL throughout, the damage would have been silent instead: 94.4% of
MobilePhoneModel, 86.8% of SearchPhrase, 19.0% of Referer and 14.9% of Title
are empty strings in this dataset, and the 16 queries that filter on <> ''
would have matched nothing.
Two configuration knobs needed sizing that the postgresql entry has no
equivalent for. max_process_memory caps everything the instance allocates and
defaults to 12 GB regardless of machine, which would hold a c7a.metal-48xl to a
twelfth of its RAM; install uses 80%, floored at the 2 GB minimum. And
query_dop, which openGauss needs above its default of 1 to use more than one
thread — worth 9x on Q34 at 100M rows — makes every worker thread take a
connection slot, so the driver's 10-connection throughput test exhausts the
default max_connections = 200 with `No free proc is available to create a new
connection`; max_connections is therefore sized from query_dop. ./start also
runs gs_ctl under setsid: it otherwise leaves the server in the caller's
process group, where anything that signals that group takes the database down.
Both entries were run end to end on the full dataset. Each gets exactly
99,997,497 rows in and answers all 43 queries with no errors and no timeouts;
the row store occupies 86.6 GB, the column store 16 GB for the table and
31.3 GB for the data directory, and the column store loads in a little over
half the time. All 43 queries are the postgresql entry's, unmodified. They were
checked query by query against clickhouse-local on a 1% slice — agreeing
everywhere except where a LIMIT cuts a run of tied sort keys, and on AVG(UserID)
where openGauss is the one that is exactly right — and the two openGauss
engines were then compared against each other on the full 100M rows, agreeing
on 34 of 43 with the other 9 being the same tie-breaking cases.
Two openGauss defects found on the way are written up in
opengauss-column/README.md, neither of which changes what the benchmark
measures:
- Q35 differs from Q34 only by a constant in the GROUP BY list, and runs
12x slower. EXPLAIN VERBOSE gives the reason in one line: the parallel
redistribution is keyed on the leading grouping column, so Q35 gets
`Distribute Key: (1)` and hashes all 100M rows onto a single worker.
query_dop = 48 makes Q35 slower than query_dop = 1; moving the constant to
the end of the list restores `Distribute Key: url` and the runtime.
- An aggregate FILTER clause over a column-store table kills the whole
gaussdb instance — no log entry, no core, redo recovery on next start.
1000 rows reproduce it, and the same statement on a row-store table is
fine.
No results yet — those need runs on the benchmark's own EC2 machines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
deployed
to
benchmark-approval
September 1, 2026 09:10 — with 
GitHub Actions
Active
GitHub Actions
Active
Contributor
|
The run of Logs:
|
The first benchmark run of both entries failed, in two unrelated ways, and neither could reproduce on a development box. Spilling hash aggregations died with `could not create temporary file ...: Too many open files`. openGauss derives max_safe_fds from the shell's `ulimit -n`, which cloud-init leaves at 1024, and then caps it again with max_files_per_process, whose default is 1000 — so the instance had 978 file descriptors for everything, and a machine with a generous default ulimit (this one had 524288) never notices. The `run` wrapper now raises the OS limit to the 1000000 openGauss's own installation guide asks for, falling back to the hard limit, and `install` raises max_files_per_process to 100000; a fresh install verified end to end now logs `max_safe_fds = 99978` instead of 978. The column-store entry spent about 30 minutes in every cold cycle — 18 queries in ten hours against 156 seconds of actual query time — while the row-store entry, same driver and machine, spent about 100 seconds. The only difference between the two configurations was the quarter of RAM the column entry gave cstore_buffers, so it no longer does: the driver restarts the server and drops the page cache before every query, so a large CU cache is never warm when it matters, and 8 GB of it on a 32 GB machine is only a way to run out of memory. The two install scripts are now identical apart from a comment. Two defensive changes in the same area, since a cold cycle is invisible in the log (the driver silences ./start and ./stop): gs_ctl now gets an explicit -t so a stuck lifecycle operation cannot eat a cycle unbounded, and PGCONNECT_TIMEOUT bounds each ./check probe, which a server still replaying WAL would otherwise accept and then sit on. The row-store entry has a third problem that is not a bug and cannot be configured away, now measured and written up in its README: its cold scan runs at about 110 MB/s regardless of the storage under it. On c6a.4xlarge that is 670-720 s for every one of the 43 queries against a ~72 GB table, where the postgresql entry answers in ~258 s against a larger one. On NVMe capable of gigabytes per second the same scan still only manages 107-123 MB/s, and heap_bulk_read_size, the seqscan pre-read knob, does not move it — at query_dop = 1 a large pre-read makes things worse, reading 25 GB to scan a 6.7 GB table. 43 queries x 3 runs x ~700 s is 25 hours, so on any machine whose RAM cannot hold the table this entry cannot finish inside the benchmark's 10-hour cap. It should complete on the large-memory instances, where only the cold run of each query comes off the disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
alexey-milovidov
deployed
to
benchmark-approval
September 1, 2026 20:14 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
|
Results for Logs:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
openGauss is Huawei's open-source (Mulan PSL v2) DBMS: a PostgreSQL 9.2.4 fork — the server still reports
9.2.4— rewritten in C++, with a thread-per-session model, a vectorized executor and a column store alongside the row engine.Two entries, because those two storage engines behave nothing alike on this workload:
opengauss— the row store a plainCREATE TABLEgives you.opengauss-column— the same setup withORIENTATION = COLUMN, which is what the vendor documentation prescribes for OLAP.Installation
Native, no container. openGauss publishes binaries for openEuler and CentOS only, but the 6.0.5 LTS openEuler 20.03 server tarball (
x86_64andaarch64) is glibc 2.28 and runs unmodified on Ubuntu once two sonames are bridged in a private directory onLD_LIBRARY_PATH:libaio.so.1, which Ubuntu renamed tolibaio.so.1t64in a transition that is an ABI no-op on 64-bit, andlibreadline.so.7, which onlygsqllinks and only for line editing. Everything else non-glibc is bundled in the tarball. Docker images exist only for the 7.0.0-RC line, not for 6.0.x LTS.The server refuses to run as root, so
installcreates the conventionalommuser and the other scripts go through a wrapper that sources the openGauss environment. Sinceommcannot traverse root's home on the benchmark VMs,create.sqlandhits.tsvreachgsqlon stdin rather than by path.Three things a naive entry gets wrong
DBCOMPATIBILITY = 'PG'is not optional. openGauss defaults to'A'(Oracle) compatibility, where the empty string is NULL, and the load stops on the first row ofhits.tsvwithnull value in column "referer" violates not-null constraint. Had the schema not beenNOT NULLthroughout the damage would have been silent instead: 94.4% ofMobilePhoneModel, 86.8% ofSearchPhrase, 19.0% ofRefererand 14.9% ofTitleare empty strings in this dataset, and the 16 queries that filter on<> ''would have matched nothing.query_dopandmax_connectionsare coupled. openGauss runs a query on one thread unlessquery_dopis raised above its default of 1 — it is worth 9x on Q34 at 100M rows — but every worker thread then takes a connection slot, so the driver's 10-connection throughput test exhausts the defaultmax_connections = 200withNo free proc is available to create a new connection.installsizes both from the thread count, andmax_process_memory(a hard per-instance ceiling that defaults to 12 GB regardless of machine, which would hold ac7a.metal-48xlto a twelfth of its RAM) fromMemTotal.gs_ctl startdoes not detach. It leaves the server in the caller's process group, where anything that later signals that group takes the database down;./startruns it undersetsid.Verification
Both entries were run end to end on the full dataset:
opengaussopengauss-column./data-size(Load times are from a busy shared machine and are not meant as results.)
All 43 queries are the
postgresqlentry's, unmodified — no rewrites, no substituted functions. They were compared query by query againstclickhouse-localon a 1% slice, agreeing everywhere except where aLIMITcuts through a run of tied sort keys and onAVG(UserID), where openGauss accumulates innumericand is the one that is exactly right. The two openGauss engines were then compared against each other on the full 100M rows: 34 of 43 identical, the other 9 the same tie-breaking cases.No results files — those need runs on the benchmark's own EC2 machines.
Two openGauss defects found on the way
Neither changes what the benchmark measures; both are written up in
opengauss-column/README.md.GROUP BYlist and runs 12x slower.EXPLAIN VERBOSEgives the reason in one line — the parallel redistribution is keyed on the leading grouping column, so Q35 getsDistribute Key: (1)and hashes all 100M rows onto a single worker.query_dop = 48makes Q35 slower thanquery_dop = 1; moving the constant to the end of the list restoresDistribute Key: urland the runtime. The query is left exactly as ClickBench specifies it.FILTERclause over a column-store table kills the wholegaussdbinstance — no log entry, no core, redo recovery on next start. 1000 rows reproduce it and the same statement on a row-store table is fine. No ClickBench query usesFILTER.Both are unfiled: the openGauss tracker is on Gitee.
🤖 Generated with Claude Code