From f394235e9a8d02727360fed636839752d9117f25 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Tue, 1 Sep 2026 13:46:40 +0000 Subject: [PATCH 1/5] oceanbase: add entries for the OceanBase column store and row store MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OceanBase is the shared-nothing, Paxos-replicated, multi-tenant DBMS Alibaba started in 2010 and open-sourced under Mulan PSL v2 in 2021: a MySQL-compatible SQL layer — `SELECT VERSION()` answers `5.7.25-OceanBase_CE-v5.0.1.0` — over an LSM-tree storage engine. Since 4.3 that engine can store a table by column instead of by row, and the two are different enough here to justify two entries: `oceanbase` is the column store, which is what the vendor's own OLAP parameter template selects, and `oceanbase-row` is the same setup with `WITH COLUMN GROUP (all columns)`. It installs natively; no container is needed. OceanBase publishes only el7/el8 RPMs, but the observer links nothing outside glibc and a libaio that ships in the `oceanbase-ce-libs` package, so `rpm2cpio | cpio` into /opt/oceanbase runs unmodified on Ubuntu — on x86_64 and aarch64 alike, so the c8g machines are covered too. Nothing is installed system-wide. The whole one-time cluster setup lives in ./install rather than ./load, because a freshly started observer serves no tenant at all: it answers "Tenant not in this server" until ALTER SYSTEM BOOTSTRAP has created the internal sys tenant, and user data then needs a tenant of its own, which needs a resource pool, which needs a unit config. All of that has to be done before the driver's first ./check. The unit is sized from what GV$OB_SERVERS reports as unassigned rather than from arithmetic on memory_limit, because the bootstrap has already given the sys tenant a unit whose size depends on the version. Configuration is obd's, the vendor deployer's: memory_limit at 80% of RAM, system_memory from obd's step function, cpu_count at nproc-2, plus the `olap` parameter and system-variable template that ships inside the RPM itself (etc/default_parameter.json, etc/default_system_variable.json) and that obd and OCP offer as a dropdown. Two of those template entries change results and not just speed: utf8mb4_bin makes LIKE and ORDER BY byte-exact, which is what the reference ClickHouse results do, and parallel_degree_policy = AUTO is what lets a query use more than one thread without a hint in the SQL. ./load goes through LOAD DATA INFILE with the APPEND hint — OceanBase's bypass (direct) load, which sorts by primary key and writes straight into major SSTables, skipping the SQL layer, the transaction layer and the memtable, and which also collects optimizer statistics on the way so no separate ANALYZE is needed. Parallelism is the tenant's core count capped at one worker per 512 MB of tenant memory: each direct-load worker holds a sort area, a macroblock writer and a 7 MB coroutine stack, and at 90 workers against a 9 GB tenant the load dies with "No memory or reach tenant memory limit" and rolls back. On the benchmark's own machines the core count is the smaller of the two limits. ./load then forces a minor freeze of the user tenants *and* of the meta tenants and waits for it, without which the entry produces no result at all. The direct load leaves ~100 MB of redo log on the meta tenant that every user tenant carries, and that stream's base_lsn -- its checkpoint -- stays at zero, so every ./start replays the whole thing; how much replay a tenant can buffer is bounded by its memtable, and a meta tenant's memtable is about 4% of the resource unit's memory, 410 MB on a 9 GB unit. The backlog does not fit, replay stalls with "CLOG pending size in task queue exceeds limit", the observer never reaches "start success", and ./check times out on all 43 queries. Measured on the unit whose meta tenant gets exactly that 410 MB: without the freeze the server had not finished starting after ten minutes, twice; with it, 27 seconds. TENANT = all covers the sys and user tenants; the meta tenants need all_meta. For the same reason in reverse, benchmark.sh raises BENCH_CHECK_TIMEOUT to 900 s -- the observer's start is not instant even with nothing to replay, and a ./check that times out aborts the whole run rather than one query. ./data-size reports DATA_DISK_IN_USE + LOG_DISK_IN_USE from GV$OB_SERVERS rather than du of the store directory: the observer preallocates both its data file and its redo log pool at startup, so du reports the reservation and would say the same thing about an empty database as about a loaded one. On the full dataset that is 21.1 GiB -- 11.9 GiB of columns plus 9.2 GiB of redo log -- against 159 GiB of du. queries.sql differs from clickhouse/queries.sql on two lines out of 43. Q29's REGEXP_REPLACE backreference has to be '$1' — OceanBase follows MySQL 8, where '\1' is the literal character 1, and left alone the query collapses every row into a single group. Q43 groups by minute, which the mysql entry's '%H:00:00' gets wrong. Every query was compared against clickhouse-local on the same rows: 33 agree exactly, eight are LIMIT cutting through tied sort keys (verified: the sort-key multisets match, and in Q24 the ten WatchIDs are the same ten), Q4 differs because ClickHouse overflows AVG(UserID) in an Int64 while OceanBase returns the exact decimal mean, and Q6 differs by 2 in 107907 because utf8mb4_bin is a PAD SPACE collation and two phrase pairs differ only in a trailing space. The string columns are VARCHAR rather than TEXT — TEXT is a LOB type in OceanBase and the benchmark's hottest columns are URL, Title and Referer — with widths at least four times the longest value the dataset actually holds, since OceanBase caps a row at 1.5 MB of declared width and 28 VARCHAR(65535) columns exceed that fivefold. Verified end to end on the full dataset: ./load gets exactly 99,997,497 rows in, the APPEND hint's online statistics land with them, all 43 queries return a result, stop/start/check/query works, and the driver's own bench_run_query and bench_concurrent_qps were run against the loaded table. The rowstore entry is validated at 1% scale. No results yet; those need runs on the benchmark's own EC2 machines. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 1 - oceanbase-row/README.md | 51 ++++++ oceanbase-row/benchmark.sh | 9 ++ oceanbase-row/check | 16 ++ oceanbase-row/create.sql | 111 +++++++++++++ oceanbase-row/data-size | 14 ++ oceanbase-row/install | 310 ++++++++++++++++++++++++++++++++++++ oceanbase-row/load | 72 +++++++++ oceanbase-row/queries.sql | 43 +++++ oceanbase-row/query | 45 ++++++ oceanbase-row/start | 27 ++++ oceanbase-row/stop | 12 ++ oceanbase-row/template.json | 11 ++ oceanbase/README.md | 307 +++++++++++++++++++++++++++++++++++ oceanbase/benchmark.sh | 9 ++ oceanbase/check | 16 ++ oceanbase/create.sql | 111 +++++++++++++ oceanbase/data-size | 14 ++ oceanbase/install | 310 ++++++++++++++++++++++++++++++++++++ oceanbase/load | 72 +++++++++ oceanbase/queries.sql | 43 +++++ oceanbase/query | 45 ++++++ oceanbase/start | 27 ++++ oceanbase/stop | 12 ++ oceanbase/template.json | 11 ++ 25 files changed, 1698 insertions(+), 1 deletion(-) create mode 100644 oceanbase-row/README.md create mode 100755 oceanbase-row/benchmark.sh create mode 100755 oceanbase-row/check create mode 100644 oceanbase-row/create.sql create mode 100755 oceanbase-row/data-size create mode 100755 oceanbase-row/install create mode 100755 oceanbase-row/load create mode 100644 oceanbase-row/queries.sql create mode 100755 oceanbase-row/query create mode 100755 oceanbase-row/start create mode 100755 oceanbase-row/stop create mode 100644 oceanbase-row/template.json create mode 100644 oceanbase/README.md create mode 100755 oceanbase/benchmark.sh create mode 100755 oceanbase/check create mode 100644 oceanbase/create.sql create mode 100755 oceanbase/data-size create mode 100755 oceanbase/install create mode 100755 oceanbase/load create mode 100644 oceanbase/queries.sql create mode 100755 oceanbase/query create mode 100755 oceanbase/start create mode 100755 oceanbase/stop create mode 100644 oceanbase/template.json diff --git a/README.md b/README.md index 9fd9423a53..5609856e67 100644 --- a/README.md +++ b/README.md @@ -310,7 +310,6 @@ Please help us add more systems and run the benchmarks on more types of VMs: - [ ] LocustDB - [ ] Manticore Search - [ ] MS SQL Server with Column Store Index (without publishing) -- [ ] OceanBase - [ ] Planetscale (without publishing) - [ ] Redshift Spectrum - [ ] Seafowl diff --git a/oceanbase-row/README.md b/oceanbase-row/README.md new file mode 100644 index 0000000000..a47418f487 --- /dev/null +++ b/oceanbase-row/README.md @@ -0,0 +1,51 @@ +This is the `oceanbase` entry with one change: `hits` is created +`WITH COLUMN GROUP (all columns)` instead of `(each column)`, so it stays in +OceanBase's row store — the LSM-tree layout every table had before 4.3 +introduced the columnstore engine, and still what a plain `CREATE TABLE` gives +you. + +Read `../oceanbase/README.md` first: the RPM unpacking, the bootstrap that has +to happen inside `./install`, the resource unit sized from `GV$OB_SERVERS`, the +`olap` parameter template, the `VARCHAR` widths and the 1.5 MB row limit, the +direct load and the `secure_file_priv` Unix-socket rule are all identical here +and are only described there. + +To run the benchmark: + +``` +./benchmark.sh +``` + +## What differs from the columnstore entry + +- `create.sql` says `WITH COLUMN GROUP (all columns)` — one group holding every + column, which is what a row is. `SHOW CREATE TABLE` is otherwise identical to + the columnstore entry's, down to `COMPRESSION = 'zstd_1.3.8'`, + `SKIP_INDEX_LEVEL = 1`, `DELTA_FORMAT = 'encoding'`, + `MERGE_ENGINE = DELETE_INSERT` and `COLLATE = utf8mb4_bin`; the table is + still sorted by `(CounterID, EventDate, UserID, EventTime, WatchID)`. Only + the physical grouping of the columns changes. + + Spelling the group out matters: with the `olap` template's + `default_table_store_format = 'column'` in force, simply *omitting* the + clause produces a columnstore table, not a rowstore one. +- `install` sets `default_table_store_format = 'row'` rather than `'column'`, + so the parameter and the schema say the same thing. + +Everything else — the same 43 queries, the same load path, the same instance +and tenant sizing — is byte-for-byte the columnstore entry, so the difference +between the two result sets is the storage format and nothing else. + +## Verification + +Validated at 1% scale on the same instance as the columnstore entry: the DDL +produces `WITH COLUMN GROUP(all columns)` with every other table option +identical, the load gets all rows in, and all 43 queries return a result. The +comparison against `clickhouse-local` lands in the same place as the +columnstore entry's — 32 of 43 identical, the rest `LIMIT` ties plus the two +semantic differences (`AVG(UserID)` overflowing in ClickHouse, and +`utf8mb4_bin` being a `PAD SPACE` collation) that `../oceanbase/README.md` +describes. + +The 100-million-row load has only been run for the columnstore entry. No +results yet — those need runs on the benchmark's own EC2 machines. diff --git a/oceanbase-row/benchmark.sh b/oceanbase-row/benchmark.sh new file mode 100755 index 0000000000..6ef40860df --- /dev/null +++ b/oceanbase-row/benchmark.sh @@ -0,0 +1,9 @@ +#!/bin/bash +export BENCH_DOWNLOAD_SCRIPT="download-hits-tsv" +# The observer's startup is not instant even when nothing has to be replayed: +# it re-reads its schema and tablet metadata, and after ./drop_caches all of +# that comes off the disk. Measured here at 27-45 s warm and a few minutes when +# the volume is busy, against the driver's 300 s default -- and a ./check that +# times out aborts the whole run, so give it room. +export BENCH_CHECK_TIMEOUT=900 +exec ../lib/benchmark-common.sh diff --git a/oceanbase-row/check b/oceanbase-row/check new file mode 100755 index 0000000000..6b6fef45ae --- /dev/null +++ b/oceanbase-row/check @@ -0,0 +1,16 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +# obclient talks to the server over TCP and needs no privileges of its own; +# only ./start, ./stop and ./install (which writes under $OB_HOME and has to +# reach the Unix socket) need root. +# +# Check the user tenant, not the sys tenant: after a restart the server accepts +# root@sys well before the tenant's log stream has replayed and its tablets are +# readable, and a query issued in that window fails. +out=$("$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" \ + -A -N -e 'SELECT 1' 2>&1) +[ "$out" = "1" ] diff --git a/oceanbase-row/create.sql b/oceanbase-row/create.sql new file mode 100644 index 0000000000..fe6587827b --- /dev/null +++ b/oceanbase-row/create.sql @@ -0,0 +1,111 @@ +CREATE TABLE hits +( + WatchID BIGINT NOT NULL, + JavaEnable SMALLINT NOT NULL, + Title VARCHAR(16384) NOT NULL, + GoodEvent SMALLINT NOT NULL, + EventTime DATETIME NOT NULL, + EventDate DATE NOT NULL, + CounterID INTEGER NOT NULL, + ClientIP INTEGER NOT NULL, + RegionID INTEGER NOT NULL, + UserID BIGINT NOT NULL, + CounterClass SMALLINT NOT NULL, + OS SMALLINT NOT NULL, + UserAgent SMALLINT NOT NULL, + URL VARCHAR(32768) NOT NULL, + Referer VARCHAR(32768) NOT NULL, + IsRefresh SMALLINT NOT NULL, + RefererCategoryID SMALLINT NOT NULL, + RefererRegionID INTEGER NOT NULL, + URLCategoryID SMALLINT NOT NULL, + URLRegionID INTEGER NOT NULL, + ResolutionWidth SMALLINT NOT NULL, + ResolutionHeight SMALLINT NOT NULL, + ResolutionDepth SMALLINT NOT NULL, + FlashMajor SMALLINT NOT NULL, + FlashMinor SMALLINT NOT NULL, + FlashMinor2 VARCHAR(512) NOT NULL, + NetMajor SMALLINT NOT NULL, + NetMinor SMALLINT NOT NULL, + UserAgentMajor SMALLINT NOT NULL, + UserAgentMinor VARCHAR(255) NOT NULL, + CookieEnable SMALLINT NOT NULL, + JavascriptEnable SMALLINT NOT NULL, + IsMobile SMALLINT NOT NULL, + MobilePhone SMALLINT NOT NULL, + MobilePhoneModel VARCHAR(512) NOT NULL, + Params VARCHAR(8192) NOT NULL, + IPNetworkID INTEGER NOT NULL, + TraficSourceID SMALLINT NOT NULL, + SearchEngineID SMALLINT NOT NULL, + SearchPhrase VARCHAR(8192) NOT NULL, + AdvEngineID SMALLINT NOT NULL, + IsArtifical SMALLINT NOT NULL, + WindowClientWidth SMALLINT NOT NULL, + WindowClientHeight SMALLINT NOT NULL, + ClientTimeZone SMALLINT NOT NULL, + ClientEventTime DATETIME NOT NULL, + SilverlightVersion1 SMALLINT NOT NULL, + SilverlightVersion2 SMALLINT NOT NULL, + SilverlightVersion3 INTEGER NOT NULL, + SilverlightVersion4 SMALLINT NOT NULL, + PageCharset VARCHAR(512) NOT NULL, + CodeVersion INTEGER NOT NULL, + IsLink SMALLINT NOT NULL, + IsDownload SMALLINT NOT NULL, + IsNotBounce SMALLINT NOT NULL, + FUniqID BIGINT NOT NULL, + OriginalURL VARCHAR(32768) NOT NULL, + HID INTEGER NOT NULL, + IsOldCounter SMALLINT NOT NULL, + IsEvent SMALLINT NOT NULL, + IsParameter SMALLINT NOT NULL, + DontCountHits SMALLINT NOT NULL, + WithHash SMALLINT NOT NULL, + HitColor CHAR NOT NULL, + LocalEventTime DATETIME NOT NULL, + Age SMALLINT NOT NULL, + Sex SMALLINT NOT NULL, + Income SMALLINT NOT NULL, + Interests SMALLINT NOT NULL, + Robotness SMALLINT NOT NULL, + RemoteIP INTEGER NOT NULL, + WindowName INTEGER NOT NULL, + OpenerName INTEGER NOT NULL, + HistoryLength SMALLINT NOT NULL, + BrowserLanguage VARCHAR(512) NOT NULL, + BrowserCountry VARCHAR(512) NOT NULL, + SocialNetwork VARCHAR(512) NOT NULL, + SocialAction VARCHAR(512) NOT NULL, + HTTPError SMALLINT NOT NULL, + SendTiming INTEGER NOT NULL, + DNSTiming INTEGER NOT NULL, + ConnectTiming INTEGER NOT NULL, + ResponseStartTiming INTEGER NOT NULL, + ResponseEndTiming INTEGER NOT NULL, + FetchTiming INTEGER NOT NULL, + SocialSourceNetworkID SMALLINT NOT NULL, + SocialSourcePage VARCHAR(2048) NOT NULL, + ParamPrice BIGINT NOT NULL, + ParamOrderID VARCHAR(512) NOT NULL, + ParamCurrency VARCHAR(512) NOT NULL, + ParamCurrencyID SMALLINT NOT NULL, + OpenstatServiceName VARCHAR(2048) NOT NULL, + OpenstatCampaignID VARCHAR(2048) NOT NULL, + OpenstatAdID VARCHAR(2048) NOT NULL, + OpenstatSourceID VARCHAR(2048) NOT NULL, + UTMSource VARCHAR(2048) NOT NULL, + UTMMedium VARCHAR(2048) NOT NULL, + UTMCampaign VARCHAR(2048) NOT NULL, + UTMContent VARCHAR(2048) NOT NULL, + UTMTerm VARCHAR(2048) NOT NULL, + FromTag VARCHAR(2048) NOT NULL, + HasGCLID SMALLINT NOT NULL, + RefererHash BIGINT NOT NULL, + URLHash BIGINT NOT NULL, + CLID INTEGER NOT NULL, + PRIMARY KEY (CounterID, EventDate, UserID, EventTime, WatchID) +) +ORGANIZATION INDEX +WITH COLUMN GROUP (all columns); diff --git a/oceanbase-row/data-size b/oceanbase-row/data-size new file mode 100755 index 0000000000..43a7c39825 --- /dev/null +++ b/oceanbase-row/data-size @@ -0,0 +1,14 @@ +#!/bin/bash +# The observer preallocates both its data file (datafile_size) and its redo log +# pool (log_disk_size) at startup, so `du` on the store directory reports the +# reservation rather than the dataset -- tens of gigabytes of untouched zeroes. +# DATA_DISK_IN_USE and LOG_DISK_IN_USE are the macroblocks and log blocks +# actually occupied, which is the number this benchmark asks for: user data, +# indexes, and transaction log. +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +"$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A -Doceanbase -N \ + -e 'SELECT SUM(DATA_DISK_IN_USE) + SUM(LOG_DISK_IN_USE) FROM GV$OB_SERVERS' diff --git a/oceanbase-row/install b/oceanbase-row/install new file mode 100755 index 0000000000..defba7bc50 --- /dev/null +++ b/oceanbase-row/install @@ -0,0 +1,310 @@ +#!/bin/bash +# OceanBase publishes only el7/el8 RPMs, but the observer links nothing outside +# glibc and a bundled libaio, so the el8 package runs unmodified on Ubuntu once +# it is unpacked with rpm2cpio. Everything lands under $OB_HOME; nothing is +# installed system-wide. +# +# Because a fresh observer serves no tenant until it has been bootstrapped, the +# whole one-time setup (bootstrap, resource unit/pool, tenant, OLAP parameter +# template) happens here rather than in ./load -- the driver runs ./install +# before its first ./check, and ./check has to be able to reach the tenant. +set -eu + +OB_VERSION=${OB_VERSION:-5.0.1.0-100000042026072912} +OBCLIENT_VERSION=${OBCLIENT_VERSION:-2.2.14-1} +LIBOBCLIENT_VERSION=${LIBOBCLIENT_VERSION:-2.2.14-2} +OB_HOME=${OB_HOME:-/opt/oceanbase} +OB_TENANT=${OB_TENANT:-bench} +OB_CLUSTER=${OB_CLUSTER:-clickbench} +OB_MYSQL_PORT=${OB_MYSQL_PORT:-2881} +OB_RPC_PORT=${OB_RPC_PORT:-2882} +# "row" for this entry, "column" for the ../oceanbase one. +OB_STORE_FORMAT=${OB_STORE_FORMAT:-row} + +# The sentinel is written as the very last thing this script does, so an install +# that died halfway -- after the observer had already written its config and +# preallocated its data file, but before the tenant existed -- is retried from +# scratch rather than left half-built for ./check to time out against. +if [ -f "$OB_HOME/.clickbench-installed" ]; then + echo "oceanbase: cluster already initialized in $OB_HOME" >&2 + exit 0 +fi + +case "$(uname -m)" in + x86_64) ob_arch=x86_64 ;; + aarch64) ob_arch=aarch64 ;; + *) echo "oceanbase: unsupported architecture $(uname -m)" >&2; exit 1 ;; +esac +mirror=${OB_MIRROR:-"https://mirrors.aliyun.com/oceanbase/community/stable/el/8/${ob_arch}"} + +sudo apt-get update -y +# rpm2cpio+cpio unpack the RPMs. The -libs package is what carries the bundled +# libaio the observer links against; without it the binary will not start. +sudo apt-get install -y wget rpm2cpio cpio + +pkgdir=$(mktemp -d) +for rpm in "oceanbase-ce-${OB_VERSION}.el8.${ob_arch}.rpm" \ + "oceanbase-ce-libs-${OB_VERSION}.el8.${ob_arch}.rpm" \ + "obclient-${OBCLIENT_VERSION}.el8.${ob_arch}.rpm" \ + "libobclient-${LIBOBCLIENT_VERSION}.el8.${ob_arch}.rpm"; do + wget --continue --tries=5 --progress=dot:giga -O "$pkgdir/$rpm" "$mirror/$rpm" +done + +unpack=$(mktemp -d) +for rpm in "$pkgdir"/*.rpm; do + (cd "$unpack" && rpm2cpio "$rpm" | cpio -idm --quiet) +done + +# The server package unpacks to /home/admin/oceanbase, obclient to /u01/obclient. +sudo rm -rf "$OB_HOME" +sudo mkdir -p "$OB_HOME" +sudo cp -a "$unpack"/home/admin/oceanbase/. "$OB_HOME/" +sudo mkdir -p "$OB_HOME/obclient" +sudo cp -a "$unpack"/u01/obclient/. "$OB_HOME/obclient/" +rm -rf "$pkgdir" "$unpack" + +# store/{clog,slog,sstable} have to exist before the first start: the observer +# does not create them and dies with OB_NO_SUCH_FILE_OR_DIRECTORY if they are +# missing. +sudo mkdir -p "$OB_HOME/store/clog" "$OB_HOME/store/slog" "$OB_HOME/store/sstable" \ + "$OB_HOME/log" "$OB_HOME/run" + +# cpio restores the RPM's uid/gid only when it runs as root, so on a machine +# where the benchmark is driven by a sudo-capable non-root user the tree ends up +# owned by that user while the observer itself runs as root -- and the observer +# refuses to start when the two differ ("current user that starts observer is +# not the same with the original one"). Normalise the ownership instead. +sudo chown -R root:root "$OB_HOME" + +# The observer submits its data I/O through libaio, and one context per disk +# thread quickly exceeds Ubuntu's default fs.aio-max-nr of 65536. +sudo sysctl -w fs.aio-max-nr=1048576 >/dev/null +sudo sysctl -w vm.max_map_count=655360 >/dev/null + +memory_kb=$(awk '/MemTotal/ {print $2}' /proc/meminfo) +threads=$(nproc) +# Free space on the filesystem that will hold the data files, in kB. +disk_kb=$(df -Pk "$OB_HOME" | awk 'NR==2 {print $4}') + +# OceanBase documents 8 GB of RAM as its minimum, and it means it: the observer +# reserves the whole memory_limit up front, and out of that it needs +# system_memory plus a resource unit for the sys tenant plus one for the +# benchmark tenant. Below 8 GB there is no split that leaves a working tenant, +# so say so here rather than let the OOM killer end the run 40 minutes in. +if [ "$memory_kb" -lt 8000000 ]; then + echo "oceanbase: needs at least 8 GB of RAM, this machine has" \ + "$((memory_kb / 1024)) MB; skipping" >&2 + exit 1 +fi + +# 80% of RAM for the whole instance. The observer reserves memory_limit as its +# own arena and never gives it back, so leaving a fifth of the machine to the +# page cache (and to reading hits.tsv during ./load) keeps the kernel OOM killer +# out of the run. +memory_limit_mb=$((memory_kb / 1024 * 4 / 5)) + +# system_memory is carved out of memory_limit for the instance's own bookkeeping +# and cannot be handed to a tenant. These are the steps obd's +# generate_general_config.py uses; picking a number by hand either starves the +# root service or wastes a tenant's worth of RAM. +if [ "$memory_limit_mb" -lt 12288 ]; then system_memory_mb=1024 +elif [ "$memory_limit_mb" -lt 20480 ]; then system_memory_mb=5120 +elif [ "$memory_limit_mb" -lt 40960 ]; then system_memory_mb=6144 +elif [ "$memory_limit_mb" -lt 61440 ]; then system_memory_mb=7168 +elif [ "$memory_limit_mb" -lt 81920 ]; then system_memory_mb=8192 +elif [ "$memory_limit_mb" -lt 102400 ]; then system_memory_mb=9216 +elif [ "$memory_limit_mb" -lt 133120 ]; then system_memory_mb=10240 +else system_memory_mb=$((memory_limit_mb * 8 / 100)) +fi + +# The data file is preallocated at startup, so ask for a fixed slab rather than +# letting it auto-extend in the middle of the load: 35% of the free space, which +# on the 500 GB benchmark volume leaves room for the 75 GB hits.tsv alongside. +# The preallocated size does not distort ./data-size -- that reports the bytes +# actually occupied (see the comment there). +datafile_mb=$((disk_kb / 1024 * 35 / 100)) +[ "$datafile_mb" -lt 20480 ] && datafile_mb=20480 + +# The redo log pool is preallocated too. obd's automatic sizing asks for three +# times the tenant memory, which is meant for a cluster taking continuous +# writes; this benchmark writes once, through a path that bypasses the redo log +# (see ./load), so one times memory_limit is ample. +log_disk_mb=$memory_limit_mb +[ "$log_disk_mb" -lt 8192 ] && log_disk_mb=8192 + +# cpu_count is what the instance believes the machine has; obd reserves two +# cores for the OS. obd also floors it at 8, which would tell a four-core +# machine it has twice the cores it does, so that part is left out. +cpu_count=$((threads > 2 ? threads - 2 : threads)) + +# Every resource unit -- the sys tenant's, created by the bootstrap, and the +# benchmark tenant's -- has to be at least __min_full_resource_pool_memory, +# which defaults to 5 GB. Two of those need 10 GB on top of system_memory, more +# than a c6a.xlarge has in total, and CREATE RESOURCE UNIT is then rejected +# outright. Keep the vendor default where it fits and drop to the 1 GB floor the +# parameter accepts only on machines where it does not. +if [ $((memory_limit_mb - system_memory_mb)) -lt 12288 ]; then + min_pool_memory=1073741824 +else + min_pool_memory=5368709120 +fi + +# Cluster-level half of the vendor's OLAP parameter template, taken verbatim +# from etc/default_parameter.json (scenario "olap") in this very package. It +# only turns off per-statement tracing and the slow-query machinery, neither of +# which means anything for a 43-query analytical sweep, and bounds the syslog. +# +# __min_full_resource_pool_memory is not from the template; see the comment +# where it is computed above. +sudo tee "$OB_HOME/bench.env" >/dev/null </dev/null <&2 + +# No default database in these connections: `oceanbase` does not exist until the +# bootstrap creates it, and obclient fails the connection outright if -D names a +# missing database. +sys() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } +ten() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } + +./start +# The bootstrap statement is itself what creates the sys tenant, so wait for +# root@sys to answer rather than for the tenant ./check looks at. This first +# start is the slow one: it preallocates datafile_size + log_disk_size. +started=no +for i in $(seq 1 600); do + if sys -N -e 'SELECT 1' >/dev/null 2>&1; then + started=yes + break + fi + sleep 1 +done +if [ "$started" != "yes" ]; then + echo "oceanbase: observer did not come up; see $OB_HOME/log/observer.log" >&2 + exit 1 +fi + +sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" + +# A freshly bootstrapped cluster has only the sys tenant, which is reserved for +# cluster metadata; user data goes in a tenant of its own, and a tenant needs a +# resource pool, which needs a unit config. +# +# Size that unit from what the server reports as unassigned rather than from +# arithmetic on memory_limit: the bootstrap gives the sys tenant a unit of its +# own, and how much it takes is a function of the version, not of anything set +# here. Asking for one core or one byte more than is free fails the CREATE with +# "resource not enough to hold 1 unit". +read -r tenant_cpu tenant_memory_mb tenant_log_disk_mb <&2 + exit 1 +fi +echo "oceanbase: tenant unit ${tenant_cpu} cpu / ${tenant_memory_mb}M memory /" \ + "${tenant_log_disk_mb}M log disk" >&2 + +sys -e "CREATE RESOURCE UNIT bench_unit + MAX_CPU = ${tenant_cpu}, MIN_CPU = ${tenant_cpu}, + MEMORY_SIZE = '${tenant_memory_mb}M', + LOG_DISK_SIZE = '${tenant_log_disk_mb}M'" +sys -e "CREATE RESOURCE POOL bench_pool + UNIT = 'bench_unit', UNIT_NUM = 1, ZONE_LIST = ('zone1')" +sys -e "CREATE TENANT ${OB_TENANT} + RESOURCE_POOL_LIST = ('bench_pool'), PRIMARY_ZONE = 'zone1' + SET ob_tcp_invited_nodes = '%'" + +# Degree of parallelism for the bulk load in ./load. The vendor's rule is "up to +# the tenant's core count", but the direct load's per-worker cost is memory as +# well as CPU -- a sort area, a macroblock writer and a 7 MB coroutine stack +# each -- so a machine with many cores and little RAM per core runs the tenant +# out of memory (ERROR 4013) halfway through the load instead of finishing it. +# Cap at one worker per 512 MB of tenant memory. On every machine this benchmark +# runs on the core count is the smaller of the two and this changes nothing. +load_parallel=${tenant_cpu%%.*} +memory_cap=$((tenant_memory_mb / 512)) +[ "$memory_cap" -lt 1 ] && memory_cap=1 +[ "$load_parallel" -gt "$memory_cap" ] && load_parallel=$memory_cap +echo "export OB_LOAD_PARALLEL=${load_parallel}" | sudo tee -a "$OB_HOME/bench.env" >/dev/null +echo "oceanbase: load parallelism ${load_parallel}" >&2 + +ready=no +for i in $(seq 1 600); do + if ten -N -e 'SELECT 1' >/dev/null 2>&1; then + ready=yes + break + fi + sleep 1 +done +if [ "$ready" != "yes" ]; then + echo "oceanbase: tenant ${OB_TENANT} never became reachable" >&2 + exit 1 +fi + +# Tenant-level half of the same OLAP template from etc/default_parameter.json. +# These are storage-engine and optimizer defaults for analytical work, not +# rewrites of anything this benchmark measures. +ten -e " +ALTER SYSTEM SET default_table_store_format = '${OB_STORE_FORMAT}'; +ALTER SYSTEM SET default_table_organization = 'HEAP'; +ALTER SYSTEM SET default_table_merge_engine = 'DELETE_INSERT'; +ALTER SYSTEM SET default_micro_block_format_version = 2; +ALTER SYSTEM SET default_skip_index_level = 1; +ALTER SYSTEM SET default_delta_format = 'encoding'; +ALTER SYSTEM SET default_load_mode = 'DISABLED'; +ALTER SYSTEM SET direct_load_allow_fallback = 0; +ALTER SYSTEM SET _io_read_batch_size = '128K'; +ALTER SYSTEM SET _io_read_redundant_limit_percentage = 50; +ALTER SYSTEM SET _io_callback_thread_count = 64; +ALTER SYSTEM SET _nested_loop_join_enabled = False; +ALTER SYSTEM SET _rtf_creator_max_row_count = 2000000000; +ALTER SYSTEM SET _force_subquery_unnest = True; +ALTER SYSTEM SET _rowsets_max_rows = 512; +ALTER SYSTEM SET max_partition_num = 65536; +" + +# Variables half of the same template, from etc/default_system_variable.json. +# The three collation lines matter beyond performance: utf8mb4_bin makes string +# comparison, ORDER BY and LIKE byte-exact, which is what the reference +# ClickHouse results these numbers get compared against do. +ten -e " +SET GLOBAL ob_query_timeout = 604800000000; +SET GLOBAL ob_trx_timeout = 604800000000; +SET GLOBAL parallel_min_scan_time_threshold = 10; +SET GLOBAL ob_sql_work_area_percentage = 30; +SET GLOBAL parallel_degree_policy = 'AUTO'; +SET GLOBAL collation_server = utf8mb4_bin; +SET GLOBAL collation_connection = utf8mb4_bin; +SET GLOBAL collation_database = utf8mb4_bin; +" + +# LOAD DATA reads a server-side file and the path has to be inside +# secure_file_priv. OceanBase only lets that variable be changed over a local +# Unix socket, never over TCP -- hence the -S connection here. +sudo "$OB_HOME/obc" -S "$OB_HOME/run/sql.sock" -uroot@"$OB_TENANT" -A \ + -e "SET GLOBAL secure_file_priv = '/'" + +sudo touch "$OB_HOME/.clickbench-installed" +./stop diff --git a/oceanbase-row/load b/oceanbase-row/load new file mode 100755 index 0000000000..f34c8313b9 --- /dev/null +++ b/oceanbase-row/load @@ -0,0 +1,72 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +ten() { "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } +sys() { "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } + +# Drop+create so the script is idempotent. +ten -e "DROP DATABASE IF EXISTS hits" +ten -e "CREATE DATABASE hits" +ten -Dhits < create.sql + +# LOAD DATA over a bypass ("direct") load: the rows are converted, sorted by +# primary key and written straight into major SSTables, skipping the SQL layer, +# the transaction layer and the memtable. That is the vendor's documented path +# for initial bulk loads, and for a columnstore table it is also what puts the +# data in its final columnar layout without waiting for a major compaction. +# +# APPEND is the shorthand for direct(true, 0) and additionally turns on online +# statistics collection, so the optimizer has table and column statistics by +# the time the first query arrives -- no separate ANALYZE pass. +# +# INFILE (server side) rather than LOCAL INFILE: a client-side load is fed +# through the SQL layer in packets and cannot use the direct path at all. +ten -Dhits -e "LOAD DATA /*+ APPEND parallel($OB_LOAD_PARALLEL) */ + INFILE '$(pwd)/hits.tsv' + INTO TABLE hits + FIELDS TERMINATED BY '\t' ESCAPED BY '\\\\' + LINES TERMINATED BY '\n'" + +rows=$(ten -Dhits -N -e 'SELECT COUNT(*) FROM hits') +echo "oceanbase: loaded $rows rows" >&2 +if [ "$rows" != "99997497" ]; then + echo "oceanbase: expected 99997497 rows, got $rows" >&2 + exit 1 +fi + +# Push the log-stream checkpoints forward before the driver starts stopping and +# starting the server between queries. Without this the entry cannot produce a +# result at all, and the failure is not obvious: +# +# The direct load leaves about a hundred megabytes of redo log on the *meta* +# tenant that every user tenant carries, and that stream's base_lsn -- its +# checkpoint -- stays at 0, so every ./start replays the whole thing. How much +# replay a tenant can buffer is bounded by its memtable, and the meta tenant's +# memtable is roughly 4% of the resource unit's memory: 410 MB on a 9 GB unit. +# The backlog does not fit, replay stalls with "CLOG pending size in task queue +# exceeds limit", the observer never reaches "start success", and ./check times +# out on every one of the 43 queries. Measured on a 9 GB unit: without the +# freeze the server never finished starting in 10 minutes; with it, 27 seconds. +# +# TENANT = all covers the sys and user tenants; the meta tenants need all_meta, +# which `all` does not include. +sys -e "ALTER SYSTEM MINOR FREEZE TENANT = all" +sys -e "ALTER SYSTEM MINOR FREEZE TENANT = all_meta" +# A minor freeze returns as soon as it is queued, so wait for the dump: every +# log stream outside the sys tenant should have a base_lsn that has moved off +# zero, which is what says its checkpoint is no longer at the beginning of the +# log. Streams shorter than one 64 MB log block never move it and are skipped. +for i in $(seq 1 300); do + stalled=$(sys -N -e "SELECT COUNT(*) FROM oceanbase.GV\$OB_LOG_STAT + WHERE tenant_id <> 1 AND base_lsn = 0 + AND end_lsn > 67108864" 2>/dev/null || echo 1) + [ "$stalled" = "0" ] && break + sleep 1 +done + +# Only remove inputs once the load has been confirmed complete. +rm -f hits.tsv +sync diff --git a/oceanbase-row/queries.sql b/oceanbase-row/queries.sql new file mode 100644 index 0000000000..361197d422 --- /dev/null +++ b/oceanbase-row/queries.sql @@ -0,0 +1,43 @@ +SELECT COUNT(*) FROM hits; +SELECT COUNT(*) FROM hits WHERE AdvEngineID <> 0; +SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits; +SELECT AVG(UserID) FROM hits; +SELECT COUNT(DISTINCT UserID) FROM hits; +SELECT COUNT(DISTINCT SearchPhrase) FROM hits; +SELECT MIN(EventDate), MAX(EventDate) FROM hits; +SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC; +SELECT RegionID, COUNT(DISTINCT UserID) AS u FROM hits GROUP BY RegionID ORDER BY u DESC LIMIT 10; +SELECT RegionID, SUM(AdvEngineID), COUNT(*) AS c, AVG(ResolutionWidth), COUNT(DISTINCT UserID) FROM hits GROUP BY RegionID ORDER BY c DESC LIMIT 10; +SELECT MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT MobilePhone, MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, COUNT(DISTINCT UserID) AS u FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10; +SELECT SearchEngineID, SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT UserID, COUNT(*) FROM hits GROUP BY UserID ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase LIMIT 10; +SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID FROM hits WHERE UserID = 435090932899640449; +SELECT COUNT(*) FROM hits WHERE URL LIKE '%google%'; +SELECT SearchPhrase, MIN(URL), COUNT(*) AS c FROM hits WHERE URL LIKE '%google%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, MIN(URL), MIN(Title), COUNT(*) AS c, COUNT(DISTINCT UserID) FROM hits WHERE Title LIKE '%Google%' AND URL NOT LIKE '%.google.%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10; +SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '$1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits; +SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS c FROM hits GROUP BY URL ORDER BY c DESC LIMIT 10; +SELECT 1, URL, COUNT(*) AS c FROM hits GROUP BY 1, URL ORDER BY c DESC LIMIT 10; +SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, COUNT(*) AS c FROM hits GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND URL <> '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10; +SELECT Title, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND Title <> '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND IsLink <> 0 AND IsDownload = 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT TraficSourceID, SearchEngineID, AdvEngineID, CASE WHEN (SearchEngineID = 0 AND AdvEngineID = 0) THEN Referer ELSE '' END AS Src, URL AS Dst, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, Src, Dst ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT URLHash, EventDate, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 3594120000172545465 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 10 OFFSET 100; +SELECT WindowClientWidth, WindowClientHeight, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND DontCountHits = 0 AND URLHash = 2868770270353813622 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10 OFFSET 10000; +SELECT DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') ORDER BY DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') LIMIT 10 OFFSET 1000; diff --git a/oceanbase-row/query b/oceanbase-row/query new file mode 100755 index 0000000000..61332f82e2 --- /dev/null +++ b/oceanbase-row/query @@ -0,0 +1,45 @@ +#!/bin/bash +# Reads a SQL query from stdin, runs it against the hits database. +# Stdout: query result. +# Stderr: query runtime in fractional seconds on the last line, parsed from +# obclient's "N rows in set (X.YYY sec)" footer. +# Exit non-zero on error. +set -e + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +query=$(cat) + +# -vvv is what makes obclient print the "N rows in set (X.YYY sec)" footer for a +# -e statement; at -vv and below there is no timing to read at all. +out=$("$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A \ + -Dhits -vvv -e "$query" 2>&1) && status=0 || status=$? + +if [ "$status" -ne 0 ] || printf '%s\n' "$out" | grep -qE '^ERROR'; then + printf '%s\n' "$out" >&2 + exit 1 +fi + +# Stdout: the result rows only. -vvv also echoes the statement, draws the box +# borders and signs off with "Bye"; the rows are the lines between the pipes. +printf '%s\n' "$out" | awk -F' *\\| *' '/^\| /{ + row = "" + for (i = 2; i < NF; i++) { row = row (i > 2 ? "\t" : "") $i } + print row +}' + +# "N rows in set (M min S sec)" for anything over a minute, "(S sec)" otherwise. +secs=$(printf '%s\n' "$out" \ + | grep -oP '\((?:[0-9.]+\s+min\s+)?[0-9.]+\s+sec\)' | tail -n1 | tr -d '()') +if [ -z "$secs" ]; then + echo "no timing in obclient output" >&2 + exit 1 +fi + +awk -v s="$secs" ' +BEGIN { + n = split(s, a, /[ \t]+/) + if (n >= 3 && a[2] == "min") { printf "%.3f\n", a[1] * 60 + a[3] } + else { printf "%.3f\n", a[1] } +}' >&2 diff --git a/oceanbase-row/start b/oceanbase-row/start new file mode 100755 index 0000000000..c7a0c6e488 --- /dev/null +++ b/oceanbase-row/start @@ -0,0 +1,27 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +# The observer reads its persisted config from etc/observer.config.bin, but +# passing the full option string on every start keeps the configuration in one +# place (./install wrote it to bench.env) instead of depending on whatever the +# last ALTER SYSTEM left behind. +# +# setsid: the observer daemonizes but stays in the caller's process group, so a +# `timeout` around the driver or a killed shell would take the database with it. +# +# The observer wants a large open-file limit: it keeps a descriptor per worker +# thread plus one per connection, and 655350 is the figure obd's own start check +# insists on. +sudo bash -c " + cd '$OB_HOME' + export LD_LIBRARY_PATH='$OB_HOME/lib' + ulimit -n 655350 + ulimit -c unlimited + exec setsid ./bin/observer \ + -p '$OB_MYSQL_PORT' -P '$OB_RPC_PORT' -z zone1 -n '$OB_CLUSTER' -c 1 \ + -d '$OB_HOME/store' -I 127.0.0.1 -r '127.0.0.1:$OB_RPC_PORT:$OB_MYSQL_PORT' \ + -o '$OB_OPTSTR' +" diff --git a/oceanbase-row/stop b/oceanbase-row/stop new file mode 100755 index 0000000000..6b3332d598 --- /dev/null +++ b/oceanbase-row/stop @@ -0,0 +1,12 @@ +#!/bin/bash + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" 2>/dev/null || true + +# SIGTERM is the observer's graceful shutdown: it stops accepting connections, +# checkpoints, and exits. The pid file is the only handle -- there is no +# `observer stop` subcommand. +if [ -f "$OB_HOME/run/observer.pid" ]; then + sudo kill -TERM "$(sudo cat "$OB_HOME/run/observer.pid")" 2>/dev/null || true +fi +exit 0 diff --git a/oceanbase-row/template.json b/oceanbase-row/template.json new file mode 100644 index 0000000000..453f4dceb4 --- /dev/null +++ b/oceanbase-row/template.json @@ -0,0 +1,11 @@ +{ + "system": "OceanBase (row store)", + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": [ + "C++", + "row-oriented", + "MySQL compatible" + ] +} diff --git a/oceanbase/README.md b/oceanbase/README.md new file mode 100644 index 0000000000..24fa68fbfa --- /dev/null +++ b/oceanbase/README.md @@ -0,0 +1,307 @@ +OceanBase is a distributed relational DBMS started inside Alibaba in 2010 and +open-sourced (Mulan PSL v2) in 2021. It is a shared-nothing, Paxos-replicated, +multi-tenant database with a MySQL-compatible SQL layer — `SELECT VERSION()` +answers `5.7.25-OceanBase_CE-v5.0.1.0` — and an LSM-tree storage engine. Since +4.3 that engine can store a table by column instead of by row, which is what +this entry uses; `../oceanbase-row` is the same setup with the row store, for +comparison. + +This is a single-node deployment: one OBServer process, one zone, one replica. + +To run the benchmark: + +``` +./benchmark.sh +``` + +## Installation + +OceanBase publishes binaries as RPMs for openEuler/CentOS only, so there is no +apt repository to point Ubuntu at. That turns out not to matter: the observer +links nothing but glibc, libm, librt, libpthread, libdl and a `libaio.so.1` +that ships inside the `oceanbase-ce-libs` package. `install` unpacks the el8 +RPMs with `rpm2cpio | cpio` into `/opt/oceanbase` and runs them from there. +Nothing is installed system-wide and no distribution package is touched apart +from `wget`, `rpm2cpio` and `cpio`. + +Four packages are needed, all from the same mirror (`OB_MIRROR` overrides it): + +- `oceanbase-ce` — the observer and `obshell`, ~1 GB unpacked; +- `oceanbase-ce-libs` — only `lib/libaio.so.1`, and without it the observer + will not start; +- `obclient` and `libobclient` — OceanBase's MariaDB-derived CLI. Ubuntu's + `mysql-client` speaks the same protocol, but `obclient` is the client the + vendor ships and tests against, and it is a 23 MB download. + +`install` also raises `fs.aio-max-nr` (the observer runs its data I/O through +libaio and wants more contexts than Ubuntu's default 65536 allows) and +`vm.max_map_count`, and normalises the ownership of `/opt/oceanbase` to root — +`cpio` only restores the RPM's uids when it runs as root, and the observer +refuses to start when the user starting it differs from the owner of its files. + +Both `x86_64` and `aarch64` RPMs exist, so this entry runs on the `c8g.*` +machines too. The mirror also carries `nonlse` aarch64 builds for CPUs without +ARMv8.1 atomics; the Gravitons this benchmark runs on have them, so the normal +build is used. + +## Bootstrapping, and why it happens in `./install` + +A freshly started observer serves no tenant at all: it accepts connections but +answers `ERROR 5150 (HY000): Tenant not in this server` until +`ALTER SYSTEM BOOTSTRAP` has created the internal `sys` tenant. User data does not belong in `sys` either — it needs +a tenant of its own, which needs a resource pool, which needs a unit config. + +That whole sequence has to be finished before the driver's first `./check`, so +`install` does it: start the observer, wait for `root@sys` to answer, bootstrap, +`CREATE RESOURCE UNIT` / `CREATE RESOURCE POOL` / `CREATE TENANT bench`, apply +the parameter template below, and stop again. Everything after that is a plain +`./start`. + +The unit is sized from what `GV$OB_SERVERS` reports as *unassigned* rather than +from arithmetic on `memory_limit`, because the bootstrap has already given the +`sys` tenant a unit and how large that unit is depends on the version. Asking +for one core or one byte more than is free fails the `CREATE` outright with +`resource not enough to hold 1 unit`. + +`./check` connects to the `bench` tenant, not to `sys`. After a restart the +server answers `root@sys` well before the tenant's log stream has replayed, and +a query issued in that window fails — checking `sys` would let the driver start +timing queries too early. + +## Configuration + +`install` sizes the instance the way `obd`, the vendor's deployer, does: + +- `memory_limit` is 80% of RAM. The observer reserves it as one arena at + startup and never returns it, so the remaining fifth is what the page cache + and the `hits.tsv` read during `./load` have to live in. +- `system_memory` — the slice of `memory_limit` reserved for the instance + itself and not assignable to a tenant — follows obd's step function of + `memory_limit` (1 GB below 12 GB, then 5, 6, 7, 8, 9, 10 GB, then 8%). +- `cpu_count` is `nproc - 2`, again as obd does — without obd's additional floor + of 8, which on a four-core machine would tell the instance it has twice the + cores it does. +- `datafile_size` is 35% of the free space on the filesystem, and + `log_disk_size` equals `memory_limit`. Both are *preallocated* at first + start, which is most of why that first start takes a couple of minutes. The + data file has to be generous rather than merely large enough for the table: + the sort that the direct load performs spills its runs into the same block + manager, so at the high-water mark it holds the finished columns and the + temporary runs at once. + obd's automatic sizing would ask for `3 x (memory_limit - system_memory) + + system_memory` of redo log; this benchmark writes the dataset once, through a + path that bypasses the redo log, so `memory_limit` is plenty. +- `__min_full_resource_pool_memory` is the one knob set against the vendor + default, and only on small machines. Every resource unit has to be at least + this large, it defaults to 5 GB, and two of them — the `sys` tenant's and the + benchmark tenant's — then need 10 GB on top of `system_memory`, which no + machine under about 24 GB of RAM can spare. Where that is the case `install` + drops it to the 1 GB floor the parameter accepts, so that + `CREATE RESOURCE UNIT` is not rejected outright; elsewhere the default + stands. + +`install` also refuses to run below 8 GB of RAM, which is the vendor's +documented minimum, so `t3a.small` and `c6a.large` produce a one-line message +rather than an OOM kill partway through the load. + +Everything else comes from `etc/default_parameter.json` and +`etc/default_system_variable.json`, files that ship inside the RPM and hold the +vendor's recommended parameter values for five workload shapes +(`express_oltp`, `complex_oltp`, `htap`, `olap`, `kv`). `install` applies the +`olap` set verbatim; `obd` and OCP offer the same thing as a dropdown when you +create a cluster and a tenant, but there is no single `ALTER SYSTEM SET +scenario` to do it in one statement, so the entries are spelled out. They are +storage-engine and optimizer defaults — column store as the default table +format, heap tables, skip-index level 1, `encoding` delta format, auto DOP, a +larger vectorized batch, bigger read batches — not rewrites of anything the +benchmark measures. `template.json` therefore says `"tuned": "no"`. + +Two of them do change results, not just speed: + +- `collation_server` / `collation_connection` become `utf8mb4_bin`. That makes + `LIKE` and `ORDER BY` byte-exact, so Q21-Q24 match the same rows as + ClickHouse does, rather than the larger case-insensitive sets the + `utf8mb4_general_ci` default would produce (and that the `mysql` and `doris` + entries do produce). +- `parallel_degree_policy = AUTO` lets the optimizer choose the degree of + parallelism per query. Without it OceanBase runs each query on one thread + unless the SQL carries a `/*+ parallel(N) */` hint, and `queries.sql` here + carries no hints at all. + +## Schema + +`create.sql` is the MySQL schema with three changes. + +`WITH COLUMN GROUP (each column)` is what puts the table in the column store. +It is spelled out in the DDL rather than left to the `default_table_store_format += column` parameter the OLAP template sets, so the file says what it builds. + +`ORGANIZATION INDEX` keeps the rows sorted by the primary key. The OLAP +template makes `HEAP` the default — in a heap table the primary key becomes a +separate unique index and the data is stored in arrival order — but sorting by +`(CounterID, EventDate, UserID, EventTime, WatchID)` is what every other +column store in this benchmark does with the same tuple (`DUPLICATE KEY` in +`doris` and `starrocks`, `SORT KEY` in `singlestore`, `ORDER BY` in +`clickhouse`), and it is what lets the Q37-Q42 `CounterID = 62` filter skip +most of the table. + +The string columns are `VARCHAR(n)` rather than `TEXT`. `TEXT` in OceanBase is +a LOB type, and putting the benchmark's hottest columns — `URL`, `Title`, +`Referer` — behind LOB indirection in a column store would be a strange thing +for a real user to do. `VARCHAR` needs a declared width, and the widths cannot +simply all be the maximum: OceanBase caps a row at 1.5 MB of *declared* width +(4 bytes per `utf8mb4` character), which 28 `VARCHAR(65535)` columns exceed +about fivefold, and `CREATE TABLE` fails with `Row size too large`. The widths +in `create.sql` are at least four times the longest value each column actually +holds in the 100 million rows, rounded up to a power of two: + +| column | longest value | declared | +| --- | --- | --- | +| `OriginalURL` | 8134 | 32768 | +| `URL` | 7391 | 32768 | +| `Referer` | 2710 | 32768 | +| `Title` | 1152 | 16384 | +| `SearchPhrase` | 1113 | 8192 | +| `Params` | 993 | 8192 | +| `OpenstatCampaignID`, `UTMCampaign`, `UTMContent`, `UTMTerm`, … | ≤ 208 | 2048 | +| `PageCharset`, `MobilePhoneModel`, `FlashMinor2`, … | ≤ 41 | 512 | +| `UserAgentMinor` | 2 | 255 | +| `HitColor` | 1 | `CHAR` | + +Those are `CHAR_LENGTH`s measured in the loaded table. A 1% sample of the same +rows understates them by up to a factor of three — `Params` peaks at 315 there +against 993 over the whole dataset — so the margin is not decoration. + +The three timestamp columns are `DATETIME`, not the `mysql` entry's +`TIMESTAMP`: `DATETIME` stores what the file says without a session-timezone +round trip. + +## Loading + +`./load` uses `LOAD DATA ... INFILE` with the `APPEND` hint, which is +OceanBase's "bypass" (direct) load: rows are converted, sorted by primary key +and written straight into major SSTables, skipping the SQL layer, the +transaction layer and the memtable. It is the documented path for an initial +bulk load, and for a columnstore table it is also what leaves the data in its +final columnar layout without waiting for a major compaction. `APPEND` is +shorthand for `direct(true, 0)` and additionally turns on online statistics +collection, so the optimizer has table and column statistics by the time the +first query arrives and there is no separate `ANALYZE` pass. + +`INFILE` reads the file on the server side. `LOAD DATA LOCAL INFILE` — what the +`mysql` entry uses — cannot take the direct path at all; it is fed through the +SQL layer in protocol packets. + +The `parallel(N)` degree is the tenant's core count, which is the vendor's rule, +but capped at one worker per 512 MB of tenant memory. Each direct-load worker +holds a sort area, a macroblock writer and a 7 MB coroutine stack, and on a +machine with many cores and little memory per core that adds up: at 90 workers +against a 9 GB tenant the load reached about 10 GB of data and then died with +`ERROR 4013 (HY001): No memory or reach tenant memory limit`, rolling the whole +thing back. On every machine this benchmark runs on the core count is the +smaller of the two limits and the cap changes nothing. + +Server-side reads have to be inside `secure_file_priv`, and OceanBase will only +let that variable be set over a **Unix socket**, never over TCP. `install` does +it through `-S /opt/oceanbase/run/sql.sock`. + +`./load` ends with a minor freeze of the user tenants *and* of the meta tenants, +and waits for it, and without that the entry produces no result at all. The +direct load leaves about a hundred megabytes of redo log on the meta tenant that +every user tenant carries, and that stream's `base_lsn` — its checkpoint — +stays at zero, so every `./start` replays the whole thing. How much replay a +tenant can buffer is bounded by its memtable, and a meta tenant's memtable is +roughly 4% of the resource unit's memory: 410 MB on a 9 GB unit. The backlog +does not fit, replay stalls with `CLOG pending size in task queue exceeds +limit`, the observer never reaches `start success`, and `./check` times out on +every one of the 43 queries. Measured on the unit whose meta tenant gets exactly +that 410 MB: without the freeze the server had not finished starting after ten +minutes, twice; with it, 27 seconds. `TENANT = all` covers the sys and user +tenants — the meta tenants need `all_meta`, which `all` does not include. + +`benchmark.sh` raises the driver's `BENCH_CHECK_TIMEOUT` from 300 s to 900 s for +the same reason in reverse: the observer's start is not instant even with +nothing to replay — it re-reads its schema and tablet metadata, which after the +driver's `drop_caches` all comes off the disk — and a `./check` that times out +aborts the whole run rather than one query. + +One thing to know if you reproduce this on a volume you do not have to yourself: +the load moves well over 100 GB through the disk — 75 GB of `hits.tsv` in, the +sort's runs out and back, then the merged columns — and OceanBase's failure +detector watches how long redo-log writes take. On a contended volume where +write latency reached tens of milliseconds it logged `clog disk may be hung`, +stopped log sync, and the load ended in `ERROR 4012 (HY000): Timeout` with +everything rolled back. The benchmark's own machines have the volume to +themselves and never come near this. + +## Data size + +`./data-size` reports `DATA_DISK_IN_USE + LOG_DISK_IN_USE` from +`GV$OB_SERVERS`, not `du` on the store directory. The observer preallocates +both the data file and the redo log pool at startup, so `du` measures the +reservation — tens of gigabytes of untouched zeroes — and would say the same +thing about an empty database as about a loaded one. The two `IN_USE` counters +are the macroblocks and log blocks actually occupied, which is the number this +benchmark asks for: user data, indexes and transaction log. + +## Query results + +`queries.sql` is the `mysql` entry's set with two edits. The other 41 lines are +byte-identical to `clickhouse/queries.sql`, the reference: + +- Q29's `REGEXP_REPLACE` backreference is `'$1'`. OceanBase follows MySQL 8 + here, where `'\1'` is not a backreference but the literal character `1`; + left alone, the query collapses every row into one group. +- Q43 groups by minute (`'%Y-%m-%d %H:%i:00'`). The reference query is + `DATE_TRUNC('minute', EventTime)`; the `mysql` entry's `'%H:00:00'` truncates + to the hour instead. + +Every query was compared against `clickhouse-local` reading the same rows. 33 +of the 43 agree exactly. The other ten: + +- eight — Q18, Q23, Q24, Q32, Q33, Q36, Q40, Q41 — are `LIMIT` cutting through + a run of tied sort keys, so which rows come back is arbitrary in both + systems. Q18 has no `ORDER BY` at all; in Q32 and Q33 the sort key is + `COUNT(*)` grouped by the unique `WatchID`, so every group ties at 1. Where + the tie can be checked, it checks out: the multiset of sort-key values is + identical, and in Q24 the ten `WatchID`s are the same ten. +- Q4, `SELECT AVG(UserID)`, differs because ClickHouse accumulates the + numerator in an `Int64` and overflows. On the 1% slice it answers + `-702352578971`, while the exact mean — which OceanBase computes in a + decimal, and which ClickHouse reproduces if you ask it for + `sum(toInt128(UserID)) / count()` — is `2532976247401878033`. +- Q6, `COUNT(DISTINCT SearchPhrase)`, answers 107905 against ClickHouse's + 107907 on a 1% slice. `utf8mb4_bin` is a `PAD SPACE` collation, so two pairs + of phrases that differ only in a trailing space compare equal. This is a + property of every MySQL-family collation available here, not of the column + store. + +## Verification + +These scripts were run end to end on the full dataset. `./load` gets exactly +99,997,497 rows in and the `APPEND` hint's online statistics land with it +(`DBA_TAB_STATISTICS` reports `num_rows = 99997497`). `./data-size` reports +21.1 GiB — 11.9 GiB of columns plus 9.2 GiB of redo log — against 159 GiB of +`du` on the same directory, which is what the preallocation looks like. All 43 +queries return a result, `./stop` → `./start` → `./check` → query works, and the +driver's own `bench_run_query` and `bench_concurrent_qps` were run against the +loaded table to check the integration rather than just the scripts. + +Two caveats about the machine that ran it, which is a shared 96-core box, not a +benchmark VM: + +- Its disk is contended, so no timing here means anything. Two earlier attempts + at the load died on it — one with `ERROR 4013` before the parallelism cap + described above existed, one with `clog disk may be hung`. +- 96 cores against a 9 GB tenant is about a tenth of the memory per core that + any machine in this benchmark has. At that ratio `parallel_degree_policy = + AUTO` picks a degree the tenant cannot afford for the three highest- + cardinality aggregations, and Q19, Q32 and Q33 fail with `ERROR 4013`. Pinning + the degree to 13 — what a 16-core machine would choose — runs all three: + 22.6 s, 74.6 s and 40.9 s. Nothing is capped in `queries.sql` for this; the + parallelism policy is left where the vendor's template puts it. + +The correctness comparison against `clickhouse-local` above was done query by +query on a 1% slice. + +No results yet — those need runs on the benchmark's own EC2 machines. diff --git a/oceanbase/benchmark.sh b/oceanbase/benchmark.sh new file mode 100755 index 0000000000..6ef40860df --- /dev/null +++ b/oceanbase/benchmark.sh @@ -0,0 +1,9 @@ +#!/bin/bash +export BENCH_DOWNLOAD_SCRIPT="download-hits-tsv" +# The observer's startup is not instant even when nothing has to be replayed: +# it re-reads its schema and tablet metadata, and after ./drop_caches all of +# that comes off the disk. Measured here at 27-45 s warm and a few minutes when +# the volume is busy, against the driver's 300 s default -- and a ./check that +# times out aborts the whole run, so give it room. +export BENCH_CHECK_TIMEOUT=900 +exec ../lib/benchmark-common.sh diff --git a/oceanbase/check b/oceanbase/check new file mode 100755 index 0000000000..6b6fef45ae --- /dev/null +++ b/oceanbase/check @@ -0,0 +1,16 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +# obclient talks to the server over TCP and needs no privileges of its own; +# only ./start, ./stop and ./install (which writes under $OB_HOME and has to +# reach the Unix socket) need root. +# +# Check the user tenant, not the sys tenant: after a restart the server accepts +# root@sys well before the tenant's log stream has replayed and its tablets are +# readable, and a query issued in that window fails. +out=$("$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" \ + -A -N -e 'SELECT 1' 2>&1) +[ "$out" = "1" ] diff --git a/oceanbase/create.sql b/oceanbase/create.sql new file mode 100644 index 0000000000..3ada9a7dfd --- /dev/null +++ b/oceanbase/create.sql @@ -0,0 +1,111 @@ +CREATE TABLE hits +( + WatchID BIGINT NOT NULL, + JavaEnable SMALLINT NOT NULL, + Title VARCHAR(16384) NOT NULL, + GoodEvent SMALLINT NOT NULL, + EventTime DATETIME NOT NULL, + EventDate DATE NOT NULL, + CounterID INTEGER NOT NULL, + ClientIP INTEGER NOT NULL, + RegionID INTEGER NOT NULL, + UserID BIGINT NOT NULL, + CounterClass SMALLINT NOT NULL, + OS SMALLINT NOT NULL, + UserAgent SMALLINT NOT NULL, + URL VARCHAR(32768) NOT NULL, + Referer VARCHAR(32768) NOT NULL, + IsRefresh SMALLINT NOT NULL, + RefererCategoryID SMALLINT NOT NULL, + RefererRegionID INTEGER NOT NULL, + URLCategoryID SMALLINT NOT NULL, + URLRegionID INTEGER NOT NULL, + ResolutionWidth SMALLINT NOT NULL, + ResolutionHeight SMALLINT NOT NULL, + ResolutionDepth SMALLINT NOT NULL, + FlashMajor SMALLINT NOT NULL, + FlashMinor SMALLINT NOT NULL, + FlashMinor2 VARCHAR(512) NOT NULL, + NetMajor SMALLINT NOT NULL, + NetMinor SMALLINT NOT NULL, + UserAgentMajor SMALLINT NOT NULL, + UserAgentMinor VARCHAR(255) NOT NULL, + CookieEnable SMALLINT NOT NULL, + JavascriptEnable SMALLINT NOT NULL, + IsMobile SMALLINT NOT NULL, + MobilePhone SMALLINT NOT NULL, + MobilePhoneModel VARCHAR(512) NOT NULL, + Params VARCHAR(8192) NOT NULL, + IPNetworkID INTEGER NOT NULL, + TraficSourceID SMALLINT NOT NULL, + SearchEngineID SMALLINT NOT NULL, + SearchPhrase VARCHAR(8192) NOT NULL, + AdvEngineID SMALLINT NOT NULL, + IsArtifical SMALLINT NOT NULL, + WindowClientWidth SMALLINT NOT NULL, + WindowClientHeight SMALLINT NOT NULL, + ClientTimeZone SMALLINT NOT NULL, + ClientEventTime DATETIME NOT NULL, + SilverlightVersion1 SMALLINT NOT NULL, + SilverlightVersion2 SMALLINT NOT NULL, + SilverlightVersion3 INTEGER NOT NULL, + SilverlightVersion4 SMALLINT NOT NULL, + PageCharset VARCHAR(512) NOT NULL, + CodeVersion INTEGER NOT NULL, + IsLink SMALLINT NOT NULL, + IsDownload SMALLINT NOT NULL, + IsNotBounce SMALLINT NOT NULL, + FUniqID BIGINT NOT NULL, + OriginalURL VARCHAR(32768) NOT NULL, + HID INTEGER NOT NULL, + IsOldCounter SMALLINT NOT NULL, + IsEvent SMALLINT NOT NULL, + IsParameter SMALLINT NOT NULL, + DontCountHits SMALLINT NOT NULL, + WithHash SMALLINT NOT NULL, + HitColor CHAR NOT NULL, + LocalEventTime DATETIME NOT NULL, + Age SMALLINT NOT NULL, + Sex SMALLINT NOT NULL, + Income SMALLINT NOT NULL, + Interests SMALLINT NOT NULL, + Robotness SMALLINT NOT NULL, + RemoteIP INTEGER NOT NULL, + WindowName INTEGER NOT NULL, + OpenerName INTEGER NOT NULL, + HistoryLength SMALLINT NOT NULL, + BrowserLanguage VARCHAR(512) NOT NULL, + BrowserCountry VARCHAR(512) NOT NULL, + SocialNetwork VARCHAR(512) NOT NULL, + SocialAction VARCHAR(512) NOT NULL, + HTTPError SMALLINT NOT NULL, + SendTiming INTEGER NOT NULL, + DNSTiming INTEGER NOT NULL, + ConnectTiming INTEGER NOT NULL, + ResponseStartTiming INTEGER NOT NULL, + ResponseEndTiming INTEGER NOT NULL, + FetchTiming INTEGER NOT NULL, + SocialSourceNetworkID SMALLINT NOT NULL, + SocialSourcePage VARCHAR(2048) NOT NULL, + ParamPrice BIGINT NOT NULL, + ParamOrderID VARCHAR(512) NOT NULL, + ParamCurrency VARCHAR(512) NOT NULL, + ParamCurrencyID SMALLINT NOT NULL, + OpenstatServiceName VARCHAR(2048) NOT NULL, + OpenstatCampaignID VARCHAR(2048) NOT NULL, + OpenstatAdID VARCHAR(2048) NOT NULL, + OpenstatSourceID VARCHAR(2048) NOT NULL, + UTMSource VARCHAR(2048) NOT NULL, + UTMMedium VARCHAR(2048) NOT NULL, + UTMCampaign VARCHAR(2048) NOT NULL, + UTMContent VARCHAR(2048) NOT NULL, + UTMTerm VARCHAR(2048) NOT NULL, + FromTag VARCHAR(2048) NOT NULL, + HasGCLID SMALLINT NOT NULL, + RefererHash BIGINT NOT NULL, + URLHash BIGINT NOT NULL, + CLID INTEGER NOT NULL, + PRIMARY KEY (CounterID, EventDate, UserID, EventTime, WatchID) +) +ORGANIZATION INDEX +WITH COLUMN GROUP (each column); diff --git a/oceanbase/data-size b/oceanbase/data-size new file mode 100755 index 0000000000..43a7c39825 --- /dev/null +++ b/oceanbase/data-size @@ -0,0 +1,14 @@ +#!/bin/bash +# The observer preallocates both its data file (datafile_size) and its redo log +# pool (log_disk_size) at startup, so `du` on the store directory reports the +# reservation rather than the dataset -- tens of gigabytes of untouched zeroes. +# DATA_DISK_IN_USE and LOG_DISK_IN_USE are the macroblocks and log blocks +# actually occupied, which is the number this benchmark asks for: user data, +# indexes, and transaction log. +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +"$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A -Doceanbase -N \ + -e 'SELECT SUM(DATA_DISK_IN_USE) + SUM(LOG_DISK_IN_USE) FROM GV$OB_SERVERS' diff --git a/oceanbase/install b/oceanbase/install new file mode 100755 index 0000000000..87254cf9b8 --- /dev/null +++ b/oceanbase/install @@ -0,0 +1,310 @@ +#!/bin/bash +# OceanBase publishes only el7/el8 RPMs, but the observer links nothing outside +# glibc and a bundled libaio, so the el8 package runs unmodified on Ubuntu once +# it is unpacked with rpm2cpio. Everything lands under $OB_HOME; nothing is +# installed system-wide. +# +# Because a fresh observer serves no tenant until it has been bootstrapped, the +# whole one-time setup (bootstrap, resource unit/pool, tenant, OLAP parameter +# template) happens here rather than in ./load -- the driver runs ./install +# before its first ./check, and ./check has to be able to reach the tenant. +set -eu + +OB_VERSION=${OB_VERSION:-5.0.1.0-100000042026072912} +OBCLIENT_VERSION=${OBCLIENT_VERSION:-2.2.14-1} +LIBOBCLIENT_VERSION=${LIBOBCLIENT_VERSION:-2.2.14-2} +OB_HOME=${OB_HOME:-/opt/oceanbase} +OB_TENANT=${OB_TENANT:-bench} +OB_CLUSTER=${OB_CLUSTER:-clickbench} +OB_MYSQL_PORT=${OB_MYSQL_PORT:-2881} +OB_RPC_PORT=${OB_RPC_PORT:-2882} +# "column" for this entry, "row" for the ../oceanbase-row one. +OB_STORE_FORMAT=${OB_STORE_FORMAT:-column} + +# The sentinel is written as the very last thing this script does, so an install +# that died halfway -- after the observer had already written its config and +# preallocated its data file, but before the tenant existed -- is retried from +# scratch rather than left half-built for ./check to time out against. +if [ -f "$OB_HOME/.clickbench-installed" ]; then + echo "oceanbase: cluster already initialized in $OB_HOME" >&2 + exit 0 +fi + +case "$(uname -m)" in + x86_64) ob_arch=x86_64 ;; + aarch64) ob_arch=aarch64 ;; + *) echo "oceanbase: unsupported architecture $(uname -m)" >&2; exit 1 ;; +esac +mirror=${OB_MIRROR:-"https://mirrors.aliyun.com/oceanbase/community/stable/el/8/${ob_arch}"} + +sudo apt-get update -y +# rpm2cpio+cpio unpack the RPMs. The -libs package is what carries the bundled +# libaio the observer links against; without it the binary will not start. +sudo apt-get install -y wget rpm2cpio cpio + +pkgdir=$(mktemp -d) +for rpm in "oceanbase-ce-${OB_VERSION}.el8.${ob_arch}.rpm" \ + "oceanbase-ce-libs-${OB_VERSION}.el8.${ob_arch}.rpm" \ + "obclient-${OBCLIENT_VERSION}.el8.${ob_arch}.rpm" \ + "libobclient-${LIBOBCLIENT_VERSION}.el8.${ob_arch}.rpm"; do + wget --continue --tries=5 --progress=dot:giga -O "$pkgdir/$rpm" "$mirror/$rpm" +done + +unpack=$(mktemp -d) +for rpm in "$pkgdir"/*.rpm; do + (cd "$unpack" && rpm2cpio "$rpm" | cpio -idm --quiet) +done + +# The server package unpacks to /home/admin/oceanbase, obclient to /u01/obclient. +sudo rm -rf "$OB_HOME" +sudo mkdir -p "$OB_HOME" +sudo cp -a "$unpack"/home/admin/oceanbase/. "$OB_HOME/" +sudo mkdir -p "$OB_HOME/obclient" +sudo cp -a "$unpack"/u01/obclient/. "$OB_HOME/obclient/" +rm -rf "$pkgdir" "$unpack" + +# store/{clog,slog,sstable} have to exist before the first start: the observer +# does not create them and dies with OB_NO_SUCH_FILE_OR_DIRECTORY if they are +# missing. +sudo mkdir -p "$OB_HOME/store/clog" "$OB_HOME/store/slog" "$OB_HOME/store/sstable" \ + "$OB_HOME/log" "$OB_HOME/run" + +# cpio restores the RPM's uid/gid only when it runs as root, so on a machine +# where the benchmark is driven by a sudo-capable non-root user the tree ends up +# owned by that user while the observer itself runs as root -- and the observer +# refuses to start when the two differ ("current user that starts observer is +# not the same with the original one"). Normalise the ownership instead. +sudo chown -R root:root "$OB_HOME" + +# The observer submits its data I/O through libaio, and one context per disk +# thread quickly exceeds Ubuntu's default fs.aio-max-nr of 65536. +sudo sysctl -w fs.aio-max-nr=1048576 >/dev/null +sudo sysctl -w vm.max_map_count=655360 >/dev/null + +memory_kb=$(awk '/MemTotal/ {print $2}' /proc/meminfo) +threads=$(nproc) +# Free space on the filesystem that will hold the data files, in kB. +disk_kb=$(df -Pk "$OB_HOME" | awk 'NR==2 {print $4}') + +# OceanBase documents 8 GB of RAM as its minimum, and it means it: the observer +# reserves the whole memory_limit up front, and out of that it needs +# system_memory plus a resource unit for the sys tenant plus one for the +# benchmark tenant. Below 8 GB there is no split that leaves a working tenant, +# so say so here rather than let the OOM killer end the run 40 minutes in. +if [ "$memory_kb" -lt 8000000 ]; then + echo "oceanbase: needs at least 8 GB of RAM, this machine has" \ + "$((memory_kb / 1024)) MB; skipping" >&2 + exit 1 +fi + +# 80% of RAM for the whole instance. The observer reserves memory_limit as its +# own arena and never gives it back, so leaving a fifth of the machine to the +# page cache (and to reading hits.tsv during ./load) keeps the kernel OOM killer +# out of the run. +memory_limit_mb=$((memory_kb / 1024 * 4 / 5)) + +# system_memory is carved out of memory_limit for the instance's own bookkeeping +# and cannot be handed to a tenant. These are the steps obd's +# generate_general_config.py uses; picking a number by hand either starves the +# root service or wastes a tenant's worth of RAM. +if [ "$memory_limit_mb" -lt 12288 ]; then system_memory_mb=1024 +elif [ "$memory_limit_mb" -lt 20480 ]; then system_memory_mb=5120 +elif [ "$memory_limit_mb" -lt 40960 ]; then system_memory_mb=6144 +elif [ "$memory_limit_mb" -lt 61440 ]; then system_memory_mb=7168 +elif [ "$memory_limit_mb" -lt 81920 ]; then system_memory_mb=8192 +elif [ "$memory_limit_mb" -lt 102400 ]; then system_memory_mb=9216 +elif [ "$memory_limit_mb" -lt 133120 ]; then system_memory_mb=10240 +else system_memory_mb=$((memory_limit_mb * 8 / 100)) +fi + +# The data file is preallocated at startup, so ask for a fixed slab rather than +# letting it auto-extend in the middle of the load: 35% of the free space, which +# on the 500 GB benchmark volume leaves room for the 75 GB hits.tsv alongside. +# The preallocated size does not distort ./data-size -- that reports the bytes +# actually occupied (see the comment there). +datafile_mb=$((disk_kb / 1024 * 35 / 100)) +[ "$datafile_mb" -lt 20480 ] && datafile_mb=20480 + +# The redo log pool is preallocated too. obd's automatic sizing asks for three +# times the tenant memory, which is meant for a cluster taking continuous +# writes; this benchmark writes once, through a path that bypasses the redo log +# (see ./load), so one times memory_limit is ample. +log_disk_mb=$memory_limit_mb +[ "$log_disk_mb" -lt 8192 ] && log_disk_mb=8192 + +# cpu_count is what the instance believes the machine has; obd reserves two +# cores for the OS. obd also floors it at 8, which would tell a four-core +# machine it has twice the cores it does, so that part is left out. +cpu_count=$((threads > 2 ? threads - 2 : threads)) + +# Every resource unit -- the sys tenant's, created by the bootstrap, and the +# benchmark tenant's -- has to be at least __min_full_resource_pool_memory, +# which defaults to 5 GB. Two of those need 10 GB on top of system_memory, more +# than a c6a.xlarge has in total, and CREATE RESOURCE UNIT is then rejected +# outright. Keep the vendor default where it fits and drop to the 1 GB floor the +# parameter accepts only on machines where it does not. +if [ $((memory_limit_mb - system_memory_mb)) -lt 12288 ]; then + min_pool_memory=1073741824 +else + min_pool_memory=5368709120 +fi + +# Cluster-level half of the vendor's OLAP parameter template, taken verbatim +# from etc/default_parameter.json (scenario "olap") in this very package. It +# only turns off per-statement tracing and the slow-query machinery, neither of +# which means anything for a 43-query analytical sweep, and bounds the syslog. +# +# __min_full_resource_pool_memory is not from the template; see the comment +# where it is computed above. +sudo tee "$OB_HOME/bench.env" >/dev/null </dev/null <&2 + +# No default database in these connections: `oceanbase` does not exist until the +# bootstrap creates it, and obclient fails the connection outright if -D names a +# missing database. +sys() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } +ten() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } + +./start +# The bootstrap statement is itself what creates the sys tenant, so wait for +# root@sys to answer rather than for the tenant ./check looks at. This first +# start is the slow one: it preallocates datafile_size + log_disk_size. +started=no +for i in $(seq 1 600); do + if sys -N -e 'SELECT 1' >/dev/null 2>&1; then + started=yes + break + fi + sleep 1 +done +if [ "$started" != "yes" ]; then + echo "oceanbase: observer did not come up; see $OB_HOME/log/observer.log" >&2 + exit 1 +fi + +sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" + +# A freshly bootstrapped cluster has only the sys tenant, which is reserved for +# cluster metadata; user data goes in a tenant of its own, and a tenant needs a +# resource pool, which needs a unit config. +# +# Size that unit from what the server reports as unassigned rather than from +# arithmetic on memory_limit: the bootstrap gives the sys tenant a unit of its +# own, and how much it takes is a function of the version, not of anything set +# here. Asking for one core or one byte more than is free fails the CREATE with +# "resource not enough to hold 1 unit". +read -r tenant_cpu tenant_memory_mb tenant_log_disk_mb <&2 + exit 1 +fi +echo "oceanbase: tenant unit ${tenant_cpu} cpu / ${tenant_memory_mb}M memory /" \ + "${tenant_log_disk_mb}M log disk" >&2 + +sys -e "CREATE RESOURCE UNIT bench_unit + MAX_CPU = ${tenant_cpu}, MIN_CPU = ${tenant_cpu}, + MEMORY_SIZE = '${tenant_memory_mb}M', + LOG_DISK_SIZE = '${tenant_log_disk_mb}M'" +sys -e "CREATE RESOURCE POOL bench_pool + UNIT = 'bench_unit', UNIT_NUM = 1, ZONE_LIST = ('zone1')" +sys -e "CREATE TENANT ${OB_TENANT} + RESOURCE_POOL_LIST = ('bench_pool'), PRIMARY_ZONE = 'zone1' + SET ob_tcp_invited_nodes = '%'" + +# Degree of parallelism for the bulk load in ./load. The vendor's rule is "up to +# the tenant's core count", but the direct load's per-worker cost is memory as +# well as CPU -- a sort area, a macroblock writer and a 7 MB coroutine stack +# each -- so a machine with many cores and little RAM per core runs the tenant +# out of memory (ERROR 4013) halfway through the load instead of finishing it. +# Cap at one worker per 512 MB of tenant memory. On every machine this benchmark +# runs on the core count is the smaller of the two and this changes nothing. +load_parallel=${tenant_cpu%%.*} +memory_cap=$((tenant_memory_mb / 512)) +[ "$memory_cap" -lt 1 ] && memory_cap=1 +[ "$load_parallel" -gt "$memory_cap" ] && load_parallel=$memory_cap +echo "export OB_LOAD_PARALLEL=${load_parallel}" | sudo tee -a "$OB_HOME/bench.env" >/dev/null +echo "oceanbase: load parallelism ${load_parallel}" >&2 + +ready=no +for i in $(seq 1 600); do + if ten -N -e 'SELECT 1' >/dev/null 2>&1; then + ready=yes + break + fi + sleep 1 +done +if [ "$ready" != "yes" ]; then + echo "oceanbase: tenant ${OB_TENANT} never became reachable" >&2 + exit 1 +fi + +# Tenant-level half of the same OLAP template from etc/default_parameter.json. +# These are storage-engine and optimizer defaults for analytical work, not +# rewrites of anything this benchmark measures. +ten -e " +ALTER SYSTEM SET default_table_store_format = '${OB_STORE_FORMAT}'; +ALTER SYSTEM SET default_table_organization = 'HEAP'; +ALTER SYSTEM SET default_table_merge_engine = 'DELETE_INSERT'; +ALTER SYSTEM SET default_micro_block_format_version = 2; +ALTER SYSTEM SET default_skip_index_level = 1; +ALTER SYSTEM SET default_delta_format = 'encoding'; +ALTER SYSTEM SET default_load_mode = 'DISABLED'; +ALTER SYSTEM SET direct_load_allow_fallback = 0; +ALTER SYSTEM SET _io_read_batch_size = '128K'; +ALTER SYSTEM SET _io_read_redundant_limit_percentage = 50; +ALTER SYSTEM SET _io_callback_thread_count = 64; +ALTER SYSTEM SET _nested_loop_join_enabled = False; +ALTER SYSTEM SET _rtf_creator_max_row_count = 2000000000; +ALTER SYSTEM SET _force_subquery_unnest = True; +ALTER SYSTEM SET _rowsets_max_rows = 512; +ALTER SYSTEM SET max_partition_num = 65536; +" + +# Variables half of the same template, from etc/default_system_variable.json. +# The three collation lines matter beyond performance: utf8mb4_bin makes string +# comparison, ORDER BY and LIKE byte-exact, which is what the reference +# ClickHouse results these numbers get compared against do. +ten -e " +SET GLOBAL ob_query_timeout = 604800000000; +SET GLOBAL ob_trx_timeout = 604800000000; +SET GLOBAL parallel_min_scan_time_threshold = 10; +SET GLOBAL ob_sql_work_area_percentage = 30; +SET GLOBAL parallel_degree_policy = 'AUTO'; +SET GLOBAL collation_server = utf8mb4_bin; +SET GLOBAL collation_connection = utf8mb4_bin; +SET GLOBAL collation_database = utf8mb4_bin; +" + +# LOAD DATA reads a server-side file and the path has to be inside +# secure_file_priv. OceanBase only lets that variable be changed over a local +# Unix socket, never over TCP -- hence the -S connection here. +sudo "$OB_HOME/obc" -S "$OB_HOME/run/sql.sock" -uroot@"$OB_TENANT" -A \ + -e "SET GLOBAL secure_file_priv = '/'" + +sudo touch "$OB_HOME/.clickbench-installed" +./stop diff --git a/oceanbase/load b/oceanbase/load new file mode 100755 index 0000000000..f34c8313b9 --- /dev/null +++ b/oceanbase/load @@ -0,0 +1,72 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +ten() { "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } +sys() { "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } + +# Drop+create so the script is idempotent. +ten -e "DROP DATABASE IF EXISTS hits" +ten -e "CREATE DATABASE hits" +ten -Dhits < create.sql + +# LOAD DATA over a bypass ("direct") load: the rows are converted, sorted by +# primary key and written straight into major SSTables, skipping the SQL layer, +# the transaction layer and the memtable. That is the vendor's documented path +# for initial bulk loads, and for a columnstore table it is also what puts the +# data in its final columnar layout without waiting for a major compaction. +# +# APPEND is the shorthand for direct(true, 0) and additionally turns on online +# statistics collection, so the optimizer has table and column statistics by +# the time the first query arrives -- no separate ANALYZE pass. +# +# INFILE (server side) rather than LOCAL INFILE: a client-side load is fed +# through the SQL layer in packets and cannot use the direct path at all. +ten -Dhits -e "LOAD DATA /*+ APPEND parallel($OB_LOAD_PARALLEL) */ + INFILE '$(pwd)/hits.tsv' + INTO TABLE hits + FIELDS TERMINATED BY '\t' ESCAPED BY '\\\\' + LINES TERMINATED BY '\n'" + +rows=$(ten -Dhits -N -e 'SELECT COUNT(*) FROM hits') +echo "oceanbase: loaded $rows rows" >&2 +if [ "$rows" != "99997497" ]; then + echo "oceanbase: expected 99997497 rows, got $rows" >&2 + exit 1 +fi + +# Push the log-stream checkpoints forward before the driver starts stopping and +# starting the server between queries. Without this the entry cannot produce a +# result at all, and the failure is not obvious: +# +# The direct load leaves about a hundred megabytes of redo log on the *meta* +# tenant that every user tenant carries, and that stream's base_lsn -- its +# checkpoint -- stays at 0, so every ./start replays the whole thing. How much +# replay a tenant can buffer is bounded by its memtable, and the meta tenant's +# memtable is roughly 4% of the resource unit's memory: 410 MB on a 9 GB unit. +# The backlog does not fit, replay stalls with "CLOG pending size in task queue +# exceeds limit", the observer never reaches "start success", and ./check times +# out on every one of the 43 queries. Measured on a 9 GB unit: without the +# freeze the server never finished starting in 10 minutes; with it, 27 seconds. +# +# TENANT = all covers the sys and user tenants; the meta tenants need all_meta, +# which `all` does not include. +sys -e "ALTER SYSTEM MINOR FREEZE TENANT = all" +sys -e "ALTER SYSTEM MINOR FREEZE TENANT = all_meta" +# A minor freeze returns as soon as it is queued, so wait for the dump: every +# log stream outside the sys tenant should have a base_lsn that has moved off +# zero, which is what says its checkpoint is no longer at the beginning of the +# log. Streams shorter than one 64 MB log block never move it and are skipped. +for i in $(seq 1 300); do + stalled=$(sys -N -e "SELECT COUNT(*) FROM oceanbase.GV\$OB_LOG_STAT + WHERE tenant_id <> 1 AND base_lsn = 0 + AND end_lsn > 67108864" 2>/dev/null || echo 1) + [ "$stalled" = "0" ] && break + sleep 1 +done + +# Only remove inputs once the load has been confirmed complete. +rm -f hits.tsv +sync diff --git a/oceanbase/queries.sql b/oceanbase/queries.sql new file mode 100644 index 0000000000..361197d422 --- /dev/null +++ b/oceanbase/queries.sql @@ -0,0 +1,43 @@ +SELECT COUNT(*) FROM hits; +SELECT COUNT(*) FROM hits WHERE AdvEngineID <> 0; +SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits; +SELECT AVG(UserID) FROM hits; +SELECT COUNT(DISTINCT UserID) FROM hits; +SELECT COUNT(DISTINCT SearchPhrase) FROM hits; +SELECT MIN(EventDate), MAX(EventDate) FROM hits; +SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC; +SELECT RegionID, COUNT(DISTINCT UserID) AS u FROM hits GROUP BY RegionID ORDER BY u DESC LIMIT 10; +SELECT RegionID, SUM(AdvEngineID), COUNT(*) AS c, AVG(ResolutionWidth), COUNT(DISTINCT UserID) FROM hits GROUP BY RegionID ORDER BY c DESC LIMIT 10; +SELECT MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT MobilePhone, MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, COUNT(DISTINCT UserID) AS u FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10; +SELECT SearchEngineID, SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT UserID, COUNT(*) FROM hits GROUP BY UserID ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase LIMIT 10; +SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID FROM hits WHERE UserID = 435090932899640449; +SELECT COUNT(*) FROM hits WHERE URL LIKE '%google%'; +SELECT SearchPhrase, MIN(URL), COUNT(*) AS c FROM hits WHERE URL LIKE '%google%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, MIN(URL), MIN(Title), COUNT(*) AS c, COUNT(DISTINCT UserID) FROM hits WHERE Title LIKE '%Google%' AND URL NOT LIKE '%.google.%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10; +SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '$1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits; +SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS c FROM hits GROUP BY URL ORDER BY c DESC LIMIT 10; +SELECT 1, URL, COUNT(*) AS c FROM hits GROUP BY 1, URL ORDER BY c DESC LIMIT 10; +SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, COUNT(*) AS c FROM hits GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND URL <> '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10; +SELECT Title, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND Title <> '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND IsLink <> 0 AND IsDownload = 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT TraficSourceID, SearchEngineID, AdvEngineID, CASE WHEN (SearchEngineID = 0 AND AdvEngineID = 0) THEN Referer ELSE '' END AS Src, URL AS Dst, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, Src, Dst ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT URLHash, EventDate, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 3594120000172545465 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 10 OFFSET 100; +SELECT WindowClientWidth, WindowClientHeight, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND DontCountHits = 0 AND URLHash = 2868770270353813622 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10 OFFSET 10000; +SELECT DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') ORDER BY DATE_FORMAT(EventTime, '%Y-%m-%d %H:%i:00') LIMIT 10 OFFSET 1000; diff --git a/oceanbase/query b/oceanbase/query new file mode 100755 index 0000000000..61332f82e2 --- /dev/null +++ b/oceanbase/query @@ -0,0 +1,45 @@ +#!/bin/bash +# Reads a SQL query from stdin, runs it against the hits database. +# Stdout: query result. +# Stderr: query runtime in fractional seconds on the last line, parsed from +# obclient's "N rows in set (X.YYY sec)" footer. +# Exit non-zero on error. +set -e + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +query=$(cat) + +# -vvv is what makes obclient print the "N rows in set (X.YYY sec)" footer for a +# -e statement; at -vv and below there is no timing to read at all. +out=$("$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A \ + -Dhits -vvv -e "$query" 2>&1) && status=0 || status=$? + +if [ "$status" -ne 0 ] || printf '%s\n' "$out" | grep -qE '^ERROR'; then + printf '%s\n' "$out" >&2 + exit 1 +fi + +# Stdout: the result rows only. -vvv also echoes the statement, draws the box +# borders and signs off with "Bye"; the rows are the lines between the pipes. +printf '%s\n' "$out" | awk -F' *\\| *' '/^\| /{ + row = "" + for (i = 2; i < NF; i++) { row = row (i > 2 ? "\t" : "") $i } + print row +}' + +# "N rows in set (M min S sec)" for anything over a minute, "(S sec)" otherwise. +secs=$(printf '%s\n' "$out" \ + | grep -oP '\((?:[0-9.]+\s+min\s+)?[0-9.]+\s+sec\)' | tail -n1 | tr -d '()') +if [ -z "$secs" ]; then + echo "no timing in obclient output" >&2 + exit 1 +fi + +awk -v s="$secs" ' +BEGIN { + n = split(s, a, /[ \t]+/) + if (n >= 3 && a[2] == "min") { printf "%.3f\n", a[1] * 60 + a[3] } + else { printf "%.3f\n", a[1] } +}' >&2 diff --git a/oceanbase/start b/oceanbase/start new file mode 100755 index 0000000000..c7a0c6e488 --- /dev/null +++ b/oceanbase/start @@ -0,0 +1,27 @@ +#!/bin/bash +set -eu + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" + +# The observer reads its persisted config from etc/observer.config.bin, but +# passing the full option string on every start keeps the configuration in one +# place (./install wrote it to bench.env) instead of depending on whatever the +# last ALTER SYSTEM left behind. +# +# setsid: the observer daemonizes but stays in the caller's process group, so a +# `timeout` around the driver or a killed shell would take the database with it. +# +# The observer wants a large open-file limit: it keeps a descriptor per worker +# thread plus one per connection, and 655350 is the figure obd's own start check +# insists on. +sudo bash -c " + cd '$OB_HOME' + export LD_LIBRARY_PATH='$OB_HOME/lib' + ulimit -n 655350 + ulimit -c unlimited + exec setsid ./bin/observer \ + -p '$OB_MYSQL_PORT' -P '$OB_RPC_PORT' -z zone1 -n '$OB_CLUSTER' -c 1 \ + -d '$OB_HOME/store' -I 127.0.0.1 -r '127.0.0.1:$OB_RPC_PORT:$OB_MYSQL_PORT' \ + -o '$OB_OPTSTR' +" diff --git a/oceanbase/stop b/oceanbase/stop new file mode 100755 index 0000000000..6b3332d598 --- /dev/null +++ b/oceanbase/stop @@ -0,0 +1,12 @@ +#!/bin/bash + +OB_HOME=${OB_HOME:-/opt/oceanbase} +source "$OB_HOME/bench.env" 2>/dev/null || true + +# SIGTERM is the observer's graceful shutdown: it stops accepting connections, +# checkpoints, and exits. The pid file is the only handle -- there is no +# `observer stop` subcommand. +if [ -f "$OB_HOME/run/observer.pid" ]; then + sudo kill -TERM "$(sudo cat "$OB_HOME/run/observer.pid")" 2>/dev/null || true +fi +exit 0 diff --git a/oceanbase/template.json b/oceanbase/template.json new file mode 100644 index 0000000000..c2168904f1 --- /dev/null +++ b/oceanbase/template.json @@ -0,0 +1,11 @@ +{ + "system": "OceanBase", + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": [ + "C++", + "column-oriented", + "MySQL compatible" + ] +} From 0cd87b976a2549510daf9ed3c881b58914b7bdde Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Tue, 1 Sep 2026 21:54:48 +0000 Subject: [PATCH 2/5] oceanbase: retry the bootstrap, and say why it failed The c6a.4xlarge run died at ALTER SYSTEM BOOTSTRAP with a bare "ERROR 4015 (HY000): System error" and nothing else in the log, so there was nothing to act on. Rerunning ./install locally with that machine's exact derived configuration -- memory_limit 25102M, system_memory 6144M, cpu_count 14, a 162 GB data file and a 24.5 GB log disk -- bootstraps fine on this box, so whatever it is does not follow from the numbers. Three changes, in order of usefulness: - On failure, dump the observer's own log: the lines mentioning bootstrap, CHECK_SERVER_EMPTY, WDIAG/EDIAG/ERROR, plus the tail of observer.log and of observer.log.wf. The next run will say what the server actually objected to instead of leaving us to guess. - Give the bootstrap up to three attempts. Retrying is not just a matter of re-running the statement: the observer decides whether it may be bootstrapped by looking for its own leftovers, and those are spread around $OB_HOME rather than confined to store/ -- etc2/, etc3/, wallet/, audit/ and two generated files in etc/. Leave any of them behind and the retry is refused with "Server is not empty but has never been bootstrapped ... has_data_version_file=TRUE", which is exactly what happened when a first attempt at this cleared only store/. reset_observer_state clears all of them, and re-bootstrapping from that state was verified to work. - Wait 30 s after root@sys first answers before bootstrapping. The SQL port comes up before the rest of the server has settled and the bootstrap's first act is a round of RPCs to every address in the rs_list; an instant start was reproduced here by putting the store on tmpfs, and while that did bootstrap cleanly, firing the statement into a server that answered a second ago is not worth the risk when the wait is free. Also fixes the shutdown wait in the retry path to use the pid file: the observer is launched as ./bin/observer from $OB_HOME, so pgrep on an absolute path never matches it. Co-Authored-By: Claude Opus 5 (1M context) --- oceanbase-row/install | 78 +++++++++++++++++++++++++++++++++++++------ oceanbase/README.md | 10 ++++++ oceanbase/install | 78 +++++++++++++++++++++++++++++++++++++------ 3 files changed, 146 insertions(+), 20 deletions(-) diff --git a/oceanbase-row/install b/oceanbase-row/install index defba7bc50..08e03a7c0a 100755 --- a/oceanbase-row/install +++ b/oceanbase-row/install @@ -185,25 +185,83 @@ echo "oceanbase: memory_limit=${memory_limit_mb}M system_memory=${system_memory_ sys() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } ten() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } -./start # The bootstrap statement is itself what creates the sys tenant, so wait for # root@sys to answer rather than for the tenant ./check looks at. This first # start is the slow one: it preallocates datafile_size + log_disk_size. -started=no -for i in $(seq 1 600); do - if sys -N -e 'SELECT 1' >/dev/null 2>&1; then - started=yes +start_and_bootstrap() { + ./start + local i started=no + for i in $(seq 1 900); do + if sys -N -e 'SELECT 1' >/dev/null 2>&1; then + started=yes + break + fi + sleep 1 + done + if [ "$started" != "yes" ]; then + echo "oceanbase: observer did not come up" >&2 + return 1 + fi + # The SQL port answers before the rest of the server has settled, and the + # bootstrap's first act is a round of RPCs to every address in the + # rs_list -- itself, here. Give that a moment rather than firing the + # statement into a half-started process. + sleep 30 + sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" +} + +# A bootstrap that fails cannot simply be retried: the observer decides whether +# it is allowed to be bootstrapped by looking for its own leftovers, and those +# are scattered around $OB_HOME rather than confined to store/ -- etc2/, etc3/, +# wallet/, audit/ and two generated files in etc/. Leave any of them in place +# and the retry is refused with "Server is not empty but has never been +# bootstrapped ... has_data_version_file=TRUE". Clearing all of them puts the +# directory back the way it was before the first ./start. +reset_observer_state() { + ./stop + # The observer is launched as ./bin/observer from $OB_HOME, so pgrep on an + # absolute path would not match it; the pid file is the reliable handle. + local i pid + for i in $(seq 1 120); do + pid=$(sudo cat "$OB_HOME/run/observer.pid" 2>/dev/null || true) + [ -z "$pid" ] && break + sudo kill -0 "$pid" 2>/dev/null || break + sleep 1 + done + sudo rm -rf "$OB_HOME/store" "$OB_HOME/log" "$OB_HOME/run" \ + "$OB_HOME/etc2" "$OB_HOME/etc3" "$OB_HOME/wallet" "$OB_HOME/audit" \ + "$OB_HOME"/etc/observer.config.bin* \ + "$OB_HOME"/etc/observer.data_version.bin* + sudo mkdir -p "$OB_HOME/store/clog" "$OB_HOME/store/slog" "$OB_HOME/store/sstable" \ + "$OB_HOME/log" "$OB_HOME/run" + sudo chown -R root:root "$OB_HOME" +} + +bootstrapped=no +for attempt in 1 2 3; do + if start_and_bootstrap; then + bootstrapped=yes break fi - sleep 1 + # Say why. Without this the benchmark log shows one bare "ERROR 4015 + # (HY000): System error" and nothing to act on. + echo "oceanbase: bootstrap attempt ${attempt} failed. Observer log says:" >&2 + sudo grep -aE "BOOTSTRAP|bootstrap|CHECK_SERVER_EMPTY|WDIAG|EDIAG|ERROR" \ + "$OB_HOME/log/observer.log" 2>/dev/null | tail -n 60 | cut -c1-500 >&2 + echo "oceanbase: ...and the last lines of the log, whatever they are:" >&2 + sudo tail -n 20 "$OB_HOME/log/observer.log" 2>/dev/null | cut -c1-500 >&2 + if [ -s "$OB_HOME/log/observer.log.wf" ]; then + echo "oceanbase: ...and the warning log:" >&2 + sudo tail -n 30 "$OB_HOME/log/observer.log.wf" 2>/dev/null | cut -c1-500 >&2 + fi + [ "$attempt" = 3 ] && break + reset_observer_state done -if [ "$started" != "yes" ]; then - echo "oceanbase: observer did not come up; see $OB_HOME/log/observer.log" >&2 +if [ "$bootstrapped" != "yes" ]; then + echo "oceanbase: could not bootstrap the cluster; giving up" >&2 exit 1 fi -sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" - # A freshly bootstrapped cluster has only the sys tenant, which is reserved for # cluster metadata; user data goes in a tenant of its own, and a tenant needs a # resource pool, which needs a unit config. diff --git a/oceanbase/README.md b/oceanbase/README.md index 24fa68fbfa..864eeb2370 100644 --- a/oceanbase/README.md +++ b/oceanbase/README.md @@ -63,6 +63,16 @@ from arithmetic on `memory_limit`, because the bootstrap has already given the for one core or one byte more than is free fails the `CREATE` outright with `resource not enough to hold 1 unit`. +The bootstrap gets up to three attempts, and a failed one is reported with the +observer's own log rather than just the client's `ERROR 4015 (HY000): System +error`, which says nothing. Retrying is not simply a matter of running the +statement again: the observer decides whether it may be bootstrapped by looking +for its own leftovers, and those are spread around `$OB_HOME` rather than +confined to `store/` — `etc2/`, `etc3/`, `wallet/`, `audit/` and two generated +files in `etc/`. Leave any of them behind and the second attempt is refused with +`Server is not empty but has never been bootstrapped … has_data_version_file= +TRUE`, so `install` clears all of them between attempts. + `./check` connects to the `bench` tenant, not to `sys`. After a restart the server answers `root@sys` well before the tenant's log stream has replayed, and a query issued in that window fails — checking `sys` would let the driver start diff --git a/oceanbase/install b/oceanbase/install index 87254cf9b8..26c2b62760 100755 --- a/oceanbase/install +++ b/oceanbase/install @@ -185,25 +185,83 @@ echo "oceanbase: memory_limit=${memory_limit_mb}M system_memory=${system_memory_ sys() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@sys -A "$@"; } ten() { sudo "$OB_HOME/obc" -h127.0.0.1 -P"$OB_MYSQL_PORT" -uroot@"$OB_TENANT" -A "$@"; } -./start # The bootstrap statement is itself what creates the sys tenant, so wait for # root@sys to answer rather than for the tenant ./check looks at. This first # start is the slow one: it preallocates datafile_size + log_disk_size. -started=no -for i in $(seq 1 600); do - if sys -N -e 'SELECT 1' >/dev/null 2>&1; then - started=yes +start_and_bootstrap() { + ./start + local i started=no + for i in $(seq 1 900); do + if sys -N -e 'SELECT 1' >/dev/null 2>&1; then + started=yes + break + fi + sleep 1 + done + if [ "$started" != "yes" ]; then + echo "oceanbase: observer did not come up" >&2 + return 1 + fi + # The SQL port answers before the rest of the server has settled, and the + # bootstrap's first act is a round of RPCs to every address in the + # rs_list -- itself, here. Give that a moment rather than firing the + # statement into a half-started process. + sleep 30 + sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" +} + +# A bootstrap that fails cannot simply be retried: the observer decides whether +# it is allowed to be bootstrapped by looking for its own leftovers, and those +# are scattered around $OB_HOME rather than confined to store/ -- etc2/, etc3/, +# wallet/, audit/ and two generated files in etc/. Leave any of them in place +# and the retry is refused with "Server is not empty but has never been +# bootstrapped ... has_data_version_file=TRUE". Clearing all of them puts the +# directory back the way it was before the first ./start. +reset_observer_state() { + ./stop + # The observer is launched as ./bin/observer from $OB_HOME, so pgrep on an + # absolute path would not match it; the pid file is the reliable handle. + local i pid + for i in $(seq 1 120); do + pid=$(sudo cat "$OB_HOME/run/observer.pid" 2>/dev/null || true) + [ -z "$pid" ] && break + sudo kill -0 "$pid" 2>/dev/null || break + sleep 1 + done + sudo rm -rf "$OB_HOME/store" "$OB_HOME/log" "$OB_HOME/run" \ + "$OB_HOME/etc2" "$OB_HOME/etc3" "$OB_HOME/wallet" "$OB_HOME/audit" \ + "$OB_HOME"/etc/observer.config.bin* \ + "$OB_HOME"/etc/observer.data_version.bin* + sudo mkdir -p "$OB_HOME/store/clog" "$OB_HOME/store/slog" "$OB_HOME/store/sstable" \ + "$OB_HOME/log" "$OB_HOME/run" + sudo chown -R root:root "$OB_HOME" +} + +bootstrapped=no +for attempt in 1 2 3; do + if start_and_bootstrap; then + bootstrapped=yes break fi - sleep 1 + # Say why. Without this the benchmark log shows one bare "ERROR 4015 + # (HY000): System error" and nothing to act on. + echo "oceanbase: bootstrap attempt ${attempt} failed. Observer log says:" >&2 + sudo grep -aE "BOOTSTRAP|bootstrap|CHECK_SERVER_EMPTY|WDIAG|EDIAG|ERROR" \ + "$OB_HOME/log/observer.log" 2>/dev/null | tail -n 60 | cut -c1-500 >&2 + echo "oceanbase: ...and the last lines of the log, whatever they are:" >&2 + sudo tail -n 20 "$OB_HOME/log/observer.log" 2>/dev/null | cut -c1-500 >&2 + if [ -s "$OB_HOME/log/observer.log.wf" ]; then + echo "oceanbase: ...and the warning log:" >&2 + sudo tail -n 30 "$OB_HOME/log/observer.log.wf" 2>/dev/null | cut -c1-500 >&2 + fi + [ "$attempt" = 3 ] && break + reset_observer_state done -if [ "$started" != "yes" ]; then - echo "oceanbase: observer did not come up; see $OB_HOME/log/observer.log" >&2 +if [ "$bootstrapped" != "yes" ]; then + echo "oceanbase: could not bootstrap the cluster; giving up" >&2 exit 1 fi -sys -e "ALTER SYSTEM BOOTSTRAP ZONE 'zone1' SERVER '127.0.0.1:${OB_RPC_PORT}'" - # A freshly bootstrapped cluster has only the sys tenant, which is reserved for # cluster metadata; user data goes in a tenant of its own, and a tenant needs a # resource pool, which needs a unit config. From bc63329e16af6fc382eecbc1a3f0499479958d97 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 2 Sep 2026 01:30:57 +0000 Subject: [PATCH 3/5] Add benchmark results for oceanbase-row (c6a.4xlarge) --- .../results/20260902/c6a.4xlarge.json | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 oceanbase-row/results/20260902/c6a.4xlarge.json diff --git a/oceanbase-row/results/20260902/c6a.4xlarge.json b/oceanbase-row/results/20260902/c6a.4xlarge.json new file mode 100644 index 0000000000..27e4aa7087 --- /dev/null +++ b/oceanbase-row/results/20260902/c6a.4xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "OceanBase (row store)", + "date": "2026-09-02", + "machine": "c6a.4xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++","row-oriented","MySQL compatible"], + "load_time": 599, + "data_size": 30813454336, + "concurrent_qps": 0.035, + "concurrent_error_ratio": 0, + "result": [ + [2.366, 0.01, 0.01], + [65.584, 66.074, 65.974], + [65.541, 66.103, 65.944], + [65.526, 66.036, 66.019], + [65.69, 65.969, 65.861], + [65.624, 66.071, 65.952], + [65.821, 66, 65.953], + [65.283, 65.688, 65.643], + [65.656, 66.567, 66.501], + [66.409, 67.116, 67.936], + [65.614, 66.052, 65.982], + [65.612, 65.97, 66.007], + [65.658, 65.934, 65.817], + [66.273, 67.094, 66.989], + [65.666, 65.95, 65.966], + [65.657, 65.975, 65.956], + [66.802, 67.59, 67.634], + [66.3, 65.613, 65.699], + [91.874, 80.706, 80.173], + [65.164, 65.594, 65.589], + [65.641, 65.976, 65.965], + [65.198, 65.59, 65.554], + [65.127, 65.619, 65.549], + [65.167, 65.573, 65.562], + [65.026, 65.287, 65.386], + [65.237, 65.541, 65.537], + [65.108, 65.581, 65.541], + [65.635, 65.956, 65.994], + [66.914, 66.767, 67.205], + [65.606, 65.974, 65.961], + [65.635, 65.929, 66.021], + [68.052, 65.253, 65.924], + [128.347, 102.125, 101.933], + [71.6, 75.595, 75.003], + [71.553, 75.981, 75.129], + [65.757, 65.973, 65.952], + [0.29, 0.028, 0.029], + [0.222, 0.015, 0.015], + [0.234, 0.012, 0.011], + [0.305, 0.061, 0.057], + [0.245, 0.012, 0.011], + [0.232, 0.014, 0.014], + [0.236, 0.016, 0.015] +] + } + \ No newline at end of file From c6a96ec739a3ab64961f04b967de9bbb17c2be78 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 2 Sep 2026 07:44:18 +0000 Subject: [PATCH 4/5] Add benchmark results for oceanbase (c8g.4xlarge) --- oceanbase/results/20260902/c8g.4xlarge.json | 60 +++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 oceanbase/results/20260902/c8g.4xlarge.json diff --git a/oceanbase/results/20260902/c8g.4xlarge.json b/oceanbase/results/20260902/c8g.4xlarge.json new file mode 100644 index 0000000000..5f405c4970 --- /dev/null +++ b/oceanbase/results/20260902/c8g.4xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "OceanBase", + "date": "2026-09-02", + "machine": "c8g.4xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++","column-oriented","MySQL compatible"], + "load_time": 943, + "data_size": 24903680000, + "concurrent_qps": 0.208, + "concurrent_error_ratio": 0, + "result": [ + [0.166, 0.002, 0.002], + [0.199, 0.004, 0.004], + [0.419, 0.033, 0.033], + [0.614, 0.021, 0.02], + [1.362, 1.565, 0.489], + [null, null, null], + [0.195, 0.009, 0.009], + [0.203, 0.011, 0.01], + [2.003, 0.701, 0.614], + [13.05, 1.128, 1.549], + [1.992, 0.089, 0.083], + [2.604, 0.099, 0.089], + [1.518, 0.59, 0.267], + [null, null, null], + [3.66, 0.31, 0.282], + [1.441, 0.587, 0.512], + [17.503, 2.052, 4.323], + [2.08, 1.745, 1.742], + [18.238, 18.656, 23.35], + [0.285, 0.004, 0.004], + [6.68, 0.256, 0.26], + [7.769, 0.168, 0.164], + [13.474, 6.169, 8.23], + [7.911, 0.572, 0.869], + [1.247, 0.07, 0.082], + [1.224, 0.082, 0.078], + [3.648, 0.099, 0.099], + [7.197, 0.133, 0.133], + [9.392, 6.765, 6.681], + [0.395, 0.017, 0.016], + [7.246, 0.299, 0.243], + [13.094, 0.392, 0.305], + [40.597, 42.463, 41.536], + [13.406, 10.888, 12.042], + [12.944, 10.3, 12.268], + [0.897, 0.603, 0.602], + [0.213, 0.027, 0.028], + [0.141, 0.011, 0.01], + [0.16, 0.006, 0.005], + [0.225, 0.043, 0.043], + [0.151, 0.007, 0.006], + [0.143, 0.009, 0.008], + [0.142, 0.01, 0.01] +] + } + \ No newline at end of file From cd6b798e741c62590dac31b4b9b6196a7c98685d Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 2 Sep 2026 09:43:56 +0000 Subject: [PATCH 5/5] Add benchmark results for oceanbase-row (c6a.2xlarge) --- .../results/20260902/c6a.2xlarge.json | 60 +++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100644 oceanbase-row/results/20260902/c6a.2xlarge.json diff --git a/oceanbase-row/results/20260902/c6a.2xlarge.json b/oceanbase-row/results/20260902/c6a.2xlarge.json new file mode 100644 index 0000000000..8d89c57a12 --- /dev/null +++ b/oceanbase-row/results/20260902/c6a.2xlarge.json @@ -0,0 +1,60 @@ +{ + "system": "OceanBase (row store)", + "date": "2026-09-02", + "machine": "c6a.2xlarge", + "cluster_size": 1, + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": ["C++","row-oriented","MySQL compatible"], + "load_time": 1054, + "data_size": 24232591360, + "concurrent_qps": 0.025, + "concurrent_error_ratio": 0, + "result": [ + [1.157, 0.015, 0.014], + [64.617, 65.089, 65.014], + [64.574, 65.039, 64.982], + [64.579, 65.066, 64.931], + [69.84, 68.5, 68.477], + [65.279, 65.5, 65.587], + [64.6, 64.979, 64.936], + [64.464, 64.673, 64.672], + [73.165, 72.944, 72.671], + [75.423, 72.315, 72.568], + [64.655, 64.953, 64.955], + [64.688, 64.92, 64.931], + [65, 65.177, 65.237], + [70.122, 70.332, 70.347], + [65.349, 66.557, 66.266], + [66.361, 66.02, 66.351], + [75.843, 73.403, 73.153], + [64.742, 64.765, 64.927], + [95.395, 88.869, 89.404], + [64.324, 64.551, 64.548], + [64.694, 64.931, 64.934], + [64.293, 64.594, 64.471], + [64.318, 64.648, 64.575], + [64.339, 64.555, 64.571], + [64.319, 64.569, 64.628], + [64.299, 64.536, 64.637], + [64.378, 64.57, 64.57], + [64.691, 64.941, 64.982], + [75.291, 69.243, 69.277], + [64.706, 64.956, 64.995], + [68.252, 64.131, 64.952], + [70.58, 67.196, 67.389], + [130.859, 113.651, 114.071], + [124.558, 99.638, 99.63], + [124.527, 99.316, 100.379], + [65.839, 65.671, 65.934], + [0.333, 0.051, 0.051], + [0.291, 0.026, 0.029], + [0.283, 0.018, 0.018], + [0.377, 0.101, 0.096], + [0.283, 0.017, 0.017], + [0.303, 0.019, 0.019], + [0.312, 0.027, 0.03] +] + } + \ No newline at end of file