Skip to content

Mixed version tests for citus#8431

Merged
eaydingol merged 12 commits intomainfrom
eag/test-clean-up
Feb 13, 2026
Merged

Mixed version tests for citus#8431
eaydingol merged 12 commits intomainfrom
eag/test-clean-up

Conversation

@eaydingol
Copy link
Copy Markdown
Collaborator

@eaydingol eaydingol commented Dec 31, 2025

Adds support for running regression tests in mixed‑version scenarios

This PR updates the regression test runner script, test schedules, and some regression tests to support running Citus in mixed‑version setups.
With these changes, regression test schedules can be executed in N/N‑1 and mixed‑version scenarios, including:

  • SQL version is N‑1
  • Library version is N‑1
  • A worker is N‑1
  • Coordinator is N‑1

For potential usage, see the workflow jobs removed in commit (0642595) and the corresponding test runs at (https://github.com/citusdata/citus/actions/runs/20746079664).

Details

Schedule refactoring
To enable N‑1 testing for the remaining multi‑node tests, this PR separates tests that drop/create the Citus extension from the existing multi_1_schedule into a new schedule multi_1_create_citus_schedule

This allows the remaining multi‑node tests to be reused safely in N‑1 scenarios.

Test changes

  • Cleanup steps after tests.
  • Minor adjustments to account for the updated schedule structure.

Perl script changes
This PR extends src/test/regress/pg_regress_multi.pl to support:

  • Creating the Citus extension at a specific version for the coordinator (the original PR that introduced versioned extension creation updated only worker logic and missed the coordinator Workflow refactor  #8361)
  • Changing the Citus library (citus.so) for specific nodes (worker, coordinator, or all)
  • Creating the Citus extension at a given version for specific nodes (worker, coordinator, or all)

These changes enable running N/N‑1 compatibility tests in multiple configurations. Sample scenarios are listed below.

Sample scenarios
To demonstrate potential usage sample jobs are added to git workflow but excluded from this PR (see previous comment on the commit and runs).

Test Citus Lib N‑1
All nodes load citus.so from version 13.2.
Test Citus SQL N‑1
All nodes create the Citus extension at version 13.2‑1.
Test Citus Worker N‑1
Only worker 1 loads citus.so from 13.2 and creates the extension at 13.2‑1.
Test Citus Coordinator N‑1
Only the coordinator loads citus.so from 13.2 and creates the extension at 13.2‑1.

Note that the following schedules from “Test Citus” are not included in N‑1 scenarios:

check-multi-1-create-citus, check-multi-mx Some tests in these schedules drop and recreate the Citus extension using the default version, which is incompatible with N‑1 setups.
check-vanilla The test preparation steps have not yet been adapted for N‑1 workflows.

Local testing
Local testing can be performed using sql/versions.sql and test_versions.sh. (removed in previous commit)
Steps:

  • Install the citus 13.2
  • Copy citus.so files from 13.2.0 to ~/citus-libs/17/v13.2.0
  • Install the version from head
  • Run ./test_versions.sh

See version_test_results.txt for example results.
Sample test runs:
CITUSLIBDIR=~/citus-libs/17/v13.2.0 CITUSVERSION=13.2-1 N1MODE=workeronly EXTRA_TESTS=versions make check-minimal
CITUSVERSION=13.2-1 N1MODE=all EXTRA_TESTS=versions make check-minimal

@eaydingol eaydingol marked this pull request as draft December 31, 2025 07:26
@codecov
Copy link
Copy Markdown

codecov Bot commented Dec 31, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.90%. Comparing base (e1875b9) to head (fa1318d).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8431      +/-   ##
==========================================
- Coverage   88.91%   88.90%   -0.02%     
==========================================
  Files         286      286              
  Lines       63103    63103              
  Branches     7908     7908              
==========================================
- Hits        56110    56100      -10     
- Misses       4735     4739       +4     
- Partials     2258     2264       +6     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@eaydingol eaydingol changed the title Split multi-1 schedule Introduce N-1 tests Jan 5, 2026
@eaydingol eaydingol force-pushed the eag/test-clean-up branch 2 times, most recently from 0a86a6b to 723789d Compare January 6, 2026 09:44
@eaydingol eaydingol changed the title Introduce N-1 tests Mixed version tests for citus Jan 6, 2026
@eaydingol eaydingol force-pushed the eag/test-clean-up branch 2 times, most recently from d6e1c53 to b44d601 Compare January 6, 2026 10:52
@eaydingol eaydingol marked this pull request as ready for review January 6, 2026 14:06
@eaydingol eaydingol requested review from ihalatci and m3hm3t January 14, 2026 20:51
@onurctirtir onurctirtir self-requested a review February 10, 2026 12:37
Copy link
Copy Markdown
Member

@onurctirtir onurctirtir left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, optionally, we can make CI happier by adding this into multi_name_resolution.sql

SET client_min_messages TO WARNING;

before the following line

DROP SCHEMA multi_name_resolution CASCADE;

Also, I got confused a bit - won't we add a workflow that can be manually triggered via Actions tab within this PR? Sth that takes Citus version and "n-1 mode" to test against as inputs?

--

Also, it'd be nicer if we can document how to run those tests locally / on CI in src/test/regress/README.md etc.

@eaydingol eaydingol merged commit 08a27dd into main Feb 13, 2026
292 of 293 checks passed
@eaydingol eaydingol deleted the eag/test-clean-up branch February 13, 2026 07:17
eaydingol added a commit that referenced this pull request Mar 3, 2026
Adds support for running regression tests in mixed‑version scenarios


This PR updates the regression test runner script, test schedules, and
some regression tests to support running Citus in mixed‑version setups.
With these changes, regression test schedules can be executed in **N/N‑1
and mixed‑version scenarios**, including:

- SQL version is N‑1
- Library version is N‑1
- A worker is N‑1
- Coordinator is N‑1

For potential usage, see the workflow jobs removed in commit
(0642595)
and the corresponding test runs at
(https://github.com/citusdata/citus/actions/runs/20746079664).


**Details**


**Schedule refactoring**
To enable N‑1 testing for the remaining multi‑node tests, this PR
separates tests that drop/create the Citus extension from the existing
multi_1_schedule into a new schedule `multi_1_create_citus_schedule`

This allows the remaining multi‑node tests to be reused safely in N‑1
scenarios.



**Test changes**
- Cleanup steps after tests.
- Minor adjustments to account for the updated schedule structure.



**Perl script changes**
This PR extends src/test/regress/pg_regress_multi.pl to support:

- Creating the Citus extension at a specific version for the coordinator
(the original PR that introduced versioned extension creation updated
only worker logic and missed the coordinator
#8361)
- Changing the Citus library (citus.so) for specific nodes (worker,
coordinator, or all)
- Creating the Citus extension at a given version for specific nodes
(worker, coordinator, or all)

These changes enable running N/N‑1 compatibility tests in multiple
configurations. Sample scenarios are listed below.


**Sample scenarios**
To demonstrate potential usage sample jobs are added to git workflow but
excluded from this PR (see previous comment on the commit and runs).

**Test Citus Lib N‑1**
All nodes load citus.so from version 13.2.
**Test Citus SQL N‑1**
All nodes create the Citus extension at version 13.2‑1.
**Test Citus Worker N‑1**
Only worker 1 loads citus.so from 13.2 and creates the extension at
13.2‑1.
**Test Citus Coordinator N‑1**
Only the coordinator loads citus.so from 13.2 and creates the extension
at 13.2‑1.

Note that the following schedules from “Test Citus” are not included in
N‑1 scenarios:

`check-multi-1-create-citus, check-multi-mx` Some tests in these
schedules drop and recreate the Citus extension using the default
version, which is incompatible with N‑1 setups.
`check-vanilla` The test preparation steps have not yet been adapted for
N‑1 workflows.

**Local testing**
Local testing can be performed using sql/versions.sql and
test_versions.sh. (removed in previous commit)
Steps:

- Install the citus 13.2
- Copy citus.so files from 13.2.0 to ~/citus-libs/17/v13.2.0
- Install the version from head
- Run ./test_versions.sh

See version_test_results.txt for example results.
Sample test runs: 
CITUSLIBDIR=~/citus-libs/17/v13.2.0 CITUSVERSION=13.2-1
N1MODE=workeronly EXTRA_TESTS=versions make check-minimal
CITUSVERSION=13.2-1 N1MODE=all EXTRA_TESTS=versions make check-minimal
eaydingol added a commit that referenced this pull request Mar 3, 2026
Adds support for running regression tests in mixed‑version scenarios


This PR updates the regression test runner script, test schedules, and
some regression tests to support running Citus in mixed‑version setups.
With these changes, regression test schedules can be executed in **N/N‑1
and mixed‑version scenarios**, including:

- SQL version is N‑1
- Library version is N‑1
- A worker is N‑1
- Coordinator is N‑1

For potential usage, see the workflow jobs removed in commit
(0642595)
and the corresponding test runs at
(https://github.com/citusdata/citus/actions/runs/20746079664).


**Details**


**Schedule refactoring**
To enable N‑1 testing for the remaining multi‑node tests, this PR
separates tests that drop/create the Citus extension from the existing
multi_1_schedule into a new schedule `multi_1_create_citus_schedule`

This allows the remaining multi‑node tests to be reused safely in N‑1
scenarios.



**Test changes**
- Cleanup steps after tests.
- Minor adjustments to account for the updated schedule structure.



**Perl script changes**
This PR extends src/test/regress/pg_regress_multi.pl to support:

- Creating the Citus extension at a specific version for the coordinator
(the original PR that introduced versioned extension creation updated
only worker logic and missed the coordinator
#8361)
- Changing the Citus library (citus.so) for specific nodes (worker,
coordinator, or all)
- Creating the Citus extension at a given version for specific nodes
(worker, coordinator, or all)

These changes enable running N/N‑1 compatibility tests in multiple
configurations. Sample scenarios are listed below.


**Sample scenarios**
To demonstrate potential usage sample jobs are added to git workflow but
excluded from this PR (see previous comment on the commit and runs).

**Test Citus Lib N‑1**
All nodes load citus.so from version 13.2.
**Test Citus SQL N‑1**
All nodes create the Citus extension at version 13.2‑1.
**Test Citus Worker N‑1**
Only worker 1 loads citus.so from 13.2 and creates the extension at
13.2‑1.
**Test Citus Coordinator N‑1**
Only the coordinator loads citus.so from 13.2 and creates the extension
at 13.2‑1.

Note that the following schedules from “Test Citus” are not included in
N‑1 scenarios:

`check-multi-1-create-citus, check-multi-mx` Some tests in these
schedules drop and recreate the Citus extension using the default
version, which is incompatible with N‑1 setups.
`check-vanilla` The test preparation steps have not yet been adapted for
N‑1 workflows.

**Local testing**
Local testing can be performed using sql/versions.sql and
test_versions.sh. (removed in previous commit)
Steps:

- Install the citus 13.2
- Copy citus.so files from 13.2.0 to ~/citus-libs/17/v13.2.0
- Install the version from head
- Run ./test_versions.sh

See version_test_results.txt for example results.
Sample test runs: 
CITUSLIBDIR=~/citus-libs/17/v13.2.0 CITUSVERSION=13.2-1
N1MODE=workeronly EXTRA_TESTS=versions make check-minimal
CITUSVERSION=13.2-1 N1MODE=all EXTRA_TESTS=versions make check-minimal
eaydingol added a commit that referenced this pull request Mar 6, 2026
Adds support for running regression tests in mixed‑version scenarios

This PR updates the regression test runner script, test schedules, and
some regression tests to support running Citus in mixed‑version setups.
With these changes, regression test schedules can be executed in **N/N‑1
and mixed‑version scenarios**, including:

- SQL version is N‑1
- Library version is N‑1
- A worker is N‑1
- Coordinator is N‑1

For potential usage, see the workflow jobs removed in commit
(0642595)
and the corresponding test runs at
(https://github.com/citusdata/citus/actions/runs/20746079664).

**Details**

**Schedule refactoring**
To enable N‑1 testing for the remaining multi‑node tests, this PR
separates tests that drop/create the Citus extension from the existing
multi_1_schedule into a new schedule `multi_1_create_citus_schedule`

This allows the remaining multi‑node tests to be reused safely in N‑1
scenarios.

**Test changes**
- Cleanup steps after tests.
- Minor adjustments to account for the updated schedule structure.

**Perl script changes**
This PR extends src/test/regress/pg_regress_multi.pl to support:

- Creating the Citus extension at a specific version for the coordinator
(the original PR that introduced versioned extension creation updated
only worker logic and missed the coordinator
#8361)
- Changing the Citus library (citus.so) for specific nodes (worker,
coordinator, or all)
- Creating the Citus extension at a given version for specific nodes
(worker, coordinator, or all)

These changes enable running N/N‑1 compatibility tests in multiple
configurations. Sample scenarios are listed below.

**Sample scenarios**
To demonstrate potential usage sample jobs are added to git workflow but
excluded from this PR (see previous comment on the commit and runs).

**Test Citus Lib N‑1**
All nodes load citus.so from version 13.2.
**Test Citus SQL N‑1**
All nodes create the Citus extension at version 13.2‑1.
**Test Citus Worker N‑1**
Only worker 1 loads citus.so from 13.2 and creates the extension at
13.2‑1.
**Test Citus Coordinator N‑1**
Only the coordinator loads citus.so from 13.2 and creates the extension
at 13.2‑1.

Note that the following schedules from “Test Citus” are not included in
N‑1 scenarios:

`check-multi-1-create-citus, check-multi-mx` Some tests in these
schedules drop and recreate the Citus extension using the default
version, which is incompatible with N‑1 setups.
`check-vanilla` The test preparation steps have not yet been adapted for
N‑1 workflows.

**Local testing**
Local testing can be performed using sql/versions.sql and
test_versions.sh. (removed in previous commit)
Steps:

- Install the citus 13.2
- Copy citus.so files from 13.2.0 to ~/citus-libs/17/v13.2.0
- Install the version from head
- Run ./test_versions.sh

See version_test_results.txt for example results.
Sample test runs:
CITUSLIBDIR=~/citus-libs/17/v13.2.0 CITUSVERSION=13.2-1
N1MODE=workeronly EXTRA_TESTS=versions make check-minimal
CITUSVERSION=13.2-1 N1MODE=all EXTRA_TESTS=versions make check-minimal
eaydingol added a commit that referenced this pull request Mar 9, 2026
## Backport mixed-version test infrastructure to release-13.2

This PR cherry-picks a series of commits from `main` that introduce
mixed-version testing support and related fixes to the `release-13.2`
branch. These changes enable running regression tests against different
Citus versions (N/N-1 scenarios), which is essential for validating
backward compatibility during phased upgrades.

### Cherry-picked commits

1. **Relax version check
([#8356](#8356 —
`c125b9bec563`
Relaxes the Citus version compatibility check to allow the installed
extension to differ by at most one minor version from the loaded
library. This enables replicas to lag by a single minor version during
phased upgrades without triggering a version mismatch for distributed
queries.

2. **Workflow refactor
([#8361](#8361 —
`b8ab8aac554c`
Refactors `.github/workflows/build_and_test.yml` to use a shared
reusable workflow (`run_tests.yml`) for the main, failure, and CDC test
suites. Introduces the `CITUSVERSION` environment variable so tests can
create the Citus extension at a specific version (e.g.,
`CITUSVERSION=13.2-1 make check`).

3. **Test refactor for columnar separation
([#8420](#8420 —
`b7a5f698ab77`
Minor test adjustments to prepare for columnar module separation, adding
necessary setup/teardown steps in several test files.

4. **Alternative lib directory for tests
([#8390](#8390 —
`2e64eb020268`
Adds support for loading `citus.so` and `columnar.so` from an
alternative directory during tests via the `CITUSLIBDIR` environment
variable (e.g., `CITUSLIBDIR=/path/to/v13.2.0 make check-minimal`).

5. **Store/restore exit status
([#8436](#8436 —
`1c5b304feeef`
Fixes regression test exit status being lost after test cleanup steps,
ensuring the correct exit code is propagated.

6. **Mixed version tests for citus
([#8431](#8431 —
`08a27ddc0579`
The core mixed-version testing infrastructure. Extends
`pg_regress_multi.pl` to support:
- Creating the Citus extension at a specific version for coordinator
and/or workers
   - Swapping the Citus shared library for specific nodes
- Running N/N-1 compatibility tests in multiple configurations (SQL N-1,
Lib N-1, Worker N-1, Coordinator N-1)

Separates tests that drop/create the Citus extension into a new
`multi_1_create_citus_schedule` to allow safe reuse of remaining
multi-node tests in N-1 scenarios.

7. **Fix IsTenantSchema when version checks are disabled
([#8480](#8480 —
`546f20661a3b`
Fixes `IsTenantSchema()` incorrectly returning `false` when
`citus.enable_version_checks` was off. The previous guard completely
disabled schema-based sharding during mixed-version testing, causing 5
test failures. The fix replaces the guard with a `CheckCitusVersion()`
call, which returns `true` unconditionally when version checks are
disabled.

### Conflicts resolved

- **`build_and_test.yml`** (commits 2 and 6): Resolved by accepting the
incoming reusable workflow pattern and the new
`check-multi-1-create-citus` / `check-tap` make targets.
- **`Makefile`** (commit 6): Resolved by accepting the new
`check-multi-1-create-citus` and `check-tap` targets in `check-full`.

### Files changed

29 files changed, +699 / -210 lines across CI workflows, test
infrastructure (`pg_regress_multi.pl`, `Makefile`), core version
checking logic (`metadata_cache.c`, `columnar_tableam.c`), tenant schema
handling, and regression test SQL/expected output files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants