Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,6 @@ src/main/java/org/mobilitydb/spark/generated/

# Derived MEOS catalog (generated in CI via the provision-meos action)
tools/meos-idl.json

# tools/refresh-from-master.sh scratch (sibling clones + libmeos prefix).
.meos-chain/
44 changes: 15 additions & 29 deletions GENERATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,42 +35,28 @@ so the catalog and jar never drift from the `run.py` that derives them.

The UDF layer is the generated `registerAll()` — zero hand-written registrations.

## Regenerating by hand
## Refreshing against upstream MEOS

CI performs the steps below via `provision-meos`. To run them yourself you need a JDK, Maven,
CMake and the MEOS build dependencies.

**1. Derive libmeos and the catalog from MobilityDB master.** Both come from one commit; see
`MEOS-API/GENERATION.md` for the two commands:
`tools/refresh-from-master.sh` runs the whole chain — deriving libmeos and the catalog from
MobilityDB master, building the JMEOS jar, and regenerating the UDF layer — in one command, on
Ubuntu with a JDK, Maven, CMake and the MEOS build dependencies installed:

```bash
MDB=~/src/MobilityDB # checkout at the commit you are deriving from
MEOSAPI=~/src/MEOS-API
cmake -S "$MDB" -B "$MDB/build" -DCMAKE_BUILD_TYPE=Release -DMEOS=ON -DALL=ON
cmake --build "$MDB/build" -j"$(nproc)"
cmake --install "$MDB/build" --prefix "$MDB/.prefix"
cd "$MEOSAPI" && MDB_SRC_ROOT="$MDB" python3 run.py "$MDB/.prefix/include"
tools/refresh-from-master.sh
```

**2. Build the JMEOS jar against that catalog and install it into the local Maven
repository** under the coordinates this build resolves — `org.jmeos:meos:1.0`:
It runs the shared `MobilityDB/MEOS-API` `refresh-jvm-chain.sh` over this repo (cloning MEOS-API
master the first time). All of that script's options pass through:

```bash
cd ~/src/JMEOS # JMEOS main
CATALOG="$MEOSAPI/output/meos-idl.json" LIBMEOS="$MDB/.prefix/lib/libmeos.so" \
tools/regen-from-catalog.sh
mvn install:install-file -Dfile=jar/JMEOS.jar \
-DgroupId=org.jmeos -DartifactId=meos -Dversion=1.0 -Dpackaging=jar
tools/refresh-from-master.sh --mdb ~/src/MobilityDB # refresh against a local MobilityDB branch
tools/refresh-from-master.sh --skip-tests # regenerate + compile, skip the tests
```

**3. Stage the catalog and build.** `tools/meos-idl.json` is derived, not committed:

```bash
cd ~/src/MobilitySpark
cp "$MEOSAPI/output/meos-idl.json" tools/meos-idl.json
LD_LIBRARY_PATH="$MDB/.prefix/lib" mvn clean test
```
This repo's last leg is `tools/refresh.conf`: the repo-root Maven module, the `org.jmeos:meos:1.0`
jar coordinates, and the `mvn … clean test` command. `generate-sources` runs `tools/codegen_jvm.py
--engine spark` over the staged `tools/meos-idl.json` and the installed jar, so the UDF layer is
regenerated by the build itself; `LD_LIBRARY_PATH` is how the tests find `libmeos.so`.

`generate-sources` runs `tools/codegen_jvm.py --engine spark --catalog tools/meos-idl.json
--jar <the installed jar> --out target/generated-sources/spark/...`, so the UDF layer is
regenerated by the build itself. `LD_LIBRARY_PATH` is how the tests find `libmeos.so`.
The chain's per-leg commands, which `refresh-jvm-chain.sh` composes, are in `MEOS-API/GENERATION.md`
(`provision-meos.sh`) and `JMEOS/GENERATION.md` (`regen-from-catalog.sh`).
35 changes: 35 additions & 0 deletions tools/refresh-from-master.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
# refresh-from-master.sh — refresh this consumer's MEOS facades against the latest MEOS API,
# end to end, with one command:
#
# tools/refresh-from-master.sh
#
# It runs the shared MobilityDB/MEOS-API refresh-jvm-chain.sh over this repo — deriving the
# catalog and libmeos from the latest MobilityDB master, building the JMEOS jar, and
# regenerating this consumer's facades. The per-consumer last leg is in tools/refresh.conf.
#
# All of refresh-jvm-chain.sh's options pass through, e.g.:
# tools/refresh-from-master.sh --mdb ~/src/MobilityDB # refresh against a local MobilityDB branch
# tools/refresh-from-master.sh --skip-tests # regenerate + compile, skip the test run
#
# MEOSAPI=<path> uses an existing MEOS-API checkout (any branch); otherwise MEOS-API master is
# cloned into the work dir. WORK_DIR overrides the scratch location (default <repo>/.meos-chain).
set -euo pipefail

HERE="$(cd "$(dirname "$0")/.." && pwd)"
WORK="${WORK_DIR:-$HERE/.meos-chain}"
MEOSAPI="${MEOSAPI:-}"

if [ -z "$MEOSAPI" ]; then
MEOSAPI="$WORK/MEOS-API"
mkdir -p "$WORK"
if [ -d "$MEOSAPI/.git" ]; then
git -C "$MEOSAPI" fetch --quiet https://github.com/MobilityDB/MEOS-API master
git -C "$MEOSAPI" checkout --quiet FETCH_HEAD
else
git clone --quiet https://github.com/MobilityDB/MEOS-API "$MEOSAPI"
fi
fi

exec "$MEOSAPI/tools/refresh-jvm-chain.sh" \
--consumer "$HERE" --meos-api "$MEOSAPI" --work-dir "$WORK" "$@"
8 changes: 8 additions & 0 deletions tools/refresh.conf
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# refresh.conf — MobilitySpark's last leg for tools/refresh-jvm-chain.sh (in MEOS-API).
# The chain (MobilityDB -> catalog + libmeos -> JMEOS jar) is shared; only these differ per
# consumer. BUILD_CMD runs from <repo>/<BUILD_DIR> with $PREFIX (the libmeos install prefix),
# $CATALOG and $JAR exported; $SKIP_TESTS is set when --skip-tests is passed.
ENGINE=spark
BUILD_DIR=.
JMEOS_COORDS=org.jmeos:meos:1.0
BUILD_CMD='LD_LIBRARY_PATH="$PREFIX/lib" mvn -B ${SKIP_TESTS:+-DskipTests} clean test'
Loading