Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
if this touches public Rust, Python, C ABI, Go, Manifest,
persisted-format, examples/docs, feature, or MSRV surfaces
- [ ] `CHANGELOG.md` updated under `Unreleased` if user-facing
- [ ] `cargo deny check` passes (licenses / advisories / bans / sources)
- [ ] `cargo deny --workspace --all-features --locked check` passes (licenses / advisories / bans / sources)
- [ ] If `ordvec-python/` changed: `cargo clippy -p ordvec-python --all-targets -- -D warnings`, then `maturin develop` + `pytest ordvec-python/tests` pass

## Notes
Expand Down
20 changes: 16 additions & 4 deletions .github/workflows/audit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@ name: audit
# Read-only token, no `run:` steps (no injection surface). Because this runs
# UNATTENDED on a cron schedule, the third-party actions are SHA-pinned (not the
# mutable @v4/@v2 tags ci.yml uses on human-triggered push/PR) so a compromised
# tag update cannot auto-execute on a scheduled runner. Scoped to
# `check advisories`.
# tag update cannot auto-execute on a scheduled runner. Scoped to `check
# advisories`, with separate locked roots for the main workspace and the
# standalone fuzz crate so neither committed lockfile can fall outside the
# advisory graph.

on:
schedule:
Expand All @@ -36,6 +38,16 @@ jobs:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
- name: Audit root workspace advisories
uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
with:
command: check advisories
command: check
arguments: --workspace --all-features --locked
command-arguments: advisories
- name: Audit standalone fuzz lockfile advisories
uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
with:
command: check
manifest-path: fuzz/Cargo.toml
arguments: --all-features --locked
command-arguments: advisories
13 changes: 12 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,8 @@ jobs:
run: cargo build
- name: cargo test
run: cargo test
- name: cargo test -p ordvec-manifest --all-features
run: cargo test -p ordvec-manifest --all-features

# ----------------------------------------------------------------------
# No-system-deps guarantee + packaging check. Fails if any forbidden
Expand Down Expand Up @@ -348,9 +350,18 @@ jobs:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
- name: Check root workspace policy
uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
with:
command: check
arguments: --workspace --all-features --locked
- name: Audit standalone fuzz lockfile advisories
uses: EmbarkStudios/cargo-deny-action@bb137d7af7e4fb67e5f82a49c4fce4fad40782fe # v2.0.20
with:
command: check
manifest-path: fuzz/Cargo.toml
arguments: --all-features --locked
command-arguments: advisories

# ----------------------------------------------------------------------
# AVX-512 coverage via Intel SDE. GitHub-hosted runners lack AVX-512, so we
Expand Down
124 changes: 102 additions & 22 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ jobs:
needs: guard
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
timeout-minutes: 35
permissions:
contents: read
actions: read
Expand All @@ -174,26 +175,105 @@ jobs:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
SHA: ${{ github.sha }}
POLL_INTERVAL_SECONDS: 30
POLL_TIMEOUT_SECONDS: 1800
run: |
set -euo pipefail
MAIN_SHA="$(gh api "repos/${REPO}/commits/main" --jq '.sha')"
echo "current main sha: ${MAIN_SHA}"
if [ "$SHA" != "$MAIN_SHA" ]; then
echo "::error::release tag points at ${SHA}, but current main is ${MAIN_SHA}. Move the tag to the current protected main HEAD and re-run."
exit 1
fi
# Require a SUCCESSFUL push run for this SHA *on main* for each workflow.
# Filtering on branch as well as head_sha stops a green run for the same
# commit on an unrelated branch from satisfying the gate.
for wf in ci.yml python.yml fuzz.yml codeql.yml actionlint.yml zizmor.yml; do
ok="$(gh api \
"repos/${REPO}/actions/workflows/${wf}/runs?head_sha=${SHA}&branch=main&event=push&status=success&per_page=20" \
--jq '[.workflow_runs[] | select(.head_branch == "main" and .event == "push" and .conclusion == "success")] | length')"
echo "successful push ${wf} runs for ${SHA} on main: ${ok}"
if [ "${ok}" -lt 1 ]; then
echo "::error::no successful push ${wf} run for ${SHA} on main. Push to main, let CI pass, then re-tag."
deadline=$((SECONDS + POLL_TIMEOUT_SECONDS))
while :; do
MAIN_SHA="$(gh api "repos/${REPO}/commits/main" --jq '.sha')"
echo "current main sha: ${MAIN_SHA}"
if [ "$SHA" != "$MAIN_SHA" ]; then
echo "::error::release tag points at ${SHA}, but current main is ${MAIN_SHA}. Move the tag to the current protected main HEAD and re-run."
exit 1
fi

all_green=true
for wf in ci.yml python.yml fuzz.yml codeql.yml actionlint.yml zizmor.yml; do
# Query every status, then inspect the latest exact-SHA push run.
# A success-only filter cannot distinguish normal run-visibility
# delay from terminal failure and was the source of issue #272.
api_error="$(mktemp)"
if runs_json="$(gh api \
"repos/${REPO}/actions/workflows/${wf}/runs?head_sha=${SHA}&branch=main&event=push&per_page=100" \
2>"$api_error")"; then
:
else
api_exit_status=$?
api_message="$(<"$api_error")"
api_status="$(sed -nE 's/.*HTTP ([0-9]{3}).*/\1/p' "$api_error" | tail -n 1)"
retryable_api_error=false
case "$api_status" in
408|429|5??)
retryable_api_error=true
;;
"")
if [ "$api_exit_status" -ne 4 ] && grep -Eiq \
'timeout|timed out|connection (reset|refused|closed)|could not resolve|temporary failure in name resolution|network is unreachable|TLS handshake|unexpected EOF|error connecting to' \
"$api_error"; then
retryable_api_error=true
fi
;;
403)
if grep -Eiq 'rate.?limit|secondary rate|abuse detection' "$api_error"; then
retryable_api_error=true
fi
;;
esac
rm -f "$api_error"
if "$retryable_api_error"; then
echo "waiting for ${wf}: transient API query failure${api_status:+ (HTTP ${api_status})}: ${api_message}"
all_green=false
continue
fi
echo "::error::${wf}: API query failed permanently${api_status:+ (HTTP ${api_status})}: ${api_message}"
exit 1
fi
rm -f "$api_error"

latest="$(jq -r --arg sha "$SHA" '
[.workflow_runs[]
| select(.head_sha == $sha and .head_branch == "main" and .event == "push")]
| sort_by([.run_number, .run_attempt, .id])
| last
| if . == null then ""
else [.status, (.conclusion // ""), (.html_url // "")] | join("|")
end
' <<<"$runs_json")"
if [ -z "$latest" ]; then
echo "waiting for ${wf}: no exact-SHA main push run is visible yet"
all_green=false
continue
fi

IFS='|' read -r status conclusion run_url <<<"$latest"
if [ "$status" != "completed" ]; then
echo "waiting for ${wf}: latest run is ${status} (${run_url})"
all_green=false
elif [ "$conclusion" = "success" ]; then
echo "green ${wf}: ${run_url}"
else
echo "::error::${wf} completed with ${conclusion:-no conclusion} for ${SHA}: ${run_url}"
exit 1
fi
done

if "$all_green"; then
FINAL_MAIN_SHA="$(gh api "repos/${REPO}/commits/main" --jq '.sha')"
echo "final current main sha: ${FINAL_MAIN_SHA}"
if [ "$SHA" != "$FINAL_MAIN_SHA" ]; then
echo "::error::main advanced from release tag ${SHA} to ${FINAL_MAIN_SHA} while CI was being checked. Tag only the new protected main HEAD after its own CI is green."
exit 1
fi
echo "all release-gated workflows are green for ${SHA} on main"
break
fi
if (( SECONDS >= deadline )); then
echo "::error::timed out after ${POLL_TIMEOUT_SECONDS}s waiting for exact-SHA main push CI. No artifacts or draft release were created."
exit 1
fi
echo "CI is not settled; polling again in ${POLL_INTERVAL_SECONDS}s"
sleep "$POLL_INTERVAL_SECONDS"
done

release-avx512:
Expand Down Expand Up @@ -264,7 +344,7 @@ jobs:

notes:
name: release notes (git-cliff) + draft Release
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
permissions:
Expand Down Expand Up @@ -302,7 +382,7 @@ jobs:

build-crate:
name: build .crate + SBOM
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -393,7 +473,7 @@ jobs:

build-wheels:
name: wheel ${{ matrix.platform.target }} (${{ matrix.platform.runner }})
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ${{ matrix.platform.runner }}
strategy:
Expand Down Expand Up @@ -536,7 +616,7 @@ jobs:

build-sdist:
name: build sdist + SBOM
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
steps:
Expand Down Expand Up @@ -609,7 +689,7 @@ jobs:

build-manifest-wheels:
name: manifest wheel ${{ matrix.platform.target }} (${{ matrix.platform.runner }})
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ${{ matrix.platform.runner }}
strategy:
Expand Down Expand Up @@ -674,7 +754,7 @@ jobs:

build-manifest-sdist:
name: build manifest sdist + SBOM
needs: guard
needs: [guard, require-ci-green]
if: needs.guard.outputs.ok == 'true'
runs-on: ubuntu-latest
steps:
Expand Down
92 changes: 63 additions & 29 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

_No unreleased changes._

## 0.7.0 - 2026-07-13
## 0.6.0 - 2026-07-19

### Added

Expand All @@ -33,34 +33,6 @@ _No unreleased changes._
read core; the sqlite registry's private duplicate hasher is deduped
onto the public helper.

### Changed

- **BREAKING (`ordvec-manifest`): deterministic manifest schema v2.** The
manifest schema version is now `ordvec.index_manifest.v2`. `manifest_id`
and `created_at` are removed from `IndexManifest`, creation omits the
optional `build` field (serialized as absent, not `null`), and auxiliary
artifact entries are sorted by
`(name, path)`, so identical bundle content serializes to byte-identical
manifests and `sha256(manifest.json)` is the bundle's content address.
Existing v1 manifests no longer parse; loading one fails with an error
naming both schema versions (zero back-compat, pre-release). Embedded
paths must now be canonical — bundle-relative, forward slashes, no `.`,
`..`, or empty segments — enforced both at creation (non-embeddable
inputs fail `create` instead of minting a manifest that fails its own
verification) and at verification (`*_path_not_canonical` codes, now
also covering calibration and encoder-distortion profile refs). Absolute
paths and escaping `..` paths remain available behind the existing
`allow_absolute_paths` / `allow_path_escape` opt-ins. The
`write_manifest_file` serialization form is documented as the single
canonical byte form: hashing and signing operate on stored bytes, and any
serializer change is a schema-version event. The sqlite report registry
drops its `manifest_id` column: the cached `verification_reports` table is
migrated in place on open (rows preserved, under one atomic transaction),
while the rebuildable `active_manifest` pointer is reset and must be
re-activated.

## 0.6.0 - 2026-07-04

### Performance

- **Batched sign candidate generation now streams the corpus once per call.**
Expand Down Expand Up @@ -117,6 +89,37 @@ _No unreleased changes._

### Changed

- **First-run and release surfaces are aligned for v0.6.0.** The root and PyPI
READMEs now lead with a tiny deterministic retrieval that prints a checked
result, show persistence/reopen, and route users to the Rust, Python, CLI,
and manifest packages. Both manifest packages include an end-to-end
create-and-verify path, and every `ordvec-manifest` command and option now
has CLI help text. Release invariants pin the current package version,
manifest schema, feature defaults, public metadata links, and first-run
examples so the next release cannot silently drift across surfaces.
- **BREAKING (`ordvec-manifest`): deterministic manifest schema v2.** The
manifest schema version is now `ordvec.index_manifest.v2`. `manifest_id`
and `created_at` are removed from `IndexManifest`, creation omits the
optional `build` field (serialized as absent, not `null`), and auxiliary
artifact entries are sorted by
`(name, path)`, so identical bundle content serializes to byte-identical
manifests and `sha256(manifest.json)` is the bundle's content address.
This is an intentional pre-1.0 minor-version break: v1 manifests from
`0.5.x` do not parse, and loading one fails with an error naming both schema
versions. Embedded paths must now be canonical — bundle-relative, forward
slashes, no `.`, `..`, or empty segments — enforced both at creation
(non-embeddable inputs fail `create` instead of minting a manifest that
fails its own verification) and at verification (`*_path_not_canonical`
codes, now also covering calibration and encoder-distortion profile refs).
Absolute paths and escaping `..` paths remain available behind the existing
`allow_absolute_paths` / `allow_path_escape` opt-ins. The
`write_manifest_file` serialization form is documented as the single
canonical byte form: hashing and signing operate on stored bytes, and any
serializer change is a schema-version event. The sqlite report registry
drops its `manifest_id` column: the cached `verification_reports` table is
migrated in place on open (rows preserved, under one atomic transaction),
while the rebuildable `active_manifest` pointer is reset and must be
re-activated.
- **ordvec-manifest: derived artifact size bounds.** Verification now bounds
every artifact read by its manifest-declared `file_size_bytes` (the manifest
itself remains hard-capped at 1 MiB and SHA-256 pins content); manifest
Expand All @@ -141,6 +144,37 @@ _No unreleased changes._
`sha256_file_bounded` no longer materialises the file in memory before
hashing.

### Fixed

- **`ordvec-manifest`: canonical content addresses are independent of
`serde_json` feature unification.** Nested extension and attestation maps are
recursively key-sorted and JSON numbers are normalized before serialization,
so downstream `preserve_order` / `arbitrary_precision` features cannot change
the same logical manifest's bytes. Manifest creation now rejects non-UTF-8
artifact paths instead of embedding the lossy replacement character.
- **`ordvec-manifest`: manifest writes are atomic and fail before replacement.**
`write_manifest_file` writes and syncs a same-directory temporary file before
atomically replacing the destination, and rejects non-finite distortion
values before touching an existing manifest. New-file mode/umask behavior and
existing portable permission bits are preserved.
- **`ordvec-manifest`: restore the declared Rust 1.89 floor for SQLite builds.**
The optional `rusqlite` dependency returns to the compatible 0.39 line after
0.40's build dependency used a library feature unavailable on Rust 1.89; the
all-features manifest suite now runs in the permanent MSRV lane.

### Security

- The release CI gate now waits, with a 30-minute fail-closed deadline, for
the latest exact-HEAD push runs instead of treating normal Actions
visibility/in-progress delay as an immediate failure. It rejects terminal
non-successes and a moving `main`, and every draft-release or artifact-build
job is directly blocked behind the gate.
- The advisory policy now roots `cargo-deny` at every workspace member and
enables every feature, closing the blind spot that omitted dev-only
benchmark dependencies. `anyhow` is updated from 1.0.102 to 1.0.103 to
clear RUSTSEC-2026-0190; the documented dev-only `bincode` 1.x
unmaintained advisory remains explicitly triaged.

## 0.5.0 - 2026-06-19

### Security
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ cargo test --no-default-features
cargo +1.89.0 build # MSRV
cargo build --locked
RUSTFLAGS="-D warnings" cargo build
cargo deny check # licenses / advisories / bans / sources
cargo deny --workspace --all-features --locked check # full locked workspace graph
```

SIMD dispatches at runtime — AVX-512 / AVX2 on x86_64, NEON on aarch64,
Expand Down
Loading
Loading