From 3f4473edaa4b61fff2747b5ab70531b9ee8cdc90 Mon Sep 17 00:00:00 2001 From: Kristofer Karlsson Date: Fri, 11 Sep 2026 14:18:47 +0200 Subject: [PATCH] connected: add incremental connectivity check via rev-list The connectivity check uses rev-list to find commits reachable from the incoming tips but not from any local ref and then walks their object closure. Commit traversal stops at the connectivity boundary, but trees and blobs reachable from that boundary still need to be walked so they can be marked uninteresting, allocating a struct object for each one. On repositories where the boundary commits have large trees, the connectivity check for small incoming changes visits and tracks more objects than needed. Add an alternative connectivity check that verifies incoming commits incrementally against their parents. Instead of traversing the full boundary closure, the new check compares each new commit's tree with its parent trees. Already trusted entries are skipped, changed subtrees are descended into recursively, and blobs are checked for existence. This approach thus avoids descending into untouched subtrees. For example, consider a commit that changes one file under lib/ and also moves an unchanged subtree from src/ to dev/: Parent tree New tree +-- src/ (aaa) +-- dev/ (aaa) +-- lib/ (bbb) +-- lib/ (ccc) +-- foo.c (ddd) +-- foo.c (ddd) +-- bar.c (eee) +-- bar.c (fff) The verifier first scans the new root and collects aaa and ccc as work items. It then scans the parent root, publishing aaa and bbb into the trusted sets and recording bbb as the comparison base for ccc. When the work list is revisited, aaa is now trusted and skipped even though it appears at a different path. The verifier descends into ccc using bbb as its parent base. Scanning bbb similarly makes ddd and eee trusted, leaving only the new fff blob to be checked for existence. Thus neither the moved subtree nor any other unchanged subtree is recursively explored; only the changed lib/ subtree is descended into, and only the new bar.c blob needs an existence check. The root trees still need to be read and scanned as comparison bases. New commits are processed with ancestors before descendants. Parents outside the incoming commit set are on the already-connected side of the boundary and provide the initial trusted bases. Once an incoming commit has been verified, its tree can in turn be used as a trusted base for descendant commits. The verifier distinguishes trusted trees from expanded trees. A trusted tree can be accepted without further verification. An expanded tree has additionally published its direct non-gitlink entries into the trusted sets. Expanded parent trees therefore need not be read again for blob-only work, but may still be reread when recursive verification needs same-path parent subtrees. The implementation adds a --verify-trees-incremental flag to rev-list, following the same pattern as --exclude-promisor-objects: a pre-setup_revisions() scan sets the flag, the post-setup_revisions() option loop skips it, and the main traversal short-circuits into the incremental verifier after collecting commits from the revision walk. Because the incremental block eagerly consumes all commits via get_revision(), the subsequent mark_edges_uninteresting() and traverse_commit_list_filtered() calls naturally find no commit work to do and only process any non-commit tips (trees, blobs) left in revs.pending. For partial clones, missing promisor objects (trees and blobs promised by a promisor remote) are silently accepted during verification instead of triggering errors. The revision walk receives --exclude-promisor-objects so promisor commits do not enter the verification set. Gate the new algorithm behind transfer.connectivityCheck=incremental, keeping full as the default. Fall back to the full algorithm for deepening fetches to keep this commit easy to reason about, though incremental mode could potentially handle them as well. p5412 results (median of 3), scaling one dimension at a time. Each modified file is in a different directory, with directories chosen round-robin. Scaling tree size (10 commits, 10 files/commit): files full incr. full/incr 5K 0.01s 0.01s 1.1x 50K 0.04s 0.01s 2.2x 200K 0.14s 0.02s 5.7x 800K 0.60s 0.04s 14.8x The full mode must traverse the boundary tree closure, which grows with overall tree size. Incremental still scans the root trees, but avoids descending into unchanged subtrees, so it grows much more slowly with repository size. Scaling commit count (200K files, 10 files/commit): commits full incr. full/incr 1 0.14s 0.01s 7.5x 10 0.15s 0.02s 6.8x 100 0.16s 0.07s 2.2x 500 0.29s 0.29s 0.9x 3000 1.26s 1.67s 0.7x 5000 1.95s 2.71s 0.7x 10000 3.74s 6.02s 0.6x With many commits the per-commit overhead of scanning both the new and parent root trees accumulates and incremental becomes slower. Breakeven is around 500 commits and the ratio stabilizes near 0.6x for this fixture. Scaling files per commit (200K files, 10 commits): files/commit full incr. full/incr 1 0.15s 0.02s 7.5x 10 0.15s 0.02s 6.8x 100 0.15s 0.04s 3.1x 500 0.26s 0.16s 1.6x 1000 0.36s 0.35s 1.0x 2000 0.66s 0.79s 0.8x Breakeven is around 1000 files/commit. At 2000 files/commit (every directory touched), incremental is about 1.2x slower. For small repositories both modes are fast enough that the difference is difficult to measure reliably. Signed-off-by: Kristofer Karlsson --- Documentation/config/transfer.adoc | 20 + Documentation/rev-list-options.adoc | 6 + Makefile | 1 + builtin/rev-list.c | 18 + connected.c | 24 + meson.build | 1 + t/meson.build | 1 + ...enerate-repo-p5412-connectivity-check.perl | 44 ++ t/perf/p5412-connectivity-check.sh | 87 +++ t/t5412-connectivity-check.sh | 598 ++++++++++++++++++ tree-verify.c | 242 +++++++ tree-verify.h | 15 + 12 files changed, 1057 insertions(+) create mode 100644 t/perf/generate-repo-p5412-connectivity-check.perl create mode 100755 t/perf/p5412-connectivity-check.sh create mode 100755 t/t5412-connectivity-check.sh create mode 100644 tree-verify.c create mode 100644 tree-verify.h diff --git a/Documentation/config/transfer.adoc b/Documentation/config/transfer.adoc index f1ce50f4a6e6ba..b9939d3bde841c 100644 --- a/Documentation/config/transfer.adoc +++ b/Documentation/config/transfer.adoc @@ -1,3 +1,23 @@ +transfer.connectivityCheck:: + Choose which algorithm to use for the connectivity check + performed during object transfer operations such as + linkgit:git-fetch[1] and linkgit:git-receive-pack[1]. + The connectivity check verifies that all objects reachable + from the incoming tips are available locally or, in a partial + clone, promised by a promisor remote. + The variants are as follows: ++ +-- +`full` (default);; + Walk the full object closure of the boundary commits. +`incremental`;; + Verify incoming commits by diffing their trees against parent + trees, recursively descending only into entries that differ. + The largest benefits occur when incoming commits change a + small fraction of a large tree closure. + Falls back to `full` for deepening fetches. +-- + transfer.credentialsInUrl:: A configured URL can contain plaintext credentials in the form `://:@/`. You may want diff --git a/Documentation/rev-list-options.adoc b/Documentation/rev-list-options.adoc index fd831f0ec64744..7964b691c4af79 100644 --- a/Documentation/rev-list-options.adoc +++ b/Documentation/rev-list-options.adoc @@ -1089,6 +1089,12 @@ we cannot get their Object ID though, an error will be raised. stronger than `--missing=allow-promisor` because it limits the traversal, rather than just silencing errors about missing objects. + +`--verify-trees-incremental`:: + (For internal use only.) Verify tree connectivity + incrementally by comparing each commit's tree against its + parent trees. Used by `check_connected()` when + `transfer.connectivityCheck` is set to `incremental`. endif::git-rev-list[] `--no-walk[=(sorted|unsorted)]`:: diff --git a/Makefile b/Makefile index d4b775953d3842..aa5b4e2b849fb6 100644 --- a/Makefile +++ b/Makefile @@ -1357,6 +1357,7 @@ LIB_OBJS += trailer.o LIB_OBJS += transport-helper.o LIB_OBJS += transport.o LIB_OBJS += tree-diff.o +LIB_OBJS += tree-verify.o LIB_OBJS += tree-walk.o LIB_OBJS += tree.o LIB_OBJS += unpack-trees.o diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 6b596231ab1ab0..7741fdec9f9b58 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -28,6 +28,7 @@ #include "commit-reach.h" #include "quote.h" #include "strbuf.h" +#include "tree-verify.h" struct rev_list_info { struct rev_info *revs; @@ -706,6 +707,7 @@ int cmd_rev_list(int argc, int bisect_find_all = 0; int use_bitmap_index = 0; int filter_provided_objects = 0; + int verify_trees_incremental = 0; const char *show_progress = NULL; int ret = 0; @@ -748,6 +750,8 @@ int cmd_rev_list(int argc, if (!strcmp(arg, "--exclude-promisor-objects")) { repo->fetch_if_missing = 0; revs.exclude_promisor_objects = 1; + } else if (!strcmp(arg, "--verify-trees-incremental")) { + verify_trees_incremental = 1; } else if (skip_prefix(arg, "--missing=", &arg)) { parse_missing_action_value(repo, arg); } else if (!strcmp(arg, "-z")) { @@ -822,6 +826,8 @@ int cmd_rev_list(int argc, if (!strcmp(arg, "--exclude-promisor-objects")) continue; /* already handled above */ + if (!strcmp(arg, "--verify-trees-incremental")) + continue; /* already handled above */ if (skip_prefix(arg, "--missing=", &arg)) continue; /* already handled above */ @@ -935,6 +941,18 @@ int cmd_rev_list(int argc, prepare_maximal_independent(&revs); + if (verify_trees_incremental) { + struct commit *commit; + struct commit_list *new_commits = NULL; + + while ((commit = get_revision(&revs)) != NULL) + commit_list_insert(commit, &new_commits); + + verify_commits_incremental(repo, &new_commits, + revs.exclude_promisor_objects); + commit_list_free(new_commits); + } + if (revs.tree_objects) mark_edges_uninteresting(&revs, show_edge, 0); diff --git a/connected.c b/connected.c index 929b9bd28d6fab..e3dcd8e2b0ff58 100644 --- a/connected.c +++ b/connected.c @@ -1,6 +1,7 @@ #define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" +#include "config.h" #include "gettext.h" #include "hex.h" #include "odb.h" @@ -67,6 +68,26 @@ static int check_connected_promisor(oid_iterate_fn fn, return 1; } +static int incremental_check_applicable(struct check_connected_options *opt) +{ + const char *algorithm = NULL; + + if (repo_config_get_string_tmp(the_repository, + "transfer.connectivitycheck", + &algorithm)) + return 0; + if (!strcasecmp(algorithm, "full")) + return 0; + if (strcasecmp(algorithm, "incremental")) + die(_("unknown transfer.connectivityCheck algorithm '%s'"), + algorithm); + + if (opt->is_deepening_fetch) + return 0; + + return 1; +} + /* * If we feed all the commits we want to verify to this command * @@ -133,6 +154,9 @@ int check_connected(oid_iterate_fn fn, void *cb_data, if (opt->progress) strvec_pushf(&rev_list.args, "--progress=%s", _("Checking connectivity")); + if (incremental_check_applicable(opt)) + strvec_push(&rev_list.args, + "--verify-trees-incremental"); rev_list.git_cmd = 1; if (opt->env) diff --git a/meson.build b/meson.build index d86f2acd2b2a46..ab24bb4b91250d 100644 --- a/meson.build +++ b/meson.build @@ -562,6 +562,7 @@ libgit_sources = [ 'transport-helper.c', 'transport.c', 'tree-diff.c', + 'tree-verify.c', 'tree-walk.c', 'tree.c', 'unpack-trees.c', diff --git a/t/meson.build b/t/meson.build index 7f53cca7d1f891..19a8246c44fc49 100644 --- a/t/meson.build +++ b/t/meson.build @@ -652,6 +652,7 @@ integration_tests = [ 't5409-colorize-remote-messages.sh', 't5410-receive-pack.sh', 't5411-proc-receive-hook.sh', + 't5412-connectivity-check.sh', 't5500-fetch-pack.sh', 't5501-fetch-push-alternates.sh', 't5502-quickfetch.sh', diff --git a/t/perf/generate-repo-p5412-connectivity-check.perl b/t/perf/generate-repo-p5412-connectivity-check.perl new file mode 100644 index 00000000000000..95469962804db9 --- /dev/null +++ b/t/perf/generate-repo-p5412-connectivity-check.perl @@ -0,0 +1,44 @@ +#!/usr/bin/perl +# +# Generate a fast-import stream for p5412 connectivity check benchmarks. +# +# Usage: generate-repo-p5412-connectivity-check.perl +# [] [] +# +# Creates one initial commit with dirs*files_per_dir files, then +# additional commits each modifying +# files in directories chosen round-robin from 1... + +use strict; +use warnings; + +my ($nd, $nf, $nc, $hot, $fpc) = @ARGV; +$hot = $nd if !$hot || $hot > $nd; +$fpc = 1 if !$fpc; + +sub data { + printf "data %d\n%s\n", length($_[0]), $_[0]; +} + +# Initial tree: one commit with nd*nf files. +printf "commit refs/heads/main\n"; +printf "committer perf now\n"; +data("initial"); +for my $d (1..$nd) { + for my $f (1..$nf) { + printf "M 100644 inline d-%04d/f-%03d\n", $d, $f; + data(sprintf "%03d%03d", $d, $f); + } +} + +# Subsequent commits (auto-chained by fast-import). +for my $i (1..$nc) { + printf "commit refs/heads/main\n"; + printf "committer perf now\n"; + data(sprintf "change-%03d", $i); + for my $j (0..$fpc-1) { + my $d = (($i + $j) % $hot) + 1; + printf "M 100644 inline d-%04d/f-001\n", $d; + data(sprintf "c%d-%d", $i, $j); + } +} diff --git a/t/perf/p5412-connectivity-check.sh b/t/perf/p5412-connectivity-check.sh new file mode 100755 index 00000000000000..d16b466dcb39a0 --- /dev/null +++ b/t/perf/p5412-connectivity-check.sh @@ -0,0 +1,87 @@ +#!/bin/sh + +test_description='performance of connectivity check modes + +Compare the default and incremental rev-list connectivity modes +directly, avoiding pack transfer noise. + +Each repository has a flat tree of many directories with 100 files +in each. Three axes are scaled independently: tree size, commit +count, and files changed per commit.' + +. ./perf-lib.sh + +test_perf_fresh_repo + +generate="$TEST_DIRECTORY/perf/generate-repo-p5412-connectivity-check.perl" + +# $1=dirs $2=files_per_dir $3=commits $4=hot_dirs (optional, default=all) +# $5=files_per_commit (optional, default=1) +test_perf_conn () { + local nd="$1" nf="$2" nc="$3" hot="${4:-$1}" fpc="${5:-1}" + local total=$(($nd * $nf)) + local name="repo-${nd}d-${nf}f-${nc}c-${hot}h-${fpc}fpc" + local label="${total} files, ${nc} commits" + if test "$hot" -lt "$nd" + then + label="$label (${hot} hot dirs)" + fi + if test "$fpc" -gt 1 + then + label="$label (${fpc} files/commit)" + fi + + test_expect_success "setup $label" ' + git init '"$name"' && + "$PERL_PATH" '"$generate"' '"$nd"' '"$nf"' '"$nc"' '"$hot"' '"$fpc"' | + git -C '"$name"' fast-import --date-format=now --quiet && + ( + cd '"$name"' && + git rev-parse main~'"$nc"' >../'"${name}"'_old && + git rev-parse main >../'"${name}"'_new && + git update-ref refs/heads/main \ + $(cat ../'"${name}"'_old) && + git repack -ad && + git config gc.auto 0 + ) + ' + + test_perf "$label (full)" ' + cat '"${name}"'_new | + git -C '"$name"' rev-list \ + --objects --stdin --not --all --quiet \ + --exclude-promisor-objects + ' + + test_perf "$label (incremental)" ' + cat '"${name}"'_new | + git -C '"$name"' rev-list --verify-trees-incremental \ + --objects --stdin --not --all --quiet \ + --exclude-promisor-objects + ' +} + +# Scaling tree size (10 commits, 10 files/commit). +test_perf_conn 50 100 10 50 10 +test_perf_conn 500 100 10 500 10 +test_perf_conn 2000 100 10 2000 10 +test_perf_conn 8000 100 10 8000 10 + +# Scaling commit count (200K files, 10 files/commit). +test_perf_conn 2000 100 1 2000 10 +test_perf_conn 2000 100 10 2000 10 +test_perf_conn 2000 100 100 2000 10 +test_perf_conn 2000 100 500 2000 10 +test_perf_conn 2000 100 3000 2000 10 +test_perf_conn 2000 100 5000 2000 10 +test_perf_conn 2000 100 10000 2000 10 + +# Scaling files per commit (200K files, 10 commits). +test_perf_conn 2000 100 10 2000 1 +test_perf_conn 2000 100 10 2000 10 +test_perf_conn 2000 100 10 2000 100 +test_perf_conn 2000 100 10 2000 500 +test_perf_conn 2000 100 10 2000 1000 +test_perf_conn 2000 100 10 2000 2000 + +test_done diff --git a/t/t5412-connectivity-check.sh b/t/t5412-connectivity-check.sh new file mode 100755 index 00000000000000..6c27b0845092e3 --- /dev/null +++ b/t/t5412-connectivity-check.sh @@ -0,0 +1,598 @@ +#!/bin/sh + +test_description='connectivity check (transfer.connectivityCheck)' +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +test_oid_cache <<-\EOF +missing sha1:0000000000000000000000000000000000000001 +missing sha256:0000000000000000000000000000000000000000000000000000000000000001 +EOF + +set_connectivity_check () { + git -C "$1" config transfer.connectivityCheck "$2" +} + +# Run a connectivity check via rev-list directly. Uses $mode +# (set by the enclosing for-loop) to choose full or incremental. +check_connected () { + flags= && + if test "$mode" = incremental + then + flags=--verify-trees-incremental + fi && + printf '%s\n' "$@" | + git rev-list $flags \ + --objects --stdin --not --all --quiet \ + --exclude-promisor-objects +} + +# Run git with a temporary index, leaving the real index untouched. +tmpgit () { + GIT_INDEX_FILE=.git/tmp-idx git "$@" +} + +# Create a commit with one file changed, without modifying HEAD, +# index, or worktree. Prints the new commit OID on stdout. +# Usage: commit_with_change +commit_with_change () { + new_blob=$(echo "$3" | git hash-object -w --stdin) && + tmpgit read-tree "$1" && + tmpgit update-index --replace \ + --cacheinfo "100644,$new_blob,$2" && + new_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + git commit-tree "$new_tree" -p "$1" -m "modify $2" +} + +# Check OIDs and optionally verify trace2 counts. +# Usage: check_connected_trace ... +# An empty string for or skips that assertion. +check_connected_trace () { + trace_file=$1 trees=$2 blobs=$3 && + shift 3 && + test_env GIT_TRACE2_EVENT="$(pwd)/$trace_file" \ + check_connected "$@" && + if test "$mode" != incremental + then + return + fi && + if test -n "$trees" + then + test_trace2_data_singular connectivity trees_loaded "$trees" \ + <"$trace_file" + fi && + if test -n "$blobs" + then + test_trace2_data_singular connectivity blobs_checked "$blobs" \ + <"$trace_file" + fi +} + +# Shared setup: a repo with several root-level files and nested dirs. +# The unchanged/ subtree (10 dirs x 10 files = 100 blobs, 11 trees) +# acts as a canary: any test asserting small tree/blob counts would +# fail dramatically if incremental accidentally walked into it. + +test_expect_success 'setup main repo' ' + git init main-repo && + ( + cd main-repo && + for i in $(test_seq 1 5) + do + echo "file $i" >"file-$i.txt" || return 1 + done && + git add file-*.txt && + git commit -m "initial" && + + mkdir -p a/b/c && + echo deep >a/b/c/deep.txt && + echo other >a/other.txt && + git add a/b/c/deep.txt a/other.txt && + git commit -m "add nested dirs" && + + for i in $(test_seq 1 10) + do + d="unchanged/dir-$i" && + mkdir -p "$d" && + for j in $(test_seq 1 10) + do + echo "$i $j" >"$d/file-$j.txt" || return 1 + done + done && + git add unchanged/ && + git commit -m "add unchanged canary subtree" + ) +' + +test_expect_success 'setup replacement object repo' ' + git init replace-test && + ( + cd replace-test && + + test_commit --no-tag original file.txt && + original=$(git rev-parse HEAD) && + orig_blob=$(git rev-parse HEAD:file.txt) && + + # Orphan replacement commit with a different tree + replacement_tree=$(echo replaced | git hash-object -w --stdin | + xargs -I{} git mktree <<-EOF + 100644 blob {} file.txt + EOF + ) && + replacement=$(git commit-tree -m "replacement" \ + "$replacement_tree") && + + git replace "$original" "$replacement" && + + # Remove the original blob so only the replacement + # tree is complete. + rm .git/objects/$(test_oid_to_path "$orig_blob") && + + # Drop branch and HEAD so --not --all does not + # exclude the original commit. + git update-ref -d refs/heads/main && + git update-ref -d HEAD && + + echo "$original" >.git/test-oid + ) +' + +missing_oid=$(test_oid missing) +original_oid=$(cat replace-test/.git/test-oid) + +for mode in full incremental +do + +# Corruption detection: craft broken object graphs and verify detection. +# All tests use main-repo without modifying its refs or worktree. + +test_expect_success "$mode: rejects commit with missing blob" ' + ( + cd main-repo && + bad_tree=$(printf "100644 blob ${missing_oid}\tfile.txt\n" | + git mktree --missing) && + bad_commit=$(git commit-tree "$bad_tree" -p HEAD -m "bad") && + test_expect_code 128 check_connected "$bad_commit" 2>err && + test_grep "missing blob object" err + ) +' + +test_expect_success "$mode: rejects commit with missing subtree" ' + ( + cd main-repo && + bad_tree=$(printf "40000 tree ${missing_oid}\tdir\n" | + git mktree --missing) && + bad_commit=$(git commit-tree "$bad_tree" -p HEAD -m "bad") && + test_expect_code 128 check_connected "$bad_commit" 2>err && + test_grep "bad tree object" err + ) +' + +test_expect_success "$mode: verifies direct tree tip" ' + ( + cd main-repo && + bad_tree=$(printf "100644 blob ${missing_oid}\tfile.txt\n" | + git mktree --missing) && + test_expect_code 128 check_connected "$bad_tree" 2>err && + test_grep "missing blob object" err + ) +' + +test_expect_success "$mode: verifies direct blob tip" ' + ( + cd main-repo && + blob_oid=$(echo "hello" | git hash-object -w --stdin) && + check_connected "$blob_oid" + ) +' + +test_expect_success "$mode: rejects missing direct blob tip" ' + ( + cd main-repo && + test_expect_code 128 check_connected \ + "$missing_oid" 2>err + ) +' + +test_expect_success PERL_TEST_HELPERS \ + "$mode: rejects blob OID reused as tree entry" ' + ( + cd main-repo && + + blob_oid=$(git rev-parse HEAD:file-1.txt) && + bin_oid=$(echo "$blob_oid" | hex2oct) && + + bad_tree=$(printf "40000 subdir\0$bin_oid" | + git hash-object -t tree -w --stdin) && + bad_commit=$(git commit-tree -p HEAD -m "child" "$bad_tree") && + + test_expect_code 128 check_connected "$bad_commit" 2>err && + test_grep "not a tree" err + ) +' + +test_expect_success PERL_TEST_HELPERS \ + "$mode: rejects tree OID reused as blob entry" ' + ( + cd main-repo && + + tree_oid=$(git rev-parse HEAD:a) && + bin_oid=$(echo "$tree_oid" | hex2oct) && + + bad_tree=$(printf "100644 fakefile\0$bin_oid" | + git hash-object -t tree -w --stdin) && + bad_commit=$(git commit-tree -p HEAD -m "child" "$bad_tree") && + + test_expect_code 128 check_connected "$bad_commit" 2>err && + test_grep "not a blob" err + ) +' + +test_expect_success "$mode: checks multiple tips" ' + ( + cd main-repo && + c1=$(commit_with_change HEAD file-1.txt "tip-a") && + c2=$(commit_with_change HEAD file-2.txt "tip-b") && + git tag -a -m "tagged" multi-tag "$c1" && + tag_oid=$(git rev-parse multi-tag) && + git tag -d multi-tag && + check_connected "$c2" "$tag_oid" + ) +' + +# Tree-diff optimization: verify trace2 counts. + +test_expect_success "$mode: handles single file change" ' + ( + cd main-repo && + oid=$(commit_with_change HEAD file-1.txt "changed") && + + # 2 trees loaded (new root + parent root), 1 blob checked. + check_connected_trace trace-flat.txt 2 1 "$oid" + ) +' + +test_expect_success "$mode: handles nested change" ' + ( + cd main-repo && + oid=$(commit_with_change HEAD a/b/c/deep.txt "deep-changed") && + # 4 new trees + 4 parent trees = 8 loaded, 1 blob checked. + check_connected_trace trace-nested.txt 8 1 "$oid" + ) +' + +test_expect_success "$mode: handles change-then-revert" ' + ( + cd main-repo && + c1=$(commit_with_change HEAD file-1.txt "revert-tmp") && + c2=$(commit_with_change "$c1" file-1.txt "file 1") && + c3=$(commit_with_change "$c2" file-1.txt "revert-final") && + + # c1: new root + parent root = 2 loads. c2: root matches + # HEAD (already trusted), skipped. c3: new root + parent + # already expanded = 1 load. Total: 3 trees, 2 blobs. + check_connected_trace trace-revert.txt 3 2 "$c3" + ) +' + +test_expect_success "$mode: handles subtree moved to another path" ' + ( + cd main-repo && + moved_tree=$(git ls-tree HEAD | + sed "s/ a$/ moved/" | + git mktree) && + moved=$(git commit-tree "$moved_tree" -p HEAD -m move) && + # New root + parent root scanned, but the moved subtree + # (same OID) is trusted and not descended into. + check_connected_trace trace-move.txt 2 0 "$moved" + ) +' + +test_expect_success "$mode: handles multi-parent merge" ' + ( + cd main-repo && + left=$(commit_with_change HEAD file-1.txt left) && + right=$(commit_with_change HEAD file-2.txt right) && + git update-ref refs/heads/left "$left" && + git update-ref refs/heads/right "$right" && + left_blob=$(git rev-parse "$left:file-1.txt") && + right_blob=$(git rev-parse "$right:file-2.txt") && + tmpgit read-tree HEAD && + tmpgit update-index --replace \ + --cacheinfo "100644,$left_blob,file-1.txt" && + tmpgit update-index --replace \ + --cacheinfo "100644,$right_blob,file-2.txt" && + merge_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + merge=$(git commit-tree "$merge_tree" \ + -p "$left" -p "$right" -m merge) && + # Both parent root trees are scanned as bases, so + # blobs from each parent are trusted without ODB checks. + check_connected_trace trace-merge.txt 3 0 "$merge" && + git update-ref -d refs/heads/left && + git update-ref -d refs/heads/right + ) +' + +test_expect_success "$mode: handles gitlink entries (submodules)" ' + ( + cd main-repo && + tmpgit read-tree HEAD && + tmpgit update-index --add \ + --cacheinfo "160000,$missing_oid,my-submodule" && + gitlink_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + gitlink_commit=$(git commit-tree "$gitlink_tree" -p HEAD \ + -m "add gitlink") && + + # Gitlink entries are skipped -- the missing submodule + # commit OID does not cause a failure. + check_connected_trace trace-gitlink.txt 2 0 "$gitlink_commit" + ) +' + +test_expect_success "$mode: handles file-to-directory transition" ' + ( + cd main-repo && + + # Parent: "foo" is a blob at root. + blob_a=$(echo "file-content" | git hash-object -w --stdin) && + tmpgit read-tree HEAD && + tmpgit update-index --add \ + --cacheinfo "100644,$blob_a,foo" && + parent_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + parent=$(git commit-tree "$parent_tree" -p HEAD \ + -m "add foo as file") && + + # Child: "foo" becomes a directory (foo/bar.txt). + blob_b=$(echo "dir-content" | git hash-object -w --stdin) && + tmpgit read-tree "$parent" && + tmpgit update-index --remove foo && + tmpgit update-index --add \ + --cacheinfo "100644,$blob_b,foo/bar.txt" && + child_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + child=$(git commit-tree "$child_tree" -p "$parent" \ + -m "foo: file to directory") && + check_connected_trace trace-f2d.txt "" "" "$child" + ) +' + +test_expect_success "$mode: handles directory-to-file transition" ' + ( + cd main-repo && + + # Parent: "bar/baz.txt" exists (bar is a directory). + blob_a=$(echo "nested" | git hash-object -w --stdin) && + tmpgit read-tree HEAD && + tmpgit update-index --add \ + --cacheinfo "100644,$blob_a,bar/baz.txt" && + parent_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + parent=$(git commit-tree "$parent_tree" -p HEAD \ + -m "add bar as directory") && + + # Child: "bar" becomes a plain file. + blob_b=$(echo "flat" | git hash-object -w --stdin) && + tmpgit read-tree "$parent" && + tmpgit update-index --remove bar/baz.txt && + tmpgit update-index --add \ + --cacheinfo "100644,$blob_b,bar" && + child_tree=$(tmpgit write-tree) && + rm -f .git/tmp-idx && + child=$(git commit-tree "$child_tree" -p "$parent" \ + -m "bar: directory to file") && + check_connected_trace trace-d2f.txt "" "" "$child" + ) +' + +# Replacement objects. + +test_expect_success "$mode: accepts with replacement objects" ' + ( + cd replace-test && + check_connected "$original_oid" + ) +' + +test_expect_success "$mode: rejects without replacement objects" ' + ( + cd replace-test && + GIT_NO_REPLACE_OBJECTS=1 && + export GIT_NO_REPLACE_OBJECTS && + test_expect_code 128 check_connected \ + "$original_oid" 2>err && + test_grep "missing blob object" err + ) +' + +test_expect_success "$mode: accepts missing promised blob" ' + test_when_finished "rm -rf prom-src prom-server.git prom-client" && + git init prom-src && + test_commit -C prom-src --no-tag base file.txt original && + test_commit -C prom-src --no-tag "add file2" file2.txt extra && + git clone --bare prom-src prom-server.git && + git -C prom-server.git config uploadpack.allowfilter true && + git -C prom-server.git config uploadpack.allowanysha1inwant true && + git clone --no-checkout --filter=blob:none \ + "file://$(pwd)/prom-server.git" prom-client && + ( + cd prom-client && + promised_blob=$(git rev-parse HEAD:file2.txt) && + test_must_fail env GIT_NO_LAZY_FETCH=1 \ + git cat-file -e "$promised_blob" && + new_tree=$(printf "100644 blob %s\tnewname.txt\n" \ + "$promised_blob" | + git mktree --missing) && + new_commit=$(git commit-tree "$new_tree" \ + -p HEAD -m "reuse promised blob") && + check_connected "$new_commit" && + test_must_fail env GIT_NO_LAZY_FETCH=1 \ + git cat-file -e "$promised_blob" + ) +' + +test_expect_success "$mode: accepts missing promised tree" ' + test_when_finished "rm -rf prom-tree-src prom-tree-server.git prom-tree-client" && + git init prom-tree-src && + mkdir -p prom-tree-src/a/b && + test_commit -C prom-tree-src --no-tag "nested dirs" a/b/file.txt deep && + git clone --bare prom-tree-src prom-tree-server.git && + git -C prom-tree-server.git config uploadpack.allowfilter true && + git -C prom-tree-server.git config uploadpack.allowanysha1inwant true && + git clone --no-checkout --filter=tree:1 \ + "file://$(pwd)/prom-tree-server.git" prom-tree-client && + ( + cd prom-tree-client && + promised_tree=$(git ls-tree HEAD -- a | + awk "{print \$3}") && + test_must_fail env GIT_NO_LAZY_FETCH=1 \ + git cat-file -e "$promised_tree" && + new_tree=$(printf "40000 tree %s\trenamed\n" \ + "$promised_tree" | + git mktree --missing) && + new_commit=$(git commit-tree "$new_tree" \ + -p HEAD -m "reuse promised tree") && + check_connected "$new_commit" && + test_must_fail env GIT_NO_LAZY_FETCH=1 \ + git cat-file -e "$promised_tree" + ) +' + +test_expect_success "$mode: verifies local commit in partial clone" ' + test_when_finished "rm -rf pc-src pc-server.git pc-client" && + git init pc-src && + test_commit -C pc-src --no-tag base file.txt && + git clone --bare pc-src pc-server.git && + git -C pc-server.git config uploadpack.allowfilter true && + git -C pc-server.git config uploadpack.allowanysha1inwant true && + git clone --filter=blob:none \ + "file://$(pwd)/pc-server.git" pc-client && + ( + cd pc-client && + local_commit=$(commit_with_change HEAD file.txt local-content) && + check_connected "$local_commit" + ) +' + +test_expect_success "$mode: respects shallow boundary" ' + test_when_finished "rm -rf shallow-src shallow" && + git init shallow-src && + test_commit -C shallow-src --no-tag base file content-1 && + mkdir shallow-src/sub && + test_commit -C shallow-src --no-tag change sub/other content-2 && + git clone --depth=1 "file://$(pwd)/shallow-src" shallow && + ( + cd shallow && + tip=$(git rev-parse HEAD) && + git for-each-ref --format="delete %(refname)" | + git update-ref --no-deref --stdin && + check_connected "$tip" + ) +' + +test_expect_success "$mode: deepening fetch succeeds" ' + test_when_finished "rm -rf deepen-src deepen-server.git deepen-client" && + git init deepen-src && + test_commit -C deepen-src --no-tag c1 file.txt && + test_commit -C deepen-src --no-tag c2 file.txt && + test_commit -C deepen-src --no-tag c3 file.txt && + git clone --bare deepen-src deepen-server.git && + git clone --depth=1 "file://$(pwd)/deepen-server.git" deepen-client && + set_connectivity_check deepen-client $mode && + test -f deepen-client/.git/shallow && + GIT_TRACE2_EVENT="$(pwd)/deepen-trace.txt" \ + git -C deepen-client fetch --deepen=2 origin main && + # Incremental falls back to full for deepening fetches, + # so the trees_loaded event should not appear. + test_grep ! trees_loaded deepen-trace.txt +' + +test_expect_success "$mode: malformed tree detected" ' + ( + cd main-repo && + echo abc >malformed-tree && + malformed_tree=$(git hash-object --literally -t tree -w \ + malformed-tree) && + malformed_commit=$(git commit-tree "$malformed_tree" \ + -p HEAD -m malformed) && + test_expect_code 128 check_connected \ + "$malformed_commit" 2>err + ) +' + +test_expect_success PERL_TEST_HELPERS \ + "$mode: mid-tree corruption detected" ' + ( + cd main-repo && + # Build a tree with one valid entry followed by garbage. + blob_oid=$(echo "valid" | git hash-object -w --stdin) && + bin_oid=$(echo "$blob_oid" | hex2oct) && + printf "100644 good\0${bin_oid}GARBAGE" >corrupt-mid-tree && + corrupt_tree=$(git hash-object --literally -t tree -w \ + corrupt-mid-tree) && + corrupt_commit=$(git commit-tree "$corrupt_tree" \ + -p HEAD -m "mid-tree corruption") && + test_expect_code 128 check_connected \ + "$corrupt_commit" 2>err && + test_grep "too-short tree object" err + ) +' + +done + +# Algorithm selection. + +test_expect_success 'invalid transfer.connectivityCheck is rejected' ' + test_when_finished "rm -rf invalid-cfg-src invalid-cfg-dst" && + git init invalid-cfg-src && + test_commit -C invalid-cfg-src --no-tag base file.txt && + git clone invalid-cfg-src invalid-cfg-dst && + test_commit -C invalid-cfg-src --no-tag update file.txt updated && + git -C invalid-cfg-dst config transfer.connectivityCheck bogus && + test_must_fail git -C invalid-cfg-dst fetch origin main 2>err && + test_grep "unknown transfer.connectivityCheck" err +' + +test_expect_success 'push uses incremental when configured' ' + test_when_finished "rm -rf int-src int-dst.git" && + git init int-src && + test_commit -C int-src --no-tag base file.txt && + git clone --bare int-src int-dst.git && + test_commit -C int-src --no-tag update file.txt updated && + set_connectivity_check int-dst.git incremental && + GIT_TRACE2_EVENT="$(pwd)/push-trace.txt" \ + git -C int-src push ../int-dst.git main && + test_trace2_data_singular connectivity trees_loaded 2 \ + path, entry->pathlen, + entry->mode); + if (cmp > 0) + break; + if (cmp < 0) { + wi++; + continue; + } + if (S_ISDIR(work[wi].entry.mode) && + S_ISDIR(entry->mode)) + oid_array_append(&work[wi].parent_trees, + &entry->oid); + return wi + 1; + } + return wi; +} + +static void verify_blob(struct repository *repo, + const struct object_id *oid, + struct verify_state *vs) +{ + if (oidset_contains(&vs->trusted_trees, oid)) + die(_("object %s is a tree, not a blob"), + oid_to_hex(oid)); + vs->blobs_checked++; + if (odb_has_object(repo->objects, oid, 0) || + (vs->exclude_promisor_objects && + is_promisor_object(repo, oid))) { + oidset_insert(&vs->trusted_blobs, oid); + return; + } + die(_("missing blob object '%s'"), oid_to_hex(oid)); +} + +static void verify_tree(struct repository *repo, + const struct object_id *new_tree_oid, + const struct oid_array *base_trees, + struct verify_state *vs, int depth) +{ + struct tree *tree; + struct tree_desc desc; + struct name_entry entry; + struct work_item *work = NULL; + size_t nr_work = 0, alloc_work = 0; + int need_subtree_bases = 0; + size_t i; + + if (depth > repo->settings.max_allowed_tree_depth) + die(_("exceeded maximum allowed tree depth")); + + if (oidset_contains(&vs->trusted_trees, new_tree_oid)) + return; + + tree = lookup_tree(repo, new_tree_oid); + if (!tree || repo_parse_tree_gently(repo, tree, 1)) { + if (odb_has_object(repo->objects, new_tree_oid, 0)) + die(_("malformed tree object %s"), + oid_to_hex(new_tree_oid)); + if (vs->exclude_promisor_objects && + is_promisor_object(repo, new_tree_oid)) { + oidset_insert(&vs->trusted_trees, new_tree_oid); + return; + } + die(_("bad tree object %s"), + oid_to_hex(new_tree_oid)); + } + + vs->trees_loaded++; + init_tree_desc(&desc, &tree->object.oid, + tree->buffer, tree->size); + + while (tree_entry(&desc, &entry)) { + if (S_ISGITLINK(entry.mode)) + continue; + if (S_ISDIR(entry.mode)) { + if (oidset_contains(&vs->trusted_trees, &entry.oid)) + continue; + need_subtree_bases = 1; + } else { + if (oidset_contains(&vs->trusted_blobs, &entry.oid)) + continue; + } + ALLOC_GROW(work, nr_work + 1, alloc_work); + work[nr_work] = (struct work_item){ .entry = entry }; + nr_work++; + } + + if (!nr_work) + goto done; + + for (i = 0; base_trees && i < base_trees->nr; i++) { + const struct object_id *base_oid = &base_trees->oid[i]; + int expanded = oidset_contains(&vs->expanded_trees, base_oid); + struct tree *base; + struct tree_desc base_desc; + struct name_entry scan_entry; + size_t wi = 0; + + if (expanded && !need_subtree_bases) + continue; + + base = lookup_tree(repo, base_oid); + if (!base || repo_parse_tree_gently(repo, base, 1)) + die(_("bad tree object %s"), + oid_to_hex(base_oid)); + + vs->trees_loaded++; + init_tree_desc(&base_desc, &base->object.oid, + base->buffer, base->size); + + while (tree_entry(&base_desc, &scan_entry)) { + if (S_ISGITLINK(scan_entry.mode)) + continue; + + if (need_subtree_bases) + wi = collect_subtree_bases(work, nr_work, + wi, &scan_entry); + + if (!expanded) { + struct oidset *set = S_ISDIR(scan_entry.mode) + ? &vs->trusted_trees + : &vs->trusted_blobs; + oidset_insert(set, &scan_entry.oid); + } + } + + if (!expanded) + oidset_insert(&vs->expanded_trees, base_oid); + + free_tree_buffer(base); + } + + for (i = 0; i < nr_work; i++) { + if (S_ISDIR(work[i].entry.mode)) { + if (!oidset_contains(&vs->trusted_trees, + &work[i].entry.oid)) + verify_tree(repo, &work[i].entry.oid, + &work[i].parent_trees, vs, + depth + 1); + } else { + if (!oidset_contains(&vs->trusted_blobs, + &work[i].entry.oid)) + verify_blob(repo, &work[i].entry.oid, vs); + } + } + +done: + oidset_insert(&vs->trusted_trees, new_tree_oid); + oidset_insert(&vs->expanded_trees, new_tree_oid); + free_tree_buffer(tree); + for (i = 0; i < nr_work; i++) + oid_array_clear(&work[i].parent_trees); + free(work); +} + +static void verify_commit_tree(struct repository *repo, + struct commit *commit, + struct verify_state *vs) +{ + struct oid_array base_trees = OID_ARRAY_INIT; + struct commit_list *p; + + for (p = commit->parents; p; p = p->next) { + const struct object_id *tree_oid; + parse_commit_or_die(p->item); + tree_oid = get_commit_tree_oid(p->item); + oidset_insert(&vs->trusted_trees, tree_oid); + oid_array_append(&base_trees, tree_oid); + } + + verify_tree(repo, get_commit_tree_oid(commit), + &base_trees, vs, 0); + oid_array_clear(&base_trees); +} + +void verify_commits_incremental(struct repository *repo, + struct commit_list **commits, + int exclude_promisor_objects) +{ + struct verify_state vs = { 0 }; + struct commit_list *iter; + unsigned nr_before; + + vs.exclude_promisor_objects = exclude_promisor_objects; + + nr_before = commit_list_count(*commits); + sort_in_topological_order(commits, REV_SORT_IN_GRAPH_ORDER); + if (commit_list_count(*commits) < nr_before) + die(_("cycle detected in incoming commit graph")); + + *commits = commit_list_reverse(*commits); + + for (iter = *commits; iter; iter = iter->next) + verify_commit_tree(repo, iter->item, &vs); + + oidset_clear(&vs.trusted_trees); + oidset_clear(&vs.trusted_blobs); + oidset_clear(&vs.expanded_trees); + trace2_data_intmax("connectivity", repo, + "trees_loaded", vs.trees_loaded); + trace2_data_intmax("connectivity", repo, + "blobs_checked", vs.blobs_checked); +} diff --git a/tree-verify.h b/tree-verify.h new file mode 100644 index 00000000000000..6aadadff704d9e --- /dev/null +++ b/tree-verify.h @@ -0,0 +1,15 @@ +#ifndef TREE_VERIFY_H +#define TREE_VERIFY_H + +struct commit_list; +struct repository; + +/* + * Verify trees of commits incrementally against their parents. + * Dies on verification failure. + */ +void verify_commits_incremental(struct repository *repo, + struct commit_list **commits, + int exclude_promisor_objects); + +#endif /* TREE_VERIFY_H */