URL detection: two gates leave 25 forms undetected and 4 resolving to the wrong VCS

[tony]

Summary

A sweep of 64 URL forms against libvcs v0.45.1 finds 35 detected correctly, 4 detected as the wrong VCS, and 25 not detected at all. The failures are not one bug — they come from two independent gates, and the two gates need different fixes. This issue consolidates the URL-detection gaps into one place.

Detection here means registry.match(url=url, is_explicit=True), which is what downstream callers use to pick a VCS backend.

Gate 1: the git scheme regex only matches http|https

RE_SCHEME in url/git.py accepts only http and https. Every other transport git supports has no matching rule, so GitURL.is_valid() returns False.

URL	git(1) supports	GitURL.is_valid
ssh://git@github.com/o/r.git	yes	False
ssh://git@github.com:22/o/r.git	yes	False
git://github.com/o/r.git	yes	False
file:///path/to/repo.git	yes	False
file://localhost/path/to/repo.git	yes	False
/path/to/repo.git	yes	False
ftp://host.xz/path/to/repo.git	yes, deprecated	False
ftps://host.xz/path/to/repo.git	yes, deprecated	False

git-clone(1) "GIT URLS" documents ssh://, git://, http[s]:// and ftp[s]://, plus the scp-like form and bare local paths. It explicitly deprecates ftp[s] ("this is inefficient and deprecated; do not use it"), so those two are low priority. ssh:// is the significant omission, since it is the form many hosts display for copy-paste.

Contrast: url/hg.py accepts http|https|ssh and url/svn.py accepts file|http|https|svn|svn+ssh. The git parser is the narrow one.

Not covered by the regex either, both documented by git-clone(1) under tilde expansion:

URL	GitURL.is_valid
git@github.com:~user/repo.git	False
ssh://git@host/~user/repo.git	False

Gate 2: hg and svn core rules are not marked explicit

is_explicit per rule, read from v0.45.1:

Parser	explicit rules	non-explicit rules
git.py	core-git-https, core-git-scp, 4 AWS CodeCommit rules, pip-url, pip-scp-url, pip-file-url	none
hg.py	pip-url, pip-file-url	core-hg, core-hg-scp
svn.py	pip-url, pip-file-url	core-svn, core-svn-scp

Because match(is_explicit=True) only consults explicit rules, hg and svn are reachable only through their pip-style spellings. Their native URLs parse but never surface:

URL	own parser is_valid	registry.match
ssh://hg@bitbucket.org/user/repo	True	no match
svn://svn.example.com/repo/trunk	True	no match

The same gate produces four cases where an http(s) URL for an hg or svn repository is claimed by git, since core-git-https is the only explicit rule that can match it:

URL	actual	detected
https://bitbucket.org/user/repo	hg	git
http://hg.example.com/repo	hg	git
https://svn.example.com/repo/trunk	svn	git
http://svn.example.com/repo/trunk	svn	git

This is the worst class in the sweep: a confident wrong answer rather than an error. Mercurial and Subversion both serve over plain http(s), so these are ordinary URLs, not exotic ones.

Flipping the flag alone regresses git

Marking core-hg, core-hg-scp, core-svn and core-svn-scp explicit fixes the two undetected rows, but makes the most common git URLs ambiguous, because three parsers then match the same string:

hg           hg           ssh://hg@bitbucket.org/user/repo
svn          svn          svn://svn.example.com/repo/trunk
AMBIG        git,hg,svn   https://github.com/pallets/flask.git
AMBIG        git,hg,svn   https://bitbucket.org/user/repo
AMBIG        git,hg,svn   git@github.com:pallets/flask.git
git          git          git+https://github.com/o/r.git

Callers that treat "more than one match" as undetectable would start failing on https://github.com/... and on scp-style URLs. So this needs a disambiguation strategy, not a one-line change.

Rule already declares a weight field described as "Higher is more likely to win", but VCSRegistry.match() never reads it — it appends every parser whose is_valid returns True and returns them unranked. Wiring weight into match() (or returning matches sorted by it) looks like the missing piece that would let the core rules be explicit without making git ambiguous.

Other gaps in the sweep

Uppercase schemes are rejected. HTTPS://github.com/o/r.git and GIT+SSH://git@host/o/r.git both fail. RFC 3986 §3.1 states schemes are case-insensitive and that implementations "should accept uppercase letters as equivalent to lowercase in scheme names for robustness".

Trailing whitespace validates, then breaks at clone. A trailing space or newline is accepted as a valid URL; a leading space is rejected. The trailing form reaches git and fails there:

fatal: '/path/to/repo.git ' does not appear to be a git repository
fatal: Could not read from remote repository.

A URL pasted from a terminal or read line-wise from a file carries exactly this, and the resulting error names no cause. Trimming on parse, or rejecting both ends, would be consistent.

git+git:// is rejected, and should stay rejected. pip discourages git, git+git and git+http because the git protocol has no authentication and http has no TLS. Noted so it is not mistaken for a gap.

Local and Windows paths are rejected, which is correct for path-vs-URL routing. /srv/git/repo.git, ./repo.git, ../repo.git, ~/code/repo.git, C:\code\repo.git, C:/code/repo.git and \\server\share\repo.git all return no match. Notably C:\... is not mistaken for scp-style despite resembling host:path, which matches git's own rule that the scp form "is only recognized if there are no slashes before the first colon". Callers that use is_valid to distinguish a URL argument from a filesystem path can rely on this. It is listed here only because a caller storing a bare local path in a config cannot then resolve it.

Minor: attribute docstrings in Rule are shifted by one. In base.py, defaults is followed by "Is the match unambiguous with other VCS systems? e.g. git+ prefix" (which describes is_explicit), and is_explicit is followed by "Weight: Higher is more likely to win" (which describes weight). weight itself ends up undocumented.

Evidence that this is unfinished rather than intentional

NPM_DEFAULT_RULES is an empty list whose own docstring gives ssh://git@github.com:npm/cli.git#v1.0.27 and git://github.com/npm/cli.git#v1.0.27 as target formats. It is never referenced anywhere else in the package. tests/url/test_git.py contains no assertion that bare ssh:// or git:// must fail; its ssh:// occurrences are all git+ssh:// fixtures plus an AWS CodeCommit comment. The only "unsupported" assertions in the suite are the is_generic=False markers for the host-specific CodeCommit rule.

Consolidation

• URL parser: ssh://, git://, ftp(s):// and bare file:// are not detected #545 is fully contained in this issue (it covered Gate 1 only) and can be closed in favour of this one.
• URL Parser: ADA compliance #495 (URL Parser: ADA compliance) proposes replacing the regex engine with WHATWG URL / ada-url parsing. That would likely subsume Gate 1 and the uppercase-scheme and whitespace items, but not Gate 2, which is about rule metadata rather than parsing. Complementary, not duplicate.
• GitPipURL: to_pip_url() #387 (GitPipURL: to_pip_url()) covers serialising back out to a pip URL — the reverse direction. Related through the pip-vs-git URL split, but distinct.
• Abstract away pip url / url pattern support #272 (Abstract away pip url / url pattern support) proposes runtime-registrable patterns and VCS detection hint callbacks. The "detection hint" idea overlaps directly with the disambiguation problem above.

Suggested order

1. Wire weight into VCSRegistry.match() so ranked matches are possible.
2. Mark the hg and svn core rules explicit, using weight to keep git winning for bare http(s) and scp-style.
3. Extend git's RE_SCHEME to ssh, git and file, plus tilde-path support.
4. Normalise scheme case and trim surrounding whitespace on parse.
5. Optionally add ftp/ftps, noting git deprecates them.

Bibliography

• git-clone(1), "GIT URLS" — the four remote schemes, scp-like syntax, tilde expansion, local paths
• Pro Git, "The Protocols" — transport trade-offs, including why the git protocol is unauthenticated
• RFC 3986 §3.1, Scheme — scheme case-insensitivity
• RFC 3986 §6.2.2.1, Case Normalization — lowercase scheme and host when normalising
• pip, VCS Support — git+*, hg+*, bzr+* scheme lists and the security note on git/git+git/git+http
• Mercurial, URL Paths — file://, http(s)://, ssh://, static-http://
• Version Control with Subversion, "Subversion in Action" — file://, svn://, svn+ssh://, http(s)://
• WHATWG URL Standard, URL parsing — referenced by URL Parser: ADA compliance #495
• npm, Git URLs as dependencies — the format the NPM_DEFAULT_RULES stub targets

[tony]

Downstream tracking issue: vcs-python/vcspull#565, covering what this would resolve for vcspull and the discriminator work that stays on that side.