fget

Download files from a list of HTTP(S) URLs with bounded concurrency, streaming writes, and predictable output paths.

• Read URLs from stdin or download a single URL with --url.
• Configure headers, an explicit proxy, timeouts, and a per-file size limit.
• Verify TLS certificates by default and report HTTP failures.
• Keep incomplete downloads in temporary files and protect existing files by default.
• Run on Windows, Linux, and macOS.

Install

Build the current checkout with Go 1.26.0 or newer:

git clone https://github.com/bp0lr/fget.git
cd fget
go build -o fget .
./fget --help

On Windows, use go build -o fget.exe . and .\fget.exe --help.

The module recommends the Go 1.27.1 toolchain. Go selects/downloads it automatically when toolchain switching is enabled. You can use GOTOOLCHAIN=local to stay on an installed, supported version. The project does not change your system-wide Go installation.

To install the latest published version into your Go binary directory:

go install github.com/bp0lr/fget@latest

Add $(go env GOPATH)/bin to PATH if necessary (or your configured GOBIN). Unreleased changes require building this checkout. go get is no longer the command for installing executables; see the Go installation guidance.

Prebuilt ZIP archives can be generated by the Release artifacts workflow. Each archive contains the binary and this README; checksums.txt contains SHA-256 hashes. These workflow artifacts are not automatically published as GitHub Releases.

Quick start

Download one file:

fget -u 'https://example.com/files/report.csv'

Read a file containing one URL per line (Bash):

fget -o downloads --unique < urls.txt

The same input in PowerShell:

Get-Content .\urls.txt | fget -o .\downloads --unique

Store files directly in a directory and set a request header:

fget -u 'https://example.com/files/report.csv' -o downloads --no-folders -H 'Accept: text/csv'

Follow redirects, use four workers, and limit each file to 10 MiB:

fget -f -w 4 -t 60 --max-size 10485760 --unique < urls.txt

Use an explicit proxy:

fget -u 'https://example.com/file.txt' -p 'http://127.0.0.1:8080'

Use --insecure only when intentionally accepting an unverified TLS certificate. --proxy accepts HTTP and HTTPS proxy URLs; proxy environment variables are not used.

Options

Run fget --help for the built-in reference.

Option	Default	Meaning
-u, --url URL	stdin	Download a single HTTP(S) URL; ignore stdin
-o, --output DIR	current directory	Base directory; see output layout below
-w, --workers N	20	Concurrent downloads, from 1 to 100
-t, --timeout N	20	HTTP request timeout in seconds, including redirects and response-body reading
-H, --header 'Name: value'	none	Repeatable request header; the last value for a name wins
-p, --proxy URL	none	Explicit HTTP(S) proxy, optionally with credentials
-f, --follow-redirect	false	Follow up to 10 redirects
-v, --verbose	false	Print each saved file and its size
-r, --random-agent	false	Pick a User-Agent from the legacy browser compatibility list
--no-folders	false	Save directly in the base directory
--unique	false	Include the host and a stable hash of the URL in filenames
--insecure	false	Disable TLS certificate verification
--overwrite	false	Replace an existing file only after downloading the new content successfully
--max-size BYTES	0	Maximum saved response-body size per file; zero means unlimited
--version		Print build version
-h, --help		Print help

The default User-Agent identifies fget and its version. An explicit -H 'User-Agent: ...' takes precedence over both the default and --random-agent. The Host header is supported. On a redirect to another origin (scheme or host/port), custom headers and credentials are removed; only User-Agent is retained. Same-origin redirects retain headers.

Output layout

For https://example.com/assets/app.js:

Arguments	Saved path
none	results/example.com/app.js
-o downloads	downloads/results/example.com/app.js
-o downloads --no-folders	downloads/app.js
--no-folders	app.js in the current directory

The existing meaning of -o is preserved: it adds results/<host> unless --no-folders is set. URL directory trees are not recreated. Redirected downloads use the original URL for naming.

URLs without a filename use index.html. Characters unsuitable for filenames, Windows device names, trailing dots/spaces, and long components are normalized. A host with a port is normalized too: example.com:8080 becomes example.com_8080.

With --unique, the format is <host>_<16-hex-URL-hash>_<filename>. Paths and query strings contribute to the hash; fragments do not. This distinguishes, for example, /a/app.js, /b/app.js, and /a/app.js?v=2. It is deterministic, so submitting the same URL again still encounters the existing file. Rare hash or normalization collisions are protected by the same no-overwrite policy.

By default, an existing destination is an error, including a collision with another worker. Use --unique to distinguish URLs or --overwrite to intentionally replace a file. With --overwrite, if multiple URLs resolve to the same destination, the last successful publication wins.

Downloads use .fget-*.part files in the destination directory, with owner-only permissions where supported. Normal errors and Ctrl+C remove these temporary files. The completed file is published only after the response is fully read and the file is synced and closed. A forced process termination may leave a .part file.

Default publication uses a hard link so another worker or process cannot be silently overwritten. The destination filesystem must support hard links (for example NTFS, APFS, or ext4); unsupported filesystems report an error. --overwrite uses rename instead, whose atomicity depends on the operating system/filesystem. Output writes are confined through Go's os.Root APIs.

Errors and scripting

Only HTTP 2xx responses count as successful downloads. A redirect without --follow-redirect is a failure; its body is not saved. Non-2xx responses, timeouts, incomplete response bodies, size-limit violations, and filesystem errors are reported even without --verbose.

Input is processed as it arrives. Blank lines are ignored, surrounding whitespace is trimmed, and lines must be shorter than 1 MiB. Invalid lines are reported while other URLs continue. Empty input is an error. Both the queue and active downloads are bounded by the worker count. The response is streamed to disk; memory does not scale with the size of each file. The size limit also applies to decompressed response bodies when Go automatically decodes gzip.

Diagnostics and the final summary go to stderr. stdout is reserved for help/version output. URL credentials and query strings are omitted from the normal URL labels in diagnostics.

Downloaded: 3; failed: 1; bytes: 12345

Exit code	Meaning
0	All downloads succeeded, or help/version was printed
1	A download or filesystem operation failed
2	Invalid arguments, invalid/empty input, or an input-reading error
130	Canceled, including Ctrl+C

Input errors take precedence over download errors; cancellation takes precedence over both. On cancellation, the summary describes completed/failed attempts, not queued URLs that never started. When calling run from Go with a blocking reader, the caller owns and closes that reader after cancellation.

Migrating from the original version

• TLS certificates are now verified. Use --insecure explicitly when required.
• HTTP errors no longer create successful output files.
• Existing files are preserved by default; use --overwrite explicitly.
• --unique now includes a URL hash, so generated filenames change.
• Invalid worker counts and timeouts are rejected rather than silently accepted or clamped.
• Errors and summaries are always visible on stderr, with meaningful exit codes.
• Files are created with owner-only permissions where supported.
• All existing flag names and the output-directory layout remain available.

Development

The project keeps one package with separate files for CLI/configuration, HTTP handling, downloads, and output naming. pflag is the only runtime dependency.

go mod download
go mod verify
go test ./... -timeout 2m -cover
go vet ./...
go test -race ./... -timeout 2m
go run golang.org/x/vuln/cmd/govulncheck@v1.7.0 ./...
go test -run '^$' -bench BenchmarkStreamingCopy -benchmem

The race detector requires a supported platform and C compiler. Tests use local HTTP/TLS servers and temporary directories; they do not download from public websites. Symlink tests skip where the OS does not allow creating symlinks. CI tests Go 1.26 and 1.27.1 on Windows, Linux, and macOS, and runs the race detector on Linux.

Format changed Go files with gofmt -w. Build or run the whole package (go build . / go run .), since the implementation now spans multiple files.

Build release artifacts

From the repository root:

go run ./scripts/release.go -version v1.0.0

This builds ZIP archives for Windows, Linux, and macOS, each on amd64 and arm64, plus dist/checksums.txt. It embeds the supplied version, uses CGO_ENABLED=0, preserves executable permissions in the ZIP, and normalizes archive timestamps. dev is also accepted for local previews. dist/ is ignored by Git. Tags matching v* and manual workflow runs generate the same artifacts without publishing a GitHub Release.

Verify an archive against checksums.txt, for example:

cd dist
sha256sum -c checksums.txt

Or in PowerShell, compare the reported hash with the matching entry:

Get-FileHash .\dist\fget_v1.0.0_windows_amd64.zip -Algorithm SHA256
Get-Content .\dist\checksums.txt

License

A project license has not been selected yet. No license file is included at this stage.