feat(modelartifacts): support bounded parallel Hugging Face file downloads

[localai-bot]

Problem

Hugging Face artifact materialization downloads every selected snapshot file sequentially through downloader.DownloadFilesWithContext. Repositories with many shards and support files therefore underutilize available network bandwidth and spend substantial time on per-file request latency.

The executor is explicitly sequential today:

for i := range tasks {
    ...
    task.URI.DownloadFileWithContext(...)
}

Proposal

Add a bounded, configurable parallelism setting for independent file downloads in a Hugging Face snapshot.

• Keep the current sequential behaviour as the default for backward compatibility.
• Add an explicit configuration surface (environment variable and application config) for the maximum number of concurrent artifact-file transfers.
• Use a bounded worker pool or errgroup with a concurrency limit.
• Preserve cancellation: first download/verification error cancels outstanding work and returns the original error.
• Preserve resumable .partial files and per-file SHA verification.
• Make manifest construction and aggregate progress concurrency-safe and deterministic (sort entries by path before writing).
• Do not parallelize chunks of a single file in this issue; only independent files/shards.

Open questions / acceptance criteria

• Choose a conservative default (likely 1 initially) and document tuning guidance.
• Ensure raw progress reports aggregate total bytes across concurrent files without exceeding the snapshot total.
• Add tests for max-concurrency enforcement, cancellation/error propagation, deterministic manifests, and resumable retry.
• Measure throughput on a representative sharded HF model and compare concurrency 1 vs 2/4/8.

Related

The current artifact materializer already resolves the full snapshot and downloads each file via the shared sequential executor. This issue intentionally does not change file-selection policy or the progress-update throttling path.

[localai-org-maint-bot]

I’m picking this up. I’ll keep concurrency opt-in (default 1), preserve cancellation/resume/verification semantics, make manifest/progress handling deterministic and concurrency-safe, and add focused tests and configuration documentation before opening a PR.