Builds often fetch dependencies directly from Git repositories. Parallel CI jobs can clone the same repositories repeatedly, downloading the same history and putting load on upstream servers. It would be useful to cache those downloads alongside the package downloads proxy already handles.
This is inspired by @montehurd’s git-cache-proxy, which keeps local bare repositories and serves them through Git’s smart HTTP protocol. Clients use a rewritten repository URL with ordinary git clone and git fetch commands.
Issue 112 raised this use case for Crystal packages and Ruby dependencies sourced directly from Git. The generic HTTP support in PR 302 and source-archive caching proposed in PR 399 cover related downloads. This proposal would add a dedicated Git handler.
An initial implementation could support:
- Read-only Git access over HTTP through configured upstreams.
- Local repository caches, refreshed as branches and tags change.
- Concurrent requests sharing one upstream refresh.
- Cache usage and upstream activity reported through the existing metrics and UI.
Git’s git-http-backend could handle the protocol, with proxy managing upstream access and repository storage. Local disk would be a reasonable starting point. Ref freshness, serving cached repositories during upstream outages, and eviction of unused repositories need some design discussion.
Git PURLs could identify dependencies by repository and pinned commit. That would also provide a path to accepting Git dependencies in proxy mirror and SBOM ingestion, so builds could prepopulate repository caches alongside registry packages. This could follow the initial clone/fetch support.
@montehurd, would you be interested in helping shape or implement this? Your experience with git-cache-proxy would help establish which build workloads and client configurations we should support first.
Builds often fetch dependencies directly from Git repositories. Parallel CI jobs can clone the same repositories repeatedly, downloading the same history and putting load on upstream servers. It would be useful to cache those downloads alongside the package downloads proxy already handles.
This is inspired by @montehurd’s git-cache-proxy, which keeps local bare repositories and serves them through Git’s smart HTTP protocol. Clients use a rewritten repository URL with ordinary
git cloneandgit fetchcommands.Issue 112 raised this use case for Crystal packages and Ruby dependencies sourced directly from Git. The generic HTTP support in PR 302 and source-archive caching proposed in PR 399 cover related downloads. This proposal would add a dedicated Git handler.
An initial implementation could support:
Git’s
git-http-backendcould handle the protocol, with proxy managing upstream access and repository storage. Local disk would be a reasonable starting point. Ref freshness, serving cached repositories during upstream outages, and eviction of unused repositories need some design discussion.Git PURLs could identify dependencies by repository and pinned commit. That would also provide a path to accepting Git dependencies in
proxy mirrorand SBOM ingestion, so builds could prepopulate repository caches alongside registry packages. This could follow the initial clone/fetch support.@montehurd, would you be interested in helping shape or implement this? Your experience with git-cache-proxy would help establish which build workloads and client configurations we should support first.