Find what you need - explore our website and developer resources
Miłosz Kosobucki
15 June 2021
After many struggles with using git LFS on repositories that need to store big files, I decided to spend some time on checking the status of the built-in partial clone functionality that could possibly let you achieve the same (as of git 2.30).
TL;DR: The building blocks are there, but server-side support is spotty and critical tooling is missing. It's not very usable yet, but it's getting there.
Normally, when you clone a git repository, all file versions throughout the whole repository history are downloaded. If you have multiple revisions of multi-GB binary files, as we have in some projects, this becomes a problem.
Partial clone lets you download only a subset of objects in the repository and defer downloading the rest, until needed. Most of the time, it means checkout.
For example, to clone a repository with blobs in only the latest version of the default branch, you can do as follows:
git clone --filter=blob:none git@example.com:repo.gitThe --filter part is crucial; it tells you what to include/omit. It's called a filter-spec, across the git docs. The specifics of how you can filter are available in git help rev-list. You can filter based on blob size, location the in tree (slow! - guess why), or both.
The remote from which you cloned will be called a "promisor" remote, so-called because it promises to fulfill requests for missing objects when requested later:
[remote "origin"]
url = git@example.com:repo.git
fetch = +refs/heads/*:refs/remotes/origin/*
promisor = true
partialclonefilter = blob:limit=1048576 # limited to 1MAs you change branches, the required files will be downloaded on-demand during checkout.
Below is a video of a partial checkout in action. Notice how the actual files are downloaded during the checkout operation, and not during clone:
I checked out Linux kernel from the GitHub mirror, through regular and partial clone, and recorded some statistics:
As you can see, there are some tradeoffs. Checking out takes longer because the actual file content has to be downloaded, not just copied from object store. There are savings in terms of initial clone and repository size because you're not storing a copy of various driver sources deprecated since the late 90s. The gains would be even more pronounced in repositories that store multiple versions of big binary files. Think evolving game assets or CI system output.
The server side needs to implement git v2 protocol. Many don't do it yet, or do it in a limited manner.
:none or sizeetc/jgit.conf). Unfortunately, we experienced some bugs when trying this out:[uploadpack]
allowFilter = trueAs you check out new revisions with big files and download them, you will end up with lots of old data from the previous versions because it's not cleaned up automatically.
Git LFS has the git lfs prune tool. No such thing yet exists for partial clones.
See this git mailing list thread.
Since you want server-side operations to happen quickly, it's best to store the git repository on a very fast storage, which also happens to be expensive. It would be nice to store big files that don't really need fast operations (you won't do diffs on textures or sound files server-side) separately.
Christian Couder of GitLab is working on something around this. It's already possible to have multiple promisor remotes queried in succession. For example,
there could be separate promisor remote backed by a CDN or cloud storage (e.g. S3). However, servers will need to learn how to push the data there when users push their trees.
See this git mailing list thread.
Since everything is fresh, you need to add some magical incantations to git commands to have it working. Ideally, some "recommended" filter should be stored server-side, so that users don't have to come up with filter spec on their own, when cloning.
Below are some useful links, if you'd like to learn more about partial cloning in git:
Currently, a lot of effort around partial cloning is driven by Christian Couder of GitLab. You can follow some of the development under the following links:
If you would like to learn Git, KDAB offers an introductory training class.
About KDAB
The KDAB Group is a globally recognized provider for software consulting, development and training, specializing in embedded devices and complex cross-platform desktop applications. In addition to being leading experts in Qt, C++ and 3D technologies for over two decades, KDAB provides deep expertise across the stack, including Linux, Rust and modern UI frameworks. With 100+ employees from 20 countries and offices in Sweden, Germany, USA, France and UK, we serve clients around the world.
Choose between learning advanced QML programming, Modern C++, integrating Rust and Qt or 3d rendering with Vulkan. The KDAB Training Day 2025 will take place in Munich on the 8th of May, right after the Qt World Summit on 6th to 7th of May.
Learn more
Stay on top of the latest news, publications, events and more.
Go to Sign-up
Learn Modern C++
Our hands-on Modern C++ training courses are designed to quickly familiarize newcomers with the language. They also update professional C++ developers on the latest changes in the language and standard library introduced in recent C++ editions.
Learn more