Big Repositories, Shallow, Partial and Sparse.

You need to fix a typo in one documentation page. The repository is large, and git clone is going to download every version of every file anyone ever committed.

Git has three ways to take less. They sound alike, and they cut different things.

• A shallow clone takes fewer commits.
• A partial clone takes the commits but leaves out file contents until you need them.
• A sparse checkout takes everything, but only puts some folders on your disk.

We will build a small repository with real history and clone it all three ways. That gives us a table of what each one cost on disk, and the commands that undo each one:

git clone --depth 1 <url>
git clone --filter=blob:none --sparse <url>
git sparse-checkout set docs

The cost of the full clone#

This is the baseline. A normal clone of our stand-in remote:

$ git clone file:///C:/.../hub.git full
$ git rev-list --count main
29
$ git count-objects -vH
...
size-pack: 4.29 MiB

size-pack is the size of the packed history on disk. 4.29 MiB for a small test repository is the number to beat.

Shallow: fewer commits#

--depth 1 asks for the latest commit and nothing before it.

$ git clone --depth 1 file:///C:/.../hub.git shallow
Cloning into 'shallow'...
$ git log --oneline
e9769cd Change 30
$ git rev-parse --is-shallow-repository
true
$ git count-objects -vH
...
size-pack: 154.67 KiB

154.67 KiB instead of 4.29 MiB. The cost is in that git log output: there is one commit. Git remembers where it stopped in a file, .git/shallow, which holds the id of the commit that has its parents cut off.

That is fine for a build server that only needs to compile the latest version. It is not fine when you want git blame, git log on a file, or git bisect, because the history they need is not there.

You can get more, or all of it, later:

$ git fetch --depth=5
$ git log --oneline | wc -l
5
$ git fetch --unshallow
$ git log --oneline | wc -l
29
$ git rev-parse --is-shallow-repository
false

--depth=5 grows the history to five commits. --unshallow fetches the rest and turns the repository back into a normal one.

Partial: all commits, no file contents#

A partial clone keeps the whole commit history but leaves out file contents, which Git calls blobs. --filter=blob:none says: no blobs yet.

$ git clone --filter=blob:none file:///C:/.../hub.git part2
Cloning into 'part2'...
$ git rev-list --count main
29
$ git count-objects -vH
...
size-pack: 176.46 KiB

All 29 commits, and 176.46 KiB on disk. A shallow clone with a single commit and this one are about the same size here, and this one still has the history.

Where did the files go? Git fetches a blob when something needs it. Checking out main needed the current files, so those came with the clone. An old version of the data file is a different matter:

$ git show HEAD~20:assets/data.txt | head -c 60
/X/SMgBvhT7YnWhpJhZn6SgfWrgGUQ6Dooe1EmkpO0fQImxUNquFAriLInCn
$ git count-objects -vH
...
size-pack: 328.08 KiB

Reading a file from 20 commits ago made Git go back to the remote and fetch that one blob. size-pack grew by about 150 KB, which is one copy of the data file. Nothing printed, so you will not see it happen.

Two things to know before you rely on this:

• The server has to allow filters. With a plain bare repository, Git warned filtering not recognized by server, ignoring and sent everything. After git config uploadpack.allowFilter true on that repository, the filter worked. GitHub supports it.
• The repository now depends on the remote. Our test clone shows why in its config:

$ git config --get-regexp "remote.origin.(promisor|partialclonefilter)"
remote.origin.promisor true
remote.origin.partialclonefilter blob:none

promisor means the remote has promised to supply what is missing. Offline, with an old file you never opened, that promise cannot be kept.

Sparse: everything, but only some folders#

A sparse checkout is about your working folder, not about what you downloaded. You tell Git to put only the folders you name on disk.

The two ideas combine well. --sparse on a clone starts with only the top-level files:

$ git clone --filter=blob:none --sparse file:///C:/.../hub.git sp
$ ls
README.md
$ git count-objects -vH
...
size-pack: 24.40 KiB

24.40 KiB. Now add the folder we work in:

$ git sparse-checkout set docs
$ ls
README.md
docs
$ git sparse-checkout list
docs

Git fetched the files in docs, and size-pack rose to 26.89 KiB. The assets and src folders are not on disk.

set replaces the list. add extends it:

$ git sparse-checkout add src
$ ls
README.md
docs
src

In Git 2.45.2, set works in what Git calls cone mode: you name whole folders, and everything inside comes with them. git config core.sparseCheckoutCone printed true here.

Finally, to go back to the whole tree:

$ git sparse-checkout disable
$ ls
README.md
assets
docs
src
$ git count-objects -vH
...
size-pack: 179.69 KiB

Git fetched the files it had skipped, and the pack grew from 26.89 KiB to 179.69 KiB.

What about Git LFS?#

Git LFS is a different tool for a different problem: a few very large files, such as models or design files, that you do not want in history at all. It stores a small text pointer in Git and the file itself elsewhere.

We ran it in a scratch repository, with the file type tracked before the file was added:

$ git lfs install --local
Updated Git hooks.
Git LFS initialized.
$ git lfs track "*.bin"
Tracking "*.bin"
$ cat .gitattributes
*.bin filter=lfs diff=lfs merge=lfs -text
$ git lfs ls-files
52ca684f48 * model.bin
$ git cat-file -p HEAD:model.bin
version https://git-lfs.github.com/spec/v1
oid sha256:52ca684f4830a78b5a958d39f26e5bd276931ae753f2e3ea8476dd9f742be4ee
size 100000

The commit holds three lines of text. The 100,000 bytes live in LFS.

It does not help with a file you already committed. In a second test, we added old.bin, committed it, and only then ran git lfs track "*.bin". git lfs ls-files printed nothing, and git cat-file -s HEAD:old.bin printed 100000, the real file. Tracking applies to new commits. History that already has the big file keeps it.

git lfs migrate, the tool for rewriting history, and GitHub's LFS storage limits were not tested here. Look those up before you rely on them.

Try it#

Here is the whole tour in seven commands. The remote is https://github.com/you/big.git. Type them one at a time.

Two things worth trying in the terminal: ask for a hint when you are stuck, and add src with git sparse-checkout add src before disabling.

Challenge: all the history, two folders

Make a new clone that has every commit but only docs and src on disk. Do it with a filter, not with --depth. Then prove that assets is missing from the disk and not from Git.

Expected result: git rev-list --count main prints the same number as in a full clone, ls shows README.md, docs and src, ls assets says No such file or directory, and git ls-files still lists assets/data.txt.

$ git clone --filter=blob:none --sparse file:///C:/.../hub.git lean
$ cd lean
$ git sparse-checkout set docs src
$ git rev-list --count main
29
$ ls assets
ls: cannot access 'assets': No such file or directory
$ git ls-files
README.md
assets/data.txt
docs/guide.md
src/main.c

What we learned#

• git clone --depth 1 takes only the latest commit. git fetch --unshallow takes the rest. With local folders, use file:/// or Git ignores --depth.
• --filter=blob:none takes every commit and fetches file contents when something needs them. The server has to allow it, and the repository depends on the remote afterwards.
• git sparse-checkout set <folders> puts only those folders on your disk. add extends the list, list shows it, disable brings back the whole tree.
• A shallow clone cuts history, a partial clone cuts file contents, a sparse checkout cuts what is on disk. They can be combined.
• Git LFS keeps large files out of history from the moment you track them, and does nothing for files already committed.

Next time you only need one folder, you will not wait for the rest.