Git speedups for large repos

[gjost]

Spend no more than 2 days on this.

The ddr-densho-1000 is really huge and this causes usability problems even when the repo is checked out locally. In particular, git status takes forever to run.
Repo has tons of files and also a long history (~4000 commits).

IDEA cp ddr-densho-1000 ddr-densho-1000new, remove .git/, git init
where does the slow come from?
TODO research git performance (num objects, size, repo age)
TODO can we set git caching interval?
TODO profile git operations
does not correlate to number of objects of phsyical size of repo
seems to be length commit history

Ways to improve git status performance (2012)
https://stackoverflow.com/questions/4994772/ways-to-improve-git-status-performance
10 GB repo on NFS on Linux. First time git status ~36min, subsequent 8min

Slow Git Performance (2021)
https://support.purestorage.com/Knowledge_Base/FlashBlade_KB/Slow_Git_Performance

OPTIONS

Shallow clone
git clone --depth=50 --no-single-branch COLLECTION

Sparse checkout
https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/
git clone COLLECTION
git sparse-checkout init --cone
git sparse-checkout set ...

Partial checkouts
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/
Blobless clones: git clone --filter=blob:none
Treeless clones: git clone --filter=tree:0

TODO Test shallow,sparse clones
TODO test on Dana's machine

[gjost]

We Put Half a Million files in One git Repository, Here’s What We Learned
https://canvatechblog.com/we-put-half-a-million-files-in-one-git-repository-heres-what-we-learned-ec734a764181
To reduce the amount of work git needs to do to find changes, we used the fsmonitor hook with Watchman so we capture changes as they happen instead of having to scan all files in the repository every time a command is run.
We also enabled feature.manyFiles, which under the hood enables the untracked cache to skip directories and files that haven’t been modified.
Git also has a built-in command (maintenance) to optimize a repository’s data, speeding up commands and reducing disk space. This isn’t enabled by default, so we register it with a schedule for daily and hourly routines.
Sparse checkout
If an engineer can tell us what they usually work on, we can craft a checkout pattern that includes all the required dependencies to run and test their code locally while keeping the checkout as small as possible.
Sparse checkout drawbacks:

• Tracked files not physically populated on disk can’t be searched through or interacted with. Accidental changes or an erroneous merge conflict might leave these files in a bad state.
• Overhead to every git checkout to check if updated file should be populated or ignored. This overhead is small with simple patterns but becomes significant with more complex ones.

https://news.ycombinator.com/item?id=31762245
Interesting

• https://github.com/microsoft/VFSForGit (defunct)
• https://github.com/microsoft/scalar
• https://github.com/twosigma/git-meta

The Case Against Monorepos (Infoworld)

Trunk-Based Development: Monorepos (https://trunkbaseddevelopment.com/monorepos)
monorepo.tools - Everything you need to know about monorepos, and the tools to build them (https://monorepo.tools)