Migrate a Large (2GB+) Repository to GitHub

If you are migrating to GitHub having a repository larger than 2GB or have files in history larger than 100MB, you might encounter some resistance from GitHub.

GH001: Large files detected

If you have files larger, in your git history, larger than 100MB, you will most likely receive and error message including "GH001: Large files detected".

The solution for this problem is to rewrite your git history.

The question is "How". After trying a few approaches and failing miserably, I finally stumbled upon an awesome repository on GitHub.

Git-Filter-Repo can clean up large files from the git history blazing fast.

Remove Large Files From Git History

After git-filter-repo installation:

1. Make a fresh clone of your repository with the flag --mirror

2. Inside the new clone folder run:‌‌‌‌‌

git filter-repo --strip-blobs-bigger-than 99MB
delete files larger than 99MB from your git history

After filter-repo is finished you will have cleaned up the bare repository.

If your repository is not too large (less than 2GB) then you can run the following command to push the repository and all branches to the new GitHub repository:

git push --mirror {github-repo-link}
push a git repository including all branches

Push a Large Repository to GitHub Incrementally

1. If your repository is too large (2GB+) then you will need to push it to GitHub incrementally with the following command (Mac OS):

git rev-list --reverse --all | ruby -ne 'x ||=0; x += 1; print $_ if x % 30000 == 0;' | xargs -I{} git push {github-repo-link} +{}:refs/heads/master
push 30k commits at a time

2. The above command will leave a few commits out so to push the rest of the master branch run:

git push {github-repo-link} +master
push rest of master branch

3. And finally push all branches:

git push --mirror {github-repo-link}
push all branches

That's it!

That's how I managed to migrate a large repository with unnecessary large files to GitHub.

Adnan Mujkanovic

Adnan Mujkanovic

Curious by nature, I love learning and grasping new concepts. My areas of interest are software development, DevOps, Economics, Philosophy and Science.