How to remove/delete a large file from commit history in the Git repository?

Question

I accidentally dropped a DVD-rip into a website project, then carelessly git commit -a -m ..., and, zap, the repo was bloated by 2.2 gigs. Next time I made some edits, deleted the video file, and committed everything, but the compressed file is still there in the repository, in history.

I know I can start branches from those commits and rebase one branch onto another. But what should I do to merge the 2 commits so that the big file doesn't show in the history and is cleaned in the garbage collection procedure?

Solution 1

Why not use this simple but powerful command?

git filter-branch --tree-filter 'rm -f DVD-rip' HEAD

The --tree-filter option runs the specified command after each checkout of the project and then recommits the results. In this case, you remove a file called DVD-rip from every snapshot, whether it exists or not.

If you know which commit introduced the huge file (say 35dsa2), you can replace HEAD with 35dsa2..HEAD to avoid rewriting too much history, thus avoiding diverging commits if you haven't pushed yet. This comment courtesy of @alpha_989 seems too important to leave out here.

See this link.

Solution 2

Use the BFG Repo-Cleaner, a simpler, faster alternative to git-filter-branch specifically designed for removing unwanted files from Git history.

Carefully follow the usage instructions, the core part is just this:

$ java -jar bfg.jar --strip-blobs-bigger-than 100M my-repo.git

Any files over 100MB in size (that aren't in your latest commit) will be removed from your Git repository's history. You can then use git gc to clean away the dead data:

$ git gc --prune=now --aggressive

The BFG is typically at least 10-50x faster than running git-filter-branch, and generally easier to use.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Solution 3

What you want to do is highly disruptive if you have published history to other developers. See “Recovering From Upstream Rebase” in the git rebase documentation for the necessary steps after repairing your history.

You have at least two options: git filter-branch and an interactive rebase, both explained below.

Using git filter-branch

I had a similar problem with bulky binary test data from a Subversion import and wrote about removing data from a git repository.

Say your git history is:

$ git lola --name-status
* f772d66 (HEAD, master) Login page
| A     login.html
* cb14efd Remove DVD-rip
| D     oops.iso
* ce36c98 Careless
| A     oops.iso
| A     other.html
* 5af4522 Admin page
| A     admin.html
* e738b63 Index
  A     index.html

Note that git lola is a non-standard but highly useful alias. With the --name-status switch, we can see tree modifications associated with each commit.

In the “Careless” commit (whose SHA1 object name is ce36c98) the file oops.iso is the DVD-rip added by accident and removed in the next commit, cb14efd. Using the technique described in the aforementioned blog post, the command to execute is:

git filter-branch --prune-empty -d /dev/shm/scratch \
  --index-filter "git rm --cached -f --ignore-unmatch oops.iso" \
  --tag-name-filter cat -- --all

Options:

• --prune-empty removes commits that become empty (i.e., do not change the tree) as a result of the filter operation. In the typical case, this option produces a cleaner history.
• -d names a temporary directory that does not yet exist to use for building the filtered history. If you are running on a modern Linux distribution, specifying a tree in /dev/shm will result in faster execution.
• --index-filter is the main event and runs against the index at each step in the history. You want to remove oops.iso wherever it is found, but it isn’t present in all commits. The command git rm --cached -f --ignore-unmatch oops.iso deletes the DVD-rip when it is present and does not fail otherwise.
• --tag-name-filter describes how to rewrite tag names. A filter of cat is the identity operation. Your repository, like the sample above, may not have any tags, but I included this option for full generality.
• -- specifies the end of options to git filter-branch
• --all following -- is shorthand for all refs. Your repository, like the sample above, may have only one ref (master), but I included this option for full generality.

After some churning, the history is now:

$ git lola --name-status
* 8e0a11c (HEAD, master) Login page
| A     login.html
* e45ac59 Careless
| A     other.html
|
| * f772d66 (refs/original/refs/heads/master) Login page
| | A   login.html
| * cb14efd Remove DVD-rip
| | D   oops.iso
| * ce36c98 Careless
|/  A   oops.iso
|   A   other.html
|
* 5af4522 Admin page
| A     admin.html
* e738b63 Index
  A     index.html

Notice that the new “Careless” commit adds only other.html and that the “Remove DVD-rip” commit is no longer on the master branch. The branch labeled refs/original/refs/heads/master contains your original commits in case you made a mistake. To remove it, follow the steps in “Checklist for Shrinking a Repository.”

$ git update-ref -d refs/original/refs/heads/master
$ git reflog expire --expire=now --all
$ git gc --prune=now

For a simpler alternative, clone the repository to discard the unwanted bits.

$ cd ~/src
$ mv repo repo.old
$ git clone file:///home/user/src/repo.old repo

Using a file:///... clone URL copies objects rather than creating hardlinks only.

Now your history is:

$ git lola --name-status
* 8e0a11c (HEAD, master) Login page
| A     login.html
* e45ac59 Careless
| A     other.html
* 5af4522 Admin page
| A     admin.html
* e738b63 Index
  A     index.html

The SHA1 object names for the first two commits (“Index” and “Admin page”) stayed the same because the filter operation did not modify those commits. “Careless” lost oops.iso and “Login page” got a new parent, so their SHA1s did change.

Interactive rebase

With a history of:

$ git lola --name-status
* f772d66 (HEAD, master) Login page
| A     login.html
* cb14efd Remove DVD-rip
| D     oops.iso
* ce36c98 Careless
| A     oops.iso
| A     other.html
* 5af4522 Admin page
| A     admin.html
* e738b63 Index
  A     index.html

you want to remove oops.iso from “Careless” as though you never added it, and then “Remove DVD-rip” is useless to you. Thus, our plan going into an interactive rebase is to keep “Admin page,” edit “Careless,” and discard “Remove DVD-rip.”

Running $ git rebase -i 5af4522 starts an editor with the following contents.

pick ce36c98 Careless
pick cb14efd Remove DVD-rip
pick f772d66 Login page

# Rebase 5af4522..f772d66 onto 5af4522
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#  x, exec = run command (the rest of the line) using shell
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.
#

Executing our plan, we modify it to

edit ce36c98 Careless
pick f772d66 Login page

# Rebase 5af4522..f772d66 onto 5af4522
# ...

That is, we delete the line with “Remove DVD-rip” and change the operation on “Careless” to be edit rather than pick.

Save-quitting the editor drops us at a command prompt with the following message.

Stopped at ce36c98... Careless
You can amend the commit now, with

        git commit --amend

Once you are satisfied with your changes, run

        git rebase --continue

As the message tells us, we are on the “Careless” commit we want to edit, so we run two commands.

$ git rm --cached oops.iso
$ git commit --amend -C HEAD
$ git rebase --continue

The first removes the offending file from the index. The second modifies or amends “Careless” to be the updated index and -C HEAD instructs git to reuse the old commit message. Finally, git rebase --continue goes ahead with the rest of the rebase operation.

This gives a history of:

$ git lola --name-status
* 93174be (HEAD, master) Login page
| A     login.html
* a570198 Careless
| A     other.html
* 5af4522 Admin page
| A     admin.html
* e738b63 Index
  A     index.html

which is what you want.

Solution 4

(The best answer I've seen to this problem is: https://stackoverflow.com/a/42544963/714112 , copied here since this thread appears high in Google search rankings but that other one doesn't)

🚀 A blazingly fast shell one-liner 🚀

This shell script displays all blob objects in the repository, sorted from smallest to largest.

For my sample repo, it ran about 100 times faster than the other ones found here.
On my trusty Athlon II X4 system, it handles the Linux Kernel repository with its 5,622,155 objects in just over a minute.

The Base Script

git rev-list --objects --all \
| git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
| awk '/^blob/ {print substr($0,6)}' \
| sort --numeric-sort --key=2 \
| cut --complement --characters=13-40 \
| numfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

When you run above code, you will get nice human-readable output like this:

...
0d99bb931299  530KiB path/to/some-image.jpg
2ba44098e28f   12MiB path/to/hires-image.png
bd1741ddce0d   63MiB path/to/some-video-1080p.mp4

🚀 Fast File Removal 🚀

Suppose you then want to remove the files a and b from every commit reachable from HEAD, you can use this command:

git filter-branch --index-filter 'git rm --cached --ignore-unmatch a b' HEAD

Solution 5

After trying virtually every answer in SO, I finally found this gem that quickly removed and deleted the large files in my repository and allowed me to sync again: http://www.zyxware.com/articles/4027/how-to-delete-files-permanently-from-your-local-and-remote-git-repositories

CD to your local working folder and run the following command:

git filter-branch -f --index-filter "git rm -rf --cached --ignore-unmatch FOLDERNAME" -- --all

replace FOLDERNAME with the file or folder you wish to remove from the given git repository.

Once this is done run the following commands to clean up the local repository:

rm -rf .git/refs/original/
git reflog expire --expire=now --all
git gc --prune=now
git gc --aggressive --prune=now

Now push all the changes to the remote repository:

git push --all --force

This will clean up the remote repository.

View more solutions

Comments

[Contango]

Do NOT run these commands unless you want to create immense pain for yourself. It deleted a lot of my original source code files. I assumed it would purge some large files from my commit history in GIT (as per the original question), however, I think this command is designed to permanently purge files from your original source code tree (big difference!). My system: Windows, VS2012, Git Source Control Provider.

[Agung Prasetyo]

Why i can't push when using git filter-branch, failed to push some refs to '[email protected]:product/myproject.git' To prevent you from losing history, non-fast-forward updates were rejected Merge the remote changes before pushing again.

[Greg Bacon]

Add the -f (or --force) option to your git push command: “Usually, the command refuses to update a remote ref that is not an ancestor of the local ref used to overwrite it. This flag disables the check. This can cause the remote repository to lose commits; use it with care.”

[Kostanos]

I used this command: git filter-branch --force --index-filter 'git rm --cached -r --ignore-unmatch oops.iso' --prune-empty --tag-name-filter cat -- --all instead of first one from your code

[Andy Hayden]

How is it different from above, why is it better?

[Kostanos]

For some reason mkljun version is not reduced git space in my case, I already had removed the files from index by using git rm --cached files. The Greg Bacon's proposition is more complete, and quite the same to this mine, but he missed the --force index for cases when you are using filter-branch for multiple times, and he wrote so much info, that my version is like resume of it.

[Roberto Tyley]

This is a wonderfully thorough answer explaining the use of git-filter-branch to remove unwanted large files from history, but it's worth noting that since Greg wrote his answer, The BFG Repo-Cleaner has been released, which is often faster and easier to use - see my answer for details.

[Tony]

@Roberto: I followed the usage instructions on the site doing a clone --mirror. When it came time to push the repo, it failed stating that I needed to pull first. I'm pretty sure there have been no commits between the time I clone and push back. If I pull, git complains that it needs a working tree inside my-repo.git. Any suggestions?

[Roberto Tyley]

@tony It's worth repeating the entire cloning & clearing procedure to see if the message asking you to pull re-occurs, but it's almost certainly because your remote server is configured to reject non-fast-forward updates (ie, it's configured to stop you from losing history - which is exactly what you want to do). You need to get that setting changed on the remote, or failing that, push the updated repo history to a brand new blank repo.

[Tony]

@RobertoTyley Thanks. I have tried it 3 different times and all resulted with the same message. So I'm also thinking that you're right about the remote server being configured to reject the non-fast-forward updates. I'll consider just pushing the updated repo to a brand new repo. Thank you!

[4Z4T4R]

After I do either of the procedures above, the remote repository (on GitHub) does NOT delete the large file. Only the local does. I force push and nada. What am I missing?

[rynop]

this also works on dirs. ... "git rm --cached -rf --ignore-unmatch path/to/dir"...

[Ehsan]

I can't just delete "pick cb14efd Remove DVD-rip" line, cause in "Remove DVD-rip" commit I did some other stuff. (in Interactive Rebase solution)

[Greg Bacon]

@Ehsan In your case, mark both commits with edit and clean up by hand in the shell.

[Weiyi]

@RobertoTyley Perfect, you save my time, thanks very much. By the way, maybe should do git push --force after your steps, otherwise the remote repo still not changed.

[Ali B]

This response helped me, except the script in the answer has a slight issue and it doesn't search in all branches form me. But the command in the link did it perfectly.

[MatrixManAtYrService]

+1 to adding git push --force. Also worth noting: force pushes may not be allowed by the remote (gitlab.com doesn't, by default. Had to "unprotect" the branch).

[kon psych]

Instead of --strip-blobs-bigger-than 100M you can also use -b 100M according to help.

[user1601201]

Not sure if BFG automatically deletes the reflog references... if not, you still need to run: git reflog expire --expire-unreachable=all as described by @Greg Bacon in his answer here. If there are still reflog entries, the data will not be removed by git gc, even with aggressive (apparently there are limits to its aggressiveness)

[Ernesto Fernandez]

Tip: If you run bfg.jar with the file declared in .gitignore, it won't be removed.

[podarok]

Much better than bfg. I was unable to clean file from a git with bfg, but this command helped

[James]

This is great. Just a note for others that you'll have to do this per branch if the large file is in multiple branches.

[matbrgz]

@Tony BFG must rewrite history to do what it does, essentially creating a whole new commit tree. This by definition mean that the commits get new sha1 hashes which is why the force push is needed, as the parent is no longer what the server expects. This is usually a GOOD thing, but in this particular case we know better.

[Syed Waqas]

@RobertoTyley: I got a general question, is BFG equally functional on Windows as much as on Linux/Mac?

[Roberto Tyley]

@WaqasShah : yes, it runs on any platform that has Java 7 or above installed. You can download Java for Windows here: java.com/en/download

[drstevok]

This really helped but I needed to use the -f option not just -rf here git rm --cached -rf --ignore-unmatch oops.iso instead of git rm --cached -r --ignore-unmatch oops.iso as per @lfender6445 below

[Richard G]

This worked for me on a local commit that I couldn't upload to GitHub. And it seemed simpler than the other solutions.

[Cameron E]

you should get this in homebrew

[oarfish]

All this does for me is create a huge .git-rewrite directory while keeping the removed files in the repo.

[MitchellK]

BFG worked an absolute charm for me. Brought a 517mb repo down to 38 Mb in just a few minutes. Nothing else worked for me prior to finding this answer.

[DaveRGP]

Undocumented issue (mostly) when given a "is repo packed" error. Use git gc on the target repo, then re-execute whatever it was you were doing with BFG. Once that was sorted worked pretty well. Could use more explicit documentation, but then I'm not the quickest learner ;p

[kristianp]

This method is waaay too slow for large repositories. It took over an hour to list the large files. Then when I go to delete files, after an hour it is only 1/3 of the way through processing the first file I want to delete.

[Nir]

Yes, its slow, but does the work... Do you know anything quicker?

[kristianp]

Haven't used it, but BFG Repo-Cleaner, as per another answer on this page.

[user189035]

How do you install that stuff? brew install bfg gives me Warning: bfg 1.12.15 is already installed ok: $ java -jar bfg.jar --strip-blobs-bigger-than 1M myrepo.git I get: Error: Unable to access jarfile bfg.jar

[Ivan Talalaev]

@DaveRGP Thanks for tip) that issue: does the repo need to be packed? definitely must be documented.

[alpha_989]

If you know the commit where you put the file in (say 35dsa2), you can replace HEAD with 35dsa2..HEAD. tree-filter is much slower than index-filter that way it wont try to checkout all the commits and rewrite them. if you use HEAD, it will try to do that.

[naitsirhc]

If your repo has any tags, you likely also want to add the flag --tag-name-filter cat to re-tag the new corresponding commits as they are rewritten, i.e., git filter-branch --index-filter 'git rm --cached --ignore-unmatch a b' --tag-name-filter cat HEAD (see this related answer)

[gabry]

+1 for BFG, I tried the "standard" method using filter-branch and it's FAR slower and in my case it didn't removed all the references to the big files...

[nruth]

Mac instructions and some other info appear in the original linked post

[eleijonmarck]

git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>' HEAD workorder right of the bat

[Ramon Vasconcelos]

Worked like a charm for me.

[stevec]

I tried this and now have "Your branch and 'origin/master' have diverged, and have 49 and 44 different commits each, respectively."

[skizzo]

This worked for me as well. Gets rid of a specific folder (in my case, one that contained files too large or a Github repo) on the repository, but keeps it on the local file system in case it exists.

[thang]

what is myrepo.git?

[Blairg23]

In your output, you state that we should run git reflog expire --expire=now --all && git gc --prune=now --aggressive

[Florian Oswald]

my favourite answer. a slight tweak to use on mac os (using gnu commands) git rev-list --objects --all \ | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \ | awk '/^blob/ {print substr($0,6)}' \ | sort --numeric-sort --key=2 \ | gnumfmt --field=2 --to=iec-i --suffix=B --padding=7 --round=nearest

[Maroun]

Is there a way to remove the "Former-commit-id" from all commits?

[ruoho ruotsi]

Worked for me! no history is left which is potentially confusing (if someone where to clone right now), make sure you have a plan to update any broken links, dependencies, etc

[Gregory Danenberg]

Master branch is protected from direct pushes. Will the procedure work as expected from private branch and PR to master?

[tedder42]

Using a free tool that has three lines of output bothers you? Better avoid open-source projects!

[Budi Mulyo]

hiks, can anyone explain more simple step.. this is to confusing to me.. :(

[Robin Manoli]

cool script with the rev-list but it didn't work for me as an alias, any idea how to do that?

[shim_mang]

@Roberto Tyley: How do I remove commits which are older than HEAD~5?

[Vasili Pascal]

Thank you, however for Mac OSX + zsh it did not work, and I modified it to a simpler version : <code> git rev-list --objects --all git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' awk '/^blob/ {print substr($0,6)}' sort --numeric-sort --key=2</code>

[eugene]

What happens to the commits that happens while we are doing git gc --prune=now --aggressive ? because it takes long time... ?

[jan-glx]

After the push you probably want to run: git fetch && git reset origin/master --soft on existing clones.

[SMPLYJR]

This works in my case. I run this on your master branch.

[Chris]

FYI BFG does not work as advertised, and filter-branch does not take that long. Running BFG 10-30 times with different branches and different configurations takes much longer.

[Noel Evans]

After running the above command, you then have to run git push --all --force to get remote's history to match the amended version you have now created locally (@stevec)

[Jossy]

Worked for me but me mindful this deletes everything after that point

[Fanchen Bao]

I tried the filter-branch methods described in the other answers, but they didn't work. After filtering, I still got file size too big error when pushing to GitHub. This solution worked, most likely because it removed the big file from ALL occurrences in ALL branches.

[AaA]

Your assumption is that file with known name need to be removed, in my case all files are called index.html (different folders) and only one of them need to be removed which I happen to know its hash

[Greg Bacon]

@AaA See the link to the blog post by Conrad Parker with the definition of git lola, also linked in the answer.

[AaA]

@GregBacon, I'm sorry, my comment is missing and I don't even remember what was the question

[Greg Bacon]

@AaA My recollection is you asked about the definitions of git lol and git lola.

[mustafabar]

I tried many other solutions, but BFS is the only one that resulted in a reduction of size "at the remote server side" and maintaining the history at the same time.

[clayg]

can you elaborate on how the "remove the large file from all commits" step worked, that was amazing!

[Kevin R.]

Thanks @clayg. I don't understand deeply the git filter-branch command, as I wrote, I just followed the GitHub documentation. What I know is that this command browses through your .git folder and find all tracks of the given file and removes it from the history.

[diman82]

How do I submit the applied changes (on my local repository) to a remote repository? Or this is not possible, and I should clone the amended repo to a new one?

[Donat]

@diman82: Best would be to make a new empty repository, set the remote repository from your cloned repo to that and push. This is common to all these answers here: You will get many new commit hashes. This is unavoidable because the commit hashes guarantee for the content and the history of a repo. The alternative way is dangerous, you could make a force push and then run gc to get rid of the files. But do not do this unless you have tested very well and you are aware of all the consequences !

[diman82]

I've already pushed (with --force option), worked well (to a cloned repository, as a precaution).

[Andrei Boyanov]

Works really well! My only question before pushing is what will happen with the closed pull requests? Are we going to loose them as the commit hashes will change ?

[davewoodhall]

Confirmed April 2021, git gc --prune=now --aggressive still does the trick !

[ahjashish]

No-one gave this answer because it does not answer the question. He wants a specific file removed from the history. Your answer nukes everything in the repo after a certain point.

[Kostas Stamos]

May also need git push origin --tags --force to remove large files from the remote in tagged releases.

[Alf Pascu]

I removed the refs/original/refs/heads/master branch created after the filter-branch using the backup+git clone steps and the remotes of the original repo were lost in the new repo, having the original repo itself as the new remote instead. I would perhaps indicate it for completeness

[Lucas]

git filter-repo --strip-blobs-bigger-than 10M worked much better on my end

[Stardust]

This is great. Want to add, for the last command, if you have a very large repo with many commits, instead of doing --all, read this answer (stackoverflow.com/a/51468389) to split up pushes (note, replace git push with git push -f). I had to do this because the pack size exceeded 2 GB trying to push everything at once. And comment two - back up everything!

[Ezh]

@mkljun, please at least remove "git push origin master --force"! First of all it is not related to the original question - author didn't ask how to edit commits and push changes to some repository. And second - this is dangerous, you really can delete a lot of files and push changes to remote repository without first check what was deleted is not a good idea.

[alper]

How does your answer differs from @Greg Bacon 's answer?

[alper]

Your solution works when I apply two times, second one after the git push --all --force , is it normal?

[Karl]

I guess it's only me that didn't realize this command will also nuke the file from the project itself, not just the git repo. Certainly worked though!

[Hammer. Wang]

This works great!

[Ng Ju Ping]

Worked for me! One question though, why is git reflog expire required?

[Gideon A.]

Thanks for the detailed answered. Solved the issue for me.

[Quantum0xE7]

why is git reflog expire required?

[Exploring]

@KevinR. you have to force push, isnt it?

[MA19]

Does this command work when working with local repo? The problem I have is that I have some large file in my local history and I cannot push to remote, so what would be my repo.git?

[Alex L]

This worked well for me. filter-repo has good documentation for more advanced cases but in mine, I just needed to get rid of big file I accidentally committed. In my case, it worked fine to duplicate the project dir, run the command in the new version, re-add the remote and push (no strictly fresh clone).

[james-see]

this should be the accepted answer now. Worked amazingly well.

[Sheldon]

Add -f after git filter-branch, if need to overwrite previous backup

[Sukombu]

The interactive rebase approach is the best one, i think. Self explaining and full control without third-party tools. Just git rm --cached file.ext did not working for me and asked for forcing. But in my case (just want to move the file to LFS) it also worked by just add an suitable .gitattributes file in the same commit.

[Williem]

this is a joke right?