How do I remove the old history from a git repository?

Question

I'm afraid I couldn't find anything quite like this particular scenario.

I have a git repository with a lot of history: 500+ branches, 500+ tags, going back to mid-2007. It contains ~19,500 commits. We'd like to remove all of the history before Jan 1, 2010, to make it smaller and easier to deal with (we would keep a complete copy of the history in an archive repository).

I know the commit that I want to have become the root of the new repository. I can't, however, figure out the correct git mojo to truncate the repo to start with that commit. I'm guessing some variant of

git filter-branch

involving grafts would be necessary; it might also be necessary to treat each of the 200+ branches we want to keep separately and then patch the repo back together (something I do know how to do).

Has anyone ever done something like this? I've got git 1.7.2.3 if that matters.

Solution 1

You can create a graft of the parent of your new root commit to no parent (or to an empty commit, e.g. the real root commit of your repository). E.g. echo "<NEW-ROOT-SHA1>" > .git/info/grafts

After creating the graft, it takes effect right away; you should be able to look at git log and see that the unwanted old commits have gone away:

$ echo 4a46bc886318679d8b15e05aea40b83ff6c3bd47 > .git/info/grafts
$ git log --decorate | tail --lines=11
commit cb3da2d4d8c3378919844b29e815bfd5fdc0210c
Author: Your Name <[email protected]>
Date:   Fri May 24 14:04:10 2013 +0200

    Another message
 
commit 4a46bc886318679d8b15e05aea40b83ff6c3bd47 (grafted)
Author: Your Name <[email protected]>
Date:   Thu May 23 22:27:48 2013 +0200

    Some message

If all looks as intended, you can utilize git filter-branch -- --all to make it permanent.

BEWARE: after doing the filter-branch step, all commit ids will have changed, so anybody using the old repo must never merge with anyone using the new repo.

Solution 2

Maybe it's too late to post a reply, but as this page is the first Google's result, it may still be helpful.

If you want to free some space in your git repo, but do not want to rebuild all your commits (rebase or graft), and still be able to push/pull/merge from people who has the full repo, you may use the git clone shallow clone (--depth parameter).

; Clone the original repo into limitedRepo
git clone file:///path_to/originalRepo limitedRepo --depth=10

; Remove the original repo, to free up some space
rm -rf originalRepo
cd limitedRepo
git remote rm origin

You may be able to shallow your existing repo, by following these steps:

; Shallow to last 5 commits
git rev-parse HEAD~5 > .git/shallow

; Manually remove all other branches, tags and remotes that refers to old commits

; Prune unreachable objects
git fsck --unreachable ; Will show you the list of what will be deleted
git gc --prune=now     ; Will actually delete your data

How to remove all git local tags?

Ps: Older versions of git didn't support clone/push/pull from/to shallow repos.

Solution 3

This method is easy to understand and works fine. The argument to the script ($1) is a reference (tag, hash, ...) to the commit starting from which you want to keep your history.

#!/bin/bash
git checkout --orphan temp $1 # create a new branch without parent history
git commit -m "Truncated history" # create a first commit on this branch
git rebase --onto temp $1 master # now rebase the part of master branch that we want to keep onto this branch
git branch -D temp # delete the temp branch

# The following 2 commands are optional - they keep your git repo in good shape.
git prune --progress # delete all the objects w/o references
git gc --aggressive # aggressively collect garbage; may take a lot of time on large repos

NOTE that old tags will still remain present; so you might need to remove them manually

remark: I know this is almost the same aswer as @yoyodin, but there are some important extra commands and informations here. I tried to edit the answer, but since it is a substantial change to @yoyodin's answer, my edit was rejected, so here's the information!

Solution 4

Try this method How to truncate git history :

#!/bin/bash
git checkout --orphan temp $1
git commit -m "Truncated history"
git rebase --onto temp $1 master
git branch -D temp

Here $1 is SHA-1 of the commit you want to keep and the script will create new branch that contains all commits between $1 and master and all the older history is dropped. Note that this simple script assumes that you do not have existing branch called temp. Also note that this script does not clear the git data for old history. Run git gc --prune=all && git repack -a -f -F -d after you've verified that you truly want to lose all history. You may also need rebase --preserve-merges but be warned that the git implementation of that feature is not perfect. Inspect the results manually if you use that.

Solution 5

As an alternative to rewriting history, consider using git replace as in this article from the Pro Git book. The example discussed involves replacing a parent commit to simulate the beginning of a tree, while still keeping the full history as a separate branch for safekeeping.

View more solutions

Comments

[seanf]

Works for me, except I had to work around the lack of "git checkout --orphan" on my version of git: bogdan.org.ua/2011/03/28/…

[ebneter]

Yes, I think you could probably do what we wanted with that, if you nuked the separate full history branch as well. (We were trying to shrink the repository.)

[aanno]

Well, after creating a '.git/info/grafts' file and filter-branch, I still needed a 'git clone --no-local --no-hardlinks' copy (make all your local tracking branches before that). Simply removing '.git/info/grafts' does not do the trick!

[aanno]

You probably want to cross-check stackoverflow.com/questions/7654822/… when you want to shrink your repository size.

[Exectron]

I tried this, but got merge conflicts in the rebase step. Strange--I wasn't expecting that merge conflicts could be possible in these circumstances.

[Exectron]

I had to do git filter-branch --tag-name-filter cat -- --all to update tags. But I've also got older tags pointing to the old history that I want to delete. How can I get rid of all those old tags? If I don't delete them, then the older history doesn't disappear and I can still see it with gitk --all.

[Marius Gedminas]

"Just create a graft of the parent of your new root commit to no parent" needs some elaboration. I tried that and failed to figure out the syntax for "no parent". Manual page claims a parent commit ID is required; using all zeroes just gives me an error.

[friederbluemle]

Use git commit --allow-empty -m "Truncate history" if the commit you checked out does not contain any files.

[friederbluemle]

In case anyone else was wondering how exactly it works, it's pretty easy: echo "<NEW-ROOT-HASH>" > .git/info/grafts

[Mike S]

Can someone explain what this means? "Just create a graft of the parent of your new root commit to no parent (or to an empty commit, eg. the real root commit of your repo)."

[Charles Martin]

I agree, explaining what a graft is would be more than useful

[rustyx]

Didn't work for me. Created a mess in history with both old and new commit IDs.

[rustyx]

How do I push this back to the remote master? When I do that I end up with both old and new history.

[Wooble]

This doesn't seem to actually remove old commits; they can bee seen in git-log and checked out.

[Jake88]

The force option actually deleted the branch for me. git filter-branch -f -- --all

[ThorSummoner]

Quoted from the linked wiki page on grafts. "As of Git 1.6.5, the more flexible git replace has been added, which allows you to replace any object with any other object, and tracks the associations via refs which can be pushed and pulled between repos." So this answer might be out of date for current versions of git.

[ThorSummoner]

I was discouraged by the answer being off-site; but it does link to the GitScm site and the tutorial that it links to is very well written and seems directly to the point of the OP's question.

[Jeff Bowman]

@ThorSummoner Sorry about that! I'll develop the answer a little more fully on-site

[Chris Maes]

that will work to remove ALL history, but not for what he asked: keep history since january 2010

[datUser]

Does this method disassociate previous tags with commits? It seemed to scramble some tags for me...

[Mitch]

Unfortunately this is not an alternative to rewriting history. There is a confusing sentence in the beginning of the article that probably gave this impression. Could that be removed from this answer? You'll see in the article that the author does rewrite the history of the truncated branch, but proposes a way of reattaching the legacy "history" branch using git replace. I believe this was corrected on another question where you posted this answer.

[wizzwizz4]

+1 This is the correct answer for newer versions of Git. (Oh, and please come back to PPCG!)

[user5359531]

What is 'temp' supposed to be? What are you supposed to pass as an argument for this? Is there an example of what these commands are supposed to look like when you actually run them? Thanks.

[user5359531]

I appreciate the explanations given for the git prune and git gc commands. Is there an explanation for the rest of the commands in the script? As it stands, it is not clear what arguments are being passed to it and what each command is doing. Thanks.

[Chris Maes]

@user5359531 thanks for your remark, I added some more comments for each command. Hope this helps.

[YPCrumble]

@ChrisMaes is git prune --progress for an older version of git? Per the docs, "In most cases, users will not need to call git prune directly, but should instead call git gc, which handles pruning along with many other housekeeping tasks."

[Chris Maes]

@ypcrumble. I don't know the exact history of those features... But note that the last commands are optional. Git GC should run automatically after a while...

[Colin]

It looks like you need at least git 1.9 for this to work. I'm not sure the exact version though, because I just went to 2.8 and it worked like a charm.

[koppor]

A discussion of git replace versus git graft is made at stackoverflow.com/q/6800692/873282

[Chris Nolet]

I believe $1 is the commit hash. (There are more details provided in the linked article).

[Warpzit]

Merge conflicts all over the place... not very usefull

[Chris Maes]

@Warpzit: at what step did you encounter those merge conflicts? It is quite strange...

[Warpzit]

@ChrisMaes at the rebase step. Kinda frustrating as this seems like the easiest solution.

[Chris Maes]

are you sure that $1 is a direct ancestor of the commit you were on when you started the script? $1 should be on the master branch (and supposing you want to purge the master branch)? Did none of the previous steps give you errors?

[Warpzit]

@ChrisMaes I'm sure it was master, I changed to a point further ahead but same issue!

[Chris Maes]

Let us continue this discussion in chat.

[luis19mx]

This was by far te easiest solution, and it's not necessary to put it on a bash file

[clapas]

Point number 1. made the difference for me. Cheers

[dvdchr]

@Warpzit, did you ever manage to find out why the conflict happened? I am also experiencing merge conflicts when rebasing..

[Warpzit]

@dvdchr Nope, but we removed old large files from history and other tweaks instead of removing whole history. Also the biggest issue was the build server which we changed to use shallow clone.

[Omn]

@Trogdor The answer should say cd limitedRepo since that is where you need to remove the reference to a non-existence origin. I've submitted an edit.

[piojo]

For reference, this was very slow (slower than the solution with graft/filter-branch), and the procedure kept failing because it needed around 60 GB of disk space, which I didn't have. This solution may work for smaller repositories, though.

[Jez]

When I try to push this shallow clone to a new repo (which I want to do because I want to get rid of my repo's history and start a new repo with a much smaller history) I get an error from Gitlab that a shallow update is not allowed. There needs to be a way to turn a shallow clone into a normal repo without restoring all of the extra history again.

[wl2776]

Does not work. git log after creating .git/info/grafts still shows initial commmit.

[DanCat]

This definitely does NOT work any longer $ git replace --convert-graft-file hint: Support for <GIT_DIR>/info/grafts is deprecated hint: and will be removed in a future Git version. hint: hint: Please use "git replace --convert-graft-file" hint: to convert the grafts into replace refs. hint: hint: Turn this message off by running hint: "git config advice.graftFileDeprecated false" And it does not appear git replace --convert-graft-file has the desired effect either.

[Scott Wiedemann]

After several attempts, I always get merge conflicts during rebase. Removing all tags did not seem to help. git version: 2.19. Does anyone know why merge conflicts even occur?

[Chris Maes]

@ScottWiedemann the merge conflicts might arise if you have a complicated history with merges. The conflicts probably arise when doing git rebase --onto temp $1 master

[ZachB]

@CraigMcQueen try using git rebase -p --onto temp $1 master (with the -p). That preserves merge commits and should avoid merge conflicts. Otherwise rebase tries to flatten merge commits.

[Nobody]

@Jez That would be the other top voted answer. This answer isn't for you if you want to permanently get rid of the history. It's for working with huge histories.

[leonbloy]

@Warpzit I got rid of merge conflicts by adding -p to the rebase command, as suggested in other answer

[Micros]

What if you want to keep a few hundred commits out of thousands? Calculating the depth can become tricky. I like the clone approach, but is there a way to target an old commit hash as initial instead of a depth number?

[Micros]

To answer my own question: git clone file:///Users/me/Projects/myProject myClonedProject --shallow-since=2016-09-02 Works like a charm!

[Ed'ka]

@Jez you can convert your shallow repo into normal one by running git filter-branch -- --all. This will change all hashes in it but after that you will be able to push it to a new repo

[Justas]

Just wanted to say thanks as it helped me in my scenario even though this might not be the right answer to the question

[bam]

@Jez Set receive.shallowupdate option for a new repo to be able to push shallow clone to it: stackoverflow.com/a/33086124/1063363

[DrStrangepork]

I followed this exactly, and all I got was the same history as before with a new branch starting at the commit I wanted to prune to with all the same history as before. No history was removed.

[danny]

Support for <GIT_DIR>/info/grafts is deprecated and will be removed in a future Git version.

[kap]

This is a nice solution. All changes remain in history, though. To remove that use e.g. this script: gist.github.com/ymollard/3f642ebda433a7cb8bd5

[Ed Randall]

fatal: Server does not support --shallow-since :(

[Joel AZEMAR]

I've had to read quite a bit to understand how to shrink my repository, indeed, it turns out that git replace is the way to go, please consider reading stackoverflow.com/questions/6800692/… I've done it through git replace and it works just fine.

[Joel AZEMAR]

Please consider using git replace instead. See stackoverflow.com/questions/6800692/…

[SherylHohman]

If you want to truncate history based on a specified date , you can use the option --shallow-since=<date> to "Create a shallow clone with a history after the specified time.", in place of --depth <depth> which "Creates a shallow clone with a history truncated to the specified number of commits."

[12431234123412341234123]

You may need the --no-single-branch option. Otherwise you lose all your other branches.

[Dmitri Zagidulin]

This answer is incredibly useful; big huge thanks to Mikko.

[mevsme]

I get $ git rebase --onto temp $1 master First, rewinding head to replay your work on top of it... Fast-forwarded temp to temp. $ git branch -D temp error: Cannot delete branch 'temp' checked out at '/home/user/Public/newspapa'

[Jason]

I got conflicts too. But a force push (git push -f) seems to deal with the problem.

[Chris Maes]

@Jason this is completely normal, since the history was rewritten.

[mochsner]

If you're reading this & running from Windows bash, format file path with drive letter included --> file:///c/Users/Marc\ Ochsner/...