Split large Git repository into many smaller ones

Question

After successfully converting an SVN repository to Git, I now have a very large Git repository that I want to break down into multiple smaller repositories and maintain history.

So, can someone help with breaking up a repo that might look like this:

MyHugeRepo/
   .git/
   DIR_A/
   DIR_B/
   DIR_1/
   DIR_2/

Into two repositories that look like this:

MyABRepo/
   .git
   DIR_A/
   DIR_B/

My12Repo/
   .git
   DIR_1/
   DIR_2/

I've tried following directions in this previous question but it doesn't really fit when trying to put multiple directories into a separate repo (Detach (move) subdirectory into separate Git repository).

Solution 1

This will setup MyABRepo; you can do My12Repo similarly of course.

git clone MyHugeRepo/ MyABRepo.tmp/
cd MyABRepo.tmp
git filter-branch --prune-empty --index-filter 'git rm --cached --ignore-unmatch DIR_1/* DIR_2/*' HEAD 

A reference to .git/refs/original/refs/heads/master remains. You can remove that up with:

cd ..
git clone MyABRepo.tmp MyABRepo

If all went well you can then remove MyABRepo.tmp.

---

If for some reason you get an error regarding .git-rewrite, you can try this:

git clone MyHugeRepo/ MyABRepo.tmp/
cd MyABRepo.tmp
git filter-branch -d /tmp/git-rewrite.tmp --prune-empty --index-filter 'git rm --cached --ignore-unmatch DIR_1/* DIR_2/*' HEAD 
cd ..
git clone MyABRepo.tmp MyABRepo

This will create and use /tmp/git-rewrite.tmp as a temporary directory, instead of .git-rewrite. Naturally, you can substitute any path you wish instead of /tmp/git-rewrite.tmp, so long as you have write permission, and the directory does not already exist.

Solution 2

You could use git filter-branch --index-filter with git rm --cached to delete the unwanted directories from clones/copies of your original repository.

For example:

trim_repo() { : trim_repo src dst dir-to-trim-out...
  : uses printf %q: needs bash, zsh, or maybe ksh
  git clone "$1" "$2" &&
  (
    cd "$2" &&
    shift 2 &&

    : mirror original branches &&
    git checkout HEAD~0 2>/dev/null &&
    d=$(printf ' %q' "$@") &&
    git for-each-ref --shell --format='
      o=%(refname:short) b=${o#origin/} &&
      if test -n "$b" && test "$b" != HEAD; then 
        git branch --force --no-track "$b" "$o"
      fi
    ' refs/remotes/origin/ | sh -e &&
    git checkout - &&
    git remote rm origin &&

    : do the filtering &&
    git filter-branch \
      --index-filter 'git rm --ignore-unmatch --cached -r -- '"$d" \
      --tag-name-filter cat \
      --prune-empty \
      -- --all
  )
}
trim_repo MyHugeRepo MyABRepo DIR_1 DIR_2
trim_repo MyHugeRepo My12Repo DIR_A DIR_B

You will need to manually delete each repository’s unneeded branches or tags (e.g. if you had a feature-x-for-AB branch, then you probably want to delete that from the “12” repository).

Solution 3

The git_split project is a simple script that does exactly what you are looking for. https://github.com/vangorra/git_split

Turn git directories into their very own repositories in their own location. No subtree funny business. This script will take an existing directory in your git repository and turn that directory into an independent repository of its own. Along the way, it will copy over the entire change history for the directory you provided.

./git_split.sh <src_repo> <src_branch> <relative_dir_path> <dest_repo>
        src_repo  - The source repo to pull from.
        src_branch - The branch of the source repo to pull from. (usually master)
        relative_dir_path   - Relative path of the directory in the source repo to split.
        dest_repo - The repo to push to.

Solution 4

Here is a ruby script that will do it. https://gist.github.com/4341033

Solution 5

Thanks for your answers but I ended up just copying the repository twice then deleting the files I didn't want from each. I am going to use the filter-branch at a later date to strip out all the commits for the deleted files since they are already version controlled elsewhere.

cp -R MyHugeRepo MyABRepo
cp -R MyHugeRepo My12Repo

cd MyABRepo/
rm -Rf DIR_1/ DIR_2/
git add -A
git commit -a

This worked for what I needed.

EDIT: Of course, the same thing was done in the My12Repo against the A and B directory. This gave me two repos with identical history up to the point I deleted the unwanted directories.

View more solutions

Comments

[Anonigan]

'git filter-branch' manpage recommends to create a fresh clone of rewritten repository instead of the last step mentioned above.

[Daenyth]

: is not a comment character in bash. You should use # instead.

[Daenyth]

This does not preserve commit history.

[Admin]

how so? I still have all the history, even for the deleted files.

[Admin]

I tried this and got an error when it was trying to delete the .git-rewrite folder at the end.

[Chris Johnsen]

@Daenyth, : is a traditional built-in command ( also specified in POSIX). It is included in bash, but it is not a comment. I specifically used it in preference to # because not all shells take # as a comment introducer in all contexts (e.g. interactive zsh without the INTERACTIVE_COMMENTS option enabled). Using : makes the whole text suitable for pasting into any interactive shell as well as saving in a script file.

[pheelicks]

Brilliant! Only solution I found that keeps all the branches intact

[kynan]

Odd, for me it stops with git remote rm origin, which always seems to return 1. Hence I replaced the && by ; for this line.

[Axel M. Garcia]

Since your requirement wasn't that repo A must pretend repo B never existed, I think this (leaving record of commits that only affected B) is an appropriate solution. Better to duplicate a little history than mangle it.

[Vertigo]

-d <path-on-another-physical-disk> worked for me and eliminated stange 'mv' failures within --tree-filter.

[bitmask]

Do you have an idea how to get the very first commit out, if it is related to an excluded path (like DIR_A, for instance)?

[unutbu]

@bitmask: Have you tried git rebase -i --root $tip (for git 1.7.12+), or this older method?

[Walter A]

Nice, $@ works for more than two dirs when needed. When finished I call git remote add origin $TARGET; git push origin master.

[Admin]

File renames history is lost, however it is how git handles renaming. Anyway, if you want to keep some directory and remove rest, there is "official" way too, with git subtree. See stackoverflow.com/questions/359424/…

[thaddeusmt]

I did not realize the full ramifications of filter-branch. For those not aware, it re-writes history, so if you plan to push the repo after you have done this, the commit hashes will be different now and it will not work.

[Tiina]

.git/refs/original/refs/heads/master what is this file?what would happen if it is remained?

[Snackoverflow]

Which history will remain in the new repository with this method? The history for all the existing files in the new repository? The history for all changes (including deleted files) in the directories DIR_A and DIR_B? What happens to the history of external file moves, e.g. from DIR_1/README.md -> DIR_A/README.md? Whatabout internal file moves such as DIR_A/MyClass.java -> DIR_1/NewNameClass.java?

[éclairevoyant]

this would be the same as the filter-branch answer above, yes? if so I assume it has the similar issue that it rewrites the entire history?