Merge two Git repositories without breaking file history
Solution 1
It turns out that the answer is much simpler if you're simply trying to glue two repositories together and make it look like it was that way all along rather than manage an external dependency. You simply need to add remotes to your old repos, merge them to your new master, move the files and folders to a subdirectory, commit the move, and repeat for all additional repos. Submodules, subtree merges, and fancy rebases are intended to solve a slightly different problem and aren't suitable for what I was trying to do.
Here's an example Powershell script to glue two repositories together:
# Assume the current directory is where we want the new repository to be created
# Create the new repository
git init
# Before we do a merge, we have to have an initial commit, so we'll make a dummy commit
git commit --allow-empty -m "Initial dummy commit"
# Add a remote for and fetch the old repo
# (the '--fetch' (or '-f') option will make git immediately fetch commits to the local repo after adding the remote)
git remote add --fetch old_a <OldA repo URL>
# Merge the files from old_a/master into new/master
git merge old_a/master --allow-unrelated-histories
# Move the old_a repo files and folders into a subdirectory so they don't collide with the other repo coming later
mkdir old_a
dir -exclude old_a | %{git mv $_.Name old_a}
# Commit the move
git commit -m "Move old_a files into subdir"
# Do the same thing for old_b
git remote add -f old_b <OldB repo URL>
git merge old_b/master --allow-unrelated-histories
mkdir old_b
dir –exclude old_a,old_b | %{git mv $_.Name old_b}
git commit -m "Move old_b files into subdir"
Obviously you could instead merge old_b into old_a (which becomes the new combined repo) if you’d rather do that – modify the script to suit.
If you want to bring over in-progress feature branches as well, use this:
# Bring over a feature branch from one of the old repos
git checkout -b feature-in-progress
git merge -s recursive -Xsubtree=old_a old_a/feature-in-progress
That's the only non-obvious part of the process - that's not a subtree merge, but rather an argument to the normal recursive merge that tells Git that we renamed the target and that helps Git line everything up correctly.
I wrote up a slightly more detailed explanation here.
Solution 2
Here's a way that doesn't rewrite any history, so all commit IDs will remain valid. The end-result is that the second repo's files will end up in a subdirectory.
-
Add the second repo as a remote:
cd firstgitrepo/ git remote add secondrepo username@servername:andsoon
-
Make sure that you've downloaded all of the secondrepo's commits:
git fetch secondrepo
-
Create a local branch from the second repo's branch:
git branch branchfromsecondrepo secondrepo/master
-
Move all its files into a subdirectory:
git checkout branchfromsecondrepo mkdir subdir/ git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} subdir/ git commit -m "Moved files to subdir/"
-
Merge the second branch into the first repo's master branch:
git checkout master git merge --allow-unrelated-histories branchfromsecondrepo
Your repository will have more than one root commit, but that shouldn't pose a problem.
Solution 3
Say you want to merge repository a
into b
(I'm assuming they're located alongside one another):
cd b
git remote add a ../a
git fetch a
git merge --allow-unrelated-histories a/master
git remote remove a
In case you want to put a
into a subdirectory do the following before the commands above:
cd a
git filter-repo --to-subdirectory-filter a
cd ..
For this you need git-filter-repo
installed (filter-branch
is discouraged).
An example of merging 2 big repositories, putting one of them into a subdirectory: https://gist.github.com/x-yuri/9890ab1079cf4357d6f269d073fd9731
More on it here.
Solution 4
A few years have passed and there are well-based up-voted solutions but I want to share mine because it was a bit different because I wanted to merge 2 remote repositories into a new one without deleting the history from the previous repositories.
-
Create a new repository in Github.
-
Download the newly created repo and add the old remote repository.
git clone https://github.com/alexbr9007/Test.git cd Test git remote add OldRepo https://github.com/alexbr9007/Django-React.git git remote -v
-
Fetch for all the files from the old repo so a new branch gets created.
git fetch OldRepo git branch -a
-
In the master branch, do a merge to combine the old repo with the newly created one.
git merge remotes/OldRepo/master --allow-unrelated-histories
Create a new folder to store all the new created content that was added from the OldRepo and move its files into this new folder.
Lastly, you can upload the files from the combined repos and safely delete the OldRepo from GitHub.
Hope this can be useful for anyone dealing with merging remote repositories.
Solution 5
I turned the solution from @Flimm this into a git alias
like this (added to my ~/.gitconfig
):
[alias]
mergeRepo = "!mergeRepo() { \
[ $# -ne 3 ] && echo \"Three parameters required, <remote URI> <new branch> <new dir>\" && exit 1; \
git remote add newRepo $1; \
git fetch newRepo; \
git branch \"$2\" newRepo/master; \
git checkout \"$2\"; \
mkdir -vp \"${GIT_PREFIX}$3\"; \
git ls-tree -z --name-only HEAD | xargs -0 -I {} git mv {} \"${GIT_PREFIX}$3\"/; \
git commit -m \"Moved files to '${GIT_PREFIX}$3'\"; \
git checkout master; git merge --allow-unrelated-histories --no-edit -s recursive -X no-renames \"$2\"; \
git branch -D \"$2\"; git remote remove newRepo; \
}; \
mergeRepo"
Related videos on Youtube
Eric Lee
Updated on March 31, 2022Comments
-
Eric Lee about 2 years
I need to merge two Git repositories into a brand new, third repository. I've found many descriptions of how to do this using a subtree merge (for example Jakub Narębski's answer on How do you merge two Git repositories?) and following those instructions mostly works, except that when I commit the subtree merge all of the files from the old repositories are recorded as new added files. I can see the commit history from the old repositories when I do
git log
, but if I dogit log <file>
it shows only one commit for that file - the subtree merge. Judging from the comments on the above answer, I'm not alone in seeing this problem but I've found no published solutions for it.Is there any way do merge repositories and leave individual file history intact?
-
Lucero over 11 yearsI'm not using Git, but in Mercurial I'd first do a convert if necessary to fix the file paths of the repos to be merged, and then force-pull one repo into the target to get the changesets, and then do a merge of the different branches. This is tested and works ;) Maybe this helps to find a solution for Git as well... compared to the subtree-merge approach I guess the convert step is different where the history is rewritten instead of just mapping a path (if I understand correctly). This then ensures a smooth merge without any special handling of file paths.
-
nacross about 10 yearsI also found this question helpful stackoverflow.com/questions/1683531/…
-
Dimitri Dewaele about 7 yearsI created a follow-up question. Might be interesting: Merge two Git repositories and keep the master history: stackoverflow.com/questions/42161910/…
-
xverges almost 7 yearsThe automated solution that worked for me was stackoverflow.com/a/30781527/239408
-
-
mholm815 about 11 yearsthis solution using
git mv
doesn't work so well. when you later use agit log
on one of the moved files you only get the commit from the move. all previous history is lost. this is becausegit mv
is reallygit rm; git add
but in one step. -
Eric Lee about 11 yearsIt's the same as any other move/rename operation in Git: from the command line you can get all of the history by doing
git log --follow
, or all of the GUI tools do that for you automatically. With a subtree merge you can't get the history for individual files, as far as I know, so this method is better. -
Jon about 10 years@EricLee When the old_b repo is merged I get a lot of merge conflicts. Is that expected? I get CONFLICT (rename/delete)
-
Eric Lee about 10 years@Jon No, if you've moved the first repo into a subdirectory first then the second repo shouldn't cause any conflicts. If it does, you've probably skipped a step somewhere or have discovered some behavior I don't know about.
-
Jon about 10 yearsI'm on Git 1.8.5 on Windows and it always causes issues when running the merge on the old_b and i'm not sure why. CONFLICT (rename/delete): DeleteFeedCommand.cs deleted in HEAD and renamed in domain/master. Version domain/master of DeleteFeedCommand.cs left in tree.
-
Jon about 10 yearsdomain/master is old_b here
-
Jon about 10 yearsI think git add -u on the conflict seems to get around it
-
Keith about 10 yearsStep 2 doesn't work for me: fatal: Not a valid object name: 'secondrepo/master'.
-
Flimm about 10 years@Keith: make sure you've added the second repo as a remote named "secondrepo", and that that repo has a branch named "master" (you can view branches on a remote repo with the command
git remote show secondrepo
) -
sksamuel about 10 yearsI had to do a fetch to bring it down as well. In between 1 and 2 I did git fetch secondrepo
-
Flimm about 10 years@monkjack: I've edited my answer to include a git fetch step. Feel free to edit the answer yourself in future.
-
DarkteK about 9 yearsThis isn't working, I think one step is missing or wrong, because I'm trying to merge like this tutorial, and all I got is just Unmerged error in many files, besides all my main files go into the sub directory folder....
-
IsmailS over 8 yearsI was merging B repo into A repo. I had a feature-in-progress branch in both repo. I wanted to pull in-progress changes of B repo to A repo. In A repo, I did
git checkout feature-in-progress
>merge master
>git merge -s recursive -Xsubtree=B old_a/feature-in-progress
. It merged properly but the newly added files inB/feature-in-progress
are missing. :( -
George over 8 yearsWhen I attempt "dir -exclude old_a | %{git mv $_.Name old_a}", I get sh.exe": dir: command not found and sh.exe": git: command not found. Using this works: ls -I old_a | xargs -I '{}' git mv '{}' old_a/
-
Mageician over 8 yearsI am using Git Bash on Windows and had the same problem as George. His command works, but it's not clear that the switches to both 'ls' and 'xargs' is a capital 'eye' and not a lower 'el'.
-
Dominique Vial about 8 yearsThis is
1
(the number One) forls
and capital 'eye' forxargs
. Thanks you for this tip! -
user3356885 about 8 yearsThis method of combining git repositories together should be carefully evaluated if you are planning on using code from new git repository for build/release process. Looks like git move operation is messing up all branches and tags on the code in newly created repository. Although you can see your tags in place but it's not usable at all.
-
rmunn about 8 yearsThis answer worked for me (or more specifically, for the colleague who just asked me how to do this). He didn't need to move any files to a subdirectory, since the two repos had different directory structures in them (C# project, merging the FooProject and BarProject repositories together -- everything was already under folders named FooProject or BarProject). Had a couple of easily-solved merge conflicts on files like
.gitignore
, but that's it. All FooProject and BarProject files are now in one repo, with full history preserved for each. -
Jeffery Utter about 8 yearsI am evaluating this option and it seems to be working quite well. One issue that I am running into is that the git logs for some files contain many more commits after merging. A file that had 4 commits in it's original repo has 400 after merging into a repo with 4000 combined commits. I'm not sure where it is picking up the other commits. Any ideas?
-
Fredrik Erlandsson almost 8 yearsFor part 5, perhaps you like to use
git merge branchfromsecondrepo -s recursive -X no-renames
so git don't tries to find renamed files. -
user5359531 almost 8 yearsAfter Step 4 and before Step 5, are you supposed to push
branchfromsecondrepo
back to its corresponding remote? Because when I went to step 5, switched back the themaster
branch, and then tried togit merge branchfromsecondrepo/master
, the changes I made in Step 4 were no longer applied, it was as if it merged straight from the remote and not from the localbranchfromsecondrepo
. I was unable to rungit merge branchfromsecondrepo
, git gave an error about it not being something that could be merged -
Flimm almost 8 years@user5359531 Nope, there's no need to push anything after step 4, no changes are made to the remote at all. If you like the result, you can push the result after all the steps in my answer have been completed. In Step 5, you should merge with
branchfromsecondrepo
, and notsecondrepo/master
orbranchfromsecondrepo/master
or any remote-tracking branch. I think you could might be confused about the differences between local branches, remote-tracking branches and branches on remotes, see this post: stackoverflow.com/a/24785777/247696 -
Mikhail Orlov over 7 yearsStep 5 requires
--allow-unrelated-histories
. -
Flimm over 7 yearsSorry @Xiong Chiamiov, I rolled back your edit because I prefer splitting a command like
git checkout -b BLA BLA2
into two commands when I'm teaching someone something. Also, your edit made the command no longer match the description of the command before it. -
Parker Coates over 7 yearsJust curious: do you really do this often enough to need an alias?
-
Eric Smith over 7 yearsOnce I've done all of those steps, can I safely delete the branch branchfromsecondrepo? Or will doing that result in loss of history?
-
Flimm over 7 years@EricSmith You can safely delete it.
git branch -d branchfromsecondrepo
won't delete the branch unless it's safe to do so. (The-D
flag forces a deletion, but-d
is safe.) -
Fredrik Erlandsson over 7 yearsNo I don't but never remember how to do it so an alias is just a way for me to remember it.
-
EML about 7 years+1, but note that if you have a .gitignore in both the source repos then you will get a conflict on the second merge, which must be fixed. Note also that someone else edited your answer to add
--allow-unrelated-histories
to your merge command. This is new, and isn't in your blog, and should probably be reverted... -
Matthew Wise about 7 yearsHad to do this and your steps above worked great, thanks! Just wondering if it is safe to remove the references to the old remotes at the end and indeed whether the old repositories can be nuked, or whether the history in the merged repo still points at the old repositories?
-
quetzalcoatl almost 7 yearsYeah.. but try changing computers and forgetting to move your aliases ;)
-
neowulf33 almost 7 yearsWhat's the value of
$GIT_PREFIX
? -
Fredrik Erlandsson almost 7 yearsgithub.com/git/git/blob/… 'GIT_PREFIX' is set as returned by running 'git rev-parse --show-prefix' from the original current directory. See linkgit:git-rev-parse[1].
-
xverges almost 7 yearsstackoverflow.com/a/30781527/239408 provides another automated approach, based on bash
-
Patrick Beard almost 7 yearsI'm using zsh rather than bash, and v2.13.0 of git. No matter what I've tried, I haven't been able to get
git filter-branch --index-filter
to work. Typically I get an error message that the .new index file doesn't exist. Does that ring any bells? -
Andrey Izman almost 7 years@PatrickBeard I don't know zsh, you can create separated file
git-add-repo.sh
with function above, at the end of the file put this linegit-add-repo "$@"
. After that you can use it from zsh likecd current/git/package
andbash path/to/git-add-repo.sh https://github.com/example/example dir/to/save
-
Patrick Beard almost 7 yearsThe problem was discussed here: stackoverflow.com/questions/7798142/…
mv "$GIT_INDEX_FILE.new" "$GIT_INDEX_FILE"
fails sometimes, so you have to add anif test
. -
Devplex over 6 yearsNote that the
--allow-unrelated-histories
option to git merge in step 5 is relatively new. Git 2.7 (the default version in Ubuntu 16.04) doesn't know it so I installed Git 2.13 from the git-core PPA. The procedure works great! -
Flimm over 6 years@MartijnHeemels For older version of Git, just omit
--allow-unrelated-histories
. See the history of this answer post. -
hussfelt over 6 yearsOn osx/macos the
dir -exclude old_a | %{git mv $_.Name old_a}
could bels | grep -v 'old_a' | xargs -I '{}' git mv '{}' old_a
-
axd over 6 yearsdoesn't always work. when merging multiple projects that have the same directory layout, there will be a massive amount of conflicts due to the merge attempt of the next subproject following a
git mv
. -
acumartini about 6 yearsIn a bash context I prefer:
find . -maxdepth 1 -not -name ".git" -and -not -name 'old_a' -exec git mv {} old_a/ \;
-
C S almost 6 yearsGreat answer, helped me out. :) . For the last step, what is the reason to do a
merge
rather than arebase
? I didgit rebase branchfromsecondrepo
to have the 1st repo start off the history of the second repo and I would imagine that's more preferable in this type of situation. -
Jeremy over 5 years
dir --ignore old_a
not-e
-
Lucademicus over 5 yearsStep 4 does not work in Windows (in Powershell), xargs command needs to be substituted. Anyone have an idea how to rewrite Step 4 for Windows? @Flimm?
-
Flimm over 5 years@Lucademicus I saw your suggested edit. Initially I approved it, but then I rolled back, as it is not strictly equivalent to the bash equivalent. I'm not sure if it handles Git ignored files correctly.
-
Célia Doolaeghe over 4 yearsThis is the only solution that worked for me to preserve git history. Don't forget to remove the remote link to old repo with
git remote rm OldRepo
. -
Mason Freed about 4 yearsI would not use this method! I tried the script, naively and verbatim (I can only blame myself for that part), and it clobbered my local git repo. The history looked mostly right, but doing a git push back to Github resulted in the dreaded "RPC failed; curl 55 SSL_write() returned SYSCALL, errno = 32" error. I tried to repair it, but it was irreparably broken. I ended up having to reconstruct things in a new local repo.
-
Andrey Izman about 4 years@MasonFreed this script creates a new git history with mix of both repos, so it can't be pushed to old repo, it require to create a new one or push with force key, means it rewrite your repo on server
-
Peter about 4 yearsAs mholm815 said, after merging the repos simply using
git log <file>
will stop on the moving commit. Eric Lee suggested usinggit log --follow <file>
but it didn't work for me (git 2.25). Instead, it showed no commits at all. To get full history for a particular file I usegit log -m --follow <file>
. -
Yuri L about 4 yearsStep 3 failed for me with syntax error. Semi-colons are missing. Fix
git filter-branch --prune-empty --tree-filter ' if [ ! -e my/new/subdir ]; then mkdir -p my/new/subdir; git ls-tree --name-only $GIT_COMMIT | xargs -I files mv files my/new/subdir; fi'
-
Stephen Turner almost 4 yearsGit documentation recommends not rebasing... git-scm.com/book/en/v2/Git-Branching-Rebasing#_rebase_peril
-
code4meow over 3 yearsI can't upvote this enough. A perfectly simple, successful, sensible solution. Thank you! And thank you @Harubiyori for the final touch.
-
Bob over 3 yearsIs it possible to do it without merge conflicts?
-
x-yuri over 3 years@Mikhail Yes, it is possible, do you see merge conflicts in the gist? If you run into merge conflicts, that means you have e.g. file
a/b/c
in both repositories. Either rename files before merge, or merge into a subdirectory, or resolve the conflicts. -
Bob over 3 yearsok. thank you. resolve the conflicts it is
-
Robert Massaioli over 2 yearsTo make the move easier, I would just
mkdir -p old_a && git mv -k * old_a
. Much simpler. -
Tim MB over 2 yearsThis worked great, except I'm missing the submodules now
-
Ishmaeel over 2 yearsthis is the perfect solution for preserving file histories without becoming dependent on
--follow
, thank you! -
Sta_Doc over 2 yearsThis solution works for me. All the previous commit messages from the other repo are kept!!
-
Astariul about 2 yearsFollowing the comment of @RobertMassaioli, if you need to move hidden files/folders as well, you can do
mkdir -p old_a && git mv -k .[!.]* old_a
-
erik-stengel about 2 yearsDo I have to execute this in PowerShell or in GitBash?