How to rebase after git-subtree add?

Question

I'm trying to learn the new git-subtree command which was added in Git 1.7.11. I seem to lose ability to rebase after I add a subtree. I have the primary repository with README file and a library repository which also has a README file. I add it to lib directory with subtree add:

$ git subtree add -P lib/mylib myliborigin master

This works fine, but now the history looks like this:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
* b99d55b Add readme
* 020e372 Initial

Now when I want to rebase my repo against origin/master and it fails because the squash commit is applied directly against its parent commit which does not apply, because it is applied to the root of the repo and not the prefix I gave it to it when adding the subtree.

The reason for this is pretty clear if I look at the squash commit. There is no information about the prefix. It is just the original mylib commits squashed together. Only the next merge commit knows anything about it, but rebase does not take it to account here.

Are there any workarounds (besides never rebasing over the subtree commits)?

Solution 1

This works in simple cases:

git rebase --preserve-merges master

Thanks to @Techlive Zheng in the comments.

---

You may see

fatal: refusing to merge unrelated histories
Error redoing merge a95986e...

Which means that git failed to automatically apply your subtree. This puts you in the situation @ericpeters described in his answer. Solution:

Re-add your subtree (use the same command you originally used):

git subtree add -P lib lib-origin master

Continue the rebase:

git rebase --continue

And you're all set!

---

If you're wondering if it worked successfully, you can compare with your original version after rebasing to make sure you didn't change anything:

git diff <ref-before-rebase> <ref-after-rebase> -- .

(the -- . at the end instructs git to only diff the files in your current directory).

---

If all else fails and you don't care about preserving the commits themselves, you can simply git cherry-pick the original subtree commit.

The commit message will look something like Add 'lib/' from commit '9767e6...' -- that's the one you want.

Solution 2

This is an old question, but I just had the same problem with my repo, and I finally found a complete solution, which (hopefully) preserves all the subtree metadata.

Suppose we had this commit tree:

B   (master) Add README.md
|     
A            Initial commit

and we forked a feature branch with a subtree residing in lib/:

git remote add -f githublib https://github.com/lib/lib.git
git subtree add --prefix lib/ githublib master --squash

It creates a merge commit D with two parents: our current master (B), and an unrelated commit F with the squashed history of the external repo. This commit also contains some git subtree metadata in its commit message (namely, git-subtree-dir and git-subtree-split).

   D     (feature) Merged commit 'F' as 'lib/'
  / \    
 /   F             Squashed 'lib/' content from GGGGGG
B        (master)  Add README.md
|     
A                  Initial commit

Later, we add some commits to both branches independently.

   E     (feature) Remove .gitignore from lib/
C  |     (master)  Add LICENSE.md
|  D               Merged commit 'F' as 'lib/'
| / \    
|/   F             Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Now we want to rebase feature onto master. Here's how:

1. Cherry-pick the commits from feature one by one to create a new copy of the feature branch on top of master.

git branch -f feature C
git checkout feature
git cherry-pick D E

E'       (feature) Remove .gitignore from lib/
|
D'                 Merged commit 'F' as 'lib/'
|
|  E               Remove .gitignore from lib/
C  |     (master)  Add LICENSE.md
|  D               Merged commit 'F' as 'lib/'
| / \    
|/   F             Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Now we have the equivalent of a rebase, but we've lost all information about the external repo, required for git subtree. To restore it:

2. Add the missing parent link as a graft, and rewrite the history of feature to make it permanent.

git checkout feature
git replace --graft D' C F
git filter-branch --tag-name-filter cat -- master..

And now we get a picture exactly equivalent to the one with started with. The old commits D and E are still out there, but they can be garbage-collected later.

E'       (feature) Remove .gitignore from lib/
|
D'                 Merged commit 'F' as 'lib/'
|\
| \                
C  \     (master)  Add LICENSE.md
|   \              
|    \   
|     F            Squashed 'lib/' content from GGGGGG
B                  Add README.md
|     
A                  Initial commit

Warning: This rewrites the history of feature, so be wary of publishing it if anybody else collaborates with you on this branch. However, since you wanted to make a rebase in the first place, you are probably aware of that :-)

Solution 3

This isn't a solution, but its the current work around I use...

Using your initial example:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d
* b99d55b Add readme
* 020e372 Initial

Rebase interactively to the 2nd commit before the subtree add:

$ git rebase -i 020e372

Delete the two subtree entries & mark edit for the prior commit:

e b99d55b Add readme

Save file/close, then when it gets to the "Add readme" commit, run the amend command:

$ git commit --amend

Then re-add your new subtree:

$ git subtree add -P lib/mylib myliborigin master

Continue the rebase:

$ git rebase --continue

Your branch should then be rebased off of master, and the subtree will be "normal" with the Squash + the Merge intact:

*   22c1fe6 (HEAD, master) Merge commit 'b6e698d9f4985825efa06dfdd7bba8d2930cd40e' as 'lib/mylib' - 
|\                                                                                                                
| * b6e698d Squashed 'lib/mylib/' content from commit d7dbd3d

Solution 4

Git 2.24.0 (released 2019-11-04) added support for git rebase --rebase-merges --strategy [strategy]. So now if you run git rebase --rebase-merges --strategy subtree [branch] when your current branch contains a subtree merge, it will Just Work now.

For my project, I've decided I might not use git subtree add, but instead throw away the second parent of the merge commit using git replace --edit. I've also used the Git book v1's obsolete "subtree" tutorial, which does the same thing but is tedious.

Solution 5

Apparently this is expected behaviour (for some perverse definition of "expected behaviour.") See: http://git.661346.n2.nabble.com/subtree-merges-lose-prefix-after-rebase-td7332850.html.

Not that this is much help to anyone. I'd love to find a workaround for this too.

View more solutions

Comments

[Vinnie Falco]

This didn't help. I still lose the prefix.

[cowbert]

What if I have the scenario where I want to squash later commits made to the subproject that are showing up in the superproject's history. For example: commit 597df58 (HEAD, master) A commit to superproject commit 9cd82bd A commit to subproject commit 059ffbe A commit to subproject commit af5209e Merge commit 'da151a...' as 'subproject' commit da151a3 Squashed 'subproject/' content from commit 4d455fb commit b10f7c9 initial commit to superproject I want to squash 9cd82bd 059ffbe with af5209e because subproject's history has already been pushed to its own repo using git subtree push...

[ericpeters]

not sure, I think I'd need a better look of what the project layout looked like, make sure to keep the terminology consistent, there are subtrees and there are submodules, but no such thing as a subproject :)

[cowbert]

well the subproject is just the result of subtree add --prefix=subproject repoB branch

[Anthony Mastrean]

Hopefully, you haven't destroyed the other repository because you thought everything would be fine later :(

[Bernhard Döbler]

What are typical values for new_place, start and end when I want to rebase a feature branch to master

[Kevin Cooper]

This certainly works, but it's a bit hacky. See my answer below for a simpler method to achieve the same result.

[Mike-E]

When you say "re-add", is the implication to also remove first? I am getting a prefix already exists message when doing the above git subtree add command.

[Kevin Cooper]

@Mike-EEE in the situation described I believe git has failed to add it automatically, so by "re-add" I mean add it yourself manually. It should not exist at that point due to the rebase.

[Mike-E]

Thanks for the context and the reply, Kevin. I have spent the day being bamboozled by this topic and my own personal plight, which derives from a seemingly innocuous git subtree pull on a remote commit comprised of a single file change that results in dozens of conflicts and errors in the parent repo/branch that only differs from said commit. In any case, I was able to get what I was looking for here: git-memo.readthedocs.io/en/latest/subtree.html Perhaps it will assist someone else.

[towo]

For the relatively simple case of "I forgot to pull some changes before adding the subtree and want to rebase", this works just as expected - thank you!

[Jimmy He]

A little late to the party, but this solution worked perfectly, thanks Kevin! I also got a prefix already exists message and it was because it contained a generated file (*.pyc) that is not version-controlled and prevented removal of the folder. Manually deleting the folder before doing git subtree add ... did the trick. Another note, if your original git subtree add ... command had a --squash, remember to include it here also so that the rebased version will look exactly the same.

[Tim Harper]

I really feel like this anti-feature needs to be advertised more clearly. The behavior is completely perplexing and everyone should know about it going in to subtrees, and yet, the documentation does not highlight it. It took me a while to find this stackoverflow article.

[Tim Harper]

In my first attempt using it, it does not "Just Work", at all. Git rebase blows up with fatal: refusing to merge unrelated histories if I try and rebase a branch with subtree commits on top of the latest master/main branch.

[nhooyr]

Above link is broken now. I believe marc.info/?l=git&m=133062667420408 is identical to what was linked.