Why does git-rebase give me merge conflicts when all I'm doing is squashing commits?

Question

We have a Git repository with over 400 commits, the first couple dozen of which were a lot of trial-and-error. We want to clean up these commits by squashing many down into a single commit. Naturally, git-rebase seems the way to go. My problem is that it ends up with merge conflicts, and these conflicts are not easy to resolve. I don't understand why there should be any conflicts at all, since I'm just squashing commits (not deleting or rearranging). Very likely, this demonstrates that I'm not completely understanding how git-rebase does its squashes.

Here's a modified version of the scripts I'm using:

---

repo_squash.sh (this is the script that is actually run):

---

rm -rf repo_squash
git clone repo repo_squash
cd repo_squash/
GIT_EDITOR=../repo_squash_helper.sh git rebase --strategy theirs -i bd6a09a484b8230d0810e6689cf08a24f26f287a

---

repo_squash_helper.sh (this script is used only by repo_squash.sh):

---

if grep -q "pick " $1
then
#  cp $1 ../repo_squash_history.txt
#  emacs -nw $1
  sed -f ../repo_squash_list.txt < $1 > $1.tmp
  mv $1.tmp $1
else
  if grep -q "initial import" $1
  then
    cp ../repo_squash_new_message1.txt $1
  elif grep -q "fixing bad import" $1
  then
    cp ../repo_squash_new_message2.txt $1
  else
    emacs -nw $1
  fi
fi

---

repo_squash_list.txt: (this file is used only by repo_squash_helper.sh)

---

# Initial import
s/pick \(251a190\)/squash \1/g
# Leaving "Needed subdir" for now
# Fixing bad import
s/pick \(46c41d1\)/squash \1/g
s/pick \(5d7agf2\)/squash \1/g
s/pick \(3da63ed\)/squash \1/g

---

I'll leave the "new message" contents to your imagination. Initially, I did this without the "--strategy theirs" option (i.e., using the default strategy, which if I understand the documentation correctly is recursive, but I'm not sure which recursive strategy is used), and it also didn't work. Also, I should point out that, using the commented out code in repo_squash_helper.sh, I saved off the original file that the sed script works on and ran the sed script against it to make sure it was doing what I wanted it to do (it was). Again, I don't even know why there would be a conflict, so it wouldn't seem to matter so much which strategy is used. Any advice or insight would be helpful, but mostly I just want to get this squashing working.

Updated with extra information from discussion with Jefromi:

Before working on our massive "real" repository, I used similar scripts on a test repository. It was a very simple repository and the test worked cleanly.

The message I get when it fails is:

Finished one cherry-pick.
# Not currently on any branch.
nothing to commit (working directory clean)
Could not apply 66c45e2... Needed subdir

This is the first pick after the first squash commit. Running git status yields a clean working directory. If I then do a git rebase --continue, I get a very similar message after a few more commits. If I then do it again, I get another very similar message after a couple dozen commits. If I do it yet again, this time it goes through about a hundred commits, and yields this message:

Automatic cherry-pick failed.  After resolving the conflicts,
mark the corrected paths with 'git add <paths>', and
run 'git rebase --continue'
Could not apply f1de3bc... Incremental

If I then run git status, I get:

# Not currently on any branch.
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
# modified:   repo/file_A.cpp
# modified:   repo/file_B.cpp
#
# Unmerged paths:
#   (use "git reset HEAD <file>..." to unstage)
#   (use "git add/rm <file>..." as appropriate to mark resolution)
#
# both modified:      repo/file_X.cpp
#
# Changed but not updated:
#   (use "git add/rm <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
# deleted:    repo/file_Z.imp

The "both modified" bit sounds weird to me, since this was just the result of a pick. It's also worth noting that if I look at the "conflict", it boils down to a single line with one version beginning it with a [tab] character, and the other one with four spaces. This sounded like it might be an issue with how I've set up my config file, but there's nothing of the sort in it. (I did note that core.ignorecase is set to true, but evidently git-clone did that automatically. I'm not completely surprised by that considering that the original source was on a Windows machine.)

If I manually fix file_X.cpp, it then fails shortly afterward with another conflict, this time between a file (CMakeLists.txt) that one version thinks should exist and one version thinks shouldn't. If I fix this conflict by saying I do want this file (which I do), a few commits later I get another conflict (in this same file) where now there's some rather non-trivial changes. It's still only about 25% of the way through the conflicts.

I should also point out, since this might be very important, that this project started out in an svn repository. That initial history very likely was imported from that svn repository.

Update #2:

On a lark (influenced by Jefromi's comments), I decided to do the change my repo_squash.sh to be:

rm -rf repo_squash
git clone repo repo_squash
cd repo_squash/
git rebase --strategy theirs -i bd6a09a484b8230d0810e6689cf08a24f26f287a

And then, I just accepted the original entries, as is. I.e., the "rebase" shouldn't have changed a thing. It ended up with the same results describe previously.

Update #3:

Alternatively, if I omit the strategy and replace the last command with:

git rebase -i bd6a09a484b8230d0810e6689cf08a24f26f287a

I no longer get the "nothing to commit" rebase problems, but I'm still left with the other conflicts.

Update with toy repository that recreates problem:

test_squash.sh (this is the file you actually run):

#========================================================
# Initialize directories
#========================================================
rm -rf test_squash/ test_squash_clone/
mkdir -p test_squash
mkdir -p test_squash_clone
#========================================================

#========================================================
# Create repository with history
#========================================================
cd test_squash/
git init
echo "README">README
git add README
git commit -m"Initial commit: can't easily access for rebasing"
echo "Line 1">test_file.txt
git add test_file.txt
git commit -m"Created single line file"
echo "Line 2">>test_file.txt 
git add test_file.txt 
git commit -m"Meant for it to be two lines"
git checkout -b dev
echo Meaningful code>new_file.txt
git add new_file.txt 
git commit -m"Meaningful commit"
git checkout master
echo Conflicting meaningful code>new_file.txt
git add new_file.txt 
git commit -m"Conflicting meaningful commit"
# This will conflict
git merge dev
# Fixes conflict
echo Merged meaningful code>new_file.txt
git add new_file.txt
git commit -m"Merged dev with master"
cd ..

#========================================================
# Save off a clone of the repository prior to squashing
#========================================================
git clone test_squash test_squash_clone
#========================================================

#========================================================
# Do the squash
#========================================================
cd test_squash
GIT_EDITOR=../test_squash_helper.sh git rebase -i HEAD@{7}
#========================================================

#========================================================
# Show the results
#========================================================
git log
git gc
git reflog
#========================================================

test_squash_helper.sh (used by test_sqash.sh):

# If the file has the phrase "pick " in it, assume it's the log file
if grep -q "pick " $1
then
  sed -e "s/pick \(.*\) \(Meant for it to be two lines\)/squash \1 \2/g" < $1 > $1.tmp
  mv $1.tmp $1
# Else, assume it's the commit message file
else
# Use our pre-canned message
  echo "Created two line file" > $1
fi

P.S.: Yes, I know some of you cringe when you see me using emacs as a fall-back editor.

P.P.S.: We do know we'll have to blow away all of our clones of the existing repository after the rebase. (Along the lines of "thou shalt not rebase a repository after it's been published".)

P.P.P.S: Can anyone tell me how to add a bounty to this? I'm not seeing the option anywhere on this screen whether I'm in edit mode or view mode.

Solution 1

All right, I'm confident enough to throw out an answer. Maybe will have to edit it, but I believe I know what your problem is.

Your toy repo test case has a merge in it - worse, it has a merge with conflicts. And you're rebasing across the merge. Without -p (which doesn't totally work with -i), the merges are ignored. This means that whatever you did in your conflict resolution isn't there when the rebase tries to cherry-pick the next commit, so its patch may not apply. (I believe this is shown as a merge conflict because git cherry-pick can apply the patch by doing a three-way merge between the original commit, the current commit, and the common ancestor.)

Unfortunately, as we noted in the comments, -i and -p (preserve merges) don't get along very well. I know that editing/rewording work, and that reordering doesn't. However, I believe that it works fine with squashes. This is not documented, but it worked for the test cases I describe below. If your case is way, way more complex, you may have a lot of trouble doing what you want, though it'll still be possible. (Moral of the story: clean up with rebase -i before merging.)

So, let's suppose we have a very simple case, where we want to squash together A, B, and C:

- o - A - B - C - X - D - E - F (master)
   \             /
    Z -----------

Now, like I said, if there were no conflicts in X, git rebase -i -p works as you'd expect.

If there are conflicts, things get a little trickier. It'll do fine squashing, but then when it tries to recreate the merge, the conflicts will happen again. You'll have to resolve them again, add them to the index, then use git rebase --continue to move on. (Of course, you can resolve them again by checking out the version from the original merge commit.)

If you happen to have rerere enabled in your repo (rerere.enabled set to true), this will be way easier - git will be able to reuse the recorded resolution from when you originally had the conflicts, and all you have to do is inspect it to make sure it worked right, add the files to the index, and continue. (You can even go one step farther, turning on rerere.autoupdate, and it'll add them for you, so the merge won't even fail). I'm guessing, however, that you didn't ever enable rerere, so you're going to have to do the conflict resolution yourself.*

_{* Or, you could try the rerere-train.sh script from git-contrib, which attempts to "Prime [the] rerere database from existing merge commits" - basically, it checks out all the merge commits, tries to merge them, and if the merge fails, it grabs the results and shows them to git-rerere. This could be time-consuming, and I've never actually used it, but it might be very helpful.}

Solution 2

If you don't mind creating a new branch, this is how I dealt with the problem:

Being on main:

# create a new branch
git checkout -b new_clean_branch

# apply all changes
git merge original_messy_branch

# forget the commits but have the changes staged for commit
git reset --soft main        

git commit -m "Squashed changes from original_messy_branch"

Solution 3

I was looking for a similar requirement , i.e. discarding intermeiate commits of my development branch , I've found this procedure worked for me.
on my working branch

git reset –hard mybranch-start-commit
git checkout mybranch-end-commit . // files only of the latest commit
git add -a
git commit -m”New Message intermediate commits discarded”

viola we have connected the latest commit to the start commit of the branch! No merge conflict issues! In my learning practice I have come to this conclusion at this stage , Is there a better approach for the purpose .

Solution 4

If you want to create exactly one commit out of a long branch of commits, some of which are merge commits, the easiest way is to reset your branch to the point before the first commit while keeping all your changes, then recommitting them:

git reset $(git merge-base origin/master @)
git add .
git commit

Replace origin/master with the name of the branch from which you branched off.

The add . is necessary because files that were newly added appear as untracked after the reset.

Solution 5

Building on @hlidka's great answer above which minimises manual intervention, I wanted to add a version that preserves any new commits on master that aren't in the branch to squash.

As I believe these could be easily lost in the git reset step in that example.

# create a new branch 
# ...from the commit in master original_messy_branch was originally based on. eg 5654da06
git checkout -b new_clean_branch 5654da06

# apply all changes
git merge original_messy_branch

# forget the commits but have the changes staged for commit
# ...base the reset on the base commit from Master
git reset --soft 5654da06       

git commit -m "Squashed changes from original_messy_branch"

# Rebase onto HEAD of master
git rebase origin/master

# Resolve any new conflicts from the new commits

View more solutions

Comments

[Cascabel]

More relevant than the scripts used is the final attempted action - it looks pretty sure to be a list of intermixed pick and squash, right? And are there any merge commits in the rebased branch? (Though you're not using rebase -p anyway)

[Ben Hocking]

I'm not sure what you mean by "final attempted action", but it is just a list of intermixed pick and squash, with the last 400 or so being all pick. There are no merges in that list, although the rebasing itself is performing its own merges. According to what I've read, "rebase -p" isn't recommended with interactive mode (which in my case ain't all that interactive, of course). From kernel.org/pub/software/scm/git/docs/git-rebase.html: "This uses the --interactive machinery internally, but combining it with the --interactive option explicitly is generally not a good idea"

[Cascabel]

By "final attempted action" I meant the list of pick/squash passed back to rebase --interactive - those are sort of a list of actions for git to attempt. I was hoping you might be able to reduce this to a single squash that was causing conflicts, and avoid all the extra complexity of your helper scripts. The other missing information is when the conflicts occur - when git applies the patches to form the squash, or when it tries to move on past the squash and apply the next patch? (And are you sure nothing bad happens with your GIT_EDITOR kludge? Another vote for simple test case.)

[Cascabel]

I really don't think you want to be using a --strategy option. All this happened without it, right? And again, can you reduce this to a single simple operation, without using your script, which produces the problem? Maybe just one of the squashes? (Unless the error happens after all of the squashes, and it doesn't happen if you do only some of them)

[Ben Hocking]

I did try it without a --strategy option, without using the script, and without even doing any squashes. (In other words a git rebase -i that was essentially a no-op.) It didn't have the first few problems (i.e., "nothing to commit"), but it stopped again at the first space/tab conflict. If I do the git rebase -i and just do that first squash, it also suffers from the "nothing to commit" strangeness.

[Cascabel]

Are you super-extra-sure there's nothing in your config? In particular, could apply.whitespace be set? (Either use git config --get apply.whitespace or check all repo, user, and system configs.) And just in case, is this with a current version of git?

[Ben Hocking]

I checked all my settings with both git config --global -l and git config -l (in the appropriate repository directory), and no whitespace stuff showed up. Just to be super-extra-sure, I checked explicitly for apply.whitespace both locally and globally. Nada. The current version I'm using is 1.7.1, although the repository history was generated with an earlier version(s), of course. On the bright side, at least I'm not missing something blindingly obvious. :) (Although ego aside, I really wish I was.)

[Ben Hocking]

By the way, I won't get offended, even if you ask questions that you think are really obvious. Sometimes the computer really is unplugged.

[Cascabel]

Huh. This is really bizarre, and I'm running out of ideas. I've never seen a no-op rebase fail. It really is no-op, right? That is, the SHA1 is an ancestor of the current commit. I actually thought it'd detect that it could just fast-forward in that case, but even if not, all it'll do for a pick is call git cherry-pick. I guess to keep narrowing it down... you could check out the commit just before the one that fails, then do that cherry-pick yourself, and see if it fails?

[Ben Hocking]

Well, I'm not sure if it's really a no-op. By no-op, I mean that I just accepted the list as is, without changing anything. There are a lot of branches that might be getting squashed down, though. That's a good idea about the cherry-picking after checking out the single commit. I'll get back to you on the results. I'm also planning on creating a simple test that contains a branch that gets merged in with a conflict that requires manual resolution.

[Ben Hocking]

Doing the single cherry-pick worked fine (i.e., it didn't reproduce the error). Just to make sure I did it correctly (it's my first cherry-pick - insert joke there), I looked at the file saved off from my rebase -i command (basically a log file), checked out the SHA-1 before the one reporting the problem, then cherry-picked the one with the problem. After that didn't recreate the problem, I then checked out one further back, and cherry-picked two additional commits (for a total of 3 cherry-picks, ending up one commit further ahead). Still didn't get the problem.

[Ben Hocking]

OK, so I've been able to recreate this with a toy repository. Code to follow.

[Cascabel]

The toy repository case is much, much easier to understand than your scripts - and shows that the scripts have nothing to do with it, just the fact that you're trying to rebase a branch containing a merge with conflict resolution (a mildly evil merge).

[Cascabel]

P.S. Looking back at my comments, I see I should've caught this sooner. I asked if you were rebasing a branch containing merges, and you said there were no merges in the interactive rebase list, which is not the same thing - there were no merges there because you didn't supply the -p option.

[Ben Hocking]

I'll definitely give this a go. Since posting this, I've also noticed in some cases, I can simply type git commit -a -m"Some message" and git rebase --continue, and it will continue on. This works even without the -p option, but it works even better with the -p option (since I'm not doing any re-ordering, it seems that -p is fine). Anyways, I'll keep you posted.

[Ben Hocking]

Setting git config --global rerere.enabled true and git config --global rerere.autoupdate true prior to running the test example does resolve the primary problem. Interestingly enough, however, it does not preserve the merges, even when specifying --preserve-merges. However, if I don't have those set, and I type git commit -a -m"Meaningful commit" and git rebase --continue while specifying --preserve-merges, it does preserve the merges. Anyways, thanks for helping me get past this problem!

[ahains]

FYI - I was just trying this and found that doing the checkout of mybranch-end-commit does not get me the deletes that occurred in the intermediate commits. So it only checks out the files that existed in mybranch-end-commit.

[jezpez]

This is a far safer solution. I also added --squash to the merge. Being: git merge --squash original_messy_branch

[Jeremy Cochoy]

I just couldn't squash / edit any commit because of merge in a 40+ commits history. This simple solution took me 10 seconds. I did this work in a new branch and when everything was fine (I did a copy just in case) I reseted by messy branch to this nice squashed commit. Perfect! :)

[JonoB]

I still got a few conflicts to manually resolve when attemping the rebase with -p Admittedly there were a lot fewer conflicts and they were just limited to the project file rather than all the code files I was getting conflicts with before. Apparently "Merge conflict resolutions or manual amendments to merge commits are not preserved." - stackoverflow.com/a/35714301/727345

[JonoB]

Firstly, this answer is fantastic, it has saved me a lot of time. But I think it assumes that master hasn't had any new commits since original_messy_branch was created. Because these new master commits will be undone by the step git reset --soft master. I would recommend specifying a commit on master to work against rather than using head, and then do a rebase afterwards to include those new commits on master

[Jp_]

Very nice solution, but in my case I'm not using rebase to squash, but to change commit messages. Any idea about this case?

[Rahul Patil]

But soft reset didn't happen for me, one of the quickest solution