git rebase, "would be overwritten", and "No changes - did you forget to use 'git add'?"

Question

git rebase does not appear to work properly in certain cases where a file is added to the repository, then removed from the repository, then added to the working directory (but not the repository).

Here's a more specific description of my problem:

• if a branch is created and switched to from some trunk,

• and a file X is added and committed in the branch,

• and subsequently X is removed and committed in the branch,

• and X is again created in the working directory, but not added or committed,

• and the trunk branch advances,

• then

• a rebase performed using the advanced trunk as the base will fail because it will refuse to overwrite X,

• and the rebase cannot be continued even if the working directory X is removed or moved out of the way.

Here's a script to reproduce my problem on the command line:

git init
echo foo > foo.txt
git add .
git commit -m 'foo'
echo foo >> foo.txt
git add .
git commit -m 'foo foo'
git checkout -b topic HEAD^
git log
echo bar > bar.txt
echo baz > baz.txt
git add .
git commit -m 'bar baz'
git rm bar.txt
git commit -m '-bar' 
echo bar > bar.txt
git rebase master 
# the following output is emitted:
# First, rewinding head to replay your work on top of it...
# Applying: bar baz
# Using index info to reconstruct a base tree...
# Falling back to patching base and 3-way merge...
# error: Untracked working tree file 'bar.txt' would be overwritten by merge.  Aborting
# Failed to merge in the changes.
# Patch failed at 0001 bar baz
# 
# When you have resolved this problem run "git rebase --continue".
rm bar.txt
git rebase --continue
# the following output is emitted:
# Applying: bar baz
# No changes - did you forget to use 'git add'?
# 
# When you have resolved this problem run "git rebase --continue".
# If you would prefer to skip this patch, instead run "git rebase --skip".
# To restore the original branch and stop rebasing run "git rebase --abort".

I know I can abort the rebase using git rebase --abort, remove bar.txt, and then git rebase master again. But how can I continue the rebase without aborting it first?

Solution 1

I found a solution: apply the troubled commit's patch manually and add it to the index.

$ patch -p1 < .git/rebase-apply/patch 
patching file bar.txt
patching file baz.txt
$ git add bar.txt baz.txt
$ git rebase --continue
Applying: bar baz
Applying: -bar

Solution 2

You need to commit before attempting to rebase. It will work sometimes if you don't, but you shouldn't do that.

If you're uncomfortable committing for some reason, you could at least use stash (which does roughly the same thing).

Solution 3

git rebase is working correctly. It is protecting your untracked file from being destroyed while visiting a commit that wants to write out a file with that same pathname.

There does not seem to be a way to retry just the most recent step of the rebase. Usually when a rebase pauses, it will leave conflicts in the index. But in this case, it can not flag this problem in the index because doing so would convert your untracked file into a tracked file. This is probably a bit of a rough spot in the UI of git rebase. You could dig into the directory it uses to store its internal state (.git/rebase-apply) and manually apply the “current” patch, but aborting and restart is probably easier (but maybe not faster if you are in the middle of rebasing a very long series).

---

If the addition of bar.txt is considered a mistake, then you might consider using git rebase -i to tease apart and drop the addition and removal of bar.txt since you are rewriting history anyway.
You will still run into the conflict, but the methods below can also be applied with git rebase -i. The script at the end would have to be split into two parts (“setup temp/” and “incorporate rebase result” since an interactive rebase will usually require multiple commands between those two sections.

---

Move the conflicting file aside temporarily and redo the rebase.

mv bar.txt +bar.txt
git rebase --abort
git rebase master

If you expect many such conflicting files then you might consider doing your rebase in a separate clone where you can be certain that you will not have any untracked files. Perhaps the trickiest part is checking that your untracked files do not conflict with the result of the rebase (the git checkout rebased-topic accomplishes this; it aborts if the untracked files conflict with the rebase result).

: "Note: will destroy
:     * 'rebased-topic' branch
:     * 'temp' directory"
( set -x &&
  : '*** Clearing temp/' &&
  rm -rf temp/ &&

  : '*** Making sure we have topic checked out' &&
  git checkout topic

  : '*** Cloning into temp/' &&
  git clone . temp && 

  : '*** Rebasing topic onto master in temp/ clone' &&
  ( cd temp &&
  git rebase origin/master
  ) &&

  : '*** Fetching rebase result from topic/ into local rebased-topic branch' &&
  git fetch -f temp topic:rebased-topic &&

  : '*** Checking rebased-topic for conflicts with untracked files' &&
  git checkout rebased-topic &&

  : '*** Resetting topic to tip of rebased-topic' &&
  git branch -f topic rebased-topic &&

  : '*** Returning to topic branch' &&
  git checkout topic &&

  : '*** Deleting rebased-topic branch' &&
  git branch -d rebased-topic &&

  : '*** Deleting temp/' &&
  rm -rf temp/
)

Comments

[Russell Silva]

Okay, but if that's the case then the error message I'm getting is extremely unhelpful, since it implies that it's possible to continue the rebase.

[mipadi]

@Russell Silva: You can continue. The issue is that, since you removed bar.txt, rebase has no work to do in that commit. git rebase --skip should solve the problem, since you now want to skip the (empty) commit. It's a bit confusing and I'm surprised git rebase even lets you proceed with a dirty work tree, but then again, git rebase isn't for the faint-hearted.

[Russell Silva]

@mipadi: The commit is not empty. In the example, baz.txt is added by the same commit that adds bar.txt. git rebase --skip results in the loss of baz.txt.

[Shannon]

In my situation, the only reason the file is untracked is because of the rebase rewind. Shouldn't rewind delete files that weren't tracked in earlier commits? There would be no data loss.

[Chris Johnsen]

@rehevkor5: I am not sure of the details of your situations, but if (for example) you have sequential commits A, B, C, and D where only the last one has the file frob (i.e. it was initially added in D and nowhere else), then Git will (temporarily) remove frob while rewinding to rebase D onto some other base (e.g. a modified B). The situation of this question is a bit more complicated; if you have some specific situation in mind, you might want to open a new question with the details.

[André Puel]

I suggest using git-apply since patch does not support the Git binary diff format.

[Szczepan Hołyszewski]

"Troubled commit" is placeholder phrasing. Please replace it with detailed instructions on how to identify the "troubled commit". If this is already happening in the sequence of commands given in your answer, please explain where exactly.

[Mahn]

This fixed it for me. When the initial "changes would be overwritten" error comes out, there should a line indicating where the patch file has been saved. e.g.: The copy of the patch that failed is found in: .git/rebase-apply/patch. That's the one you want to run via git apply, e.g. git apply .git/rebase-apply/patch. Then git add . to stage everything and finally git rebase --continue

[Rots]

I also recommend adding the -3 for the git apply to use the 3-way merge, also see answer: stackoverflow.com/a/58412727/301049