How to remove files that are listed in the .gitignore but still on the repository?

Question

I have some files in my repository that should be ignored, i added them to the .gitignore but, of course, they are not removed from my repository.

So my question is, is there a magic command or script using filter-branch that can rewrite my history and remove all these files easily? Or simply a command that will create a commit that will remove them ?

Solution 1

You can remove them from the repository manually:

git rm --cached file1 file2 dir/file3

Or, if you have a lot of files:

git rm --cached `git ls-files -i -c --exclude-from=.gitignore`

But this doesn't seem to work in Git Bash on Windows. It produces an error message. The following works better:

git ls-files -i -c --exclude-from=.gitignore | xargs git rm --cached  

In PowerShell on Windows this works even better (handles spaces in path and filenames):

git ls-files -i -c --exclude-from=.gitignore | %{git rm --cached $_}

Regarding rewriting the whole history without these files, I highly doubt there's an automatic way to do it.
And we all know that rewriting the history is bad, don't we? :)

Solution 2

An easier way that works regardless of the OS is to do

git rm -r --cached .
git add .
git commit -m "Drop files from .gitignore"

You basically remove and re-add all files, but git add will ignore the ones in .gitignore.

Using the --cached option will keep files in your filesystem, so you won't be removing files from your disk.

Note: Some pointed out in the comments that you will lose the history of all your files. I tested this with git 2.27.0 on MacOS and it is not the case. If you want to check what is happening, check your git diff HEAD~1 before you push your commit.

Solution 3

As the files in .gitignore are not being tracked, you can use the git clean command to recursively remove files that are not under version control.

Use git clean -xdn to perform a dry run and see what will be removed.
Then use git clean -xdf to execute it.

Basically, git clean -h or man git-clean(in unix) will give you help.

Be aware that this command will also remove new files that are not in the staging area.

Hope it helps.

Solution 4

I did a very straightforward solution by manipulating the output of the .gitignore statement with sed:

cat .gitignore | sed '/^#.*/ d' | sed '/^\s*$/ d' | sed 's/^/git rm -r /' | bash

Explanation:

1. print the .gitignore file
2. remove all comments from the print
3. delete all empty lines
4. add 'git rm -r ' to the start of the line
5. execute every line.

Solution 5

"git clean"^(man) and git ls-files -i^(man) had confusion around working on or showing ignored paths inside an ignored directory, which has been corrected with Git 2.32 (Q2 2021).

That means the 2021 version of the accepted answer would be:

git ls-files -i -c --exclude-from=.gitignore | xargs git rm --cached  
                ^^

See commit b548f0f, commit dd55fc0, commit aa6e1b2, commit a97c7a8, commit 2e4e43a, commit b338e9f, commit 7fe1ffd, commit 7f9dd87 (12 May 2021) by Elijah Newren (newren).
See commit 4e689d8 (12 May 2021) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit 33be431, 20 May 2021)}

ls-files: error out on -i unless -o or -c are specified

^{Signed-off-by: Elijah Newren}

ls-files --ignored^(man) can be used together with either --others or --cached.

After being perplexed for a bit and digging in to the code, I assumed that ls-files -i was just broken and not printing anything and I had a nice patch ready to submit when I finally realized that -i can be used with --cached to find tracked ignores.

While that was a mistake on my part, and a careful reading of the documentation could have made this more clear, I suspect this is an error others are likely to make as well.
In fact, of two uses in our testsuite, I believe one of the two did make this error.
In t1306.13, there are NO tracked files, and all the excludes built up and used in that test and in previous tests thus have to be about untracked files.
However, since they were looking for an empty result, the mistake went unnoticed as their erroneous command also just happened to give an empty answer.

-i will most the time be used with -o, which would suggest we could just make -i imply -o in the absence of either a -o or -c, but that would be a backward incompatible break.
Instead, let's just flag -i without either a -o or -c as an error, and update the two relevant testcases to specify their intent.

That means without -c, you would get (starting with Git 2.32, Q2 2021):

fatal: ls-files -i must be used with either -o or -c

Note: this is still a work in progress, since it was reverted in Git 2.32-rc2 but fixed with commit 2c9f1bf, commit 1df046b (27 May 2021) by Junio C Hamano (gitster).
See commit 906fc55 (27 May 2021) by Elijah Newren (newren).
See commit eef8148 (27 May 2021) by Derrick Stolee (derrickstolee).
^{(Merged by Junio C Hamano -- gitster -- in commit 329d63e, 28 May 2021)}

dir: introduce readdir_skip_dot_and_dotdot() helper

^{Signed-off-by: Elijah Newren}

View more solutions

Comments

[Intrepidd]

There are a lot of them, is there a way to delete them without having to specifying their names?

[Nate Bundy]

Unfortunately the Git Bash on Windows command doesn't work with paths that contain spaces

[Admin]

@NateBundy if you're referring to the fact that xargs won't work with spaces, most command line utilities get around that by using special flags so that whitespace won't matter. Off the top of my head I don't remember what the flags are for git ls-files and xargs (I think it might be -0 for xargs), but you can look them up.

[Roger Oba]

This should be the accepted answer. It actually works (the accepted answer didn't work for me, on macOS), and it's way cleaner.

[Ken Williams]

This answer isn't applicable - the OP says the files in .gitignore are being tracked.

[StackAttack]

wrong, wrong, and wrong. :( sorry no insight, yeah git needs to add a feature that doesn't make this particular thing - which totally sucks ass and eats up tons of coding time - not suck so bad

[snh_nl]

How does this effect other users on the repo executing a pull - as I read it the folder we dont want to track anymore is deleted??? How can we prevent this - we just want to untrack

[Emmanuel]

BEWARE! This permanently deletes all untracked files, rather than just removing from the branch

[IgorGanapolsky]

sed is straightforward?

[JosanSun]

git rm --cached git ls-files -i --exclude-from=.gitignore`` seems working now. Windows10, git version 2.19.1

[Toast]

None of these work in Windows, they all just print the usage for git rm

[Sundypha]

the git clean -xdn is a dry run which won't delete. the next one will.

[Auspice]

-1: This is a highly misleading answer - the original poster wanted to remove from the repository, not remove the files completely. I was this close to deleting a load of dynamic files required by my IDE but not required to be in the repo.

[Owen Cao]

"git ls-files -i --exclude-from=.gitignore" is very helpful, it tells me what files are excluded by .ignore.

[Jan Potužník]

I was happy to find these commands, BUT it does not really remove the files from repository. When i removed 2300 kB of images, repository size dropped only 10 kB. So it cannot be used, for example, to make repository smaller and faster to transfer.

[chintogtokh]

And if you have way too many files to remove and bash is complaining that the arguments are too long, you can successively do git rm --cached `git ls-files -i --exclude-from=.gitignore` | head -n 100

[Haidar Zeineddine]

This will mark all the files as changed with a fake commit message !

[gtatr]

what do you mean by fake commit message? It's a real commit message :P You can change the message of course, depending on your needs...

[Maytham Fahmi]

the last part (code) for windows is a working solution for my usage case.

[Mariusz Bialobrzeski]

Just for a completeness sake you need git push command at the end.

[agrath]

Don't do this, it will remove history on all the files

[phoog]

@agrath What if you don't care about that history? In my present situation the files in question are log files from debugging sessions. Why waste space with their history?

[Rafael Valero]

Siilar example with another problem: stackoverflow.com/a/63534476/7127519

[gtatr]

@agrath where did you test this? Which version of git? Which OS? It is not the case for me on MacOS and git 2.27.0

[Rawheiser]

+1 for scripting and explaining it. Explaining the danger of running it right into bash without review may have been an oversight.

[marcvangend]

Note that regardless of your current directory, --exclude-from= is relative to the git root. So if you want to do this for a .gitignore in a subdirectory, use --exclude-from=[subdirectory]/.gitignore.

[Logan Besecker]

Just to clarify a bit because the git status after doing git rm -r --cached . git add . can look a bit scary: These three commands that @gtatr provides essentially delete files from git that were previously tracked, but have since been added to your .gitignore file. When I first ran this, I saw a bunch of files and freaked out for a second but upon further inspection I noticed they were all files that were listed in my .gitignore file.

[stomy]

cmd git rm --cached -r dirName worked in Windows 7 to recursively delete all files in a folder.

[Thom Howard]

On Windows 10 Git bash command line the command "git rm --cached git ls-files -i --exclude-from=.gitignore" worked fine. Thanks for the answer!

[MaxiReglisse]

do not forget to commit: git commit -m "Drop files from .gitignore" !

[CoryCoolguy]

Uhhhhhh what do I do if git rm --cached `git ls-files -i -c --exclude-from=.gitignore` gives me -bash: /usr/bin/git: Argument list too long 😬 (not my repo!) // I checked and there's 24862 files that should have been ignored but weren't. Wish me luck!