How to substitute text from files in git history?

Question

I've always used an interface based git client (smartGit) and thus don't have much experience with the git console.

However, I now face the need to substitute a string in all .txt files from history (so, not erasing the whole file but just substituting a string). I found the following command:

git filter-branch --tree-filter 'git ls-files -z "*.php" |xargs -0 perl -p -i -e "s#(PASSWORD1|PASSWORD2|PASSWORD3)#xXxXxXxXxXx#g"' -- --all

I tried this, and unfortunately noticed that while the password did get changed, all binary files got corrupted. Images, etc. would all be corrupted.

Is there a better way to do this that won't corrupt my binary files?

Thanks.

EDIT:

I got mixed up with something. The actual code that caused binary files to get corrupted was:

$ git filter-branch --tree-filter "find . -type f -exec sed -i -e 's/originalpassword/newpassword/g' {} \;"

The code at the top actually removed all files with my password strangely enough.

Solution 1

You can avoid touching undesired files by passing -name "pattern" to find.

This works for me:

git filter-branch --tree-filter "find . -name '*.php' -exec sed -i -e \
    's/originalpassword/newpassword/g' {} \;"

Solution 2

I'd recommend using the BFG Repo-Cleaner, a simpler, faster alternative to git-filter-branch specifically designed for rewriting files from Git history.

You should carefully follow these steps here: https://rtyley.github.io/bfg-repo-cleaner/#usage - but the core bit is just this: download the BFG's jar (requires Java 7 or above) and run this command:

$ java -jar bfg.jar  --replace-text replacements.txt -fi *.php  my-repo.git

The replacements.txt file should contain all the substitutions you want to do, in a format like this (one entry per line - note the comments shouldn't be included):

PASSWORD1 # Replace literal string 'PASSWORD1' with '***REMOVED***' (default)
PASSWORD2==>examplePass         # replace with 'examplePass' instead
PASSWORD3==>                    # replace with the empty string
regex:password=\w+==>password=  # Replace, using a regex
regex:\r(\n)==>$1               # Replace Windows newlines with Unix newlines

Your entire repository history will be scanned, and .php files (under 1MB in size) will have the substitutions performed: any matching string (that isn't in your latest commit) will be replaced.

Full disclosure: I'm the author of the BFG Repo-Cleaner.

Solution 3

With Git 2.24 (Q4 2019), git filter-branch (and BFG) is deprecated.

newren/git-filter-repo does NOT do what you want.
It has an example that is ALMOST what you want in its example section:

cd repo
git filter-repo --path-glob '*.txt' --replace-text expressions.txt

with expressions.txt:

literal:originalpassword==>newpassword

However, WARNING: As Hasturkun adds in the comments

Using --path-glob (or --path) causes git filter-branch to only keep files matching those specifications.
The functionality to only replace text in specific files is available in bfg-ish as -fi, or the lint-history script.
Otherwise, it looks like this is only currently possible with a custom commit callback.
See newren/git-filter-repo issue 74

Which makes senses, considering the --replace-text option is itself a blob callback.

Solution 4

I created a file at /usr/local/git/findsed.sh , with the following contents:

find . -name 'githubDirToSubmodule.sh' -exec sed -i '' -e 's/What I want to remove//g' {} \;

I ran the command:

git filter-branch --tree-filter "sh /usr/local/git/findsed.sh"

Explanation of commands

When you run git filter-branch, this goes through each revision that you ever committed, one by one. --tree-filter runs the findsed.sh script on each committed revision, saves it, then progresses to the next revision.

The find command finds a specific file or set of files and executes (-exec) the sed editor on that file. sed is a command that takes the regex after s/ and replaces it with the string between / and /g (blank in my example). {} is a reference to the files path that was given by the find command. The file path is fed to sed, so that sed knows what to work on. \; just ends the -exec command.

Seperating the shell script and command out into seperate pieces allows for less complication when it comes to quotes '' or "".

Peculiarities

I successfully implemented this on a mac, and apparently sed is a particular (older?) version on macs. This matters, as it sometimes behaves differently. Make sure to do sed -i '' or else it was adding a "-e" to the end of files, thinking that that was what i wanted to name my backup files. -i '' says dont make backup files, just edit the files in place and no backup file needed.

Specifying -name 'filename.sh' helped me avoid another issue that I could not solve. There was another file with .sh and that file ended without a newline character. sed for some reason, would add a newline character to the end, despite the 's/blah/blah/g' not matching anything in that file. So instead of figuring out that issue, I just told the find to ignore all other files.

Additional commands that work

Additionally, I found these commands to work in the findsed.sh file (only one command at a time, not multple, so comment # the others out):

find . -name '.publishNewZenPackFromGithub.sh.swp' -exec rm -f {} \;
find . -name '*' -exec grep -H PassToRemove {} \;

Enjoy!

Solution 5

More info on git-filter-repo

https://stackoverflow.com/a/58252169/895245 gives the basics, here is some more info.

Install

As of git 2.5 at least it is not shipped with mainline git so:https://superuser.com/questions/1563034/how-do-you-install-git-filter-repo/1589985#1589985

python3 -m pip install --user git-filter-repo

Usage tips

Here is the more common approach I tend to use:

git filter-repo --replace-text <(echo 'my_password==>xxxxxxxx') HEAD

where:

• Bash process substitution allows us to not create a file for simple replaces
• HEAD makes it affect only the current branch

Modify only a range of commits

How to modify only a range of commits with git filter-repo instead of the entire branch history?

git filter-repo --replace-text <(echo 'my_password==>xxxxxxxx') --refs HEAD~2..HEAD

Replace using the Python API

For more complex replacements, you can use the Python API, see: How to use git filter-repo as a library with the Python module interface?

View more solutions

Comments

[Jimmy]

Doesn't solve your problem, but this is similar to a question I asked a while back: stackoverflow.com/questions/2225454/…

[Sukhpreet Singh Alang]

Indeed, there are many answers on how to remove files. I need to substitute a string though.

[Sukhpreet Singh Alang]

@Jimmy Cuadra, please see my edit, I actually used a different script, got mixed up. Maybe it helps you in getting the right command.

[Sukhpreet Singh Alang]

What would this one liner look like?

[Ben Jackson]

The exact thing you're passing to --tree-filter '...' right now.

[Cascabel]

Good advice; passing an actual executable script to filter-branch is often much easier than trying to deal with all the quoting.

[Sukhpreet Singh Alang]

I am on windows though, does it support bat scripts?

[Sukhpreet Singh Alang]

please see my edit, I actually used a different script, got mixed up.

[Volte]

I tried this, but looking at the git history, all the files remain the same... Do I have to 'rebase' or something (I'm so new) and if so how do I do that?

[jweyrich]

@Volte Most likely the regular expression you're using is not matching anything. This command will rewrite the repository history (like a rebase), provided that the expression matches something.

[Volte]

You were right. Turned out I was searching for .php files when I meant to be searching for .h :P That's what I get for blind-copy-paste haha. Cheers.

[test30]

unbelieveable! BFG is incredible!

[Bane]

This just helped me tremendously. Thank you for such an awesome project. I donated too. Thank you again.

[Roberto Tyley]

Thanks @Bane - really glad it helped, and thanks for supporting the project!

[Dan]

+1 as I just used BFG for fixing an SQL script that the dev team let get out of control. Though you need to highlight the default of under 1MB on your usage page Roberto, it took a long while of head scratching before it was apparent that that was why my text replace wasn't happening.

[Amedee Van Gasse]

The newlines replacement is a life saver when migrating a mixed Windows/Linux team from Subversion to Git!

[klor]

I tried using BFG on my repo, but I get "BFG aborting: No refs to update - no dirty commits found??" I use this: ---> java -jar /C/Users/user/bfg-1.12.12.jar --filter-content-including '*.{php,inc,sql,txt,htm,html,js,css,xml}' --replace-text /w/Dev/\!GIT/CVS_ident_replace.txt /w/Dev/\!GIT/repo CVS_ident_replace.txt: regex:(//\s*\$Id:).*\$==>$1\$ # Replace CVS Ident

[theyuv]

@RobertoTyley After doing the steps outlined in rtyley.github.io/bfg-repo-cleaner/#usage, how should we update our local history? I did git pull origin master. Was this a mistake? Should I have done git clone? Because after pushing again I noticed that the replacements made with bfg are gone.

[Kay]

It'd be great if examples like the above were listed on the BFG website! I had to google this SO question again to find them.

[luke]

Your script doesn't work for me (in Cygwin on Windows). However this works: git filter-branch --tree-filter "find . -name '*.php' -type f -exec sed -i -e 's/originalpassword/newpassword/g' {} \;"

[David J Eddy]

This saved my @$$ ! TY @jweyrich , shor tsweet one liner for the win.

[Felipe Correa]

@RobertoTyley Is there any way to do this only for files on certain path of the repo? I have several files with the same extension, but I want to do the cleanup only on the ones under an specific directory.

[Kaus2b]

this wasnt working, so I went through the documentation. You have a small typo. Inside the expressions.txt it should be literal:originalpassword==>newpassword

[VonC]

@KausUntwale Thank you. I have edited the answer accordingly. Don't hesitate to edit it if you see anything else.

[Eric]

In my case it was necessary to use the command with -- --all at the end as for example all tags on my original branch weren't properly copied to the resulting branch

[user760900]

how would one place a \n in the newpassword section?

[Kay]

Why is this not part of the official BFG docs? I'd send a PR but after a cursory look at the repo it seems there are many open PRs which sought to improve the project's documentation/README, which makes me feel there's no point.

[Kay]

Wow, I should actually have said "Why is this STILL not part of the official BFG docs?"... Just realised I'd already commented in the same vein almost 4 (!) years ago, lol(sob).

[VonC]

A Link to my own answer and more useful details in your own answer? All right... Automatic upvote. Those Necromancer badges won't create themselves.

[Ciro Santilli OurBigBook.com]

@VonC I forgot to upvote yours!!! I was meaning to do it!!!

[Otzen]

I tried this on a repo, the result was a repo with a single commit, and with only the file mentioned in --path-glob. I expected that the many many commits in my repo was still there and files not matched by the glob was untouched.

[VonC]

@Otzen It should have worked the way you expected. Not sure what went wrong there.

[s.k]

How to echo more than one replacement expression in your one-line solution?

[Ciro Santilli OurBigBook.com]

@s.k <(echo 'my_password==>xxxxxxxx'; echo 'my_password2==>xxxxxxxx') or <(printf my_password==>xxxxxxxx\nmy_password2==>xxxxxxxx\n) should both work.

[Hasturkun]

Using --path-glob (or --path) causes git filter-branch to only keep files matching those specifications. The functionality to only replace text in specific files is available in bfg-ish as -fi, or the lint-history script. Otherwise, it looks like this is only currently possible with a custom commit callback. See also github.com/newren/git-filter-repo/issues/74

[VonC]

@Hasturkun Thank you. I have included your comment in the answer for more visibility. And added the link to the lint-history script.

[marcolopes]

BFG didn't work. IT found the text, but still present on the revisions. Also, tried to DELETE the file, and nothing...

[marcolopes]

This takes forever... :\

[jay p]

Just stumbled upon this, great project! Does exactly what I want it to do in the shortest amount of time, thanks mate