Can I read and write to the same file in Linux without overwriting it?

bash shell stdout stdin

5,464

Solution 1

There is a utility called sponge that is a part of the moreutils suite. It was made for this exact purpose.

grep -v ec2 ~/.ssh/known_hosts | sponge ~/.ssh/known_hosts

Solution 2

Redirections are done by the shell, before the command runs. This means that the shell is told to truncate the file before grep gets a chance to read it. There is no way around this if you are using shell redirection.

5,464

cwd

Updated on September 18, 2022

Comments

cwd over 1 year
Possible Duplicate:
How can I make iconv replace the input file with the converted output?

I frequently connect to amazon ec2 using their public DNS names
(ec2-12-34-56-78.compute-1.amazonaws.com) and because of this my known_hosts file gets overwhelmed with a lot of ec2 entries I will never use again.

I know I could probably use sed -i to edit in place but I wanted to use grep and so I did this:
```
grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/known_hosts
```
That leaves known_hosts as an empty file. If I do:
```
grep -v ec2 ~/.ssh/known_hosts > ~/.ssh/tmp
mv ~/.ssh/tmp ~/.ssh/known_hosts
```
then things are fine, but I am confused why reading and writing to the same file leaves it blank, and if there is any way around this when using grep, cat, etc.
- Admin about 12 years
  
  Even if the program were doing the redirection itself, opening the file for read and then for write before actually reading from it would result in reading nothing (unless the language's standard library speculatively prebuffered, but that means you can't configure the buffer first). What were you expecting to happen?
- Admin about 12 years
  
  Sounds good. Thanks. Why is this not part of your answer?
- Admin about 12 years
  
  Because it's actually a related issue, not the original problem (the shell is doing the file create, so grep would not have a chance to do anything about it anyway). I'm just confused as to how it would be expected to work in the first place.
- Admin about 12 years
  
  This is not the first time I've experienced this, so I don't "expect" to to work anymore, but initially I assumed that input flows from left to right and you operate on it on the left side and then would output it to the file on the right side. Guess part of the problem is that years ago I was a windows user and something similar would work in DOS.
- Admin about 12 years
  
  Even DOS (2 and up) would do "the wrong thing" in most circumstances, although it at least would do the speculative prebuffer in many cases so you would see it appeart to work if the file was small enough; moreover, DOS 2 and later had shell redirection in COMMAND.COM and so had the overwrite happen before the command even ran.
- Admin about 12 years
  
  Well thank goodness for sponge then :)
- Admin about 12 years
  
  What's wrong with sed -i /ec2/d?
- Admin about 12 years
  
  @ephemient - nothing is wrong with that for this specific example.
- Admin about 12 years
  
  @Gilles - I believe that question is written with too much specificity to be found or read when searching for the core of the problem being discussed here.
- Admin about 12 years
  
  @Gilles - wouldn't you want to have the more versatile question stay open?
- Admin about 12 years
  
  @cwd It's better to use the one with the best answers. (I'm not sure I picked the right one, I have an easier time finding my own answers so I'm biased.) But it doesn't matter much.
Shawn J. Goff almost 10 years

cat can write to its output at any time; if it writes past the point at whatever is on the other side of the pipe has read, then you will read in what you wrote instead of the original file contents. That is what makes sponge special. It buffers its input until stdin closes, then writes.
Sebastian over 9 years

sponge is also part of sbase