Read from and write to file from stdin and stdout
The redirection operator to open a file in read+write mode without truncation is <>
in all Bourne-like shells (that maps to open(file, O_RDWR|O_CREAT)
(though zsh
also throws in a O_NOCTTY
) or fopen(file, "w+")
):
exec 3<> "$file"
opens the $file
on file descriptor 3 in read+write mode (without truncating it and creating it if it didn't exist).
However, only ksh93
and zsh
have seeking operators. dd
can seek, but not backwards. And note no shell except zsh
can have NUL bytes in their variables.
In zsh
:
zmodload zsh/system
exec 3<> $file
sysread -i 3 -c 2 var # a read() of 2 bytes
sysseek -u 3 0 # seek back to beginning
# or sysseek -u 3 -w current -2 # to seek back 2 bytes
syswrite -o 3 something-else
exec 3<&- # close
In ksh93
:
exec 3<> "$file"
var=$(dd bs=2 count=1 <&3 2>/dev/null; echo .)
var=${var%?}
exec 3<#((0)) # seek to beginning
# or exec 3<#((CUR-2)) # to seek back 2 bytes
print -ru3 something-else
Portably, you could still open the file several times, for each offset you want, like here to read and write 2 bytes at offset 2 (provided they're not bytes with value 0 if not using zsh
):
var=$(dd bs=2 count=1 skip=1 < "$file"; echo .)
var=${var%?}
printf %s something-else | dd bs=2 seek=1 1<> "$file"
Or:
printf %s something-else | dd bs=2 seek=1 of="$file" conv=notrunc
To read and write to the same file, ksh93
has two other interesting redirection operators:
tr 01 10 < file >; file
Would store the output of tr
in a temporary file and if tr
is successful, rename that to file
(beware the file is created anew, so with possibly different permissions and ownership).
tr -d 0 < file 1<>; file
Same as the standard/Bourne tr -d 0 < file 1<> file
except that if tr
succeeds, file
is truncated where tr
finished writing. You can use that for filter commands that produce less output than they read input, or more precisely commands that would not read data that they've previously written.
And zsh
has the =(...)
form of process substitution which you can use as:
mv =(tr 01 10 < file) file
(with similar effect and caveats as ksh93
's >;
). Or:
cp =(tr 01 10 < file) file
which would preserve attributes of file
but means an extra copy.
Now if you need to read and write at the same offset using the same file descriptor and neither zsh nor ksh93 are available, you could always revert to perl
/python
/ruby
...
perl -e '
open F, "<>", "file" or die "open: $!";
read F, $var, 1;
seek F, 0, 0;
print F "something-else"'
Now, after re-reading the updated version of your question, it looks like your file is behaving more like a socket or bidirectional pipe, and not like a regular, seekable file.
In which case, it could be just a matter of:
socat - file:your-file
or:
(cat >&3 3>&- & cat <&3 3<&-) 3<> your-file
to feed data from and to that file as read from/to stdin/stdout.
Note that each cat
reads/writes to its own copy of the file descriptor 3 open by the shell, but they share the same open file description so it should be equivalent.
Related videos on Youtube
Mark Smith
Updated on September 18, 2022Comments
-
Mark Smith over 1 year
I need to continually read from and write to a (virtual) file in bash, without closing the file in between. I'm looking for a program which will open the file for read and write, read from stdin and write what it reads to the file AND read from the file and write it to stdout. A bit like
netcat
but for files.The file in question is a virtual file in /sys (a custom thing in something I'm working on) to which one writes a command and reads to receive the response - where the write and subsequent read must be done in the same session, i.e. on the same file descriptor without closing it in between.
Easy to do in C -
open("file", O_RDWR)
,write()
,read()
- note: no need toseek()
.Is there a standard GNU tool or do I need to write one?
-
Bjorn Munch about 8 yearsMaybe you can used tee to achieve what you need? It will not read from the file though, but will copy to stdout what it writes to the file.
-
G-Man Says 'Reinstate Monica' about 8 yearsIf the suggestions that you've gotten so far aren't satisfactory to you, try explaining what you want to do a bit more clearly. For example, are you starting with an empty file (or maybe a non-existent file), reading from stdin, and writing the data to both the file and stdout? Or is there something already in the file when you start? If so, do you want the new data to overwrite the old data? In this case, do you want data from stdin to end up on stdout (after being written to the file), or do you want only the original contents of the file to be written to stdout?
-
G-Man Says 'Reinstate Monica' about 8 yearsPlease do not respond in comments; edit your question to make it clearer and more complete.
-
Stéphane Chazelas about 8 yearsIs that file behaving like a regular file (seekable, and
read()
return 0 bytes (EOF) when the end is reached), or like a bi-directional pipe or socket (where what you write is not what you may read again after seeking and whereread()
blocks when there's no available input)?
-
-
Mark Smith about 8 yearsThanks for the reply. I've edited my question to make it more explicit. Firstly, I need to write to the file before reading. (Before AND after would be OK too.) Secondly the reads and writes need to be done on the same file descriptor.
-
Stéphane Chazelas about 8 yearsHow is
command < filename | tee >(sponge filename)
solving the half translated problem? -
Mark Smith about 8 yearsThanks for the reply. I can't open the file several times: the write and read must be on the same file descriptor. I'll have a play with the
exec 3<> /sys/blah
technique and see if I can get anywhere, thanks. -
cuonglm about 8 yearsIt's worth to note that
>;
and<>;
can not be used withexec
. -
clerksx about 8 years@StéphaneChazelas It doesn't. :-) That why I said "even with translation", not "with translation".
-
Stéphane Chazelas about 8 yearsNote that if
$TMPDIR
is on the same filesystem asfilename
, thenfilename
is created anew. Whilesponge
will try and reproduce permissions, that will not be the case for other file attributes (ownership, ACLs, extended attributes...).