File size limit exceeded in bash
5,083
See duplicate question in serverfault:
The pipe version needs many more temporary files. You can inspect this quickly with the strace utility.
The pipe version use a quick exploding number of temporary files:
for i in {1..200000} ; do echo $i ; done |strace sort -n |& grep -e 'open.*/tmp/'
open("/tmp/sortb9Mhqd", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
open("/tmp/sortqKOVvG", O_RDWR|O_CREAT|O_EXCL, 0600) = 3
open("/tmp/sortb9Mhqd", O_RDONLY) = 3
open("/tmp/sortqKOVvG", O_RDONLY) = 4
The file version doesn't use temporary files for the same data set. For bigger data sets it use extremely less temporary files.
for i in {1..200000} ; do echo $i ; done >/tmp/TESTDATA ; strace sort -n /TMP/TESTDATA |& grep -e 'open.*/tmp/'
Author by
yboren
Updated on September 18, 2022Comments
-
yboren over 1 year
I have tried this shell script on a SUSE 10 server, kernel 2.6.16.60, ext3 filesystem.
The script has a line like this:
cat file | awk '{print $1" "$2" "$3}' | sort -n > result
The file's size is about 3.2G, and I get the following error message:
File size limit exceeded
.In this shell,
ulimit -f
is unlimited.After I change script into this:
cat file | awk '{print $1" "$2" "$3}' >tmp sort -n tmp > result
the problem is gone.
I don't know why, can anyone help me with an explanation?
-
Admin over 11 yearswhat happens if you get rid of the cat? awk is quite capable of reading a file without cat's help.
awk '{print $1" "$2" "$3}' file | sort -n > result
. How much RAM & swap do you have? 64-bit or 32-bit system? -
Admin over 11 yearsDon't crosspost, stuff like this tends to happen. Closing here
-
-
jw013 over 11 yearsBut how does more files relate to "File (singular) size exceeded"?
-
Mårten over 11 yearsMore temporary files mean also more stress to the partition containing
/tmp
. -
yboren over 11 yearseven if i use the sort -T option to specify a big-enough-directory ,the error still happen