Pipe buffer size is 4k or 64k?
Solution 1
The other answers tell you that the pipe size is 64 KB. The reason why PIPE_BUF is 4KB is that PIPE_BUF is the largest size for which writes are guaranteed to be atomic. See http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html
Solution 2
It's programmable now
As of Linux 2.6.35 you can use fcntl(2) with the F_SETPIPE_SZ
operation to set the pipe buffer up to /proc/sys/fs/pipe-max-size
. This is by default 1 MB; see proc(5).
Solution 3
In my experience, the single write test produced a total size of 65536, yet when I wrote 2700 at a time, I could only write 16 times, and then the next attempt stalls. I figure that the 'atomic' write needs to be within one 4K block, and that for each of my writes, it goes to the next full block to satisfy the request. So, in effect, the useable pipe size depends on the size of your writes.
Solution 4
Looks like the kernel use up to 16 buffers which adds up to 64k. See this link for an explanation of the ulimit output vs actual buffer size
Gui13
Updated on July 09, 2022Comments
-
Gui13 almost 2 years
I read in multiple places that the default buffer size for a pipe is 4kB (for instance, here), and my
ulimit -a
tends to confirm that statement:$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 15923 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 // 8 * 512B = 4kB POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
But when I use a little program to test the buffer size (by writing into the pipe until the write() blocks), I see a limit of 64kB!
See this program:
#include <stdio.h> #include <unistd.h> #include <limits.h> int main(void) { int tube[2]; char c = 'c'; int i; fprintf(stdout, "Tube Creation\n"); fprintf(stdout, "Theoretical max size: %d\n", PIPE_BUF); if( pipe(tube) != 0) { perror("pipe"); _exit(1); } fprintf(stdout, "Writing in pipe\n"); for(i=0;; i++) { fprintf(stdout, "%d bytes written\n", i+1); if( write(tube[1], &c, 1) != 1) { perror("Write"); _exit(1); } } return 0; }
And its output:
$ ./test_buf_pipe Tube Creation Theoretical max size: 4096 Writing in pipe 1 bytes written 2 bytes written 3 bytes written 4 bytes written [...] 65535 bytes written [blocks here]
It strongly suggests that the pipe buffer size is actually 64k! What is happening here??
-
Gui13 over 13 yearsRah, same link as the one I pointed in my question. Didn't see the line stating 64k... Anyway why ulimit states 4K is a mysery to me...
-
Gui13 over 11 yearsYes, but no. The pipe size is advertized by
ulimit
is still 4kB on Kernel 3.2 (the one I have here). My question was about PIPE_BUF, which is what janneb answered: PIPE_BUF is a constant and is the maximum size of a write which will be atomic when performed (i.e., thread-safe ;)). -
Stephane Chazelas over 8 yearsOnly ksh and bash ulimit builtin report that pipe size, but it's a not a limit as set by setrlimit, and it's not the size of the pipe buffers,it's just PIPE_BUF, the maximum size for which a write() to a pipe is guaranteed to be atomic.
-
user1767754 over 6 yearsDoes this mean by setting more then 4KB you can still guarantee that the write is atomic?
-
Subomi almost 6 yearsDoes this imply that the buffers can go up arbitrarily? But the guaranteed atomic write is limiited to 4K?