Can I increase the system pipe buffer max?

Question

I'd like to make a FIFO pipe with a buffer of ~5MB. I know the default FIFO pipe buffer max in linux is around 1MB. I see that it lives in /proc/sys/fs/pipe-max-size

I tried to set it as follows:

sudo sysctl fs.pipe-max-size=4194304

Then I see that the value has indeed been changed:

$ cat  /proc/sys/fs/pipe-max-size
4194304

Then I created a new FIFO pipe, but I didn't notice any improvement in performance. It seems to fill up at the same speed as the previous 1MB FIFO pipe. So I'm unsure that my new FIFO pipe actually has a 4MB buffer.

How do I 1) increase the system FIFO pipe buffer max and 2) Create a FIFO pipe that uses this buffer max?

Solution 1

Your command changes the maximum buffer size, not the default one.

From the pipe(7) manpage:

/proc/sys/fs/pipe-max-size (since Linux 2.6.35)

The maximum size (in bytes) of individual pipes that can be set by users without the CAP_SYS_RESOURCE capability.

and:

Since Linux 2.6.11, the pipe capacity is 16 pages (i.e., 65,536 bytes in a system with a page size of 4096 bytes). Since Linux 2.6.35, the default pipe capacity is 16 pages, but the capacity can be queried and set using the fcntl(2) F_GETPIPE_SZ and F_SETPIPE_SZ operations.

Thus, unless you call the fcntl(F_SETPIPE_SZ) system call on the opened pipe, it will stay at its default capacity: 64 kB. For this, you must use a language that offers a binding to syscalls (C/C++, Python, PHP, perl,... but not sh/bash).

Solution 2

Following on from xhienne's answer, this perl script will set the size of an existing open fifo:

#!/usr/bin/perl
# usage: name-of-open-fifo size-in-bytes
# http://unix.stackexchange.com/a/353761/119298
use strict;
use Fcntl;
my $fifo = shift @ARGV or die "usage: fifo size";
my $size = shift @ARGV or die "usage: fifo size";
open(FD, $fifo) or die "cannot open";
printf "old size %d\n",fcntl(\*FD, Fcntl::F_GETPIPE_SZ, 0);
my $new = fcntl(\*FD, Fcntl::F_SETPIPE_SZ, int($size));
die "failed" if $new<$size;
printf "new size %d\n",$new;

Put this in a file, say ~/setfifo, do chmod +x on it, and run it after you have created and opened your fifo, eg:

$ mkfifo /tmp/fifo
$ cat -n <>/tmp/fifo & 
$ ~/setfifo /tmp/fifo 1048576
 old size 65536
 new size 1048576

---

If your perl does not yet have the constants F_GETPIPE_SZ and F_SETPIPE_SZ, you can just use the appropriate numbers found by looking through the C files in /usr/include/. They are respectively 1024+8 and 1024+7. Here's the resulting perl script:

#!/usr/bin/perl
# usage: name-of-open-fifo size-in-bytes
# http://unix.stackexchange.com/a/353761/119298
use strict;
# int fcntl(int fd, int cmd, ...) F_GETPIPE_SZ,void F_SETPIPE_SZ,int
# /usr/include/asm-generic/fcntl.h #define F_LINUX_SPECIFIC_BASE 1024
# /usr/include/linux/fcntl.h #define F_SETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 7)
sub F_SETPIPE_SZ{ 1024+7; }
sub F_GETPIPE_SZ{ 1024+8; }
my $fifo = shift @ARGV or die "usage: fifo size";
my $size = shift @ARGV or die "usage: fifo size";
open(FD, $fifo) or die "cannot open";
printf "old size %d\n",fcntl(\*FD, F_GETPIPE_SZ, 0);
my $new = fcntl(\*FD, F_SETPIPE_SZ, int($size));
die "failed" if $new<$size;
printf "new size %d\n",$new;

Comments

[user2989813]

I should add I'm running the latest Debian Jessie if that matters.

[user2989813]

Thanks for the advice. Now I know I need to set the size of an actual opened FIFO pipe. I tried to run: perl -MFcntl -e 'fcntl(STDIN, 1031, 1048576) or die $!' <> /dev/xconsole but got -bash: /dev/xconsole: Permission denied. Do you know why?

[meuh]

It seems that Debian Jessie perl is at version 5.20.2 at the moment, and I was using a more recent 5.22.2 on Fedora, so you do not yet have these constants by name. I've updated my answer with a version which should work for you.

[rph]

The following page shows a solution using python: golinuxhub.com/2018/05/…

[rph]

@meuh If I run your script twice, in both runs I get: "old size 65536". In the second run the pipe size should be initially 1MB. Any reason for this?

[meuh]

@rkioji It works for me, the 2nd run has the new big size as "old". Make sure you dont close the fifo between the 2 runs.