Terminal does not accept pasted or typed lines of more than 1024 characters

Question

When I enter text on stdin in an OS X Terminal, a single line is limited to 1024 characters. For example, cat > /dev/null beeps after I type (or paste) a line longer than this, and refuses to accept more characters. A problematic example is when I want to count characters from pasted text with cat | wc -c: the cat blocks at the first long line.

This seems to be a general problem with pasting to stdin. Can this observed stdin limitation of 1024 characters per line be removed or pushed to a higher limit?

I need this because I want to paste text that has lines longer than 1024 characters.

I could also use a "heredoc" << EOT and paste my long lines without any problem, but then the text appears in my shell history, which I don't want.

Solution

Probably a limit of the terminal device line discipline internal line editor buffer.

You should be able to enter long lines by pressing Ctrl+D in the middle of it (so the currently entered part be sent to cat and the line editor flushed), or by disabling that line editor altogether.

For instance, if using zsh:

STTY=-icanon cat > file

Note that then you can't use Backspace or any other editing capability. You'd also need to press Ctrl-C to stop cat.

With other shells:

s=$(stty -g); stty -icanon; cat > file

Followed by:

stty "$s"

Or just:

stty -icanon; cat > file
stty sane

Of course, things like

cat | wc -l

or

wc -l

won't work. Because Ctrl+C kills all the processes in the foreground process group.

You could do:

STTY=-icanon cat | (trap '' INT; wc -l)

Or as suggested by @mikeserv:

STTY='eol " "' wc -l

That way, the buffer will be flushed every time you enter space. You're still in canonical mode, so you can still edit words (as opposed to lines) and use Ctrl+D to signify EOF.

Or:

STTY='-icanon min 0 time 30' wc -l

EOF will come 3 seconds after you stop typing. Or:

STTY=-icanon sed -n '/^EOF$/q;p' | wc -l

And enter EOF (the 3 letters on a line on its own) to end the input.

As suggested by Gilles, where possible (as in generally not a telnet/ssh session for instance), use pbpaste instead of pasting. (That's on OSX; under X11, call xsel or xclip.):

pbpaste | wc -l

That will also avoid problems with some control characters (like ^C) that may be found in the copy-paste buffer.

Comments

[Admin]

That's annoying. I guess the Heredoc approach is a tolerable work-around, since you can tell the shell to not add a command line that starts with a space. See Is there any way to keep a command from being added to your history? for details for various shells, including zsh.

[Admin]

This is a badly phrased question. In the Z shell the limit is not 1024 characters. You're actually talking about a limit that is not implemented by the Z Shell, nor in force when the Z shell is the program that is reading input from the terminal. In this case, it is cat that is doing the reading, and the Z shell isn't involved. This ground has been covered before, with cat, at unix.stackexchange.com/questions/131105 .

[Admin]

Call pbpaste instead of pasting. (That's on OSX; under X11, call xsel or xclip.)

[Admin]

@JdeBP My question was obviously not about zsh but about the cat command not working like I needed: I was giving details about my environment (zsh, OS X Terminal), in case they mattered.

[Admin]

@Gilles Your solution deserves to be put in one of the answers. I will let you do it if you want, otherwise I will do it: it can really be useful.

[Admin]

@Gilles A pbpaste/xclip solution should mention the fact that it does not work on a remote host, since it does not have the same clipboard as the local host.

[Admin]

you can use a clipboard manager as mentioned, or you can switch to non-canonical input as mentioned, or you can try a program that can do that stuff for you. You might have luck with less -f /dev/tty | out or better with ex - ^Mo^M<paste>^[:wq! where the ^escapes in the last stand in for real keypresses. I think less should basically set the terminal to raw and handle its own line wrapping - and definitely ex/vi/open do... well, definitely vi/open do (ex is definitely canonical).

[Admin]

@mikeserv Thank you for the ideas. The less - f /dev/tty does not work, for me, unfortunately. Another idea, though, for remote host pasting, is to uuencode first (so as to get short lines).

[Eric O Lebigot]

This does indeed allow me to paste long lines. However, it does not work for the general case of pasting to stdin: for example, STTY=-icanon cat | wc -c does not produce any result (I guess that the necessary Ctrl-C kills wc).

[mikeserv]

@EOL - Well, i figured wc was waiting on the end of a line. INT and TSTP will both flush i/o - but TSTP can be recovered. CTRL-Z + fg may help - but can you not just do CTRL-J? (it's the newline char). Alternatively you can: cat | ( trap '' INT; wc -c), but that assumes wc is well-behaved enough to die at the end of its input - probably a safe-assumption, though, given its purpose. dd, as mentioned, will always write on INT, and tee can be told to -ignore interrupts. Honestly, your heredoc idea was best, though - look at HIST_NO_STORE in man zshall. shly: set -o nolog.

[Eric O Lebigot]

Thank you for all the ideas. Ctrl-J does not quit cat, so wc is never run (the pasted text already contains many newlines, so it is fortunate that cat does not quit on newline :) ). Your trap idea works (after STTY=…). It's a little bit involved, for simply counting characters, though. On a local machine, pbpaste | wc -c and friends is good.

[Stéphane Chazelas]

@EOL, see edit for other alternatives.

[mikeserv]

@EOL - i dunno.. i don't think there are newlines in the cut buffer - which, i think, is the source of your problem. I think probably there are line-feeds or returns - or, actually, since i think Macs only honor returns by default, maybe the source of the problem is reversed? Anyway, if there are end-of-line chars in the cut buffer the canonical terminal should not hang as it does - each \n/\r (whichever) should flush the buffer when pasted. Switch wc to -l and see what it reports, also consider setting a literal stty eol <char> and/or i(cr|nl)(nl|cr) as appropriate.

[mikeserv]

@EOL - I wrote an answer here about how the stty eol setting might be used in some situations to overcome a canonical line-discipline's maximum line-length. It may be applicable to your problem as well.