Trailing spaces when copying from console

whitespace terminal vim tmux clipboard

11,457

Solution 1

You'll have spaces at the end of the line when selecting and copying from the terminal if the application displayed spaces at that spot. Applications may display spaces in order to erase what was there before. Terminals have commands to delete a whole line or delete characters to the right of the cursor; applications choose between that and displaying spaces based on what they consider most efficient. For example, if you type some stuff at a prompt, then press Backspace, the application (e.g. the shell) is likely to overwrite the last character with a space.

If you have an X11 connection, you can use xsel or xclip to copy a file to the local clipboard.

Experimentally, Vim seems to go through pains not to display lines ending with spaces (even when the buffer contains a line that ends with spaces). So copying from that is an option if you don't have an X11 connection.

An alternative would be to post-process after copying:

xsel | sed 's/  *$//' | xsel

Solution 2

That's actually several questions, and none of the answers do more than skim over the interesting parts:

when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line.
When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely.

Most terminals (like xterm) store the data which you can select on the screen. There's no hidden part behind it which tells the terminal that the application meant to fill in a background.

Applications update the screen by moving the cursor around, writing text (which may include actual spaces — or tabs which the terminal renders as spaces) and erasing parts of the screen.

Erasure is a special problem (for select/paste), since many terminals (such as xterm) will fill the erased area of the screen with the current color (the back color erase (bce) feature in the terminal description). But at the same time, erased areas no longer have characters stored in those positions of the screen. Terminals which use the on-screen representation will let you select everything except for the erased area. (As a special case, your terminal could have erased regions surrounded by text, which it will pretend are spaces for selection).

All of that's a nuisance, and long ago xterm provided a feature for ignoring the trailing spaces. Most other terminals don't provide that. And since it's an option, it may not be turned on if you happened to run xterm. Most terminal developers that copy xterm's features don't copy the options.

Going to tmux: it doesn't support the bce feature. Its developers decided not to. So normal applications running in tmux will produce trailing blanks. When you override the terminal description, it confuses tmux, since it doesn't know that vim is really assuming that the background will be cleared using the current color. You just get the erasure. No color.

On the other hand, GNU screen's developers decided a while back to support the feature. It's optional...

Solution 3

Copy & pasting from a terminal screen is never going to be fully reliable because it's dealing with screen output instead of original source material. If some applications echo text to the terminal in an unusual fashion and it results in the terminal not being able to guess what the original text was, there's probably not much you or the terminal can do about it.

A lot of information about the original text is potentially lost when it is rendered to a terminal: for example whether an area of white-space was produced by a tab or by a series of spaces, or whether two rows of text were originally one long line that got wrapped or two separate lines.

The terminal tries to do its best to let you copy & paste the original text that got echoed to the terminal but it cannot always know.

As an experiment, try this:

Use less to view a file that contains very long lines that wrap across multiple terminal lines.
Triple-click one of those lines (for select-whole-line). less selected the entire logical line that spans multiple physical lines and if you pasted it elsewhere it would be preserved as one long line.
Press j some number of lines so that part of that long line disappears off the top of the screen.
Press k one or more times to scroll the whole logical line back into view.
Triple-click the logical line again. This time only one physical line is selected. That is because less repainted the screen physical line by physical line and the terminal no longer has any way of knowing that the physical lines were connected together.
Now if you were to drag-select the entire logical line manually and paste it elsewhere, you would find that it has embedded newlines.

YMMV on that experiment because your terminal (or version of less) might be more or less clever than mine.

Generally, the dumber the software that produced the output, the better your chances are of being able to copy & paste exactly the original material. cat, for one, is pretty much as dumb as it gets.^{(You understand of course that "dumb" is a compliment!)}

If you get an extra space character at the end of a line under tmux, it's probably because tmux is actually echoing that character. Remember that tmux does its own terminal emulation and then re-emits new terminal sequences to render to the underlying terminal. Perhaps it is echoing that space character because it believes there might be a need in some cases to overwrite some other character it thinks might be there. Whatever the reason, the terminal probably has no way of knowing that space character is not really part of the original content.

Solution 4

In a lot of modern Terminal applications there are settings like "Trim trailing spaces" in mouse/copy options. Those are present for Konsole (KDE) & Yakuake.

Solution 5

Using Fedora 17 Linux and the konsole terminal window, I open vim with the syntastic vim add-on, I was able to reproduce this error consistently. I put these three python lines in vim:

a = "generic assignment"
b = "cursor is on this line"
c = "generic assignment"

Then I visually select those lines and paste it here:

a = "generic assignment"
b = "cursor is on this line"                                                                                     
c = "generic assignment"

Notice how the second line has tons of spaces copied over. That sure gets annoying.

Lines 1 and 3 are copied as expected, but the second line copies spaces until the rightmost spot of the terminal window.

Work-around Solution:

Move the highlighted line (the vim cursor) away from the lines you want to copy before selecting it with the mouse. Then the extra spaces don't appear on that one line.

I'm guessing it is the syntax and color highlighting add-ons that are causing these issues.

View more solutions

11,457

x-yuri

Updated on September 18, 2022

Comments

x-yuri over 1 year
It's this annoying behavior I've been experiencing here and there occasionally: when you select text with mouse in console (that is, copy it), paste it, and realize you've got extra spaces at the end of each line. That is,
```
line 1                                                                          
line 2                                                                          
```
instead of
```
line 1
line 2
```
So, not just one space at the end of each line.

I couldn't reliably reproduce the issue, and couldn't find the answer. I believe with some software it manifests itself only after a while.

But I've just noticed, that when I open the same file in vim, first right from console, then from tmux, it works out well in the former case. And doesn't in the latter one. Considering TERM=xterm-256color in console, and TERM=screen-256color in tmux, my conjecture is that it has to do with terminal not doing it properly, or not enabling applications to do it properly. Quite a vague conjecture, I presume. So, the first question is, "What exactly is causing it?"

And the other one is, "How do I go about it?" The worst case is when the file is located remotely. I used to copy it locally and open it with gedit lately. Now I supposedly have an option to open it in new console (since I'm mostly working in tmux sessions), and copy from there. Could this be done any simpler?

When I run vim from tmux with TERM=xterm-256color vim, it behaves strangely. Like not drawing background where there is no text. And it doesn't seem okay to me to change TERM variable (making software think it's dealing with other terminal).

When editing local file, I usually do :!gedit %.
x-yuri almost 9 years

You made me come up with a workaround: cat a file, and copy the output of cat. At least in my particular case it works. Thanks :) But if it's a long file... Well, I was thinking about mounting remote filesystem (sshfs), then I can edit a file with gedit for example as if it were a local one.
x-yuri almost 9 years

So copying from that is an option if you don't have an X11 connection. Not with screen-like terminal emulators, AFAICT (TERM=screen*). Speaking of xsel, good point, I'm now thinking of creating a vim command, which outputs shell command to be issued locally, like :XSelCommand % -> ssh server cat path/to/file | xsel. That seems better, than mounting remote filesystems, or copying files locally.
x-yuri almost 9 years

So, with a command like this: command! XSelCommand echo 'ssh ' . system('hostname')[:-2] . ' ' . shellescape('sh -c ''cat "$1"'' -- ' . shellescape(expand('%'))) . ' | xsel', or like this one: command! -range=% RXSelCommand echo 'ssh ' . system('hostname')[:-2] . ' ' . shellescape('sh -c ''cat "$1" | sed -n "<line1>,<line2>p"'' -- ' . shellescape(expand('%'))) . ' | xsel', one can copy files' contents, not what was output. The easier option, however, would be rather to open the file in local vim over ssh and do :%!xsel. Or do :r!ssh HOST CMD, and again :%!xsel.
x-yuri over 8 years

I've managed to run into two issues with xsel so far: 1) :%!xsel doesn't work in vim, but :%!xsel -i does work, supposedly because vim redirects stdout as well as stdin; one more option is :w !xsel, 2) copying files over 4000 bytes doesn't work when pasting to chrome's textarea for example, but works for gedit; the tab hangs for 10 or so seconds, but nothing is pasted after that; do you know by any chance what kind of magic number that is?
x-yuri over 8 years

...And nothing of the sort happens when copying from gedit to chrome with middle mouse button.
Gilles 'SO- stop being evil' over 8 years

@x-yuri I don't know why there would be a size limit. Try changing xsel/middle-mouse-button to xsel -b/Ctrl+V, that uses a slightly different selection mechanism which may be differently buggy.
x-yuri over 8 years

Actually, the same happens with xsel -b, so using gedit for recopying the text is the only workaround so far.
Gilles 'SO- stop being evil' over 8 years

@x-yuri I've can reproduce this problem when copying text with xsel -b and pasting into Chrome. I don't know what causes it, probably a Chrome bug since other programs don't have this problem. But this has nothing to do with this question.
x-yuri over 7 years

The general idea seems to be clear, but I might not be understanding the particulars of some statements, or not sure if I do, like... Terminals which use the on-screen representation What are other types of terminals? Are they covered in your answer? As a special case, your terminal could have erased regions surrounded by text, which it will pretend are spaces for selection Here you probably mean "marked as erased". And what kind of regions are these? Spaces in the middle of the line? Empty lines surrounded by text? These are probably the least clear ones.
Marius over 7 years

Other types: applications (such as curses) which do higher-levels "know" if they're clearing the whole screen or repainting a line, and have the data to reconstruct things in more than one way. "Marked as erased": for example xterm may show a blank for erased areas, but actually has no character.
Marius over 7 years

fwiw, "screen" and "tmux" are common examples of applications which can manipulate the terminal so that selection behaves differently.
Matija Nalis over 5 years

If data you want to copy is in the file, you can use xsel -bi < filename to safely copy all of it... Use recent version of xsel, though
x-yuri about 5 years

Occasionally I get less lines then in the original (newlines doesn't always propagate). You might want to check this after doing the replacements.
zentenk over 4 years

I've finally found the right working command: with Mint, when you select with the mouse under vim, your solution doesnt work. This one works xsel -b -o | sed 's/[ \t]*$//' | xsel -b (= using -b option to operate on the clipboard selection)!!