Heredocument error I can't understand ("delimited by end of file")

Question

I execute the following code in Bash:

bash /dev/fd/10 10<<-'SES'

    cat <<EMR >> /etc/apache2/apache2.conf
        #
        <Directory /var/www/>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
        </Directory>
        EMR

SES

But the execution breaks and I get:

warning: here-document at line 2 delimited by end-of-file (wanted `EMR')

• I don't understand what the error means and I already added - between the operator and the name of the containing heredoc to allow tabulations.
• I made sure there are no spaces after the opener (operator+name) or after the delimiter.

Note: If you copy the code above to test it in your computer, you need to change all leading spaces to tabs (tabulations) as StackExchange changed the original tabulations to spaces.

---

Possible causes to rule out:

There are several possible causes for this, here are some I've ruled out:

1. Problematic hidden characters:

I edit the file with Notepad++ and I turned on the "Show all symbols" mode to make sure all indents are tabulation based (see red arrows) and that all End Of Line chars (EOLs) are LF unix based (there aren't any CR chars):

Moreover, the encoding is UTF-8, so it's not an encoding problem.

2. Pseudo dash symbol (<<-):

It seems as the added hyphen (<<-) doesn't do its job of stripping all leading tabs because when I manually delete all leading tabs whatsoever (or all leading tabs before each delimiter) the heredoc does work as expected. One can suggest that the hyphen isn't really a dash symbol, but why won't it be? I have no way to validate that.

3. Bash bug:

Other people not-using Windows, and using different Bash-containing Linux distros didn't have this, so this is most likely not a Bash bug. I've also tested this with the Bash development section at GNU.

---

Corrupted pasting of copied data from Notepad++ might be it:

If I paste the heredoc from Notepad++ into a nano file, it seems the leading tabs are all in place - they aren't deleted are transformed into spaces in pasting.

Moreover, cat script.sh | grep "^ " and cat script.sh | grep -x '\s*EMR' (when done in the files directory), come out empty.

Yet, in a later pasting I found another corruption problem that is most likley to cause this (see my answer).

Solution 1

The problem undoubtedly originates in Windows. It deals with the TTY window-size; The narrower the window size is, the more Line Feed (LF) char and Carriage Return (CR) char combos will be added in pasting per the Window's narrowness. Such combos are represented as CRLF.

Solution

If I understand Nano's maintainer Benno Schulenberg correct, Nano will naturally convert CRLF combos into Trailing Whitespace chars (^J chars).

As suggested to me by Benno, adding the following code in the end of /etc/nanorc solved it:

bind ^J enter main

Clarifying the solution

On the one hand, it will disable formation of ^J chars that break the code; on the other hand, it will add only Line Feeds (LF chars), applicable in Linux (unlike the Windows CRLF combos), to the copied data.

The solely added LF chars are good as they prevent the pasted data to appear as one long line of text.

---

• I also reported this problem in length to the WSL development team. WSL is the Windows Subsystem for Linux project.

Solution 2

The image you posted shows carriage returns (CR in the image). This is a DOS-formatted text file. Use dos2unix or some similar utility to convert the file to a proper Unix text file.

As per comment below: If you absolutely need to do development in Notepad++ on Windows (I would personally recommend doing the development on the same OS that you're targeting, if at all possible), then the program may be told to save text files with Unix newlines by selecting "Unix (LF)" in the "New Documents" preferences:

Make sure that your file has a newline (LF) at the end of the last line. All lines in a Unix text file (and a shell script is a text file) needs a terminating newline at the end. If the last line doesn't have one, the file is technically actually not a text file but binary.

See also, for example, the answers to "Remove ^M character from log files"

Solution 3

Unfortunately, the design of terminal emulators (such as the terminal window on your Ubuntu box) mostly doesn't allow for good paste support; in particular, for the most part, paste works exactly like typing the same thing (if you were a very quick typist). Most stuff you'd think is some special signal to the program running in the terminal is actually in-band data—that is, just a character. If your paste contains that character, the program can't tell the difference between you typing the character and pasting it.

So you can't really use paste to transfer files around, unless they contain no characters that could confuse the receiving program. In particular, the tabs your paste contain are confusing bash—it's seeing a tab and attempting to do file or command completion. Just like when you type a tab.

The cat > filename trick may well fail if there were a control-D (yes, that's a character) in the pasted contents. But if you avoid control characters (other than newline and tab), it should work.

Unix folks typically use file transfer programs to move files around. The basic one we mostly use is scp or sftp, both of these are part of SSH (and are thus fully encrypted, authenticated, etc.—same security guarantees as SSH provides).

The easiest way of working between Windows and an Ubuntu VM (or vice versa) is probably to use your hypervisor's (VM product's) file sharing features—most have them. You can pick a directory on the host, and it'll be visible inside the guest. With WSL, you can access all your Windows files under /mnt/<drive letter>/. If your hypervisor doesn't have that feature, the second-easiest is to use an ordinary file share. You can mount Windows file shares from Ubuntu, or install Samba on Ubuntu to make file shares to Windows.

Depending on what you're doing, there may be better approaches. For example, automated deployment. Maybe you want to have it some that you check the file in to git, and then it's automatically deployed on your Ubuntu box when you do a git push? This can be as simple as using git hooks, or much more complicated—there are entire software packages dedicated to it.

Finally, there is a big advantage to writing these scripts to files, not just pasting them in to a shell: the file serves as documentation of what you did. Did something go wrong? Unless it went really wrong, check the file to see why. That lets you determine exactly what went wrong, often the first step to repairing the damage. Need to do it again? Just source the file a second time. Need to do something similar? Edit the file, then source it.

If you must paste a here-doc:

If you want to paste in to bash directly, the best bet is to make sure it contains no control characters other than newline. Pasting this should work:

bash /dev/fd/10 10<<'SES'

cat <<EMR >> /etc/apache2/apache2.conf
#
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride All
Require all granted
</Directory>
EMR

SES

If you must have the indentation, you can use spaces and sed instead of the leading - to strip it (note here that went ahead and used stdin instead of fd 10 because the syntax is easier and I doubt it matters):

sed -e 's/^\s\+//' <<'SES' | bash

    cat <<EMR >> /etc/apache2/apache2.conf
    #
    <Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
    </Directory>
    EMR

SES

That uses sed's search-replace functionality (s/PATTERN/REPLACEMENT/) to look at the beginning of the line (^) for whitespace (\s) repeated one or more times (\+) and replaces it with nothing. In other words, it trims off all leading spaces on each line. The -e option to sed explicitly identifies this is a sed script to run—it's not actually required, sed will interpret its first argument as script anyway—but I just personally find it clearer to always use -e.

Note that "whitespace" above includes spaces and tabs (and a few more characters that aren't really relevant here). That sed will trim them all off, regardless of the mix used.

Of course, I have no idea why you're pasting a command in to bash telling it to run bash to run cat.

Comments

[phemmer]

Works for me (as @ikkachu mentioned, using tabs, not spaces).

[Admin]

They are, but they are only recognizable to those who can recognize them at the time. I didn't know cr is necessarily a "Carriage return" or even not part of an EOL of some kind, nor did I ever expected that Notepad++ (which is the most usable text editor I know in the Windows world) would include these for files with the .sh extension. I believe I would further investigate this if there weren't answers that fast and learn it (BTW, I found these chars only after turning on the hidden character display - In Notepad++ it's under View >> Show symbol >> Show all characters).

[Admin]

I hope this Q&A session will also be a good lesson for Notepad++ users facing the same problem and lacking the same knowledge I lacked before reading Kusalananda's answer.

[tripleee]

Is there any reason to pass the script to Bash as a here document?

[Admin]

There is, but I have no desire to discuss this here.

[Kusalananda]

This question is more and more hard to follow. I suggest that you rather than adding further updates to the question (or carry on conversations in various comments), consolidate it into a coherent unity. At this point in time I'm not even sure it's on topic for this forum.

[derobert]

Your "something interesting" is indeed interesting. Could you describe the exact steps you're taking to run this on your Ubuntu box? Also, if this isn't the process, can you try: 1. write it to a file; 2. run it with source filename; 3. if it doesn't work, provide the output of od -c filename

[derobert]

@Benia Let me check my understanding—what you're doing is copying the text from Notepad++, opening a terminal on your Ubuntu (VM I guess?), and pasting it there? And sometimes that works and other times it doesn't? If that's correct, before pasting to the terminal, run cat > filename, then paste, then type control-D (if a prompt doesn't appear after control-D, try a second control-D). That should write whatever you pasted to filename. You can then immediately try source filename to run it.

[derobert]

(Also, you could try od -c script.sh to see if anything funny got in there).

[Admin]

Yes that's what I do, and yes it only sometimes happened. I now did what you said in two different systems (cat > filename [Paste content and click CTRL+D] && source filename, in both cases it did worked. That's really interesting @derobert.

[derobert]

@Benia bash (or whatever shell you're using) does various processing of incoming tab characters you type (e.g., for command and filename completion)—it might be corrupting them. Pasting to a terminal often looks exactly like typing to the program running in the terminal.

[Admin]

Yes, it is Bash (I now edited the question and declared that). Responding to your last comment, is there a faster way to use this "safe mode" you showed me (cat > filename [Paste content and click CTRL+D] && source filename) to run the heredoc without such random corruption? Please publish an answer with that, not only it's a milestone, it's really the basis for a working answer.

[ilkkachu]

@Benia I wouldn't call tab completion "corrupting" input any more that it's "corruption" that inputting ^A^K deletes the line you so carefully entered. An interactive shell will have features meant for interactive users, not for entering arbitrary data. There are some ways to detect pasted data from typed data, but they require support from the application and possibly the terminal. In general it's better to just not use an interactive shell if you don't want the interactive features.

[Kusalananda]

@Benia It may be that, or it isn't a real dash (minus). In general, I'd advice to edit Unix text files on a Unix system.

[Kusalananda]

@Benia I'll retract that comment since you say that it does work when you manually remove the tabs. The inner here-document shouldn't need the dash though, right?

[Philippos]

@Benia Your script works here with tabs, but not with spaces. Did you check whether Notepad++ does something like expanding tabs to spaces on writing and vice versa on reading? That way you think there are tabs, but there are spaces indeed. Check with grep "^ " script.sh

[Philippos]

@Benia I'm on debian 8. If you get nothing, it's not spaces instead of tabs. Another try: what about grep -x '\s*EMR'?

[derobert]

@Benia edited, I think that answers all your questions. You might find looking at the revision history easier than re-reading the whole thing. Also, maybe you should ask a question about your logistical reasons—possibly they're fixable. I don't anything about them, so I can't really advise there. You could try our chat room to see if people think they'd make a good question. chat.stackexchange.com/rooms/26/dev-chat

[derobert]

@Benia last solution contains no tabs, only spaces. The ^ in the sed pattern means beginning of line. So that's one or more (\+) whitespace characters (\s) at the beginning of the line (^). (sorry for brief response, using SE phone app)

[derobert]

@Benia no need for a bigger bounty, I think I've got enough worthless Internet points already. Tabs don't work because bash does special things with them. Note how when you type a space in the shell, you get a space. Type a tab, you don't get a tab.

[derobert]

@Benia BTW: I think part of the problem is I've been confused about how much you know about shell scripting—that bash /dev/fd/10 10<<-'SES' bit isn't something I'd expect a novice/beginner to come up with. But I guess maybe that's copy & paste? Or you just somehow never came across sed.

[derobert]

@Benia I don't know where to find a tutor, unfortunately (and I'm not available and further doubt I'm really qualified to be a tutor, I have no experience as an educator).

[derobert]

@Benia it strips off all whitespace (which includes tabs). So if the tabs get through, it's fine with that. Same if only some of the tabs do, or if you use a mix of spaces and tabs. Though if bash's tab handling breaks your paste in others ways (e.g., completes something) it'll probably appear to work, but leave you with unexpected lines in your Apache config.

[derobert]

@Benia From your original problem, that it sometimes works and sometimes doesn't, it seems that bash is sometimes doing something to the tabs. (I'm personally surprised it sometimes works). The normal thing bash does with tabs is tab completion—for example, try pasting readl<TAB> in to the terminal. At least here, I get readlink instead—bash has performed tab completion. What I'm worried about isn't that you'll get partial execution, but that you'll get some weird corruption from bash doing a tab completion. I'm not entirely sure it'll happen, but avoiding it is easy (just use spaces).

[Admin]

Would you believe this if I had told you? Now it doesn't work with the code you gave (same error given).

[derobert]

@Benia doesn't work with tabs or with spaces? If with tabs, well I was surprised it ever worked... If with spaces, that is weird. Either way, I wonder what changed...

[Admin]

I didn't try with spaces, I usually work with tabs.

[Admin]

Dear @derobert, please see my answer.

[Admin]

I found it!!! I found the problem finally!!! Please see my new answer. This is it!!!

[Admin]

Your aiming was right --- These are CR chars created by Windows as a response to the TTY window size. Please see my answer below if you want to expand on this Windows behavior.