Bash function that accepts input from parameter or pipe

Question

I am working with an ASP.NET web application written in C# deployed to IIS/Windows Server 2008 R2. The application uses NHibernate to interact with an Oracle database running on a networked unix server.

It seems that writes being made by the application to the database have no effect.

If I manually edit the value of a record in the Oracle database, the new value is reflected by the application. However, if I attempt to change a value using the application’s custom “save” functionality, the changes are not reflected in the database. It seems like reads are succeeding, but writes are failing.

Why do writes seem to be failing?

More information:

No obvious error messages are received (ie. the application does not throw an exception and it seems to continue running as if everything is fine).

Another instance of this application is running on IIS/Windows Server 2003. This instance can write to the Oracle database (ie. the changes can immediately be seen in the database by using a database viewer after clicking “save”).

The code is virtually identical between the 2003/2008 applications. However, on the 2008 server, I am using newer versions of Oracle libraries and I changed to target architecture of the visual studio projects from ‘Any CPU’ to ‘x86’ (the 2008 server is 64-bit while the 2003 server is 32-bit).

Disclaimer:

I have very limited experience working with IIS, NHibernate, Oracle databases, Windows Server 2003, and Windows Server 2008 R2. I do, however, have slightly more experience working with C#, ASP.NET web applications, Visual Studio, and MSSQL databases).

Solution 1

See Stéphane Chazelas's answer for a better solution.

---

You can use /dev/stdin to read from standard input

b64decode()
{
    if (( $# == 0 )) ; then
        base64 --decode < /dev/stdin
        echo
    else
        base64 --decode <<< "$1"
        echo
    fi
}

• $# == 0 checks if number of command line arguments is zero
• base64 --decode <<< "$1" one can also use herestring instead of using echo and piping to base64

Solution 2

Here it should just be:

b64decode() {
  if [ "$#" -gt 0 ]; then
    # concatenated arguments fed via a pipe.
    printf %s "$@" | base64 --decode
  else
    base64 --decode  # read from stdin
  fi
  ret=$?
  echo # add one newline character
  return "$ret" # return with base64's exit status to report decoding
                # errors if any.
}

In any case, do not use base64 --decode < /dev/stdin. At best (on most systems) < /dev/stdin does nothing, it just does the equivalent of dup2(0,0) (duplicating fd 0 onto fd 0, a no-op).

But on Linux or Cygwin, < /dev/stdin does not work properly here. On those systems, opening /dev/stdin is not like duplicating stdin, it reopens from scratch and independently the same file as is currently opened on stdin.

So if previously stdin was pointing in the middle of some regular file, after < /dev/stdin, you'll end up with stdin now pointing at the start of that file. If stdin was the writing end of a pipe (which it shouldn't under normal circumstances), you'll end up with it being the reading end. If it was a socket, then it will fail as sockets cannot be opened. Same if you don't have permission to open that file for reading (for instance because the permissions changed or the file was originally opened on stdin using different credentials).

And after base64 --decode < /dev/stdin returns, the current position of stdin within the file (for seekable file input) will have been left untouched, not left at the end (or wherever base64 stopped reading) since base64's stdin was on a different open file description.

Solution 3

Sundeep's answer works for base64 because that utility does not support multiple lines. A more general fix for the more general case

• target utility allowing multiple commandline arguments
• pipe or redirect containing multiple lines

is something like

my_function() {
    if (( ${#} == 0 )) ; then
        while read -r line ; do
            target_utility "${line}"
        done
    else
        target_utility "${@}"
    fi
}

Solution 4

People are talking about how base64 --decode might really be a placeholder for a more complicated command — perhaps a pipeline or a compound command — but nobody does anything about it (*).  Well, here’s a non-recursive solution in which the main commands appear only once:

b64decode() {
        if [ "$#" -ne 0 ]
        then
                printf '%s\n' "$1"
        else
                cat
        fi | base64 --decode
        echo
} 

The logic is fairly simple:

• If there is an argument, write it to stdout.
  • (printf is safer than echo.)
  • The question doesn’t say what to do if there are two or more arguments, so I’m ignoring that case.
• If not, run a cat.
  • The lonesome cat isn’t useless.  UUOCs are typically characterized by having exactly one filename argument; this one has none.  It connects the input to the function (which is the input of the if statement) to the output of the if statement (which is the input to the base64 –decode statement), much like the cat in this slightly similar answer.

But some people are allergic to cats, so here’s a feline-free version:

b64decode() {
    (
        if [ "$#" -ne 0 ]
        then
                exec <<< "$1"
        fi
        base64 --decode
        echo
    )
}

• This uses a here-string, so it works in bash, and I guess it will work in ksh (ksh93 or higher), mksh and zsh, but it won’t work in other POSIX-compliant shells.
• The logic is similar to the first version — if there is an argument, redirect stdin to a here-string of the argument, so base64 will see it as stdin.
• Otherwise, do nothing (leave stdin alone).
• If the user says b64decode < file.txt, that (obviously) sets the stdin for the b64decode to be the file.txt file.  But, if we do any form of exec <…, that sets the stdin for the entire shell, globally.  So, after base64 has read the here-string, the shell will get an end-of-file (EOF) and exit.  So, we run the entire function in a subshell.  Of course this means that the processing code cannot do anything that changes the shell’s environment.

_______________
(*) Except for kbulgrien, who handled it with recursive code, which strikes me as inappropriate.

Solution 5

Both Sundeep's and TomRoche's answers were illuminating and appreciated, but in a different situation target_utility "${@}" could represent more complex code. Duplicating non-trivial code in both the if and in the else could create unnecessary issues. Though the OP's issue may not present a problem ideally resolved through use of recursion, other reader's problems may benefit from using it, or, from considering use of an wrapper function:

my_function() {
    if (( ${#} == 0 )) ; then
        while read -r __my_function ; do
            my_function "${__my_function}"
        done
    else
        target_utility "${@}"
    fi
}

This answer is one of various possible solutions for a "Bash function that accepts input from parameter or pipe" since the OP indicated [in a comment] that base64 was not the actual problem domain. As such, it makes no attempt to assure input is able to be directly read from a file.

Per various comments, there is concern this and similar other templates may not properly handle lines with leading white space; that IFS = may be required before the read. In fact, presence of any character the shell is sensitive to may require special handling before it hits this function. As-is, this code functions properly where leading white space is known to not occur, and where data sensitivities are pre-handled by the emitter.

This was not just a textbook example; it is used in an actual, production script. target_utility is a simple moniker for the actual code fragment. This answer does not presume to be a universal template without need for additional developer sagacity.

(It seems reasonable to assume users are often expected to occasionally adapt answers rather than expect to use them verbatim.)

View more solutions

Comments

[OldProgrammer]

Who knows. Please show some code of the persistence action.

[Felice Pollano]

As a pure divination, sounds like you are not committing a transaction.

[yohosuff]

With all due respect, the application running on the 2003 server uses the exact same persistence code as that running on the 2008 server, and the writes are definitely working on the 2003 server, so I do not believe it is a problem with the code (unless it needs to be written differently because of the different environment, but that seems unlikely to me). Please correct me if I am wrong about this.

[Alessio]

seems like a pointless function when base64 and bash can do all that anyway. why write a function just to avoid using the -d or --decode option? if you really must have something called b64decode then alias b64decode='base64 --decode'. b64d would be shorter, though and save even more typing.

[tyrondis]

You are right, this was just used as an example, though.

[Pablo A]

Probably duplicate on stackoverflow

[tyrondis]

Thanks! Can you explain the advantage of heredoc here?

[Sundeep]

just a clean syntax imo, echo and pipe might be faster.. see unix.stackexchange.com/questions/59007/… and it is herestring, I made a mistake

[tyrondis]

Doesn't seem to work with files containing multiple lines though.

[Sundeep]

does the base64 command support multiple lines?

[tyrondis]

No, it appears it does not.

[Julie Pelletier]

You could pipe the input through tr -d "\n" to remove lines breaks.

[Alessio]

does base64 support multiple input lines? of course it does, it would be pretty useless if it didn't. see for yourself: ls -l /usr/bin/ | base64 | base64 -d

[Sundeep]

@cas, thanks.. I just tried by saving output of ls -l /usr/bin/ | base64 to a file.. and then gave it as input to b64decode.. it works.. so not sure which case is failing for OP...

[Alessio]

btw, +1. yours is a good answer to a question that only makes any sense if it's just a standin for much more complicated function.

[chepner]

You don't actually need the < /dev/stdin; without a file, base64 will simply read from the standard input it inherits from its parent, which is /dev/stdin.

[Jeff Schaller]

What does "BSIP" mean?

[Jeff Schaller]

There are several un-defined commands in your function; hopefully there's a "BSIP" library of some kind that defines them?

[Scott - Слава Україні]

(1) No indication of what BSIP is. (Google doesn’t know what it is, and neither does Jeff.) (2) A lot of code to handle a fairly simple problem, with no explanation. (3) It’s not clear that this even answers the question. At best, it might be a wrong answer.

[G-Man Says 'Reinstate Monica']

(1) It seems unfortunate to use recursive code for a function that isn’t inherently recursive. (2) It seems unfortunate to use recursive code and not state in the text portion of your answer that you are using recursive code. (3) It seems unfortunate that you have followed TomRoche’s lead and written a loop, when the question does not appear to call for one.

[ZeeLoony]

It is a long-standing grammar rule that suggests acronyms not be used without first elaborating what they stand for. Refusal to expand one on a well-meaning query seems... well... difficult.

[ZeeLoony]

pypi.org/project/bsif "BSIF (ByStar Shell-based/bash-based Integration Facilities) Namespace: A name base for a collection of bash libraries and bash commands for autonomous services intgration [sic]. Support For support, ideas, suggestions, criticism, comments and questions; please contact the author/maintainer Mohsen Banan at: " <redacted /> ... "Installation pip install bsip"

[ZeeLoony]

Also unclear why the answer is criticized for providing a "more general case" since the OP said "You are right, this [base64] was just used as an example, though. – tyrondis Aug 5 '16 at 22:45".

[G-Man Says 'Reinstate Monica']

(1) Yes, my wording was snarky and belligerent.  I apologize.  (2) I still have an issue with the (IMO, unsupported) interpretation that b64decode is supposed to invoke target_utility (base64 --decode) with arguments, and that b64decode < file.txt is supposed to read file.txt one line at a time and invoke target_utility N times.  (3) As @Stéphane Chazelas pointed out in a comment on TomRoche’s answer, read should be IFS= read (to support input lines that begin with space or tab).  (4) Thank you for cleaning up Mohsen Banan’s answer.

[G-Man Says 'Reinstate Monica']

@StéphaneChazelas: This is so confusing!  IMHO, it was a bad design decision to make quoting optional in [[…]] — it tells people that quoting isn’t so important, and so they try to memorize the rules for when quotes are required.  Forbidding quotes in $((…)) is even worse.  As you said, “one side effect of omitting quotes … is that it can send a wrong message to beginners: that it may be all right not to quote variables.” … (Cont’d)

[G-Man Says 'Reinstate Monica']

(Cont’d) …  And perhaps worse of all, the rules for ((…)) and $((…)) are different — see Shell Arithmetic Expansion with Quotes.  I can get if (( "$#" == 0 )) to work; more to the point; I can’t get it to fail. Why do you say that I must not use quotes?

[Stéphane Chazelas]

Hmmm, actually $((...)) and ((...)) are indeed different in that regard in mksh.

[ZeeLoony]

The title of the question is what triggered the process of finding this Q/A for material that aided development of the above to solve a real life problem described by the title. The OP declared that base64 decode was not the "real" problem; pedantic constraint of answers to a particular "example" seems less helpful. When this question and its answers were key to helping solve real problems, alternate answers can be gifts to the community in recognition of the fact that many more people will use this Q/A to solve problems. Since the answer is on-topic per the title, I feel it is "game on".

[ZeeLoony]

More boiler-plate added to the answer, but the value of what really seems to be nitpicking seems questionable even if there is also value in a judicious hunt for theoretical ideals. In the end, this site is about helping community. All the associated answers and comments are available for a visitor to peruse and consider. It is decidedly less valuable to the community when all answers are identical. If one had to deal with the drain of such critique for every answer, perhaps site usage would go down. People have lives to lead and the OP is apparently already happy. IMO, it is time to move on.

[G-Man Says 'Reinstate Monica']

@StéphaneChazelas: On the contrary, I’m using 4.1. … … … … For a specific example, I can say, (("adams"="42")), (("adams=42")), (("adams"="17"+"25")) or (("adams"="17+25")), and I get adams set to 42.  But of course adams=$(("42")) fails.

[Stéphane Chazelas]

@G-Man, hmmm. Quick tests here seem to concur with you (though a=$(("42")) now works in 4.4+), I'll have to double check on the machine I ran tests the other day to understand what I did and why I said that.