Unit testing for shell scripts
Solution 1
UPDATE 2019-03-01: My preference is bats now. I have used it for a few years on small projects. I like the clean, concise syntax. I have not integrated it with CI/CD frameworks, but its exit status does reflect the overall success/failure of the suite, which is better than shunit2 as described below.
PREVIOUS ANSWER:
I'm using shunit2 for shell scripts related to a Java/Ruby web application in a Linux environment. It's been easy to use, and not a big departure from other xUnit frameworks.
I have not tried integrating with CruiseControl or Hudson/Jenkins, but in implementing continuous integration via other means I've encountered these issues:
- Exit status: When a test suite fails, shunit2 does not use a nonzero exit status to communicate the failure. So you either have to parse the shunit2 output to determine pass/fail of a suite, or change shunit2 to behave as some continuous integration frameworks expect, communicating pass/fail via exit status.
- XML logs: shunit2 does not produce a JUnit-style XML log of results.
Solution 2
Wondering why nobody mentioned BATS. It's up-to-date and TAP-compliant.
Describe:
#!/usr/bin/env bats
@test "addition using bc" {
result="$(echo 2+2 | bc)"
[ "$result" -eq 4 ]
}
Run:
$ bats addition.bats
✓ addition using bc
1 tests, 0 failures
Solution 3
Roundup by @blake-mizerany sounds great, and I should make use of it in the future, but here is my "poor-man" approach for creating unit tests:
- Separate everything testable as a function.
- Move functions into an external file, say
functions.sh
andsource
it into the script. You can usesource `dirname $0`/functions.sh
for this purpose. -
At the end of
functions.sh
, embed your test cases in the below if condition:if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then fi
-
Your tests are literal calls to the functions followed by simple checks for exit codes and variable values. I like to add a simple utility function like the below to make it easy to write:
function assertEquals() { msg=$1; shift expected=$1; shift actual=$1; shift if [ "$expected" != "$actual" ]; then echo "$msg EXPECTED=$expected ACTUAL=$actual" exit 2 fi }
Finally, run
functions.sh
directly to execute the tests.
Here is a sample to show the approach:
#!/bin/bash
function adder()
{
return $(($1+$2))
}
(
[[ "${BASH_SOURCE[0]}" == "${0}" ]] || exit 0
function assertEquals()
{
msg=$1; shift
expected=$1; shift
actual=$1; shift
/bin/echo -n "$msg: "
if [ "$expected" != "$actual" ]; then
echo "FAILED: EXPECTED=$expected ACTUAL=$actual"
else
echo PASSED
fi
}
adder 2 3
assertEquals "adding two numbers" 5 $?
)
Solution 4
I recently released new testing framework called shellspec.
shellspec is BDD style testing framework. It's works on POSIX compatible shell script including bash, dash, ksh, busybox etc.
Of course, the exit status reflects the result of running of the specs and it's has TAP-compliant formatter.
The specfile is close to natural language and easy to read, and also it's shell script compatible syntax.
#shellcheck shell=sh
Describe 'sample'
Describe 'calc()'
calc() { echo "$(($*))"; }
It 'calculates the formula'
When call calc 1 + 2
The output should equal 3
End
End
End
Solution 5
In addition to roundup and shunit2 my overview of shell unit testing tools also included assert.sh and shelltestrunner.
I mostly agree with roundup author's critique of shunit2 (some of it subjective), so I excluded shunit2 after looking at the documentation and examples. Although, it did look familiar having some experience with jUnit.
In my opinion shelltestrunner is the most original of the tools I've looked at since it uses simple declarative syntax for test case definition. As usual, any level of abstraction gives some convenience at the cost of some flexibility. Even though, the simplicity is attractive I found the tool too limiting for the case I had, mainly because of the lack of a way to define setup/tearDown actions (for example, manipulate input files before a test, remove state files after a test, etc.).
I was at first a little confused that assert.sh only allows asserting either output or exit status, while I needed both. Long enough to write a couple of test cases using roundup. But I soon found the roundup's set -e
mode inconvenient as non-zero exit status is expected in some cases as a means of communicating the result in addition to stdout, which makes the test case fail in said mode. One of the samples shows the solution:
status=$(set +e ; rup roundup-5 >/dev/null ; echo $?)
But what if I need both the non-zero exit status and the output? I could, of course, set +e
before invocation and set -e
after or set +e
for the whole test case. But that's against the roundup's principle "Everything is an Assertion". So it felt like I'm starting to work against the tool.
By then I've realized the assert.sh's "drawback" of allowing to only assert either exit status or output is actually a non-issue as I can just pass in test
with a compound expression like this
output=$($tested_script_with_args)
status=$?
expected_output="the expectation"
assert_raises "test \"$output\" = \"$expected_output\" -a $status -eq 2"
As my needs were really basic (run a suite of tests, display that all went fine or what failed), I liked the simplicity of assert.sh, so that's what I chose.
Related videos on Youtube
gareth_bowles
Updated on July 05, 2022Comments
-
gareth_bowles almost 2 years
Pretty much every product I've worked on over the years has involved some level of shell scripts (or batch files, PowerShell etc. on Windows). Even though we wrote the bulk of the code in Java or C++, there always seemed to be some integration or install tasks that were better done with a shell script.
The shell scripts thus become part of the shipped code and therefore need to be tested just like the compiled code. Does anyone have experience with some of the shell script unit test frameworks that are out there, such as shunit2 ? I'm mainly interested in Linux shell scripts for now; I'd like to know how well the test harness duplicate the functionality and ease of use of other xUnit frameworks, and how easy it is to integrate with continuous build systems such as CruiseControl or Hudson.
-
todo about 11 yearsRoundup formalizes some tasks/tags for you. Once you get over the learning hump, it's quite useful. Personally, I like haridsv's approach better, because it doesn't require me to install another pkg. I have apply this approach to shell script and python testings.
-
pahaz almost 8 yearsYou can also check bashtest util: github.com/pahaz/bashtest (it`s simple way to write bash tests)
-
sobolevn over 6 yearsCheckout this overview of almost every possible tool: medium.com/wemake-services/…
-
gareth_bowles over 6 years@sobolevn very nice article, thanks!
-
-
gareth_bowles over 11 yearsNice, thanks - I really like the structured programming approach to shell scripts.
-
Henk Langeveld about 11 yearsVery good observation there: 'Separate everything testable as a function'. It's important even if you don't (yet) write a test for it. It is a major factor in improving code readability. A couple of years back I started writing my (ksh) script according to this principle. Write everything as a function.
-
haridsv about 11 years@HenkLangeveld Won't the
exit 0
cause the "sourcee" (the one sourcing this) to exit as well? I think you are looking forreturn
here. -
Henk Langeveld about 11 years@haridsv Good question. When you source the file, you don't want to run the tests. By collecting all test-related code into a subshell, we don't spoil the runtime environment of the 'consumer'. That
exit 0
only terminates the subshell. -
haridsv about 11 years@HenkLangeveld oops, you are right, I didn't pay attention to the parenthesis you added.
-
iconoclast almost 11 yearsThis is brilliant. Thanks for the detailed explanation.
-
Noah Sussman almost 9 years+1 for BATS! Judging by its github metrics (12 contributors, almost 2000 stars), this is the tool most people choose as their bash xUnit test runner.
-
Ethan Post over 7 yearsPete, I am very close to releasing a unit test library which works for bash and ksh, very simple to use. Can you send me a link to the Junit-style XML log someplace so I can see what it contains? Also would you want exit status for a single failing test or at the end when completed, if > 0 test fail exit 1? The way I do it now is that there is a function you can call to get the # of tests which passed/failed.
-
Pete TerMaat over 7 years@EthanPost I no longer use/require XML logs. You might find help by googling for things like "xunit xml format". Things like this: xunit.github.io/docs/format-xml-v2.html
-
ssc over 6 years+1: I've used BATS before a while back and was not entirely overwhelmed. Found this excellent (independent ?) Getting Started document today, followed it through and am now totally sold on BATS.
-
Andy over 5 yearsAs I've read this answer my first impulse was "Nah, the maintainer doesn't respond to issues and it has some severe bugs but then I discovered that it has been forked since I've used it last time"
-
Flamefire about 5 yearsshunit now also exits with correct exit code: github.com/kward/shunit2/blob/…
-
Koichi Nakashima almost 5 yearsSupported JUnit formatter in version 0.16.0. and also supported coverage, parallel execution and more.