What Unix commands can be used as a semaphore/lock?

Question

I want to run multiple Bash shell scripts in parallel. However, I want to avoid race conditions. What Unix commands are truly atomic that I could use for this purpose, and how can I use them?

Solution 1

If lockfile is not installed on your system, then mkdir will do the work: it's an atomic operation, and it fails if the directory already exists (as long as you don't add the -p command-line switch).

create_lock_or_wait () {
  path="$1"
  wait_time="${2:-10}"
  while true; do
        if mkdir "${path}.lock.d"; then
           break;
        fi
        sleep $wait_time
  done
}

remove_lock () {
  path="$1"
  rmdir "${path}.lock.d"
}

Solution 2

flock(1)

#!/bin/bash

# Makes sure we exit if flock fails.
set -e

(
  # Wait for lock on /var/lock/.myscript.exclusivelock (fd 200) for 10 seconds
  flock -x -w 10 200

  # Do stuff

) 200>/var/lock/.myscript.exclusivelock

This ensures that code between "(" and ")" is run only by one process at a time and that the process does wait for a lock too long.

Solution 3

lockfile(1) looks like a good candidate, though beware that it's part of the procmail package, which you may not have installed on your machine yet. It's a popular enough package that it should be packaged for your system if it's not installed yet. Three of the four systems I checked have it, and the other has it available.

Using it is simple:

#!/bin/sh
LOCKFILE=$HOME/.myscript/lock
mkdir -p `dirname $LOCKFILE`

echo Waiting for lock $LOCKFILE...
if lockfile -1 -r15 $LOCKFILE
then
    # Do protected stuff here
    echo Doing protected stuff...

    # Then, afterward, clean up so another instance of this script can run
    rm -f $LOCKFILE
else
    echo "Failed to acquire lock!  lockfile(1) returned $?"
    exit 1
fi

The options I've given make it retry once a second for up to 15 seconds. Drop the "-r" flag if you want it to wait forever.

Solution 4

The system call mkdir() is atomic on POSIX filesystems. So, using the mkdir command in such a way that it involves exactly one call to mkdir() would achieve your purpose. (IOW, don't use mkdir -p). The corresponding unlock is rmdir of course.

Caveat emptor: mkdir() might not be atomic on network filesystems.

Solution 5

Maybe the lockfile command will do what you need.

lockfile ~/.config/mylockfile.lock
.....
rm -f important.lock

View more solutions

Comments

[Lucas Jones]

Just for reference - the man page: linux.die.net/man/1/lockfile. :)

[heyman]

Nice one, didn't know about it. However, it's apparently Linux-specific...

[Alex B]

@Riccardo, FreeBSD has a similar command: lockf(1).

[Charley]

lockf(1) doesn't work in the way used in this example, though. It can't take a file descriptor number as an argument.

[Alexej Magura]

is rmdir therefore also atomic?

[Jay]

Be aware that (according to the manpage), "Once a file is locked, the lock must be touched at least once every five minutes or the lock will be considered stale, and subsequent lock attempts will succeed."

[Benjamin W.]

This seems to delete the wrong file.

[HappyFace]

@RiccardoMurri brew install discoteq/discoteq/flock

[Socowi]

Shorter but equivalent: flock -w 10 /var/lock/.myscript.exclusivelock -c '# do stuff'. No need for ( ... ) 200> ... if you only execute a few simple commands. However, if you want to modify the environment (for instance, assign a variable inside the lock for later use) then use { ... } 200> ....

[christianlc]

this is the correct answer - flock is available on all linux distros and as @HappyFace commented, available as ok'ish port for bsd

[christianlc]

10 years late, but the correct answer then, and now ,is flock, discussed below (unix.stackexchange.com/a/291/343121)