monitor files (à la tail -f) in an entire directory (even new ones)
Solution 1
You can tail multiple files with… multitail.
multitail -Q 1 'directory/*'
-Q 1 PATTERN
means to check for new content in existing or new files matching PATTERN every 1 second. Lines from all files are shown in the same window, use -q
instead of -Q
to have separate windows.
Solution 2
xtail
is also an alternative. Its man page describes it as:
Xtail monitors one or more files, and displays all data written to a file since command invocation. It is very useful for monitoring multiple logfiles simultaneously. If an entry given on the command line is a directory, all files in that directory will be monitored, including those created after the xtail invocation. If an entry given on the command line doesn’t exist, xtail will watch for it and monitor it once created. When switching files in the display, a banner showing the pathname of the file is printed.
An interrupt character (usually CTRL/C or DEL) will display a list of the most recently modified files being watched. Send a quit signal (usually CTRL/backslash) to stop xtail.
Solution 3
No idea about a shell solution, but (assuming Linux1
) inotify
could be the way to go... see this example imitating tail -F
(using pyinotify
), maybe it can be used as a basis for following an entire directory.
In general, inotify
can monitor directories (citing man 7 inotify
)
The following bits can be specified in mask when calling inotify_add_watch(2) and may be returned in the mask field returned by read(2):
IN_ACCESS File was accessed (read) (*). IN_ATTRIB Metadata changed, e.g., permissions, timestamps, extended attributes, link count (since Linux 2.6.25), UID, GID, etc. (*). IN_CLOSE_WRITE File opened for writing was closed (*). IN_CLOSE_NOWRITE File not opened for writing was closed (*). IN_CREATE File/directory created in watched directory (*). IN_DELETE File/directory deleted from watched directory (*). IN_DELETE_SELF Watched file/directory was itself deleted. IN_MODIFY File was modified (*). IN_MOVE_SELF Watched file/directory was itself moved. IN_MOVED_FROM File moved out of watched directory (*). IN_MOVED_TO File moved into watched directory (*). IN_OPEN File was opened (*).
When monitoring a directory, the events marked with an asterisk (*) above can occur for files in the directory, in which case the name field in the returned inotify_event structure identifies the name of the file within the directory.
(...and pyinotify
closely follows theses options)
1
: BSDs have a similar thing, kqueue
. Maybe a cross-platform solution is achievable using GIO (Python bindings) as abstraction layer since it can, beside inotify
, also use kqueue
Solution 4
Also you can watch directory with watch
watch -n0,1 "ls -lrt /directory/ | tail"
Solution 5
I wrote a quick one that fulfils the need.
#!/bin/bash
LOG_PATTERN=$1
BASE_DIR=$(dirname $LOG_PATTERN* | head -1)
run_thread (){
echo Running thread
tail -F $LOG_PATTERN* &
THREAD_PID=$!
}
# When someone decides to stop the script - killall children
cleanup () {
pgrep -P $$ | xargs -i kill {}
exit
}
trap cleanup SIGHUP SIGINT SIGTERM
if [ $# -ne 1 ]; then
echo "usage: $0 <directory/pattern without * in the end>"
exit 1
fi
# Wait for the directory to be created
if [ ! -d $BASE_DIR ] ; then
echo DIR $BASE_DIR does not exist, waiting for it...
while [ ! -d $BASE_DIR ] ; do
sleep 2
done
echo DIR $BASE_DIR is now online
fi
# count current number of files
OLD_NUM_OF_FILES=$(ls -l $LOG_PATTERN* 2>/dev/null | wc -l)
# Start Tailing
run_thread
while [ 1 ]; do
# If files are added - retail
NUM_FILES=$(ls -l $LOG_PATTERN* 2>/dev/null | wc -l)
if [ $NUM_FILES -ne $OLD_NUM_OF_FILES ]; then
OLD_NUM_OF_FILES=$NUM_FILES
kill $THREAD_PID
run_thread
fi
sleep 1
done
santiagozky
Updated on September 18, 2022Comments
-
santiagozky almost 2 years
I normally watch many logs in a directory doing
tail -f directory/*
. The problem is that a new log is created after that, it will not show in the screen (because*
was expanded already).Is there a way to monitor every file in a directory, even those that are created after the process has started?
-
Itamar about 9 yearsActually, you can ommit the sleep 1 in the main loop, will be more snappy to get new files. But I don't like that kind of busy waits
-
edpaez almost 7 yearsLink is broken, but I think is the same as: manpages.ubuntu.com/manpages/zesty/man1/xtail.1.html
-
jimbobmcgee almost 7 yearsMinor nitpick:
watch
redraws the screen in the alternative buffer, with the first x lines of output from the command. Across a number of files with no changes, if the earlier files do not change, thetail
might render the same thing each time, so you get the appearance of no additional entries, as they are drawn in later files 'below' the bottom of the screen. For a short file, though, this is fine... -
trs over 5 yearsThis does not give a solution to the original problem. This just outputs (the last few lines of) a directory listing (repeatedly, always current - thanks to
watch
), instead of the latest lines of all files (including new ones) in that directory. -
Ned64 over 4 yearsDue to the
sleep
this is not a busy wait but soft on the CPU, just polling. One could change it tosleep 0.2s
(GNU sleep) or whatever to make it faster if required. -
Gilles 'SO- stop being evil' about 4 years@Nish Indeed I don't think multitail implements any form of directory recursion.
-
Chris Prince almost 4 yearsThis is great! It would be even more useful if I could use it more like
tail
-- without any of the ncurses control. I.e., so I could just scroll upwards like normally and see the prior contents. In my use case, I don't care which file the output is coming from. But I do want to easily scroll back upwards. I'm currently using a commandmultitail -d -Q 1 'directory/*'
but don't get any scrolling upwards. -
Metamorphic about 3 yearsThis was the most useful answer for me. Inotify and the
inotifywait
command don't work on Sshfs paths, but then you can dossh HOST inotifywait ...
... -
Urasquirrel almost 3 yearslooks like this is no longer maintained