Script to monitor folder for new files?
Solution 1
You should consider using inotifywait
, as an example:
inotifywait -m /path -e create -e moved_to |
while read dir action file; do
echo "The file '$file' appeared in directory '$dir' via '$action'"
# do something with the file
done
In Ubuntu inotifywait
is provided by the inotify-tools
package. As of version 3.13 (current in Ubuntu 12.04) inotifywait
will include the filename without the -f option. Older versions may need to be coerced. What is important to note is that the -e
option to inotifywait
is the best way to do event filtering. Also, your read
command can assign the positional output into multiple variables that you can choose to use or ignore. There is no need to use grep/sed/awk to preprocess the output.
Solution 2
In case you came here for a simple, fast, handy solution reading the title, you could use watch
watch -n 0.1 ls <your_folder>
Monitors your folder and lists you everything in it every 0.1 seconds
Drawback
Non-scriptable (For scripting options, have a look at other answers)
Not real time, so if a file was created and deleted in less than 0.1 second, then this would not work, watch
only supports minimum of 0.1 seconds.
Solution 3
I just cooked up this, and see no huge problems with it, other than a tiny chance of missing files in between checks.
while true
do
touch ./lastwatch
sleep 10
find /YOUR/WATCH/PATH -cnewer ./lastwatch -exec SOMECOMMAND {} \;
done
If your file processing doesn't take too long, you should not miss any new file. You could also background the activities... It's not bullet proof, but it serves some purposes without external tools like inotify.
Solution 4
I prefer incron
, as its easier to manage. Essentially it's a service that leverages inotify
and you can setup configurations to take action based on file change operations.
Ex:
<directory> <file change mask> <command or action> options
/var/www/html IN_CREATE /root/scripts/backup.sh
You can see a full example here: http://www.cyberciti.biz/faq/linux-inotify-examples-to-replicate-directories/
Solution 5
entr
Using entr
is the new way to do this (it's cross platform). Note entr
doesn't use polling giving it a huge advantage over many of the alternatives.
Uses
kqueue(2)
orinotify(7)
to avoid polling.entr
was written to make rapid feedback and automated testing natural and completely ordinary.
On BSD it uses pledge(2)
You can install it with
apt-get install entr
dnf install entr
brew install entr
You can track a directory for new additions using
while $(true); do
# echo ./my_watch_dir | entr -dnr echo "Running trigger..."
echo ./my_watch_dir | entr -dnr ##MY COMMAND##
done;
Options explained (from the docs),
-d
Track the directories of regular files provided as input and exit if a new file is added. This option also enables directories to be specified explicitly. Files with names beginning with ‘.’ are ignored.-n
Run in non-interactive mode. In this mode entr does not attempt to read from the TTY or change its properties.-r
Reload a persistent child process. As with the standard mode of operation, a utility which terminates is not executed again until a file system or keyboard event is processed.SIGTERM
is used to terminate the utility before it is restarted. A process group is created to prevent shell scripts from masking signals.entr
waits for the utility to exit to ensure that resources such as sockets have been closed. Control of the TTY is not transferred the child process.
Note entr
isn't polling there. It's after there is an operation on an inode it hasn't seen. Note this isn't a failsafe mechanism because the inode could be recycled to a different file name entirely. Ie,
mkdir foo;
cd foo;
touch bar;
touch baz;
ls -1i;
echo "delete bar; add quz"
rm bar;
touch quz;
ls -1i;
See the inotifywait
answer for a better, and more powerful method of doing this. Albeit with a much worse interface.
Related videos on Youtube
ihatetoregister
Updated on September 18, 2022Comments
-
ihatetoregister over 1 year
How can I immediately detect when new files added to a folder within a bash script? I would like the script to process files as soon as they are created in the folder. Are there any methods aside from scheduling a cron job that checks for new files each minute or so?
-
ztank1013 over 12 yearsAre you going to remove files from the folder once they are processed?
-
Gilles 'SO- stop being evil' over 12 yearspossible duplicate of How to run a command when a directory's contents are updated?
-
George Birbilis over 3 yearsactually the answers at that other question mentioned as possible duplicate are less diversified and not that helpful in case of cross-platform need. In fact for cross-platform I'd vote for fswatch (emcrisostomo.github.io/fswatch), also mentioned at a comment in another answer below
-
-
ihatetoregister over 12 yearsGreat! The
inotifywait
was just what I wanted. -
daisy over 11 yearsWhy is fflush() necessary here? Since a newline is already created I think
-
enzotib over 11 years@warl0ck: I think
awk
does not print each line as it can, but it waits for a bigger buffer (containing more than one line) before it decides to output something. -
Lugoues about 11 yearsJust want to update this. You do not need awk to achieve this. you can filter the events with '-e create' and get only the filename by doing '-f %f' or the full path using '-f %w%f'. So the first line of the above script becomes: inotifywait -m /path -f %w%f -e create |
-
mbaljeetsingh about 10 years@Lugoues and now when you try to use -f you get
The '--filename' option no longer exists. The option it enabled in earlier versions of inotifywait is now turned on by default.
So, you only have to doinotifywait -m /path -e create |
I'm going to try and edit this answer. -
Ankit almost 9 yearsThis solution works for mounted remote filesystems. inotify-tools developer(s) is working on fuse (or was in mid 2014).
-
Admin almost 9 yearsNow there is also a portable tool for it called
fswatch
. I did not write it, but it's open source and I use it. -
Michael Sacchi over 8 yearsGood catch. I improved it a bit to support spaces in filenames.
-
Michael Sacchi over 8 yearsAbsolutely. That's the way to go. Not really sure why I went down that road, I use -exec routinely.
-
Farhan almost 8 yearsits not realtime. realtime is always best
-
alightholder over 7 yearsYou shouldn't ever use
ls
for scripting. Usefind
or simple globbing instead: mywiki.wooledge.org/ParsingLs -
agrublev about 7 yearsFor this in need
sudo apt-get install inotify-tools
-
Devs love ZenUML almost 7 yearsBest solution if
inotify
is not available. I would add-type f
to filter out files only. Otherwise the folder will also be returned. -
SDsolar over 6 yearsYep - the
-f filename
option is great. So then the only remaining question is how to get this to start upon reboot. I am going to use this with my solar plant toos.system("ssh me@mysystem ' ( touch /home/me/alarms/low24 ) '")
so then the creation of this file will cause the master computer to useespeak
and announce the low voltage. It already sends me an email but since my system already speaks the time at the top of the hour it has all the rest. askubuntu.com/questions/977613/… -
MKatleast3 about 6 yearsEven thou this is not a real-time, sleeping just a second is enough for me. Most importantly, this solution can watch newly created folders and files. This is great especially for monitoring Hadoop steps log which create new folders for every new step.
-
Wender almost 6 yearscan you explain me the "while" line? Where did "path", "action" and "file" come from?
-
Tim over 5 years@Wender inotfiywait outputs 3 pieces of information on a single line when triggered. The 'read' bash builtin reads the input line and assigns each of the three pieces of information to a variable. Thus the first piece is assigned to the variable path, the second to action, and the third to file. Having assigned values to those variables, they are then available to be used later (like on the echo line). More information: tldp.org/LDP/Bash-Beginners-Guide/html/sect_08_02.html
-
Joabe Lucena about 5 yearsThat was exactly what I was trying to remember! Thanks a lot!!
-
Maksym Ganenko about 5 yearsPlease replace name
path
withinread path action file
withfilepath
because it may overwrite$PATH
environment variable and so all your non-builtin commands within loop will report command not found. It happened within my fresh Arch Linux zsh terminal and I spent a lot of time googling and experimenting until I found the reason. -
bob over 4 yearsincron never worked for me. based on SE questions about it, it seems like many people have unresolved issues with it.
-
kodmanyagha about 4 yearsthis is very cool. I like it
-
Bob almost 4 yearsFor CentOS:
yum --enablerepo=epel -y install inotify-tools
-
ArcanisGK507 almost 4 yearsthis not work if the director is end Point of FTP point mount.
-
George Birbilis over 3 yearsFor Solaris do note the comment "The behaviour depends on the monitor. Solaris File Events Notification will tell you that a directory has changed when a file has been created, but won't give you any more details. It's up to the receiver to scan the directory contents and see what file has been created." at github.com/emcrisostomo/fswatch/issues/228
-
George Birbilis over 3 yearsAlso, note the license is GPL v3.0, so you should avoid statically linking to libfswatch (but dynamically link to its shared library instead) or prefer using the fswatch utility instead if you don't want to be forced to share the source code of your app too. Regarding having to share fswatch's source code, guess sharing a URL to its source code repository should be enough (that one has the most up to date version of the code anyway).
-
MaXi32 over 3 yearsi use inotify for my bash script and most of the script have awk, sed. and reading again this answer carefully, I felt regret not reading about how to implement that 'built-in' variables path, action, file instead of using grep, etc.. thanks a lot for this answer
-
user3821306 about 3 yearsI needed a bunch of other options
inotifywait -me modify,create,delete /home/Mansoor/ITB/uploads
. This works on Windows WSL2, ChromeOS, RHEL even Android! stackoverflow.com/questions/50168397/… -
Thanos M almost 3 yearsmaybe this is a dumb question, but how can it achieve that without using polling
-
Dawid Wolski over 2 yearsThanks. I've updated the link.
-
clay over 2 yearshow to run it as background daemon?
-
MrE about 2 years
entr
is great to restart a process when files changes, but it's not able to give you the name of the file that changed or was added to a folder: it gives the first file in the folder that was modified, which is not useful