Recursive wildcards in GNU make?

Question

It's been a while since I've used make, so bear with me...

I've got a directory, flac, containing .FLAC files. I've got a corresponding directory, mp3 containing MP3 files. If a FLAC file is newer than the corresponding MP3 file (or the corresponding MP3 file doesn't exist), then I want to run a bunch of commands to convert the FLAC file to an MP3 file, and copy the tags across.

The kicker: I need to search the flac directory recursively, and create corresponding subdirectories in the mp3 directory. The directories and files can have spaces in the names, and are named in UTF-8.

And I want to use make to drive this.

Solution 1

I would try something along these lines

FLAC_FILES = $(shell find flac/ -type f -name '*.flac')
MP3_FILES = $(patsubst flac/%.flac, mp3/%.mp3, $(FLAC_FILES))
.PHONY: all
all: $(MP3_FILES)
mp3/%.mp3: flac/%.flac
    @mkdir -p "$(@D)"
    @echo convert "$<" to "[email protected]"

A couple of quick notes for make beginners:

• The @ in front of the commands prevents make from printing the command before actually running it.
• $(@D) is the directory part of the target file name ([email protected])
• Make sure that the lines with shell commands in them start with a tab, not with spaces.

Even if this should handle all UTF-8 characters and stuff, it will fail at spaces in file or directory names, as make uses spaces to separate stuff in the makefiles and I am not aware of a way to work around that. So that leaves you with just a shell script, I am afraid :-/

Solution 2

You can define your own recursive wildcard function like this:

rwildcard=$(foreach d,$(wildcard $(1:=/*)),$(call rwildcard,$d,$2) $(filter $(subst *,%,$2),$d))

The first parameter ($1) is a list of directories, and the second ($2) is a list of patterns you want to match.

Examples:

To find all the C files in the current directory:

$(call rwildcard,.,*.c)

To find all the .c and .h files in src:

$(call rwildcard,src,*.c *.h)

---

This function is based on the implementation from this article, with a few improvements.

Solution 3

If you're using Bash 4.x, you can use a new globbing option, for example:

SHELL:=/bin/bash -O globstar
list:
  @echo Flac: $(shell ls flac/**/*.flac)
  @echo MP3: $(shell ls mp3/**/*.mp3)

This kind of recursive wildcard can find all the files of your interest (.flac, .mp3 or whatever). O

Solution 4

My solution is based on the one above, uses sed instead of patsubst to mangle the output of find AND escape the spaces.

Going from flac/ to ogg/

OGGS = $(shell find flac -type f -name "*.flac" | sed 's/ /\\ /g;s/flac\//ogg\//;s/\.flac/\.ogg/' )

Caveats:

1. Still barfs if there are semi-colons in the filename, but they're pretty rare.
2. The $(@D) trick won't work (outputs gibberish), but oggenc creates directories for you!

Solution 5

FWIW, I've used something like this in a Makefile:

RECURSIVE_MANIFEST = `find . -type f -print`

The example above will search from the current directory ('.') for all "plain files" ('-type f') and set the RECURSIVE_MANIFEST make variable to every file it finds. You can then use pattern substitutions to reduce this list, or alternatively, supply more arguments into find to narrow what it returns. See the man page for find.

View more solutions

Comments

[Admin]

Any reason for selecting make for this purpose? I'd have thought writing a bash script would be simpler

[Roger Lipscombe]

(...or I could write it in Ruby or Python). I'd like to have a play with make beyond the basics, and this is a 'project' I have open right now.

[P Shved]

@Neil, make's concept as pattern-based file system transformation is the best way to approach the original problem. Perhaps implementations of this approach have its limitations, but make is closer to implementing it than bare bash.

[Admin]

@Pavel Only if it works!

[ndim]

@Pavel Well, a sh script that walks through the list of flac files (find | while read flacname), makes a mp3name from that, runs "mkdir -p" on the dirname "$mp3name", and then, if [ "$flacfile" -nt "$mp3file"] converts "$flacname" into "$mp3name" is not really magic. The only feature you are actually losing compared to a make based solution is the possibility to run N file conversions processes in parallel with make -jN.

[ndim]

Admittedly, make's declarative approach is nicer than any imperative language could ever offer.

[Admin]

@ndim That's the first time I have ever heard make's syntax be described as "nice" :-)

[Jens]

Using make and having spaces in file names are contradictory requirements. Use a tool appropriate for the problem domain.

[Gabriel Devillers]

Related: stackoverflow.com/questions/3774568/…

[dmckee --- ex-moderator kitten]

This is where I was going...fast fingers for the win. Though it looks like you may be able to do something clever with vpath. Must study that one of these days.

[Roger Lipscombe]

Doesn't appear to work when the directories have spaces in the names.

[Roger Lipscombe]

Didn't realize that I'd have to shell out to find to get the names recursively...

[ndim]

Oh. Spaces. Well, make will not work with spaces. Make syntax uses spaces for its own purposes.

[dmckee --- ex-moderator kitten]

@Roger: No it doesn't. There is a Grouch Marx skit involving a doctor... But I suppose that the file naming is not in your control.

[Roger Lipscombe]

What, even in the filenames? Ick.

[Roger Lipscombe]

Fair enough; I'll investigate other options. Marking this as the answer anyway.

[Roger Lipscombe]

See the follow up question about rake, instead: stackoverflow.com/questions/2483418/recursive-wildcards-in-r‌​ake

[SystemParadox]

Does make not use lazy evaluation on dependency lists? I put something like this into my Makefile and it still runs the find command even if I tell make to build a target that doesn't need it (and yes, I am using '=' instead of ':=').

[ecmanaut]

Perfect. I needed a variant on it that turns one subtree's coffee into another subtree's javascript: gist.github.com/johan/5490763

[Jeroen]

This doesn't seem to work for me. I've copied the exact function and it still won't look recursively.

[Admin]

I am using GNU Make 3.81, and it seems to work for me. It won't work if any of the filenames have spaces in them, though. Note that the filenames it returns have paths relative to the current directory, even if you are only listing files in a subdirectory.

[paulkon]

Is there a way to parallelize and/or asynchronize the conversion recipe? It's rather slow when I run it in sequence but when I send all the file names to the convert command directly it's many fold faster.

[ndim]

@PaulKonova: Run make -jN. For N use the number of conversions which make should run in parallel. Caution: Running make -j without an N will start all conversion processes at once in parallel which might be equivalent to a fork bomb.

[paulkon]

Yeah, that works. Also, in this particular case, is there a way to remove old directories and files from the MP3 directory which are no longer in the FLAC directory? To have the MP3 directory maintain parity with the FLAC directory?

[ndim]

@PaulKonova Not with ´make`.

[mems]

FLAC_FILES = $(shell find flac -type f -name '*.flac') remove the / after flac folder if you don't want something like that (double slashes): flac//file.flac

[Tomasz Gandor]

This is truly an example, that make is a Turing Tar Pit (see here: yosefk.com/blog/fun-at-the-turing-tar-pit.html). It is not even that hard, but one has to read this: gnu.org/software/make/manual/html_node/Call-Function.html and then "understand recurrence". YOU had to write this recursively, in the verbatim sense; it's not the everyday understanding of "automatically include stuff from subdirs". It's actual RECURRENCE. But remember - "To understand recurrence, you have to understand recurrence".

[akauppi]

To me, even just $(wildcard flac/**/*.flac) seems to work. OS X, Gnu Make 3.81

[lahwran]

I tried $(wildcard ./**/*.py) and it behaved the same as $(wildcard ./*/*.py). I don't think make actually supports **, and it just doesn't fail when you use two *s next to each other.

[kenorb]

@lahwran It should when you invoking commands via Bash shell and you've enabled globstar option. Maybe you're not using GNU make or something else. You may also try this syntax instead. Check the comments for some suggestions. Otherwise it's a thing for the new question.

[lahwran]

@kenorb no no, I didn't even try your thing because I wanted to avoid shell invocation for this particular thing. I was using akauppi's suggested thing. The thing I went with looked like larskholte's answer, though I got it from somewhere else because the comments here said this one was subtly broken. shrug :)

[kenorb]

@lahwran In this case ** won't work, because the extended globbing is a bash/zsh thing.

[Adrian Torrie]

What is the line .PHONY: all for?

[ndim]

@Adrian: The .PHONY: all line tells make that the recipe for the all target is to be executed even if there is a file called all newer than all the $(MP3_FILES).

[Mel]

thank you, been trying to do exactly this for an hour already. glad I stumbled upon this.

[user1129682]

@TomaszGandor You don't have to understand recurrence. You have to understand recursion and in order to do that you must first understand recursion.

[Tomasz Gandor]

My bad, I fell for a linguistic false-friend. Comments can't be edited after such a long time, but I hope everybody got the point. And they also understand recursion.

[Sơn Trần-Nguyễn]

Just want to chime in that $(wildcard flac/**/*.flac) does not work for me on OS X Catalina with Gnu Make 4.2.1. Which is a bummer :/

[kenorb]

@SơnTrần-Nguyễn Make sure to enable it with shopt -s globstar and try on shell first. Check option by shopt | grep globstar.

[Sơn Trần-Nguyễn]

@kenorb Catalina switched the shell from bash to zsh. I did test ls css/**/*.css and it worked.

[Rodney]

This is more portable than calling a shell function. Also I was able to use it to get a list of directories $(sort $(dir $(call rwildcard,src,*))) (where src is my top level dir)

[Kenn Sebesta]

Despite the tar pit, this is the right answer because it is portable and does not depend on shell commands.

[Kenn Sebesta]

Interestingly, this command seems to fail for search strings which have an _ in them. For instance, $(call rwildcard,src,hw*.h) works, but $(call rwildcard,src,hw_*.h) does not. This is not the case for a normal wildcard call, e.g. $(wildcard src/hw_*.h).