ZSH — Loop files, ignore certain patterns (no ls or find)

Question

I just finished reading an article titled "Why you shouldn't parse the output of ls". I want to loop each file in a directory, and the article states that the following is one of the most basic and ideal ways.

for f in *; do
    echo $f
    # …do stuff
done

The one thing I do not understand is how can I ignore certain files and directories? I use ZSH and have turned extendedglob on. Because of this, I thought this would be viable: for f in *^.jpg; do…, but it only echoes "*^.jpg", no files.

How can I keep the loop mostly as is, but incorporate the ability to exclude certain patterns? Bonus question: can you make the glob recursive?

Solution

Zsh provides very nice ways to enumerate files. They're documented in the manual under “Filename Generation”, but the zsh manual isn't very easy to follow.

By default, if there are no matching files, you'll get an error, which is often desirable on the command line, but not in scripts. To disable this error, either put this near the beginning of your script to turn on the null_glob option:

setopt null_glob
…
for x in *; do …

or use the N glob qualifier:

# Reliably iterate over non-dot files
for x in *(N); do …

Either snippet above iterates over all the files in the current directory that are not dot files. If there is no such file, the loop doesn't run at all (unlike sh where the loop body unhelpfully runs once on the unexpanded glob *, and default zsh where the glob causes an error).

To include dot files in the iteration, either turn on the got_glob option or use the D glob qualifier.

# Reliably iterate over all files in the current directory
for x in *(DN); do …

I recommend the glob qualifier method, even if it's more verbose, because it will keep behaving the same way even if you copy-paste that snippet to a script that didn't turn on the same options.

To exclude certain files based on their name, you can use the glob operators ^ and ~. Note that these require setopt extended_glob in your script² (this is not inherited from your runtime environment or your .zshrc¹). For example, to exclude *.jpg files, either write *~*.jpg or ^*.jpg.

(Note that what you wrote, *^.jpg, would by default cause an error in zsh. If you're seeing *^.jpg printed out, then either you're running zsh with nomatch turned off (it's on by default), which is normally only done when emulating sh or ksh, or else you're actually running that script under an sh shell, probably because it's missing a shebang line at the top.)

setopt extended_glob
# Iterate over all the files in the current directory except *.jpg and .*
for x in *~*.jpg(N); do …

If you want to traverse subdirectories recursively, it's easy: just use **/.

setopt extended_glob
# Iterate over all the files in the current directory and its (grand-)*children except *.jpg and .*
for x in **/*~*.jpg(N); do …

Through glob qualifiers, you can also select files based on conditions other than their name. For example, to loop over regular files only, excluding directories, symbolic links, etc.:

# Iterate over all regular files in the current directory and its (grand-)*children except *.jpg and .*
for x in **/*(.DN); do …

You can even run arbitrary code to decide whether to include a file or not with the e or + glob qualifier. However, when the wildcard pattern is used in a for loop, it's clearer to put the filtering code at the top of the loop body.

¹ _{You could put setopt extended_glob in ~/.zshenv, but I strongly recommend against using .zshenv, because any script that makes assumptions about what's in there would break on a different machine or a different account.}
² _{If you're writing a zsh completion function, these run in an environment where extended_glob is turned on.}

Comments

[Kusalananda]

Have you actually enabled that option in the script that you are running? It would not be enough to enable it in the interactive shell that you use to launch the script from.

[Kusalananda]

Also, are you actually running the script with zsh? Ordinarily, the zsh shell would complain when no names matches the pattern.

[Audun Olsen]

Placing setopt extendedglob inside does give the error "setopt: command not found", hmm. My script begins with #!/bin/zsh, so I would guess it runs w/ zsh, yes.

[Audun Olsen]

I seem to have done the error of calling the script prefaced with sh not zsh. Is it conventional to call scripts with zsh filename.sh?

[Kusalananda]

It is conventional to call a scripts with the correct interpreter. A Perl script with perl, a zsh script with zsh, a bash script with bash, etc. It is also conventional to make script executable and add a proper #!-line at the top pointing to the correct interpreter so that you can just use ./filename.sh.

[Audun Olsen]

I see, of course. I mostly see sh and the latter. But with the latter always seem to give permission errors. I'll look more into it, thanks!

[Kusalananda]

Note that the script would have to be made executable with chmod +x filename.sh for you to be able to use ./filename. sh.

[Stéphane Chazelas]

*^.jpg would expand to all the files whose name ends in ^.jpg without extendedglob and all files with extentedglob, even file.jpg as that's for instance the empty string (matched by *) followed by file.jpg (matched by ^.jpg as that's not .jpg) (or file.jpg followed by the empty string, or file.jp followed by g, etc.).