How can I use inverse or negative wildcards when pattern matching in a unix/linux shell?

Question

Say I want to copy the contents of a directory excluding files and folders whose names contain the word 'Music'.

cp [exclude-matches] *Music* /target_directory

What should go in place of [exclude-matches] to accomplish this?

Solution 1

In Bash you can do it by enabling the extglob option, like this (replace ls with cp and add the target directory, of course)

~/foobar> shopt extglob
extglob        off
~/foobar> ls
abar  afoo  bbar  bfoo
~/foobar> ls !(b*)
-bash: !: event not found
~/foobar> shopt -s extglob  # Enables extglob
~/foobar> ls !(b*)
abar  afoo
~/foobar> ls !(a*)
bbar  bfoo
~/foobar> ls !(*foo)
abar  bbar

You can later disable extglob with

shopt -u extglob

Solution 2

The extglob shell option gives you more powerful pattern matching in the command line.

You turn it on with shopt -s extglob, and turn it off with shopt -u extglob.

In your example, you would initially do:

$ shopt -s extglob
$ cp !(*Music*) /target_directory

The full available extended globbing operators are (excerpt from man bash):

If the extglob shell option is enabled using the shopt builtin, several extended pattern matching operators are recognized.A pattern-list is a list of one or more patterns separated by a |. Composite patterns may be formed using one or more of the following sub-patterns:

• ?(pattern-list)
  Matches zero or one occurrence of the given patterns
• *(pattern-list)
  Matches zero or more occurrences of the given patterns
• +(pattern-list)
  Matches one or more occurrences of the given patterns
• @(pattern-list)
  Matches one of the given patterns
• !(pattern-list)
  Matches anything except one of the given patterns

So, for example, if you wanted to list all the files in the current directory that are not .c or .h files, you would do:

$ ls -d !(*@(.c|.h))

Of course, normal shell globing works, so the last example could also be written as:

$ ls -d !(*.[ch])

Solution 3

Not in bash (that I know of), but:

cp `ls | grep -v Music` /target_directory

I know this is not exactly what you were looking for, but it will solve your example.

Solution 4

If you want to avoid the mem cost of using the exec command, I believe you can do better with xargs. I think the following is a more efficient alternative to

find foo -type f ! -name '*Music*' -exec cp {} bar \; # new proc for each exec



find . -maxdepth 1 -name '*Music*' -prune -o -print0 | xargs -0 -i cp {} dest/

Solution 5

A trick I haven't seen on here yet that doesn't use extglob, find, or grep is to treat two file lists as sets and "diff" them using comm:

comm -23 <(ls) <(ls *Music*)

comm is preferable over diff because it doesn't have extra cruft.

This returns all elements of set 1, ls, that are not also in set 2, ls *Music*. This requires both sets to be in sorted order to work properly. No problem for ls and glob expansion, but if you're using something like find, be sure to invoke sort.

comm -23 <(find . | sort) <(find . | grep -i '.jpg' | sort)

Potentially useful.

View more solutions

Comments

[Adam Rosenfield]

This does a recursive copy, which is different behavior. It also spawns a new process for each file, which can be very inefficient for a large number of files.

[dland]

The cost of spawning a process is approximately zero compared to all the IO that copying each file generates. So I'd say this is good enough for occasional usage.

[Daniel Bungert]

Default ls will put multiple files per line, which probably isn't going to give the right results.

[Adam Rosenfield]

Only when stdout is a terminal. When used in a pipeline, ls prints one filename per line.

[SpoonMeiser]

ls only puts multiple files per line if outputting to a terminal. Try it yourself - "ls | less" will never have multiple files per line.

[Adam Rosenfield]

This does a recursive find, which is different behavior than what OP wants.

[Vinko Vrsalovic]

Some workarounds for the process spawning: stackoverflow.com/questions/186099/…

[ejgottl]

use "-maxdepth 1" to avoid recursion.

[ejgottl]

use backticks to get the analog of the shell wild card expansion: cp find -maxdepth 1 -not -name '*Music*' /target_directory

[ejgottl]

note that find will also grab dot (hidden) files. This is different behavior than shell wildcard expansion. Obviously you can filter these out too....

[Ishbir]

It won't work for filenames containing spaces (or other white spcace characters).

[Erick Robertson]

I like this feature: ls /dir/*/!(base*)

[Elijah Lynn]

How do you include everything () and also exclude !(b)?

[Thedward]

This will work as long as your file names don't have any tabs, newlines, more than one space in a row, or any backslashes. While those are pathological cases, it is good to be aware of the possibility. In bash you can use while IFS='' read -r filename , but then newlines are still a problem. In general it is best not to use ls to enumerate files; tools like find are much better suited.

[Thedward]

Without any additional tools: for file in *; do case ${file} in (*Music*) ;; (*) cp "${file}" /target_directory ; echo ;; esac; done

[Noldorin]

How would you match, say, everything starting with f, except foo?

[kennytm]

@Noldorin,ElijahLynn See stackoverflow.com/questions/14822102/…

[weberc2]

Why is this disabled by default?

[osirisgothra]

shopt -o -u histexpand if you need to look for files with exclamation points in them -- on by default, extglob is off by default so that it doesn't interfere with histexpand, in the docs it explains why this is so. match everything that starts with f except foo: f!(oo), of course 'food' would still match (you would need f!(oo*) to stop things that begin in 'foo' or, if you want to get rid of certain things ending in '.foo' use !(.foo), or prefixed: myprefix!(.foo) (matches myprefixBLAH but not myprefixBLAH.foo)

[Graham Perks]

Why is this disabled? Just to be ultra compatible, says stackoverflow.com/questions/17191622/…. Would be safe to echo shopt -s extglob >>~/.bashrc

[James Haigh]

@osirisgothra: Why not use ‘set +H’ (or ‘set +o histexpand’) instead of ‘shopt -o -u histexpand’?

[osirisgothra]

It's just a more verbose way of doing things, good for scripting when you are not sure of who will be doing the reading and don't wish to explain to a thousand incompetent newcomers that + does not mean disable, etc, etc...

[Robert Siemer]

@osirisgothra extglob does not really interfere with histexpand, because if both are on, !(...) is ignored by history expansion. – Thus, both keep working, but !(...) is directed to extglob then.

[Big McLargeHuge]

What's the reason for -d?

[Ishbir]

@Koveras for the case that one of the .c or .h files is a directory.

[zrajm]

Doh! There was a bug (I matched against *.txt instead of just *). Fixed now.

[avtomaton]

use -maxdepth 1 for non-recursive?

[Admin]

One problem with shell expansion for file globbing, is that it expands the command line before it executes the program. This mean you can end up with a very long command line, and anyone else on the system can potentially see everything you've specified on the command line using ps auxwwf (for example). Furthermore, even if snooping users is not a concern, the command line may simply reach a limit, and the options get truncated, or the command fails to be executed. A more robust approach is always to use find -print0 | xargs -0 as mentioned in another answer.

[davidwebca]

I found this to be the cleanest solution without having to enable / disable shell options. The -maxdepth option would be recommended in this post to have the result needed by the OP, but it all depends on what you're trying to accomplish.

[Mark Stosberg]

One of the benefits of the exclusion is not to traverse the directory in the first place. This solution does two traversals of sub-directories-- one with the exclusion and one without.

[James M. Lay]

Very good point, @MarkStosberg. Although, one fringe benefit of this technique is you could read exclusions from an actual file, e.g. comm -23 <(ls) exclude_these.list

[tripleee]

The cp manual page on MacOS has an -a option but it does something entirely different. Which platform supports this?

[tripleee]

mywiki.wooledge.org/ParsingLs lists a number of additional reasons why you should avoid this.

[tripleee]

Using find in backticks will break in unpleasant ways if it finds any nontrivial file names.

[spurra]

@DaveKennedy It's to list everything in the current directory D, but not the content of subdirectories that may be contained in the directory D.

[dhill]

Shouldn't the first line be -u instead of -s or skipped?

[vitalii]

It uses 2 loops, don't use that ever. With find use -exec like find . -not -name "*Music*" -exec cp "{}" /target/dir \;

[zeawoas]

is there also a way to combine positive and negative wildcards?

[Thomas Seeling]

Why does this work with ; to separate the commands, but not as GLOBIGNORE=xxx ls *

[Aryeh Leib Taurog]

find . -type f ! -name '*Music*' | cpio pd /target_directory