Copy group of files (Filename*) to backup (Filename*.bak)

Question

Background

In Linux you can:

• List a group of files with ls Filename*
• Remove a group of files with rm Filename*
• Move a group of files with mv Filename* /New/Directory
• But you cannot copy a group of files with: cp Filename* *.bak

Change Linux cp command to copy group of files

I have a group of files I'd like to copy without entering the names one by one and using the cp command:

$ ls gmail-meta3*
gmail-meta3                          gmail-meta3-REC-1558392194-26467821
gmail-meta3-LAB-1558392194-26467821  gmail-meta3-YAD-1558392194-26467821

How can I use something like the old DOS command copy gmail-meta3* *.bak?

I don't want to type similar command four times:

cp gmail-meta3-LAB-1558392194-26467821 gmail-meta3-LAB-1558392194-26467821.bak

I'm looking for a script/function/app that accepts parameters for old and new filename group and not something with hard-coded filenames. For example, a user can type:

copy gmail-meta3* *.bak

or they might type:

copy gmail-meta3* save-*

Solution 1

I wrote this one-liner into my ~/.bashrc. Much better answers using find can be posted I suppose. Even better answers could be written in in C. Hopefully this Q&A gets the ball rolling for better answers:

cps () {
    # cps "Copy Splat", copy group of files to backup, ie "cps Filename .bak"
    # Copies Filename1 to Filename1.bak, Filename2 to Filename2.bak, etc.
    # If Filename1.bak exists, don't copy it to Filename1.bak.bak
    for f in "$1"*; do [[ ! "$f" == *"$2" ]] && cp -a "$f" "$f$2"; done

    # OLD version comments suggested to remove 
    # ls "$1"* | while read varname; do cp -a "$varname" "$varname$2"; done
}

• for f in "$1"*; do : $1 is the gmail-meta3 parameter and f is the list of files matching. Combined this means for gmail-meta3, gmail-meta3-LAB-9999, etc. do the following
• [[ ! "$f" == *"$2" ]] && : $f is the same as f above. $2 is the .bak parameter passed. Combined this means if the filename doesn't end in .bak (because we don't want to copy .bak and create .bak.bak) then do the following
• cp -a "$f" "$f$2"; copy gmail-meta3 to gmail-meta3.bak, etc.
• done : loop back and grab next filename in gmail-meta3* list.

---

cps gmail-meta3 .bak Sample Output

Using the question as an example here is how it looks in action:

───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 05:46 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ cps gmail-meta3 .bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ ll gmail-meta3*
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3
-rw-rw-r-- 1 rick rick 26467821 May 20 16:43 gmail-meta3.bak
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821
-rw-rw-r-- 1 rick rick      643 May 20 16:43 gmail-meta3-LAB-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821
-rw-rw-r-- 1 rick rick    49607 May 20 16:44 gmail-meta3-REC-1558392194-26467821.bak
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821
-rw-rw-r-- 1 rick rick   728954 Jun 27 17:04 gmail-meta3-YAD-1558392194-26467821.bak
───────────────────────────────────────────────────────────────────────────────────────────
rick@alien:~/gmail$ 

Note: This uses the -a flag with the cp command to preserve timestamps and give you better grasp of your file backups.

Notice how the file copies have the exact same date and time as the originals. If the -a parameter was omitted they would be given the current date and time and wouldn't look like a true backup except that the file size would be the same.

Solution 2

Here is an example of one atypical usage of sed, that is applicable for this task:

sed -i.bak '' file-prefix*

In this way, actually, sed will not change the files, because we didn't provided any commands '', but due to the option -i[suffix] it will create a backup copy of each file. I found this approach when I was searching Is there any way to create backup copy of a file, without type its name twice?

Solution 3

You can use find:

find . -max-depth 1 -name 'gmail-meta3*' -exec cp "{}" "{}.bak" \;

That will find in the current directory . all files with a name matching the glob pattern (mind the single quotes around the pattern to prevent shell globbing). For each file found, it will exec cp from name to name.bak. The \; at the end ensures it will do each file individually instead of passing all of them at once. The max depth as 1 only searches the cuurent directory instead of recursing down.

Solution 4

You can use a for loop with bash. Normally, I would just type it as a one-liner because this isn't a task I perform often:

for f in test* ; do cp -a "$f" "prefix-${f}.ext" ; done

However, if you need it as a script:

cps() {
   [ $# -lt 2 ] && echo "Usage: cps REGEXP FILES..." && return 1

   PATTERN="$1" ; shift

   for file in "$@" ; do
      file_dirname=`dirname "$file"`
      file_name=`basename "$file"`
      file_newname=`echo "$file_name" | sed "$PATTERN"`

      if [[ -f "$file" ]] && [[ ! -e "${file_dirname}/${file_newname}" ]] ; then
         cp -a "$file" "${file_dirname}/${file_newname}"
      else
         echo "Error: $file -> ${file_dirname}/${file_newname}"
      fi
   done
}

Usage is similar to rename. To test:

pushd /tmp
mkdir tmp2
touch tmp2/test{001..100}     # create test files
ls tmp2
cps 's@^@prefix-@ ; s@[email protected]@' tmp2/test*    # create backups
cps 's@[email protected]@' tmp2/test*    # more backups ... will display errors
ls tmp2
\rm -r tmp2                   # cleanup
popd

Solution 5

The closest you will likely get to the DOS paradigm is mcp (from the mmv package):

mcp 'gmail-meta3*' 'gmail-meta3#1.bak'

If zsh is available, its contributed zmv module is perhaps a little closer:

autoload -U zmv

zmv -C '(gmail-meta3*)' '$1.bak'

---

I'd avoid ls regardless - a variant on your own answer that's safe for whitespace (including newlines) would be

printf '%s\0' gmail-meta3* | while IFS= read -r -d '' f; do cp -a -- "$f" "$f.bak"; done

or perhaps

printf '%s\0' gmail-meta3* | xargs -0 -I{} cp -a -- {} {}.bak

View more solutions

Comments

[qwr]

The issue looks to me using glob operator twice which none of your other commands use. bash isn't smart enough to handle it.

[Mike S]

@qwr, the fact that bash expands metacharacters and tokenizes the input prior to handing it off to a command to be exec'ed is part of the design of the UNIX shells. To somehow try and program an exception for the cp command would break the entire consistency of bash, which would not be smart at all. As an exercise, try and figure out what is happening here, and why it is the shell's metacharacter expansion that makes it so: touch aa ab ba; mkdir bb; cp a* b*; ls *

[WinEunuuchs2Unix]

@MikeS Thanks for the pointers. Yesterday someone else said you can use a wildcard * for the source filenames but not for the target filenames. As such as substitute wildcard (I think ## was suggested but I'm leaning towards %) will have to be used for the target. I think this is what you are reinforcing? I didn't expect to change the cp command at all. Just create a wrapper script called copy that emulated (within reason) the DOS copy command.

[Mike S]

@WinEunuuchs2Unix that person was correct. The shell metacharacters are independent of the commands. So all wildcards will try and match all files that match the pattern. If you're trying to make a general-purpose "match everything and copy them to whatever they were but add this suffix" program, then yes, putting an unescaped metacharacter as the target will probably not do what you want. Because all metacharacters in the shell command line are expanded. If you KNOW for certain that the target metacharacter will never form a match, you could use it- because the shell can't expand it.

[Mike S]

...but that would be an ugly thing to do. Better to use a special character. % or underscore are good, they're generally not metacharacters (but be careful of using % in a crontab file; it's special there).

[istrasci]

If you are interested, I have a script that does this with several config options, as well as one that does the inverse (remove the .bak from *.bak files).

[WinEunuuchs2Unix]

I understand mmv is the package but in comments you say the command is mcp but then in the command you use mmv which is also a command in mmv package. I like the direction of the printf examples and in a polished script I'd ensure $1 and $2 were passed. +1 for getting ball rolling :)

[steeldriver]

@WinEunuuchs2Unix apologies - the mcp/mmv was a brainfart. Actually mcp is just a synonym for mmv -c

[WinEunuuchs2Unix]

Pfft no worries. If I had a dollar for every typo I've made I'd be a millionaire :) I'd like clarification on printf command I've never used really. Are you saying printf '%s\0' "$1"* would work if gmail-meta3 was passed as parameter 1?

[steeldriver]

@WinEunuuchs2Unix I'd probably let the calling context do the globbing i.e. cps gmail-meta3* and then write printf '%s\0 "$@" | while ...` in the function. Or simply use for f; do cp -- "$f" "$f.bak"; done (like xiota's answer, but as a function)

[WinEunuuchs2Unix]

Will it work with find . -max-depth 1 -name '"$1"'' -exec cp "{}" "{}$2" \; when $1 is source and $2 is extension?

[cbojar]

$2 should be ok to substitute so long as it is reasonable. $1 could be trickier as we can't do variable substitution inside single quotes. I'm not sure offhand but it might be possible to use $1 in double quotes since the pattern is stored in a string.

[WinEunuuchs2Unix]

Nice coding but it's kind of hard after decades of using Source then Target to switch copy command to Target then Source

[pLumo]

Added the function with the suffix in the back.

[WinEunuuchs2Unix]

Thanks that is more intuitive. Calling it a suffix is accurate the way my answer coded it but it's really a target or destination. Other users might want to use: copy gmail-meta3* old-meta3*. In my answer I couldn't figure out how to get * into the destination name like my question requested...

[pLumo]

problem is that * is interpreted by the shell, so the function will not know about it. You would need some other character or quote it, then replace it with the original filename within the function.

[WinEunuuchs2Unix]

I guess the # could be used as a replacement wildcard for *? So you could type copy filenames# save-#. I think you'd want the wildcard character to be the same for source and target.

[pLumo]

I added one with pattern, that should be pretty much what you want, but I let the filenames be chosen from the shell outside the function, so I use * for the files and # for the pattern. I think it's better.

[qwr]

don't people always recommend against parsing ls

[Mr. Sam]

Since you mention find I assume you’re aware of the dangers of parsing ls? But in your case neither is necessary: just do for file in "$1"*; do copy -a "$file" "$file$2"; done — this is completely safe and much simpler than any kind of indirection via ls or find and a while loop.

[WinEunuuchs2Unix]

I love rsycnc so I upvoted but, a simpler method would be cp Filename* /path/to/backup/dir because files wouldn't need *.bak uniquifier if they were in a separate directory.

[WinEunuuchs2Unix]

FYI: $ time for f in gmail-meta3* ; do cp -a "$f" "${f}.bak" ; done = real 0m0.046s

[WinEunuuchs2Unix]

Um no I don't want to optimize for time. It is 0.046 seconds which means 0 seconds for human perception. I was just trying to show how I was testing posted answers and passing along interesting tidbits to viewers who looked at the sed command above. Or at least I was interested in comparing sed to cp....

[WinEunuuchs2Unix]

Your solution with cp is faster than solution with sed though. So it's a cause for celebration :)

[Mike S]

It probably doesn't matter what storage device you have for these tests, if this is happening on a Linux machine. The kernel is extremely good at buffering files in memory. That's what the "buff/cache" value is when using the free command. Actual writes to the device take place at a moment chosen by an algorithm that takes into account age of the cache and memory pressure on the machine. If you're trying multiple tests, then the first file reads will come off the disk but subsequent reads will most likely come straight out of memory (see sync; echo 3 > /proc/sys/vm/drop_caches).

[G-Man Says 'Reinstate Monica']

(1) -a is a non-standard test operator.  Why not use -e?  (2) “Unable to create temporary directory.” is a somewhat misleading error message.  (3) Why not just use mktemp -d?  (4) You should test exit statuses.  For example, you should say ! mkdir "$FOLDER" && echo "Unable to create temporary directory." && return 1 or mkdir "$FOLDER" || { echo "Unable to create temporary directory."; return 1;}.  Likewise for cp and rename (and maybe even pushd, if you want to be careful).  … (Cont’d)

[G-Man Says 'Reinstate Monica']

(Cont’d) …  (5) Arrggghhhh!  Don’t say $@; say "$@". (5b) There’s no need to use { and } when you reference variables the way you are doing ("${FOLDER}", "${PATTERN}" and "${file}"); just do "$FOLDER", "$PATTERN" and "$file". (6) This assumes that files are in the current directory.  cps 's/$/.bak/' d/foo will copy d/foo to foo.bak in the current directory, not d/foo.bak.

[xiota]

@G-Man 1. Done. 2. N/A b/c scrapped the rename script. 3. N/A. 4. Tried. Testing exit statuses doesn't work. Don't know why. 5. Done. 6. Rewrote to work with files that may not be contained in current directory.

[xiota]

@WinEunuuchs2Unix Rewrote to work with files that aren't in the current directory and to use regular expressions via sed. May be slower than previous versions.

[G-Man Says 'Reinstate Monica']

@xiota: Well, that’s a bit of a shame; I thought the idea of piggybacking on the existing rename program was brilliant, even if the implementation was flawed. But +1 for making the improvements.

[0x5453]

Note that with zmv you can use "wildcard replacement" mode, which I find a bit easier to grok: zmv -W -C 'gmail-meta3*' '*.bak'

[xiota]

@G-Man Couldn't figure out how to fix problems you pointed out while still using rename.

[WinEunuuchs2Unix]

@KonradRudolph Thanks for your suggestion. I implemented and tested your suggestion with a couple minor changes.

[steeldriver]

@0x5453 thanks - I wasn't aware of that option

[pa4080]

Yesterday I made few tests with large files over 2GB and - yes, this approach is relatively slow than using of the cp command, but I can't say there is any significant performance difference.

[WinEunuuchs2Unix]

Although I upvoted your answer and all the others in this thread I regret to say I've accepted my own answer tonight. I used cps last .bak in the command line and four files in my directory were copied. I didn't need to remember sed command syntax you posted here two years ago. It's rather brilliant though.