What's wrong with my bash script to keep last x files and delete the rest?

Question

I have this bash script which nicely backs up my database on a cron schedule:

#!/bin/sh

PT_MYSQLDUMPPATH=/usr/bin
PT_HOMEPATH=/home/philosop
PT_TOOLPATH=$PT_HOMEPATH/philosophy-tools
PT_MYSQLBACKUPPATH=$PT_TOOLPATH/mysql-backups
PT_MYSQLUSER=*********
PT_MYSQLPASSWORD="********"
PT_MYSQLDATABASE=*********
PT_BACKUPDATETIME=`date +%s`
PT_BACKUPFILENAME=mysqlbackup_$PT_BACKUPDATETIME.sql.gz
PT_FILESTOKEEP=14

$PT_MYSQLDUMPPATH/mysqldump -u$PT_MYSQLUSER -p$PT_MYSQLPASSWORD --opt $PT_MYSQLDATABASE | gzip -c > $PT_MYSQLBACKUPPATH/$PT_BACKUPFILENAME

Problem with this is that it will keep dumping the backups in the folder and not clean up old files. This is where the variable PT_FILESTOKEEP comes in. Whatever number this is set to thats the amount of backups I want to keep. All backups are time stamped so by ordering them by name DESC will give you the latest first.

Can anyone please help me with the rest of the BASH script to add the clean up of files? My knowledge of bash is lacking and I'm unable to piece together the code to do the rest.

Solution 1

First, be sure you are in right folder:

if [ -z $PT_MYSQLBACKUPPATH ]; then
 echo "No PT_MYSQLBACKUPPATH set. Exit"
 exit 1
fi
cd $PT_MYSQLBACKUPPATH
if [ $? != 0 ]; then
 echo "cd to PT_MYSQLBACKUPPATH failed. Exit"
 exit 1
fi

You can remove files older than n, in your case:

find -mtime +14 -delete

Deletes files older than 14 days.

More complicated (definitely not optimal, though) solution for your question:

# Get list of newest files. If newest files are first, use head -n 14 instead of 
# head.
files=(`ls | sort | tail -n 14`)
# Loop over all files in this folder
for i in *; do 
 preserve=0; 
 #Check whether this file is in files array:
 for a in ${files[@]}; do 
  if [ $i == $a ]; then 
   preserve=1; 
  fi; 
 done; 
 # If it wasn't, delete it (or in this case, print filename)
 if [ $preserve == 0 ]; then 
  echo $i; # test first, then change this to "rm $i"
 fi;
done

Solution 2

You could try this one:

ls -r1 $PT_MYSQLBACKUPPATH/ | tail -n +$(($PT_FILESTOKEEP+1)) | xargs rm

ls -r1 will list all files in reverse order, one file per line.

tail -n +$number filters the first $number-1 files of the list out (resp. displays all files beginning from $number till the last one).

xargs will execute rm with all file names from standard input.

Solution 3

Here is my usage of inspiration from this post:

#!/bin/bash
# Thu Jun 28 13:22:53 CEST 2012
# ${DESTDIR}/files2keep.sh
# Keep the 3 yungest files
# mra at miracleas.dk , deployed on RHEL 6.
InitValues(){
TODAY=`date +"%Y%m%d"`
NOW=`date +"%H%M"`
DESTDIR=/mnt/dbdmp
LOGFILE=?{0}-${TODAY}-${NOW}.log
}
BackupFileMaintenance(){
KEEPFILES=(`ls -lrt ${DESTDIR}/*mysqldump.sql.gz| tail -n 3| awk '{print $9}'`)
    for i in `ls -lrt ${DESTDIR}/*mysqldump.sql.gz | awk '{print $9}'`; do
    preserve=0 
    #Check whether this file is in files array:
        for a in ${KEEPFILES[@]}; do
                if [ $i == $a ]; then
                preserve=1 
                fi 
        done 
    if [ $preserve == 0 ]; then
    echo $i; # then change this to "rm -f $i" after test
    fi
    done
}
InitValues
BackupFileMaintenance
exit

Solution 4

Regarding answer from bmk, when available ls -t1 is safer than -r1 (sort by modification time rather than arbitrary file order)

Also on some versions of tail syntax is tail -n +$number (option -n is needed)

As a bonus, combining both find and ls, here is a way to remove files older than 30 days but keep at least 15 files:

ls -t1 $PT_MYSQLBACKUPPATH/|tail -n +16| xargs -n1 basename|xargs -n1 -I{} find $PT_MYSQLBACKUPPATH/ -mtime +30 -name {} -delete

View more solutions

Comments

[Scott]

This isnt what I was really after but actually will work better than what I had in mind. Before I test how do I make sure this is run in PT_MYSQLBACKUPPATH. Don't want to run this outside of that otherwise end up deleteing a whole load of files. Could be a disaster...

[Olli]

@Brady: I added example for checking environment.

[user1686]

Do not ever use for i in $(ls) or var=($(ls)) (Hint: for i in *) unless you can be 400% sure that the filenames will never contain spaces or anything like that.

[Scott]

@Olli - Ok that script is now working without error however I won't be able to tell if the 14 day deletion is working untill I have some old files in there. Will re-visit in 14 days if there is an issue. Thanks for the help.

[Olli]

@grawity: true, stupid me. However, how to do that with var=..., with sort and tail? (Feel free to edit answer instead of giving correct solution in comments)

[Dennis Williamson]

ls | sort sorts by name. Use ls -t | head -n 14 (it will still fail, though, for filenames that contain whitespace).

[Scott]

Thanks for your reply bmk but I have gone with using find -mtime +14 -delete provided by Olli

[DevSolar]

+1 because this very precisely answers the question.

[slhck]

Some explanation of the code you're posting would be great. Can you edit your answer?