Shell script to keep backups for last 3 days, last 3 weeks and last 3 months

Question

Am not entirely sure how to do this due to my lack of shell knowledge.

NOW=$(date +"%Y.%m.%d.%T")

tar czf /backups/web_backup_$NOW.tgz /web/

Can you please assist me to delete old backups so that it only keeps:

1. Last 3 days
2. One backup each from last 3 weeks
3. One backup each for last 3 months

Solution 1

something like this: (just an idea)

mkdir -p monthly
mkdir -p weekly

ln backup_$NOW.tgz weekly/

# find current month
month=$(date +%Y-%m-)
# find the first file of the current month in the weekly folder
first_monthly=$(ls --sort=time -1 weekly/*$month* 2>/dev/null | tail -1)
# and put it in the monthly folder
ln -f $first_monthly monthly/

# we need only 5 weekly backups
ls --sort=time -1 weekly/* 2>/dev/null | tail -n+6 >> /tmp/deletebackups.txt
# we need only 5 monthly backups
ls --sort=time -1 monthly/* 2>/dev/null | tail -n+6 >> /tmp/deletebackups.txt

# delete the extra files
#rm $(cat /tmp/deletebackups.txt) 2>/dev/null
xargs --arg-file /tmp/deletebackups.txt rm

Solution 2

Look at rsnapshot. It might do what you want out of the box.

Solution 3

If you use rsnapshot in the first place, it performs proper backup retention on the fly (it creates that structure of hourly, daily, weekly, ... snapshots for you).

If you do not create snapshots with rsnapshot or already have a considerably large set of snapshots and want to thin it out in hindsight, then have a look at timegaps. It is designed exactly this use case (simple implementation of backup retention policies). According to your use case, you could invoke timegaps like that:

$ timegaps days3,weeks3,months3 *.tgz

It would then identify those tgz files that need to be rejected (based on file modification time) and write those to stdout for review or further processing (non-invasive read-only mode). Timegaps can also --move these files or directlyor --delete them. If required, it can parse the item creation time from the file name (have a look at the help message) instead of retrieving it via stat().

Disclaimer: I am the author of timegaps.

Comments

[Dennis Williamson]

You need to do tail -n+6 to keep 5. There's a doubled "=" in the assignment to first_monthly. The last line would be better as: xargs --arg-file /tmp/deletebackups.txt rm.

[elated]

@Dennis sorry but my lack of shell scripting knowledge makes it difficult for me to even make these changes. can you please post the final script.

[Jure1873]

hmm...I've updated my post with Dennis Willimasons corrections, but I think you will be better off using a utility that already does what you want. Maybe sbackup or rsnapshot?

[Phil Hollenback]

You definitely want to use rsnapshot.