Can I ignore files by pattern in deja-dup (Backup)?

Question

My Déjà Dup backups have become quiet large and I noticed they contain a huge number of unnecessary files (e.g. *.pyc files, **__pycache__ folders and other build related temporary stuff).

I know that I can ignore specific folders, but is there a way to exclude files and or folders by patterns?

I thought there might be more options usable trough a configuration file, but Déjà Dup doesn't use one. So I looked at duplicity (the CLI it is based on), but the man page doesn't mention a configuration file either. I know that duplicity can ignore files and folders based on patterns (--exclude, --exclude-filelist), but I have no idea how to combine this with Déjà Dup.

Do I have to ditch Déjà Dup and use duplicity manually? Or is there a way to set the needed options, so that they are used automatically, when duplicity is used by Déjà Dup?

Solution 1

I am using a variation of Ian Puleston answer, much simplified as the proposed script was not working for me. Refer to his answer, but use this content instead for the duplicity shim script:

#!/bin/bash

# Shim script run from deja-dup in place of duplicity 
# which forces the exclusions read from ~/.config/deja-dup-excludes 
# using the standard filelist exclusion mechanism

/usr/bin/duplicity --include=$HOME/.cache/deja-dup/metadata --exclude-filelist=$HOME/.config/deja-dup-excludes "$@"

This is tested on Ubuntu 18.04 and 20.04, and it uses duplicity's standard --exclude-filelist command, which supports some extended shell globbing patterns (for older duplicity versions you can swap it with --exclude-globbing-filelist). It's important that the exclusion appears first in order for everything to work. Be careful with what you exclude, no liabilities accepted, use at your own risk.

Notes about --include=$HOME/.cache/deja-dup/metadata

--include=$HOME/.cache/deja-dup/metadata is hardcoded in deja-dups invocation of duplicity. The file is created during backup and used for basic sanity checking, thus required to be included in the backup. If it's not, deja-dup fails with the following error at the end of the backup creation:

'metadata' file not found when creating backup ("Could not restore ‘/home/user/.cache/deja-dup/metadata’: File not found in backup"

Thus the --include=$HOME/.cache/deja-dup/metadata argument is required (an must come first) in the above shim script in case your exclusion rules in ~/.config/deja-dup-excludes happen to exclude $HOME/.cache (e.g. via /**/.cache). See the FILE SELECTION section in duplicity's manpage for details.

Solution 2

You can edit the exclude list like:

gsettings get org.gnome.DejaDup exclude-list
# remove comment to execute
# gsettings set org.gnome.DejaDup exclude-list ['path1', 'path2']

Source: https://answers.launchpad.net/deja-dup/+question/280954

I tried to add patterns like '**/.git' and '**/build' into that list, like this:

gsettings get org.gnome.DejaDup exclude-list > exclude-list
gedit exclude-list
gsettings set org.gnome.DejaDup exclude-list "`cat exclude-list`"

But to me it seems like the **'s were not passed to duplicity. So instead I ended up doing seaches like

locate "/home/*/.svn"
locate "/home/*/build"

and added them to the exclude-list manually

Solution 3

There is no way currently with Deja Dup to do advanced filtering like that. See upstream bug https://bugs.launchpad.net/deja-dup/+bug/374274

Solution 4

Using ** patterns do not (any longer) work because deja-dub escapes [?* characters in duplicity command. See https://git.launchpad.net/deja-dup/tree/libdeja/tools/duplicity/DuplicityJob.vala#n303 :

  string escape_duplicity_path(string path)
  {
    // Duplicity paths are actually shell globs.  So we want to escape anything
    // that might fool duplicity into thinking this isn't the real path.
    // Specifically, anything in '[?*'.  Duplicity does not have escape
    // characters, so we surround each with brackets.
    string rv;
    rv = path.replace("[", "[[]");
    rv = rv.replace("?", "[?]");
    rv = rv.replace("*", "[*]");
    return rv;
  }

  void process_include_excludes()
  {
    expand_links_in_list(ref includes, true);
    expand_links_in_list(ref excludes, false);

    // We need to make sure that the most specific includes/excludes will
    // be first in the list (duplicity uses only first matched dir).  Includes
    // will be preferred if the same dir is present in both lists.
    includes.sort((CompareFunc)cmp_prefix);
    excludes.sort((CompareFunc)cmp_prefix);

    foreach (File i in includes) {
      var excludes2 = excludes.copy();
      foreach (File e in excludes2) {
        if (e.has_prefix(i)) {
          saved_argv.append("--exclude=" + escape_duplicity_path(e.get_path()));
          excludes.remove(e);
        }
      }
      saved_argv.append("--include=" + escape_duplicity_path(i.get_path()));
      //if (!i.has_prefix(slash_home_me))
      //  needs_root = true;
    }
    foreach (File e in excludes) {
      saved_argv.append("--exclude=" + escape_duplicity_path(e.get_path()));
    }

    // TODO: Figure out a more reasonable way to order regexps and files.
    // For now, just stick regexps in the end, as they are more general.
    foreach (string r in exclude_regexps) {
      saved_argv.append("--exclude=" + r);
    }

    saved_argv.append("--exclude=**");
  }

Solution 5

1. install dconf-editor

sudo apt install dconf-editor

2. run dconf-editor as normal user. (don't use sudo)

dconf-editor

3. locate org -> gnome -> deja-dup -> exclude-list
4. set Custom value to (replace leo with your user name)

['$TRASH', '$DOWNLOAD', '/home/leo/.anaconda', '/home/leo/**/node_modules', '/home/leo/**/__pycache__', '/home/leo/**/*.pyc']

5. You might need to reboot/re-signin. I run Screenshot which automatically update the value. I don't know why, maybe someone else can explain.

---

Screenshots:

Replace leo with your user name

View more solutions

Comments

[esmail]

org -> gnome -> deja-dup a.k.a. org.gnome.DejaDup. Not working for me, though.

[NOAH MIGRALA]

you can edit the exclude list like: gsettings get org.gnome.DejaDup exclude-list gsettings set org.gnome.DejaDup exclude-list ['path1', 'path2']

[Salim B]

I guess you use your include_list.txt file by calling duplicity directly from the command line? Do you happen to know of any way to make such a pattern list work with Ubuntu's default Déjà Dup GUI? (BTW: It seems rather confusing to me naming a file containing inclusion and exclusion patterns include_list.txt...)

[musicformellons]

I tried this and the ~/**/node_modules does show in the 'Folder to ignore', but still they are backed-up..., so does not seem to work...

[musicformellons]

As Michael Terry is the maintainer of deja-dup, this is 'final'. See also this more recent issue in the deja-dup repo which confirms wildcards are still unsupported.

[Joël]

Looking at manpages, it seems that indeed duplicity provides --include-filelist option with the syntax presented here (look for FILE SELECTION), however the manpage for the GUI provides very few options (well, GUI-related options).

[Francois]

the ms-windows registry was such a good idea that some linux developers felt the need to borrow the idea

[jessexknight]

I made a script to export a long list of the (~350) excluded paths. However, it seems this really slows down the backup, though CPU is about the same.

[Brad]

Can you please update your answer to include the command line you use to kick off your backups? I'm not clear on how to use this file include_list.txt. According to Deja_dup glob paths are not supported

[Brad]

org.gnome.deja-dup doesn't exist for me. did you have to add this schema manually? see my post here

[ying]

when you set the list, you need to enclose the list with quote

[Bruno Bossola]

Thank you for the idea. This particular implementation was not working for me, but I added a simplified version that works on my system (Ubuntu 18). See askubuntu.com/a/1378432/347084

[Salim B]

This answer doesn't work because Deja Dup escapes the asterisks when passing the exclude list to the underlying tool duplicity, cf. gitlab.gnome.org/World/deja-dup/-/issues/112#note_912936