Permissions restoring from Time Machine - Finder copy vs "cp" copy

Question

Note: this question was starting to sprawl so I rewrote it.

I have a folder that I'm trying to restore from a Time Machine backup. Using cp -R works fine, but certain folders cannot be restored with either the Time Machine UI or Finder.

Other users have reported similar errors and the cp -R workaround was suggested (e.g. Restoring from Time Machine - Permissions Error). But I wanted to understand:

1. Why cp -R works when the Finder and the Time Machine UI do not.
2. Whether I could prevent the errors by changing file permissions before the backup.

There do indeed seem to be some permissions that Finder works with and some that it does not. I've narrowed the errors down to folders with the user ben (that's me) and the group wheel.

Here's a simplified reproduction. I have four folders with the owner/group combinations I've seen so far:

ben ~/Desktop/test $ ls -lea
total 16
drwxr-xr-x   7 ben   staff   238 27 Nov 14:31 .
drwx------+ 17 ben   staff   578 27 Nov 14:29 ..
 0: group:everyone deny delete
-rw-r--r--@  1 ben   staff  6148 27 Nov 14:31 .DS_Store
drwxr-xr-x   3 ben   staff   102 27 Nov 14:30 ben-staff
drwxr-xr-x   3 ben   wheel   102 27 Nov 14:30 ben-wheel
drwxr-xr-x   3 root  admin   102 27 Nov 14:31 root-admin
drwxr-xr-x   3 root  wheel   102 27 Nov 14:31 root-wheel

Each contains a single file called file with the same owner/group:

ben ~/Desktop/test $ cd ben-staff
ben ~/Desktop/test/ben-staff $ ls -lea
total 0
drwxr-xr-x  3 ben  staff  102 27 Nov 14:30 .
drwxr-xr-x  7 ben  staff  238 27 Nov 14:31 ..
-rw-r--r--  1 ben  staff    0 27 Nov 14:30 file

In the backup, they look like this:

ben /Volumes/Deimos/Backups.backupdb/Ben’s MacBook Air/Latest/Macintosh HD/Users/ben/Desktop/test $ ls -leA
total 16
-rw-r--r--@ 1 ben   staff  6148 27 Nov 14:34 .DS_Store
 0: group:everyone deny write,delete,append,writeattr,writeextattr,chown
drwxr-xr-x@ 3 ben   staff   102 27 Nov 14:51 ben-staff
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
drwxr-xr-x@ 3 ben   wheel   102 27 Nov 14:51 ben-wheel
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
drwxr-xr-x@ 3 root  admin   102 27 Nov 14:52 root-admin
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown
drwxr-xr-x@ 3 root  wheel   102 27 Nov 14:52 root-wheel
 0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown

Of these, ben-staff can be restored with Finder without errors. root-wheel and root-admin ask for my password and then restore without errors. But ben-wheel does not prompt for my password and gives the error:

The operation can’t be completed because you don’t have permission to access “file”.

Interestingly, I can restore the file from this folder by dragging it directly to my local drive (instead of dragging its parent folder), but when I do so its permissions are changed to ben/staff.

Here are the permissions after the restore for the three folders that worked correctly, and the file from ben-wheel that was changed to ben/staff.

ben ~/Desktop/test-restore $ ls -leA
total 16
-rw-r--r--@ 1 ben   staff  6148 27 Nov 14:46 .DS_Store
drwxr-xr-x  3 ben   staff   102 27 Nov 14:30 ben-staff
-rw-r--r--  1 ben   staff     0 27 Nov 14:30 file
drwxr-xr-x  3 root  admin   102 27 Nov 14:31 root-admin
drwxr-xr-x  3 root  wheel   102 27 Nov 14:31 root-wheel

Can anyone explain this behaviour? Why do Finder and the Time Machine UI break with the ben / wheel permissions? And why does cp -R work (even without sudo)?

Solution 1

The issue here is that Finder implements special handling of files restored from Time Machine to preserve all their permissions, which failed for files owned by the current user's account, but not a group he's a member of.

---

Usually, when copying files using cp, their attributes aren't retained. This can be changed using the -p option.

-p Cause cp to preserve the following attributes of each source file in the copy: modification time, access time, file flags, file mode, user ID, and group ID, as allowed by permissions. Access Control Lists (ACLs) and Extended Attributes (EAs), including resource forks, will also be preserved.

In both cases, you either copy all or none or these metadata. cp is clever enough to restore them only after all contained files have been processed ([source, see near If -p is in effect, set all the attributes).

If you don't have root permissions, ownership is not retained. The reason for this is that only root can make files owned by users not him and by groups he's not a member of.

---

To make Time Machine backups viewable but otherwise immutable in Finder, they are protected by Access Control Lists denying all users all modification permissions.

0: group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown

Since other attributes (other ACLs, extended attributes, file dates and permissions) should be retained when restoring from backup, special handling of these folders is required in Finder. It must remove a specific ACL entry, but retain everything else.

Additionally, Apple apparently wants Finder to retain all ownership information when copying files and directories from backup. This includes the group membership.

---

If your account is not the owner of the directories in question, Finder requests an administrator password and hands the copying off to its elevated privileges helper program Locum. This does not happen when you are the owner of a file in the backup set.

Now, one of the following cases happens:

• You are not the owner of the file: Finder asks for your password, Locum restores all permissions just as in the backup.
• You are the owner and a member of the file's group: Finder copies the file over and restores the group.
• You are the owner but not a member of the file's group: Finder copies the file over and fails to restore its group.

It's like it's trying to chown username:groupname the file, and prints an error if it fails — which it does if you're neither root (sudo) nor username and a member of groupname.

It behaves slightly differently if you're not copying folders, but files: While file ownership is retained, group membership is discarded if you're not a member of the group. This is what you saw when only copying a single file.

---

The obvious solutions to this problem:

• Prevent file ownerships that cause restoration to fail (i.e. owned by you and a group you're not a member of — most of the time, this isn't useful anyway)
• Make yourself a member of that particular group at least temporarily. Unfortunately I wasn't able to do this using dscl from the command line in a way that had an effect without logout or restart. Another side effect is that with wheel, you might run into problems with permissions depending on your system configuration.

Solution 2

Aside from regular UNIX file permissions (user/group/everyone each having their own read/write/execute permissions) Mac OS X also uses Access Control Lists (ACLs) that allow much more granular file/folder permissions settings.

Time Machine adds (prepends) the following ACL to all files:

group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown

(The above ACL is folder-specific but maps to regular files like so: add_file = write, add_subdirectory = append, delete_child = <none>)

This means that all files/folders inside a Time Machine backup are locked for everyone (even the root user).

If you restore your files manually from a Time Machine Backup (i.e. if you do not use the Migration Assistant) then all your files will keep those pesky Time Machine ACLs attached to them.

It is quite easy to remove the Time Machine ACLs. You have three options:

1) Swing the axe and remove the ACLs from your files/folders entirely (nothing wrong with that but not a very cautious strategy)

2) Remove the first entry from the ACLs of your files/folders (more cautious but not a perfect solution)

3) Remove specific restrictions from the ACLs of your files/folders (probably your best bet if you want to preserve non-Time Machine-imposed ACLs)

For all three options you need the Terminal which you will find in /Applications/Utilities

Option 1: Here's how you swing the axe:

If you know you have only your personal files in a folder called "My Recovered Files" on your Desktop and you know that those files don't have/need any fancy ACLs then you can type the following into your Terminal window:

chmod -R -N ~/Desktop/My\ Recovered\ Files

(if you don't know the Terminal-way of specifying a file/folder simply drag and drop the file/folder you want onto the Terminal window and the Terminal will type the correct file/folder name for you)

Option 2: Here's how you remove the first ACL entry

Same example as above. You have a folder called "My Recovered Files" on your Desktop. But in this case you have a few files with custom ACLs that you want to preserve. Type the following into the Terminal window:

chmod -R -a# 0 ~/Desktop/My\ Recovered\ Files

What makes the above solution "dangerous" is that it is not idempotent. An idempotent operation is an operation that can be applied over and over without changing the result after it has been applied once. Kind of like multiplying a number by 1. You can keep doing it but the result is always the same.

Why does that matter? Well, let's say that you have a file that already had an ACL before Time Machine prepended its own ACL entry. If you run the above command twice then you will have removed both the Time Machine ACL as well as the ACL that you probably didn't want to lose.

Plus the above solution is also not ideal for Time Machine files that are mixed in with other files. If any of these other (non-Time Machine) files have ACLs then the above command will remove those ACLs.

Option 3: Here's how you remove specific restrictions from an ACL

Aside from being able to specify which number entry of an ACL you want to remove you can also specify the specific restrictions you want to remove. So you could do this:

chmod -R -a "group:everyone deny add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown" ~

("~" means "my home directory", i.e. if your username is bob then "~" = "/Users/bob")

The above command is idempotent which means that you can run it over and over without ill effect. In fact, anybody can run it at any time. If there are no files/folders that have been locked by Time Machine in your home directory nothing will happen.

OK. I hope this was helpful for some people. I had the same problem with Time Machine permissions yesterday so I figured that I'd share what I found out.

Solution 3

Actually, you should use the tmutil to restore files from your Time Machine backups. This way you won't have to remove the extended attributes etc.

From the Mac OS X man page:

tmutil restore [-v] src ... dst

Restore the item src, which is inside a snapshot, to the location dst. The dst argument mimics the destination path semantics of the cp tool. You may provide multiple source paths to restore. The last path argument must be a destination.

When using the restore verb, tmutil behaves largely like Finder. Custom Time Machine metadata (extended security and other) will be removed from the restored data, and other metadata will be preserved.

Root privileges are not strictly required to perform restores, but tmutil does no permissions preflighting to determine your ability to restore src or its descendants. Therefore, depending on what you're restoring, you may need root privileges to perform the restore, and you should know this ahead of time. This is the same behavior you would encounter with other copy tools such as cp or ditto. When restoring with tmutil as root, ownership of the restored items will match the state of the items in the backup.

So basically you can use it like the cp command without worrying about the internal rest.

In most cases you want to restore with sudo, when you don't know, if you are the owner of the files or if you have write permissions for the target directory.

Comments

[Admin]

Finder should ask you for the administrator password...

[Admin]

I don't think that Finder should need the administrator password as cp doesn't need to be sudoed.

[Admin]

And I like your idea, but I can confirm that cp -a /Volumes/Deimos/Backups.backupdb/Ben’s\ MacBook\ Air/Latest/Macintosh\ HD/opt/osdev . still works.

[Admin]

I've added the permissions of some folders that work above. Yes, I just set up Time Machine this morning - it's the same computer and it's the default Time Machine volume.

[Admin]

Can you please restate your question? (In a comment responding to this? Its getting a bit confusing.

[Ben Challenor]

Thanks for the fantastic answer! One thing I don't understand though - if Apple has to special case Time Machine anyway, why doesn't Finder always ask for your password, so that it can always get permissions right? And would it be worth reporting this in Radar?

[HikeMike]

@BenChallenor Wouldn't you tire of always entering your password, no matter if required or not? 95+% of all file restorations are probably a user's personal data with full access permissions. They probably just failed to test this uncommon special condition. It's definitely a candidate for Apple Bug Reporter — but you need to decide whether it's actually worth your time. Personally, I'm not a fan of Radar, as non-fatal issues are usually ignored in my experience.

[HikeMike]

Cannot reproduce. How are you restoring files so that their Time Machine ACLs are retained?

[Sherwood Botsford]

I have had extensive problems with tmutil aborting an operation at an arbitrary file on large (>100 G) restores.