Safe way of synchronising Dropbox with backup

Question

I use Dropbox but also like to back up my files to an external hard drive. I am currently doing a one way sync from Dropbox to the external drive every hour. This is very inefficient as the application rechecks all the checksums each time. However, I do this as I don't understand what Dropbox does if you copy a large file to it and then half way through the copying try to copy the file to a backup.

In my set up, I copy the files to my Dropbox folder in location A, and then copy the files from the Dropbox folder in location B to the external hard drive. So Dropbox is basically used as a way to transfer the files to nearer the external hard drive.

Is there a safe and guaranteed way of backing up Dropbox files that doesn't suffer from this sort of possible race condition?

---

Further clarification: The question is if Dropbox updates the local copies atomically (using locks or renaming or some other trick) and preserves file attributes. However, I believe (now that someone told me) that attributes such as modified time, created time etc. are not carried over between machines.

To be clear, this is not a question about uploading or how Dropbox handles updates on its servers. This is only a question about downloading and whether there is any time period when the local copy of a file is incomplete or inconsistent. The only sensible way I can see to avoid inconsistency would be if a copy is made of a local file that is being modified, the copy is changed, the old copy deleted and the new copy renamed. Is that actually what happens? If the file is very large this has obvious space problems so I don't know if they do it.

Solution

It seems that dropbox does download a file into cache and then after it is completely downloaded it then copies and deletes the old version locally. Attributes such as modified time, created time etc are not carried over between machines but the local "last modified" date on the windows machine will do to see what has changed. This is enough to let a standard backup utility to its work.

Comments

[Ramhound]

There are many tools that exist that will allow to sync two folders under specific conditions. I suggest you use one of those tools to do this. These tools will allow you to verify the checksums are actually different before the job is performed which prevents the situation you are concerned about.

[graffe]

@Ramhound This is what I do at the moment but it means that it is rechecking every single file on both sides each time which is slow.

[Ramhound]

Except...The files are local so the scan should be fast. The program I use only updates files that were changed since the last run of the job.

[graffe]

How do you stop it from copying half synched files? I mean large files which dropbox is in the process of downloading to your dropbox folder.

[Ramhound]

If you mean files I have placed in the folder and Dropbox starts to upload to my account which are then modified, this isn't really a problem, since I designed my job to run once every single day. This allows any file to be fully uploaded to my account and only after its complete do I update the file.

[graffe]

No sorry. I mean files that dropbox is in the middle of copying from its online server to my local dropbox folder (see comments below the answer for example).

[graffe]

In my case I copy the files to my dropbox folder in location A, and then copy the files from the dropbox folder in location B to the hard drive. This is the possible problem.

[user1301428]

I see. In this case, if I understand correctly, if you copy the file to the hard drive from the dropbox folder b, then it means that the file has already finished uploading on dropbox servers, so your data should already be synced. Does this make sense to you?

[graffe]

Right so that's the question. Can a large file ever be in the dropbox folder at location b half copied? Because I am copying from the local drive/dropbox folder at location B to the external drive.

[user1301428]

Not that I know. The only thing that can be "half copied" is a folder, if you copy it before all the files in it have been synced. But in the case of a unique, large file, then I think there is no problem :)

[graffe]

This is great if true but... :) It seems hard to imagine how a 1GB file could just appear instantly in your local dropbox folder. Unless there is some clever locking mechanism.

[user1301428]

Let's say there were no locking mechanism, now that I think about it, dropbox gives you the ability to sync your files on LAN, so if you need to sync large files you could use that, to reduce the risk of having half files.

[Ramhound]

Half files cannot exist on Windows. The folder DropBox is using to sync with the online content wouldn't start to sync the file until it was complete.

[graffe]

@Ramhound You mean that the file is downloaded to a temporary location locally by dropbox and then renamed to the correct name when it is completely downloaded?

[graffe]

The same issue arises when updating an existing file. The question is basically if dropboxes updates atomically (using locks or renaming or some other trick). So when the "updating file" symbol shows on the file icon, are the files intact (old version) until the new version is fully downloaded, at which point the file is replaced in full and the file attributes (time etc.) are updated too?

[Ramhound]

@felipa - You are not being clear with what you want to know. A system within Windows cannot be incomplete. So the file being uploaded to dropbox is complete on your local system, Dropbox wouldn't get rid of the CURRENT FILE, until the file being updated is FULLY UPLOADED this is common sense. So whats your question?

[user1301428]

@Ramhound I think that he's trying to ask if a file, during the update, is "incrementally" modified during the download of a new file, or if it stays the time until the whole new file has been downloaded, at which point the old file becomes the new one, so to say

[Ramhound]

@user1301428 - Dropbox does not update the file until its complete on their end. Of course there is a reason this question was closed and I am starting to understand the reason.

[user1301428]

@Ramhound but I think that the op wanted to know how the update process works after the file has been updated on Dropbox's end. Oh well, I don't know, let's just wait for him to explain ^^

[graffe]

@user1301428 You are exactly right. I am talking only about the download part. I am not talking about uploading at all. The question is simply whether during the download part where either a new file is being downloaded from the dropbox servers or an existing file on the local system is being updated from the dropbox servers, is there ever even a short time period where the local copy is incomplete/inconsistent. If it works by making a local copy, updating that and then renaming then I guess not. But does it because that could be quite expensive space wise locally?