How to sync a local folder with a folder in a Google Cloud Platform bucket?

Question

I want to sync a folder in one of my buckets (or even the whole bucket) and a local folder in more than one windows computer. I don't mind running some script in the background. Is there any way to do this? I haven't found any resources out there. I've tried using CrossFTP, an FTP client that supports the Google Cloud Platform, but syncing is only available when paying those 30 bucks and I'd rather avoid that to be honest. FileZilla is a nice option too, but I can't seem to get it up and working with Google. Any ideas? Any help or suggestions would be greatly appreciated. Thanks in advance.

Solution 1

You can use gsutil and write a script which can run at intervals to upload/sync

To upload: https://cloud.google.com/storage/docs/gsutil/commands/cp#description

gsutil cp *.txt gs://my-bucket

For sync: https://cloud.google.com/storage/docs/gsutil/commands/rsync

gsutil rsync -d -r data gs://mybucket/data

For your use case, sync up to one of the buckets and then sync back to all of your machines.

Solution 2

To update the available options GCP now offers Data Transfer Service that can be scheduled to move on prem data to the cloud bucket.

Comments

[Cottand]

Thanks for the quick answer. So, I could manage gsutil to write the actual thing, but how do I write the script? Do I create a file and subsequently edit it? If so, where? etc, I know these might seem like stupid questions but I'm completely new to this, so please bear with me. Thanks again!

[Vikram Tiwari]

Here's quick tutorial on how to write bash scripts: linux.com/LEARN/writing-simple-bash-script

[Vikram Tiwari]

Rather setup a cron job! askubuntu.com/questions/2368/how-do-i-set-up-a-cron-job

[Cottand]

Great that's all I needed. Cheers!

[Cottand]

Thing is, if I only use gsutil cp to upload, any files I put on my directory will be uploaded (cool) but the ones I delete won't get deleted on the server, right? I'd have to use gsutil rsyinc -d for that somehow I guess, but without deleting the whole of the directory I'm uploading to? Supposing I manage to arrange that, when a file gets deleted on the server, to delete it on the local directory I'd have to gsutil rsync -d again and that'd re-download the rest of the files too, is this correct?

[Cottand]

Oooh the -d option only deletes extra files, ok, cool that solves that 2nd issue.

[Cottand]

And how do I run the script periodically on windows? The cron job doesn't really seem to apply tbh. The directory has to be on a windows machine.

[Vikram Tiwari]

Something like this should be enough: blogs.technet.microsoft.com/heyscriptingguy/2012/08/11/…

[dliu]

Maybe can also try gcsfuse? it basically mount the bucket to your local file system so you can access your bucket like a folder (it's in beta): github.com/GoogleCloudPlatform/gcsfuse