Copy files from Google Compute Engine instance to Google Cloud Storage bucket

Question

Is there a way to copy files from a Google Compute Engine instance directly to a Google Cloud Storage bucket? There doesn't seem to be any info on the gcloud compute copy-files help page or in the Google Cloud Storage documentation.

The only examples I am seeing are to download it locally and then upload it again, which doesn't make sense for me if there are very big files.

Solution

The best way to do this is to SSH into the instance and use the gsutil command to copy files directly from the GCE instance to a GCS bucket.

Keep in mind the instance needs to have Google Cloud Storage "write scope" which is a setting you need to create when you first create the instance OR you can add later using a service account.

If you're using a machine image that was provided by Google, gsutil is already installed on the VM instance.

Example:

gsutil cp file1 file2 gs://bucket

If you have a lot of files to upload, you can parallelize via -m:

gsutil -m cp file1 file2 gs://bucket

If you want to recursively upload a directory, use -r:

gsutil cp -r dir1 gs://bucket

See the docs for gsutil cp for more information.