How to run Google gsutil using Python
Note that the proper and official way to interact with Google Cloud Storage is to make use of the Google Cloud Client Library for Python
and not running the gsutil
command through subprocess.Popen
.
If you are not setting up merely some tests I would suggest you to follow from the beginning this way if there is not any technological constrain that makes this way impracticable.
You can check at the following links the relative Overview and Documentation. A small example taken from the Documentation can be the following:
from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket('<your-bucket-name>')
blob = bucket.blob('my-test-file.txt')
blob.upload_from_string('this is test content!')
You can find a further example here using google-cloud-python with the Datastore and Cloud Storage to manage expenses.
Related videos on Youtube
alphanumeric
Updated on June 04, 2022Comments
-
alphanumeric almost 2 years
After installing and configuring Google Cloud SDK
gsutil
command can be run by simply typing its name and the argument(-s) using Windows cmd.Here is the example:
"C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin\gcloud" version
But the same command fails if run using Python subprocess. With subprocess's
shell
argument set to True theImportError
occurs:import subprocess cmd = '"C:/Program Files (x86)/Google/Cloud SDK/google-cloud-sdk/bin/gsutil" version' p = subprocess.Popen(cmd, shell=True)
.....
ImportError: No module named site
With subprocess's
shell
argument set to False then theWindowsError: [Error 2] The system cannot find the file specified
occurs:p = subprocess.Popen(cmd, shell=False)
Is there a way to run
gsutil
on Windows using Python? -
HoosierDaddy over 4 yearsThe python API does not allow use of the
-m
option for parallelism, as far as I know. So there are reasons for usingsubprocess
and thegsutil
command. -
tfad334 over 4 years@UricSou: You can share client instances across threads because the storage client uses the
requests
library. Just create client instances aftermultiprocessing.Pool
. -
CpILL about 4 yearsAlso, the python API is dead slow comapred to the commandline
-
MAC about 2 years@GalloCedrone how to upload folder using above approach?