Automating pydrive verification process

Question

I am trying to automate the GoogleAuth process when using the pydrive library (https://pypi.python.org/pypi/PyDrive).

I've set up the pydrive and the google API such that my secret_client.json works but it requires web authentication for gdrive access every time i run my script:

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive

gauth = GoogleAuth()
gauth.LocalWebserverAuth()

drive = GoogleDrive(gauth)

textfile = drive.CreateFile()
textfile.SetContentFile('eng.txt')
textfile.Upload()
print textfile

drive.CreateFile({'id':textfile['id']}).GetContentFile('eng-dl.txt')

eng.txt is just a textfile. Moreover when I try to use the above script while I am logged into another account. It doesn't upload the eng.txt into my gdrive that generated the secret_client.json but the account that was logged in when I authorize the authentication

From the previous post, I've tried the following to automate the verification process but it's giving error messages:

import base64, httplib2
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive

from apiclient.discovery import build
from oauth2client.client import SignedJwtAssertionCredentials
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive

#gauth = GoogleAuth()
#gauth.LocalWebserverAuth()

# from google API console - convert private key to base64 or load from file
id = "464269119984-j3oh4aj7pd80mjae2sghnua3thaigugu.apps.googleusercontent.com"
key = base64.b64decode('COaV9QUlO1OdqtjMiUS6xEI8')

credentials = SignedJwtAssertionCredentials(id, key, scope='https://www.googleapis.com/auth/drive')
credentials.authorize(httplib2.Http())

gauth = GoogleAuth()
gauth.credentials = credentials

drive = GoogleDrive(gauth)

drive = GoogleDrive(gauth)

textfile = drive.CreateFile()
textfile.SetContentFile('eng.txt')
textfile.Upload()
print textfile

drive.CreateFile({'id':textfile['id']}).GetContentFile('eng-dl.txt')

Error:

Traceback (most recent call last):
  File "/home/alvas/git/SeedLing/cloudwiki.py", line 29, in <module>
    textfile.Upload()
  File "/usr/local/lib/python2.7/dist-packages/pydrive/files.py", line 216, in Upload
    self._FilesInsert(param=param)
  File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 53, in _decorated
    self.auth.Authorize()
  File "/usr/local/lib/python2.7/dist-packages/pydrive/auth.py", line 422, in Authorize
    self.service = build('drive', 'v2', http=self.http)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apiclient/discovery.py", line 192, in build
    resp, content = http.request(requested_url)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 132, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 475, in new_request
    self._refresh(request_orig)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 653, in _refresh
    self._do_refresh_request(http_request)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 677, in _do_refresh_request
    body = self._generate_refresh_request_body()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 861, in _generate_refresh_request_body
    assertion = self._generate_assertion()
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 977, in _generate_assertion
    private_key, self.private_key_password), payload)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/crypt.py", line 131, in from_string
    pkey = crypto.load_pkcs12(key, password).get_privatekey()
OpenSSL.crypto.Error: [('asn1 encoding routines', 'ASN1_get_object', 'header too long')]

My authentication on gdrive api looks like this:

How could I use pydrive such that I do not need to authenticate everytime I use it?

How to allow automatic authentication such that the python script using the pydrive script will only upload to the account that generated the secret_client.json and not the currently logged on account on the internet browser?

Solution 1

First, you're misunderstanding one very important bit of how this works:

when I try to use the above script while I am logged into another account. It doesn't upload the eng.txt into my gdrive that generated the secret_client.json but the account that was logged in when I authorize the authentication

This is exactly how it's supposed to work. You, as the developer, distribute client_secret.json with your application, and that file is used by PyDrive to authenticate the application with Google. Google wants to know how many API requests are being made by each application out there for all sorts of reasons (metrics, charge the account, revoke access, etc.), so it requires the application to authenticate itself.

Now, when your application runs LocalWebserverAuth, it's authenticating the client with Google. The client, of course, is the person actually using your application. In this case, the developer and client are the same person (you), but imagine your want to distribute your application to a million different people. They need to be able to authenticate themselves and upload files to their own Drive account, rather that having them all end up in yours (the developer), who provided client_secret.json.

That said, it's really just a very minor change to make it so your app doesn't have to ask the client to authenticate every time you run the app. You just need to use LoadCredentialsFile and SaveCredentialsFile.

from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive

gauth = GoogleAuth()
# Try to load saved client credentials
gauth.LoadCredentialsFile("mycreds.txt")
if gauth.credentials is None:
    # Authenticate if they're not there
    gauth.LocalWebserverAuth()
elif gauth.access_token_expired:
    # Refresh them if expired
    gauth.Refresh()
else:
    # Initialize the saved creds
    gauth.Authorize()
# Save the current credentials to a file
gauth.SaveCredentialsFile("mycreds.txt")

drive = GoogleDrive(gauth)

textfile = drive.CreateFile()
textfile.SetContentFile('eng.txt')
textfile.Upload()
print textfile

drive.CreateFile({'id':textfile['id']}).GetContentFile('eng-dl.txt')

Solution 2

An alternative way is to use a custom auth flow by writing a setting.yaml file into the working directory. And this method works better as LocalWebserverAuth() will generate a token that expires in just one hour and there is no refresh token.

A sample settings.yaml file looks like this

client_config_backend: file
client_config:
    client_id: <your_client_id>
    client_secret: <your_secret>

save_credentials: True
save_credentials_backend: file
save_credentials_file: credentials.json

get_refresh_token: True

oauth_scope:
    - https://www.googleapis.com/auth/drive
    - https://www.googleapis.com/auth/drive.install

With this file, you still have to use a browser to complete authentication for the first time, and after that a credentials.json file will be generated in the working directory with a refresh token.

This method works better if you are trying to automate your script on server

Solution 3

This whole thread helped me a lot, but after I implemented all the solutions presented here one more issue came along: LocalWebserverAuth() won't get the refresh token.

If you open the "mycreds.txt" generated after you implement @dano's code, you'll see that the "refresh token" will be set to "null". After a couple of hours, the token expires and you get the following and end up having to manually authenticate again.

The error:

raise RefreshError('No refresh_token found.') pydrive.auth.RefreshError: No refresh_token found.Please set access_type of OAuth to offline.

The solution for that is to force the approval_promt and set access_type to offline on the flow params of the GoogleAuth.

Here's how I got no more errors:

gauth = GoogleAuth()

# Try to load saved client credentials
gauth.LoadCredentialsFile("mycreds.txt")

if gauth.credentials is None:
    # Authenticate if they're not there

    # This is what solved the issues:
    gauth.GetFlow()
    gauth.flow.params.update({'access_type': 'offline'})
    gauth.flow.params.update({'approval_prompt': 'force'})

    gauth.LocalWebserverAuth()

elif gauth.access_token_expired:

    # Refresh them if expired

    gauth.Refresh()
else:

    # Initialize the saved creds

    gauth.Authorize()

# Save the current credentials to a file
gauth.SaveCredentialsFile("mycreds.txt")  

drive = GoogleDrive(gauth)

Thank you all!

Solution 4

This is just to complete @wang892 post above (I have not enough reputation to comment).

That answer helped me to automate my script (not having to reauthenticate each time I run it).

But as I used the sample settings.yaml file available in PyDrive documentation, I ran into problems (due to my complete ignorance about how oauth works).

That sample file contains these lines, which I think were limiting my PyDrive script to access only to files and folders created by itself (see PyDrive issue #122 for details):

Limited access:

oauth_scope:
  - https://www.googleapis.com/auth/drive.file
  - https://www.googleapis.com/auth/drive.install

When I changed those lines the problem was solved (I had to remove my stored credentials and ran the script to reauthorise it, just once again).

With these new lines my script has now access to all files in my Google Drive:

Full access:

oauth_scope:
  - https://www.googleapis.com/auth/drive

A bit more about this in PyDrive issue #108, which enlighted me a lot.

Solution 5

If the credentials are not in place, this code generates an input box with two options:

• Browser authentication(which you need to do just once)

• Upload of the credentials file (this file will be generated the fist time you choose for Browser authentication

Now it is easy to share the notebook, which will just run without asking for authorization, since it will be using the credentials saved in the mycreds.txt from the local environment. However, if the runtime crashes or is reset, that file will be lost and it need to be inserted again via the input box above. Of course you can do this again via the Browser authentication, but if you redistribute the mycreds.txt to the people that are using the notebook, they can use the Upload function to insert the credentials to the local environment.

The final few lines just provide an example of how a csv file from the authenticated drive can be uploaded and used in the notebook.

#Install the required packages and fix access to my Google drive account
!pip install pydrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials


#Checks for file with Google authentication key, if the file is not in place, it asks to authenticate via the browser
gauth = GoogleAuth()
if os.path.isfile("mycreds.txt") is False:
    choice = input ("Do you want to: U) Upload authentication file (mycreds.txt). B) Browser authentication (only possible for owner of the connected Google drive folder). [U/B]? : ")
    if choice == "U":
          print ("Upload the mycreds.txt file")
          from google.colab import files
          files.upload()      
    elif choice == "B":
          auth.authenticate_user()
          gauth.credentials = GoogleCredentials.get_application_default()
          gauth.SaveCredentialsFile("mycreds.txt")

gauth.LoadCredentialsFile("mycreds.txt")
if gauth.access_token_expired:
    gauth.Refresh()
else: gauth.Authorize()

#Now you can easily use the files from your drive by using their ID  
drive = GoogleDrive(gauth)
download = drive.CreateFile({'id': '1KRqYpR9cteX-ZIwhdfghju6_wALl4'})
download.GetContentFile('my_data.csv')
data_frame = pd.read_csv('my_data.csv')

View more solutions

Comments

[Gregg]

What information goes in the mycreds.txt file and do you still require the client_secrets.json ? When I use the above code with the client_secrets.json it still requests a verification code, or have I missed the boat totally ?

[dano]

@Gregg You still need client_secrets.json to authenticate the App. mycreds.txt will store tokens that are can be used to authenticate the user. So the first time you run this code, you'll need to enter a verification code via your web browser. The second time, it should use the credentials saved in mycreds.txt.

[Joe]

This made my day. Thank you :)

[iChux]

Just like @Joe you made my day. I'm grateful.

[Sakeer]

@dano You just awesome

[Townheadbluesboy]

Hi - I'm just starting out with this stuff - following the quickstart and adding your very helpful suggestion. I noticed 'setting.yaml' should be 'settings.yaml' - then it worked lovely.

[avg]

Thanks, this works like a charm! The PyDrive Quickstart guide is not too clear with the details.

[avg]

How does this method compare with @wang892 's method? Will I need to keep authorising manually after the token expires?

[James Shapiro]

Could you provide some code similar to the example that dano provided above that shows how your authentication code actually works?

[Sulli]

this is almost what I need but the authentification part does not work for some reason. I can generate mycreds.txt without problems but then the script ends without message and without mounting google drive

[elvis-chuks]

What if i want all files uploaded to a central drive (my own google drive), what approach would i take?

[pmalbu]

The addition of access_type: offline and approval_prompt: force made my day! Thank you so much! Brilliant 🎉 Btw, where is the documentation for the Gauth Flow stuff? I couldn't find it..

[agenis]

perso, i had to have a settings.yaml file with get_refresh_token : True to have it work

[Kevin R.]

agree with @Sakeer you awesome!

[King]

hi may i know if where i can get mycreds.txt? thanks

[PAYRE Quentin]

Thanks a lot you saved my day

[sh37211]

What is eng.txt and how do we create it prior to running this? line 22, in <module> textfile.SetContentFile('eng.txt') File "/home/me/envs/myenv/lib/python3.9/site-packages/pydrive/fil‌​es.py", line 169, in SetContentFile self.content = open(filename, 'rb') FileNotFoundError: [Errno 2] No such file or directory: 'eng.txt'

[dano]

@sh37211 It's just a file that the question asker was trying to upload. It has nothing to do with the authentication process.

[sh37211]

@dano thanks for your reply! lol I should've read the question more carefully instead of just skipping to the answer! d'oh.

[Meet Gondaliya]

what information is present in the mycreds.txt file ?

[dano]

@MeetGondaliya See my comment above where I answered that same question.

[Meet Gondaliya]

Hey, I used this script successfully, but when I used it in a pythonanywhere hosted website, gauth.LocalWebserverAuth() doesnt work. No authentication window is opening and also I am getting an error that cannot find mycreds.txt file even though I used the same code. Can you think of anything which might cause this error?