How to authenticate with gcloud big query using a json credentials file?

Question

In the gcloud documentation for google bigquery, it states that authentication can be determined from from_service_account_json.

I've tried the following

from gcloud import bigquery
client = bigquery.Client.from_service_account_json('/Library/gcloud_api_credentials.json')

The json file looks like the following (Note: Scrambled credentials so these are now fake).

{"type": "service_account",
"project_id": "example_project",
  "private_key_id": "c7e371776ab6e2dsfafdsaff97edf9377178c8",
  "private_key": "-----BEGIN PRIVATE KEY-----\nXXXXXXXXAgEAAoIBAQDBIAaPzRVOT12J\nfPKzJ/tgVTJWHWiO29C/OpjRbd4WUFNEfX3mdShYoVo9+F7u3QSlCxA4SgZmro0s\n/yQKyhCrpcmrmWJyr5VqqIvqScJVJzBvLHCto/JoydCSzYQJz89vltZ11NZE1n+W\ncvF2ygX9oM/dyUK/1eVsJOIoj4qj2jpRuQhxgh0ag2HVYlmAfnP9wfTdLeoV0e/z\n0WtBNesCrekh/ooldbxcDa6KvLJpBKu6BJg0nYWtnB0Q2hEZl1msNDs9lsMIWI/4\nRwz4iEv67HDTDaTerF89sFsPW5dyybnspxuNXlNSjtdABpHTU4mm1bwgRSEomuyZ\nptMp+Ck/AgMBAAECggEAYXyDE2/Yw3D4rIBwGtOh5tzazfGGaCa51u4DWiR1qRnf\nDoGyXfTh6ut4HDQV58mVoKJXC7MCTC7sgLxCO1gI1jyX8pE4yt+rMu3lwtx8xnND\n1bq/HkfWA+Nr787iSfmmopOnqk9H4fPM7+sRzH7qSz8z7qa7lQ6qAs8TgpoCx/Zt\nQsfAUDBFHP2x2msngZVkW2rZiCJyiWDE62KI7uxlUXlnqkC9j6vXL+n/SBmVGYSn\n2crQ0CONdBXvk753E4e3AhhA7VUcQorhEnYGFcsI2L9s1mK2SOGnakAukyZpoqQG\n//UmCX/lSu4SpfKi+NFedtQLfZqpfU6tOhS90OzYwQKBgQD1v1FdJ05X92RzRlSb\nQUUqWNKC3OwSlFAEZnBrpY8xSlK8alLXQRQWhBjdugtnfRfvPokILoxtZcgC6rUY\ntyauXPdDbv8+U4+YtcRSf3ZqNQmIajwt/pS/aj9CpY+ZL62codVmXOnpjiaIEZLk\nECJC947oYpxBb6k5OIrecZHuzwKBgQDJLq5H4i3JNpmIHiC/xgDcL9ROrPJBZhaS\fdafsdafddsffdsf/+\nCzA/tpS/dn0i5bf4E/SFWJdFHXk/rIooEGypFNw9e2Sw5rElI3B9DYKkXpHWDLvS\ne0vHm7s6kQKBgQDAH0ZY8XXKR9r8VCupBSoxpSdOr7DQBatZBU02m0SLYHO2YdlJ\n99Lgl31pOx3XVnDz1YwEHYAjtsLbpOrX27gd8QhOa+SYsXM+DDzj0kqdVcGSKJvL\nyZA45bN3Q0A0npW4Mr1HPbSRp1RwZGzWZ9ahTDEpdI58ifNejNnkdh2E+QKBgQCQ\nkSDfuuJX8wpEzBGQfUJk92HC33h+ElQp8MOyl+2qrzTt5KQIkJ+7gbG1F2lQ/Qhc\nqaIDTPTozJds1YfpLARc/fdsfddfd/ovcaSayzV8dEGeXDKU7GrVds5\niwPqRHk3HRJwJHYrT9onoqu8XdDCVggp6ju8RmY2YQKBgQC+r2aGJTOE5BNEwoZa\nT5q5lda+ADFm9JQj9zcF6OI4UFQqJaW0KVlPdr2pH5h6PoRtyVsZ5ZbDSjbZ2+Jp\nT1fW54HoK2S+faXWEmHPIYbRGSSDRJ/Kphu28D4C8Jo8YJKn4rwd8dnWmYC/73cx\naqUiJ7TdFLUYNwl/tg/XGR3PTQ==\n-----END PRIVATE KEY-----\n",
  "client_email": "[email protected]",
  "client_id": "123422110336236066294",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/example-project%40appspot.gserviceaccount.com"
}

I get the below error:

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 2217, in <module>
    globals = debugger.run(setup['file'], None, None)
  File "/Applications/PyCharm.app/Contents/helpers/pydev/pydevd.py", line 1643, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/API/google_big_query_api.py", line 69, in <module>
    try2()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/Shippy/API/google_big_query_api.py", line 62, in try2
    client = bigquery.Client.from_service_account_json('/Library/gcloud_api_credentials.json')
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/client.py", line 59, in from_service_account_json
    credentials = get_for_service_account_json(json_credentials_path)
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/credentials.py", line 128, in get_for_service_account_json
    json_credentials_path, scopes=scope)
  File "build/bdist.macosx-10.6-x86_64/egg/oauth2client/service_account.py", line 195, in from_json_keyfile_name
  File "build/bdist.macosx-10.6-x86_64/egg/oauth2client/service_account.py", line 171, in _from_parsed_json_keyfile
  File "build/bdist.macosx-10.6-x86_64/egg/oauth2client/service_account.py", line 108, in __init__
  File "build/bdist.macosx-10.6-x86_64/egg/oauth2client/util.py", line 163, in scopes_to_string
TypeError

I think I may have incorrectly created the json credentials file, but I'm not sure how that is suppose to be made.

I've also tried to download the json file from google, but now get the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-79ea8cfc1391> in <module>()
----> 1 client = bigquery.Client.from_service_account_json('/Users/Chris/Desktop/Shippy-abc820f485e5.json')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/client.pyc in from_service_account_json(cls, json_credentials_path, *args, **kwargs)
     57         if 'credentials' in kwargs:
     58             raise TypeError('credentials must not be in keyword arguments')
---> 59         credentials = get_for_service_account_json(json_credentials_path)
     60         kwargs['credentials'] = credentials
     61         return cls(*args, **kwargs)

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/credentials.pyc in get_for_service_account_json(json_credentials_path, scope)
    126     """
    127     return ServiceAccountCredentials.from_json_keyfile_name(
--> 128         json_credentials_path, scopes=scope)
    129 
    130 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in from_json_keyfile_name(cls, filename, scopes)
    193         with open(filename, 'r') as file_obj:
    194             client_credentials = json.load(file_obj)
--> 195         return cls._from_parsed_json_keyfile(client_credentials, scopes)
    196 
    197     @classmethod

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in _from_parsed_json_keyfile(cls, keyfile_dict, scopes)
    169         credentials = cls(service_account_email, signer, scopes=scopes,
    170                           private_key_id=private_key_id,
--> 171                           client_id=client_id)
    172         credentials._private_key_pkcs8_pem = private_key_pkcs8_pem
    173         return credentials

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in __init__(self, service_account_email, signer, scopes, private_key_id, client_id, user_agent, **kwargs)
    106         self._service_account_email = service_account_email
    107         self._signer = signer
--> 108         self._scopes = util.scopes_to_string(scopes)
    109         self._private_key_id = private_key_id
    110         self.client_id = client_id

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/util.pyc in scopes_to_string(scopes)
    161         return scopes
    162     else:
--> 163         return ' '.join(scopes)
    164 
    165 

TypeError: 

I've also tried downloading the p12 file

bigquery.Client.from_service_account_p12('[email protected]','/Users/Chris/Desktop/Shippy-e824e0396911.p12')

I then receive the below error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-e71586550945> in <module>()
----> 1 client = bigquery.Client.from_service_account_p12('[email protected]','/Users/Chris/Desktop/Shippy-e824e0396911.p12')

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/client.pyc in from_service_account_p12(cls, client_email, private_key_path, *args, **kwargs)
     92             raise TypeError('credentials must not be in keyword arguments')
     93         credentials = get_for_service_account_p12(client_email,
---> 94                                                   private_key_path)
     95         kwargs['credentials'] = credentials
     96         return cls(*args, **kwargs)

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/gcloud-0.10.1-py2.7.egg/gcloud/credentials.pyc in get_for_service_account_p12(client_email, private_key_path, scope)
    157     """
    158     return ServiceAccountCredentials.from_p12_keyfile(
--> 159         client_email, private_key_path, scopes=scope)
    160 
    161 

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in from_p12_keyfile(cls, service_account_email, filename, private_key_password, scopes)
    276         return cls._from_p12_keyfile_contents(
    277             service_account_email, private_key_pkcs12,
--> 278             private_key_password=private_key_password, scopes=scopes)
    279 
    280     @classmethod

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in _from_p12_keyfile_contents(cls, service_account_email, private_key_pkcs12, private_key_password, scopes)
    245         signer = crypt.Signer.from_string(private_key_pkcs12,
    246                                           private_key_password)
--> 247         credentials = cls(service_account_email, signer, scopes=scopes)
    248         credentials._private_key_pkcs12 = private_key_pkcs12
    249         credentials._private_key_password = private_key_password

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/service_account.pyc in __init__(self, service_account_email, signer, scopes, private_key_id, client_id, user_agent, **kwargs)
    106         self._service_account_email = service_account_email
    107         self._signer = signer
--> 108         self._scopes = util.scopes_to_string(scopes)
    109         self._private_key_id = private_key_id
    110         self.client_id = client_id

/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/oauth2client-2.0.0.post1-py2.7.egg/oauth2client/util.pyc in scopes_to_string(scopes)
    161         return scopes
    162     else:
--> 163         return ' '.join(scopes)
    164 
    165 

TypeError: 

Solution

From https://googlecloudplatform.github.io/gcloud-python/latest/gcloud-api.html:

json_credentials_path (string) – The path to a private key file (this file was given to you when you created the service account). This file must contain a JSON object with a private key and other credentials information (downloaded from the Google APIs console).

It looks like the file should have been downloaded when you created the service account. Let's look at service account creation. Hiding in the storage docs we find a nice step-by-step for creating a JSON credential file.

From https://cloud.google.com/storage/docs/authentication#generating-a-private-key:

To generate a private key in JSON or PKCS12 format:

1. Open the list of existing credentials in the Google Cloud Platform Console.
2. Click New credentials and select Service account.
3. In the Create service account window, select the Key type, either JSON or P12.
4. Click Create.
5. A New public/private key pair window is displayed and the private key for the Key type you selected is downloaded automatically. If you selected a P12 key, the private key's password ("notasecret") is displayed.
6. Click Close.