I need to securely store a username and password in Python, what are my options?

Question

I'm writing a small Python script which will periodically pull information from a 3rd party service using a username and password combo. I don't need to create something that is 100% bulletproof (does 100% even exist?), but I would like to involve a good measure of security so at the very least it would take a long time for someone to break it.

This script won't have a GUI and will be run periodically by cron, so entering a password each time it's run to decrypt things won't really work, and I'll have to store the username and password in either an encrypted file or encrypted in a SQLite database, which would be preferable as I'll be using SQLite anyway, and I might need to edit the password at some point. In addition, I'll probably be wrapping the whole program in an EXE, as it's exclusively for Windows at this point.

How can I securely store the username and password combo to be used periodically via a cron job?

Solution 1

I recommend a strategy similar to ssh-agent. If you can't use ssh-agent directly you could implement something like it, so that your password is only kept in RAM. The cron job could have configured credentials to get the actual password from the agent each time it runs, use it once, and de-reference it immediately using the del statement.

The administrator still has to enter the password to start ssh-agent, at boot-time or whatever, but this is a reasonable compromise that avoids having a plain-text password stored anywhere on disk.

Solution 2

The python keyring library integrates with the CryptProtectData API on Windows (along with relevant API's on Mac and Linux) which encrypts data with the user's logon credentials.

Simple usage:

import keyring

# the service is just a namespace for your app
service_id = 'IM_YOUR_APP!'

keyring.set_password(service_id, 'dustin', 'my secret password')
password = keyring.get_password(service_id, 'dustin') # retrieve password

Usage if you want to store the username on the keyring:

import keyring

MAGIC_USERNAME_KEY = 'im_the_magic_username_key'

# the service is just a namespace for your app
service_id = 'IM_YOUR_APP!'  

username = 'dustin'

# save password
keyring.set_password(service_id, username, "password")

# optionally, abuse `set_password` to save username onto keyring
# we're just using some known magic string in the username field
keyring.set_password(service_id, MAGIC_USERNAME_KEY, username)

Later to get your info from the keyring

# again, abusing `get_password` to get the username.
# after all, the keyring is just a key-value store
username = keyring.get_password(service_id, MAGIC_USERNAME_KEY)
password = keyring.get_password(service_id, username)  

Items are encrypted with the user's operating system credentials, thus other applications running in your user account would be able to access the password.

To obscure that vulnerability a bit you could encrypt/obfuscate the password in some manner before storing it on the keyring. Of course, anyone who was targeting your script would just be able to look at the source and figure out how to unencrypt/unobfuscate the password, but you'd at least prevent some application vacuuming up all passwords in the vault and getting yours as well.

Solution 3

There are a few options for storing passwords and other secrets that a Python program needs to use, particularly a program that needs to run in the background where it can't just ask the user to type in the password.

Problems to avoid:

1. Checking the password in to source control where other developers or even the public can see it.
2. Other users on the same server reading the password from a configuration file or source code.
3. Having the password in a source file where others can see it over your shoulder while you are editing it.

Option 1: SSH

This isn't always an option, but it's probably the best. Your private key is never transmitted over the network, SSH just runs mathematical calculations to prove that you have the right key.

In order to make it work, you need the following:

• The database or whatever you are accessing needs to be accessible by SSH. Try searching for "SSH" plus whatever service you are accessing. For example, "ssh postgresql". If this isn't a feature on your database, move on to the next option.
• Create an account to run the service that will make calls to the database, and generate an SSH key.
• Either add the public key to the service you're going to call, or create a local account on that server, and install the public key there.

Option 2: Environment Variables

This one is the simplest, so it might be a good place to start. It's described well in the Twelve Factor App. The basic idea is that your source code just pulls the password or other secrets from environment variables, and then you configure those environment variables on each system where you run the program. It might also be a nice touch if you use default values that will work for most developers. You have to balance that against making your software "secure by default".

Here's an example that pulls the server, user name, and password from environment variables.

import os

server = os.getenv('MY_APP_DB_SERVER', 'localhost')
user = os.getenv('MY_APP_DB_USER', 'myapp')
password = os.getenv('MY_APP_DB_PASSWORD', '')

db_connect(server, user, password)

Look up how to set environment variables in your operating system, and consider running the service under its own account. That way you don't have sensitive data in environment variables when you run programs in your own account. When you do set up those environment variables, take extra care that other users can't read them. Check file permissions, for example. Of course any users with root permission will be able to read them, but that can't be helped. If you're using systemd, look at the service unit, and be careful to use EnvironmentFile instead of Environment for any secrets. Environment values can be viewed by any user with systemctl show.

Option 3: Configuration Files

This is very similar to the environment variables, but you read the secrets from a text file. I still find the environment variables more flexible for things like deployment tools and continuous integration servers. If you decide to use a configuration file, Python supports several formats in the standard library, like JSON, INI, netrc, and XML. You can also find external packages like PyYAML and TOML. Personally, I find JSON and YAML the simplest to use, and YAML allows comments.

Three things to consider with configuration files:

1. Where is the file? Maybe a default location like ~/.my_app, and a command-line option to use a different location.
2. Make sure other users can't read the file.
3. Obviously, don't commit the configuration file to source code. You might want to commit a template that users can copy to their home directory.

Option 4: Python Module

Some projects just put their secrets right into a Python module.

# settings.py
db_server = 'dbhost1'
db_user = 'my_app'
db_password = 'correcthorsebatterystaple'

Then import that module to get the values.

# my_app.py
from settings import db_server, db_user, db_password

db_connect(db_server, db_user, db_password)

One project that uses this technique is Django. Obviously, you shouldn't commit settings.py to source control, although you might want to commit a file called settings_template.py that users can copy and modify.

I see a few problems with this technique:

1. Developers might accidentally commit the file to source control. Adding it to .gitignore reduces that risk.
2. Some of your code is not under source control. If you're disciplined and only put strings and numbers in here, that won't be a problem. If you start writing logging filter classes in here, stop!

If your project already uses this technique, it's easy to transition to environment variables. Just move all the setting values to environment variables, and change the Python module to read from those environment variables.

Solution 4

After looking though the answers to this and related questions, I've put together some code using a few of the suggested methods for encrypting and obscuring secret data. This code is specifically for when the script has to run without user intervention (if the user starts it manually, it's best to have them put in the password and only keep it in memory as the answer to this question suggests). This method isn't super-secure; fundamentally, the script can access the secret info so anyone who has full system access has the script and its associated files and can access them. What this does do id obscures the data from casual inspection and leaves the data files themselves secure if they are examined individually, or together without the script.

My motivation for this is a project that polls some of my bank accounts to monitor transactions - I need it to run in the background without me re-entering passwords every minute or two.

Just paste this code at the top of your script, change the saltSeed and then use store() retrieve() and require() in your code as needed:

from getpass import getpass
from pbkdf2 import PBKDF2
from Crypto.Cipher import AES
import os
import base64
import pickle


### Settings ###

saltSeed = 'mkhgts465wef4fwtdd' # MAKE THIS YOUR OWN RANDOM STRING

PASSPHRASE_FILE = './secret.p'
SECRETSDB_FILE = './secrets'
PASSPHRASE_SIZE = 64 # 512-bit passphrase
KEY_SIZE = 32 # 256-bit key
BLOCK_SIZE = 16  # 16-bit blocks
IV_SIZE = 16 # 128-bits to initialise
SALT_SIZE = 8 # 64-bits of salt


### System Functions ###

def getSaltForKey(key):
    return PBKDF2(key, saltSeed).read(SALT_SIZE) # Salt is generated as the hash of the key with it's own salt acting like a seed value

def encrypt(plaintext, salt):
    ''' Pad plaintext, then encrypt it with a new, randomly initialised cipher. Will not preserve trailing whitespace in plaintext!'''

    # Initialise Cipher Randomly
    initVector = os.urandom(IV_SIZE)

    # Prepare cipher key:
    key = PBKDF2(passphrase, salt).read(KEY_SIZE)

    cipher = AES.new(key, AES.MODE_CBC, initVector) # Create cipher

    return initVector + cipher.encrypt(plaintext + ' '*(BLOCK_SIZE - (len(plaintext) % BLOCK_SIZE))) # Pad and encrypt

def decrypt(ciphertext, salt):
    ''' Reconstruct the cipher object and decrypt. Will not preserve trailing whitespace in the retrieved value!'''

    # Prepare cipher key:
    key = PBKDF2(passphrase, salt).read(KEY_SIZE)

    # Extract IV:
    initVector = ciphertext[:IV_SIZE]
    ciphertext = ciphertext[IV_SIZE:]

    cipher = AES.new(key, AES.MODE_CBC, initVector) # Reconstruct cipher (IV isn't needed for edecryption so is set to zeros)

    return cipher.decrypt(ciphertext).rstrip(' ') # Decrypt and depad


### User Functions ###

def store(key, value):
    ''' Sore key-value pair safely and save to disk.'''
    global db

    db[key] = encrypt(value, getSaltForKey(key))
    with open(SECRETSDB_FILE, 'w') as f:
        pickle.dump(db, f)

def retrieve(key):
    ''' Fetch key-value pair.'''
    return decrypt(db[key], getSaltForKey(key))

def require(key):
    ''' Test if key is stored, if not, prompt the user for it while hiding their input from shoulder-surfers.'''
    if not key in db: store(key, getpass('Please enter a value for "%s":' % key))


### Setup ###

# Aquire passphrase:
try:
    with open(PASSPHRASE_FILE) as f:
        passphrase = f.read()
    if len(passphrase) == 0: raise IOError
except IOError:
    with open(PASSPHRASE_FILE, 'w') as f:
        passphrase = os.urandom(PASSPHRASE_SIZE) # Random passphrase
        f.write(base64.b64encode(passphrase))

        try: os.remove(SECRETSDB_FILE) # If the passphrase has to be regenerated, then the old secrets file is irretrievable and should be removed
        except: pass
else:
    passphrase = base64.b64decode(passphrase) # Decode if loaded from already extant file

# Load or create secrets database:
try:
    with open(SECRETSDB_FILE) as f:
        db = pickle.load(f)
    if db == {}: raise IOError
except (IOError, EOFError):
    db = {}
    with open(SECRETSDB_FILE, 'w') as f:
        pickle.dump(db, f)

### Test (put your code here) ###
require('id')
require('password1')
require('password2')
print
print 'Stored Data:'
for key in db:
    print key, retrieve(key) # decode values on demand to avoid exposing the whole database in memory
    # DO STUFF

The security of this method would be significantly improved if os permissions were set on the secret files to only allow the script itself to read them, and if the script itself was compiled and marked as executable only (not readable). Some of that could be automated, but I haven't bothered. It would probably require setting up a user for the script and running the script as that user (and setting ownership of the script's files to that user).

I'd love any suggestions, criticisms or other points of vulnerability that anyone can think of. I'm pretty new to writing crypto code so what I've done could almost certainly be improved.

Solution 5

There's not much point trying to encrypt the password: the person you're trying to hide it from has the Python script, which will have the code to decrypt it. The fastest way to get the password will be to add a print statement to the Python script just before it uses the password with the third-party service.

So store the password as a string in the script, and base64 encode it so that just reading the file isn't enough, then call it a day.

View more solutions

Comments

[Naftuli Kay]

I'll need to edit the username and password periodically and I'll be wrapping the whole thing in an EXE for Windoze; I've edited the post to reflect this. Should I simply base64 it wherever I do end up storing it?

[Naftuli Kay]

Unfortunately, it's Windows, I'll be wrapping it in an EXE, and I'll need to change the password every so often, so hard-coding it won't be an option.

[Aishik kirtaniya]

Windows still has file system permissions. Store the password in an external file and remove everyone's access excluding your own. You probably also have to remove their administrative privileges.

[wberry]

I agree that "encrypting" the password does not help, since the plain-text password must anyway be obtained in an automated fashion, and therefore must be obtainable from whatever is stored. But there are viable approaches.

[Voo]

Yeah using permissions is the only reliable security option here. Obviously any administrator will still be able to access the data (at least on windows/usual linux distributions) but then that's a battle already lost.

[Naftuli Kay]

+1, that makes a lot of sense. I could always build a UI for it which essentially asks the user for his password on boot, that way it's never stored on disk and is safe from prying eyes.

[Sepero]

It's true. When password decryption is automated, then that's just as good as having a plain text password. The real security is in locking down the user account with access. The best that can be done is to give read-only permissions to only that user account. Possibly create a special user, specifically and only for that service.

[ArtOfWarfare]

This looks like the best choice to me, but I feel this answer is way too incomplete to accept it, given it's lacking any actual examples.

[ArtOfWarfare]

There are some examples for using these functions in Python over here: stackoverflow.com/questions/463832/using-dpapi-with-python

[Stevoisiak]

How should the username be stored? Does keyring support retrieving both the username and password?

[Stevoisiak]

@DustinWyatt Clever use of get_password for the username. Although, I think you should begin the answer with the original simplified example of keyring.set_password() and keyring.get_password()

[Umar.H]

Thought I recognized your name, you were on the beginners & experts panel on TalkPython, as a beginner, your message really resonated with me, thanks!

[Ciasto piekarz]

keyring is not part of python standard library

[Dustin Wyatt]

@Ciastopiekarz Did something about the answer lead you to believe that it was part of the standard library?

[a_sid]

Hello. If your project already uses this technique, it's easy to transition to environment variables. I know how to set environment variables in Windows 10 manually but can access them from my python code using os.getenv(). How should we do this if the code is being shared? If the code is downloaded by another developer, how should he/she make sure the environment variables are already set for him?

[Don Kirkby]

I try to pass a reasonable default value to os.getenv(), @a_sid, so the code will at least run for a user who hasn't set the environment variables. If there is no good default value, raise a clear error when you get None. Other than that, put clear comments in the settings file. If I misunderstood something, I suggest you ask a separate question.

[Kebman]

Does keyring safely scrub the password from logs and memory afterwords?

[Dustin Wyatt]

@Kebman Keyring reads/writes the credentials from your system wallet. That's it. I'm not sure how useful it would be to have such scrubbing functionality anyway. As the answer notes anything running in your system account can already read your wallet. If something can read your memory or log files it's probably capable of running under your account.

[alper]

Is it safe to save and use pickle object? I was saving my authenticated object as pickle and re-use it

[gruentee]

I adapted the above script for my purposes: gist.github.com/gruentee/6849a331bf74a97eda826743c274b3ba