django setting environment variables in unittest tests

Question

I want to be able to set environment variables in my Django app for tests to be able to run. For instance, my views rely on several API keys.

There are ways to override settings during testing, but I don't want them defined in settings.py as that is a security issue.

I've tried in my setup function to set these environment variables, but that doesn't work to give the Django application the values.

class MyTests(TestCase):
    def setUp(self):
        os.environ['TEST'] = '123'  # doesn't propogate to app

When I test locally, I simply have an .env file I run with

foreman start -e .env web

which supplies os.environ with values. But in Django's unittest.TestCase it does not have a way (that I know) to set that.

How can I get around this?

Solution 1

As @schillingt noted in the comments, EnvironmentVarGuard was the correct way.

from test.test_support import EnvironmentVarGuard # Python(2.7 < 3)
from test.support import EnvironmentVarGuard # Python >=3
from django.test import TestCase

class MyTestCase(TestCase):
    def setUp(self):
        self.env = EnvironmentVarGuard()
        self.env.set('VAR', 'value')

    def test_something(self):
        with self.env:
            # ... perform tests here ... #
            pass

This correctly sets environment variables for the duration of the context object with statement.

Solution 2

The test.support.EnvironmentVarGuard is an internal API that might be changed from version to version with breaking (backward incompatible) changes. In fact, the entire test package is internal use only. It was explicitly stated on the test package documentation page that it's for internal testing of core libraries and NOT a public API. (see links below)

You should use patch.dict() in python's standard lib unittest.mock. It can be used as a context manager, decorator or class decorator. See example code below copied from the official Python documentation.

import os
from unittest.mock import patch
with patch.dict('os.environ', {'newkey': 'newvalue'}):
    print(os.environ['newkey'])  # should print out 'newvalue'
    assert 'newkey' in os.environ  # should be True
assert 'newkey' not in os.environ  # should be True

Update: for those who doesn't read the documentation thoroughly and might have missed the note, read more test package notes at

https://docs.python.org/2/library/test.html or

https://docs.python.org/3/library/test.html

Solution 3

If you are loading your environment variables in Django's settings.py file like this:

import os
ENV_NAME = os.environ.get('ENV_NAME', 'default')

You could use this:

from django.test import TestCase, override_settings

@override_settings(ENV_NAME="super_setting")
def test_...(self):

Solution 4

Using EnvironmentVarGuard is not a good solution as it fails in some environments and works in others. see example below.

A better solution is what was suggested by erewok that requires making use of the unittest.mock in python3.

Assuming using unittest

from unittest.mock import patch
class TestCase(unittest.TestCase):

    def setUp(self):
        self.env = patch.dict('os.environ', {'hello':'world'})

    def test_scenario_1(self):
        with self.env:
            self.assertEqual(os.environ.get('hello'), 'world')

```

Solution 5

I use py.test as my test runner, and it allows you to create a pytest.ini file in which you can specify a particular settings file to use while running tests.

See documentation on this here:

http://pytest-django.readthedocs.org/en/latest/configuring_django.html#pytest-ini-settings

I recommend py.test in general as a test runner, because it supports different types of test classes and even simple functions, and it's pretty easy to set up fixtures or other code that runs before and after tests.

View more solutions

Comments

[lollercoaster]

This is how to change settings.py files, not environment variables. My API calls, for example to AWS, etc are made with constructors that look for environment variables, not django settings.

[erewok]

You said, "There are ways to override settings during testing, but I don't want them defined in settings.py as that is a security issue." I fail to see how defining variables in a settings file that is used only by your test runner is a security issue.

[lollercoaster]

I should have been clearer. There are two issues. 1) security, not committing credentials, 2) my workflow with boto and other APIs usings environment variables so I'd like to be able to use those. I was just drawing the analogy with settings.py

[erewok]

Your tests are going to use actual API keys and credentials? They're actually going to communicate with third-party APIs?

[lollercoaster]

Yes, absolutely. It's rather hard to test whether an S3 bucket is writeable, etc without using API credentials.

[erewok]

I have never considered testing third-party APIs.

[lollercoaster]

No, you aren't testing 3rd party APIs. You are testing you configuration of services on those third party APIs and whether the interface to those APIs that you have set up works as expected with all the corner cases. This is more than perfectly acceptable practice, it would be insane not to.

[erewok]

Aside from configuration, we typically test application code that communicates with APIs using mocked interfaces. I don't think I'd want my ci-server making calls to APIs. I don't want it accidentally sending email or talking to the world in other ways either.

[Nate]

Throws an import error. Additionally, the documentation for EnvironmentVarGuard states: "Warning The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python."

[medmunds]

python 3 moved it to from test.support import EnvironmentVarGuard. However, if you'd rather not depend on internal-use-only code, you could copy the python 2.7 implementation of EnvironmentVarGuard into your own code -- it's pretty straightforward.

[Devy]

@seb in the link you provided, it's anchored to class test.test_support.EnvironmentVarGuard, in other words, it's part of the test package, which is a regression test package for Python. And then scroll your page ALL THE WAY UP, read the note in the second line right after the title: Note The test package is meant for internal use by Python only. It is documented for the benefit of the core developers of Python. Any use of this package outside of Python’s standard library is discouraged as code mentioned here can change or be removed without notice between releases of Python.

[s g]

This is a very good answer as it's a common practice to pull os.environ variables into settings.py, and then use those variables (a-la from django.conf import settings) throughout the code. If your code uses, for example, my_obj = MyClass(arg=settings.MY_ENV_VAR), then you'll need to use the @override_settings() wrapper to correctly set your MY_ENV_VAR in unit tests. If you set your os.environ vars in any other way, my_obj will not get instantiated properly

[jamesc]

As per the docs docs.python.org/3/library/… : "If you use this technique you must ensure that the patching is “undone” by calling stop." This example does not "unpatch" the environment so could break other tests.

[Hammad]

great answer to use django's own override_settings