Unit, Integration, and Functional Testing

Unit testing is, not surprisingly, the act of testing a "unit" in your application. In this context, a "unit" is often a function or a method of a class instance. The unit is also referred to as a "unit under test".

The goal of a single unit test is to test only some permutation of the "unit under test". If you write a unit test that aims to verify the result of a particular codepath through a Python function, you need only be concerned about testing the code that lives in the function body itself. If the function accepts a parameter that represents a complex application "domain object" (such as a resource, a database connection, or an SMTP server), the argument provided to this function during a unit test need not be and likely should not be a "real" implementation object. For example, although a particular function implementation may accept an argument that represents an SMTP server object, and the function may call a method of this object when the system is operating normally that would result in an email being sent, a unit test of this codepath of the function does not need to test that an email is actually sent. It just needs to make sure that the function calls the method of the object provided as an argument that would send an email if the argument happened to be the "real" implementation of an SMTP server object.

An integration test, on the other hand, is a different form of testing in which the interaction between two or more "units" is explicitly tested. Integration tests verify that the components of your application work together. You might make sure that an email was actually sent in an integration test.

A functional test is a form of integration test in which the application is run "literally". You would have to make sure that an email was actually sent in a functional test, because it tests your code end to end.

It is often considered best practice to write each type of tests for any given codebase. Unit testing often provides the opportunity to obtain better "coverage": it's usually possible to supply a unit under test with arguments and/or an environment which causes all of its potential codepaths to be executed. This is usually not as easy to do with a set of integration or functional tests, but integration and functional testing provides a measure of assurance that your "units" work together, as they will be expected to when your application is run in production.

The suggested mechanism for unit and integration testing of a Pyramid application is the Python unittest module. Although this module is named unittest, it is actually capable of driving both unit and integration tests. A good unittest tutorial is available within Dive Into Python 3 by Mark Pilgrim.

Pyramid provides a number of facilities that make unit, integration, and functional tests easier to write. The facilities become particularly useful when your code calls into Pyramid-related framework functions.

Test Set Up and Tear Down

Pyramid uses a "global" (actually thread local) data structure to hold two items: the current request and the current application registry. These data structures are available via the pyramid.threadlocal.get_current_request() and pyramid.threadlocal.get_current_registry() functions, respectively. See Thread Locals for information about these functions and the data structures they return.

If your code uses these get_current_* functions or calls Pyramid code which uses get_current_* functions, you will need to call pyramid.testing.setUp() in your test setup and you will need to call pyramid.testing.tearDown() in your test teardown. setUp() pushes a registry onto the thread local stack, which makes the get_current_* functions work. It returns a Configurator object which can be used to perform extra configuration required by the code under test. tearDown() pops the thread local stack.

Normally when a Configurator is used directly with the main block of a Pyramid application, it defers performing any "real work" until its .commit method is called (often implicitly by the pyramid.config.Configurator.make_wsgi_app() method). The Configurator returned by setUp() is an autocommitting Configurator, however, which performs all actions implied by methods called on it immediately. This is more convenient for unit testing purposes than needing to call pyramid.config.Configurator.commit() in each test after adding extra configuration statements.

The use of the setUp() and tearDown() functions allows you to supply each unit test method in a test case with an environment that has an isolated registry and an isolated request for the duration of a single test. Here's an example of using this feature:

import unittest
from pyramid import testing

class MyTest(unittest.TestCase):
    def setUp(self):
        self.config = testing.setUp()

    def tearDown(self):
        testing.tearDown()

The above will make sure that get_current_registry() called within a test case method of MyTest will return the application registry associated with the config Configurator instance. Each test case method attached to MyTest will use an isolated registry.

The setUp() and tearDown() functions accept various arguments that influence the environment of the test. See the pyramid.testing API for information about the extra arguments supported by these functions.

If you also want to make get_current_request() return something other than None during the course of a single test, you can pass a request object into the pyramid.testing.setUp() within the setUp method of your test:

import unittest
from pyramid import testing

class MyTest(unittest.TestCase):
    def setUp(self):
        request = testing.DummyRequest()
        self.config = testing.setUp(request=request)

    def tearDown(self):
        testing.tearDown()

If you pass a request object into pyramid.testing.setUp() within your test case's setUp, any test method attached to the MyTest test case that directly or indirectly calls get_current_request() will receive the request object. Otherwise, during testing, get_current_request() will return None. We use a "dummy" request implementation supplied by pyramid.testing.DummyRequest because it's easier to construct than a "real" Pyramid request object.

Test setup using a context manager

An alternative style of setting up a test configuration is to use the with statement and pyramid.testing.testConfig() to create a context manager. The context manager will call pyramid.testing.setUp() before the code under test and pyramid.testing.tearDown() afterwards.

This style is useful for small self-contained tests. For example:

import unittest

class MyTest(unittest.TestCase):

    def test_my_function(self):
        from pyramid import testing
        with testing.testConfig() as config:
            config.add_route('bar', '/bar/{id}')
            my_function_which_needs_route_bar()

What?

Thread local data structures are always a bit confusing, especially when they're used by frameworks. Sorry. So here's a rule of thumb: if you don't know whether you're calling code that uses the get_current_registry() or get_current_request() functions, or you don't care about any of this, but you still want to write test code, just always call pyramid.testing.setUp() in your test's setUp method and pyramid.testing.tearDown() in your tests' tearDown method. This won't really hurt anything if the application you're testing does not call any get_current* function.

Using the Configurator and pyramid.testing APIs in Unit Tests

The Configurator API and the pyramid.testing module provide a number of functions which can be used during unit testing. These functions make configuration declaration calls to the current application registry, but typically register a "stub" or "dummy" feature in place of the "real" feature that the code would call if it was being run normally.

For example, let's imagine you want to unit test a Pyramid view function.

from pyramid.httpexceptions import HTTPForbidden

def view_fn(request):
    if request.has_permission('edit'):
        raise HTTPForbidden
    return {'greeting':'hello'}

Note

This code implies that you have defined a renderer imperatively in a relevant pyramid.config.Configurator instance, otherwise it would fail when run normally.

Without doing anything special during a unit test, the call to has_permission() in this view function will always return a True value. When a Pyramid application starts normally, it will populate an application registry using configuration declaration calls made against a Configurator. But if this application registry is not created and populated (e.g., by initializing the configurator with an authorization policy), like when you invoke application code via a unit test, Pyramid API functions will tend to either fail or return default results. So how do you test the branch of the code in this view function that raises HTTPForbidden?

The testing API provided by Pyramid allows you to simulate various application registry registrations for use under a unit testing framework without needing to invoke the actual application configuration implied by its main function. For example, if you wanted to test the above view_fn (assuming it lived in the package named my.package), you could write a unittest.TestCase that used the testing API.

import unittest
from pyramid import testing

class MyTest(unittest.TestCase):
    def setUp(self):
        self.config = testing.setUp()

    def tearDown(self):
        testing.tearDown()

    def test_view_fn_forbidden(self):
        from pyramid.httpexceptions import HTTPForbidden
        from my.package import view_fn
        self.config.testing_securitypolicy(userid='hank',
                                           permissive=False)
        request = testing.DummyRequest()
        request.context = testing.DummyResource()
        self.assertRaises(HTTPForbidden, view_fn, request)

    def test_view_fn_allowed(self):
        from my.package import view_fn
        self.config.testing_securitypolicy(userid='hank',
                                           permissive=True)
        request = testing.DummyRequest()
        request.context = testing.DummyResource()
        response = view_fn(request)
        self.assertEqual(response, {'greeting':'hello'})

In the above example, we create a MyTest test case that inherits from unittest.TestCase. If it's in our Pyramid application, it will be found when pytest is run. It has two test methods.

The first test method, test_view_fn_forbidden tests the view_fn when the security policy forbids the current user the edit permission. Its third line registers a "dummy" "non-permissive" authorization policy using the testing_securitypolicy() method, which is a special helper method for unit testing.

We then create a pyramid.testing.DummyRequest object which simulates a WebOb request object API. A pyramid.testing.DummyRequest is a request object that requires less setup than a "real" Pyramid request. We call the function being tested with the manufactured request. When the function is called, pyramid.request.Request.has_permission() will call the "dummy" security policy we've registered through testing_securitypolicy(), which denies access. We check that the view function raises a HTTPForbidden error.

The second test method, named test_view_fn_allowed, tests the alternate case, where the security policy allows access. Notice that we pass different values to testing_securitypolicy() to obtain this result. We assert at the end of this that the view function returns a value.

Note that the test calls the pyramid.testing.setUp() function in its setUp method and the pyramid.testing.tearDown() function in its tearDown method. We assign the result of pyramid.testing.setUp() as config on the unittest class. This is a Configurator object and all methods of the configurator can be called as necessary within tests. If you use any of the Configurator APIs during testing, be sure to use this pattern in your test case's setUp and tearDown; these methods make sure you're using a "fresh" application registry per test run.

See the pyramid.testing chapter for the entire Pyramid-specific testing API. This chapter describes APIs for registering a security policy, registering resources at paths, registering event listeners, registering views and view permissions, and classes representing "dummy" implementations of a request and a resource.

See also

See also the various methods of the Configurator documented in pyramid.config that begin with the testing_ prefix.

Creating Integration Tests

In Pyramid, a unit test typically relies on "mock" or "dummy" implementations to give the code under test enough context to run.

"Integration testing" implies another sort of testing. In the context of a Pyramid integration test, the test logic exercises the functionality of the code under test and its integration with the rest of the Pyramid framework.

Creating an integration test for a Pyramid application usually means invoking the application's includeme function via pyramid.config.Configurator.include() within the test's setup code. This causes the entire Pyramid environment to be set up, simulating what happens when your application is run "for real". This is a heavy-hammer way of making sure that your tests have enough context to run properly, and tests your code's integration with the rest of Pyramid.

See also

See also Including Configuration from External Sources

Writing unit tests that use the Configurator API to set up the right "mock" registrations is often preferred to creating integration tests. Unit tests will run faster (because they do less for each test) and are usually easier to reason about.

Creating Functional Tests

Functional tests test your literal application.

In Pyramid, functional tests are typically written using the WebTest package, which provides APIs for invoking HTTP(S) requests to your application. We also like pytest and pytest-cov to provide simple testing and coverage reports.

Regardless of which testing package you use, be sure to add a tests_require dependency on that package to your application's setup.py file. Using the project myproject generated by the starter cookiecutter as described in Creating a Pyramid Project, we would insert the following code immediately following the requires block in the file myproject/setup.py.

requires = [
    'plaster_pastedeploy',
    'pyramid',
    'pyramid_jinja2',
    'pyramid_debugtoolbar',
    'waitress',
]

tests_require = [
    'WebTest',
    'pytest',
    'pytest-cov',
]

Remember to change the dependency.

    zip_safe=False,
    extras_require={
        'testing': tests_require,
    },
    install_requires=requires,

As always, whenever you change your dependencies, make sure to run the correct pip install -e command.

$VENV/bin/pip install -e ".[testing]"

In your myproject project, your package is named myproject which contains a views package containing a default.py module, which in turn contains a view function my_view that returns an HTML body when the root URL is invoked:

from pyramid.view import view_config


@view_config(route_name='home', renderer='myproject:templates/mytemplate.jinja2')
def my_view(request):
    return {'project': 'myproject'}

Test configuration and fixtures are defined in conftest.py. In the following example, we define a test fixture.

@pytest.fixture
def testapp(app):
    testapp = webtest.TestApp(app, extra_environ={
        'HTTP_HOST': 'example.com',
    })

    return testapp

This fixture is used in the following example functional tests, to demonstrate invoking the above view:

def test_root(testapp):
    res = testapp.get('/', status=200)
    assert b'Pyramid' in res.body

def test_notfound(testapp):
    res = testapp.get('/badurl', status=404)
    assert res.status_code == 404

When these tests are run, each test method creates a "real" WSGI application using the main function in your myproject.__init__ module, using WebTest to wrap that WSGI application. It assigns the result to res.

In the test named test_root, the TestApp's GET method is used to invoke the root URL. An assertion is made that the returned HTML contains the text Pyramid.

In the test named test_notfound, the TestApp's GET method is used to invoke a bad URL /badurl. An assertion is made that the returned status code in the response is 404.

See the WebTest documentation for further information about the methods available to a webtest.app.TestApp instance.