Virtual Environments¶
This document contains information about how to use Python virtual
environments with mod_wsgi. You can use a Python virtual environment
created using virtualenv and virtualenvwrapper, or if using Python 3,
the pyvenv or python -m venv commands.
The purpose of a Python virtual environments is to allow one to create multiple distinct Python environments for the same version of Python, but with different sets of Python modules and packages installed. It is recommended that you always use Python virtual environments and not install additional Python packages direct into your Python installation.
A Python virtual environment is also required where it is necessary to run multiple WSGI applications which have conflicting requirements as to what version of a Python module or package needs to be installed. They can also be used when distinct mod_wsgi daemon process groups are used to host WSGI applications for different users and each user needs to be able to separately install their own Python modules and packages.
How you configure mod_wsgi or setup your WSGI application script file for a Python virtual environment will depend on your specific requirements. The more common scenarios are explained below.
Location of the Virtual Environment¶
Whichever method you use to create a Python virtual environment, before you use it with mod_wsgi, you should validate what the location of the Python virtual environment is. If using virtualenvwrapper this may be a non obvious directory hidden away under your home directory.
The way to determine the location of the Python virtual environment is to activate the Python virtual environment from an interactive shell so it is being used, and then run the command:
python -c 'import sys; print(sys.prefix)'
This will output the directory path you will use when setting up mod_wsgi
to use the Python virtual environment. For the purposes of the examples
below, it is assumed the location of any Python virtual environments are
under the /usr/local/venvs directory. A specific Python virtual
environment may thus return for sys.prefix:
/usr/local/venvs/example
Note that this should be the root directory of the Python virtual
environment, which in turn contains the bin and lib directories for
the Python virtual environment. It is a common mistake when setting up a
Python virtual environment with mod_wsgi to use the full path to the
python executable instead of the root directory. That will not work, so
do not use the path for the python executable as the location of the
Python virtual environment, it has to be the root directory.
Do be aware that the user that Apache runs your code as will need to be able to access the Python virtual environment. On some Linux distributions, the home directory of a user account is not accessible to other users. Rather than change the permissions on your home directory, it might be better to consider locating your WSGI application code and any Python virtual environment outside of your home directory.
Virtual Environment and Python Version¶
When using a Python virtual environment with mod_wsgi, it is very important that it has been created using the same Python installation that mod_wsgi was originally compiled for. It is not possible to use a Python virtual environment to force mod_wsgi to use a different Python version, or even a different Python installation.
You cannot for example force mod_wsgi to use a Python virtual environment
created using Python 3.5 when mod_wsgi was originally compiled for Python
2.7. This is because the Python library for the Python installation it was
originally compiled against is linked directly into the mod_wsgi module.
In other words, Python is embedded within mod_wsgi. When mod_wsgi is used
it does not run the command line python program to run the interpreter
and thus why you can’t force it to use a different Python installation.
The problem in trying to force mod_wsgi to use a different Python installation than what it was compiled for, even where it is the same Python version, is that the Python installation may itself not have been compiled with the same options. This is especially a problem when it comes to issues around how Python stores Unicode characters in memory.
The end result is that if you want to use a different Python installation or version than what mod_wsgi was originally compiled for, you would need to re-install mod_wsgi such that it is compiled for the Python installation or version you do want to use. Do not try and use a Python virtual environment from one Python installation or version with mod_wsgi, when mod_wsgi was compiled for a different one.
Daemon Mode (Single Application)¶
The preferred way of setting up mod_wsgi is to run each WSGI application in its own daemon process group. This is called daemon mode. A typical configuration for running a WSGI application in daemon mode would be:
WSGIDaemonProcess myapp
WSGIProcessGroup myapp
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias / /some/path/project/myapp.wsgi
<Directory /some/path/project>
Require all granted
</Directory>
The WSGIDaemonProcess directive defines the daemon process group. The
WSGIProcessGroup directive indicates that the WSGI application should be
run within the defined daemon process group.
As only the single application is being run within the daemon process
group, the WSGIApplicationGroup directive is also being used. When this
is used with the %{GLOBAL} value, it forces the WSGI application to run
in the main Python interpreter context of each process. This is preferred
in this scenario as some third party packages for Python which include C
extensions will not run in the Python sub interpreter contexts which
mod_wsgi would use by default. By using the main Python interpreter context
you eliminate the possibility of such third party packages for Python
causing problems.
To modify the configuration for this scenario to use a Python virtual
environment, all you need to do is add the python-home option to the
WSGIDaemonProcess directive resulting in:
WSGIDaemonProcess myapp python-home=/usr/local/venvs/myapp
All the additonal Python packages and modules would then be installed into that Python virtual environment.
Daemon Mode (Multiple Applications)¶
If instead of running each WSGI application in a separate daemon process group as is the recommended practice, you are running multiple WSGI applications in one daemon process group, a different approach to using Python virtual environments is required.
For this scenario there are various ways the configuration could be set up. If mounting each WSGI application explicitly you might be using:
WSGIDaemonProcess myapps
WSGIProcessGroup myapps
WSGIScriptAlias /myapp3 /some/path/project/myapp3.wsgi
WSGIScriptAlias /myapp2 /some/path/project/myapp2.wsgi
WSGIScriptAlias / /some/path/project/myapp1.wsgi
<Directory /some/path/project>
Require all granted
</Directory>
If instead the directory containing the WSGI application script files is being mounted, you might be using:
WSGIDaemonProcess myapps
WSGIProcessGroup myapps
WSGIScriptAlias / /some/path/project/
<Directory /some/path/project>
Require all granted
</Directory>
The use of the WSGIDaemonProcess and WSGIProcessGroup is the same as
before, however the WSGIApplicationGroup directive is not being used.
When the WSGIApplicationGroup directive isn’t being used to override
which Python interpreter context is being used, each WSGI application will
be run in its own Python sub interpreter context of the processes. This is
necessary as often WSGI application frameworks (Django being a prime
example), do not support running more than one instance of a WSGI
application using the framework, in the same Python interpreter context at
the same time.
In this scenario of running multiple WSGI applications in the same daemon process group, more than one change is possibly required. The changes required depend on whether or not all WSGI applications should share the same Python virtual environment.
If all of the WSGI applications should share the same Python virtual
environment, then the same change as was performed above for the single
application case would be made. That is, add the python-home option
to the WSGIDaemonProcess directive:
WSGIDaemonProcess myapp python-home=/usr/local/venvs/myapps
All the additonal Python packages and modules that any of the WSGI applications required would then be installed into that Python virtual environment. Because it is a shared environment, they must all use the same version of any specific Python package or module.
If instead of all WSGI applications using the same Python virtual environment each needed their own, then a change will instead need to be made in each of the WSGI script files for the applications.
How this is done will depend on how the Python virtual environment is created.
If the Python virtual environment is created using virtualenv or virtualenvwrapper, the WSGI script for each application should be modified to include code of the following form:
python_home = '/usr/local/envs/myapp1'
activate_this = python_home + '/bin/activate_this.py'
execfile(activate_this, dict(__file__=activate_this))
Because each WSGI application is to use a separate Python virtual
environment, the value of the python_home variable would be set
differently for each WSGI script file, with it referring to the root
directory of the respective Python virtual environments.
This code should be placed in the WSGI script file before any other module
imports in the WSGI script file, with the exception of from __future__
imports used to enable Python feature flags.
Important to note is that when the Python virtual environment is activated
from within the WSGI script, what happens is a bit different to when the
python-home option to WSGIDaemonProcess is used.
When activating the Python virtual environment from within the WSGI script
file, only the site-packages directory from the Python virtual
environment is being used. This directory will be added to the Python
module search path, along with any additional directories related to the
site-packages directory registered using .pth files present in the
site-packages directory. This will be placed at the start of the
existing sys.path.
The consequence of this is that the Python virtual environment isn’t completely overriding the original Python installation the Python virtual environment was created from. This means that if the main Python installation had additional Python packages installed they will also potentially be visible to the WSGI application.
That this occurs could cause confusion as you might for example think you
had all the packages you require listed in your requirements.txt file
for pip, but didn’t and so a package may not have been installed. If
that package was installed in the main Python installation, it would be
picked up from there, but it might be the wrong version and have
dependencies on versions of other packages for which you have different
versions installed in your Python virtual environment and which are found
instead of those in the main Python installation.
To avoid such problems, when activating the Python virtual environment
from within the WSGI script file, it is necessary to still set the
python-home option of the WSGIDaemonProcess directive, but set it to
an empty Python virtual environment which has had no additional packages
installed:
WSGIDaemonProcess myapp python-home=/usr/local/venvs/empty
By doing this, the main Python installation will not be consulted and instead it will fallback to the empty Python virtual environment. This Python virtual environment should remain empty and you should not install additional Python packages or modules into it, or you will cause the same sort of conflicts that can arise with the main Python installation when it was being used.
When needing to activate the Python virtual environment from within the
WSGI script file as described, it is preferred that you be using the either
virtualenv or virtualenvwrapper to create the Python virtual
environment. This is because they both provide the activate_this.py
script file which does all the work of setting up sys.path. When you
use either pyvenv or python -m venv with Python 3, no such
activation script is provided.
So use virtualenv or virtualenvwrapper if you can. If you cannot for
some reason and are stuck with pyvenv or python -m venv, you can
instead use the following code in the WSGI script file:
python_home = '/usr/local/envs/myapp1'
import sys
import site
# Calculate path to site-packages directory.
python_version = '.'.join(map(str, sys.version_info[:2]))
site_packages = python_home + '/lib/python%s/site-packages' % python_version
# Add the site-packages directory.
site.addsitedir(site_packages)
As before this code should be placed in the WSGI script file before any
other module imports in the WSGI script file, with the exception of from
__future__ imports used to enable Python feature flags.
When using this method, do be aware that the additions to the Python module
search path are made at the end of sys.path. For that reason, you must
set the python-home option to WSGIDaemonProcess to the location of
an empty Python virtual environment. If you do not do this, any additional
Python package installed in the main Python installation will hide those in
the Python virtual environment for the application.
There is extra code you could add which would reorder sys.path to make
it work in an equivalent way to the activate_this.py script provided
when you use virtualenv or virtualenvwrapper but it is messy and more
trouble than it is worth:
python_home = '/usr/local/envs/myapp1'
import sys
import site
# Calculate path to site-packages directory.
python_version = '.'.join(map(str, sys.version_info[:2]))
site_packages = python_home + '/lib/python%s/site-packages' % python_version
site.addsitedir(site_packages)
# Remember original sys.path.
prev_sys_path = list(sys.path)
# Add the site-packages directory.
site.addsitedir(site_packages)
# Reorder sys.path so new directories at the front.
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
It is better to avoid needing to manually activate the Python virtual
environment from inside of a WSGI script by using a separate daemon process
group per WSGI application. At the minimum, at least avoid pyvenv and
python -m venv.
Embedded Mode (Single Application)¶
The situation for running a single WSGI application in embedded mode is not
much different to running a single WSGI application in daemon mode. In the
case of embedded mode, there is though no WSGIDaemonProcess directive.
The typical configuration when running a single WSGI application in embedded module might be:
WSGIScriptAlias / /some/path/project/myapp.wsgi
WSGIApplicationGroup %{GLOBAL}
<Directory /some/path/project>
Require all granted
</Directory>
The WSGIDaemonProcess and WSGIProcessGroup directives are gone, but
the WSGIApplicationGroup directive is still used to force the WSGI
application to run in the main Python interpreter context of each of the
Apache worker processes. This is to avoid those issues with some third
party packages for Python with C extensions as mentioned before.
In this scenario, to set the location of the Python virtual environment
to be used, the WSGIPythonHome directive is used:
WSGIPythonHome /usr/local/envs/myapp
Note that if the WSGI application is being setup within the context of an
Apache VirtualHost, the WSGIPythonHome cannot be placed inside of
the VirtualHost. Instead it must be placed outside of all
VirtualHost definitions. This is because it applies to the whole Apache
instance and not just the single VirtualHost.
Embedded Mode (Multiple Applications)¶
Running multiple applications in embedded mode is also similar to when running multiple WSGI applications in one daemon process group. You still need to ensure each WSGI application runs in its own Python sub interpreter context to avoid potential issues with Python web frameworks that don’t allow more than one WSGI application to be using it at the same time in a Python interpreter context.
If mounting each WSGI application explicitly you might be using:
WSGIScriptAlias /myapp3 /some/path/project/myapp3.wsgi
WSGIScriptAlias /myapp2 /some/path/project/myapp2.wsgi
WSGIScriptAlias / /some/path/project/myapp1.wsgi
<Directory /some/path/project>
Require all granted
</Directory>
If instead the directory containing the WSGI application script files is being mounted, you might be using:
WSGIScriptAlias / /some/path/project/
<Directory /some/path/project>
Require all granted
</Directory>
In this scenario, to set the location of the Python virtual environment
to be used by all WSGI application, the WSGIPythonHome directive is used:
WSGIPythonHome /usr/local/envs/myapps
If the WSGI application is being setup within the context of an Apache
VirtualHost, the WSGIPythonHome cannot be placed inside of the
VirtualHost. Instead it must be placed outside of all VirtualHost
definitions. This is because it applies to the whole Apache instance and
not just the single VirtualHost.
If each WSGI application needs its own Python virtual environment, then
activation of the Python virtual environment needs to be performed in the
WSGI script itself as explained previously for the case of daemon mode
being used. The WSGIPythonHome directive should be used to refer to an
empty Python virtual environment if needed to ensure that any additional
Python packages in the main Python installation don’t interfere with what
packages are installed in the Python virtual environment for each WSGI
application.
Adding Additional Module Directories¶
The python-home option to WSGIDaemonProcess and the
WSGIPythonHome directive are the preferred way of specifying the
location of the Python virtual environment to be used. If necessary,
activation of the Python virtual environment can also be performed from the
WSGI script file itself.
If you need to add additional directories to search for Python packages or modules this can also be done. You may want to do this where you need to specify where the actual WSGI application is located, where a WSGI script file needs to import application specific modules.
If you are using daemon mode and want to add additional directories to the
Python module search path, you can use the python-path option to
WSGIDaemonProcess:
WSGIDaemonProcess myapp python-path=/some/path/project
This option would be in addition to the python-home option used to
specify where the Python virtual environment is located.
If you are using embedded mode, you can use the WSGIPythonPath
directive:
WSGIPythonPath /some/path/project
This directive is in addition to the WSGIPythonHome directive used to
specify where the Python virtual environment is located.
In either case, if you need to specify more than one directory, they can be separated using a ‘:’ character.
If you are having to activate the Python virtual enviromment from within a
WSGI script and need to add additional directories to the Python module
search path, you should modify sys.path directly from the WSGI script
file.
Note that prior practice was that these ways of setting the Python module
search path were used to specify the location of the Python virtual
environment. Specifically, they were used to add the site-packages
directory of the Python virtual environment. You should not do that.
The better way to specify the location of the Python virtual environment is
using the python-home option of the WSGIDaemonProcess directive for
daemon mode, or the WSGIPythonHome directive for embedded mode. These
ways of specifying the Python virtual environment have been available since
mod_wsgi 3.0 and Linux distributions have not shipped such an old version
of mod_wsgi for quite some time. If you are using the older way, please
update your configurations.