How to use Conda as your Python environment

Read the Docs supports Conda as an environment management tool, along with Virtualenv. Conda support is useful for people who depend on C libraries, and need them installed when building their documentation.

This work was funded by Clinical Graphics – many thanks for their support of open source.

Activating conda

Conda support is available using a Configuration file tutorial, see conda.

Our Docker images use Miniconda, a minimal conda installer. After specifying your project requirements using a conda environment.yml file, Read the Docs will create the environment (using conda env create) and add the core dependencies needed to build the documentation.

Creating the environment.yml

There are several ways of exporting a conda environment:

  • conda env export will produce a complete list of all the packages installed in the environment with their exact versions. This is the best option to ensure reproducibility, but can create problems if done from a different operative system than the target machine, in our case Ubuntu Linux.

  • conda env export --from-history will only include packages that were explicitly requested in the environment, excluding the transitive dependencies. This is the best option to maximize cross-platform compatibility, however it may include packages that are not needed to build your docs.

  • And finally, you can also write it by hand. This allows you to pick exactly the packages needed to build your docs (which also results in faster builds) and overcomes some limitations in the conda exporting capabilities.

For example, using the second method for an existing environment:

$ conda activate rtd38
(rtd38) $ conda env export --from-history | tee environment.yml
name: rtd38
channels:
  - defaults
  - conda-forge
dependencies:
  - rasterio==1.2
  - python=3.8
  - pytorch-cpu=1.7
prefix: /home/docs/.conda/envs/rtd38

Read the Docs will override the name and prefix of the environment when creating it, so they can have any value, or not be present at all.

Tip

Bear in mind that rasterio==1.2 (double ==) will install version 1.2.0, whereas python=3.8 (single =) will fetch the latest 3.8.* version, which is 3.8.8 at the time of writing.

Warning

Pinning Sphinx and other Read the Docs core dependencies is not yet supported by default when using conda (see this GitHub issue for discussion). If your project needs it, request that we enable the CONDA_APPEND_CORE_REQUIREMENTS feature flag.

Effective use of channels

Conda packages are usually hosted on https://anaconda.org/, a registration-free artifact archive maintained by Anaconda Inc. Contrary to what happens with the Python Package Index, different users can potentially host the same package in the same repository, each of them using their own channel. Therefore, when installing a conda package, conda also needs to know which channels to use, and which ones take precedence.

If not specified, conda will use defaults, the channel maintained by Anaconda Inc. and subject to Anaconda Terms of Service. It contains well-tested versions of the most widely used packages. However, some packages are not available on the defaults channel, and even if they are, they might not be on their latest versions.

As an alternative, there are channels maintained by the community that have a broader selection of packages and more up-to-date versions of them, the most popular one being conda-forge.

To use the conda-forge channel when specifying your project dependencies, include it in the list of channels in environment.yml, and conda will rank them in order of appearance. To maximize compatibility, we recommend putting conda-forge above defaults:

name: rtd38
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  # Rest of the dependencies

Tip

If you want to opt out the defaults channel completely, replace it by nodefaults in the list of channels. See the relevant conda docs for more information.

Making builds faster with mamba

One important thing to note is that, when enabling the conda-forge channel, the conda dependency solver requires a large amount of RAM and long solve times. This is a known issue due to the sheer amount of packages available in conda-forge.

As an alternative, you can instruct Read the Docs to use mamba, a drop-in replacement for conda that is much faster and reduces the memory consumption of the dependency solving process.

To do that, add a .readthedocs.yaml configuration file with these contents:

.readthedocs.yaml
version: 2

build:
  os: "ubuntu-20.04"
  tools:
    python: "mambaforge-22.9"

conda:
  environment: environment.yml

You can read more about the build.tools.python configuration in our documentation.

Mixing conda and pip packages

There are valid reasons to use pip inside a conda environment: some dependency might not be available yet as a conda package in any channel, or you might want to avoid precompiled binaries entirely. In either case, it is possible to specify the subset of packages that will be installed with pip in the environment.yml file. For example:

name: rtd38
channels:
  - conda-forge
  - defaults
dependencies:
  - rasterio==1.2
  - python=3.8
  - pytorch-cpu=1.7
  - pip>=20.1  # pip is needed as dependency
  - pip:
    - black==20.8b1

The conda developers recommend in their best practices to install as many requirements as possible with conda, then use pip to minimize possible conflicts and interoperability issues.

Warning

Notice that conda env export --from-history does not include packages installed with pip, see this conda issue for discussion.

Compiling your project sources

If your project contains extension modules written in a compiled language (C, C++, FORTRAN) or server-side JavaScript, you might need special tools to build it from source that are not readily available on our Docker images, such as a suitable compiler, CMake, Node.js, and others.

Luckily, conda is a language-agnostic package manager, and many of these development tools are already packaged on conda-forge or more specialized channels.

For example, this conda environment contains the required dependencies to compile Slycot on Read the Docs:

name: slycot38
channels:
  - conda-forge
  - defaults
dependencies:
  - python=3.8
  - cmake
  - numpy
  - compilers

Troubleshooting

If you have problems on the environment creation phase, either because the build runs out of memory or time or because some conflicts are found, you can try some of these mitigations:

  • Reduce the number of channels in environment.yml, even leaving conda-forge only and opting out of the defaults adding nodefaults.

  • Constrain the package versions as much as possible to reduce the solution space.

  • Use mamba, an alternative package manager fully compatible with conda packages.

  • And, if all else fails, request more resources.

Custom Installs

If you are running a custom installation of Read the Docs, you will need the conda executable installed somewhere on your PATH. Because of the way conda works, we can’t safely install it as a normal dependency into the normal Python virtualenv.

Warning

Installing conda into a virtualenv will override the activate script, making it so you can’t properly activate that virtualenv anymore.