Installing extensions in parallel (experimental!)¶
Since EasyBuild v4.5.0 (initial) support is available for installing extensions in parallel, that is to run the installation command for multiple extensions that are ready to install at the same time in the background, to exploit the availability of multiple cores.
Note: this is an experimental feature. Some of the mentioned functionality may be subject to change or be prone to errors.
Configuration¶
To let EasyBuild install extensions in parallel, the
parallel-extensions-install configuration option must be enabled.
In addition, since the support for installing extensions in parallel is
an experimental feature, the experimental configuration option must be
also be enabled (see also Experimental features).
How many extension installations should be started (at maximum) at the
same time is controlled by the parallel configuration option.
Requirements¶
For EasyBuild to be able to install a list of extensions in parallel, two requirements must be met:
- The easyblock(s) that are used for installing the extensions must
have support for determining which installed extensions are required
to fulfil the dependencies for a particular extension. This is done
via the
required_depsproperty method. - EasyBuild must be able to start a single shell command
(asynchronously, so it runs in the background) to perform the
installation of the extension via the
run_asyncmethod, and check whether the command has completed via theasync_cmd_checkmethod.
For an example of how required_deps, run_async, and
async_cmd_check can be implemented, see the
RPackage
easyblock.
Caveats and Known issues¶
(last update: EasyBuild v4.5.0)
There are a couple of caveats and known issues with installing extensions in parallel, which we hope to resolve in future EasyBuild releases.
This is also why the support for installing extensions in parallel is currently marked as an experimental feature.
Only works for R extensions¶
The support for installing extensions in parallel currently only works
for R extensions, since only the RPackage easyblock (which is used to
install R packages as extensions) implements the necessary methods (see
Requirements).
Supporting other types of extensions (Python, Perl, etc.) requires that
the corresponding easyblocks that are used to install those extensions
(like PythonPackage, PerlModule) are enhanced to support determining
required dependencies and starting the installation command
asynchronously.
List of extensions must be self-contained¶
The EasyBuild framework currently enforces that all required
dependencies for a particular extension are included in the list of
extensions that is being installed (specified via the exts_list
easyconfig parameter). Extensions provided by dependencies are not taken
into account (yet).
As a result, trying to install an
R-bundle-Bioconductor
easyconfig with parallel-extensions-install enabled will result in an
error reporting missing required dependencies.
Skipping of installed extensions is still done sequentially¶
Checking which extensions are already installed for which the
installation can be skipped when --skip (see
Installing additional extensions using -k/-skip) is being used
is still done sequentially.
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.
Sanity check for extensions is still run sequentially¶
The sanity checks that are done for each extension are still done through a sequential loop over the list of extensions.
Additional work is needed to also run these checks in parallel across the cores that are available to EasyBuild.