Shell scripts and Lmod

Some application provide shell scripts to initialize their use. The drawbacks of this approach is that applications would have to provide scripts for each shell and there was typically no way to unload the application. Also users of shells other than bash or (t)csh were also usually out of luck.

Lmod, like other environment module system are used by tools that are not shells such as cmake, R, perl, python, etc. So application shell scripts there weren’t helpful there either. Modules provide a way to support the other shells and non-shell applications.

Lmod has provided sh_to_modulefile to convert scripts to modulefiles. New in version 8.6+, Lmod provides support for source_sh () to source shells scripts inside a modulefile. This provides several features at a cost. It means that it can be used by all module applications and it can be unloaded. The cost is that Lmod is evaluating the shell script in a subshell extract the module commands every time that module is loaded. So it is typically better to convert it once with sh_to_modulefile.

Converting shell scripts to modulefiles

Lmod provides a script called sh_to_modulefile which will convert a script to a modulefile. An example is:

% $LMOD_DIR/sh_to_modulefile  ./foo.sh > foo_1.0.lua

or:

% $LMOD_DIR/sh_to_modulefile  --output foo_1.0.lua ./foo.sh

This program defaults to generating a lua based modulefile. It is possible to generate a TCL modulefile with:

% $LMOD_DIR/sh_to_modulefile  --to TCL --output foo_1.0 ./foo.sh

See:

% $LMOD_DIR/sh_to_modulefile  --help

for all the options.

The way it works is that remembers the initial environment and runs the script. The program then compares the initial environment and generate environment. The output is a report of the environment changes.

As of version 8.6, Lmod now tracks changes to shell aliases and shell functions and writes them to the generated modulefile.

Converting scripts once with this command is usually best. However, some scripts depend on dynamic environment variable that change between users such as the values of $HOME or $USER. In this case, the use of the source_sh () modulefile function can be helpful.

Using source_sh ()

The feature of sourcing shell scripts inside a modulefile was introduced in Tmod 4.6+. It has be shamelessly studied and re-implemented in Lmod 8.6+. In Lmod, this feature re-uses much of the code that implements sh_to_modulefile. This code does the following when performing a module load.

  1. Gets the current environment, shell aliases and shell functions

  2. Sources the shell script with arguments

  3. Compares the new environment to extract module commands

The resulting modules commands are stored in the user environment inside the module table which can be shown by running $ module –mt.

When unloading or showing, the module commands are extracted from the module table and used to unload the changes that the script caused. In other words, the shell script is only evaluated when loaded. not on unload.

Note: Occasionly, application scripts will provide a “deactivate” that a site might be temped to use like this:

if (mode() == "unload") then
   source_sh("bash", "app_script deactivate")
end

Do not do this! the function source_sh expects to find the module function in the module table in the environment. It is better to do this for load and unload:

source_sh("bash", "app_script activate")

and let Lmod unload the scripts via the generated module functions.

Sites can dynamically build the shell script and argument string. It is important however that this string be the same for both load and unload because this string is part of the access method to extract the commands from the module table.

Assumptions that Lmod makes about scripts used by source_sh ()

Lmod assumes that these scripts DO NOT have module commands or change $MODULEPATH.

Calling the shell script directly inside a modulefile

Site can also use the execute{} function inside a modulefile. This function add the shell script text at the end of the string that is evaluated by the shell. This execute command only makes sense if the evaluation is by the appropriate shell.

If a site wants to place the shell command first then they can use the print() statement as this will appear first. For example, to have the script appear before the unset environment commands then do this:

if (mode() == "unload") then
   print("app_script deactivate")
end

The string “app_script deactivate” will be generated before any other environment commands will generated.