Tracking Module Usage¶
Once you have a module system, it can be important to know what modules your users are using or not. This ability collect in some fashion has existed for a long time. What is new here is a complete solution: Using syslog to track module usage. Then collect this data into a database.
There are a number of steps but all are easy. The following is an overview. I am going to explain our setup and point out where you site may be different. In this setup I will be assuming that you are using a MySQL database and Rsyslog (version 5.8)
For a cluster, it is common to have each node in that cluster to send its syslog messages to a single central machine (called master in this discussion). I will be sending the module usage message through syslog to a separate machine. That separate machine will collect the module tracking data into a log file that just contains tracking data. Finally this data is written into the database.
Also provided is a script to analyze the data as shown below:
$ analyzeLmodDB --sqlPattern '%fftw%' counts --start '2015-01-01 --end '2015-02-01'
Module path Distinct Users
----------- --------------
/apps/intel13/mvapich2_1_9/modulefiles/fftw3/3.3.2 151
/apps/intel13/mvapich2_1_9/modulefiles/fftw2/2.1.5 62
/apps/intel13/impi_4_1/modulefiles/fftw3/3.3.2 45
/apps/intel13/impi_4_1/modulefiles/fftw2/2.1.5 19
This shows number of users of any fftw module for the first month of 2015.
Detailed Steps¶
Step 1¶
Use SitePackage.lua to send a message to syslog.:
--------------------------------------------------------------------------
-- load_hook(): Here we record the any modules loaded.
local hook = require("Hook")
local uname = require("posix").uname
local cosmic = require("Cosmic"):singleton()
local syshost = cosmic:value("LMOD_SYSHOST")
local s_msgT = {}
local function load_hook(t)
-- the arg t is a table:
-- t.modFullName: the module full name: (i.e: gcc/4.7.2)
-- t.fn: The file name: (i.e /apps/modulefiles/Core/gcc/4.7.2.lua)
-- use syshost from configuration if set
-- otherwise extract 2nd name from hostname: i.e. login1.stampede2.tacc.utexas.edu
local host = syshost
if (not host) then
local i,j, first
i,j, first, host = uname("%n"):find("([^.]*)%.([^.]*)%.")
end
if (mode() ~= "load") then return end
local msg = string.format("user=%s module=%s path=%s host=%s time=%f",
os.getenv("USER"), t.modFullName, t.fn, uname("%n"),
epoch())
s_msgT[t.modFullName] = msg
end
hook.register("load", load_hook)
local function report_loads()
for k,msg in pairs(s_msgT) do
lmod_system_execute("logger -t ModuleUsageTracking -p local0.info " .. msg)
end
end
ExitHookA.register(report_loads)
This code uses two “hook” functions. The first is load_hook. This means that every load will saved. The second hook is called at exit. If there were no errors then any module loads are reported by sending a syslog message with the tag “ModuleUsageTracking”
Please read the file src/SitePackage.lua to see how to use the environment variable LMOD_PACKAGE_PATH to point to your own SitePackage.lua.
You should check to see that Lmod finds your SitePackage.lua. If you do:
$ module --config
and it reports:
Modules based on Lua: Version X.Y ...
by Robert McLay mclay@tacc.utexas.edu
Description Value
----------- -----
...
Site Pkg location standard
Then you haven’t set things up correctly.
Step 2¶
Have “master” send the tracking messages to a separate computer.
You can add the following to master’s /etc/rsyslog.conf file:
if $programname contains 'ModuleUsageTracking' then @module_usage_tracking
&~
Where you change “module_usage_tracking” into a real machine name. Adding this to rsyslog.conf will direct all syslog messages to be sent to the “module_usage_tracking” machine.
Remember to restart the rsyslog daemon on master.
Step 3¶
On the “module_usage_tracking” machine you add to /etc/rsyslog.conf the following:
# read in include files
$IncludeConfig /etc/rsyslog.d/*.conf...
Then in /etc/rsyslog.d/moduleTracking.conf:
$Ruleset remote
if $programname contains 'ModuleUsageTracking' then /var/log/moduleUsage.log
$Ruleset RSYSLOG_DefaultRuleset
# provides UDP syslog reception
$ModLoad imudp
$InputUDPServerBindRuleset remote
$UDPServerRun 514
The above commands are in the language of rsyslog version 5.8. What this says is accept outside syslog messages on port 514 and if any are tagged with “ModuleUsageTracking” then write them to /var/log/moduleUsage.log
Remember to restart the rsyslog daemon on the “module_usage_tracking” machine.
Step 4¶
Create the file /etc/logrotate.d/moduleUsage:
/var/log/moduleUsage.log{
missingok
copytruncate
rotate 4
daily
create 644 root root
notifempty
}
This will log rotate the moduleUsage.log. Remember to restart the logrotate daemon. Note that it will be the second day before the log is rotated. On Centos machines, it seems that the log rotate happens at about 3am.
Step 5¶
I found the following site helpful in getting the MySQL database setup:
http://zetcode.com/db/mysqlpython/
Install MySQL db. Create a mysql root password. Then create an account in the database like this:
$ mysql -u root -p Enter password: mysql> CREATE DATABASE lmod; mysql> CREATE USER 'lmod'@'localhost' IDENTIFIED BY 'test623'; mysql> USE lmod; mysql> GRANT ALL ON lmod.* TO 'lmod'@'localhost'; mysql> flush privileges; mysql> quit;
You will want to change ‘test623’ to some other password. You’ll also probably want to allow access to this database from outside machines as well.
Use the “conf_create” program from the contrib/tracking_module_usage directory to create a file containing the access information for the db:
$ ./conf_create Database host: Database user: lmod Database pass: Database name: lmod
Where you’ll have to fill in the correct name for the database host and password. This creates a file named lmod_db.conf which is used by createDB.py, analyzeLmodDB and other programs to access the database.
Make sure your python knows about the MySQLdb module. Please use pip or something similar if it is unavailable.
Create the database by running the createDB.py program.:
$ ./createDB.py
Step 6¶
If you have more than one cluster and you want to store them in the same database you might want to modify the store_module_data program found in the contrib/tracking_module_usage directory. It assumes that host names are of the form: node_name.cluster_name.something.something and the current store_module_data program picks off the second field in the hostname. If your site names things differently you should modify that routine to match your needs.
I use a cron job to load the moduleUsage.log-* files. This is the script I use:
#!/bin/bash PATH=<path_to_python3>:$PATH cd ~mclay/load_module_usage for i in /var/log/moduleUsage.log-*; do ./store_module_data $i if [ "$?" -eq 0 ]; then rm -f $i fi done
Where <path_to_python3> has a python3 that can also import MySQLdb python module. If it is not already installed, you can do:
$ pip3 install mysqlclient
Also you’ll probably want to change ~mclay/load_module_usage to where ever you have the store_module_data program and lmod_db.conf files.
I am running this cron job on the “module_usage_tracking” machine at 5am every morning. This is after the log rotation has been done.
Step 7¶
Once data is being written to the database you can now start analyzing the data. You can use SQL commands directly into the MySQL data base or you can use the supplied script found in the contrib/tracking_module_usage directory: analyseLmodDB:
% ./analyzeLmodDB --help
usage: analyzeLmodDB [-h] [--dbname DBNAME] [--syshost SYSHOST]
[--start STARTDATE] [--end ENDDATE]
[--sqlPattern SQLPATTERN]
cmdA [cmdA ...]
positional arguments:
cmdA commands: counts, usernames, modules_used_by
optional arguments:
-h, --help show this help message and exit
--dbname DBNAME lmod db name
--syshost SYSHOST system host name
--start STARTDATE start date
--end ENDDATE end date
--sqlPattern SQLPATTERN sql pattern for matching
There are three kinds of reports this program will report. Only one command at a time.
counts: Report the number of distinct users of a particular module:
$ analyzeLmodDB --sqlPattern '%fftw%' --start '2015-01-01 --end '2015-02-01' counts Module path Distinct Users ----------- -------------- /apps/intel13/mvapich2_1_9/modulefiles/fftw3/3.3.2 151 /apps/intel13/mvapich2_1_9/modulefiles/fftw2/2.1.5 62 /apps/intel13/impi_4_1/modulefiles/fftw3/3.3.2 45 /apps/intel13/impi_4_1/modulefiles/fftw2/2.1.5 19To get all modules loaded in a date range do:
$ analyzeLmodDB --sqlPattern '%' --start '2015-01-01 --end '2015-02-01' counts
usernames: Report users of a particular pattern:
$ ./analyzeLmodDB --sqlPattern '%/apps/modulefiles/settarg%' usernames Module path User Name ----------- --------- /opt/apps/modulefiles/settarg/5.8 user1 /opt/apps/modulefiles/settarg/5.8 user2 /opt/apps/modulefiles/settarg/5.8 user3 /opt/apps/modulefiles/settarg/5.8.1 mclay /opt/apps/modulefiles/settarg/5.9.1 user5
modules_used_by: Report the modules used by a particular user:
$ ./analyzeLmodDB --start '2015-01-01 --end '2015-02-01' --sqlPattern 'mclay' modules_used_by Module path User Name ----------- --------- /opt/apps/gcc4_9/modulefiles/boost/1.55.0.lua mclay /opt/apps/gcc4_9/modulefiles/mvapich2/2.1 mclay /opt/apps/gcc4_9/mvapich2_2_1/modulefiles/phdf5/1.8.16.lua mclay /opt/apps/gcc4_9/mvapich2_2_1/modulefiles/pmetis/4.0.2.lua mclay /opt/apps/intel13/modulefiles/boost/1.55.0.lua mclay /opt/apps/intel13/modulefiles/mvapich2/1.9a2 mclay
Tracking user loads and not dependent loads¶
Some sites would like to track the modules loaded by users directly and not the dependent loads. If your site wished to do that then look at the directory in the source tree: contrib/more_hooks. In that directory is a SitePackage.lua file as well as README.md which explains how to just track user loads.
