Caution
Buildbot no longer supports Python 2.7 on the Buildbot master.
3.3.22. Statistics Service
The statistics service (or stats service) is implemented in buildbot.statistics.stats_service
.
Please see stats-service
for more information.
Here is a diagram demonstrating the workings of the stats service:
3.3.22.1. Stats Service
- class buildbot.statistics.stats_service.StatsService
An instance of this class functions as a
BuildbotService
. The instance of the running service is initialized in the master configuration file (seestats-service
for more information). The running service is accessible everywhere in Buildbot via theBuildMaster
. The service is available atself.master.namedServices['<service-name>']
. It takes the following initialization arguments:storage_backends
A list of storage backends. These are instance of subclasses of
StatsStorageBase
.name
(str) The name of this service. This name can be used to access the running instance of this service using
self.master.namedServices[name]
.
Please see
stats-service
for examples.- checkConfig(self, storage_backends)
storage_backends
A list of storage backends.
This method is called automatically to verify that the list of storage backends contains instances of subclasses of
StatsStorageBase
.
- reconfigService(self, storage_backends)
storage_backends
A list of storage backends.
This method is called automatically to reconfigure the running service.
- registerConsumers(self)
Internal method for this class called to register all consumers (methods from Capture classes) to the MQ layer.
- stopService(self)
Internal method for this class to stop the stats service and clean up.
- removeConsumers(self)
Internal method for this class to stop and remove consumers from the MQ layer.
- yieldMetricsValue(self, data_name, post_data, buildid)
data_name
(str) The name of the data being sent for storage.
post_data
A dictionary of key-value pairs that is sent for storage.
buildid
The integer build id of the current build. Obtainable in all
BuildSteps
.
This method should be called to post data that is not generated and stored as build-data in the database. This method generates the
stats-yield-data
event to the mq layer which is then consumed inpostData
.
3.3.22.2. Storage backends
Storage backends are responsible for storing any statistics/data sent to them. A storage backend will generally be some sort of a database-server running on a machine.
Note
This machine may be different from the one running BuildMaster
.
Data is captured according to the master config file and then is sent to each of the storage backends provided by the master configuration (see stats-service
).
Each storage backend has a Python client defined as part of buildbot.statistics.storage_backends
to aid in posting data by StatsService
.
Currently, only InfluxDB is supported as a storage backend.
- class buildbot.statistis.storage_backends.base.StatsStorageBase
An abstract class for all storage services. It cannot be directly initialized - it would raise a
TypeError
otherwise.- thd_postStatsValue(self, post_data, series_name, context)
post_data
A dict of key-value pairs that is sent for storage. The keys of this dict can be thought of as columns in a database and the value is the data stored for that column.
series_name
(str) The name of the time-series for this statistic.
context
(Optional) Any other contextual information about the data. It is a dict of key-value pairs.
An abstract method that needs to be implemented by every child class of this class. Not doing so will result in a
TypeError
when starting Buildbot.
- class buildbot.statistics.storage_backends.influxdb_client.InfluxStorageService
InfluxDB is a distributed time series database that employs a key-value pair storage system.
This class is a Buildbot client to the InfluxDB storage backend. It is available in the configuration as
statistics.InfluxStorageService
. It takes the following initialization arguments:url
(str) The URL where the service is running.
port
(int) The port on which the service is listening.
user
(str) Username of an InfluxDB user.
password
(str) Password for
user
.db
(str) The name of database to be used.
captures
A list of instances of subclasses of
Capture
. This tells which stats are to be stored in this storage backend.name=None
(Optional) (str) The name of this storage backend.
- thd_postStatsValue(self, post_data, series_name, context={})
post_data
A dict of key-value pairs that is sent for storage. The keys of this dict can be thought of as columns in a database and the value is the data stored for that column.
series_name
(str) The name of the time-series for this statistic.
context
(Optional) Any other contextual information about the data. It is a dict of key-value pairs.
This method constructs a dictionary of data to be sent to InfluxDB in the proper format and then sends the data to the InfluxDB instance.
3.3.22.3. Capture Classes
Capture classes are used for declaring which data needs to captured and sent to storage backends for storage.
- class buildbot.statistics.capture.Capture
This is the abstract base class for all capture classes. Not to be used directly. It’s initialized with the following parameters:
- routingKey
(tuple) The routing key to be used by
StatsService
to register consumers to the MQ layer for the subclass of this class.
- callback
The callback registered with the MQ layer for the consumer of a subclass of this class. Each subclass must provide a default callback for this purpose.
- _defaultContext(self, msg)
A method for providing default context to the storage backends.
- consume(self, routingKey, msg)
This is an abstract method - each subclass of this class should implement its own consume method. If not, then the subclass can’t be instantiated. The consume method, when called (from the mq layer), receives the following arguments:
- routingKey
The routing key which was registered to the MQ layer. Same as the
routingKey
provided to instantiate this class.
- msg
The message that was sent by the producer.
- _store(self, post_data, series_name, context)
This is an abstract method of this class. It must be implemented by all subclasses of this class. It takes the following arguments:
- post_data
(dict) The key-value pair being sent to the storage backend.
- series_name
(str) The name of the series to which this data is stored.
- context
(dict) Any additional information pertaining to the data being sent.
- class buildbot.statistics.capture.CapturePropertyBase
This is a base class for both
CaptureProperty
andCapturePropertyAllBuilders
and abstracts away much of the common functionality between the two classes. It cannot be initialized directly as it contains an abstract method and raisesTypeError
if tried. It is initialized with the following arguments:- property_name
(str) The name of property needed to be recorded as a statistic. This can be a regular expression if
regex=True
(see below).
- callback=None
The callback function that is used by
CaptureProperty.consumer
to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this:
- regex=False
If this is set to
True
, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
- consume(self, routingKey, msg)
The consumer for all CaptureProperty classes described below. This method filters out the correct properties as per the configuration file and sends those properties for storage. The subclasses of this method do not need to implement this method as it takes care of all the functionality itself. See
Capture
for more information.
- _builder_name_matches(self, builder_info)
This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the
consume
method mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:- builder_info
(dict) The dictionary returned by the data API containing the builder information.
- class buildbot.statistics.capture.CaptureProperty
The capture class for capturing build properties. It is available in the configuration as
statistics.CaptureProperty
.It takes the following arguments:
- builder_name
(str) The name of builder in which the property is recorded.
- property_name
(str) The name of property needed to be recorded as a statistic.
- callback=None
The callback function that is used by
CaptureProperty.consumer
to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (seeCapturePropertyBase
for more information).
- regex=False
If this is set to
True
, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
- _builder_name_matches(self, builder_info)
See
CapturePropertyBase
for more information on this method.
- class buildbot.statistics.capture.CapturePropertyAllBuilders
The capture class to use for capturing build properties on all builders. It is available in the configuration as
statistics.CapturePropertyAllBuilders
.It takes the following arguments:
- property_name
(str) The name of property needed to be recorded as a statistic.
- callback=None
The callback function that is used by
CaptureProperty.consumer
to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this (seeCapturePropertyBase
for more information).
- regex=False
If this is set to
True
, then the property name can be a regular expression. All properties matching this regular expression will be sent for storage.
- _builder_name_matches(self, builder_info)
See
CapturePropertyBase
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildTimes
A base class for all Capture classes that deal with build times (start/end/duration). Not to be used directly. It’s initialized with:
- builder_name
(str) The name of builder whose times are to be recorded.
- callback
The callback function that is used by a subclass of this class to post-process data before formatting it and sending it to the appropriate storage backends. A default callback is provided for this. Each subclass must provide a default callback that is used in initialization of this class should the user not provide a callback.
- consume(self, routingKey, msg)
The consumer for all subclasses of this class. See
Capture
for more information.
- _retValParams(self, msg)
This is an abstract method which needs to be implemented by subclasses. This method needs to return a list of parameters that will be passed to the
callback
function. See individual buildCaptureBuild*
classes for more information.
- _err_msg(self, build_data, builder_name)
A helper method that returns an error message for the
consume
method.
- _builder_name_matches(self, builder_info)
This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the
consume
method mentioned above. It checks whether a builder is allowed to send build times to the storage backend according to the configuration file. It takes one argument:- builder_info
(dict) The dictionary returned by the data API containing the builder information.
- class buildbot.statistics.capture.CaptureBuildStartTime
A capture class for capturing build start times. It takes the following arguments:
- builder_name
(str) The name of builder whose times are to be recorded.
- callback=None
The callback function for this class. See
CaptureBuildTimes
for more information.The default callback:
- _retValParams(self, msg)
Returns a list containing one Python datetime object (start time) from
msg
dictionary.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildStartTimeAllBuilders
A capture class for capturing build start times from all builders. It is a subclass of
CaptureBuildStartTime
. It takes the following arguments:- callback=None
The callback function for this class. See
CaptureBuildTimes
for more information.The default callback:
See
CaptureBuildStartTime.__init__
for the definition.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildEndTime
A capture class for capturing build end times. Takes the following arguments:
- builder_name
(str) The name of builder whose times are to be recorded.
- callback=None
The callback function for this class. See
CaptureBuildTimes
for more information.The default callback:
- _retValParams(self, msg)
Returns a list containing two Python datetime object (start time and end time) from
msg
dictionary.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildEndTimeAllBuilders
A capture class for capturing build end times from all builders. It is a subclass of
CaptureBuildEndTime
. It takes the following arguments:- callback=None
The callback function for this class. See
CaptureBuildTimes
for more information.The default callback:
See
CaptureBuildEndTime.__init__
for the definition.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildDuration
A capture class for capturing build duration. Takes the following arguments:
- builder_name
(str) The name of builder whose times are to be recorded.
- report_in='seconds'
Can be one of three:
'seconds'
,'minutes'
, or'hours'
. This is the units in which the build time will be reported.
- callback=None
The callback function for this class. See
CaptureBuildTimes
for more information.The default callback:
- default_callback(start_time, end_time)
It returns the duration of the build as per the
report_in
argument. It receives the following arguments:
- _retValParams(self, msg)
Returns a list containing one Python datetime object (end time) from
msg
dictionary.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureBuildDurationAllBuilders
A capture class for capturing build durations from all builders. It is a subclass of
CaptureBuildDuration
. It takes the following arguments:- callback=None
The callback function for this class. See
CaptureBuildTimes
for more.The default callback:
See
CaptureBuildDuration.__init__
for the definition.
- _builder_name_matches(self, builder_info)
See
CaptureBuildTimes
for more information on this method.
- class buildbot.statistics.capture.CaptureDataBase
This is a base class for both
CaptureData
andCaptureDataAllBuilders
and abstracts away much of the common functionality between the two classes. Cannot be initialized directly as it contains an abstract method and raisesTypeError
if tried. It is initialized with the following arguments:- data_name
(str) The name of data to be captured. Same as in
yieldMetricsValue
.
- callback=None
The callback function for this class.
The default callback:
The default callback takes a value
x
and return it without changing. As such,x
itself acts as thepost_data
sent to the storage backends.
- consume(self, routingKey, msg)
The consumer for this class. See
Capture
for more.
- _builder_name_matches(self, builder_info)
This is an abstract method and needs to be implemented by all subclasses of this class. This is a helper method to the
consume
method mentioned above. It checks whether a builder is allowed to send properties to the storage backend according to the configuration file. It takes one argument:- builder_info
(dict) The dictionary returned by the data API containing the builder information.
- class buildbot.statistics.capture.CaptureData
A capture class for capturing arbitrary data that is not stored as build-data. See
yieldMetricsValue
for more. Takes the following arguments for initialization:- data_name
(str) The name of data to be captured. Same as in
yieldMetricsValue
.
- builder_name
(str) The name of the builder on which the data is captured.
- callback=None
The callback function for this class.
The default callback:
See
CaptureDataBase
of definition.
- _builder_name_matches(self, builder_info)
See
CaptureDataBase
for more information on this method.
- class buildbot.statistics.capture.CaptureDataAllBuilders
A capture class to capture arbitrary data on all builders. See
yieldMetricsValue
for more. It takes the following arguments:- data_name
(str) The name of data to be captured. Same as in
yieldMetricsValue
.
- callback=None
The callback function for this class.
- _builder_name_matches(self, builder_info)
See
CaptureDataBase
for more information on this method.