2.5.12.42. Transferring Files

class buildbot.steps.transfer.FileUpload

class buildbot.steps.transfer.FileDownload

Most of the work involved in a build will take place on the worker. But occasionally it is useful to do some work on the buildmaster side. The most basic way to involve the buildmaster is simply to move a file from the worker to the master, or vice versa. There are a pair of steps named FileUpload and FileDownload to provide this functionality. FileUpload moves a file up to the master, while FileDownload moves a file down from the master.

As an example, let’s assume that there is a step which produces an HTML file within the source tree that contains some sort of generated project documentation. And let’s assume that we run nginx web server on the buildmaster host for serving static files. We want to move this file to the buildmaster, into a /usr/share/nginx/www/ directory, so it can be visible to developers. This file will wind up in the worker-side working directory under the name docs/reference.html. We want to put it into the master-side /usr/share/nginx/www/ref.html, and add a link to the HTML status to the uploaded file.

from buildbot.plugins import steps

f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.FileUpload(workersrc="docs/reference.html",
                           masterdest="/usr/share/nginx/www/ref.html",
                           url="http://somesite/~buildbot/ref.html"))

The masterdest= argument will be passed to os.path.expanduser, so things like ~ will be expanded properly. Non-absolute paths will be interpreted relative to the buildmaster’s base directory. Likewise, the workersrc= argument will be expanded and interpreted relative to the builder’s working directory.

Note

The copied file will have the same permissions on the master as on the worker, look at the mode= parameter to set it differently.

To move a file from the master to the worker, use the FileDownload command. For example, let’s assume that some step requires a configuration file that, for whatever reason, could not be recorded in the source code repository or generated on the worker side:

from buildbot.plugins import steps

f.addStep(steps.FileDownload(mastersrc="~/todays_build_config.txt",
                             workerdest="build_config.txt"))
f.addStep(steps.ShellCommand(command=["make", "config"]))

Like FileUpload, the mastersrc= argument is interpreted relative to the buildmaster’s base directory, and the workerdest= argument is relative to the builder’s working directory. If the worker is running in ~worker, and the builder’s builddir is something like tests-i386, then the workdir is going to be ~worker/tests-i386/build, and a workerdest= of foo/bar.html will get put in ~worker/tests-i386/build/foo/bar.html. Both of these commands will create any missing intervening directories.

Other Parameters

The maxsize= argument lets you set a maximum size for the file to be transferred. This may help to avoid surprises: transferring a 100MB coredump when you were expecting to move a 10kB status file might take an awfully long time. The blocksize= argument controls how the file is sent over the network: larger blocksizes are slightly more efficient but also consume more memory on each end, and there is a hard-coded limit of about 640kB.

The mode= argument allows you to control the access permissions of the target file, traditionally expressed as an octal integer. The most common value is probably 0o755, which sets the x executable bit on the file (useful for shell scripts and the like). The default value for mode= is None, which means the permission bits will default to whatever the umask of the writing process is. The default umask tends to be fairly restrictive, but at least on the worker you can make it less restrictive with a --umask command-line option at creation time (Worker Options).

The keepstamp= argument is a boolean that, when True, forces the modified and accessed time of the destination file to match the times of the source file. When False (the default), the modified and accessed times of the destination file are set to the current time on the buildmaster.

The url= argument allows you to specify an url that will be displayed in the HTML status. The title of the url will be the name of the item transferred (directory for DirectoryUpload or file for FileUpload). This allows the user to add a link to the uploaded item if that one is uploaded to an accessible place.

For FileUpload, the urlText= argument allows you to specify the url title that will be displayed in the web UI.

Transferring Directories

class buildbot.steps.transfer.DirectoryUpload

To transfer complete directories from the worker to the master, there is a BuildStep named DirectoryUpload. It works like FileUpload, just for directories. However it does not support the maxsize, blocksize and mode arguments. As an example, let’s assume an generated project documentation, which consists of many files (like the output of doxygen or epydoc). And let’s assume that we run nginx web server on buildmaster host for serving static files. We want to move the entire documentation to the buildmaster, into a /usr/share/nginx/www/docs directory, and add a link to the uploaded documentation on the HTML status page. On the worker-side the directory can be found under docs:

from buildbot.plugins import steps

f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.DirectoryUpload(workersrc="docs",
                                masterdest="/usr/share/nginx/www/docs",
                                url="~buildbot/docs"))

The DirectoryUpload step will create all necessary directories and transfers empty directories, too.

The maxsize and blocksize parameters are the same as for FileUpload, although note that the size of the transferred data is implementation-dependent, and probably much larger than you expect due to the encoding used (currently tar).

The optional compress argument can be given as 'gz' or 'bz2' to compress the datastream.

For DirectoryUpload the urlText= argument allows you to specify the url title that will be displayed in the web UI.

Note

The permissions on the copied files will be the same on the master as originally on the worker, see option buildbot-worker create-worker --umask to change the default one.

Transferring Multiple Files At Once

class buildbot.steps.transfer.MultipleFileUpload

In addition to the FileUpload and DirectoryUpload steps there is the MultipleFileUpload step for uploading a bunch of files (and directories) in a single BuildStep. The step supports all arguments that are supported by FileUpload and DirectoryUpload, but instead of a the single workersrc parameter it takes a (plural) workersrcs parameter. This parameter should either be a list, something that can be rendered as a list or a string which will be converted to a list. Additionally it supports the glob parameter if this parameter is set to True all arguments in workersrcs will be parsed through glob and the results will be uploaded to masterdest.:

from buildbot.plugins import steps

f.addStep(steps.ShellCommand(command=["make", "test"]))
f.addStep(steps.ShellCommand(command=["make", "docs"]))
f.addStep(steps.MultipleFileUpload(workersrcs=["docs", "test-results.html"],
                                   masterdest="/usr/share/nginx/www/",
                                   url="~buildbot"))

The url= parameter, can be used to specify a link to be displayed in the HTML status of the step.

The way URLs are added to the step can be customized by extending the MultipleFileUpload class. The allUploadsDone method is called after all files have been uploaded and sets the URL. The uploadDone method is called once for each uploaded file and can be used to create file-specific links.

import os

from buildbot.plugins import steps

class CustomFileUpload(steps.MultipleFileUpload):
    linkTypes = ('.html', '.txt')

    def linkFile(self, basename):
        name, ext = os.path.splitext(basename)
        return ext in self.linkTypes

    def uploadDone(self, result, source, masterdest):
        if self.url:
            basename = os.path.basename(source)
            if self.linkFile(basename):
                self.addURL(self.url + '/' + basename, basename)

    def allUploadsDone(self, result, sources, masterdest):
        if self.url:
            notLinked = [src for src in sources if not self.linkFile(src)]
            numFiles = len(notLinked)
            if numFiles:
                self.addURL(self.url, '... %d more' % numFiles)

For MultipleFileUpload the urlText= argument allows you to specify the url title that will be displayed in the web UI.

2.5.12.43. Transferring Strings

class buildbot.steps.transfer.StringDownload

class buildbot.steps.transfer.JSONStringDownload

class buildbot.steps.transfer.JSONPropertiesDownload

Sometimes it is useful to transfer a calculated value from the master to the worker. Instead of having to create a temporary file and then use FileDownload, you can use one of the string download steps.

from buildbot.plugins import steps, util

f.addStep(steps.StringDownload(util.Interpolate("%(src::branch)s-%(prop:got_revision)s\n"),
        workerdest="buildid.txt"))

StringDownload works just like FileDownload except it takes a single argument, s, representing the string to download instead of a mastersrc argument.

from buildbot.plugins import steps

buildinfo = {
    'branch': Property('branch'),
    'got_revision': Property('got_revision')
}
f.addStep(steps.JSONStringDownload(buildinfo, workerdest="buildinfo.json"))

JSONStringDownload is similar, except it takes an o argument, which must be JSON serializable, and transfers that as a JSON-encoded string to the worker.

from buildbot.plugins import steps

f.addStep(steps.JSONPropertiesDownload(workerdest="build-properties.json"))

JSONPropertiesDownload transfers a json-encoded string that represents a dictionary where properties maps to a dictionary of build property name to property value; and sourcestamp represents the build’s sourcestamp.