Caution

Buildbot no longer supports Python 2.7 on the Buildbot master.

3.3.9. WWW Server

3.3.9.1. History and Motivation

One of the goals of the ‘nine’ project is to rework Buildbot’s web services to use a more modern, consistent design and implement UI features in client-side JavaScript instead of server-side Python.

The rationale behind this is that a client side UI relieves pressure on the server while being more responsive for the user. The web server only concentrates on serving data via a REST interface wrapping the Data API. This removes a lot of sources of latency where, in previous versions, long synchronous calculations were made on the server to generate complex pages.

Another big advantage is live updates of status pages, without having to poll or reload. The new system uses Comet techniques in order to relay Data API events to connected clients.

Finally, making web services an integral part of Buildbot, rather than a status plugin, allows tighter integration with the rest of the application.

3.3.9.2. Design Overview

The www service exposes three pieces via HTTP:

  • A REST interface wrapping Data API;

  • HTTP-based messaging protocols wrapping the Messaging and Queues interface; and

  • Static resources implementing the client-side UI.

The REST interface is a very thin wrapper: URLs are translated directly into Data API paths, and results are returned directly, in JSON format. It is based on JSON API. Control calls are handled with a simplified form of JSONRPC 2.0.

The message interface is also a thin wrapper around Buildbot’s MQ mechanism. Clients can subscribe to messages, and receive copies of the messages, in JSON, as they are received by the buildmaster.

The client-side UI is an AngularJS application. Buildbot uses the Python setuptools entry-point mechanism to allow multiple packages to be combined into a single client-side experience. This allows frontend developers and users to build custom components for the web UI without hacking Buildbot itself.

Python development and AngularJS development are very different processes, requiring different environment requirements and skillsets. To maximize hackability, Buildbot separates the two cleanly. An experienced AngularJS hacker should be quite comfortable in the www/ directory, with a few exceptions described below. Similarly, an experienced Python hacker can simply download the pre-built web UI (from pypi!) and never venture near the www/ directory.

URLs

The Buildbot web interface is rooted at its base URL, as configured by the user. It is entirely possible for this base URL to contain path components, e.g., http://build.example.org/buildbot/, if hosted behind an HTTP proxy. To accomplish this, all URLs are generated relative to the base URL.

Overall, the space under the base URL looks like this:

  • / – The HTML document that loads the UI

  • /config – Returns implementation-defined master configuration used by the frontend UI. The same data may be embedded directly into the HTML document returned by the / URL.

  • /api/v{version} – The root of the REST APIs, each versioned numerically. Users should, in general, use the latest version.

  • /ws – The WebSocket endpoint to subscribe to messages from the mq system.

  • /sse – The server sent event endpoint where clients can subscribe to messages from the mq system.

3.3.9.3. REST API

Rest API is described in its own section.

3.3.9.4. Server-Side Session

The web server keeps a session state for each user, keyed on a session cookie. This session is available from request.getSession(), and data is stored as attributes. The following attributes may be available:

user_info

A dictionary maintained by the authentication subsystem. It may have the following information about the logged-in user:

  • username

  • email

  • full_name

  • groups (a list of group names)

As well as additional fields specific to the user info implementation.

The contents of the user_info dictionary are made available to the UI as config.user.

3.3.9.5. Message API

Currently, messages are implemented with two protocols, WebSockets and server sent events.

WebSocket

WebSocket is a protocol for arbitrary messaging to and from a browser. As an HTTP extension, the protocol is not yet well supported by all HTTP proxy technologies. Although, it has been reported to work well used behind the https protocol. Only one WebSocket connection is needed per browser.

The client can connect using the url ws[s]://<BB_BASE_URL>/ws.

The protocol used is a simple in-house protocol based on json. The structure of a command from the client is as follows:

{ "cmd": "<command name>", '_id': <id of the command>, "arg1": arg1, "arg2": arg2 }

  • cmd is used to reference a command name

  • _id is used to track the response, can be any unique number or string, generated by the client. It needs to be unique per websocket session.

Response is sent asynchronously, reusing _id to track which command is responded.

Success answer example would be:

{ "msg": "OK", "_id": 1, "code": 200 }

Error answer example would be:

{ "_id": 1, "code": 404, "error": "no such command 'poing'" }

The client can send several commands without waiting for a response.

Responses are not guaranteed to be sent in order.

Several commands are implemented:

ping
{ "_id": 1, "cmd": "ping" }

The server will respond with a “pong” message:

{ "_id": 1, "msg": "pong", "code": 200 }
startConsuming

Start consuming events that match path. paths are described in the Messaging and Queues section. For size optimization reasons, paths are joined with “/”, and with the None wildcard replaced by “*”.

{ "_id": 1, "cmd": "startConsuming", "path": "change/*/*" }

Success answer example will be:

{ "msg": "OK", "_id": 1, "code": 200 }
stopConsuming

Stop consuming events that were previously registered with path.

{ "_id": 1, "cmd": "stopConsuming", "path": "change/*/*" }

Success answer example will be:

{ "msg": "OK", "_id": 1, "code": 200 }

The client will receive events as websocket frames encoded in json with the following format:

{ "k": key, "m": message }

Server Sent Events

SSE is a simpler protocol than WebSockets and is more REST compliant. It uses the chunk-encoding HTTP feature to stream the events. SSE also does not work well behind an enterprise proxy, unless you use the https protocol.

The client can connect using following endpoints:

  • http[s]://<BB_BASE_URL>/sse/listen/<path>: Start listening to events on the http connection. Optionally, setup a first event filter on <path>. The first message send is a handshake, giving a uuid that can be used to add or remove event filters.

  • http[s]://<BB_BASE_URL>/sse/add/<uuid>/<path>: Configure a sse session to add an event filter

  • http[s]://<BB_BASE_URL>/sse/remove/<uuid>/<path>: Configure a sse session to remove an event filter

Note that if a load balancer is setup as a front end to buildbot web masters, the load balancer must be configured to always use the same master given a client IP address for /sse endpoint.

The client will receive events as sse events, encoded with the following format:

event: event
data: { "key": <key>, "message": <message> }

The first event received is a handshake, and is used to inform the client about the uuid to use for configuring additional event filters

event: handshake
data: <uuid>