Skip to content

Infrastructure administration

Treeherder has four apps, and those deployments and their databases are managed by cloudOps. All deployments, which the exception of treeherder-taskcluster-staging, ingest data from the same pulse guardian queues and the environments are mostly the same with a few key differences: production has user generated data from code and performance sheriffs, and the database size is much larger than the other deployments and the ingestion is slower.

Common Operations

Deploying Treeherder

Stage is set to auto-deploy from the master branch, and production from the production branch. Prototype deploys from a separate prototype branch, but it is not kept in sync with master (this will require anyone pushing their code to prototype to reset it to lastest master when they're done).

Production deploys are a manual process that is performed by a Treeherder admin roughly twice a week by running a task in Jenkins (under Treeherder -> status -> hover over latest master commit in table (stage:deploy column) and click proceed to initiate the promotion).

What's deployed will show what commits are currently on prototype, stage and production.

Note

To access treeherder-prod in Jenkins, cloudOps need to grant you access and you'll need to follow these steps to set it up before the url will work.

Using Prototype

The prototype branch is useful and recommended for testing changes that might impact users - such as schema changes, major rewrites or very large prs, and for modifications to cron jobs or to the data ingestion pipeline. However, it's important to note that any schema changes will need to be reset after testing which might involve having cloudOps manually deleting tables or columns, and potentially modifying the django_migrations table so that it matches the current state of django migration files (in fact, this applies to all deployments).

Access to push to the prototype branch requires special permission and an admin can grant access in the repository branch settings.

Note

Failed tests that have run on the CI will not block deployment to the prototype instance.

Reverting deployments

If a production promotion needs to be reverted, cloudOps can do it (ping whomever is listed as main contact in #treeherder-ops slack channel) or a Treeherder admin can do it in the Jenkins console. Click on the link of a previous commit (far left column) that was deployed to stage and select "rebuild" button on the left side nav.

Managing scheduled tasks, celery queues and environment variables

Changing any of these involves a kubernetes change; you can either open a pull request with the change to the cloudops-infra repo if you have access or file a bugzilla bug and someone from cloudOps will do it for you.

Database Management - cloudOps

These are managed by cloudOps and any deletion of data or access to replica or prototype needs to be arranged through them by filing a bugzilla bug.

Granting access to the read-only replica

One of the ways in which we allow users to access Treeherder data is via direct access to our read-only MySQL stage replica (only a restricted group of performance test team members and Treeherder maintainers should have access to the prototype database). Mozilla's ReDash instance use this approach. Only cloudOps can grant access.

Generate the password like this:

python -c "import string; import secrets; print(secrets.token_hex(30))"

-- Adjust the username and password accordingly.
CREATE USER 'myuser' IDENTIFIED BY 'PASSWORD >=30 CHARACTERS LONG' REQUIRE SSL;

-- Paswords for auth_user are randomly generated by Django and never used/exposed due to SSO.
grant SELECT on treeherder.*

Afterwards provide the user with the newly created credentials and the hostname of the read-only replica (mysql://<name>:<password>@35.247.25.202/treeherder), making sure to emphasise the need to connect using TLS.

Warning

These credentials will also work on the master production instance, so take care to provide the hostname of the replica and not the master - or their queries will run on the instance used by Treeherder, affecting its performance.

Changing MySQL passwords

To change the password for a MySQL account (only cloudOps can do this):

  1. Set the new password [according to the MySQL documentation].

    For example to change the current user's password:

    SET PASSWORD = 'NEW PASSWORD AT LEAST 30 CHARACTERS LONG';

    Or to change another user's password:

    SET PASSWORD FOR 'another_user' = 'NEW PASSWORD AT LEAST 30 CHARACTERS LONG';