Considerations for Production Deployments

This document contains a number of suggestions for deploying MediaGoblin in actual production environments. Consider “Deploying MediaGoblin” for a basic overview of how to deploy MediaGoblin.

Deploy with Paste

The MediaGoblin WSGI application instance you get with ./lazyserver.sh is not ideal for a production MediaGoblin deployment. Ideally, you should be able to use an “init” or “control” script to launch and restart the MediaGoblin process.

Use the following command as the basis for such a script:

CELERY_ALWAYS_EAGER=true \
 /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
 /srv/mediagoblin.example.org/mediagoblin/paste.ini \
 --pid-file=/var/run/mediagoblin.pid \
 --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543

The above configuration places MediaGoblin in “always eager” mode with Celery, this means that submissions of content will be processed synchronously, and the user will advance to the next page only after processing is complete. If we take Celery out of “always eager mode,” the user will be able to immediately return to the MediaGoblin site while processing is ongoing. In these cases, use the following command as the basis for your script:

CELERY_ALWAYS_EAGER=false \
 /srv/mediagoblin.example.org/mediagoblin/bin/paster serve \
 /srv/mediagoblin.example.org/mediagoblin/paste.ini \
 --pid-file=/var/run/mediagoblin.pid \
 --server-name=fcgi fcgi_host=127.0.0.1 fcgi_port=26543

Separate Celery

MediaGoblin uses Celery to handle heavy and long-running tasks. Celery can be launched in two ways:

  1. Embedded in the MediaGoblin WSGI application [1]. This is the way ./lazyserver.sh does it for you. It’s simple as you only have to run one process. The only bad thing with this is that the heavy and long-running tasks will run in the webserver, keeping the user waiting each time some heavy lifting is needed as in for example processing a video. This could lead to problems as an aborted connection will halt any processing and since most front-end web servers will terminate your connection if it doesn’t get any response from the MediaGoblin WSGI application in a while.

  2. As a separate process communicating with the MediaGoblin WSGI application via a broker. This offloads the heavy lifting from the MediaGoblin WSGI application and users will be able to continue to browse the site while the media is being processed in the background.

To launch Celery separately from the MediaGoblin WSGI application:

  1. Make sure that the CELERY_ALWAYS_EAGER environment variable is unset or set to false when launching the MediaGoblin WSGI application.

  2. Start the celeryd main process with

    CELERY_CONFIG_MODULE=mediagoblin.init.celery.from_celery ./bin/celeryd

Set up sentry to monitor exceptions

We have a plugin for raven integration, see the “raven plugin” documentation.

Use an Init Script

Look in your system’s /etc/init.d/ or /etc/rc.d/ directory for examples of how to build scripts that will start, stop, and restart MediaGoblin and Celery. These scripts will vary by distribution/operating system.

These are scripts provided by the MediaGoblin community:

Debian
Arch Linux