Monitoring and Debugging gevent Applications

gevent applications are often long-running server processes. Beginning with version 1.3, gevent has special support for monitoring such applications and getting visibility into them.

Tip

For some additional tools, see the comments on issue 1021.

The Monitor Thread

gevent can be configured to start a native thread to watch over each hub it creates. Out of the box, that thread has support to watch two things, but you can add your own functions to be called periodically in this thread.

Blocking

When the monitor thread is enabled, by default it will watch for greenlets that block the event loop for longer than a configurable time interval. When such a blocking greenlet is detected, it will print a report to the hub’s exception_stream. It will also emit the gevent.events.EventLoopBlocked event.

See also

gevent.util.assert_switches()

For a scoped version of this.

Memory Usage

Optionally, you can set a memory limit. The monitor thread will check the process’s memory usage every memory_monitor_period seconds, and if it is found to exceed this value, the gevent.events.MemoryUsageThresholdExceeded event will be emitted. If in the future memory usage declines below the configured value, the gevent.events.MemoryUsageUnderThreshold event will be emitted.

Important

psutil must be installed to monitor memory usage.

Visibility

Tip

Insight into the monkey-patching process can be obtained by observing the events gevent.monkey emits.

It is sometimes useful to get an overview of all existing greenlets and their stack traces. The function gevent.util.print_run_info() will collect this info and print it (gevent.util.format_run_info() only collects and returns this information). The greenlets are organized into a tree based on the greenlet that spawned them.

The print_run_info function is commonly hooked up to a signal handler to get the application state at any given time.

For each greenlet the following information is printed:

• Its current execution stack

• If it is not running, its termination status and gevent.Greenlet.value or gevent.Greenlet.exception

• The stack at which it was spawned

• Its parent (usually the hub)

• Its minimal_ident

• Its name

• The spawn tree locals (only for the root of the spawn tree).

• The dicts of all gevent.local.local objects that are used in the greenlet.

The greenlet tree itself is represented as an object that you can also use for your own purposes: gevent.util.GreenletTree.

Profiling

The github repository nylas/nylas-perftools has some gevent-compatible profilers.

• stacksampler is a sampling profiler meant to be run in a greenlet in your server process and exposes data through an HTTP server; it is designed to be suitable for production usage.

• py2devtools is a greenlet-aware tracing profiler that outputs data that can be used by the Chrome dev tools; it is intended for developer usage.