Generating Deferreds¶

Class overview¶

This is an overview API reference for Deferred from the point of creating a Deferred and firing its callbacks and errbacks. It is not meant to be a substitute for the docstrings in the Deferred class, but can provide guidelines for its use.

There is a parallel overview of functions used by calling function which the Deferred is returned to at Using Deferreds .

What Deferreds don’t do: make your code asynchronous¶

Deferreds do not make the code magically not block.

Let’s take this function as an example:

from twisted.internet import defer

TARGET = 10000

def largeFibonnaciNumber():
    # create a Deferred object to return:
    d = defer.Deferred()

    # calculate the ten thousandth Fibonnaci number

    first = 0
    second = 1

    for i in xrange(TARGET - 1):
        new = first + second
        first = second
        second = new
        if i % 100 == 0:
            print "Progress: calculating the %dth Fibonnaci number" % i

    # give the Deferred the answer to pass to the callbacks:
    d.callback(second)

    # return the Deferred with the answer:
    return d

import time

timeBefore = time.time()

# call the function and get our Deferred
d = largeFibonnaciNumber()

timeAfter = time.time()

print "Total time taken for largeFibonnaciNumber call: %0.3f seconds" % \
      (timeAfter - timeBefore)

# add a callback to it to print the number

def printNumber(number):
    print "The %dth Fibonacci number is %d" % (TARGET, number)

print "Adding the callback now."

d.addCallback(printNumber)

You will notice that despite creating a Deferred in the largeFibonnaciNumber function, these things happened:

• the “Total time taken for largeFibonnaciNumber call” output shows that the function did not return immediately as asynchronous functions are expected to do; and

• rather than the callback being added before the result was available and called after the result is available, it isn’t even added until after the calculation has been completed.

The function completed its calculation before returning, blocking the process until it had finished, which is exactly what asynchronous functions are not meant to do. Deferreds are not a non-blocking talisman: they are a signal for asynchronous functions to use to pass results onto callbacks, but using them does not guarantee that you have an asynchronous function.

Advanced Processing Chain Control¶

• pause()

  Cease calling any methods as they are added, and do not respond to callback , until self.unpause() is called.

• unpause()

  If callback has been called on this Deferred already, call all the callbacks that have been added to this Deferred since pause was called.

  Whether it was called or not, this will put this Deferred in a state where further calls to addCallbacks or callback will work as normal.

Returning Deferreds from synchronous functions¶

Sometimes you might wish to return a Deferred from a synchronous function. There are several reasons why, the major two are maintaining API compatibility with another version of your function which returns a Deferred, or allowing for the possibility that in the future your function might need to be asynchronous.

In the Using Deferreds reference, we gave the following example of a synchronous function:

synch-validation.py

def synchronousIsValidUser(user):
    '''
    Return true if user is a valid user, false otherwise
    '''
    return user in ["Alice", "Angus", "Agnes"]

While we can require that callers of our function wrap our synchronous result in a Deferred using maybeDeferred , for the sake of API compatibility it is better to return a Deferred ourselves using defer.succeed :

from twisted.internet import defer

def immediateIsValidUser(user):
    '''
    Returns a Deferred resulting in true if user is a valid user, false
    otherwise
    '''

    result = user in ["Alice", "Angus", "Agnes"]

    # return a Deferred object already called back with the value of result
    return defer.succeed(result)

There is an equivalent defer.fail method to return a Deferred with the errback chain already fired.

Integrating blocking code with Twisted¶

At some point, you are likely to need to call a blocking function: many functions in third party libraries will have long running blocking functions. There is no way to ‘force’ a function to be asynchronous: it must be written that way specifically. When using Twisted, your own code should be asynchronous, but there is no way to make third party functions asynchronous other than rewriting them.

In this case, Twisted provides the ability to run the blocking code in a separate thread rather than letting it block your application. The twisted.internet.threads.deferToThread function will set up a thread to run your blocking function, return a Deferred and later fire that Deferred when the thread completes.

Let’s assume our largeFibonnaciNumber function from above is in a third party library (returning the result of the calculation, not a Deferred) and is not easily modifiable to be finished in discrete blocks. This example shows it being called in a thread, unlike in the earlier section we’ll see that the operation does not block our entire program:

def largeFibonnaciNumber():
    """
    Represent a long running blocking function by calculating
    the TARGETth Fibonnaci number
    """
    TARGET = 10000

    first = 0
    second = 1

    for i in xrange(TARGET - 1):
        new = first + second
        first = second
        second = new

    return second

from twisted.internet import threads, reactor

def fibonacciCallback(result):
    """
    Callback which manages the largeFibonnaciNumber result by
    printing it out
    """
    print "largeFibonnaciNumber result =", result
    # make sure the reactor stops after the callback chain finishes,
    # just so that this example terminates
    reactor.stop()

def run():
    """
    Run a series of operations, deferring the largeFibonnaciNumber
    operation to a thread and performing some other operations after
    adding the callback
    """
    # get our Deferred which will be called with the largeFibonnaciNumber result
    d = threads.deferToThread(largeFibonnaciNumber)
    # add our callback to print it out
    d.addCallback(fibonacciCallback)
    print "1st line after the addition of the callback"
    print "2nd line after the addition of the callback"

if __name__ == '__main__':
    run()
    reactor.run()

Possible sources of error¶

Deferreds greatly simplify the process of writing asynchronous code by providing a standard for registering callbacks, but there are some subtle and sometimes confusing rules that you need to follow if you are going to use them. This mostly applies to people who are writing new systems that use Deferreds internally, and not writers of applications that just add callbacks to Deferreds produced and processed by other systems. Nevertheless, it is good to know.