Deferred Reference¶

Multiple callbacks¶

Multiple callbacks can be added to a Deferred. The first callback in the Deferred’s callback chain will be called with the result, the second with the result of the first callback, and so on. Why do we need this? Well, consider a Deferred returned by twisted.enterprise.adbapi - the result of a SQL query. A web widget might add a callback that converts this result into HTML, and pass the Deferred onwards, where the callback will be used by twisted to return the result to the HTTP client. The callback chain will be bypassed in case of errors or exceptions.

from twisted.internet import reactor, defer

class Getter:
    def gotResults(self, x):
        """
        The Deferred mechanism provides a mechanism to signal error
        conditions.  In this case, odd numbers are bad.

        This function demonstrates a more complex way of starting
        the callback chain by checking for expected results and
        choosing whether to fire the callback or errback chain
        """
        if self.d is None:
            print "Nowhere to put results"
            return

        d = self.d
        self.d = None
        if x % 2 == 0:
            d.callback(x*3)
        else:
            d.errback(ValueError("You used an odd number!"))

    def _toHTML(self, r):
        """
        This function converts r to HTML.

        It is added to the callback chain by getDummyData in
        order to demonstrate how a callback passes its own result
        to the next callback
        """
        return "Result: %s" % r

    def getDummyData(self, x):
        """
        The Deferred mechanism allows for chained callbacks.
        In this example, the output of gotResults is first
        passed through _toHTML on its way to printData.

        Again this function is a dummy, simulating a delayed result
        using callLater, rather than using a real asynchronous
        setup.
        """
        self.d = defer.Deferred()
        # simulate a delayed result by asking the reactor to schedule
        # gotResults in 2 seconds time
        reactor.callLater(2, self.gotResults, x)
        self.d.addCallback(self._toHTML)
        return self.d

def printData(d):
    print d

def printError(failure):
    import sys
    sys.stderr.write(str(failure))

# this series of callbacks and errbacks will print an error message
g = Getter()
d = g.getDummyData(3)
d.addCallback(printData)
d.addErrback(printError)

# this series of callbacks and errbacks will print "Result: 12"
g = Getter()
d = g.getDummyData(4)
d.addCallback(printData)
d.addErrback(printError)

reactor.callLater(4, reactor.stop)
reactor.run()

Visual Explanation¶

1. Requesting method (data sink) requests data, gets Deferred object.

2. Requesting method attaches callbacks to Deferred object.

1. When the result is ready, give it to the Deferred object. .callback(result) if the operation succeeded, .errback(failure) if it failed. Note that failure is typically an instance of a twisted.python.failure.Failure instance.

2. Deferred object triggers previously-added (call/err)back with the result or failure . Execution then follows the following rules, going down the chain of callbacks to be processed.

   • Result of the callback is always passed as the first argument to the next callback, creating a chain of processors.

   • If a callback raises an exception, switch to errback.

   • An unhandled failure gets passed down the line of errbacks, this creating an asynchronous analog to a series to a series of except: statements.

   • If an errback doesn’t raise an exception or return a twisted.python.failure.Failure instance, switch to callback.

Unhandled Errors¶

If a Deferred is garbage-collected with an unhandled error (i.e. it would call the next errback if there was one), then Twisted will write the error’s traceback to the log file. This means that you can typically get away with not adding errbacks and still get errors logged. Be careful though; if you keep a reference to the Deferred around, preventing it from being garbage-collected, then you may never see the error (and your callbacks will mysteriously seem to have never been called). If unsure, you should explicitly add an errback after your callbacks, even if all you do is:

# Make sure errors get logged
from twisted.python import log
d.addErrback(log.err)

Handling possible Deferreds in the library code¶

Here is an example of a synchronous user authentication function that might be passed to authenticateUser :

synch-validation.py

def synchronousIsValidUser(user):
    '''
    Return true if user is a valid user, false otherwise
    '''
    return user in ["Alice", "Angus", "Agnes"]

However, here’s an asynchronousIsValidUser function that returns a Deferred:

from twisted.internet import reactor, defer

def asynchronousIsValidUser(user):
    d = defer.Deferred()
    reactor.callLater(2, d.callback, user in ["Alice", "Angus", "Agnes"])
    return d

Our original implementation of authenticateUser expected isValidUser to be synchronous, but now we need to change it to handle both synchronous and asynchronous implementations of isValidUser . For this, we use maybeDeferred to call isValidUser , ensuring that the result of isValidUser is a Deferred, even if isValidUser is a synchronous function:

from twisted.internet import defer

def printResult(result):
    if result:
        print "User is authenticated"
    else:
        print "User is not authenticated"

def authenticateUser(isValidUser, user):
    d = defer.maybeDeferred(isValidUser, user)
    d.addCallback(printResult)

Now isValidUser could be either synchronousIsValidUser or asynchronousIsValidUser .

It is also possible to modify synchronousIsValidUser to return a Deferred, see Generating Deferreds for more information.

Motivation¶

A Deferred may take any amount of time to be called back; in fact, it may never be called back. Your users may not be that patient. Since all actions taken when the Deferred completes are in your application or library’s callback code, you always have the option of simply disregarding the result when you receive it, if it’s been too long. However, while you’re ignoring it, the underlying operation represented by that Deferred is still chugging along in the background, possibly consuming resources such as CPU time, memory, network bandwidth and maybe even disk space. So, when the user has closed the window, hit the cancel button, disconnected from your server or sent a “stop” network message, you will want to announce your indifference to the result of that operation so that the originator of the Deferred can clean everything up and free those resources to be put to better use.

Cancellation for Applications which Consume Deferreds¶

Here’s a simple example. You’re connecting to an external host with an endpoint , but that host is really slow. You want to put a “cancel” button into your application to terminate the connection attempt, so the user can try connecting to a different host instead. Here’s a simple sketch of such an application, with the actual user interface left as an exercise for the reader:

def startConnecting(someEndpoint):
    def connected(it):
        "Do something useful when connected."
    return someEndpoint.connect(myFactory).addCallback(connected)
# ...
connectionAttempt = startConnecting(endpoint)
def cancelClicked():
    connectionAttempt.cancel()

Obviously (I hope), startConnecting is meant to be called by some UI element that lets the user choose what host to connect to and then constructs an appropriate endpoint (perhaps using twisted.internet.endpoints.clientFromString ). Then, a cancel button, or similar, is hooked up to the cancelClicked .

When connectionAttempt.cancel is invoked, that will:

1. cause the underlying connection operation to be terminated, if it is still ongoing

2. cause the connectionAttempt Deferred to be completed, one way or another, in a timely manner

3. likely cause the connectionAttempt Deferred to be errbacked with CancelledError

You may notice that that set of consequences is very heavily qualified. Although cancellation indicates the calling API’s desire for the underlying operation to be stopped, the underlying operation cannot necessarily react immediately. Even in this very simple example, there is already one thing that might not be interruptable: platform-native name resolution blocks, and therefore needs to be executed in a thread; the connection operation can’t be cancelled if it’s stuck waiting for a name to be resolved in this manner. So, the Deferred that you are cancelling may not callback or errback right away.

A Deferred may wait upon another Deferred at any point in its callback chain (see “Handling…asynchronous results”, above). There’s no way for a particular point in the callback chain to know if everything is finished. Since multiple layers of the callback chain may wish to cancel the same Deferred, any layer may call .cancel() at any time. The .cancel() method never raises any exception or returns any value; you may call it repeatedly, even on a Deferred which has already fired, or which has no remaining callbacks.

The main reason for all these qualifications, aside from specific examples, is that anyone who instantiates a Deferred may supply it with a cancellation function; that function can do absolutely anything that it wants to. Ideally, anything it does will be in the service of stopping the operation your requested, but there’s no way to guarantee any exact behavior across all Deferreds that might be cancelled. Cancellation of Deferreds is best effort. This may be the case for a number of reasons:

1. The Deferred doesn’t know how to cancel the underlying operation.

2. The underlying operation may have reached an uncancellable state, because some irreversible operation has been done.

3. The Deferred may already have a result, and so there’s nothing to cancel.

Calling cancel() will always succeed without an error regardless of whether or not cancellation was possible. In cases 1 and 2 the Deferred may well errback with a twisted.internet.defer.CancelledError while the underlying operation continues. Deferred s that support cancellation should document what they do when cancelled, if they are uncancellable in certain edge cases, etc..

If the cancelled Deferred is waiting on another Deferred , the cancellation will be forwarded to the other Deferred .

Default Cancellation Behavior¶

All Deferreds support cancellation. However, by default, they support a very rudimentary form of cancellation which doesn’t free any resources.

Consider this example of a Deferred which is ignorant of cancellation:

operation = Deferred()
def x(result):
    print("Hooray, a result:" + repr(x))
operation.addCallback(x)
# ...
def operationDone():
    operation.callback("completed")

A caller of an API that receives operation may call cancel on it. Since operation does not have a cancellation function, one of two things will happen.

1. If operationDone has been called, and the operation has completed, nothing much will change. operation will still have a result, and there are no more callbacks, so there’s no observable change in behavior.

2. If operationDone has not yet been invoked, then operation will be immediately errbacked with a CancelledError .

   However, once it’s cancelled, there’s no way to tell operationDone not to run; it will eventually call operation.callback later. In normal operation, issuing callback on a Deferred that has already called back results in an AlreadyCalledError , and this would cause an ugly traceback that could not be caught. Therefore, .callback can be invoked exactly once, causing a no-op, on a Deferred which has been cancelled but has no canceller. If you call it multiple times, you will still get an AlreadyCalledError exception.

Creating Cancellable Deferreds: Custom Cancellation Functions¶

Let’s imagine you are implementing an HTTP client, which returns a Deferred firing with the response from the server. Cancellation is best achieved by closing the connection. In order to make cancellation do that, all you have to do is pass a function to the constructor of the Deferred (it will get called with the Deferred that is being cancelled):

class HTTPClient(Protocol):
    def request(self, method, path):
        self.resultDeferred = Deferred(
            lambda ignore: self.transport.abortConnection())
        request = b"%s %s HTTP/1.0\r\n\r\n" % (method, path)
        self.transport.write(request)
        return self.resultDeferred

    def dataReceived(self, data):
        # ... parse HTTP response ...
        # ... eventually call self.resultDeferred.callback() ...

Now if someone calls cancel() on the Deferred returned from HTTPClient.request() , the HTTP request will be cancelled (assuming it’s not too late to do so). Care should be taken not to``callback()`` a Deferred that has already been cancelled.

Other behaviours¶

DeferredList accepts three keyword arguments that modify its behaviour:fireOnOneCallback , fireOnOneErrback and consumeErrors . If fireOnOneCallback is set, the DeferredList will immediately call its callback as soon as any of its Deferreds call their callback. Similarly, fireOnOneErrback will call errback as soon as any of the Deferreds call their errback. Note that DeferredList is still one-shot, like ordinary Deferreds, so after a callback or errback has been called the DeferredList will do nothing further (it will just silently ignore any other results from its Deferreds).

The fireOnOneErrback option is particularly useful when you want to wait for all the results if everything succeeds, but also want to know immediately if something fails.

The consumeErrors argument will stop the DeferredList from propagating any errors along the callback chains of any Deferreds it contains (usually creating a DeferredList has no effect on the results passed along the callbacks and errbacks of their Deferreds). Stopping errors at the DeferredList with this option will prevent “Unhandled error in Deferred” warnings from the Deferreds it contains without needing to add extra errbacks [1] . Passing a true value for the consumeErrors parameter will not change the behavior of fireOnOneCallback or fireOnOneErrback .

gatherResults¶

A common use for DeferredList is to “join” a number of parallel asynchronous operations, finishing successfully if all of the operations were successful, or failing if any one of the operations fails. In this case, twisted.internet.defer.gatherResults is a useful shortcut:

from twisted.internet import defer
d1 = defer.Deferred()
d2 = defer.Deferred()
d = defer.gatherResults([d1, d2], consumeErrors=True)
def printResult(result):
    print result
d.addCallback(printResult)
d1.callback("one")
# nothing is printed yet; d is still awaiting completion of d2
d2.callback("two")
# printResult prints ["one", "two"]

The consumeErrors argument has the same meaning as it does for NEEDS A TITLE : if true, it causes gatherResults to consume any errors in the passed-in Deferreds. Always use this argument unless you are adding further callbacks or errbacks to the passed-in Deferreds, or unless you know that they will not fail. Otherwise, a failure will result in an unhandled error being logged by Twisted. This argument is available since Twisted 11.1.0.

Basic Callback Functions¶

• addCallbacks(self, callback[, errback, callbackArgs, callbackKeywords, errbackArgs, errbackKeywords])

  This is the method you will use to interact with Deferred. It adds a pair of callbacks “parallel” to each other (see diagram above) in the list of callbacks made when the Deferred is called back to. The signature of a method added using addCallbacks should be myMethod(result, *methodArgs, **methodKeywords) . If your method is passed in the callback slot, for example, all arguments in the tuple callbackArgs will be passed as *methodArgs to your method.

  There are various convenience methods that are derivative of addCallbacks. I will not cover them in detail here, but it is important to know about them in order to create concise code.

  • addCallback(callback, *callbackArgs, **callbackKeywords)

    Adds your callback at the next point in the processing chain, while adding an errback that will re-raise its first argument, not affecting further processing in the error case.

    Note that, while addCallbacks (plural) requires the arguments to be passed in a tuple, addCallback (singular) takes all its remaining arguments as things to be passed to the callback function. The reason is obvious: addCallbacks (plural) cannot tell whether the arguments are meant for the callback or the errback, so they must be specifically marked by putting them into a tuple. addCallback (singular) knows that everything is destined to go to the callback, so it can use Python’s “*” and “**” syntax to collect the remaining arguments.

  • addErrback(errback, *errbackArgs, **errbackKeywords)

    Adds your errback at the next point in the processing chain, while adding a callback that will return its first argument, not affecting further processing in the success case.

  • addBoth(callbackOrErrback, *callbackOrErrbackArgs, **callbackOrErrbackKeywords)

    This method adds the same callback into both sides of the processing chain at both points. Keep in mind that the type of the first argument is indeterminate if you use this method! Use it for finally: style blocks.

Chaining Deferreds¶

If you need one Deferred to wait on another, all you need to do is return a Deferred from a method added to addCallbacks. Specifically, if you return Deferred B from a method added to Deferred A using A.addCallbacks, Deferred A’s processing chain will stop until Deferred B’s .callback() method is called; at that point, the next callback in A will be passed the result of the last callback in Deferred B’s processing chain at the time.

If this seems confusing, don’t worry about it right now – when you run into a situation where you need this behavior, you will probably recognize it immediately and realize why this happens. If you want to chain deferreds manually, there is also a convenience method to help you.

• chainDeferred(otherDeferred)

  Add otherDeferred to the end of this Deferred’s processing chain. When self.callback is called, the result of my processing chain up to this point will be passed to otherDeferred.callback . Further additions to my callback chain do not affect otherDeferred

  This is the same as self.addCallbacks(otherDeferred.callback, otherDeferred.errback)