Using TLS in Twisted

Overview

This document describes how to secure your communications using TLS (Transport Layer Security) — also known as SSL (Secure Sockets Layer) — in Twisted servers and clients. It assumes that you know what TLS is, what some of the major reasons to use it are, and how to generate your own certificates. It also assumes that you are comfortable with creating TCP servers and clients as described in the server howto and client howto . After reading this document you should be able to create servers and clients that can use TLS to encrypt their connections, switch from using an unencrypted channel to an encrypted one mid-connection, and require client authentication.

Using TLS in Twisted requires that you have pyOpenSSL installed. A quick test to verify that you do is to run from OpenSSL import SSL at a python prompt and not get an error.

Twisted provides TLS support as a transport — that is, as an alternative to TCP. When using TLS, use of the TCP APIs you’re already familiar with, TCP4ClientEndpoint and TCP4ServerEndpoint — or reactor.listenTCP and reactor.connectTCP — is replaced by use of parallel TLS APIs (many of which still use the legacy name “SSL” due to age and/or compatibility with older APIs). To create a TLS server, use SSL4ServerEndpoint or listenSSL . To create a TLS client, use SSL4ClientEndpoint or connectSSL .

TLS provides transport layer security, but it’s important to understand what “security” means. With respect to TLS it means three things:

1. Identity: TLS servers (and sometimes clients) present a certificate, offering proof of who they are, so that you know who you are talking to.

2. Confidentiality: once you know who you are talking to, encryption of the connection ensures that the communications can’t be understood by any third parties who might be listening in.

3. Integrity: TLS checks the encrypted messages to ensure that they actually came from the party you originally authenticated to. If the messages fail these checks, then they are discarded and your application does not see them.

Without identity, neither confidentiality nor integrity is possible. If you don’t know who you’re talking to, then you might as easily be talking to your bank or to a thief who wants to steal your bank password. Each of the APIs listed above with “SSL” in the name requires a configuration object called (for historical reasons) a contextFactory. (Please pardon the somewhat awkward name.) The contextFactory serves three purposes:

1. It provides the materials to prove your own identity to the other side of the connection: in other words, who you are.

2. It expresses your requirements of the other side’s identity: in other words, who you would like to talk to (and who you trust to tell you that you’re talking to the right party).

3. It allows you to specify certain specialized options about the way the TLS protocol itself operates.

The requirements of clients and servers are slightly different. Both can provide a certificate to prove their identity, but commonly, TLS servers provide a certificate, whereas TLS clients check the server’s certificate (to make sure they’re talking to the right server) and then later identify themselves to the server some other way, often by offering a shared secret such as a password or API key via an application protocol secured with TLS and not as part of TLS itself.

Since these requirements are slightly different, there are different APIs to construct an appropriate contextFactory value for a client or a server.

For servers, we can use twisted.internet.ssl.CertificateOptions. In order to prove the server’s identity, you pass the privateKey and certificate arguments to this object. twisted.internet.ssl.PrivateCertificate.options() is a convenient way to create a CertificateOptions instance configured to use a particular key and certificate.

For clients, we can use twisted.internet.ssl.optionsForClientTLS(). This takes two arguments, hostname (which indicates what hostname must be advertised in the server’s certificate) and optionally trustRoot. By default, optionsForClientTLS tries to obtain the trust roots from your platform, but you can specify your own.

You may obtain an object suitable to pass as the trustRoot= parameter with an explicit list of twisted.internet.ssl.Certificate or twisted.internet.ssl.PrivateCertificate instances by calling twisted.internet.ssl.trustRootFromCertificates(). This will cause optionsForClientTLS to accept any connection so long as the server’s certificate is signed by at least one of the certificates passed.

Note

Currently, Twisted only supports loading of OpenSSL’s default trust roots. If you’ve built OpenSSL yourself, you must take care to include these in the appropriate location. If you’re using the OpenSSL shipped as part of macOS 10.5-10.9, this behavior will also be correct. If you’re using Debian, or one of its derivatives like Ubuntu, install the ca-certificates package to ensure you have trust roots available, and this behavior should also be correct. Work is ongoing to make platformTrust — the API that optionsForClientTLS uses by default — more robust. For example, platformTrust should fall back to the “certifi” package if no platform trust roots are available but it doesn’t do that yet. When this happens, you shouldn’t need to change your code.

TLS echo server and client

Now that we’ve got the theory out of the way, let’s try some working examples of how to get started with a TLS server. The following examples rely on the files server.pem (private key and self-signed certificate together) and public.pem (the server’s public certificate by itself).

TLS echo server

echoserv_ssl.py

#!/usr/bin/env python
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

import sys

import echoserv

from twisted.internet import defer, protocol, ssl, task
from twisted.python import log
from twisted.python.modules import getModule


def main(reactor):
    log.startLogging(sys.stdout)
    certData = getModule(__name__).filePath.sibling("server.pem").getContent()
    certificate = ssl.PrivateCertificate.loadPEM(certData)
    factory = protocol.Factory.forProtocol(echoserv.Echo)
    reactor.listenSSL(8000, factory, certificate.options())
    return defer.Deferred()


if __name__ == "__main__":
    import echoserv_ssl

    task.react(echoserv_ssl.main)

This server uses listenSSL to listen for TLS traffic on port 8000, using the certificate and private key contained in the file server.pem. It uses the same echo example server as the TCP echo server — even going so far as to import its protocol class. Assuming that you can buy your own TLS certificate from a certificate authority, this is a fairly realistic TLS server.

TLS echo client

echoclient_ssl.py

#!/usr/bin/env python
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

import echoclient

from twisted.internet import defer, endpoints, protocol, ssl, task
from twisted.python.modules import getModule


@defer.inlineCallbacks
def main(reactor):
    factory = protocol.Factory.forProtocol(echoclient.EchoClient)
    certData = getModule(__name__).filePath.sibling("public.pem").getContent()
    authority = ssl.Certificate.loadPEM(certData)
    options = ssl.optionsForClientTLS("example.com", authority)
    endpoint = endpoints.SSL4ClientEndpoint(reactor, "localhost", 8000, options)
    echoClient = yield endpoint.connect(factory)

    done = defer.Deferred()
    echoClient.connectionLost = lambda reason: done.callback(None)
    yield done


if __name__ == "__main__":
    import echoclient_ssl

    task.react(echoclient_ssl.main)

This client uses SSL4ClientEndpoint to connect to echoserv_ssl.py. It also uses the same echo example client as the TCP echo client. Whenever you have a protocol that listens on plain-text TCP it is easy to run it over TLS instead. It specifies that it only wants to talk to a host named "example.com", and that it trusts the certificate authority in "public.pem" to say who "example.com" is. Note that the host you are connecting to — localhost — and the host whose identity you are verifying — example.com — can differ. In this case, our example server.pem certificate identifies a host named “example.com”, but your server is proably running on localhost.

In a realistic client, it’s very important that you pass the same “hostname” your connection API (in this case, SSL4ClientEndpoint) and optionsForClientTLS. In this case we’re using “localhost” as the host to connect to because you’re probably running this example on your own computer and “example.com” because that’s the value hard-coded in the dummy certificate distributed along with Twisted’s example code.

Connecting To Public Servers

Here is a short example, now using the default trust roots for optionsForClientTLS from platformTrust.

check_server_certificate.py

import sys

from twisted.internet import defer, endpoints, error, protocol, ssl, task


def main(reactor, host, port=443):
    options = ssl.optionsForClientTLS(hostname=host.decode("utf-8"))
    port = int(port)

    class ShowCertificate(protocol.Protocol):
        def connectionMade(self):
            self.transport.write(b"GET / HTTP/1.0\r\n\r\n")
            self.done = defer.Deferred()

        def dataReceived(self, data):
            certificate = ssl.Certificate(self.transport.getPeerCertificate())
            print("OK:", certificate)
            self.transport.abortConnection()

        def connectionLost(self, reason):
            print("Lost.")
            if not reason.check(error.ConnectionClosed):
                print("BAD:", reason.value)
            self.done.callback(None)

    return endpoints.connectProtocol(
        endpoints.SSL4ClientEndpoint(reactor, host, port, options), ShowCertificate()
    ).addCallback(lambda protocol: protocol.done)


task.react(main, sys.argv[1:])

You can use this tool fairly simply to retrieve certificates from an HTTPS server with a valid TLS certificate, by running it with a host name. For example:

$ python check_server_certificate.py www.twistedmatrix.com
OK: <Certificate Subject=www.twistedmatrix.com ...>
$ python check_server_certificate.py www.cacert.org
BAD: [(... 'certificate verify failed')]
$ python check_server_certificate.py dornkirk.twistedmatrix.com
BAD: No service reference ID could be validated against certificate.

Note

To properly validate your hostname parameter according to RFC6125, please also install the “service_identity” and “idna” packages from PyPI. Without this package, Twisted will currently make a conservative guess as to the correctness of the server’s certificate, but this will reject a large number of potentially valid certificates. service_identity implements the standard correctly and it will be a required dependency for TLS in a future release of Twisted.

Using startTLS

If you want to switch from unencrypted to encrypted traffic mid-connection, you’ll need to turn on TLS with startTLS on both ends of the connection at the same time via some agreed-upon signal like the reception of a particular message. You can readily verify the switch to an encrypted channel by examining the packet payloads with a tool like Wireshark .

startTLS server

starttls_server.py

from twisted.internet import defer, endpoints, protocol, ssl, task
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule


class TLSServer(LineReceiver):
    def lineReceived(self, line):
        print("received: ", line)
        if line == b"STARTTLS":
            print("-- Switching to TLS")
            self.sendLine(b"READY")
            self.transport.startTLS(self.factory.options)


def main(reactor):
    certData = getModule(__name__).filePath.sibling("server.pem").getContent()
    cert = ssl.PrivateCertificate.loadPEM(certData)
    factory = protocol.Factory.forProtocol(TLSServer)
    factory.options = cert.options()
    endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
    endpoint.listen(factory)
    return defer.Deferred()


if __name__ == "__main__":
    import starttls_server

    task.react(starttls_server.main)

startTLS client

starttls_client.py

from twisted.internet import defer, endpoints, protocol, ssl, task
from twisted.protocols.basic import LineReceiver
from twisted.python.modules import getModule


class StartTLSClient(LineReceiver):
    def connectionMade(self):
        self.sendLine(b"plain text")
        self.sendLine(b"STARTTLS")

    def lineReceived(self, line):
        print("received: ", line)
        if line == b"READY":
            self.transport.startTLS(self.factory.options)
            self.sendLine(b"secure text")
            self.transport.loseConnection()


@defer.inlineCallbacks
def main(reactor):
    factory = protocol.Factory.forProtocol(StartTLSClient)
    certData = getModule(__name__).filePath.sibling("server.pem").getContent()
    factory.options = ssl.optionsForClientTLS(
        "example.com", ssl.PrivateCertificate.loadPEM(certData)
    )
    endpoint = endpoints.HostnameEndpoint(reactor, "localhost", 8000)
    startTLSClient = yield endpoint.connect(factory)

    done = defer.Deferred()
    startTLSClient.connectionLost = lambda reason: done.callback(None)
    yield done


if __name__ == "__main__":
    import starttls_client

    task.react(starttls_client.main)

startTLS is a transport method that gets passed a contextFactory. It is invoked at an agreed-upon time in the data reception method of the client and server protocols. The server uses PrivateCertificate.options to create a contextFactory which will use a particular certificate and private key (a common requirement for TLS servers).

The client creates an uncustomized CertificateOptions which is all that’s necessary for a TLS client to interact with a TLS server.

Client authentication

Server and client-side changes to require client authentication fall largely under the dominion of pyOpenSSL, but few examples seem to exist on the web so for completeness a sample server and client are provided here.

TLS server with client authentication via client certificate verification

When one or more certificates are passed to PrivateCertificate.options, the resulting contextFactory will use those certificates as trusted authorities and require that the peer present a certificate with a valid chain anchored by one of those authorities.

A server can use this to verify that a client provides a valid certificate signed by one of those certificate authorities; here is an example of such a certificate.

ssl_clientauth_server.py

#!/usr/bin/env python
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

import sys

import echoserv

from twisted.internet import defer, protocol, ssl, task
from twisted.python import log
from twisted.python.modules import getModule


def main(reactor):
    log.startLogging(sys.stdout)
    certData = getModule(__name__).filePath.sibling("public.pem").getContent()
    authData = getModule(__name__).filePath.sibling("server.pem").getContent()
    authority = ssl.Certificate.loadPEM(certData)
    certificate = ssl.PrivateCertificate.loadPEM(authData)
    factory = protocol.Factory.forProtocol(echoserv.Echo)
    reactor.listenSSL(8000, factory, certificate.options(authority))
    return defer.Deferred()


if __name__ == "__main__":
    import ssl_clientauth_server

    task.react(ssl_clientauth_server.main)

Client with certificates

The following client then supplies such a certificate as the clientCertificate argument to optionsForClientTLS, while still validating the server’s identity.

ssl_clientauth_client.py

#!/usr/bin/env python
# Copyright (c) Twisted Matrix Laboratories.
# See LICENSE for details.

import echoclient

from twisted.internet import defer, endpoints, protocol, ssl, task
from twisted.python.modules import getModule


@defer.inlineCallbacks
def main(reactor):
    factory = protocol.Factory.forProtocol(echoclient.EchoClient)
    certData = getModule(__name__).filePath.sibling("public.pem").getContent()
    authData = getModule(__name__).filePath.sibling("server.pem").getContent()
    clientCertificate = ssl.PrivateCertificate.loadPEM(authData)
    authority = ssl.Certificate.loadPEM(certData)
    options = ssl.optionsForClientTLS("example.com", authority, clientCertificate)
    endpoint = endpoints.SSL4ClientEndpoint(reactor, "localhost", 8000, options)
    echoClient = yield endpoint.connect(factory)

    done = defer.Deferred()
    echoClient.connectionLost = lambda reason: done.callback(None)
    yield done


if __name__ == "__main__":
    import ssl_clientauth_client

    task.react(ssl_clientauth_client.main)

Notice that these two examples are very, very similar to the TLS echo examples above. In fact, you can demonstrate a failed authentication by simply running echoclient_ssl.py against ssl_clientauth_server.py; you’ll see no output because the server closed the connection rather than echoing the client’s authenticated input.

TLS Protocol Options

For servers, it is desirable to offer Diffie-Hellman based key exchange that provides perfect forward secrecy. The ciphers are activated by default, however it is necessary to pass an instance of DiffieHellmanParameters to CertificateOptions via the dhParameters option to be able to use them.

For example,

from twisted.internet.ssl import CertificateOptions, DiffieHellmanParameters
from twisted.python.filepath import FilePath
dhFilePath = FilePath('dh_param_1024.pem')
dhParams = DiffieHellmanParameters.fromFile(dhFilePath)
options = CertificateOptions(..., dhParameters=dhParams)

Another part of the TLS protocol which CertificateOptions can control is the version of the TLS or SSL protocol used. By default, Twisted will configure it to use TLSv1.2 or later and disable the insecure SSLv3 protocol. Manual control over protocols can be helpful if you need to support legacy SSLv3 systems, or you wish to restrict it down to just the strongest of the TLS versions.

You can ask CertificateOptions to use a more secure default minimum than Twisted’s by using the raiseMinimumTo argument in the initializer:

from twisted.internet.ssl import CertificateOptions, TLSVersion
options = CertificateOptions(
    ...,
    raiseMinimumTo=TLSVersion.TLSv1_3)

This will always negotiate a minimum of TLSv1.3, but will negotiate higher versions if Twisted’s default is higher. This usage will stay secure if Twisted updates the minimum to some hypothetical future TLS version, rather than causing your application to use the now theoretically insecure minimum you set.

If you need a strictly hard range of TLS versions you wish CertificateOptions to negotiate, you can use the insecurelyLowerMinimumTo and lowerMaximumSecurityTo arguments in the initializer:

from twisted.internet.ssl import CertificateOptions, TLSVersion
options = CertificateOptions(
    ...,
    insecurelyLowerMinimumTo=TLSVersion.TLSv1_0,
    lowerMaximumSecurityTo=TLSVersion.TLSv1_2)

This will cause it to negotiate between TLSv1.0 and TLSv1.2, and will not change if Twisted’s default minimum TLS version is raised. Note that this may not work at all, due to your version of OpenSSL potentially limiting the availability of deprecated or broken TLS versions. It is highly recommended not to set lowerMaximumSecurityTo unless you have a peer that is known to misbehave on newer TLS versions, and to only set insecurelyLowerMinimumTo when Twisted’s minimum is not acceptable. Using these two arguments to CertificateOptions may make your application’s TLS insecure if you do not review it frequently, and should not be used in libraries.

SSLv3 support is still available and you can enable support for it if you wish. As an example, this supports all TLS versions and SSLv3:

from twisted.internet.ssl import CertificateOptions, TLSVersion
options = CertificateOptions(
    ...,
    insecurelyLowerMinimumTo=TLSVersion.SSLv3)

Future OpenSSL versions may completely remove the ability to negotiate the insecure SSLv3 protocol, and this will not allow you to re-enable it.

Additionally, it is possible to limit the acceptable ciphers for your connection by passing an IAcceptableCiphers object to CertificateOptions. Since Twisted uses a secure cipher configuration by default, it is discouraged to do so unless absolutely necessary.

Application Layer Protocol Negotiation (ALPN) and Next Protocol Negotiation (NPN)

ALPN and NPN are TLS extensions that can be used by clients and servers to negotiate what application-layer protocol will be spoken once the encrypted connection is established. This avoids the need for extra custom round trips once the encrypted connection is established. It is implemented as a standard part of the TLS handshake.

NPN is supported from OpenSSL version 1.0.1. ALPN is the newer of the two protocols, supported in OpenSSL versions 1.0.2 onward. These functions require pyOpenSSL version 0.15 or higher. To query the methods supported by your system, use twisted.internet.ssl.protocolNegotiationMechanisms(). It will return a collection of flags indicating support for NPN and/or ALPN.

twisted.internet.ssl.CertificateOptions and twisted.internet.ssl.optionsForClientTLS() allow for selecting the protocols your program is willing to speak after the connection is established.

On the server-side you will have:

from twisted.internet.ssl import CertificateOptions
options = CertificateOptions(..., acceptableProtocols=[b'h2', b'http/1.1'])

and for clients:

from twisted.internet.ssl import optionsForClientTLS
options = optionsForClientTLS(hostname=hostname, acceptableProtocols=[b'h2', b'http/1.1'])

Twisted will attempt to use both ALPN and NPN, if they’re available, to maximise compatibility with peers. If both ALPN and NPN are supported by the peer, the result from ALPN is preferred.

For NPN, the client selects the protocol to use; For ALPN, the server does. If Twisted is acting as the peer who is supposed to select the protocol, it will prefer the earliest protocol in the list that is supported by both peers.

To determine what protocol was negotiated, after the connection is done, use TLSMemoryBIOProtocol.negotiatedProtocol. It will return one of the protocol names passed to the acceptableProtocols parameter. It will return None if the peer did not offer ALPN or NPN.

It can also return None if no overlap could be found and the connection was established regardless (some peers will do this: Twisted will not). In this case, the protocol that should be used is whatever protocol would have been used if negotiation had not been attempted at all.

Warning

If ALPN or NPN are used and no overlap can be found, then the remote peer may choose to terminate the connection. This may cause the TLS handshake to fail, or may result in the connection being torn down immediately after being made. If Twisted is the selecting peer (that is, Twisted is the server and ALPN is being used, or Twisted is the client and NPN is being used), and no overlap can be found, Twisted will always choose to fail the handshake rather than allow an ambiguous connection to set up.

An example of using this functionality can be found in this example script for clients and this example script for servers.

Related facilities

twisted.protocols.amp supports encrypted connections and exposes a startTLS method one can use or subclass. twisted.web has built-in TLS support in its client , http , and xmlrpc modules.

Conclusion

After reading through this tutorial, you should be able to:

• Use listenSSL and connectSSL to create servers and clients that use TLS

• Use startTLS to switch a channel from being unencrypted to using TLS mid-connection

• Add server and client support for client authentication