This is the second part of the Twisted tutorial Twisted from Scratch, or The Evolution of Finger .
In this section of the tutorial, our finger server will continue to sprout features: the ability for users to set finger announces, and using our finger service to send those announcements on the web, on IRC and over XML-RPC. Resources and XML-RPC are introduced in the Web Applications portion of the Twisted Web howto . More examples using twisted.words.protocols.irc can be found in Writing a TCP Client and the Twisted Words examples .
Now that port 1079 is free, maybe we can use it with a different server, one which will let people set their messages. It does no access control, so anyone who can login to the machine can set any message. We assume this is the desired behavior in our case. Testing it can be done by simply:
% nc localhost 1079 # or telnet localhost 1079
moshez
Giving a tutorial now, sorry!
^D
# But let's try and fix setting away messages, shall we?
from twisted.application import service, strports
from twisted.internet import protocol, reactor, defer
from twisted.protocols import basic
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerFactory(protocol.ServerFactory):
protocol = FingerProtocol
def __init__(self, users):
self.users = users
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
class FingerSetterProtocol(basic.LineReceiver):
def connectionMade(self):
self.lines = []
def lineReceived(self, line):
self.lines.append(line)
def connectionLost(self, reason):
user = self.lines[0]
status = self.lines[1]
self.factory.setUser(user, status)
class FingerSetterFactory(protocol.ServerFactory):
protocol = FingerSetterProtocol
def __init__(self, fingerFactory):
self.fingerFactory = fingerFactory
def setUser(self, user, status):
self.fingerFactory.users[user] = status
ff = FingerFactory({b'moshez': b'Happy and well'})
fsf = FingerSetterFactory(ff)
application = service.Application('finger', uid=1, gid=1)
serviceCollection = service.IServiceCollection(application)
strports.service("tcp:79", ff).setServiceParent(serviceCollection)
strports.service("tcp:1079", fsf).setServiceParent(serviceCollection)
This program has two protocol-factory-TCPServer pairs, which are
both child services of the application. Specifically,
the setServiceParent
method is used to define the two TCPServer services as children
of application , which implements IServiceCollection . Both
services are thus started with the application.
The previous version had the setter poke at the innards of the finger factory. This strategy is usually not a good idea: this version makes both factories symmetric by making them both look at a single object. Services are useful for when an object is needed which is not related to a specific network server. Here, we define a common service class with methods that will create factories on the fly. The service also contains methods the factories will depend on.
The factory-creation methods, getFingerFactory
and getFingerSetterFactory , follow this pattern:
Instantiate a generic server
factory, twisted.internet.protocol.ServerFactory .
Set the protocol class, just like our factory class would have.
Copy a service method to the factory as a function attribute. The
function won’t have access to the factory’s self , but
that’s OK because as a bound method it has access to the
service’s self , which is what it needs.
For getUser , a custom method defined in the service gets
copied. For setUser , a standard method of
the users dictionary is copied.
Thus, we stopped subclassing: the service simply puts useful methods and attributes inside the factories. We are getting better at protocol design: none of our protocol classes had to be changed, and neither will have to change until the end of the tutorial.
As an application service, this new finger service implements the IService interface and can be started and stopped in a standardized manner. We’ll make use of this in the next example.
# Fix asymmetry
from twisted.application import service, strports
from twisted.internet import protocol, reactor, defer
from twisted.protocols import basic
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerSetterProtocol(basic.LineReceiver):
def connectionMade(self):
self.lines = []
def lineReceived(self, line):
self.lines.append(line)
def connectionLost(self,reason):
user = self.lines[0]
status = self.lines[1]
self.factory.setUser(user, status)
class FingerService(service.Service):
def __init__(self, users):
self.users = users
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
def setUser(self, user, status):
self.users[user] = status
def getFingerFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerProtocol
f.getUser = self.getUser
return f
def getFingerSetterFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerSetterProtocol
f.setUser = self.setUser
return f
application = service.Application('finger', uid=1, gid=1)
f = FingerService({b'moshez': b'Happy and well'})
serviceCollection = service.IServiceCollection(application)
strports.service("tcp:79", f.getFingerFactory()
).setServiceParent(serviceCollection)
strports.service("tcp:1079", f.getFingerSetterFactory()
).setServiceParent(serviceCollection)
Most application services will want to use the Service base class, which implements
all the generic IService behavior.
This version shows how, instead of just letting users set their messages, we can read those from a centrally managed file. We cache results, and every 30 seconds we refresh it. Services are useful for such scheduled tasks.
listings/finger/etc.users
# Read from file
from twisted.application import service, strports
from twisted.internet import protocol, reactor, defer
from twisted.protocols import basic
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerService(service.Service):
def __init__(self, filename):
self.users = {}
self.filename = filename
def _read(self):
with open(self.filename, "rb") as f:
for line in f:
user, status = line.split(b':', 1)
user = user.strip()
status = status.strip()
self.users[user] = status
self.call = reactor.callLater(30, self._read)
def startService(self):
self._read()
service.Service.startService(self)
def stopService(self):
service.Service.stopService(self)
self.call.cancel()
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
def getFingerFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerProtocol
f.getUser = self.getUser
return f
application = service.Application('finger', uid=1, gid=1)
f = FingerService('/etc/users')
finger = strports.service("tcp:79", f.getFingerFactory())
finger.setServiceParent(service.IServiceCollection(application))
f.setServiceParent(service.IServiceCollection(application))
Since this version is reading data from a file (and refreshing the data
every 30 seconds), there is no FingerSetterFactory and thus
nothing listening on port 1079.
Here we override the standard startService
and stopService hooks in
the Finger service, which is set up as a child service of the
application in the last line of the code. startService
calls _read , the function responsible for reading the
data; reactor.callLater is then used to schedule it to
run again after thirty seconds every time it is
called. reactor.callLater returns an object that lets us
cancel the scheduled run in stopService using
its cancel method.
The same kind of service can also produce things useful for other
protocols. For example, in twisted.web, the factory itself
(Site ) is almost
never subclassed — instead, it is given a resource, which
represents the tree of resources available via URLs. That hierarchy is
navigated by Site
and overriding it dynamically is possible with getChild .
To integrate this into the Finger application (just because we can), we set
up a new TCPServer that calls the Site factory and retrieves resources via a
new function of FingerService named getResource .
This function specifically returns a Resource object with an overridden getChild method.
# Read from file, announce on the web!
from twisted.application import service, strports
from twisted.internet import protocol, reactor, defer
from twisted.protocols import basic
from twisted.web import resource, server, static
import cgi
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class FingerResource(resource.Resource):
def __init__(self, users):
self.users = users
resource.Resource.__init__(self)
# we treat the path as the username
def getChild(self, username, request):
"""
'username' is L{bytes}.
'request' is a 'twisted.web.server.Request'.
"""
messagevalue = self.users.get(username)
if messagevalue:
messagevalue = messagevalue.decode("ascii")
if username:
username = username.decode("ascii")
username = cgi.escape(username)
if messagevalue is not None:
messagevalue = cgi.escape(messagevalue)
text = '<h1>{}</h1><p>{}</p>'.format(username, messagevalue)
else:
text = '<h1>{}</h1><p>No such user</p>'.format(username)
text = text.encode("ascii")
return static.Data(text, 'text/html')
class FingerService(service.Service):
def __init__(self, filename):
self.filename = filename
self.users = {}
def _read(self):
self.users.clear()
with open(self.filename, "rb") as f:
for line in f:
user, status = line.split(b':', 1)
user = user.strip()
status = status.strip()
self.users[user] = status
self.call = reactor.callLater(30, self._read)
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
def getFingerFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerProtocol
f.getUser = self.getUser
return f
def getResource(self):
r = FingerResource(self.users)
return r
def startService(self):
self._read()
service.Service.startService(self)
def stopService(self):
service.Service.stopService(self)
self.call.cancel()
application = service.Application('finger', uid=1, gid=1)
f = FingerService('/etc/users')
serviceCollection = service.IServiceCollection(application)
f.setServiceParent(serviceCollection)
strports.service("tcp:79", f.getFingerFactory()
).setServiceParent(serviceCollection)
strports.service("tcp:8000", server.Site(f.getResource())
).setServiceParent(serviceCollection)
This is the first time there is client code. IRC clients often act a lot like
servers: responding to events from the network. The Client Service will make
sure that severed links will get re-established, with intelligent
tweaked exponential back-off algorithms. The IRC client itself is simple: the
only real hack is getting the nickname from the factory
in connectionMade .
# Read from file, announce on the web, irc
from twisted.application import internet, service, strports
from twisted.internet import protocol, reactor, defer, endpoints
from twisted.words.protocols import irc
from twisted.protocols import basic
from twisted.web import resource, server, static
import cgi
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class IRCReplyBot(irc.IRCClient):
def connectionMade(self):
self.nickname = self.factory.nickname
irc.IRCClient.connectionMade(self)
def privmsg(self, user, channel, msg):
user = user.split('!')[0]
if self.nickname.lower() == channel.lower():
d = self.factory.getUser(msg.encode("ascii"))
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
message = message.decode("ascii")
irc.IRCClient.msg(self, user, msg + ': ' + message)
d.addCallback(writeResponse)
class FingerService(service.Service):
def __init__(self, filename):
self.filename = filename
self.users = {}
def _read(self):
self.users.clear()
with open(self.filename, "rb") as f:
for line in f:
user, status = line.split(b':', 1)
user = user.strip()
status = status.strip()
self.users[user] = status
self.call = reactor.callLater(30, self._read)
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
def getFingerFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerProtocol
f.getUser = self.getUser
return f
def getResource(self):
def getData(path, request):
user = self.users.get(path, b"No such users <p/> usage: site/user")
path = path.decode("ascii")
user = user.decode("ascii")
text = '<h1>{}</h1><p>{}</p>'.format(path, user)
text = text.encode("ascii")
return static.Data(text, 'text/html')
r = resource.Resource()
r.getChild = getData
return r
def getIRCBot(self, nickname):
f = protocol.ClientFactory()
f.protocol = IRCReplyBot
f.nickname = nickname
f.getUser = self.getUser
return f
def startService(self):
self._read()
service.Service.startService(self)
def stopService(self):
service.Service.stopService(self)
self.call.cancel()
application = service.Application('finger', uid=1, gid=1)
f = FingerService('/etc/users')
serviceCollection = service.IServiceCollection(application)
f.setServiceParent(serviceCollection)
strports.service("tcp:79", f.getFingerFactory()
).setServiceParent(serviceCollection)
strports.service("tcp:8000", server.Site(f.getResource())
).setServiceParent(serviceCollection)
internet.ClientService(
endpoints.clientFromString(reactor, "tcp:irc.freenode.org:6667"),
f.getIRCBot('fingerbot')).setServiceParent(serviceCollection)
FingerService now has another new
function, getIRCbot , which returns
a ClientFactory . This factory in turn will
instantiate the IRCReplyBot protocol. The IRCBot is
configured in the last line to connect
to irc.freenode.org with a nickname
of fingerbot .
By
overriding irc.IRCClient.connectionMade , IRCReplyBot
can access the nickname attribute of the factory that
instantiated it.
In Twisted, XML-RPC support is handled just as though it was another resource. That resource will still support GET calls normally through render(), but that is usually left unimplemented. Note that it is possible to return deferreds from XML-RPC methods. The client, of course, will not get the answer until the deferred is triggered.
# Read from file, announce on the web, irc, xml-rpc
from twisted.application import internet, service, strports
from twisted.internet import protocol, reactor, defer, endpoints
from twisted.words.protocols import irc
from twisted.protocols import basic
from twisted.web import resource, server, static, xmlrpc
import cgi
class FingerProtocol(basic.LineReceiver):
def lineReceived(self, user):
d = self.factory.getUser(user)
def onError(err):
return b'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
self.transport.write(message + b'\r\n')
self.transport.loseConnection()
d.addCallback(writeResponse)
class IRCReplyBot(irc.IRCClient):
def connectionMade(self):
self.nickname = self.factory.nickname
irc.IRCClient.connectionMade(self)
def privmsg(self, user, channel, msg):
user = user.split('!')[0]
if self.nickname.lower() == channel.lower():
d = self.factory.getUser(msg.encode("ascii"))
def onError(err):
return 'Internal error in server'
d.addErrback(onError)
def writeResponse(message):
message = message.decode("ascii")
irc.IRCClient.msg(self, user, msg+ ': ' + message)
d.addCallback(writeResponse)
class FingerService(service.Service):
def __init__(self, filename):
self.filename = filename
self.users = {}
def _read(self):
self.users.clear()
with open(self.filename, "rb") as f:
for line in f:
user, status = line.split(b':', 1)
user = user.strip()
status = status.strip()
self.users[user] = status
self.call = reactor.callLater(30, self._read)
def getUser(self, user):
return defer.succeed(self.users.get(user, b"No such user"))
def getFingerFactory(self):
f = protocol.ServerFactory()
f.protocol = FingerProtocol
f.getUser = self.getUser
return f
def getResource(self):
def getData(path, request):
user = self.users.get(path, b"No such user <p/> usage: site/user")
path = path.decode("ascii")
user = user.decode("ascii")
text = '<h1>{}</h1><p>{}</p>'.format(path, user)
text = text.encode("ascii")
return static.Data(text, 'text/html')
r = resource.Resource()
r.getChild = getData
x = xmlrpc.XMLRPC()
x.xmlrpc_getUser = self.getUser
r.putChild('RPC2', x)
return r
def getIRCBot(self, nickname):
f = protocol.ClientFactory()
f.protocol = IRCReplyBot
f.nickname = nickname
f.getUser = self.getUser
return f
def startService(self):
self._read()
service.Service.startService(self)
def stopService(self):
service.Service.stopService(self)
self.call.cancel()
application = service.Application('finger', uid=1, gid=1)
f = FingerService('/etc/users')
serviceCollection = service.IServiceCollection(application)
f.setServiceParent(serviceCollection)
strports.service("tcp:79", f.getFingerFactory()
).setServiceParent(serviceCollection)
strports.service("tcp:8000", server.Site(f.getResource())
).setServiceParent(serviceCollection)
internet.ClientService(
endpoints.clientFromString(reactor, "tcp:irc.freenode.org:6667"),
f.getIRCBot('fingerbot')).setServiceParent(serviceCollection)
Instead of a web browser, we can test the XMLRPC finger using a simple
client based on Python’s built-in xmlrpclib , which will access
the resource we’ve made available at localhost/RPC2 .
# testing xmlrpc finger
try:
# Python 3
from xmlrpc.client import Server
except ImportError:
# Python 2
from xmlrpclib import Server
server = Server('http://127.0.0.1:8000/RPC2')
print(server.getUser('moshez'))