Networking is an essential task in software applications nowadays. Many
programming languages have support for network programming to various extents.
While the core libraries of most languages allow low-level socket programming,
other libraries and third-party extensions often facilitate higher-level
Internet protocols. For example, Java has a standard API to access sockets and
send emails (via the
URL class), but other common Internet
protocols are available through external libraries such as JavaMail, JTelnet,
and JTA. Perl has a native Unix-style interface to sockets and convenient core
modules such as
Net::FTP. To access Telnet
programmatically in Perl, the CPAN module
Net::Telnet is a good
Python is an exception--it has very good built-in support for both socket
and various Internet protocols, including POP3, SMTP, FTP, Telnet, and Gopher.
The Python core distribution contains many networking modules, such as
components in a high-level programming language, these modules encapsulate the
complexity of the underlying protocols and are very convenient to use. Twisted is
also another powerful networking framework, which, unlike the core
networking modules, adopts an asynchronous approach in the networking
This article introduces basic client-side networking using both core Python modules and the Twisted framework. For its example, I will show how to send, receive, and delete emails, and conduct Telnet sessions. I have written two functionally equivalent examples, one using the core modules (mail-core.py) and another using Twisted (mail-twisted.py), with both start, stop, and interact with a server to process emails. These programs work with any standard-compliant SMTP and POP3 servers in sending and retrieving of emails. The starting and stopping of server are specific to the Apache James mail server, which I choose as a local testing server due to its ease of installation and its shutdown procedure in a Telnet session.
The core module
smtplib provides a
SMTP class that
encapsulates the interactions to a SMTP server to send emails. Essentially,
you create an
SMTP instance with the address of the server
sendmail to send the mail(s), and finally close
the SMTP connection by the
def sendMail(host, addr, to, subject, content): import smtplib from email.MIMEText import MIMEText print "Sending mail from", addr, "to", to, "...", server = smtplib.SMTP(host) msg = MIMEText(content) msg["Subject"] = subject msg["From"] = addr msg["To"] = to server.sendmail(addr, [to], msg.as_string()) server.quit() print "done."
sendmail method takes parameters of the sender's address,
receivers's addresses in a list, and the content of the message. Because the
message content should be in the MIME format, use the convenient class
email.MIMEText) to create a text
message. Create a
MIMEText with the message body, specify the
subject, from, and to addresses with the dictionary-like syntax, and take it
as a string when passing to
This testing server accepts and relays emails from anyone, and this is the
default configuration of the Apache James server. Although it is fine for a local
testing server, most, if not all, SMTP servers in the Internet mandate certain
security measures to fight spam. To use a
SMTP object with a
server that requires authentication, invoke the
password) method before sending any message. This method keeps silent on
success and raises an exception when the authentication fails.
Retrieving emails is inherently more complex than sending: it involves
identification of the user, getting the number of messages, and retrieving or
deleting the messages. The
POP3 class in the
module provides methods to do this:
def display(host, user, passwd, deletion = 0): import poplib, email pop3 = poplib.POP3(host) pop3.user(user) pop3.pass_(passwd) num = len(pop3.list()) print user, "has", num, "messages" format = "%-3s %-15s %s" if num > 0: if deletion: print "Deleting", num, "messages", for i in range(1, num+1): pop3.dele(i) print ".", print " done." else: print format % ("Num", "From", "Subject") for i in range(1, num+1): str = string.join(pop3.top(i, 1), "\n") msg = email.message_from_string(str) print format % (i, msg["From"], msg["Subject"]) pop3.quit()
Like the SMTP class, you can create a
POP3 instance by
specifying the mail server. Terminate the POP3 session with the
quit method. Log in to the POP3 account with the methods
pass_ with parameters of the user name and
To get the number of messages in the server, you may use either the
stat method or the
list method. The
stat method returns a tuple of two values: the number of messages
and the total mailbox size. The
list method, used in the example,
returns the message list in the following form:
(response, ['mesg_num octets', ...], octets)
Here the second value is a list of strings, each stating the number and size of a message in the mailbox. The number of messages is the size of this list of strings.
Retrieve a message wholly with the method
or partially with
top(message_index, number_of_lines). In both
methods, the message index is 1-based and returns a message in the
(response, ['line', ...], octets)
Typically, the second value in the results is the most interesting: it is a
list of the lines of the message. The example program needs only to retrieve
the header information, so it gets string containing all header plus one line
of body text with
string.join(pop3.top(i, 1), "\n"), then uses
email.message_from_string to parse the string to build a MIME
message object. From the MIME message, you can fetch the email subject, sender
address, and receiver address(es) via the standard dictionary syntax.
To delete a message in the mailbox, use the method
dele(message_index). It sets the deletion flag of the specified
message and then the server actually does the deletion when you close the POP3
session. If you have a program calling
dele on a message but the
message persists, check that the program actually invokes
The Apache James mail server comes with a batch file to start it. As a
convenient option, the example program can invoke the batch file via a call to
def start(): print "Starting server...", os.system("c:/apps/bin/james.bat") print "done." def stop(host): import telnetlib print "Stoping server...", telnet = telnetlib.Telnet(host, 4555) telnet.read_until("Login id:") telnet.write("root\n") telnet.read_until("Password:") telnet.write("root\n") telnet.read_until("Welcome") telnet.write("shutdown\n") telnet.close() print "done."
Shutting down the server, on the other hand, requires a Telnet session.
Manually, you can do it through a Telnet client program connected to port 4555
of the server. Enter the user name as
root, password as
root, and the command
shutdown, respectively. To
shutdown the server programmatically in Python, use class
from the core module
To establish a Telnet session, create a
Telnet object with the
Telnet server address and port number specified. To interact with the telnet
server, interleave calls to methods
read_until method reads data from the
server until it receives a specified string, while the
method sends a string to the server. Note that the Apache James mail server
expects a trailing carriage return in the string from the Telnet client, so the
"\n" to the end of the parameter to
write. The Telnet session ends with the invocation of method
close on the
Beyond the core Python modules, Twisted is a networking framework in a different style. As demonstrated in the previous example, Python's core networking modules use a procedural approach. To perform a task, your code must invoke a method that holds the thread of execution until the task either completes successfully or fails. Such a method is straightforward to use since it is synchronous and communicates any results of execution to its caller by means of some return value. The code to perform a sequence of tasks simply invokes the appropriate methods one by one, possibly with some structural constructs and checking of return values.
On the other hand, the Twisted framework adopts a different approach of asynchronous invocation. A method call will schedule a task to do in the framework's execution thread, returning control to its caller immediately and before the completion of the task. An event-driven mechanism communicates the results of the execution. The object returned by the asynchronous method call can register success and failure callbacks that will be invoked when the scheduled task completes successfully and fails, respectively. Performing a sequence of tasks is relatively more complex, since you must typically define each task in a method registered as the success callback to the object returned by the previous task's method.
In Twisted, get used to receiving a
twisted.internet.defer) object from an asynchronous method call.
There is also a
DeferredList object which can watch for
asynchronous method calls completing or failing. The engine
twisted.internet) controls the
framework's execution thread. Start it and shut it down with the methods
twisted.mail.smtp) method is
the workhorse for sending mails in Twisted. It takes similar parameters as the
sendmail method of a
smtplib.SMTP object, with the
additional first parameter of the SMTP host name. The return value is a
Deferred object to which you can attach success and failure
def sendMail(host, addr, to, subject, content): from twisted.mail.smtp import sendmail from email.MIMEText import MIMEText print "Sending mail from", addr, "to", to, "..." msg = MIMEText(content) msg["Subject"] = subject msg["From"] = addr msg["To"] = to return sendmail(host, addr, [to], msg.as_string()) def main(): ... elif "s" == sys.argv: print len(addrs), "messages to be sent." dlist =  for addr in addrs: toaddr = user + "@" + host text = "Test mail: " + addr + " to " + toaddr dlist.append( sendMail(host, addr, toaddr, text, text) ) DeferredList(dlist).addBoth(lambda _: reactor.stop()) reactor.run()
To send more than one mails with
sendmail, you don't need to
attach the callbacks to each of the returned
Deferreds in a list to create a
object. The code then attaches a callback to that single
DeferredList object via its
addBoth method. It will
fire when all the
sendmail actions succeed or any of them fails.
The callback simply stops the Twisted's execution thread by
reactor.stop(). Note that the tasks scheduled or registered by
addBoth are not executing until the call
reactor.run(), which starts Twisted's execution thread.
Programming a POP3 client to retrieve mails in the Twisted framework is more
complex and takes more code. The logic to retrieve mails is all in a class that
Due to the event-driven nature of Twisted, it's easier to define a method for
each step, register the success, and failure callbacks to the method to enter
the next step and handle any error, respectively.
from twisted.internet.protocol import ClientCreator from twisted.mail.pop3client import POP3Client class MyPOP3Client(POP3Client): def serverGreeting(self, msg): POP3Client.serverGreeting(self, msg) self.login(self.myuser, self.mypass).addCallbacks( self.do_stat, errorHandler) def do_stat(self, result): self.stat().addCallbacks(self.do_retrieve, errorHandler)
In the class
MyPOP3Client, the first step to get mails is the
serverGreeting method, which Twisted will invoke when the client
starts. This method invokes the superclass's
then logs in to the POP3 server with a user name and password. The
login method returns a
Deferred object, invoking the
addCallbacks method to register the
(called upon successful login), and the
(called on login error).
do_stat method invokes
stat method to perform a POP3 STAT command, and registers the next
do_retrieve. Because the call to method
is asynchronous, it cannot return its results to the caller with return values.
Instead, it passes the results as arguments to the success callback registered
stat method. The second parameter to the
do_retrieve method is a list, of which the first element is the
number of messages in the POP3 account.
def do_retrieve(self, stats): self.format = "%-3s %-15s %s" self.num_messages = stats self.cur_message = 0 print self.myuser, "has", self.num_messages, "messages" if self.num_messages > 0: if deletion: print "Deleting", self.num_messages, "messages", self.delete(0).addCallbacks(self.do_delete_msg, errorHandler) else: print self.format % ("Num", "From", "Subject") self.retrieve(0).addCallbacks(self.do_retrieve_msg, errorHandler) else: reactor.stop() def do_retrieve_msg(self, lines): msg = email.message_from_string("\r\n".join(lines)) print self.format % (self.cur_message, msg["From"], msg["Subject"]) self.cur_message += 1 if (self.cur_message < self.num_messages): self.retrieve(self.cur_message).addCallbacks( self.do_retrieve_msg, errorHandler) else: reactor.stop()
If there is no message in the mailbox, the code calls
reactor.stop to tell Twisted to shutdown. Otherwise, it invokes
retrieve(0) to get the first message. Its success callback,
do_retrieve_msg, handles the message by displaying its summary,
and then retrieves the next message. Because the method
do_retrieve_msg gets invoked for all subsequent messages, the code
uses an instance variable,
cur_message, to keep track of the
current message number and to determine when it has handled all messages. When
it has processed everything, it stops the Twisted main loop.
Because the logic of mail retrieval is similar to deletion, both features
are in the same class,
MyPOP3Client. The instance variable
deletion denotes the current mode of working. You can see its
__init__, along with the user name and password.
More interesting is the setting of
allowInsecureLogin to true,
which allows login to a server without authentication challenge non-encrypted
def __init__(self): self.myuser = user self.mypass = passwd self.deletion = deletion self.allowInsecureLogin = True def do_delete_msg(self, str): print ".", self.cur_message += 1 if (self.cur_message < self.num_messages): self.delete(self.cur_message).addCallbacks( self.do_delete_msg, errorHandler) else: print " done." q = self.quit() q.addCallbacks(lambda _: reactor.stop(), errorHandler)
To delete a mail, call the
delete method of class
POP3Client. Similar to the core
poplib module, this
method just marks the mail for deletion, and the actual deletion occurs when
you send the POP3 command
QUIT to the server, as with the
quit method. Finally, Twisted's execution thread stops when the
quitting action completes.
pop3 = ClientCreator(reactor, MyPOP3Client) d = pop3.connectTCP(host, 110) reactor.run()
With the implementation of the desired mail handling in class
MyPOP3Client, you can launch the client. With its descriptive
name, the class
twisted.internet.protocol) provides a convenient way to start a
communication client. This code passes the
reactor and the
MyPOP3Client class to create a
begins the mail retrieval by calling
connectTCP with the specified
server and port number. Twisted's execution loop then kicks off by
do_retrieve_msg repeatedly for all messages is
conceptually tedious and lengthy, when compared to the
DeferredList mechanism which keeps track of several actions and
gets notified when all actions complete, as in the case of sending mails.
However, collecting the
Deferreds of multiple calls to
POP3Client in a
simply does not work in Twisted (Versions 2.2.0 and 2.4.0). The success
callback never gets invoked (see mail-twisted.py).
Twisted can power a Telnet client in a way similar to, but simpler than, the
POP3 client. the Telnet conversation logic goes in a subclass of
def stop(host): from twisted.internet.protocol import ClientCreator from twisted.conch.telnet import Telnet class MyTelnet(Telnet): def dataReceived(self, data): if "Login id:" in data: self._write("root\n") elif "Password:" in data: self._write("root\n") elif "Welcome" in data: d = self._write("shutdown\n") def connectionLost(self, reason): reactor.stop() print "done." mytelnet = ClientCreator(reactor, MyTelnet) d = mytelnet.connectTCP(host, 4555) reactor.run()
MyTelnet overrides two methods of class
Telnet. The first method,
dataReceived, is called
when data arrives at the client. It checks the data received and calls the
_write method to send the user name, password, or the shutdown
command accordingly to the server. The second method is
connectionLost, which Twisted calls when the server closes the
telnet session. In that case, the program simply terminates the Twisted
execution loop. The Telnet client starts by using the
class, connected to port 4555 of the James mail server.
The two functionally equivalent programs, one using Python core modules and the other using the Twisted framework, significantly differ from each other in terms of programming style and the amount of code. Then when should you use either of the two options?
For basic programs such as the command-line client of this example, the Python core networking modules are more desirable due to the simplicity and performance advantages. However, most real-world networking programs are very complex, and Twisted's asynchronous programming model is more effective. For example, BitTorrent, the popular peer-to-peer file sharing client that performs massive parallel downloading of data chunks from different sources, uses Twisted. Twisted also works well in programs with graphical user interface (GUI), because its asynchronous nature fits more seamlessly with the event-driven programming models of modern GUI frameworks. In fact, Twisted has integration with popular GUI frameworks including PyGTK, Qt, Tkinter, WxPython, and Win32.
The other area where Twisted shines is in server programming. A typical network server uses multithreading so that it can handle multiple clients concurrently. The asynchronous mechanism of Twisted alleviates the creation and handling of threads by server programs. In addition, Twisted provides several protocols on which to build new networking services, enabling rapid development of complex servers. One such project is Quotient, which adopts Twisted to build a multiprotocol messaging server that supports a variety of protocols and services including SMTP, POP3, IMAP, webmail, and SIP.
Kendrew Lau is a consultant in Hong Kong, with focus on Java, Linux, and other OSS technologies.
Return to the Python DevCenter.
Copyright © 2009 O'Reilly Media, Inc.