76
1 INTERNET CLIENT PROGRAMMING USING PYTHON CUAUHTEMOC CARBAJAL ITESM CEM APRIL 06, 2013
1
INTERNET CLIENT PROGRAMMING USING PYTHON
C U A U H T E M O C C A R B A J A LI T E S M C E M
A P R I L 0 6 , 2 0 1 3
INTRODUCTION
• In the previous lecture we took a look at low-level networking communication protocols using sockets. • This type of networking is at the heart of most of the
client/server protocols that exist on the Internet today (FTP, NNTP, SMTP, POP3, IMAP, etc.). • These protocols work in a way much like the
client/server examples of the previous lecture. • The only difference is that now we have taken
lower-level protocols such as TCP/IP and created newer, more specific protocols on top of them to implement these higher-level services.
2
NETWORK CLIENT LIBRARIES
• Python offers a rich variety of network client code• Email: smtplib, poplib, imaplib
• rfc822 and email modules handle content
• File transfer: ftplib
• Web: httplib, urllib• More on these later
• Network news: nntplib
• Telnet: telnetlib
3
GENERAL CLIENT STRATEGY
• Library usually defines an object class• Create an instance of the object to interact with
the server• Call the instance's methods to request particular
interactions
4
USING SMTPLIB
• s = smtplib.SMTP([host[, port]])• Create SMTP object with given connection
parameters• r = s.sendmail(from, to, msg
[, mopts[, ropts]])
• from : sender address• to : list of recipient addresses• msg : RFC822-formatted message (including all
necessary headers)• mopts, ropts : ESMTP option lists
5
SMTP EXAMPLE
import smtplib, socket
frad = "[email protected]"
toads = ["[email protected]",
msg = """To: Various recipients
From: Steve Holden <[email protected]>
Hello. This is an RFC822 mail message.
"""
6
SMTP EXAMPLE (CONT)
try:server = smtplib.SMTP('10.0.0.1')result = server.sendmail(frad, toads, msg)server.quit()if result:
for r in result.keys():print "Error sending to", rrt = result[r]print "Code", rt[0], ":", rt[1]
else:print "Sent without errors"
except smtplib.SMTPException, arg:print "Server could not send mail", arg
7
SENDING EMAILS FROM THE RASPBERRY PI
• There are many cases when it can be very useful to be able to send emails from the Raspberry Pi to arbitrary recipients. • This is not the same as having a real
MTA running on the Pi (like Sendmail, Postfix, Exim, QMail, etc.), which can also receive and store emails.
• In the following we are only going to cover the possibility of sending emails, not receiving. • In most cases this is enough, as
people tend to use GMail, Yahoo! Mail and other major email service providers and they store their emails on the servers of these providers.
• Still, sending out emails from the Raspberry Pi can come in handy in many situations.
8
SENDING EMAILS FROM THE RASPBERRY PI (CONT)
• You could have some sensors connected to the GPIO pins of the Pi and you could program the Pi to send you an email • when the temperature in the room rises above or drops below certain
threshold values, • when a gas sensor registers unwanted gas leaks or • when the measured voltage of a monitored battery becomes too low.
• You could also have your Pi send you daily or weekly emails with summarized system data.
• Or maybe you could connect a webcam to the Raspberry Pi and set up some motion detection software, which would send you an email as soon as it detects motion in a given area of your house.
• Maybe you are hosting a WordPress website on your Raspberry Pi and you would like to provide your readers with the possibility to subscribe to the posts.
• This all means that the Pi needs to be able to send out emails.9
PYTHON SENDING EMAIL USING SMTP
• Simple Mail Transfer Protocol (SMTP) is a protocol which handles sending e-mail and routing e-mail between mail servers.• Python provides smtplib module which defines an
SMTP client session object that can be used to send mail to any Internet machine with an SMTP or ESMTP listener daemon.• For normal use, you should only require the
initialization/connect, sendmail(), and quit() methods. An example is included below.
10
PYTHON SENDING EMAIL USING SMTP
• Here is a simple syntax to create one SMTP object which can later be used to send an email:
• Parameters:• host: This is the host running your SMTP server. You can specify
the host’s IP address or a domain name like tutorialspoint.com. This is optional argument.
• port: If you are providing host argument then you need to specify a port where SMTP server is listening. Usually this port would be 25.
• local_hostname: If your SMTP server is running on your local machine then you can specify just localhost as of this option.
import smtplib
smtpObj = smtplib.SMTP( [host [, port [, local_hostname]]] )
11
SENDMAIL
SMTP.sendmail(from_addr, to_addrs, msg[, mail_options, rcpt_options])
• Arguments:• an RFC 822 from-address string, • a list of RFC 822 to-address strings (a bare string will be treated
as a list with 1 address), • and a message string.
• The caller may pass a list of ESMTP options (such as 8bitmime) to be used in MAIL FROM commands as mail_options. • ESMTP options (such as DSN commands) that should be used
with all RCPT commands can be passed as rcpt_options. (If you need to use different ESMTP options to different recipients you have to use the low-level methods such as mail(), rcpt() and data() to send the message.)
12
EHLO
SMTP.ehlo([hostname])
• Identify yourself to an ESMTP server using EHLO. • The hostname argument defaults to the fully qualified domain
name of the local host. Examine the response for ESMTP option and store them for use by has_extn(). Also sets several informational attributes: the message returned by the server is stored as the ehlo_resp attribute, does_esmtp is set to true or false depending on whether the server supports ESMTP, and esmtp_features will be a dictionary containing the names of the SMTP service extensions this server supports, and their parameters (if any).
• Unless you wish to use has_extn() before sending mail, it should not be necessary to call this method explicitly. It will be implicitly called by sendmail() when necessary.
13
STARTTLS
SMTP.starttls([keyfile[, certfile]])
• Put the SMTP connection in TLS (Transport Layer Security) mode. All SMTP commands that follow will be encrypted. You should then call ehlo() again.
• If keyfile and certfile are provided, these are passed to the socket module’s ssl() function.
• If there has been no previous EHLO or HELO command this session, this method tries ESMTP EHLO first.
14
STMPLIB EXAMPLE
import smtplib
to = '[email protected]'gmail_user = '[email protected]'gmail_pwd = '28/Julio/1955'smtpserver = smtplib.SMTP("smtp.gmail.com",587)smtpserver.ehlo()smtpserver.starttls()smtpserver.ehlosmtpserver.login(gmail_user, gmail_pwd)header = 'To:' + to + '\n' + 'From: ' + gmail_user + '\n' + 'Subject:testing \n'print headermsg = header + '\n this is test msg from my RPi \n\n'smtpserver.sendmail(gmail_user, to, msg)print 'done!'smtpserver.close()
email-example2.py 15
PIR MOVEMENT DETECTOR
• A PIR movement detector and an Arduino are used to have the Arduino communicate with a Python program running on a computer (or RPi) to send an email whenever movement is detected by the sensor.
http://learn.adafruit.com/arduino-lesson-17-email-sending-movement-detector 16
ARDUINO SETUP
17
ARDUINO CODE
int pirPin = 7;int minSecsBetweenEmails = 60; // 1 minlong lastSend = -minSecsBetweenEmails * 1000l;
void setup(){
pinMode(pirPin, INPUT);Serial.begin(9600);
}
18
ARDUINO CODE (CONT)
void loop() {long now = millis();if (digitalRead(pirPin) == HIGH){if (now > (lastSend + minSecsBetweenEmails * 1000l)){Serial.println("MOVEMENT");lastSend = now;
}else{Serial.println("Too soon");
}}delay(500);
}19
PYTHON CODE
import timeimport serialimport smtplibTO = '[email protected]'GMAIL_USER = '[email protected]'GMAIL_PASS = 'putyourpasswordhere'SUBJECT = 'Intrusion!!'TEXT = 'Your PIR sensor detected movement' ser = serial.Serial('/dev/ttyACM0', 9600)
20
PYTHON CODE (CONT)
def send_email():print("Sending Email")smtpserver = smtplib.SMTP("smtp.gmail.com",587)smtpserver.ehlo()smtpserver.starttls()smtpserver.ehlosmtpserver.login(GMAIL_USER, GMAIL_PASS)header = 'To:' + TO + '\n' + 'From: ' + GMAIL_USERheader = header + '\n' + 'Subject:' + SUBJECT + '\n‘print headermsg = header + '\n' + TEXT + ' \n\n‘smtpserver.sendmail(GMAIL_USER, TO, msg)smtpserver.close()
21
PYTHON CODE (CONT)
while True:message = ser.readline()print(message)if message[0] == 'M' :
send_email()time.sleep(0.5)
22
POST OFFICE PROTOCOL
• POP is an application-layer Internet standard protocol used to “pop” (copy) messages from a central mail server to your local computer.
• Although most POP clients have an option to leave mail on server after download, e-mail clients using POP generally connect, retrieve all messages, store them on the user's PC as new messages, delete them from the server, and then disconnect.
• A POP3 server listens on well-known port 110. • Encrypted communication for POP3 is either requested
after protocol initiation, using the STLS command, if supported, or by POP3S, which connects to the server using Transport Layer Security (TLS) or Secure Sockets Layer (SSL) on well-known TCP port 995.
23
POP3 PROTOCOL CLIENT
• This module defines a class, POP3 which encapsulates a connection to a POP3 server and implements the protocol as defined in RFC 1725.
• The POP3 class supports both the minimal and optional command sets.
• Additionally, this module provides a class POP3_SSL, which provides support for connecting to POP3 servers that use SSL as an underlying protocol layer.
• Note that POP3, though widely supported, is obsolescent.
• The implementation quality of POP3 servers varies widely, and too many are quite poor.
• If your mailserver supports IMAP, you would be better off using the imaplib.IMAP4 class, as IMAP servers tend to be better implemented.
24
USING POPLIB
• p = poplib.POP3(host[, port])• Creates a POP object with given connection
parameters• p.user(username)• Provide username to server
• p.pass_(password)• Provide password to server
• p.stat()• Returns (# of msgs, # of bytes)
25
USING POPLIB (CONT)
• p.retr(msgnum)• returns (response, linelist, bytecount)
• p.dele(msgnum)• Marks the given message for deletion
• p.quit()• Terminate the connection• Server actions pending deletes and unlocks the
mailbox
26
POPLIB EXAMPLE
import poplib, rfc822, sys, StringIOSRVR = 'pop.googlemail.com'PORT = '995'USER = "[email protected]"PASS = "28/Julio/1955"try:
p = poplib.POP3_SSL(SRVR, PORT)except:
print "Can't contact %s" % (SRVR, )sys.exit(-1)
try:print p.user(USER)print p.pass_(PASS)
except:print "Authentication failure"sys.exit(-2)
27
POPLIB EXAMPLE (CONT)
msglst = p.list()[1]for m in msglst:
mno, size = m.split()lines = p.retr(mno)[1]print "----- Message %s" % (mno, )file = StringIO.StringIO("\r\n".join(lines)) msg = rfc822.Message(file)body = file.readlines()addrs = msg.getaddrlist("to")for rcpt, addr in addrs:
print "%-15s %s" % (rcpt, addr)print len(body), "lines in message body"
print "-----"p.quit()
pop-example.py (C3PO)
28
FTP
• The File Transfer Protocol (FTP) was once among the most widely used protocols on the Internet, invoked whenever a user wanted to transfer files between Internet-connected computers.
• There were four primary activities that it once powered.1. File download: documents, source code, media like images or
movies• Lists of “anonymous” FTP servers that allowed public access were circulated• You logged into them with the username “anonymous” or “ftp,” and then—
out of politeness, so they would know who was using their bandwidth—you typed your e-mail address as the password.
• FTP was always the protocol of choice when files needed to be moved between computer accounts, since trying to transfer large files with Telnet clients was often a dicey proposition.
2. Anonymous upload. • Many organizations wanted outsiders to be able to submit documents or
files, and their solution was to set up FTP servers that allowed files to be written into a directory whose contents could not, then, be listed back again. That way, users could not see (and hopefully could not guess!) the names of the files that other users had just submitted and get to them before the site administrators did.
29
FTP (CONT)
3. To support the synchronization of entire trees of files between computer accounts.
• By using a client that provided for “recursive” FTP operations, users could push entire directory trees from one of their accounts to another, and server administrators could clone or install new services without having to re-build them from scratch on a new machine.
• When using FTP like this, users were generally not aware of how the actual protocol worked, or of the many separate commands needed to transfer so many different files: instead, they hit a button and a large batch operation would run and then complete.
4. Interactive, full-fledged file management• The early FTP clients presented a command-line prompt that felt something
like a Unix shell account itself, and—as we shall see—the protocol borrows from shell accounts both the idea of a “current working directory” and of a cd command to move from one directory to another. Later clients mimicked the idea of a Mac-like interface, with folders and files drawn on the computer screen.
• But in either case, in the activity of filesystem browsing the full capabilities of FTP finally came into play: it supported not only the operations of listing directories and uploading and downloading files, but of creating and deleting directories, adjusting file permissions, and re-naming files.
30
FTP ISSUES
• Lack of security: • not only files, but usernames and passwords are sent completely in the
clear and can be viewed by anyone observing network traffic.• Long-running connections:• FTP user tends to make a connection, choose a working directory, and do
several operations all over the same network connection. Modern Internet services, with millions of users, prefer protocols like HTTP that consist of short, completely self-contained requests, instead of long-running FTP connections that require the server to remember things like a current working directory.
• Filesystem security:• The early FTP servers, instead of showing users just a sliver of the host
filesystem that the owner wanted exposed, tended to simply expose the entire filesystem, letting users cd to / and snoop around to see how the system was configured. True, you could run the server under a separate ftp user and try to deny that user access to as many files as possible; but many areas of the Unix filesystem need to be publicly readable simply so that normal users can use the programs there.
• While servers were eventually written that exposed only part of the host filesystem, this more or less violated the original intention: that an FTP session would look like a Telnet command-line prompt, even down to the fact that full pathnames were used that started at the filesystem root.
31
WHAT TO USE INSTEAD OF FTP
• Today, there are better alternatives than the FTP protocol for pretty much anything you could want to do with it. • You will still occasionally see URLs that start with ftp:,
but they are becoming quite rare. • Use this protocol either because you have a legacy
need to speak FTP from your Python program, or because you want to learn more about file transfer protocols in general and FTP is a good, historical place to start.
32
SO WHAT ARE THE ALTERNATIVES?
1. For file download, HTTP is the standard protocol on today’s Internet, protected with SSL when necessary for security. Instead of exposing system-specific file name conventions like FTP, HTTP supports system-independent URLs.
2. Anonymous upload is a bit less standard, but the general tendency is to use a form on a web page that instructs the browser to use an HTTP POST operation to transmit the file that the user selects.
3. File synchronization has improved immeasurably since the days when a recursive FTP file copy was the only common way to get files to another computer. Instead of wastefully copying every file, modern commands like rsync or rdist efficiently compare files at both ends of the connection and copy only the ones that are new or have changed.
4. Full filesystem access is actually the one area where FTP can still commonly be found on today’s Internet: thousands of cut-rate ISPs continue to support FTP, despite its insecurity, as the means by which users copy their media and (typically) PHP source code into their web account. A much better alternative today is for service providers to support SFTP instead. 33
FTP CLIENTS AND SERVERS ON THE INTERNET
• The protocol works as follows:• Client contacts the FTP server on the remote host• Client logs in with username and password (or anonymous
and e-mail address)• Client performs various file transfers or information requests• Client completes the transaction by logging out of the
remote host and FTP server
34The client and server communicate using the FTP protocol on the command or control port data; is transferred using the data port.
USING FTPLIB
• f = ftplib.FTP(host[,user[,passwd[,acct]]])• Creates an FTP object
• f.dir(directory)
• Send directory listing to standard output
• f.cwd(directory)
• Change to given directory• f.mkd(directory)
• Create directory on server• f.pwd()
• Returns current directory on server
35
USING FTPLIB (CONTINUED)
• retrbinary(command, callback[,maxblocksize[, rest]])
• Retrieve a file in binary mode• command - an FTP command• E.g. "RETR myfile.dat"
• callback - processes each block• maxblocksize – how much data per
block• rest – restart position
36
USING FTPLIB (CONTINUED)
• f.retrlines(command[, callback])
• Retrieves a file in text mode• command - an FTP command• E.g. "RETR myfile.txt"
• callback - processes each line as an argument• Default callback prints line to standard
output• f.storlines(command, file)
• Sends content of file line-by-line• f.storbinary(command, file, blocksize)
• Sends content of file block-by-block
37
FTPLIB CLASSES
class ftplib.FTP([host[, user[, passwd[, acct[, timeout]]]]])• Return a new instance of the FTP class. When host is given, the
method call connect(host) is made. When user is given, additionally the method call login(user, passwd, acct) is made (where passwd and acct default to the empty string when not given). The optional timeout parameter specifies a timeout in seconds for blocking operations like the connection attempt (if is not specified, the global default timeout setting will be used).
class ftplib.FTP_TLS([host[, user[, passwd[, acct[, keyfile[, certfile[, timeout]]]]]]])• A FTP subclass which adds TLS support to FTP as described in RFC
4217. Connect as usual to port 21 implicitly securing the FTP control connection before authenticating. Securing the data connection requires the user to explicitly ask for it by calling the prot_p() method. keyfile and certfile are optional – they can contain a PEM formatted private key and certificate chain file name for the SSL connection.
38
EXAMPLE USING FTP
>>> from ftplib import FTP>>> f = FTP('ftp.ibiblio.org')>>> print "Welcome:", f.getwelcome()Welcome: 220 ProFTPD Server>>> f.login()"230-\n Welcome to ftp.ibiblio.org......Anonymous access granted, restrictions apply">>> print "Current working directory:", f.pwd()Current working directory: />>> f.quit
39
DOWNLOADING AN ASCII FILE
#!/usr/bin/env python# ASCII download – ftp-text_example.py# Downloads README from remote and writes it to disk.import osfrom ftplib import FTPif os.path.exists('README'):raise IOError('refusing to overwrite your README file')
def writeline(data):fd.write(data)fd.write(os.linesep)
f = FTP('ftp.kernel.org')f.login()f.cwd('/pub/linux/kernel')fd = open('README', 'w')f.retrlines('RETR README', writeline)fd.close()f.quit() ftp-text_example.py C3PO
40
EXAMPLE USING FTP
>>> from ftplib import FTP >>> ftp = FTP('ftp.kernel.org') # connect to host, default port>>> ftp.login() # user anonymous, passwd anonymous@ >>> f.cwd('/pub')>>> ftp.retrlines('LIST') # list directory contents -r--r--r-- 1 ftp ftp 578 Mar 18 2003 README_ABOUT_BZ2_FILESdrwxr-xr-x 6 ftp ftp 4096 Dec 01 2011 distdrwxr-xr-x 13 ftp ftp 4096 Nov 16 2011 linuxdrwxr-xr-x 3 ftp ftp 4096 Sep 23 2008 mediadrwxr-xr-x 17 ftp ftp 4096 Jun 06 2012 scmdrwxr-xr-x 2 ftp ftp 4096 Dec 01 2011 sitedrwxr-xr-x 13 ftp ftp 4096 Nov 27 2011 softwaredrwxr-xr-x 3 ftp ftp 4096 Apr 30 2008 tools'226 Directory send OK.‘>>> ftp.retrbinary('RETR README_ABOUT_BZ2_FILES',open('README','wb').write) '226 Transfer complete.' >>> ftp.quit()
41
EXAMPLE USING FTP_TLS
>>> from ftplib import FTP_TLS>>> ftps = FTP_TLS('ftp.python.org') >>> ftps.login() # login anonymously before securing control channel >>> ftps.prot_p() # switch to secure data connection >>> ftps.retrlines('LIST') # list directory content securely total 9 drwxr-xr-x 8 root wheel 1024 Jan 3 1994 . drwxr-xr-x 8 root wheel 1024 Jan 3 1994 .. drwxr-xr-x 2 root wheel 1024 Jan 3 1994 bin drwxr-xr-x 2 root wheel 1024 Jan 3 1994 etcd-wxrwxr-x 2 ftp wheel 1024 Sep 5 13:43 incoming drwxr-xr-x 2 root wheel 1024 Nov 17 1993 lib drwxr-xr-x 6 1094 wheel 1024 Sep 13 19:07 pub drwxr-xr-x 3 root wheel 1024 Jan 3 1994 usr-rw-r--r-- 1 root root 312 Aug 1 1994 welcome.msg '226 Transfer complete.' >>> ftps.quit()
42
ABBREVIATED FTPLIB EXAMPLE
class Writer:
def __init__(self, file):
self.f = open(file, "w")
def __call__(self, data):
self.f.write(data)
self.f.write('\n')
print data
FILENAME = "AutoIndent.py"
writer = Writer(FILENAME)
import ftplib
ftp = ftplib.FTP('127.0.0.1', 'book', 'bookpw')
ftp.retrlines("RETR %s" % FILENAME, writer)
43
HTTP AND HTML LIBRARIES
• Python applications are often web-based• htmllib, HTMLParser – HTML parsing• httplib – HTTP protocol client• urllib, urllib2 – multiprotocol client• SimpleHTTPServer, CGIHTTPServer –SocketServer-based servers
• cgi, cgitb – CGI scripting assistance• Various web samples also available
44
USING URLLIB
• f = urllib.urlopen(URL)• Create file-like object that allows you to read the
identified resource• urlretrieve(url[, filename[,
reporthook[, data]]])
• Reads the identified resource and store it as a local file• See documentation for further details
• This is very convenient for interactive use
45
INTERACTIVE URLLIB SESSION
• Useful for testing & quick interactions
>>> import urllib>>> f = urllib.urlopen("http://www.python.org/")>>> page = f.read()>>> len(page)21801>>> h=f.info()>>> h.getheader("Server")'Apache/2.2.16 (Debian)'>>> h.getheaders("Date")['Mon, 01 Apr 2013 20:32:46 GMT']>>> h.type'text/html'
46
USING URLLIB2
• urllib has limitations - difficult to• Include authentication• Handle new protocols/schemes• Must subclass urllib.FancyURLOpener and bind
an instance to urllib._urlopener
• urllib2 is intended to be more flexible• The price is added complexity• Many applications don't need the complexity
47
URLLIB2.REQUEST CLASS
• Instance can be passed instead of a URL to the urllib2.urlopen() function
• r = Request(url, data=None, headers={})• r.add_header(key, value)• Can only add one header with a given key
• r.set_proxy(host, scheme )• Sets the request to use a given proxy to access the
given scheme• r.add_data(data)• Forces use of POST rather than GET• Requires http scheme
48
SERVING HTTP
• Several related modules:• BaseHTTPServer defines• HTTPServer class• BaseHTTPRequestHandler class
• SimpleHTTPServer defines• SimpleHTTPRequestHandler class
• CGIHTTPServer defines• CGIHTTPRequestHandler class
• All request handlers use the standard HTTPServer.BaseHTTPRequestHandler
49
THE SIMPLEST WEB SERVER …
import CGIHTTPServer, BaseHTTPServer
httpd = BaseHTTPServer.HTTPServer(('', 8888),
CGIHTTPServer.CGIHTTPRequestHandler)
httpd.serve_forever()
• Uses the basic HTTP server class• Request handler methods implement
the HTTP PUT/GET/HEAD requests• Yes, this really works!
50
WHAT IS CGI ?
• The Common Gateway Interface, or CGI, is a set of standards that define how information is exchanged between the web server and a custom script.
• The CGI specs are currently maintained by the NCSA and NCSA defines CGI is as follows:
• The Common Gateway Interface, or CGI, is a standard for external gateway programs to interface with information servers such as HTTP servers.
• The current version is CGI/1.1 and CGI/1.2 is under progress.
51
PYTHON CGI PROGRAMMING
52
WEB BROWSING
• To understand the concept of CGI, lets see what happens when we click a hyper link to browse a particular web page or URL.• Your browser contacts the HTTP web server and demand for
the URL ie. filename.• Web Server will parse the URL and will look for the filename
in if it finds that file then sends back to the browser otherwise sends an error message indicating that you have requested a wrong file.
• Web browser takes response from web server and displays either the received file or error message.
53
WEB BROWSING (CONT)
• However, it is possible to set up the HTTP server so that whenever a file in a certain directory is requested that file is not sent back; instead it is executed as a program, and whatever that program outputs is sent back for your browser to display.
• This function is called the Common Gateway Interface or CGI and the programs are called CGI scripts. These CGI programs can be a Python Script, PERL Script, Shell Script, C or C++ program etc.
CGI Architecture Diagram
54
WEB SERVER SUPPORT & CONFIGURATION
• Before you proceed with CGI Programming, make sure that your Web Server supports CGI and it is configured to handle CGI Programs. All the CGI Programs to be executed by the HTTP server are kept in a pre-configured directory. This directory is called CGI Directory and by convention it is named as /var/www/cgi-bin. By convention CGI files will have extention as .cgi but you can keep your files with python extension .py as well.
• By default, the Linux server is configured to run only the scripts in the cgi-bin directory in /var/www. If you want to specify any other directory to run your CGI scripts, comment the following lines in the httpd.conf file:<Directory "/var/www/cgi-bin">
AllowOverride NoneOptions ExecCGIOrder allow,denyAllow from all
</Directory>
<Directory "/var/www/cgi-bin">Options All</Directory>
• Here I assumed that you have Web Server up and running successfully and you are able to run any other CGI program like Perl or Sheel etc.
55
FIRST CGI PROGRAM
• Here is a simple link which is linked to a CGI script called hello.py. This file is being kept in /var/www/cgi-bin directory and it has following content. Before running your CGI program make sure you have chage mode of file using chmod 755 hello.py UNIX command to make file executable.#!/usr/bin/pythonprint "Content-type:text/html\r\n\r\n"print '<html>'print '<head>'print '<title>Hello Word - First CGI Program</title>'print '</head>'print '<body>'print '<h2>Hello Word! This is my first CGI program</h2>'print '</body>'print '</html>'
• If you click hello.py then this produces following output:Hello Word! This is my first CGI program
56
57
STANDARD CGI SUPPORT
• cgi module provides input handling• Recent (2.2) changes make things easier• cgitb module traps errors• Easier to diagnose problems
• Gives complete Python traceback
• Situation previously complicated by differences in multi-valued form inputs• Had to check, and program different actions (string vs list)
• Python is excellent for producing HTML!
58
THE CGI.FIELDSTORAGE CLASS
• Makes web client's input accessible• Consumes input, so only instantiate once!• Handles method GET or POST• Optional argument retains blank values
• f.getfirst(name, default=None)• Returns first (only) input value with given name
• f.getlist(name)• Returns a list of all values with given name
59
ERROR HANDLING
• Should use for all CGI scripts!import cgitb; cgitb.enable()
• Traps any errors, producing legible trace
60
SAMPLE CGI SCRIPT
#!/usr/bin/pythonimport cgi, cgitb; cgitb.enable()fields = ["subnum", "reviewer", "comments"]
form = cgi.FieldStorage()vlist = []for f in fields:
vlist.append("%s=%s" % (f, form.getfirst(f)))
print pgtmpl = """Content-Type: text/html
<html><body><head><title>Hello!</title></head>%s</body></html>""" % "<br>".join(vlist)
Support module for Common Gateway Interface (CGI) scripts.
Store a sequence of fields, reading multipart/form-data.
This method always returns only one value associated with form field name.
Return a string which is the concatenation of the strings in the iterable iterable.
61
COURSE SUMMARY
• Reviewed principles of networking• Contrasted TCP and UDP features• Shown how Python programs access networking
functionality• Given examples of client and server program
structures• Demonstrated some Python network libraries• Given pointers to other network functionality
62
SOCKET PROGRAMMING EXAMPLE
• The socket module• Provides access to low-level network programming functions.• Example: A server that returns the current time
• Notes:• Socket first opened by server is not the same one used to exchange data.• Instead, the accept() function returns a new socket for this (’client’ above).• listen() specifies max number of pending connections.
# Time server programfrom socket import *import times = socket(AF_INET, SOCK_STREAM) # Create TCP sockets.bind(("",8888)) # Bind to port 8888s.listen(5) # Start listeningwhile 1:client,addr = s.accept() # Wait for a connectionprint "Got a connection from ", addrclient.send(time.ctime(time.time())) # Send time backclient.close()
63
SOCKET PROGRAMMING EXAMPLE (CONT)
• Client Program• Connect to time server and get current time
• Key Points• Once connection is established, server/client communicate
using send() and recv().• Aside from connection process, it’s relatively straightforward.• Of course, the devil is in the details.• And are there ever a LOT of details.
# Time client programfrom socket import *s = socket(AF_INET,SOCK_STREAM) # Create TCP sockets.connect((“127.0.0.1",8888)) # Connect to servertm = s.recv(1024) # Receive up to 1024 bytess.close() # Close connectionprint "The time is", tm
64
THE SOCKET MODULE
• This is used for all low-level networking• Creation and manipulation of sockets• General purpose network functions (hostnames, data
conversion, etc...)• A direct translation of the BSD socket interface.
• Utility Functions
• Comments• Network order for integers is big-endian.• Host order may be little-endian or big-endian (depends on the
machine).
socket.gethostbyname(hostname) # Get IP address for a hostsocket.gethostname() # Name of local machinesocket.ntohl(x) # Convert 32-bit integer to host ordersocket.ntohs(x) # Convert 16-bit integer to host ordersocket.htonl(x) # Convert 32-bit integer to network ordersocket.htons(x) # Convert 16-bit integer to network order
65
THE SOCKET MODULE (CONT)
• The socket(family, type, proto) function• Creates a new socket object.• family is usually set to AF_INET• type is one of:• SOCK_STREAM Stream socket (TCP)• SOCK_DGRAM Datagram socket (UDP)
• SOCK_RAW Raw socket• proto is usually only used with raw sockets• IPPROTO_ICMP• IPPROTO_IP• IPPROTO_RAW• IPPROTO_TCP• IPPROTO_UDP
• Comments• Currently no support for IPv6 (although its on the way).• Raw sockets only available to processes running as root.
66
THE SOCKET MODULE (CONT)
• socket methods• s.accept() # Accept a new connection• s.bind(address) # Bind to an address and port• s.close() # Close the socket• s.connect(address) # Connect to remote socket• s.fileno() # Return integer file descriptor• s.getpeername() # Get name of remote machine• s.getsockname() # Get socket address as (ipaddr,port)• s.getsockopt(...) # Get socket options• s.listen(backlog) # Start listening for connections• s.makefile(mode) # Turn socket into a file object• s.recv(bufsize) # Receive data• s.recvfrom(bufsize) # Receive data (UDP)• s.send(string) # Send data• s.sendto(string, address) # Send packet (UDP)• s.setblocking(flag) # Set blocking or nonblocking mode• s.setsockopt(...) # Set socket options• s.shutdown(how) # Shutdown one or both halves of connection
• Comments• There are a huge variety of configuration/connection options.• You’ll definitely want a good reference at your side.
67
THE SOCKETSERVER MODULE
• Provides a high-level class-based interface to sockets• Encapsulates each protocol in a class (TCPServer, UDPServer, etc.)• Provides a series of handler classes that specify additional server behavior.• To create a network service, need to inherit from both a protocol and handler class.
• Example
• Comments• The module provides a number of specialized server and handler types.• Ex: ForkingTCPServer, ThreadingTCPServer, StreamRequestHandler, etc.
# Simple time serverimport SocketServerimport time# This class actually implements the server functionalityclass TimeHandler(SocketServer.BaseRequestHandler):def handle(self):self.request.send(time.ctime(time.time()))# Create the serverserver = SocketServer.TCPServer(("",8888),TimeHandler)server.serve_forever()
68
COMMON NETWORK PROTOCOLS
• Modules are available for a variety of network protocols• ftplib FTP protocol• smtplib SMTP (mail) protocol• nntplib News• gopherlib Gopher• poplib POP3 mail server• imaplib IMAP4 mail server• telnetlib Telnet protocol• httplib HTTP protocol
• Comments• These modules are built using sockets, but operate on a very
low-level.• Require a good understand of the underlying protocol.• But can be quite powerful if you know exactly what you are
doing.
69
THE HTTPLIB MODULE
• Implements the HTTP 1.0 protocol• Can use to talk to a web server.
• HTTP in two bullets• Client (e.g., a browser) sends a request to the server
GET /index.html HTTP/1.0Connection: Keep-AliveHost: www.python.orgUser-Agent: Mozilla/4.61 [en] (X11; U; SunOS 5.6 sun4u)[blank line]
• Server responds with something like this:HTTP/1.0 200 OKContent-type: text/htmlContent-length: 72883Headers: blah[blank line]Data...
70
THE HTTPLIB MODULE (CONT)
• Making an HTTP connectionimport httplibh = httplib.HTTP("www.python.org")h.putrequest('GET','/index.html')h.putheader('User-Agent','Lame Tutorial Code')h.putheader('Accept','text/html')h.endheaders()errcode,errmsg, headers = h.getreply()f = h.getfile() # Get file object for reading datadata = f.read()f.close()
• Comments• Some understanding of HTTP is required.• Only HTTP/1.0 is currently supported.• Most of the other protocol modules work in a similar manner.
71
THE URLLIB MODULE
• A high-level interface to HTTP and FTP• Provides a file-like object that can be used to connect to
remote serversimport urllibf = urllib.urlopen("http://www.python.org/index.html")data = f.read()f.close()
• Utility functionsurllib.quote(str) # Quotes a string for use in a URLurllib.quote_plus(str) # Also replaces spaces with ’+’urllib.unquote(str) # Opposite of quote()urllib.unquote_plus(str) # Opposite of quote_plus()urllib.urlencode(dict) # Turns a dictionary of key=value
# pairs into a HTTP query-string• Examples
urllib.quote("beazley@cs") # Produces "beazley%40cs"urllib.unquote("%23%21/bin/sh") # Produces "/bin/sh"
72
THE URLPARSE MODULE
Functions for manipulating URLs• URL’s have the following general format• scheme:/netloc/path;parameters?query#fragment
• urlparse(urlstring) - Parses a URL into componentsimport urlparset = urlparse.urlparse("http://www.python.org/index.html")# Produces (’http’,’www.python.org’,’/index.html’,’’,’’,’’)
• urlunparse(tuple) - Turns tuple of components back into a URL stringurl = urlparse.urlunparse(('http','www.python.org','foo.html','bar=spam',''))# Produces "http://www.python.org/foo.html?bar=spam"
• urljoin(base, url) - Combines a base and relative URLurlparse.urljoin("http://www.python.org/index.html","help.html")# Produces "http://www.python.org/help.html"
73
CGI SCRIPTING
• CGI Overview• Common protocol web servers use to run external programs in
response to HTTP requests.• Typical uses: forms processing, dynamic content generation
• How it works• You write some sort of form in your HTML document
<form method="GET" action="cgi-bin/spam.cgi">Your name: <input type="text" name="name" size=30><p>Your email: <input type="text" name="email" size=40><p><input type="submit" value="Submit"></form>
• This gets translated into request with parametersGET /cgi-bin/spam.cgi?name=Dave+Beazley&email=beazley%40cs HTTP/1.0
• Web-server (e.g., Apache) launches CGI program and passes parameters
• That program writes to stdout to produce the web-page.74
WEATHER CONDITION
import pywapiimport string
yahoo_result = pywapi.get_weather_from_yahoo(‘10001')
print "Yahoo says: It is " + string.lower(yahoo_result['condition']['text']) + " and " + yahoo_result['condition']['temp'] + "C now in New York.\n\n"
The module provides a python wrapper around the Yahoo! Weather API.
http://code.google.com/p/python-weather-api/
python-weather-api.py C3PO75
RPITWIT
• rpitwit 0.2.0• Download• rpitwit-0.2.0.tar.gz
• Remote control your RaspberryPI from Twitter.
• RPiTwit is a really simple script to execute remote shell commands on your RaspberryPI (or any other Linux distribution with python) using your twitter account.
76
