Quick black box testing example

There's an ongoing debate on the agile-testing mailing list on whether it's better to have a 'black box' or a 'white box' view into the system under test. Some are of the opinion that black boxes are easier to test, while others (Ron Jeffries in particular) say that one would like to 'open up' one's boxes, especially in an agile environment. I suspect that the answer, as always, is somewhere in the middle -- both white-box and black-box testing are critical and valuable in their own right.

I think that it's in combining both types of tests that developers and testers will find the confidence that the software under test is stable and relatively free of bugs. Developers do white-box testing via unit tests, while testers do mostly black-box testing (or maybe gray-box, since they usually do have some insight into the inner workings of the application) via functional, integration and system testing. Let's not forget load/performance/stress testing too...They too can be viewed as white-box (mostly in the case of performance testing) vs. black-box (load/stress testing), as I wrote in a previous post.

I want to include in this post my answer to a little example posted by Ron Jeffries. Here's what he wrote:

Let's explore a simple example. Suppose we have an application that includes an interface (method) whose purpose is to find a "matching" record in a collection, if one exists. If none exists, the method is to return null.

The collection is large. Some users of this method have partial knowledge of the collection's order, so that they know that the record they want, if it is in there at all, occurs at or after some integer index in the collection.

So the method accepts a value, let's say a string /find/, to match the record on, and an integer /hint/, to be used as a hint to start the search. The first record in the table is numbered zero. The largest meaningful /hint/ value is therefore N-1, where N is the number of records in the table.

We want the search to always find a record if one exists, so that if /hint/ is wrong, but /find/ is in some record, we must still return a matching record, not null.

Now then. Assuming a black box, what questions do we want to ask, what tests do we want to write, against our method

public record search(string find, int hint)?

And here's my answer:

I'll take a quick stab at it. Here's what I'd start by doing (emphasis on start):

1. Generate various data sets to run the 'search' method against.

1a. Vary the number of items in the collection: create collections with 0, 1, 10, 100, 1000, 10000, 100000, 1 million items for starters; it may be the case that we hit an operating system limit at some point, for example if the items are files in the same directory (ever done an ls only to get back a message like "too many arguments"?)

1b. For each collection in 1a., generate several orderings: increasing order, decreasing order, random, maybe some other statistical distributions.

1c. Vary the length of the names of the items in the collection: create collections with 0, 1, 10, 100, 1000 items, where the names of the items are generated randomly with lengths between 1 and 1000 (arbitrary limit, which may change as we progress testing).

1d. Generate item names with 'weird' characters (especially /, \, :, ; -- since they tend to be used as separators by the OS).

1e. Generate item names that are Unicode strings.

2. Run (and time) the 'search' method against the various collections generated in 1. Make sure you cover cases such as:

2a. The item we search for is not in the collection: verify that the search method returns Null.

2b. The item we search for is in position p, where p can be 0, N/2, N-1, N.

2c. For each case in 2b, specify a hint of 0, p-1, p, p+1, N-1: verify that in all combinations of 2b and 2c, the search method returns the item in position p.

2d. Investigate the effect of item naming on the search. Does the search method work correctly when item names keep getting longer? When the item names contain 'weird' or Unicode characters?

2e. Graph the running time of the search method against collection size, when the item is or is not in the collection (so you generate 2 graphs). See if there is any anomaly.

2f. Run the tests in 2a-2d in a loop, to see if the search method produces a memory leak.

2g. Monitor various OS parameters (via top, vmstat, Windows PerfMon) to see how well-behaved the search functionality is in regards to the resources on that machine.

2h. See how the search method behaves when other resource-intensive processes are running on that machine (CPU-, disk-, memory-, network- intensive).

If the collection of records is kept in a database, then I can imagine a host of other stuff to test that is database-related. Same if the collection is retrieved over the network.

As I said, this is just an initial stab at testing the search method. I'm sure people can come up with many more things to test. But I think this provides a pretty solid base and a pretty good automated test suite for the AUT.

I can think of many more tests that should be run if the search application talks to a database, or if it retrieves the search results via a Web service for example. I guess this all shows that a tester's life is not easy :-) -- but this is all exciting stuff at the same time!

Django cheat sheet

Courtesy of James: Django cheat sheet. I went trough the first 2 parts of the Django tutorial and I have to say I'm very impressed. Can't wait to give it a try on a real Web application.

Slides from 'py library overview' presentation

I presented an overview of the py library last night at our SoCal Piggies meeting. Although I didn't cover all the tools in the py library, I hope I managed to heighten the interest in this very useful collection of modules. You can find the slides here. Kudos again to Holger Krekel and Armin Rigo, the main guys behind the py lib.

And while we're on this subject, let's make py.test the official unit test framework for Django!!! (see the open ticket on this topic)

Installing Python 2.4.1 and cx_Oracle on AIX

I just went through the pain of getting the cx_Oracle module to work on an AIX 5.1 server running Oracle 9i, so I thought I'd jot down what I did, for future reference.

First of all, I had ORACLE_HOME set to /oracle/OraHome1.

1. Downloaded the rpm.rte package from the AIX Toolbox Download site.
2. Installed rpm.rte via smit.
3. Downloaded (from the same AIX Toolbox Download site) and installed the following RPM packages, in this order:

rpm -hi gcc-3.3.2-5.aix5.1.ppc.rpm
rpm -hi libgcc-3.3.2-5.aix5.1.ppc.rpm
rpm -hi libstdcplusplus-3.3.2-5.aix5.1.ppc.rpm
rpm -hi libstdcplusplus-devel-3.3.2-5.aix5.1.ppc.rpm
rpm -hi gcc-cplusplus-3.3.2-5.aix5.1.ppc.rpm

4. Made a symlink from gcc to cc_r, since many configuration scripts find cc_r as the compiler of choice on AIX, and I did not have it on my server.

ln -s /usr/bin/gcc /usr/bin/cc_r

5. Downloaded Python-2.4.1 from python.org.
6. Installed Python-2.4.1 (note that the vanilla ./configure failed, so I needed to run it with --disable-ipv6):

gunzip Python-2.4.1.tgz
tar xvf Python-2.4.1.tar
cd Python-2.4.1
./configure --disable-ipv6
make
make install

7. Downloaded cx_Oracle-4.1 from sourceforge.net.
8. Installed cx_Oracle-4.1 (note that I indicated the full path to python, since there was another older python version on that AIX server):

bash-2.05a# /usr/local/bin/python setup.py install
running install
running build
running build_ext
building 'cx_Oracle' extension
creating build
creating build/temp.aix-5.1-2.4
cc_r -pthread -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I/oracle/OraHome1/rdbms/demo -I/oracle/OraHome1/rdbms/public -I/oracle/OraHome1/network/public -I/usr/local/include/python2.4 -c cx_Oracle.c -o build/temp.aix-5.1-2.4/cx_Oracle.o -DBUILD_TIME="July 15, 2005 14:49:28"
In file included from /oracle/OraHome1/rdbms/demo/oci.h:2138,
           from cx_Oracle.c:9:
/oracle/OraHome1/rdbms/demo/oci1.h:148: warning: function declaration isn't a prototype
In file included from /oracle/OraHome1/rdbms/demo/ociap.h:190,
           from /oracle/OraHome1/rdbms/demo/oci.h:2163,
           from cx_Oracle.c:9:
/oracle/OraHome1/rdbms/public/nzt.h:667: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2655: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2664: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2674: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2683: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2692: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2701: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2709: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/public/nzt.h:2719: warning: function declaration isn't a prototype
In file included from /oracle/OraHome1/rdbms/demo/oci.h:2163,
           from cx_Oracle.c:9:
/oracle/OraHome1/rdbms/demo/ociap.h:6888: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/demo/ociap.h:9790: warning: function declaration isn't a prototype
/oracle/OraHome1/rdbms/demo/ociap.h:9796: warning: function declaration isn't a prototype
In file included from Variable.c:93,
           from Cursor.c:211,
           from Connection.c:303,
           from SessionPool.c:132,
           from cx_Oracle.c:73:
DateTimeVar.c: In function `DateTimeVar_SetValue':
DateTimeVar.c:81: warning: unused variable `status'
creating build/lib.aix-5.1-2.4
/usr/local/lib/python2.4/config/ld_so_aix cc_r -pthread -bI:/usr/local/lib/python2.4/config/python.exp build/temp.aix-5.1-2.4/cx_Oracle.o -L/oracle/OraHome1/lib -lclntsh -o build/lib.aix-5.1-2.4/cx_Oracle.so -s
ld: 0711-317 ERROR: Undefined symbol: .OCINumberFromInt
ld: 0711-317 ERROR: Undefined symbol: .OCINumberFromReal
ld: 0711-317 ERROR: Undefined symbol: .OCINumberFromText
ld: 0711-317 ERROR: Undefined symbol: .OCINumberToReal
ld: 0711-317 ERROR: Undefined symbol: .OCINumberToText
ld: 0711-317 ERROR: Undefined symbol: .OCINumberToInt
ld: 0711-317 ERROR: Undefined symbol: .OCIParamGet
ld: 0711-317 ERROR: Undefined symbol: .OCIDescriptorFree
ld: 0711-317 ERROR: Undefined symbol: .OCIAttrGet
ld: 0711-317 ERROR: Undefined symbol: .OCIStmtExecute
ld: 0711-317 ERROR: Undefined symbol: .OCISessionGet
ld: 0711-317 ERROR: Undefined symbol: .OCIServerDetach
ld: 0711-317 ERROR: Undefined symbol: .OCITransRollback
ld: 0711-317 ERROR: Undefined symbol: .OCISessionEnd
ld: 0711-317 ERROR: Undefined symbol: .OCISessionRelease
ld: 0711-317 ERROR: Undefined symbol: .OCIHandleFree
ld: 0711-317 ERROR: Undefined symbol: .OCIHandleAlloc
ld: 0711-317 ERROR: Undefined symbol: .OCIAttrSet
ld: 0711-317 ERROR: Undefined symbol: .OCITransStart
ld: 0711-317 ERROR: Undefined symbol: .OCISessionPoolCreate
ld: 0711-317 ERROR: Undefined symbol: .OCIErrorGet
ld: 0711-317 ERROR: Undefined symbol: .OCIEnvCreate
ld: 0711-317 ERROR: Undefined symbol: .OCINlsNumericInfoGet
ld: 0711-317 ERROR: Undefined symbol: .OCISessionPoolDestroy
ld: 0711-317 ERROR: Undefined symbol: .OCITransCommit
ld: 0711-317 ERROR: Undefined symbol: .OCITransPrepare
ld: 0711-317 ERROR: Undefined symbol: .OCIBreak
ld: 0711-317 ERROR: Undefined symbol: .OCIUserCallbackRegister
ld: 0711-317 ERROR: Undefined symbol: .OCIUserCallbackGet
ld: 0711-317 ERROR: Undefined symbol: .OCIServerAttach
ld: 0711-317 ERROR: Undefined symbol: .OCISessionBegin
ld: 0711-317 ERROR: Undefined symbol: .OCIStmtRelease
ld: 0711-317 ERROR: Undefined symbol: .OCIDescriptorAlloc
ld: 0711-317 ERROR: Undefined symbol: .OCIDateTimeConstruct
ld: 0711-317 ERROR: Undefined symbol: .OCIDateTimeCheck
ld: 0711-317 ERROR: Undefined symbol: .OCIDateTimeGetDate
ld: 0711-317 ERROR: Undefined symbol: .OCIDateTimeGetTime
ld: 0711-317 ERROR: Undefined symbol: .OCILobGetLength
ld: 0711-317 ERROR: Undefined symbol: .OCILobWrite
ld: 0711-317 ERROR: Undefined symbol: .OCILobTrim
ld: 0711-317 ERROR: Undefined symbol: .OCILobRead
ld: 0711-317 ERROR: Undefined symbol: .OCILobFreeTemporary
ld: 0711-317 ERROR: Undefined symbol: .OCILobCreateTemporary
ld: 0711-317 ERROR: Undefined symbol: .OCIDefineByPos
ld: 0711-317 ERROR: Undefined symbol: .OCIStmtGetBindInfo
ld: 0711-317 ERROR: Undefined symbol: .OCIStmtPrepare2
ld: 0711-317 ERROR: Undefined symbol: .OCIStmtFetch
ld: 0711-317 ERROR: Undefined symbol: .OCIBindByName
ld: 0711-317 ERROR: Undefined symbol: .OCIBindByPos
ld: 0711-345 Use the -bloadmap or -bnoquiet option to obtain more information.
collect2: ld returned 8 exit status
running install_lib

At this point, I did a lot of Google searches to find out why the loader emits these errors. I finally found the solution: Oracle 9i installs the 64-bit libraries in $ORACLE_HOME/lib and the 32-bit libraries in $ORACLE_HOME/lib32. Since setup.py is looking by default in $ORACLE_HOME/lib (via -L/oracle/OraHome1/lib), it finds the 64-bit libraries and it fails with the above errors. The quick hack I found was to manually re-run the last command that failed and specify -L/oracle/OraHome1/lib32 instead of -L/oracle/OraHome1/lib (I think the same effect can be achieved via environment variables such as LIBPATH).

bash-2.05a# /usr/local/lib/python2.4/config/ld_so_aix cc_r -pthread -bI:/usr/local/lib/python2.4/config/python.exp build/temp.aix-5.1-2.4/cx_Oracle.o -L/oracle/OraHome1/lib32 -lclntsh -o build/lib.aix-5.1-2.4/cx_Oracle.so -s

Then I re-ran setup.py in order to copy the shared library to the Ptyhon site-packages directory:

bash-2.05a# /usr/local/bin/python setup.py install
running install
running build
running build_ext
running install_lib
copying build/lib.aix-5.1-2.4/cx_Oracle.so -> /usr/local/lib/python2.4/site-packages

At this point I was able to import cx_Oracle at the Python prompt:

bash-2.05a# /usr/local/bin/python
Python 2.4.1 (#1, Jul 15 2005, 14:44:07)
[GCC 3.3.2] on aix5
Type "help", "copyright", "credits" or "license" for more information.
>>> import cx_Oracle
>>> dir(cx_Oracle)
['BINARY', 'BLOB', 'CLOB', 'CURSOR', 'Connection', 'Cursor', 'DATETIME', 'DataError', 'DatabaseError', 'Date', 'DateFromTicks', 'Error', 'FIXED_CHAR', 'FNCODE_BINDBYNAME', 'FNCODE_BINDBYPOS', 'FNCODE_DEFINEBYPOS', 'FNCODE_STMTEXECUTE', 'FNCODE_STMTFETCH', 'FNCODE_STMTPREPARE', 'IntegrityError', 'InterfaceError', 'InternalError', 'LOB', 'LONG_BINARY', 'LONG_STRING', 'NUMBER', 'NotSupportedError', 'OperationalError', 'ProgrammingError', 'ROWID', 'STRING', 'SYSDBA', 'SYSOPER', 'SessionPool', 'TIMESTAMP', 'Time', 'TimeFromTicks', 'Timestamp', 'TimestampFromTicks', 'UCBTYPE_ENTRY', 'UCBTYPE_EXIT', 'UCBTYPE_REPLACE', 'Warning', '__doc__', '__file__', '__name__', 'apilevel', 'buildtime', 'connect', 'makedsn', 'paramstyle', 'threadsafety', 'version']

py lib gems: greenlets and py.xml

I've been experimenting with various tools in the py library lately, in preparation for a presentation I'll give to the SoCal Piggies group meeting this month. The py lib is choke-full of gems that are waiting to be discovered. In this post, I'll talk a little about greenlets, the creation of Armin Rigo. I'll also briefly mention py.xml.

Greenlets implement coroutines in Python. Coroutines can be seen as a generalization of generators, and it looks like the standard Python libray will support them in the future via 'enhanced generators' (see PEP 342). Coroutines allow you to exit a function by 'yielding' a value and switching to another function. The original function can then be re-entered, and it will continue execution from exactly where it left off.

The greenlet documentation offers some really eye-opening examples of how they can be used to implement generators for example. Another typical use case for greenlets/coroutines is turning asynchronous or event-based code into normal sequential control flow code -- the Python Desktop Server project has a good example of exactly such a transformation.

I've also been reading and looking at the code from Armin's EuroPython talk on greenlets. The talk itself must have been highly entertaining, since it is presented as a PyGame-based game. In one of the code examples I downloaded, I noticed yet another application of the asynchronous-to-sequential transformation, this time related to parsing XML data. In a few lines of code, Armin showed how to turn an asynchronous, Expat-based parsing mechanism into a generator that yields the XML elements one at a time. This approach combines the advantages of 1) using a stream oriented parser (and thus being able to process large amounts of XML data via handlers) with 2) using a generator to expose the XML parsing code in the shape of an iterator.

Here is Armin's code which I saved in a module called iterxml.py (I made a few minor modifications to make the code more general-purpose):

from py.magic import greenlet
import xml.parsers.expat

def send(arg):
    greenlet.getcurrent().parent.switch(arg)

# 3 handler functions
def start_element(name, attrs):
    send(('START', name, attrs))
def end_element(name):
    send(('END', name))
def char_data(data):
    data = data.strip()
    if data:
        send(('DATA', data))

def greenparse(xmldata):
    p = xml.parsers.expat.ParserCreate()
    p.StartElementHandler = start_element
    p.EndElementHandler = end_element
    p.CharacterDataHandler = char_data
    p.Parse(xmldata, 1)

def iterxml(xmldata):
    g = greenlet(greenparse)
    data = g.switch(xmldata)
    while data is not None:
        yield data
        data = g.switch()

Consumers of this code can pass a string containing an XML document to the iterxml function and then use a for loop to iterate through the elements yielded by the function, like this:

for data in iterxml(xmldata):
    print data

When iterxml first executes, it instantiates a greenlet object and associates it with the greenparse function. Then it 'switches' into the greenlet and thus is calls that function with the given xmldata argument. There is nothing out of the ordinary in the greenparse function, which simply assigns the 3 handler functions to the xpat parser object, then calls its Parse method. However, the 3 handler functions all use greenlets via the send method, which sends the parsed data to the parent of the current greenlet. The parent in this case is the iterxml function, which yields the data at that point, then switches back into the greenparse function. The handler functions then get called again whenever a new XML element is encountered, and the switching back and forth continues until there is no more data to be parsed.

I've wanted for a while to check out the REST API offered by upcoming.org (which is a free alternative to meetup.com), so I used it in conjunction with the XML parsing stuff via greenlets.

Here's some code that uses the iterxml module to parse the response returned by upcoming.org when a request for searching the events in the L.A. metro area is sent to their server:

import sys, urllib
from iterxml import iterxml
import py

baseurl = "http://www.upcoming.org/services/rest/"
api_key = "YOUR_API_KEY_HERE"
metro_id = "1" # L.A. metro area
log = py.log.Producer("")

def get_venue_info(venue_id):
    method = "venue.getInfo"
    request = "%s?api_key=%s&method=%s&venue_id=%s" % 
        (baseurl, api_key, method, venue_id)
    response = urllib.urlopen(request).read()
    for data in iterxml(response):
        if data[0] == 'START' and data[1] == 'venue':
            attr = data[2]
            venue_info = "%(name)s in %(city)s" % attr
            break
    return venue_info

def search_events(keywords):
    method = "event.search"
    request = "%s?api_key=%s&method=%s&metro_id=%s&search_text=%s" % 
        (baseurl, api_key, method, metro_id, keywords)
    response = urllib.urlopen(request).read()
    for data in iterxml(response):
        if data[0] == 'START' and data[1] == 'event':
            attr  = data[2]
            log("\n" + "-" * 80)
            log.EVENT("%(name)s" % attr)
            log.WHAT("%(description)s" % attr)
            log.WHERE(get_venue_info(attr['venue_id']))
            log.WHEN("%(start_date)s @ %(start_time)s" % attr)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "Usage: %s " % sys.argv[0]
        sys.exit(1)

    keywords = "%%20".join(sys.argv[1:])
    search_events(keywords)

An upcoming.org API key is automatically generated for you when you click here.

I tested the script by searching for Python-related events in L.A.:

./upcoming_search.py python

--------------------------------------------------------------------------------
[EVENT] SoCal Piggies July Meeting
[WHAT] Monthly meeting of the Southern California Python Interest Group.
[WHERE] USC in Los Angeles
[WHEN] 2005-07-26 @ 19:00:00

Note that I'm also using the py.log facilities I mentioned in a previous post. The only thing I needed to do was to instantiate a log object via log = py.log.Producer("") and then use it via keywords such as EVENT, WHAT, WHERE and WHEN. Since I didn't declare any log consumer, the default consumer is used, which prints its messages to stdout. Each message string is nicely prefixed by the corresponding keyword.

I'm still experimenting with greenlets, and I'm sure I'll use them in the future especially for event-based GUI code.

I want to also briefly touch on py.xml, a tool that allows you to generate XML and HTML documents almost painlessly from your Python code.

Here's the XML returned by the event.search method of upcoming.org when called with a search text of 'python':

<rsp stat="ok" version="1.0">
<event id="24868" name="SoCal Piggies July Meeting" description="Monthly meeting of the Southern California Python Interest Group." start_date="2005-07-26" end_date="0000-00-00" start_time="19:00:00" end_time="21:00:00" personal="0" selfpromotion="0" metro_id="1" venue_id="7425" user_id="14959" category_id="4" date_posted="2005-07-13" />
</rsp>

And here's how to generate the same XML output with py.xml:

import py

class ns(py.xml.Namespace):
    "my custom xml namespace"

doc = ns.rsp(
    ns.event(
        id="24868",
        name="SoCal Piggies July Meeting",
        description="Monthly meeting of the Southern California Python Interest Group.",
        start_date="2005-07-26",
        end_date="0000-00-00",
        start_time="19:00:00",
        end_time="21:00:00",
        personal="0",
        selfpromotion="0",
        metro_id="1",
        venue_id="7425",
        user_id="14959",
        category_id="4",
        date_posted="2005-07-13"),
    stat="OK",
    version="1.0",
    )

print doc.unicode(indent=2).encode('utf8')

The code above prints:

<rsp stat="OK" version="1.0">
  <event category_id="4" date_posted="2005-07-13" description="Monthly meeting of the Southern California Python Interest Group." end_date="0000-00-00" end_time="21:00:00" id="24868" metro_id="1" name="SoCal Piggies July Meeting" personal="0" selfpromotion="0" start_date="2005-07-26" start_time="19:00:00" user_id="14959" venue_id="7425"/></rsp>

As the py.xml documentation succintly puts it, positional arguments are child-tags and keyword-arguments are attributes. In addition, indentation is also available via the argument to the unicode method.

I intend to cover other tools from the py library in future posts. Stay tuned for discussions on py.execnet, little-known aspects of py.test and more!

Recommended reading: Jason Huggins's blog

I recently stumbled on Jason's blog via the Thoughtworks RSS feed aggregator. Jason is the creator of Selenium and a true Pythonista. His latest post on using CherryPy, SQLObject and Cheetah for creating a 'Ruby on Rails'-like application is very interesting and entertaining. Highly recommended! Hopefully the Subway guys will heed Jason's advice of focusing more on "ease of installation and fancy earth-shatteringly beautiful 10 minute setup movies" -- this is one area in which it's hard to beat the RoR guys, but let's at least try it!

Keyword-based logging with the py library

I've been experimenting lately with the logging functionality recently introduced in the py library. Most of the functionality is still in early experimental stage, but the main ideas are pretty well defined. I want to review some of these ideas here and hopefully get some feedback.

Holger Krekel, the main py library developer, introduced the idea of keyword-based logging and of logging producers and consumers. A log producer is an object used by the application code to send messages to various log consumers. The consumers can be STDOUT, STDERR, log files, syslog, the Windows Event Log, etc. When you create a log producer, you define a set of keywords that are then used to both route the logging messages to consumers, and to prefix those messages.

Here's a quick example of code that works with the current svn version of the py library (which you can get from http://codespeak.net/svn/py/dist). I saved the following code in a file called logmodule.py:

# logmodule.py

import py

log = py.log.Producer("logmodule")

def runcmd(cmd):
    log.cmd(cmd)
    # real code follows

def post2db(data):
    log.post2db(data)
    # real code follows

if __name__ == "__main__":
    runcmd("Running command CMD1")
    post2db("PASS")

I started by instantiating a log object of class py.log.Producer with the keyword "logmodule". Since I didn't associate any consumer with the log producer, a default consumer will be used, which prints all messages to STDOUT.

The log producer object does not have any pre-defined methods associated with it. Instead, you can invoke arbitrarily-named methods. When accessing a brand new method (such as log.cmd or log.post2db), another producer object is returned on the fly, with the name of the method added as a keyword to the already-existent set of keywords. When calling the returned object as a function with the log message as the argument, the keywords are looked up to see if there is any log consumer associated with them. If no consumer is found, the default one is used and the message is pri nted to STDOUT.

Running the code above produces:

[logmodule:cmd] Running command CMD1
[logmodule:post2db] PASS

What happened behind the scenes when invoking log.cmd("Running command CMD1") was that a log producer object called log.cmd was created with ('logmodule', 'cmd') as the set of keywords. When calling the object as a function with the message as the argument, the message was by default printed to STDOUT, since no consumer was found to be associated with those keywords.
The message was also prefixed with the keywords, joined by ":".

Now let's assume we don't want any messages to be logged when using the logmodule module directly. To achieve this, we associate a null consumer with the "logmodule" keyword by adding the following line somewhere at the top of our module :

py.log.setconsumer("logmodule", None)

(NOTE: the name of the setconsumer function might change, but the idea will be the same, namely that any keyword or set of keywords can be associated with log consumers).

Now, when we invoke log.cmd("Running command CMD1"), the keyword matching logic will first look for a consumer associated with ('logmodule', 'cmd'), since this is the complete set of keywords. If it doesn't find it, it looks for a consumer associated with 'logmodule'. In our case, it will find the consumer we designated (None), so it will use that consumer and not log the message anywhere.

The simple mechanism of associating log consumers with keywords offers a tremendous amount of flexibility in configuring logging for an application. The association keywords-consumers can be done at run time, via a configuration file or via command-line arguments, and the application code doesn't need to know where logging gets routed. As an example, let's assume that we want to use the 2 top-level functions runcmd and post2db from the logmodule module, and at the same time we want to log messages to other consumers. Here is some code I saved in a file called use_logmodule.py:

# use_logmodule.py

import py
from logmodule import runcmd, post2db

py.log.setconsumer("logmodule", py.log.STDERR)
py.log.setconsumer("logmodule post2db", None)

runcmd("Running command CMD2")
post2db("FAIL")

My goal was to print all messages to STDERR by default, and not print any messages associated with the "post2db" keyword (in a real life situation, those messages could be associated with some database-management code for example).

To achieve my goal, I associated a consumer that uses STDERR with the keyword "logmodule" and a "/dev/null"-type consumer (i.e. None) with the keywords "logmodule post2db". Now when I run the above code, I get the following message printed to STDERR:

[logmodule:cmd] Running command CMD2

I could just as easily have routed the logging messages to a log file by associating a producer of type py.log.Path (currently there are 3 types of pre-defined consumers offered by the py.log package: py.log.STDOUT, py.log.STDERR and py.log.Path; in the future, they will be augmented with other types of consumers such as syslog, Windows Event Logs, etc.). Here's how to log all "post2db"-related messages to a file:

py.log.setconsumer("logmodule post2db", py.log.Path("/tmp/logmodule.out"))

When running the code in use_logmodule.py, the following line will be printed to /tmp/logmodule.out:

[logmodule:post2db] FAIL

A question you might have at this point is how does all this relate to a more traditional use of logging, based on severity levels such as debug, info, warn, error, critical. Let's assume we want our application code to issue calls to log.debug, log.info and log.error, while being able at the same time to direct these various levels of severity to different logging mechanisms. We could for example log debug messages to a log file, informational messages to the console (STDOUT or STDERR), and error messages to both a log file and the console. Here's one way to achieve this via the py.log keyword-based approach. I saved the following code in a file called config.py:

# config.py

import sys
import py

log = py.log.Producer("")
log.debug = py.log.Producer("logfile")
log.info = py.log.Producer("console")
log.error = py.log.Producer("console logfile")

logfile = "/tmp/myapp.out"
py.log.setconsumer("logfile", py.log.Path(logfile))
py.log.setconsumer("console", py.log.STDOUT)

l = open(logfile, 'a', 1)

def console_logfile_logger(msg):
    print >>sys.stdout, str(msg)
    print >>l, str(msg)

py.log.setconsumer("console logfile", console_logfile_logger)

Most of this stuff should look familiar by now. I instantiated an object called log of class Producer, with no keywords associated with it. This object will serve as the default producer object, so the application will be able to call log(some_msg) and have some_msg printed to STDOUT by default. Next, I created the debug, info and error attributes for the log object. This will enable me to call log.debug(msg), log.info(msg) and log.error(msg) in my application code, without necessarily caring where those messages get routed. As it stands right now in the configuration file, the log.debug object is declared as a log.Producer object associated with the keyword "logfile". Similarly, the log.info object is associated with the keyword "console" and the log.error object is associated with both keywords "console logfile".

The next step is defining consumers for the various keywords. For "logfile" I set a consumer of type Path, pointing to the file /tmp/myapp.out. For "console" I set the STDOUT consumer, and for "console logfile" I set a custom consumer that prints its argument (the message to be logged) to STDOUT and also appends it to the log file. As an aside, I want to note that consumers are simply functions that take a message as an argument and manage the message as they see fit (if you look in the testing code for the py.log module in the py library, you'll see consumers that simply append the message to a list for example).

Here is an example of application code that uses the above config.py file:

# appcode.py

import config

log = config.log

log.debug("DEBUG MESSAGE")
log.info("INFO MESSAGE")
log.error("ERROR MESSAGE")
log.warn("WARNING MESSAGE")

Running appcode.py produces:

[console] INFO MESSAGE
[console:logfile] ERROR MESSAGE
[warn] WARNING MESSAGE

At the same time, the file /tmp/myapp.out contains:

[logfile] DEBUG MESSAGE
[console:logfile] ERROR MESSAGE

Note that we didn't declare log.warn anywhere. It was dynamically transformed by py.log into a producer object with keyword "warn" and printed by default to STDOUT, since the "warn" keyword didn't have any consumer associated with it.

I didn't want to complicate things too much in the code above, but in my opinion it would be better to have 'debug', 'info' and 'error' as part of the keywords that also prefix each message, so that you can see at a glance what kind of severity level the message has. To achieve that, you would change the declarations for log.debug, log.info and log.error in config.py to:

log.debug = py.log.Producer("logfile").debug
log.info = py.log.Producer("console").info
log.error = py.log.Producer("console logfile").error

or alternatively to:

log.debug = py.log.Producer("logfile debug")
log.info = py.log.Producer("console info")
log.error = py.log.Producer("console logfile error")

Now if you run appcode.py, you'll see this printed on the screen:

[console:info] INFO MESSAGE
[console:logfile:error] ERROR MESSAGE
[warn] WARNING MESSAGE

This also makes it easier to turn off debugging for example. All you need to do is to associate the null consumer with the keywords "logfile debug":

py.log.setconsumer("logfile debug", None)

Note that the consumer associated with the single keyword "logfile" is still the Path consumer that prints to /tmp/myapp.out, so non-debug messages would still get routed to that file.

In conclusion, I think that the keyword-based approach works really well for the logging needs of any application. I emphasize once more that the py.log code is still in its infancy and as such is prone to changes (including naming changes), so I wouldn't rely on py.log just yet for production code. But the main idea of associating log producers with log consumers via keywords should be the same, and this is I think the novelty of this approach.

One enhancement that I've discussed with Holger is to have a way to specify that you want multiple keywords to be routed each to a different consumer. So for example, if you declared

log.error = py.log.Producer("console logfile")

a call to log.error(some_msg) would try to match the "console" keyword individually to a consumer, and the "logfile" keyword to another consumer.

Right now, the keyword matching logic only looks for a single consumer that matches either ("console", "logfile") or just ("console",), so you need to declare a custom consumer such as the console_logfile_logger function I described above.

In the multiple-keyword approach, you wouldn't need to do that, instead you would just associate "logfile" with a consumer of type Path, "console" with a consumer of type STDOUT/STDERR, and both consumers would then receive the message when log.error(some_msg) was called.

If you think you can use the keyword-based approach to logging in your own application code, or if you have a use case that you would like to see implemented via py.log, or if you have any suggestions at all, please leave a comment here of send email to py-dev at codespeak dot net.

Handling the 'Path' Windows registry value correctly

A recent recipe posted to the Python Cookbook site shows how to manipulate environment variables in Windows by modifying the registry. I've been using something similar in my code, mainly for adding directories on the fly to the 'Path' environment variable by modifying the \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Environment\Path registry value.

Here are two simple functions for getting and setting a registry value of type string (REG_SZ):

def get_registry_value(reg, key, value_name):
k = _winreg.OpenKey(reg, key)
value = _winreg.QueryValueEx(k, value_name)[0]
_winreg.CloseKey(k)
return value

def set_registry_value(reg, key, value_name, value):
k = _winreg.OpenKey(reg, key, 0, _winreg.KEY_WRITE)
value_type = _winreg.REG_SZ
_winreg.SetValueEx(k, value_name, 0, value_type, value)
_winreg.CloseKey(k)

To add a directory dir to the Path registry value, you would do:

reg = _winreg.ConnectRegistry(None, _winreg.HKEY_LOCAL_MACHINE)
key = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
path = get_registry_value(reg, key, "Path")
path += ";" + dir
set_registry_value(reg, key, "Path", path)

However, there was a problem with this approach. If you look at a typical Path registry value (using for example regedit), you'll see that it contains directories such as %SystemRoot% and %SystemRoot%\system32, where %SystemRoot% is a variable which has the Windows system directory as its value. When I modified the Path value, I mistakenly set its type to REG_SZ, and in consequence the command prompt interpreter did not replace the %SystemRoot% variable with its value. The unfortunate side effect of all this was that typical Windows binaries residing in the system32 directory, such as ping, ipconfig, etc., were not found anymore by the command prompt.

As a workaround, I replaced %SystemRoot% with os.environ['SYSTEMROOT'] in the path variable, like this:

 dirs = path.split(';')
 try:
     systemroot = os.environ['SYSTEMROOT']
 except KeyError:
     pass
 else:
     dirs = [re.sub('%SystemRoot%', systemroot, dir)
             for dir in dirs]
 path = ';'.join(dirs)

A much better approach, suggested by Ori Berger in a comment to the recipe I mentioned, is to set the value type for the Path registry value to REG_EXPAND_SZ, and not to REG_SZ. This makes variables like %SystemRoot% be automatically expanded by the command prompt interpreter. So the modified version of my set_registry_value function is now:

def set_registry_value(reg, key, value_name, value, value_type=_winreg.REG_SZ):
 k = _winreg.OpenKey(reg, key, 0, _winreg.KEY_WRITE)
 _winreg.SetValueEx(k, value_name, 0, value_type, value)
 _winreg.CloseKey(k)

For modifying the 'Path' value, you would now do:

reg = _winreg.ConnectRegistry(None, _winreg.HKEY_LOCAL_MACHINE)
key = r"SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
path = get_registry_value(reg, key, "Path")
path += ";" + dir
set_registry_value(reg, key, "Path", path, _winreg.REG_EXPAND_SZ)

Of course, the whole registry value manipulation code can be neatly packaged in a class, like here.

Installing the Firebird database on a 64-bit RHEL Linux server

I just finished the painful exercise of installing Firebird on an Intel E64MT server running RHEL 3. I first tried installing the RPM versions of Firebird from here -- tried both the 1.0 version and the 1.5 version of the database, in both Classic and SuperServer incarnations. In all cases, the RPMs installed without error, but when I tried to install the KInterbasDB Python module (and also the DBD::Interbase Perl module), the loader complained about the Firebird shared libraries, which were obviously not 64-bit. So I ended up installing Firebird from source. Here's what I did:

1. Downloaded and bunzip-ed firebird-1.5.2.4731.tar.bz2.
2. Ran ./configure, then make and make install in the firebird-1.5.2.4731 directory.

By default, this installs the Classic version of Firebird in /usr/local/firebird (the Classic version allows processes to connect directly to database files; the SuperServer version requires processes to open a socket to a server process, which then accesses the database files).

The 'make install' command also asks for a password for the SYSDBA user. If you happen to forget the password, you can always retrieve it by looking at the file /usr/local/firebird/SYSDBA.password as root. You change the password by running /usr/local/firebird/bin/changeDBAPassword.sh.

3. Downloaded the kinterbasdb-3.1.src.tar.gz Python module and installed it via python setup.py install.
4. Downloaded and installed the mxDateTime Python module from here (this module is required by kinterbasdb.)
5. Tested kinterbasdb against the example employee database shipped with Firebird:

>>> import kinterbasdb
>>> conn = kinterbasdb.connect(database='/usr/local/firebird/examples/employee.fdb')
>>> cursor = conn.cursor()
>>> cursor.execute('select * from country')
>>> print cursor.fetchall()
[('USA', 'Dollar'), ('England', 'Pound'), ('Canada', 'CdnDlr'), ('Switzerland','SFranc'), ('Japan', 'Yen'), ('Italy', 'Lira'), ('France', 'FFranc'), ('Germany', 'D-Mark'), ('Australia', 'ADollar'), ('Hong Kong', 'HKDollar'), ('Netherlands', 'Guilder'), ('Belgium', 'BFranc'), ('Austria', 'Schilling'), ('Fiji', 'FDollar')]

At this point, I created a Firebird database file called test.gdb, owned by user root and group root. When I connected to this database as root via the isql utility in /usr/local/firebird, everything looked good:

[root@localhost]# /usr/local/firebird/bin/isql
Use CONNECT or CREATE DATABASE to specify a database
SQL> connect test.gdb;
Database: test.gdb
SQL>quit;

However, when I tried to connect to the database as a non-root user, I got this error:

$ /usr/local/firebird/bin/isql
Use CONNECT or CREATE DATABASE to specify a database
SQL> connect test.gdb;
Statement failed, SQLCODE = -551

no permission for read-write access to database test.gdb

What I needed to do was to change ownership and permissions for the test.gdb file so that it was owned by user firebird and group firebird, and it was group-writeable:

chown firebird.firebird test.gdb
chmod 664 test.gdb

My goal was to have the Firebird database accessible from a CGI script running from Apache.
The httpd process was running as user apache, so I had to add this user to the firebird group in /etc/group. I then restarted httpd to take into account the new group membership.

At this point, from my CGI script, I was able to open a connection to test.gdb via:

conn = kinterbasdb.connect(database='test.gdb', user='SYSDBA',
password='password')

I found the firebird-devel mailing list very helpful in solving some of the issues I faced. In particular, this message allowed me to get past the non-root user problems.

Third SoCal Piggies meeting

The SoCal Piggies had their third meeting at USC on May 17th. A record number of 7 Piggies attended (Daniel, Howard, Mark, Grig, Brian, Titus, Diane), joined by a guest from Caltech. It was nice to meet Howard, Brian and Mark and to talk about Python while munching on a fairly decent Pizza Hut nourishment.

Titus Brown presented part 2 of his Quixote 2.0 tutorial. He showed a few introductory examples, then delved into more advanced stuff, showing how to manage sessions with Quixote. One thing I took away from it is that it takes remarkably little code. Generally speaking, Quixote seems very crisp and clean, and Titus' tutorials are very good in smoothing the learning curve, which can otherwise be steep.

Titus then spent some time talking about his Web app. testing tool twill. We couldn't get a live demo unfortunately, but it was an interesting discussion nevertheless, covering things such as the cmd module, pyparsing, and especially a great use of metaclasses. Titus ended his presentation talking about his Cucumber module, another magical metaclass-heavy module that provides a mapping between Python code and SQL statements, hiding the complexity of SQL away from the programmer.

Daniel Arbuckle was next, and he gave us an overview of iterators, generators and continuations. He highly recommended the little-used itertools module, which provides functions designed to work with iterators. The functions are similar to standard Python built-ins like map and range and allow you to write code that is faster memory-efficient. Daniel then covered generators, which can be seen as a way to express iterators using functions as opposed to classes (if you see a yield statement anywhere in a function, that function is a generator). However, there are many other and sometimes slightly brain-melting uses of generators, such as cooperative multitasking and state machine control.

Daniel ended with a mind-boggling discussion of continuations, which (luckily or unluckily?) are not and will not be offered as a feature of Python, due to GvR's dislike of them. Especially interesting was the parallel Daniel made between continuations and the HTTP GET mechanism.

Many thanks to Titus and Daniel for their presentations.

The next meeting will be at USC again, on a date to be announced. Topics for next meeting:

• CherryPy tutorial (if Greg McClure will be able to attend the meeting)
• SimpleTAL and Pyrex tutorials (Diane)
• Others?

Installing and using cx_Oracle on Unix

Here's a mini-HOWTO on installing the cx_Oracle Python module on Unix systems (tested on Linux and Solaris.) I always forget one of these steps every time I install this module on one of my systems, so I thought I'd write them down. Other people might find this useful, so here goes:

1. Download cx_Oracle-4.1.tar.gz from here.
2. Become root on the target system.
3. Make sure you have the ORACLE_HOME environment variable defined for user root and pointing to the correct Oracle installation directory on your server. This step is necessary because the cx_Oracle installation process looks for headers and libraries under ORACLE_HOME.
4. Install cx_Oracle by running 'python setup.py install'.
5. Become user oracle, since this is the user you'll most likely want to run python scripts as when interacting with Oracle databases. Make sure you have the following line in oracle user's .bash_profile (or similar for other shells):

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib

If you don't add ORACLE_HOME/lib to your LD_LIBRARY_PATH, you'll get an error such as:

>>> import cx_Oracle
Traceback (most recent call last):
File "", line 1, in ?
ImportError: ld.so.1: python: fatal: libclntsh.so.9.0: open failed: No such file or directory

Now you're ready to use the cx_Oracle module. Here's a bit of code that shows how to connect to an Oracle database and run a SELECT query:

import cx_Oracle
from pprint import pprint

connection = cx_Oracle.Connection("%s/%s@%s" % (dbuser, dbpasswd, oracle_sid))
cursor = cx_Oracle.Cursor(connection)
sql = "SELECT * FROM your_table"
cursor.execute(sql)
data = cursor.fetchall()
print "(name, type_code, display_size, internal_size, precision, scale, null_ok)"
pprint(cursor.description)
pprint(data)
cursor.close()
connection.close()

New features in Selenium 0.3

A new version of the Selenium Web app test tool has just been released by Jason Huggins and the rest of the team at ThoughtWorks. I'll highlight some of the new and updated features.

New TestRunner look and feel

The most striking feature in version 0.3 is a new and improved look for the TestRunner. Click here for a screenshot.


Simplified directory structure

Another important improvement is a much simplified directory structure for the Selenium package. The main Selenium package contains the 'BrowserBot' JavaScript engine and nothing else. All the other language-specific drivers/bindings are available for download separately.

Here's all you need to do to get started with Selenium 0.3:

1. Download selenium-0.3.0.zip from the download page
2. Unzip the archive. This will create a directory called selenium-0.3.0, with 2 sub-directories: doc and selenium
3. Copy the selenium directory somewhere where it can be served by the Web server hosting your Web app
4. Open http://www.yourwebserver.com/selenium/TestRunner.html -- you will see a screen similar to the screenshot above
5. Start tinkering with the default test suite shipped with Selenium.

To add your own test suite, create an HTML file containing a table with links to your tests:

Custom Test Suite

Test1

Save this file as CustomTestSuite.html in the tests subdirectory of the selenium directory.

In the table above, Test1 is a link to the actual test file, which is another HTML file (called Test1.html) containing your custom test table. To have Selenium run your custom test suite, pass a ‘test’ parameter to TestRunner.html by opening an URL such as this:

http://www.yourwebserver.com/selenium/
TestRunner.html?test=./tests/CustomTestSuite.html
(note: this is a single line)

Improved browser support

In addition to Firefox/Mozilla on Windows/Linux/Mac OS X and IE on Windows, Selenium now supports Konqueror on Linux. There is also experimental support for Safari on Mac OS X.

Standalone server for win32 platforms

A standalone server is available for win32 platforms. This used to be known as the 'Twisted Server' implementation of Selenium, but now the fact that it is based on Twisted is hidden behind the scenes, so that new users don't have to jump through the various hoops of installing Python/Twisted/etc.

The main advantage of using the standalone version of Selenium is that it provides a workaround for the JavaScript 'same origin policy' security limitation. The same origin policy aims to prevent Cross-Site-Scripting attacks by preventing documents or scripts loaded from one origin from getting or setting properties of a document from a different origin. The net effect of all this is that in order to be able to use Selenium to test your Web application, you need to deploy the Selenium engine/framework on the server hosting the AUT (application under test). With the standalone Selenium server, you don't have this limitation and you can start doing what all new users to Selenium seem to want to do: test Google searches :-)

For users on non-Windows platforms, the Twisted Server version is still available via Subversion. See my previous blog entry for a tutorial on using the Twisted Server version. Many concepts discussed in there still apply to the standalone server version, in particular how to compose special URLs for testing 3rd party Web sites.

Plone Product version available for download

While Selenium has been available as a Plone Product version for quite some time, it was never part of the 'official' download page. Now you can find it among the Selenium 0.3 downloads as selenium-0.3.0-python-plone.zip.

For details on using Selenium with Plone, see 2 of my tutorials: part 1 and part 2.

Finally, here is a screen recording movie in AVI format that shows Selenium in action (note: choose 'Full screen' in your media player for better resolution.) The test run goes in 'Walk' mode through a new user registration test in a default, out-of-the-box installation of Plone. Enjoy!

New version of sparkplot on sparkplot.org

Due to positive feedback to my sparkplot post, I registered sparkplot.org and made it into a home for the sparkplot module.

I also added another option to sparkplot: -t or --transparency. Setting this option produces a transparent background for the generated PNG image (thanks to usagi for the suggestion).

I was glad to see somebody actually using the sparkplot module at the T&J blog. It looks like visualizing trading data is a natural application of sparklines. I also liked the term used by T to describe the act of embedding sparklines into blog entries: datablogging.

sparkplot: creating sparklines with matplotlib

Edward Tufte introduced sparklines in a sample chapter of his upcoming book "Beautiful Evidence". In his words, sparklines are "small, high-resolution graphics embedded in a context of words, numbers, images. Sparklines are data-intense, design-simple, word-sized graphics." Various implementations of sparklines that use different graphics packages have already been published, and probably the best known one is a PHP implementation that uses the GD backend. In what follows, I'll show some examples of sparkline graphics created via matplotlib by the sparkplot Python module I wrote.

Example 1

Since a picture is worth a thousand words, here is the Los Angeles Lakers' road to their NBA title in 2002. Wins are pictured with blue bars and losses with red bars. Note how easy it is to see the streaks for wins and losses.

The Lakers' 2004 season was their last with Shaq, when they reached the NBA finals and lost to Detroit (note the last 3 losses which sealed their fate in the finals).

Compare those days of glory with their abysmal 2005 performance, with only 2 wins in the last 21 games. Also note how the width of the last graphic is less than the previous 2, a consequence of the Lakers not making the playoffs this year.

Example 2

The southern oscillation is defined as the barometric pressure difference between Tahiti and the Darwin Islands at sea level. The southern oscillation is a predictor of El Nino which in turn is thought to be a driver of world-wide weather. Specifically, repeated southern oscillation values less than -1 typically defines an El Nino.

Here is a sparkline for the southern oscillation from
1955to 1992 (456 sample data points obtained from NIST). The sparkline is plotted with a horizontal span drawn along the x axis covering data values between -1 and 0, so that values less than -1 can be more clearly seen.

Example 3
Here is the per capita income in California from 1959 to 2003.
And here is the "real" per capita income (adjusted for inflation) in California, from 1959 to 2003.

Example 4

Here is the monthly distribution of messages sent to comp.lang.py from 1994 to 2004, plotted per year. Minimum and maximum values are shown with blue dots and labeled in the graphics.

Year		Total
1994		3,018
1995		4,026
1996		8,378
1997		12,910
1998		19,533
1999		24,725
2000		42,961
2001		55,271
2002		56,750
2003		64,548
2004		56,184

There was an almost constant increase in the number of messages per year, from 1994 to 2004, the only exception being 2004, when there were fewer message than in 2002 and 2003.

Details on using sparkplot

1) Install the Numeric Python module (required by matplotlib)
2) Install matplotlib
3) Prepare data files: sparkplot simplistically assumes that its input data file contains just 1 column of numbers
4) Run sparkplot.py. Here are some command-line examples to get you going:

- given only the input file and no other option, sparkplot.py will generate a gray sparkline with the first and last data points plotted in red:

sparkplot.py -i CA_real_percapita_income.txt

produces:

The name of the output file is by default .png. It can be changed with the -o option.

The plotting of the first and last data points can be disabled with the --noplot_first and --noplot_last options.

- given the input file and the label_first, label_last, format=currency options, sparkplot.py will generate a gray sparkline with the first and last data points plotted in red and with the first and last data values displayed in a currency format:

sparkplot.py -i CA_real_percapita_income.txt --label_first --label_last --format=currency

produces:

The currency symbol is $ by default, but it can be changed with the --currency option.

- given the input file and the plot_min, plot_max, label_min, label_max, format=comma options, sparkplot.py will generate a gray sparkline with the first and last data points plotted in red, with the min. and max. data points plotted in blue, and with the min. and max. data values displayed in a 'comma' format (e.g. 23,456,789):

sparkplot.py -i clpy_1997.txt --plot_min --plot_max --label_min --label_max --format=comma

produces:

- given the input file and the type=bars option, sparkplot.py will draw blue bars for the positive data values and red bars for the negative data values:

sparkplot.py -i lakers2005.txt --type=bars

produces:

As a side note, I think bar plots look better when the data file contains a relatively large number of data points, and the variation of the data is relatively small. This type of plots works especially well for sports-related graphics, where wins are represented as +1 and losses as -1.

- for other options, run sparkplot.py -h

I hope the sparkplot module will prove to be useful when you need to include sparkline graphics in your Web pages. All the caveats associated with alpha-level software apply :-) Let me know if you find it useful. I'm very much a beginner at using matplotlib, and as I become more acquainted with it I'll add more functionality to sparkplot.

Finally, kudos to John Hunter, the creator of matplotlib. I found this module extremely powerful and versatile. For a nice introduction to matplotlib, see also John's talk at PyCon05.

Note: the Blogger template system might have something to do with the fact that the graphics are shown with a border; when included in a "normal", white-background HTML page, there is no border and they integrate more seamlessly into the text.

Update 5/2/05: Thanks to Kragen Sitaker for pointing out a really simple solution to the "borders around images" problem -- just comment out the CSS definition for .post img in the Blogger template.

More on performance vs. load testing

I recently got some comments/questions related to my previous blog entry on performance vs. load vs. stress testing. Many people are still confused as to exactly what the difference is between performance and load testing. I've been thinking more about it and I'd like to propose the following question as a litmus test to distinguish between these two types of testing: are you actively profiling your application code and/or monitoring the server(s) running your application? If the answer is yes, then you're engaged in performance testing. If the answer is no, then what you're doing is load testing.

Another way to look at it is to see whether you're doing more of a white-box type testing as opposed to black-box testing. In the white-box approach, testers, developers, system administrators and DBAs work together in order to instrument the application code and the database queries (via specialized profilers for example), and the hardware/operating system of the server(s) running the application and the database (via monitoring tools such as vmstat, iostat, top or Windows PerfMon). All these activities belong to performance testing.

The black box approach is to run client load tools against the application in order to measure its responsiveness. Such tools range from lightweight, command-line driven tools such as httperf, openload, siege, Apache Flood, to more heavy duty tools such as OpenSTA, The Grinder, JMeter. This type of testing doesn't look at the internal behavior of the application, nor does it monitor the hardware/OS resources on the server(s) hosting the application. If this sounds like the type of testing you're doing, then I call it load testing.

In practice though the 2 terms are often used interchangeably, and I am as guilty as anyone else of doing this, since I called one of my recent blog entries "HTTP performance testing with httperf, autobench and openload" instead of calling it more precisely "HTTP load testing". I didn't have access to the application code or the servers hosting the applications I tested, so I wasn't really doing performance testing, only load testing.

I think part of the confusion is that no matter how you look at these two types of testing, they have one common element: the load testing part. Even when you're profiling the application and monitoring the servers (hence doing performance testing), you still need to run load tools against the application, so from that perspective you're doing load testing.

As far as I'm concerned, these definitions don't have much value in and of themselves. What matters most is to have a well-established procedure for tuning the application and the servers so that you can meet your users' or your business customers' requirements. This procedure will use elements of all the types of testing mentioned here and in my previous entry: load, performance and stress testing.

Here's one example of such a procedure. Let's say you're developing a Web application with a database back-end that needs to support 100 concurrent users, with a response time of less than 3 seconds. How would you go about testing your application in order to make sure these requirements are met?

1. Start with 1 Web/Application server connected to 1 Database server. If you can, put both servers behind a firewall, and if you're thinking about doing load balancing down the road, put the Web server behind the load balancer. This way you'll have one each of different devices that you'll use in a real production environment.

2. Run a load test against the Web server, starting with 10 concurrent users, each user sending a total of 1000 requests to the server. Step up the number of users in increments of 10, until you reach 100 users.

3. While you're blasting the Web server, profile your application and your database to see if there are any hot spots in your code/SQL queries/stored procedures that you need to optimize. I realize I'm glossing over important details here, but this step is obviously highly dependent on your particular application.

Also monitor both servers (Web/App and Database) via command line utilities mentioned before (top, vmstat, iostat, netstat, Windows PerfMon). These utilities will let you know what's going on with the servers in terms of hardware resources. Also monitor the firewall and the load balancer (many times you can do this via SNMP) -- but these devices are not likely to be a bottleneck at this level, since they usualy can deal with thousands of connections before they hit a limit, assuming they're hardware-based and not software-based.

This is one of the most important steps in the whole procedure. It's not easy to make sense of the output of these monitoring tools, you need somebody who has a lot of experience in system/network architecture and administration. On Sun/Solaris platforms, there is a tool called the SE Performance Toolkit that tries to alleviate this task via built-in heuristics that kick in when certain thresholds are reached and tell you exactly what resource is being taxed.

4. Let's say your Web server's reply rate starts to level off around 50 users. Now you have a repeatable condition that you know causes problems. All the profiling and monitoring you've done in step 3, should have already given you a good idea about hot spots in your applicationm about SQL queries that are not optimized properly, about resource status at the hardware/OS level.

At this point, the developers need to take back the profiling measurements and tune the code and the database queries. The system administrators can also increase server performance simply by throwing more hardware at the servers -- especially more RAM at the Web/App server in my experience, the more so if it's Java-based.

5. Let's say the application/database code, as well as the hardware/OS environment have been tuned to the best of everybody's abilities. You re-run the load test from step 2 and now you're at 75 concurrent users before performance starts to degrade.

At this point, there's not much you can do with the existing setup. It's time to think about scaling the system horizontally, by adding other Web servers in a load-balanced Web server farm, or adding other database servers. Or maybe do content caching, for example with Apache mod_cache. Or maybe adding an external caching server such as Squid.

One very important product of this whole procedure is that you now have a baseline number for your application for this given "staging" hardware environment. You can use the staging setup for nightly peformance testing runs that will tell you whether changes in your application/database code caused an increase or a decrease in performance.

6. Repeat above steps in a "real" production environment before you actually launch your application.

All this discussion assumed you want to get performance/benchmarking numbers for your application. If you want to actually discover bugs and to see if your application fails and recovers gracefully, you need to do stress testing. Blast your Web server with double the number of users for example. Unplug network cables randomly (or shut down/restart switch ports via SNMP). Take out a disk from a RAID array. That kind of thing.

The conclusion? At the end of the day, it doesn't really matter what you call your testing, as long as you help your team deliver what it promised in terms of application functionality and performance. Performance testing in particular is more art than science, and many times the only way to make progress in optimizing and tuning the application and its environment is by trial-and-error and perseverance. Having lots of excellent open source tools also helps a lot.

HTTP performance testing with httperf, autobench and openload

Update 02/26/07
--------
The link to the old httperf page wasn't working anymore. I updated it and pointed it to the new page at HP. Here's a link to a PDF version of a paper on httperf written by David Mosberger and Tai Jin: "httperf -- a tool for measuring Web server performance".

Also, openload is now OpenWebLoad, and I updated the link to its new home page.
--------

In this post, I'll show how I conducted a series of performance tests against a Web site, with the goal of estimating how many concurrent users it can support and what the response time is. I used a variety of tools that measure several variables related to HTTP performance.

• httperf is a benchmarking tool that measures the HTTP request throughput of a web server. The way it achieves this is by sending requests to the server at a fixed rate and measuring the rate at which replies arrive. Running the test several times and with monotonically increasing request rates, one can see the reply rate level off when the server becomes saturated, i.e., when it is operating at its full capacity.
• autobench is a Perl wrapper around httperf. It runs httperf a number of times against a Web server, increasing the number of requested connections per second on each iteration, and extracts the significant data from the httperf output, delivering a CSV format file which can be imported directly into a spreadsheet for analysis/graphing.
• openload is a load testing tool for Web applications. It simulates a number of concurrent users and it measures transactions per second (a transaction is a completed request to the Web server) and response time.
  

I ran a series of autobench/httperf and openload tests against a Web site I'll call site2 in the following discussion (site2 is a beta version of a site I'll call site1). For comparison purposes, I also ran similar tests against site1 and against www.example.com. The machine I ran the tests from is a Red Hat 9 Linux server co-located in downtown Los Angeles.

I won't go into details about installing httperf, autobench and openload, since the installation process is standard (configure/make/make install or rpm -i).

Here is an example of running httperf against www.example.com:

# httperf --server=www.example.com --rate=10 --num-conns=500

httperf --client=0/1 --server=www.example.com --port=80 --uri=/ --rate=10 --send-buffer=4096 --recv-buffer=16384 --num-conns=500 --num-calls=1
Maximum connect burst length: 1

Total: connections 500 requests 500 replies 500 test-duration 50.354 s

Connection rate: 9.9 conn/s (100.7 ms/conn, <=8 concurrent connections)
Connection time [ms]: min 449.7 avg 465.1 max 2856.6 median 451.5 stddev 132.1
Connection time [ms]: connect 74.1
Connection length [replies/conn]: 1.000

Request rate: 9.9 req/s (100.7 ms/req)
Request size [B]: 65.0

Reply rate [replies/s]: min 9.2 avg 9.9 max 10.0 stddev 0.3 (10 samples)
Reply time [ms]: response 88.1 transfer 302.9
Reply size [B]: header 274.0 content 54744.0 footer 2.0 (total 55020.0)
Reply status: 1xx=0 2xx=500 3xx=0 4xx=0 5xx=0

CPU time [s]: user 15.65 system 34.65 (user 31.1% system 68.8% total 99.9%)
Net I/O: 534.1 KB/s (4.4*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

The 3 arguments I specified on the command line are:

• server: the name or IP address of your Web site (you can also specify a particular URL via the --uri argument)
• rate: specifies the number of HTTP requests/second sent to the Web server -- indicates the number of concurrent clients accessing the server
• num-conns: specifies how many total HTTP connections will be made during the test run -- this is a cumulative number, so the higher the number of connections, the longer the test run
  

Here is a detailed interpretation of an httperf test run. In short, the main numbers to look for are the connection rate, the request rate and the reply rate. Ideally, you would like to see that all these numbers are very close to the request rate specified on the command line. If the actual request rate and the reply rate start to decline, that's a sign your server became saturated and can't handle any new connections. That could also be a sign that your client became saturated, so that's why it's better to test your client against a fast Web site in order to gauge how many outgoing HTTP requests can be sustained by your client.

Autobench is a simple Perl script that facilitates multiple runs of httperf and automatically increases the HTTP request rate. Configuration of autobench can be achieved for example by means of the ~/.autobench.conf file. Here is how my file looks like:

# Autobench Configuration File

# host1, host2
# The hostnames of the servers under test
# Eg. host1 = iis.test.com
# host2 = apache.test.com

host1 = testhost1
host2 = testhost2

# uri1, uri2
# The URI to test (relative to the document root). For a fair comparison
# the files should be identical (although the paths to them may differ on the
# different hosts)

uri1 = /
uri2 = /

# port1, port2
# The port number on which the servers are listening

port1 = 80
port2 = 80

# low_rate, high_rate, rate_step
# The 'rate' is the number of number of connections to open per second.
# A series of tests will be conducted, starting at low rate,
# increasing by rate step, and finishing at high_rate.
# The default settings test at rates of 20,30,40,50...180,190,200

low_rate = 10
high_rate = 50
rate_step = 10

# num_conn, num_call
# num_conn is the total number of connections to make during a test
# num_call is the number of requests per connection
# The product of num_call and rate is the the approximate number of
# requests per second that will be attempted.

num_conn = 200
#num_call = 10
num_call = 1

# timeout sets the maximimum time (in seconds) that httperf will wait
# for replies from the web server. If the timeout is exceeded, the
# reply concerned is counted as an error.

timeout = 60

# output_fmt
# sets the output type - may be either "csv", or "tsv";

output_fmt = csv

## Config for distributed autobench (autobench_admin)
# clients
# comma separated list of the hostnames and portnumbers for the
# autobench clients. No whitespace can appear before or after the commas.
# clients = bench1.foo.com:4600,bench2.foo.com:4600,bench3.foo.com:4600

clients = localhost:4600

The only variable I usually tweak from one test run to another is num_conn, which I set to the desired number of total HTTP connections to the server for that test run. In the example file above it is set to 200.

I changed the default num_call value from 10 to 1 (num_call specifies the number of HTTP requests per connection; I like to set it to 1 to keep things simple). I started my test runs with low_rate set to 10, high_rate set to 50 and rate_step set to 10. What this means is that autobench will run httperf 5 times, starting with 10 requests/sec and going up to 50 requests/sec in increments of 10.

When running the following command line...

# autobench --single_host --host1=www.example.com --file=example.com.csv

...I got this output and this CSV file.

Here is a graph generated via Excel from the CSV file obtained when running autobench against www.example.com for a different test run, with 500 total HTTP connections (the CSV file is here):



A few things to note about this typical autobench run:

• I chose example.com as an example of how an "ideal" Web site should behave
• the demanded request rate (in requests/second) starts at 10 and goes up to 50 in increments of 5 (x-axis)
• for each given request rate, the client machine makes 500 connections to the Web site
• the achieved request rate and the connection rate correspond to the demanded request rate
• the average and maximum reply rates are roughly equal to the demanded request rate
• the reponse time is almost constant, around 100 msec
• the are no HTTP errors

What this all means is that the example.com Web site is able to easily handle up to 50 req/sec. The fact that the achieved request rate and the connection rate increase linearly from 10 to 50 also means that the client machine running the test is not the bottleneck. If the demanded request rate were increased to hundreds of req/sec, then the client will not be able to keep up with the demanded requests and it will become the bottleneck itself. In these types of situations, one would need to use several clients in parallel in order to bombard the server with as many HTTP requests as it can handle. However, the client machine I am using is sufficient for requests rates lower than 50 req/sec.

Here is an autobench report for site1 (the CSV file is here):



Some things to note about this autobench run:

• I specified only 200 connections per run, so that the server would not be over-taxed
• the achieved request rate and the connection rate increase linearly with the demanded request rate, but then level off around 40
• there is a drop at 45 req/sec which is probably due to the server being temporarily overloaded
• the average and maximum reply rates also increase linearly, then level off around 39 replies/sec
• the response time is not plotted, but it also increases linearly from 93 ms to around 660 ms

To verify that 39 is indeed the maximum reply rate that can be achieved by the Web server, I ran another autobench test starting at 10 req/sec and going up to 100 req/sec in increments of 10 (the CSV file is here):



Observations:

• the reply rate does level off around 39 replies/sec and actually drops to around 34 replies/sec when the request rate is 100
• the response time (not plotted) increases linearly from 97 ms to around 1.7 sec

We can conclude that the current site1 Web site can sustain up to around 40 requests/second before it becomes saturated. At higher request rates, the response time increases and users will start experiencing time-outs.

Here is an autobench report for site2 (the CSV file is here):



Some things to note about this autobench run:

• the achieved request rate and the connection rate do not increase with the demanded request rate; instead, they are both almost constant, hovering around 6 req/sec
• the average reply rate also stays relatively constant at around 6 replies/sec, while the maximum reply rate varies between 5 and 17
• there is a dramatic increase in response time (not plotted) from 6 seconds to more than 18 seconds

From this initial run, we can see that the average reply rate does not exceed 6-7 replies/second, so this seems to be the limit for the site2 Web site. In order to further verify this hypothesis, I ran another autobench test, this time going from 1 to 10 requests/second, in increments of 1. Here is the report (the CSV file is here):



Some things to note about this autobench run:

• the achieved request rate and the connection rate increase linearly with the demanded request rate from 1 to 6, then level off around 6
• the average reply rate is almost identical to the connection rate and also levels off around 6
• the maximum reply rate levels off around 8
• the reponse time (not plotted) increases from 226 ms to 4.8 seconds

We can conclude that the site2 Web site can sustain up to 7 requests/second before it becomes saturated. At higher request rates, the response time increases and users will start experiencing time-outs.

Finally, here are the results of a test run that uses the openload tool in order to measure transactions per second (equivalent to httperf's reply rate) and reponse time (the CSV file is here):



Some notes:

• the transaction rate levels off, as expected, around 6 transactions/sec
• the average response time levels off around 7 seconds, but the maximum response time varies considerably from 3 to around 20 seconds, reaching up to 30 seconds

These results are consistent with the ones obtained by running httperf via autobench. From all these results, we can safely conclude that in its present state, the site2 Web site is not ready for production, unless more than 6-7 concurrent users are never expected to visit the site at the same time. The response time is very high and the overall user experience is not a pleasant one at this time. Also, whenever I increased the load on the site (for example by running autobench with 200 through 500 connections per run), the site became almost instantly un-responsive and ended up sending HTTP errors back to the client.

Conclusion

The tools I described are easy to install and run. The httperf request/reply throughput measurements in particular prove to be very helpful in pinpointing HTTP bottlenecks. When they are corroborated with measurements from openload, an overall picture emerges that is very useful in assessing HTTP performance numbers such as concurrent users and response time.

Update

I got 2 very un-civil comments from the same Anonymous Coward-type poster. This poster called my blog entry "amateurish" and "recklessly insane" among other things. One slightly more constructive point made by AC is a question: why did I use these "outdated" tools and not other tools such as The Grinder, OpenSTA and JMeter? The answer is simple: I wanted to use command-line-driven, lightweight tools that can be deployed on any server, with no need for GUIs and distributed installations. If I were to test a large-scale Web application, I would certainly look into the heavy-duty tools mentioned by the AC. But the purpose of my post was to show how to conduct a very simple experiment that can still furnish important results and offer a good overall picture about a Web site's behavior under moderate load.