Interview with Jerry Weinberg at Citerus

Read it here. As usual, Jerry Weinberg has many thought-provoking things to say. My favorite:

"Q: If you're the J.K Rowling of software development, who's Harry P then?

A: Well, first of all, I'm not a billionaire, so it's probably not correct to say I'm the J.K. Rowling of software development. But if I were, I suspect my Harry Potter would be a test manager, expected to do magic but discounted by software developers because "he's only a tester." As for Voldemort, I think he's any project manager who can't say "no" or hear what Harry is telling him."

Testers are finally redeemed :-)

Vonnegut's last interview

Via Tim Ferriss's blog, an inspiring interview with Kurt Vonnegut. BTW, if you haven't read Tim's '4-hour workweek' book, I highly recommend it. It will make you green with envy, but it will also offer you some good ideas about organizing your work and your life a bit differently.

PyCon'08 Testing Tutorial proposal

It's that time of the year, when the PyCon organizers are asking for talk, panel, and tutorial proposals. Titus and I are thinking about doing a three-peat of our Testing tutorial, but this time....with a twist. Read about it on Titus's blog; then send us your code/application that you'd like to test, or problems you have related to testing. Should be lots of fun.

Pybots updates

Time for the periodical update on the Pybots project. Since my last post in July, John Hampton added a buildslave running Gentoo on x86 and testing Trac and SQLAlchemy. A belated thank you to John.

I also had to disable the tests for bzr dev on my RH 9 buildslave, because for some reason they were leaving a lot of orphaned/zombie processes around.

With help from Jean-Paul Calderone from the Twisted team, we managed to get the Twisted buildslave (running RH 9) past some annoying multicast-related failures. Jean-Paul had me add an explicit iptables rule to allow multicast traffic. The rule is:

iptables -A INPUT -j ACCEPT -d 225.0.0.0/24

This seemed to have done the trick. There are some Twisted unit tests that still fail -- some of them are apparently due to the fact that raising string exceptions is now illegal in the Python trunk (2.6). Jean-Paul will investigate and I'll report on the findings -- after all, this type of issues is exactly why we set up the Pybots farm in the first place.

As usual, I end with a plea to people interested in running Pybots buidlslaves to either send a message to the mailing list, or contact me directly at grig at gheorghiu dot net.

Compiling mod_python on RHEL 64 bit

I just went through the fairly painful exercise of compiling mod_python 3.3.1 on a 64-bit RHEL 5 server. RHEL 5 ships with Python 2.4.3 and mod_python 3.2.8. I needed mod_python to be compiled against Python 2.5.1. I had already compiled and installed Python 2.5.1 from source into /usr/local/bin/python2.5. The version of Apache on that server is 2.2.3.

I first tried this:

# tar xvfz mod_python-3.3.1.tar.gz
# cd mod_python-3.3.1
# ./configure --with-apxs==/usr/sbin/apxs --with-python=/usr/local/bin/python2.5
# make

...at which point I got this ugly error:

/usr/lib64/apr-1/build/libtool --silent --mode=link gcc -o mod_python.la \
-rpath /usr/lib64/httpd/modules -module -avoid-version    finfoobject.lo \
hlistobject.lo hlist.lo filterobject.lo connobject.lo serverobject.lo util.lo \
tableobject.lo requestobject.lo _apachemodule.lo mod_python.lo\
-L/usr/local/lib/python2.5/config -Xlinker -export-dynamic -lm\
-lpython2.5 -lpthread -ldl -lutil -lm
/usr/bin/ld: /usr/local/lib/python2.5/config/libpython2.5.a(abstract.o):
relocation R_X86_64_32 against `a local symbol' can not be used when making a shared object;
recompile with -fPIC
/usr/local/lib/python2.5/config/libpython2.5.a: could not read symbols: Bad value
collect2: ld returned 1 exit status
apxs:Error: Command failed with rc=65536

I googled around for a bit, and I found this answer courtesy of Martin von Loewis. To quote:

It complains that some object file of Python wasn't compiled
with -fPIC (position-independent code). This is a problem only if
a) you are linking a static library into a shared one (mod_python, in this case), and
b) the object files in the static library weren't compiled with -fPIC, and
c) the system doesn't support position-dependent code in a shared library

As you may have guessed by now, it is really c) which I
blame. On all other modern systems, linking non-PIC objects
into a shared library is supported (albeit sometimes with a
performance loss on startup).

So your options are
a) don't build a static libpython, instead, build Python
with --enable-shared. This will give you libpython24.so
which can then be linked "into" mod_python
b) manually add -fPIC to the list of compiler options when
building Python, by editing the Makefile after configure has run
c) find a way to overcome the platform limitation. E.g. on
Solaris, the linker supports an impure-text option which
instructs it to accept relocations in a shared library.

You might wish that the Python build process supported
option b), i.e. automatically adds -fPIC on Linux/AMD64.
IMO, this would be a bad choice, since -fPIC itself usually
causes a performance loss, and isn't needed when we link
libpython24.a into the interpreter (which is an executable,
not a shared library).

Therefore, I'll close this as "won't fix", and recommend to
go with solution a).

So I proceeded to reconfigure Python 2.5 via './configure --enable-shared', then the usual 'make; make install'. However, I hit another snag right away when trying to run the new python2.5 binary:

# /usr/local/bin/python
python: error while loading shared libraries: libpython2.5.so.1.0: cannot open shared object file: No such file or directory

I remembered from other issues I had similar to this that I have to include the path to libpython2.5.so.1.0 (which is /usr/local/lib) in a ldconfig configuration file.

I created /etc/ld.so.conf.d/python2.5.conf with the contents '/usr/local/lib' and I ran

# ldconfig

At this point, I was able to run the python2.5 binary successfully.

I then re-configured and compiled mod_python with

# ./configure --with-apxs=/usr/sbin/apxs --with-python=/usr/local/bin/python2.5
# make

Finally, I copied mod_python.so from mod_python-3.3.1/src/.libs to /etc/httpd/modules and restarted Apache.

Not a lot of fun, that's all I can say.

Update 10/23/07

To actually use mod_python, I had to also copy the directory mod_python-3.3.1/lib/python/mod_python to /usr/local/lib/python2.5/site-packages. Otherwise I would get lines like these in the apache error_log when trying to hit a mod_python-enabled location:

[Mon Oct 22 19:41:20 2007] [error] make_obcallback: \
could not import mod_python.apache.\n \
ImportError: No module named mod_python.apache
[Mon Oct 22 19:41:20 2007] [error] make_obcallback:
Python path being used \
"['/usr/local/lib/python2.5/site-packages/setuptools-0.6c6-py2.5.egg', \
'/usr/local/lib/python25.zip', '/usr/local/lib/python2.5', \
'/usr/local/lib/python2.5/plat-linux2', \
'/usr/local/lib/python2.5/lib-tk', \
'/usr/local/lib/python2.5/lib-dynload', '/usr/local/lib/python2.5/site-packages']".
[Mon Oct 22 19:41:20 2007] [error] get_interpreter: no interpreter callback found.

Update 01/29/08

I owe Graham Dumpleton (the creator of mod_python and mod_wsgi) an update to this post. As he added in the comments, instead of manually copying directories around, I could have simply said:

make install

and the installation would have properly updated the site-packages directory under the correct version of python (2.5 in my case) -- this is because I specified that version in the --with-python option of ./configure.

Another option for the installation, if you want to avoid copying the mod_python.so file in the Apache modules directory, and only want to copy the Python files in the site-packages directory, is:

make install_py_lib

Update 06/18/10

From Will Kessler:

"You might also want to add a little note though. The error message may actually be telling you that Python itself was not built with --enable-shared. To get mod_python-3.3.1 working you need to build Python with -fPIC (use enable-shared) as well."

What's more important: TDD or acceptance testing?

The answer, as far as I'm concerned, is 'BOTH'. Read these entertaining blog posts to see why: Roy Osherove's JAOO conference writeup (his take on Martin Fowler's accent cracked me up), Martin Jul's take on the pull-no-punches discussions on TDD between Roy O. and Jim Coplien, and also Martin Jul's other blog post on why acceptance tests are important.

As I said before, holistic testing is the way to go.

Roy Osherove book on "The art of unit testing"

Just found out from Roy Osherove's blog that his book on "The Art of Unit Testing" is available for purchasing online -- well, the first 5 chapters are, but then you get the next as they're being published. Roy uses NUnit to illustrate unit testing concepts and techniques, but that shouldn't deter you from buying the book, because the principles are pretty much the same in all languages. I'm a long time reader of Roy's blog and I can say this is good stuff, judging by his past posts on unit testing and mock testing techniques.

Beware of timings in your tests

Finally I get to write a post about testing. Here's the scenario I had to troubleshoot yesterday: a client of ours has a Web app that uses a java applet for FTP transfers to a back-end server. The java applet presents a nice GUI to end-users, allowing them to drag and drop files from their local workstation to the server.

The problem was that some file transfers were failing in a mysterious way. We obviously looked at the network connectivity between the user reporting the problem initially and our data center, then we looked at the size of the files he was trying to transfer (he thought files over 10 MB were the culprit). We also looked at the number of files transferred, both multiple files in one operation and single files in consecutive operations. We tried transferring files using both a normal FTP client, and the java applet. Everything seemed to point in the direction of 'works for me' -- a stance well-known to testers around the world. All of a sudden, around an hour after I started using the java applet to transfer files, I got the error 'unable to upload one or more files', followed by the message 'software caused connection abort: software write error'. I thought OK, this may be due to web sessions timing out after an hour. I did some more testing, and the second time I got the error after half an hour. I also noticed that I let some time pass between transfers. This gave me the idea of investigating timeout setting on the FTP server side (which was running vsftpd). And lo and behold, here's what I found in the man page for vsftpd.conf:

idle_session_timeout
The timeout, in seconds, which is the maximum time a remote client may spend between FTP commands. If the timeout triggers, the remote client is kicked off.

Default: 300

My next step was of course to wait 5 minutes between file transfers, and sure enough, I got the 'unable to upload one or more files' error.

Lesson learned: pay close attention to the timing of your tests. Also look for timeout settings both on the client and on the server side, and write corner test cases accordingly.

In the end, it was by luck that I discovered the cause of the problems we had, but as Louis Pasteur said, "Chance favors the prepared mind". I'll surely be better prepared next time, timing-wise.

Barack Obama is now a connection

That's the message I see on my LinkedIn home page. How could this be possible, you ask? Well, yesterday I checked out my home page, and I noticed the 'featured question of the day' asked by Barack Obama himself (of course, the question was "how can the next president better help small businesses and entrepreneurs thrive".) A co-worker decided to send a LinkedIn invite to Barack. A little while later, he got the acceptance in his inbox. I followed his example, just for fun, and what do you know, I got back the acceptance in a matter of SECONDS, not even minutes! It seems that B.O. has set his LinkedIn account to accept each and every invite he gets. I guess when you're running for president, every little statistic counts. He already has 500+ connections, and I'm sure the time will come when he'll brag to the other candidates that his LinkedIn account is bigger than theirs.

The bottom line is that YOU TOO can have Barack as your connection, if only to brag to your friends about it.

I must be bored out of my mind

...otherwise why would I have taken the time to get my nerd score?





Now YOU do it!

Security testing book review on Dr. Dobbs site

I wrote a review for "The Art of Security Testing" a while ago for Dr. Dobbs. I found out only now that it's online at the Dr. Dobbs's Portal site. Read it here.

Weinberg on Agile

A short but sweet PM Boulevard interview with Jerry Weinberg on Agile management/methods. Of course, he says we need to drop the A and actually drop 'agile' altogether at some point, and just talk about "normal, sensible, professional methods of developing software." Count me in.

Jakob Nielsen on fancy formatting and fancy words

Just received the latest Alertbox newsletter from Jakob Nielsen. The topic is "Fancy Formatting, Fancy Words = Ignored". I'd have put 2 equal signs in there, but anyway....The 'ignored' in question is your web site, if you're trying to draw attention to important facts/figures by using red bold letters and pompous language. Nielsen's case study in the article is the U.S. Census Bureau's homepage, which displayed the current population of the US in big red bold letters, and called it "Population clock". As a result, users were confused as to the meaning of that number, and what's more, they didn't bother to even read the full number, because they thought it's an ad of some sort. Interesting stuff.

Some notes from the August SoCal Piggies meeting

Read them here.

Put your Noonhat on

You may have seen this already, but here's another short blurb from me: Brian Dorsey, a familiar face to those of you who have been at the last 2 or 3 PyCon conferences, has launched a Django-based Web site he called Noonhat. The tagline says it all: "Have a great lunch conversation". It's a simple but original, fun and hopefully viral idea: you specify your location on a map, then you indicate your availability for lunch, and Noonhat puts you in touch with other users who have signed up and are up for lunch in your area at that time.

This has potential not only for single people trying to find a date, but also for anybody who's unafraid of stepping out of their comfort zone and strike interesting conversations over lunch. Brian and his site have already been featured on a variety of blogs and even in mainstream media around Seattle. Check out the Noonhat blog for more details.

Well done, Brian, and may your site prosper (of course, the ultimate in prosperity is being bought by Google :-)

Fuzzing in Python

I just bought "Fuzzing: Brute Force Vulnerability Discovery" and skimmed it a bit. I was pleasantly surprised to see that Python is the language of choice for many fuzzing tools, and clearly the favorite language of the authors, since they implemented many of their tools in Python. See the fuzzing.org site/blog also, especially the Fuzzing software page. Sulley in particular seems a very powerful fuzzing framework. I need to look more into it (so much cool stuff, so little time.)

Update: got through the first 5-6 chapters of the book. Highly entertaining and educational so far.

Werner Vogels talk at QCon

Werner Vogels is the CTO of Amazon. You can watch a talk he gave at the QCon conference on the topics of Availability and Consistency. The bottom line is that, as systems scale (and for amazon.com that means hundreds of thousands of systems), you have to pick 2 of the following 3: Consistency, Availability, Partitioning (actually the full name of the third one is "Tolerance to network partitioning.) This is called the CAP theorem, and Eric Brewer from Inktomi first came up with it.

Vogels pretty much equated partitioning with failure. Failure is inevitable, so you have to choose it out of those 3 properties. You're left with a choice between consistency and availability, or between ACID and BASE. According to Vogels, it turns out there's also a middle-of-the-road approach, where you choose a specific approach based on the needs of a particular service. He gave the example of the checkout process on amazon.com. When customers want to add items to their shopping cart, you ALWAYS want to honor that request (obviously because that's $$$ in the bank for you). So you choose high availability, and you hide errors from the customers in the hope that the system will sort out the errors at a later stage. When the customer hits the 'Submit order' button, you want high consistency for the next phase, because several sub-systems access that data at the same time (credit card processing, shipping and handling, reporting, etc.).

I also liked the approach Amazon takes when splitting people into teams. They have the 2-pizza rule: if it takes more than 2 pizzas to feed a team, it means the team is too large and needs to be split up. This equates to about 8 people per team. They actually make architectural decisions based on team size. If a feature is deemed to large to be comprehended by a team of 8 people, they split the feature into smaller pieces that can be digested more easily. Very agile approach :-)

Anyway, good presentation, highly recommended.

Automating tasks with pexpect

I started to use pexpect for some of the automation needs I have, especially for tasks that involve logging into a remote device and running commands there. I found the module extremely easy to use, and the documentation on the module's home page is very good. Basically, if you follow the recipe shown there for logging into an FTP server, you're set.

A couple of caveats I discovered so far:

• make sure you specify correctly the text you expect back; even an extra space can be costly, and make your script wait forever; you can add '.*' to the beginning or to the end of the text you're expecting to make sure you're catching unexpected characters
• if you want to print the output from the other side of the connection, use child.before (where child is the process spawned by pexpect)

Here's a complete script for logging into a load balancer and showing information about a load balanced server and its real servers:

#!/usr/bin/env python

import pexpect

def show_virtual(child, virtual):
    child.sendline ('show server virtual %s' % virtual)
    child.expect('SSH@MyLoadBalancer>')
    print child.before

def show_real(child, real):
    child.sendline ('show server real %s' % real)
    child.expect('SSH@MyLoadBalancer>')
    print child.before

virtuals = ['www.mysite.com']
reals = ['web01', 'web02']

child = pexpect.spawn ('ssh myadmin@myloadbalancer')
child.expect ('.* password:')
child.sendline ('mypassword')
child.expect ('SSH@MyLoadBalancer>')

for virtual in virtuals:
    show_virtual(child, virtual)

for real in reals:
    show_real(child, real)

child.sendline ('exit')

Think twice before working from a Starbucks

Here's an eye-opening article talking about a tool called Hamster that sniffs wireless traffic and reveals plain-text cookies which can then be used to impersonate users. The guy running the tool was able to log in into some poor soul's Gmail account during a BlackHat presentation.

Pretty scary, and it makes me think twice before firing up my laptop in a public wireless hotspot. The people who wrote Hamster, from Errata Security, already released another tool called Ferret, which intercepts juicy bits of information -- they call it 'information seepage'. You can see a presentation on Ferret here. They're supposed to release Hamster into the wild any day now.

Update: If the above wasn't enough to scare you, here's another set of wireless hacking tools called Karma (see the presentation appropriately called "All your layers are belong to us".)

That's what I call system testing

According to news.com, the IT systems for the 2008 Olympics in Beijing will be put through rigorous testing which will take more than 1 year! The people at Atos Origin, the company in charge of setting up the IT for the 2008 Olympics, clearly know what they are doing.

It's also interesting that the article mentions insiders as a security threat -- namely, that insiders will try to print their own accreditation badges, or do it for their friends, etc. As always, the human factor is the hardest to deal with. They say they resort to extensive background checks for the 2,500 or so IT volunteers, but I somehow doubt that will be enough.