Posts

Showing posts from 2008

Some issues when restoring files using duplicity

I blogged a while back about how to do incremental encrypted backups to S3 using duplicity. I've been testing the restore procedure for some of my S3 backups, and I had a problem with the way duplicity deals with temporary directories and files it creates during the restore.

By default, duplicity will use the system default temporary directory, which on Unix is usually /tmp. If you have insufficient disk space in /tmp for the files you're trying to restore from S3, the restore operation will eventually fail with "IOError: [Errno 28] No space left on device".

One thing you can do is create another directory on a partition with lots of disk space, and specify that directory in the duplicity command line using the --tempdir command line option. Something like: /usr/local/bin/duplicity --tempdir=/lotsofspace/temp

However, it turns out that this is not sufficient. There's still a call to os.tmpfile() buried in the patchdir.py module installed by duplicity. Consequently, …

Working with Amazon EC2 regions

Now that Amazon offers EC2 instances based in data centers in Europe, there is one more variable that you need to take into account when using the EC2 API: the concept of 'region'. Right now there are 2 regions to choose from: us-east-1 (based of course in the US on the East Coast), and the new region eu-west-1 based in Western Europe. Knowing Amazon, they will probably launch data centers in other regions across the globe -- Asia, South America, etc.

Each region has several availability zones. You can see the current ones in this nice article from the AWS Developer Zone. The default region is us-east-1, with 3 availability zones (us-east-1a, 1b and 1c). If you don't specify a region when you call an EC2 API tool, then the tool will query the default region. That's why I was baffled when I tried to launch a new AMI in Europe; I was calling 'ec2-describe-availability-zones' and it was returning only the US ones. After reading the article I mentioned, I realized I…

Deploying EC2 instances from the command line

I've been doing a lot of work with EC2 instances lately, and I wrote some simple wrappers on top of the EC2 API tools provided by Amazon. These tools are Java-based, and I intend to rewrite my utility scripts in Python using the boto library, but for now I'm taking the easy way out by using what Amazon already provides.

After downloading and unpacking the EC2 API tools, you need to set the following environment variables in your .bash_profile file:
export EC2_HOME=/path/to/where/you/unpacked/the/tools/api
export EC2_PRIVATE_KEY = /path/to/pem/file/containing/your/ec2/private/key
export EC2_CERT = /path/to/pem/file/containing/your/ec2/cert
You also need to add $EC2_HOME/bin to your PATH, so the command-line tools can be found by your scripts.

At this point, you should be ready to run for example:
# ec2-describe-images -o amazon
which lists the AMIs available from Amazon.

If you manage more than a handful of EC2 AMIs (Amazon Machine Instances), it quickly becomes hard to keep track of th…

New job at OpenX

I meant to post this for a while, but haven't had the time, because...well, it's a new job, so I've been quite swamped. I started 2 weeks ago as a system engineer at OpenX, a company based in Pasadena, whose main product is an Open Source ad server. I am part of the 'black ops' team, and my main task for now is to help with deploying and scaling the OpenX Hosted service within Amazon EC2 -- which is just one of several cloud computing providers that OpenX uses (another one is AppNexus for example).

Lots of Python involved in this, lots of automation, lots of testing, so all this makes me really happy :-)

Here is some stuff I've been working on, which I intend to post on with more details as time permits:

* command-line provisioning of EC2 instances
* automating the deployment of the OpenX application and its pre-requisites
* load balancing in EC2 using HAProxy
* monitoring with Hyperic
* working with S3-backed file systems

I'll also start working soon with slack, a…

The sad state of open source monitoring tools

I've been looking lately at open source network monitoring tools. I'm not impressed at all by what I've seen so far. Pretty much the least common denominator when it comes to this type of tools is Nagios, which is not a bad tool (I used it a few years ago), but did you see its Web interface? It's soooooo 1999 -- think 'Perl CGI scripts'!

A slew of other tools are based on the Nagios engine, and are trying hard to be more pleasing to the eye -- Opsview and GroundWork are some examples. Opsview seems just a wrapper around Nagios, with not a lot of improvements in terms of both functionality and UI.

I looked at the GroundWork screencast and it seemed promising, but when I tried to install it I had a very unpleasant experience. First of all, the install script uses curses (did those guys hear about unattended installs?), and requires Java 1.5. Although I had both Java 1.5 and 1.6 on my CentOS server, and JAVA_HOME set correctly, it didn't stop the installer from …

Issues with Ubuntu 8.10 on Lenovo T61p laptop

I got a new Lenovo ThinkPad T61p, and of course I promptly installed Ubuntu Ibex 8.10 on it. The first day I used it, I had no issues, but this morning it froze no less than 3 times, and each time the Caps Lock light flashed. I googled around, and I found what I hope is the solution in this post on the Ubuntu forums. It seems that this is the core issue:

System lock-ups with Intel 4965 wireless

The version of the iwlagn wireless driver for Intel 4965 wireless chipsets included in Linux kernel version 2.6.27 causes kernel panics when used with 802.11n or 802.11g networks. Users affected by this issue can install the linux-backports-modules-intrepid package, to install a newer version of this driver that corrects the bug. (Because the known fix requires a new version of the driver, it is not expected to be possible to include this fix in the main kernel package.)

As recommended, I did 'apt-get install backports-modules-intrepid' and I rebooted. That was around 1 hour ago, and I h…

Python and MS Azure

You've probably heard by now of Microsoft's entry in the cloud computing race, dubbed Azure. What I didn't know until I saw it this morning on InfoQ was that Microsoft encourages the use of languages and tools other than their official ones. Here's what they say on the 'What is the Azure Service Platform' page:

"Windows Azure is an open platform that will support both Microsoft and non-Microsoft languages and environments. Windows Azure welcomes third party tools and languages such as Eclipse, Ruby, PHP, and Python."

While you and I may think MS says this just for marketing/PR purposes, it turns out they are walking the walk a bit. I was glad to see in the InfoQ article that a Microsoft guy wrote a Python wrapper on top of the Azure Data Storage APIs. Note that this is classic CPython, not IronPython. I assume more interesting stuff can be done with IronPython.

"phrase from nearest book" meme

Via Elliot:

Grab the nearest book.Open it to page 56.Find the fifth sentence.Post the text of the sentence in your journal along with these instructions.Don’t dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.Here's mine, from 'Kim' by Kipling:

"A little later a marriage procession would strike into the Grand Trunk with music and shoutings, and a smell of marigold and jasmine stronger even than the reek of the dust."

Not bad, I like it :-)

Migrating SSL certs from IIS to Apache

If you ever find yourself facing this task, use this guide -- worked perfectly for me.

I work for this guy

Image
Best Halloween costume I've ever seen. You rock Nate!!!

This is depressing: Ken Thompson is also a googler

Just found out that Ken Thompson, one of the creators of Unix, works at Google. You can see his answers to various questions addressed to Google engineers by following the link with his name on this page.

So let's see, Google has hired:

* Ken Thompson == Unix
* Vint Cerf == TCP/IP
* Andrew Morton == #2 in Linux
* Guido van Rossum == Python
* Ben Collins-Sussman and Brian Fitzpatrick == subversion
* Bram Moolenaar == vim

...and I'm sure there are countless others that I missed.

If this isn't a march towards world domination, I don't know what is :-)

The case of the missing profile photo

Earlier today I posted a blog entry, then I went to view it on my blog, only to notice that my profile photo was conspicuously absent. I double-checked the URL for the source of the image -- it was http://agile.unisonis.com/gg.jpg. Then I remembered that I recently migrated agile.unisonis.com to my EC2 virtual machine. I quickly ssh-ed into my EC2 machine and saw that the persistent storage volume was not mounted. I ran uptime and noticed that it only showed 8 hours, so the machine had somehow been rebooted. In my experiments with setting up that machine, I had failed to add a line to /etc/fstab that causes the persistent storage volume to be mounted after the rebooted. Easily rectified:

echo "/dev/sds /ebs1 ext3 defaults 0 0" >> /etc/fstab

I connected to my EC2 environment with ElasticFox and saw that the EBS volume was still attached to my machine instance as /dev/sds, so I mounted it via 'mount /dev/sds/ /ebs1', then restart…

Recommended book: "Scalable Internet Architectures"

One of my co-workers, Nathan, introduced me to this book -- "Scalable Internet Architectures" by Theo Schlossnagle. I read it in one sitting. Recommended reading for anybody who cares about scaling their web site in terms of both web/application servers and database servers. It's especially appropriate in our day and age, when cloud computing is all the rage (more on this topic in another series of posts). My preferred chapters were "Static Content Serving" (talks about wackamole and spread) and "Static Meets Dynamic" (talks about web proxy caches such as squid).

I wish the database chapter contained more in-depth architectural discussions; instead, the author spends a lot of time showing a Perl script that is supposed to illustrate some of the concepts in the chapter, but falls very short of that in my opinion.

Overall though, highly recommended.

Example Django app needed

Dear lazyweb, I need a good sample Django application (with a database backend) to run on Amazon EC2. If the application has Ajax elements, even better.

Comments with suggestions would be greatly appreciated!

Update on EC2 and EBS

I promised I'll give an update on my "Experiences with Amazon EC2 and EBS" post from a month ago. Well, I just got an email from Amazon, telling me:

Greetings from Amazon Web Services,

This e-mail confirms that your latest billing statement is available on the AWS web site. Your account will be charged the following:

Total: $73.74

So there you have it. That's how much it cost me to run the new SoCal Piggies wiki, as well as some other small sites, with very little traffic. Your mileage will definitely vary, especially if you run a high-traffic site.

I also said I'll give an update on running a MySQL database on EBS. It turns out it's really easy. On my Fedora Core 8 AMI, I did this:

* installed mysql packages via yum:

yum -y install mysql mysql-server mysql-devel

* moved the default data directory for mysql (/var/lib/mysql) to /ebs1/mysql (where /ebs1 is the mount point of my 10 GB EBS volume), then symlinked /ebs1/mysql back to /var/lib, so that everything continues…

Presubmit testing at Google

Here is an interesting blog post from Marc Kaplan, test engineering manager at Google, on their strategy of running what they call 'presubmit tests' -- tests that are run automatically before the code gets checked in. They include performance tests, and they compare the performance of the new code with baselines from the previous week, then report back nice graphs showing the delta. Very cool.

"Unmaintained Free Software" wiki

Thanks to Heikki Toivonen, who left a comment to my previous post and pointed me to this Unmaintained Free Software wiki. Python-related projects on that site are here. Hmmm...RPM is an unmaintained Python project? Don't think so. That site could use some love...maybe it is itself in need of a maintainer? This seems like a good Google App Engine project -- to put together a similar site with a database back-end, showing unmaintained Open Source projects....

Know of any Open Source projects that need maintainers?

I got an email on a testing-related mailing list from somebody who would like to take over an Open Source project with no current maintainer. Here's a fragment of that email:

"Folks,
As I am interested in brushing up on my coding skills, so I would
appreciate your help in identifying an existing orphan/dormant
open-source tool/toolset project who needs an owner/maintainer.

I am especially interested in software process-oriented tools that
fill a hole in an agile development/test/management tool stack."

If anybody knows of such projects, especially with a testing or agile bent, please leave a comment here. Thanks!

Getting around the Firefox port-blocking annoyance

Firefox 3.x has introduced something I'm sure they call a 'feature', but is a major annoyance for any sysadmin and developer -- they block access to ports other than 80. I thought IE was the only browser that was brain-dead that way, but Firefox has proved me wrong. Anyway, here's a simple recipe for getting around this:

1) go to about:config in the Firefox address bar
2) right click, choose new->string
3) enter the name network.security.ports.banned.override and the value 1-65535
4) there is no step 4

Experiences with Amazon EC2 and EBS

I decided to port some of the sites I've been running for the last few years on a dedicated server (running RHEL9) to an Amazon EC2 AMI (which stands for 'Amazon Machine Image'). I also wanted to use some more recent features offered by Amazon in conjunction with their EC2 platform -- such as the permanent block-based storage AKA the Elastic Block Store (EBS), and also the permanent external IP addresses AKA the Elastic IPs.

To get started, I used a great blog post on 'Persistent Django on Amazon EC2 and EBS' by Thomas Brox Røst. I will refer here to some of the steps that Thomas details in his post; if you want to follow along, you're advised to read his post.

1) Create an AWS account and sign up for the EC2 service.

2) Install the ElasticFox Firefox extension -- the greatest thing since sliced bread in terms of managing EC2 AMIs. To run the ElasticFox GUI, go to Tools->ElasticFox in Firefox; this will launch a new tabbed window showing the GUI. From now on, I…

Back up your Windows desktop to S3 with SecoBackup

I found SecoBackup to be a good tool for backing up a Windows machine to S3. They have a 'free community edition' version, but you will still pay more in terms of your S3 costs than what you would normally pay to Amazon. You basically sign up for the SecoBackup service 'powered by AWS' and you pay $0.20 per GB of storage and $0.20 per GB of bandwidth -- so double what you'd pay if you stored it directly on S3. You don't even need to have an Amazon S3 account, they take care of it transparently for you.

I think this is a good tool for backing up certain files on Windows-based desktops. For example I back up my Quicken files from within a Windows XP virtual image that I run inside VMWare workstation on top of my regular Ubuntu Hardy desktop.

Ruby refugees flocking to Python?

I just wanted to put it out there that I know at least one person who was very fired up about Ruby, only to find out that all the available Ruby jobs are for Ruby-on-Rails programmers. He doesn't like Web programming, so what was he to do? You guessed it -- he started to learn Python :-)

RTFL

No, this is not a misspelling for ROTFL, but rather a variant of RTFM. It stands for Read The F...riendly Log. It's a troubleshooting technique that is very basic, yet surprisingly overlooked. I use it all the time, and I just want to draw attention to it in case you find yourself stumped by a problem that seems mysterious.

Here are some recent examples from my work.

Apache wouldn't start properly

A 'ps -def | grep http' would show only the main httpd process, with no worker processes. The Apache error log showed these lines:

Digest: generating secret for digest authentication

A google search for this line revealed this article:

http://www.raptorized.com/2006/08/11/apache-hangs-on-digest-secret-generation/

It turns out the randomness/entropy on that box had been exhausted. I grabbed the rng-tools tar.gz from sourceforge, compiled and installed it, then ran

rngd -r /dev/urandom

...and apache started its worker processes instantly.

Cannot create InnoDB tables in MySQL

Here, all…

This just in: Google releases Mox

...which is a YAPMOF (yet another Python mock object framework). Find it here, and read more about it here. I'll definitely check it out soon.

Zach and sugarbot going strong in Google SoC

Zach Riggle has made very strong progress with his Google SoC project, sugarbot. The goal of the project is to create a tool that runs and tests OLPCSugar activities automatically. Zach needed a way to hook into the PyGTK code that Sugar is based on. After looking at various tools, he settled on kiwi. He managed to have sugarbot run as an activity inside Sugar, then launch any other activities that need to be tested. The test scripts are kept by an XML-RPC server that Zach wrote, and sugarbot-based clients get them from the server and run them. Just these last couple of days, Zach also managed to get the sugarbot activity launch automatically when the Sugar environment starts up.

You can see a screencast that Zach put together, as well as a list of his accomplishments so far, in this blog post. In the screencast, Zach shows how he automates the launching and testing of two Sugar activities, the Calculator and the Terminal. Very cool stuff.

It's been a pleasure mentoring Zach on his …

Celtics use Ubuntu to beat Lakers

Excerpt from the Associated Press article about the Celtics-Lakers game last night:

"It was a group effort by this gang in green, which bonded behind Rivers, who borrowed an African word ubuntu (pronounced Ooh-BOON-too) and roughly means "I am, because we are" in English, as the Celtics' unifying team motto. The Celtics gave the Lakers a 12-minute crash course of ubuntu in the second quarter.Boston outscored Los Angeles 34-19, getting 11 field goals on 11 assists. The Celtics toyed with the Lakers, outworking the Western Conference's best inside and out and showing the same kind of heart that made Boston the center of pro basketball's universe in the '60s. "It's not what you thought, but it's still nice to see that the ubuntu concept is used successfully in sports too. I wonder what parallel we can make between the Lakers' game last night and an operating system. The Windows Blue Screen of Death comes to mind.

Security testing for agile testers

I've been asked by Lisa Crispin to contribute a few paragraphs on security testing to an upcoming book on agile testing that she and Janet Gregory are co-authoring. Here's what I came up with:

Security testing is a broad topic that cannot be possibly covered in a few paragraphs. Whole books have been devoted to this subject. Here we will try to at least provide some guidelines and pointers to books and tools that might prove useful to agile teams interested in security testing.

Just like functional testing, security testing can be viewed and conducted from two perspectives: from the inside out (white-box testing) and from the outside in (black-box testing).

Inside-out security testing assumes that the source code for the application under test is available to the testers. The code can be analyzed statically with a variety of tools that try to discover common coding errors which can make the application vulnerable to attacks such as buffer overflows or format string attacks. (Reso…

Tools for troubleshooting Web app performance

I came across this blog post which talks about 15 tools that can make your life easier when you need to troubleshoot the performance of your Web application. I knew about most of them, but a new addition to my arsenal is definitely wbox -- think of it as an HTTP-based ping. Very simple, but extremely useful.

What does your Wordle look like?

Image
The meme du jour seems to be Wordle tag clouds. I couldn't resist generating one out of the text on the first page of my blog. Here it is, in all its splendor:



You would think I'm very self-centered, since my first and last names appear so prominently. But I think it's because every blog post ends with "posted by Grig Gheorghiu at ". The next biggest word is Python, so there's some redemption for me right there :-)

Incremental backups to Amazon S3

Based on this great blog post by Tim McCormack, I managed to write some scripts that back up files to Amazon S3. The files are encrypted with GnuPG and rsync-ed to S3 using a Python-based tool called duplicity.

Here's what I did in order to get all this going on a CentOS 5.1 server running Python 2.5.

1) Signed up for Amazon S3 and got the AWS_ACCESS_KEY_ID and the AWS_SECRET_ACCESS_KEY.

2) Downloaded and installed the following packages: boto, GnuPGInterface, librsync, duplicity. All of them except librsync are Python-based, so they can be installed via 'python setup.py install'. For librsync you need to use './configure; make; make install'.

3) Generated a GPG key pair using "gpg --gen-key". Made a note of the hex fingerprint of the key (you can list the fingerprints of your keys via "gpg --fingerprint").

4) Wrote a simple boto-based Python script to create and list S3 buckets (the equivalent of directories in S3 parlance). Note that boto uses SSL,…

Compiling Python 2.5 with SSL support

If you compile Python 2.5.x from source, you need to jump through some hoops so that SSL support is enabled. Googling around, I found Patrick Altman's excellent blog post talking about this very issue.

In my case, I needed to enable SSL support for Python 2.5.2 on CentOS 5.1. I already had the openssl development libraries installed:

# yum list installed | grep ssl
mod_ssl.i386 1:2.2.3-11.el5_1.cento installed
openssl.i686 0.9.8b-8.3.el5_0.2 installed
openssl-devel.i386 0.9.8b-8.3.el5_0.2 installed

Here's what I did next, following Patrick's post:

1) edited Modules/Setup.dist from the Python 2.5.2 source distribution and made sure the correct lines were put back in (they were commented out by default):

_socket socketmodule.c

# Socket module helper for SSL support; you must comment out the other
# socket line above, and possibly edit the SSL variable:
#SSL=/usr/local/ssl
_ssl _ssl.c \
-DUSE_SSL -I$…

Encrypting a Linux root partition with LUKS and DM-CRYPT

One of our customers needed to have his Linux laptop's root partition encrypted. We found a HOWTO on achieving this with RHEL5, and we adapted it for CentOS 5. The technique is based on LUKS and DM-CRYPT. Kudos to my colleague Chris Evans for going through the exercise of getting this to work on CentOS 5 and for producing the documentation that follows, which I'm posting here hoping that it will benefit somebody at some point.

* Boot off of a Live CD, I used Fedora Core 9 Preview
* Find out which disk is which; for me /dev/sda was the external usb, and /dev/sdb was the internalsfdisk -d /dev/sdb | sfdisk /dev/sda
pvcreate --verbose /dev/sda2
vgextend --verbose VolGroup00 /dev/sda2
pvmove --verbose /dev/sdb2 /dev/sda2 # This takes ages
vgreduce --verbose VolGroup00 /dev/sdb2
pvremove --verbose /dev/sdb2
fdisk /dev/sdb
* Change the partition type to 83 for /dev/sdb2
* Here is when you get to choose the password that will protect your partition:
cryptsetup --verify-passphrase --key-size 256…

Notes from the latest SoCal Piggies meeting

...have been posted to the "Happenings in Python User groups" blog.

Update: Ben Bangert sent me the slides he used. You can download or view the PDF from here.

Guido open sources Code Review app running on GAPE

Not sure why this wasn't publicized more, but Guido van Rossum announced today that he open sourced the code for Code Review, a Google AppEngine app he released last week. Code Review is based on Mondrian, the internal code review tool that Guido wrote for Google. The relationship between the two apps in terms of features is: Code Review < Mondrian.

The code for Code Review is part of a Google code project called Rietveld. I haven't looked at it yet, but I'll certainly do so soon, just to see the master's view on how to write a GAPE application.

Ruby to Python bytecode compiler

Kumar beat me to it, but I'll mention it here too: Why the Lucky Stiff published a Ruby-to-Python-bytecode compiler, as well as tools to decompile the byte code into source code. According to the README file, he based his work on blog posts by Ned Batchelder related to dissecting Python bytecode. I wholeheartedly agree with Why's comment at the end of the README file:

You know, it's crazy that Python
and Ruby fans find themselves
battling so much. While syntax
is different, this exercise
proves how close they are to
each other! And, yes, I like
Ruby's syntax and can think much
better in it, but it would be
nice to share libs with Python
folk and not have to wait forever
for a mythical VM that runs all
possible languages.

Special guest for next SoCal Piggies meeting

We'll have the SoCal Piggies meeting this Thursday May 1st at the Gorilla Nation office in Culver City. Our special guest will be Ben Bangert, the creator of Pylons, who will give us an introduction to his framework. We'll also have a presentation from Pablo Noego from Gorilla Nation on a chat application he wrote using Google App Engine. We'll probably also have an informal discussion on Python mock testing tools and techniques.

BTW, I am putting together a Google code project for mock testing techniques in Python, in preparation for a presentation I would like to give to the group at some point. I called the project moctep, in honor of that ancient Egyptian deity, the protector of testers (or mockers, or maybe both). It doesn't have much so far, but there's some sample code you can browse through in the svn repository if you're curious. I'll be adding more meat to it soon.

Anyway, if you're a Pythonista who happens to be in the L.A. area on Thursday, pl…

"OLPC Automated Testing" project accepted for SoC

I'm happy to say that Zach Riggle's application for this year's Google Summer of Code, "OLPC Project Automated Testing", was accepted. I'm looking forward to mentoring Zach, and having Titus as a backup mentor. There's some very cool stuff that can be done in this area, and I hope that at the end of the summer we'll have some solid automated testing techniques and tools that can be applied to any Python project, not only to the OLPC Sugar environment. Stay tuned for more info on this project. BTW, here is the list of PSF-sponsored applications accepted for this years' SoC.

Come work for RIS Technology

We just posted this on craigslist, but it never hurts to blog about it too. If you're interested, send an email to techjobs at ristech.net. You and I might get to work together on the same team!

Open Source Tech Top Guns Wanted

Are you a passionate Linux user? Are you running the latest Ubuntu alpha release on your laptop just because you can? Are you wired to the latest technologies -- things like Amazon EC2/S3 and Google AppEngine? Are you a virtuoso when it comes to virtualization (Xen/VMWare)?

Do you program in Python? Do you take hard problems as personal challenges and don't give up until you solve them?

RIS Technology Inc. is a rapidly growing Los Angeles-based premium managed hosting provider that hosts and manages internet applications for medium to large size organizations nationwide. We have grown consistently at 100% each of the past four years and are currently hiring for additional growth at our corporate operations center near LAX, in Los Angeles, CA. We have immed…

Google App Engine feels constrictive

I've been toying a bit with Google App Engine. I was lucky enough to score one of the 10,000 developer accounts. I first went through their tutorial, which was fine. Then I tried to port a simple application that I used to run from the command line, which queried a range of IP addresses for their reverse DNS names. No luck. I was using the dnspython module, which in turn uses the Python socket module -- and socket is not available within the Google App Engine sandbox environment.

Also, I was talking to Michał on rewriting the Cheesecake service to run on Google App Engine, but he pointed out that cron jobs are not allowed, so that won't work either... It seems that with everything I've tried with GAE I've run into a wall so far. I know it's a 'paradigm change' for Web development, but still, I can't help wishing I had my favorite Python modules to play with.

What has your experience been with GAE so far? I know Kumar wrote a cool PyPI mirror in GAE, but I…

Ubuntu Gutsy woes with Intel 801 graphics card

I just upgraded my Dell Inspiron 6000 laptop to Ubuntu Gutsy last night. My graphics card is based on the Intel 810 chipset. After the upgrade, everything graphics-related was dog-slow. Scrolling in Firefox was choppy, IM-ing was choppy, even typing at the console was choppy. Surprisingly, I didn't find a lot of solutions to this problem. But many people on Ubuntu forums suggested disabling compiz/xgl, so that's what I ended up doing. In fact, I uninstalled all compiz and xgl-related packages, rebooted, and graphics became snappy again. Now back to trying to write an application to run on THE GOOGLE.

Meme du jour: shell history

Here's mine from my Ubuntu laptop:

$ history|awk '{a[$2]++ } END{for(i in a){print a[i] " " i}}' |sort -rn|head
121 cd
91 ssh
82 ls
46 vi
28 python
26 scp
16 dig
12 more
7 twistd
6 rm

Steve Loughran on 'Farms, Fabrics and Clouds'

Yesterday I and my colleagues at RIS Technology had the pleasure of attending a remote presentation given to us by Steve Loughran, who works as a researcher at HP Labs and is also a committer on the Ant project. I had seen Steve's slides from a presentation he gave at the University of Bristol on 'Farms, Fabrics and Clouds' back in December 2007, and I have been pestering him via email ever since, hoping to have him release a screencast. After much back and forth, Steve offered to simply present for now directly to us via Skype. He did it out of the goodness of his heart, but both he and I realized that there's a nice little business opportunity in this type of presentation: you release the slides with no audio, then you get hired to present to interested parties in person, remotely, via Skype and a shared set of slides, with a Q&A session at the end. Everybody wins in this scenario. Filing it in the 'ideas worth trying' category.

To come back to Steve's…

For you students interested in GSoC

If you're a student and you want to apply for a Python-related project for Google Summer of Code 2008, Matt Harrison has just the project for you. The project has to do with branch coverage analysis and reporting. Matt is willing to mentor too. It's a really good opportunity, so don't hesitate to apply. Hurry up though, the deadline is April 8th.

TurboGears and Pylons finally merging

This has been a long time coming, and fans of both projects have been eagerly waiting for it, but it's finally happened. Not sure if you've seen the announcements from Kevin Dangoor, Mark Ramm and Ben Bangert on their projects' mailing lists, but basically they boil down to "we feel like after the sprints at PyCon we made enough progress so that we can pull the trigger on merging the source code from the 2 projects in one common trunk." They make it sound like it was purely a technological problem, but I have my doubts about that. I think it was driven in part by the increasing popularity of Django. Unifying TurboGears and Pylons is a somewhat desperate measure to chip away at the Django market share. We'll see if it works or not. Check out the brand new page of the TurboPylons project.

ReviewBoard: open source code review tool

Via Marc Hedlund's post on O'Reilly Radar, here's an open source code review tool from VMWare: ReviewBoard. For all of us non-googlers out there, it's probably the next best thing to Guido's Mondrian (question: why has that tool not been released as open source?). Check out the sweet screenshots. The kicker though is that it uses Python and Django. Way to go, VMWare!

Python code complexity metrics and tools

There's a buzz in the air around code complexity, metrics, code coverage, etc. It started with Matt Harrison's PyCon presentation, then Ned Batchelder jumped in with a nice McCabe cyclomatic complexity computation/visualization tool, and now David Stanek posted about his pygenie tool -- which also measures the McCabe cyclomatic complexity of Python code. Now it's time to unify all these ideas in one powerful tool that computes not only complexity but also path or at least branch coverage. This would make a nice Google Summer of Code project. Too bad the deadline for 2008 GSoC applications is in 7 hours...Maybe for next year.

Update: David Goodger left a comment pointing me to Martin Blais's snakefood package, which computes and shows dependencies for your Python code. It's a good complement to the tools I mentioned above.

Recommended testing conference: CAST 2008

If you're a tester and are serious about learning and advancing in your trade, I warmly recommend the CAST 2008 conference which will be held in Toronto, July 14-16. The theme of the conference is "Beyond the Boundaries: Interdisciplinary Approaches to Software Testing" and the keynote speaker is none other than Jerry Weinberg. And it's REALLY hard to get Jerry Weinberg to speak at a conference, so you might as well take advantage of this opportunity. For more details on CAST 2008, download the PDF brochure.

It's a good time to be a Python programmer

We had the SoCal Piggies meeting at the Disney Animation Studios last night. It was a great meeting -- great presentations from Disney engineers on how they use Python at Disney (and they use it A LOT!), great food, great turnout, and great atmosphere. Let me tell you -- the Disney Animation Studios are *lush*. Thanks to Paul Hildebrandt for organizing the meeting.

I'll probably blog separately about the technical content of the presentations, but for now I just wanted to comment on the fact that everybody seems to be hiring Python programmers -- Gorilla Nation and Virgin Charter are just two companies in the L.A. area that are aggressively looking to hire Python talent. Another thing: we used to have difficulties in finding venues for our meetings. We used to meet at either USC or Caltech, and around 10-12 people max. would show up. Now companies are clamoring for organizing the meetings at their offices, and we have 20-30 people in the audience, with many new faces at every meeti…

Easy parsing with pyparsing

If you haven't used Paul McGuire's pyparsing module yet, you've been missing out on a great tool. Whenever you hit a wall trying to parse text with regular expressions or string operations, 'think pyparsing'.

I had the need to parse a load balancer configuration file and save certain values in a database. Most of the stuff I needed was fairly easily obtainable with regular expressions or Python string operations. However, I was stumped when I encountered a line such as:

bind http "Customer Server 1" http "Customer Server 2" http

This line 'binds' a 'virtual server' port to one or more 'real servers' and their ports (I'm using here this particular load balancer's jargon, but the concepts are the same for all load balancers.)

The syntax is 'bind' followed by a word denoting the virtual server port, followed by one or more pairs of real server names and ports. The kicker is that the real server names can be eithe…

PyCon08 gets great coverage

Reports on the death of the PyCon conference as a community experience have been greatly exaggerated. I personally have never seen any PyCon edition as well covered in the blogs aggregated in Planet Python as the 2008 PyCon. If you don't believe me, maybe you'll believe Google Blog Search. I think the Python community is alive and well, and ready to rock at PyCon conferences for the foreseeable future. I'm looking forward to PyCon09 in Chicago, and then probably in San Francisco for 2010/11.

PyCon presenters, unite!

If you gave a talk at PyCon and haven't uploaded your slides to the official PyCon website yet, but you have posted them online somewhere else, please leave a comment to this post with the location of your slides. I'm helping Doug Napoleone upload the slides, since some authors have experienced issues when trying to upload the slides using their PyCon account. Thanks!

Links to resources from PyCon talks

I took some notes at the PyCon talks I've been to, and I'm gathering links to resources referenced in these talks. Hopefully they'll be useful to somebody (I know they will be to me at least.)

"MPI Cluster Programming with Python and Amazon EC2" by Pete Skomoroch

* slides in PDF format
* Message Passing Interface (MPI) modules for Python: mpi4py, pympi
* ElasticWulf project (Beowulf-like setup on Amazon EC2)
* IPython1: parallel computing in Python
* EC2 gotchas

"Like Switching on the Light: Managing an Elastic Compute Cluster with Python" by George Belotsky

* S3FS: mount S3 as a local file system using Fuse (unstable)
* EC2UI: Firefox extension for managing EC2 clusters
* S3 Organizer: Firefox extension for managing S3 storage
* bundling an EC2 AMI and storing it to S3
* the boto library, which allows programmatic manipulation of Amazon Web services such as EC2, S3, SimpleDB etc. (a python-boto package is available for most Linux distributions too; for examp…