Consulting opportunities

If you're a Django or Selenium expert and live in the Los Angeles area (or even if you live in a different area, but can meet periodically with clients in Los Angeles), please send me an email at grig at gheorghiu.net. I know of a couple of great consulting gigs.

Dilbertian Information Dashboard

Do NOT do this :-)

Eliminating dependencies with regenerative build tools

This just in from Michael Feathers of "Working Effectively with Legacy Code" fame: a blog post on regenerative build tools. In the post, Michael describes an eye-opening practice related to continuous integration. Some smart people had the idea of running a script as part of a continuous build system that would comment out #include lines, one at a time, and then run the build. If the build succeeded, it meant that the include line in question was superfluous and thus could be deleted. Very interesting idea.

I bet this idea could be easily applied to Python projects, where you would comment out import statements and see if your unit test suite still passes. Of course, you can combine it with snakefood, a very interesting dependency graphing tool just released by Martin Blais. And you can also combine it with fault injection tools (aka fuzzers) such as Pester -- which belongs to the Jester family of tools also mentioned by Michael Feathers in his blog post.

Brian Marick has a new blog

If you're serious about testing, you need to read Brian Marick's blog, which he recently moved to exampler.com from the old testing.com URL. Brian says:

" This is the another step in my multi-decade switch from testing.com to exampler.com. "Exampling", though not a verb, is a better description of what I do now. It includes testing, but has a larger scope."

And by the way, the style of technical writing that Brian describes in his latest post bugs me no end too...

Resetting MySQL account passwords

I recently needed to reset the MySQL root account password. Here are the steps, for future reference:

1) Stop mysqld, for example via 'sudo /etc/init.d/mysqld stop'

2) Create text file /tmp/mysql-init with the following contents (note that the file needs to be in a location that is readable by user mysql, since it will be read by the mysqld process running as that user):

SET PASSWORD FOR 'root'@'localhost' = PASSWORD('newpassword');

3) Start mysqld_safe with following option, which will set the password to whatever was specified in /tmp/mysql-init:

$ sudo /usr/bin/mysqld_safe --init-file=/tmp/mysql-init &

4) Test connection to mysqld:

$ sudo mysql -uroot -pnewpassword

5) If connection is OK, restart mysqld server:

$ sudo /etc/init.d/mysqld restart

Also for future reference, here's how to reset a normal user account password in MySQL:

Connect to mysqld as root (I assume you know the root password):

$ mysql -uroot -prootpassword

Use the SET PASSWORD command:

mysql> SET PASSWORD for 'myuser'@'localhost' = PASSWORD('newuserpassword');

Apache virtual hosting with Tomcat and mod_jk

In a previous post I talked about "Configuring Apache 2 and Tomcat 5.5 with mod_jk". I'll revisit some of the topics in there, but within a slightly different scenario.

Let's say you want to configure virtual hosts in Apache, with each virtual host talking to a different Tomcat instance via the mod_jk connector. Each virtual host serves up a separate application via an URL such as http://www.myapp.com. This URL needs to be directly mapped to a Tomcat application. This is a fairly important requirement, because you don't want to go to a URL such as http://www.myapp.com/somedirectory to see your application. This means that your application will need to be running in the ROOT of the Tomcat webapps directory.

You also want Apache to serve up some static content, such as images.

Running multiple instances of Tomcat has a couple of advantages: 1) you can start/stop your Tomcat applications independently of each other, and 2) if a Tomcat instance goes down in flames, it won't take with it the other ones.

Here is a recipe that worked for me. My setup is: CentOS 4.4, Apache 2.2 and Tomcat 5.5, with mod_jk tying Apache and Tomcat together (mod_jk2 has been deprecated).

Scenario: we want www.myapp1.com to go to a Tomcat instance running on port 8080, and www.myapp2.com to go to a Tomcat instance running on port 8081. Apache will serve up www.myapp1.com/images and www.myapp2.com/images.

1) Install Apache and mod_jk. CentOS has the amazingly useful yum utility (similar to apt-get for you Debian/Ubuntu fans), which makes installing packages a snap:

# yum install httpd
# yum install mod_jk-ap20

2) Get the tar.gz for Tomcat 5.5 -- you can download it from the Apache Tomcat download site. The latest 5.5 version as of now is apache-tomcat-5.5.23.tar.gz.

3) Unpack apache-tomcat-5.5.23.tar.gz under /usr/local. Rename apache-tomcat-5.5.23 to tomcat8080. Unpack the tar.gz one more time, rename it to tomcat8081.

4) Change the ports tomcat is listening on for the instance that will run on port 8081.

# cd /usr/local/tomcat8081/conf
- edit server.xml and change following ports:
8005 (shutdown port) -> 8006
8080 (non-SSL HTTP/1.1 connector) -> 8081
8009 (AJP 1.3 connector) -> 8010

There are other ports in server.xml, but I found that just changing the 3 ports above does the trick.

I won't go into the details of getting the 2 Tomcat instances to run. You need to create a tomcat user, make sure you have a Java JDK or JRE installed, etc., etc.

The startup/shutdown scripts for Tomcat are /usr/local/tomcat808X/bin/startup.sh|shutdown.sh.

I will assume that at this point you are able to start up the 2 Tomcat instances. The first one will listen on port 8080 and will have an AJP 1.3 connector (used by mod_jk) listening on port 8009. The second one will listen on port 8081 and will have the AJP 1.3 connector listening on port 8010.

5) Deploy your applications.

Let's say you have war files called app1.war for your first application and app2.war for your second application. As I mentioned in the beginning of this post, your goal is to serve up these applications directly under URLs such as http://www.myapp1.com, as opposed to http://www.myapp1.com/app1. One solution I found for this is to rename app1.war to ROOT.war and put it in /usr/local/tomcat8080/webapps. Same thing with app2.war: rename it to ROOT.war and put it in /usr/local/tomcat8081/webapps.

You may also need to add one line to the Tomcat server.xml file, which is located in /usr/local/tomcat808X/conf. The line in question is the one starting with Context, and you need to add it to the Host section similar to this one in server.xml. I say 'you may also need' because I've seen cases where it worked without it. But better safe than sorry. What the Context element does is it specifies ROOT as the docBase of your Web application (similar if you will to the Apache DocumentRoot directory).

    <Host name="localhost" appBase="webapps"
     unpackWARs="true" autoDeploy="true"
     xmlValidation="false" xmlNamespaceAware="false">
      <Context path="" docBase="ROOT" debug="0"/>

At this point, if you restart the 2 Tomcat instances, you should be able to go to http://www.myapp1.com:8080 and http://www.myapp2.com:8081 and see your 2 Web applications.

6) Create Apache virtual hosts for www.myapp1.com and www.myapp2.com and tie them to the 2 Tomcat instances via mod_jk.

Here is the general mod_jk section in httpd.conf -- note that it needs to be OUTSIDE of the virtual host sections:

#
# Mod_jk settings
#
# Load mod_jk module
LoadModule    jk_module  modules/mod_jk.so
# Where to find workers.properties
JkWorkersFile conf/workers.properties
# Where to put jk logs
JkLogFile     logs/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel    emerg
# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
# JkOptions indicate to send SSL KEY SIZE,
JkOptions     +ForwardKeySize +ForwardURICompat -ForwardDirectories
# JkRequestLogFormat set the request format
JkRequestLogFormat     "%w %V %T"

Note that the section above has an entry called JkWorkersFile, referring to a file called workers.properties, which I put in /etc/httpd/conf. This file contains information about so-called workers, which correspond to the Tomcat instances we're running on that server. Here are the contents of my workers.properties file:

#
# This file provides minimal jk configuration properties needed to
# connect to Tomcat.
#
# The workers that jk should create and work with
#

workers.tomcat_home=/usr/local/tomcat8080
workers.java_home=/usr/lib/jvm/java
ps=/
worker.list=app1, app2

worker.app1.port=8009
worker.app1.host=localhost
worker.app1.type=ajp13
worker.app1.lbfactor=1

worker.app2.port=8010
worker.app2.host=localhost
worker.app2.type=ajp13
worker.app2.lbfactor=1

The file declares 2 workers that I named app1 and app2. The first worker corresponds to the AJP 1.3 connector running on port 8009 (which is part of the Tomcat instance running on port 8080), and the second worker corresponds to the AJP 1.3 connector running on port 8010 (which is part of the Tomcat instance running on port 8081).

The way Apache ties into Tomcat is that each of the VirtualHost sections configured for www.app1.com and www.app2.com declares a specific worker. Here is the VirtualHost section I have in httpd.conf for www.app1.com:

<VirtualHost *:80>
ServerName www.app1.com
DocumentRoot "/usr/local/tomcat8080/webapps/ROOT"
<Directory "usr/local/tomcat8080/webapps/ROOT">
  # Options Indexes FollowSymLinks MultiViews
  Options None
  AllowOverride None
  Order allow,deny
  allow from all
</Directory>
ErrorLog logs/app1-error.log
CustomLog logs/app1-access.log combined
# Send ROOT app. to worker named app1
JkMount  /* app1
JkUnMount  /images/* app1
RewriteEngine On
RewriteRule ^/(images/.+);jsessionid=\w+$ /$1
</VirtualHost>

The 2 important lines as far as the Apache/mod_jk/Tomcat configuration is concerned are:

JkMount /* app1
JkUnMount /images/* app1

The line "JkMount /* app1" tells Apache to send everything to the worker app1, which then ties into the Tomcat instance on port 8080.

The line "JkUnMount /images/* app1" tells Apache to handle everything under /images itself -- which was one of our goals.

At this point, you need to restart Apache, for example via 'sudo service httpd restart'. If everything went well, you should be able to go to http://www.myapp1.com and http://www.myapp2.com and see your 2 Web applications running merrily.

You may have noticed a RewriteRule in each of the 2 VirtualHost sections in httpd.conf. What happens with many Java-based Web application is that when a user first visits a page, the application does not know yet if the user has cookies enabled or not, so the application will use a session ID mechanism fondly known as jsessionid. If the user does have cookies enabled, the application will not use jsessionid the second time a page is loaded. If cookies are not enabled, the application (Tomcat in our example) will continue generating URLs such as

http://www.myapp1.com/images/myimage.gif;jsessionid=0E45D13A0815A172BD1DC1D985793D02

In our example, we told Apache to process all URLs that start with 'images'. But those URLs have already been polluted by Tomcat with jsessionid the very first time they were hit. As a result, Apache was trying to process them, and was failing miserably, so images didn't get displayed the first time a user hit a page. If the user refreshed the page, images would get displayed properly (if the user had cookies enabled).

The solution I found for this issue was to use a RewriteRule that would get rid of the jsessionid in every URL that starts with 'images'. This seemed to do the trick.

That's about it. I hope this helps somebody. It's the result of some very intense googling :-)

If you have comments or questions, please leave them here and I'll try to answer them.

JRuby buzz

I wish I could put "Jython buzz" as the title of my post, but unfortunately I can't seem to detect any Jython buzz anywhere. JRuby though seems to generate a lot of it, judging by this InfoQ article on Mingle, a commercial application based on JRuby and created by ThoughtWorks Studios.

One thing I found very interesting in the InfoQ article was that ThoughtWorks preferred to develop Mingle with JRuby (which is the JVM-based version of Ruby) over writing it on top of Ruby on Rails. They cite ease of deployment as a factor in favor of JRuby:

"In particular, the deployment story for Ruby on Rails applications is still significantly more complex than it should be. This is fine for a hosted application where the deployment platform is in full control of a single company, but Mingle isn't going to be just hosted. Not only is it going to need to scale ‘up’ to the sizes of Twitter (okay, that's wishful thinking and maybe it won't need to scale that much) but it's also going to need to scale ‘down’ to maybe a simple Windows XP machine with just a gig of RAM. On top of that, it's going to be installed by someone who doesn't understand anything about Ruby on Rails deployment and, well, possibly not much about deployment either."

They continue by saying that their large commercial customers wanted to be able to deploy Mingle by dropping a Java .war file under any of the popular Java application servers.

So, for all the talk about Ruby on Rails and the similarly hot Python frameworks, Java and J2EE are far from dead.

Here's wishing that Jython will start generating the same amount of buzz.

Michael Dell uses Ubuntu on his home laptop

Found this on Planet Ubuntu, which is buzzing with the news that Dell will be offering laptops preloaded with Feisty Fawn. I still use Edgy on my Inspiron 6000, but I'll probably upgrade to Feisty soon. Or maybe I'll just wait for a Gutsy Gibbon to burst on the scene :-)

What programming language are *you*?

Not sure what this really says about me, but I am PHP according to the survey below...(which I stumbled upon via Brian Marick's blog)


Which Programming Language are You?

PyCon07 Testing Tools Tutorial slides up

Spurred by a request from David Brochu, I put the PyCon 07 Testing Tools Tutorial slides up on agilistas.org. Titus's slides are named titus-tutorial-a.pdf through -e.pdf. My slides are only one PDF file, as my portion of the tutorial consisted mainly in Selenium and FitNesse demos. Enjoy!

Mounting local file systems using the 'bind' mount type

Sometimes paths are hardcoded in applications -- let's say you have the path to the Apache DocumentRoot directory hardcoded inside a web application to /home/apache/www.mysite.com. You can't change the code of the web app, but you want to migrate it. You don't want to use the same path on the new server, for reasons of standardization across servers. Let's say you want to set DocumentRoot to /var/www/www.mysite.com.

But /home is NFS-mounted, so that all users can have their home directory kept in one place. One not-so-optimal solution would be to create an apache directory under /home on the NFS server. At that point, you can create a symlink to /var/www/www.mysite.com inside /home/apache. This is suboptimal because the production servers will come to depend on the NFS-mounted directory. You would like to keep things related to your web application local to each server running that application.

A better solution (suggested by my colleague Chris) is to mount a local directory, let's call it /opt/apache_home, as /home/apache. Since the servers are already using automount, this is a question of simply adding this line as the first line in /etc/auto.home:

apache -fstype=bind :/opt/apache_home

/etc/auto.home was already referenced in /etc/auto.master via this line:

/home /etc/auto.home

Note that we're using the neat trick of mounting a local file system via the 'bind' mount type. This can be very handy in situations where symbolic links don't help, because you want to reference a real directory, not a file pointing to a directory. See also this blog post for other details and scenarios where this trick is helpful.

Now all applications that reference /home/apache will actually use /opt/apache_home.

For the specific case of the DocumentRoot scenario above, all we needed to do at this point was to create a symlink inside /opt/apache_home, pointing to the real DocumentRoot of /var/www/www.mysite.com.

Dell to offer pre-installed Linux on desktops

I knew about the Dell survey which asked people whether they'd like to see Linux pre-installed on Dell desktops and laptops. Looks like more than 70% of the respondents said yes (see this BBC story) -- so Dell is going for it. Now the question is which flavor(s) of Linux will be offered. From what I remember, the survey mentioned Fedora Core, Ubuntu and OpenSuse. Regardless, this is a pretty big win for Linux.

OLPC and the Romanian politicians

Interesting blog post from Jani Monoses on how the Romanian parliament rejected the country's participation in the OLPC program. All the arguments centered around cost and lack of applications such as....MS Word! As Jani says -- cluelessness abounds.

Having seen Ivan Krstic's keynote on OLPC at PyCon this year, I realize that the One Laptop Per Child program is mainly about re-introducing kids to their intuitive ways of learning, through play, peer activities and free exploration, as opposed to the centralized, one-to-many teaching method that is used in schools everywhere. The laptop becomes in this case just a tool for facilitating the new ways of learning -- or I should say the old ways, since this is what kids do naturally. But this is one of those disruptive ideas that is hard to grasp by serious grown-up people, especially politicians...

File sharing with Apache and WebDAV

If you want to share files from a Linux box to Windows clients, Samba is a popular solution. However, it can also be done with Apache and WebDAV. Here is a short HOWTO.

1) Let's say we want to share files in a directory named /usr/share/myfiles. I created a sub-directory called dav in that directory, and then I ran:

# chmod 775 dav
# chgrp apache dav

2) Make sure httpd.conf loads the mod_dav modules:

LoadModule dav_module modules/mod_dav.so
LoadModule dav_fs_module modules/mod_dav_fs.so

3) Create an Apache password file (if you want to use basic authentication) and a user -- let's call it webdav:

# htpasswd -c /etc/httpd/conf/.htpasswd webdav

4) Create a virtual host entry in httpd.conf, similar to this one:

<VirtualHost *>
ServerName share.mydomain.com
DocumentRoot "/usr/share/myfiles"
<Directory "/usr/share/myfiles">
    Options Indexes FollowSymLinks MultiViews
    AllowOverride AuthConfig
    Order allow,deny
    allow from all
</Directory>
ErrorLog share-error.log
CustomLog share-access.log combined

DavLockDB /tmp/DavLock

<Location /dav>
    Dav On
    AuthType Basic
    AuthName DAV
    AuthUserFile /etc/httpd/conf/.htpasswd
    Require valid-user
</Location>
</VirtualHost>

5) Restart httpd, verify that if you go to http://share.mydomain.com/dav you are prompted for a user name and password, and that once you get past the security dialog you can see something like 'Index of /dav'.

Now it's time to configure your Windows client to see the shared WebDAV resource. On the Windows client, either:

• go to "My network connections" and add a new connection, or
• go to Windows Explorer->Tools->Map Network Drive, then click on "Signup for online storage or connect to a network server"

Either option will bring up the "Add Network Place Wizard".

• Click Next, then select "Choose another network location", then click Next.
• For "Internet or network address", set http://share.mydomain.com/dav. At this point you'll be prompted for a user name/password; specify the ones you defined above.
• After mapping the resource, you should be able to read/write to it.

CAVEAT

Sometimes the Windows dialog asking for a user name and password will say "connecting to share.mydomain.com" and will keep asking you for the user name/password. The dialog is supposed to show the text you set in AuthName (DAV in my case). If it doesn't, click Cancel, then try again. You can also try to force HTTP basic authentication (as opposed to Windows authentication, which is what Windows tries to do) by specifying http://share.mydomain.com:80/dav as the URL. See also this entry on the WebDAV Wikipedia page.

Resources

• Apache2 mod_dav documentation
  
• Article by Martin Brown on "Enabling WebDAV in Apache"
  
• Setting up Apache2 and WebDAV on Debian

Ubuntu "command not found" magic

Via Alan Pope's blog: Ubuntu Edgy and above includes a "command not found magic" -- a bash hook that intercepts 'command not found' errors and replaces them with more useful messages, such as what packages you need to install to get that command. I tried it on my Edgy laptop and what do you know, it actually works.

First you need to apt-get the command-not-found package:

$ sudo apt-get install command-not-found
Reading package lists... Done
Building dependency tree   
Reading state information... Done
The following extra packages will be installed:
command-not-found-data
The following NEW packages will be installed:
command-not-found command-not-found-data
0 upgraded, 2 newly installed, 0 to remove and 15 not upgraded.
Need to get 471kB/475kB of archives.
After unpacking 6263kB of additional disk space will be used.
Do you want to continue [Y/n]? y
Get:1 http://us.archive.ubuntu.com edgy/universe command-not-found-data 0.1.0 [471kB]
Fetched 300kB in 2s (109kB/s)              
Selecting previously deselected package command-not-found-data.
(Reading database ... 92284 files and directories currently installed.)
Unpacking command-not-found-data (from .../command-not-found-data_0.1.0_i386.deb) ...
Selecting previously deselected package command-not-found.
Unpacking command-not-found (from .../command-not-found_0.1.0_all.deb) ...
Setting up command-not-found-data (0.1.0) ...
Setting up command-not-found (0.1.0) ...

Then you need to open a new shell window, so that the hook gets installed. In that window, try running some commands which are part of packages that you don't yet have installed. For example:

$ nmap
The program 'nmap' is currently not installed, you can install it by typing:
sudo apt-get install nmap

$ snort
The program 'snort' can be found in the following packages:
* snort-pgsql
* snort-mysql
* snort
Try: sudo apt-get install 

Pretty cool, huh. And of course the bash hook is written in Python.

Founder of Debian joins Sun

Interesting announcement from Ian Murdoch, the founder of Debian: he's joining Sun as "Chief Operating Platforms Officer" (now that's a mouthful.) I've been really skeptical so far about Sun's embrace of Open Source, but this seems like a major step in the right direction.

CheeseRater - voting in the CheeseShop

Found via Joe Gregorio's blog: CheeseRater, a slick-looking Django app that lets you vote on CheeseShop packages. Interesting idea. Since tagging/redditing/social networking/web2.0ing/etc. are all the rage these days, this might prove more popular than the automatic scoring that Michał is doing with the Cheesecake Service. Maybe we can combine the two....that would be interesting.

Stone soup as a cure for broken windows

Stones are used to break windows, but in this insightful blog post, Dave Nicolette shows how making stone soup (i.e. getting everybody to contribute a bit of something) can help deal with the broken windows syndrome. Getting everybody to contribute is an art I've been trying to master myself, but like any art, it's far from easy...

A few good agile men

Kumar sent me a link to a hilarious blog post: A few good managers. It's a gem. Excerpt:

"Marketing: "Did you cut the automated, edit sync [insert favorite feature here] feature?"

Development: "I did the job I was hired to do."

Marketing: "Did you cut the automated, edit sync feature?"
Development: "I delivered the release on time."

Marketing: "Did you cut the automated, edit sync feature?"
Development: "You're g%$#@*& right I did!""

I'm Ubuntu counted!

I just registered as Ubuntu user #12343 at Ubuntu Counter. Pretty cool number!