Thursday, February 09, 2006

Installing a Python package to a custom location with setuptools

Update 2/10/06

According to PJE, the author of setuptools, my approach below is horribly wrong. Although I personally don't see why what I attempted to show below is so mind-blowingly stupid (according again to PJE; see his 2nd comment to this post), I do respect his wish of pointing people instead to the official documentation of setuptools, in particular to the Custom Installation Location section.

OK, maybe I see why he says my approach is stupid -- it would require modifying the PYTHONPATH for each and every package you install. To be honest, I never used easy_install this way, I always had root access on my machines, so the non-root scenario is not something I see every day. The PYTHONPATH hack below can be avoided if you go through the motions of setting up an environment for easy_install, as explained in the Custom Installation instructions.

So, again, if you are really interested in getting easy_install to work as simply as possible when you don't have root access to your box, please READ THE OFFICIAL INSTRUCTIONS (and pretend the link is blinking to attract your attention.)

If you still want to read my original post, warts and all, here it is:

My previous post on setuptools generated a couple of comments from people who pointed out that I didn't really read Joe Gregorio's post well enough; they said his main problem was not being able to install Routes using setuptools as a non-root user, since he doesn't have root access to his server (BTW, Joe, if you're reading this, JohnCompanies offers great Linux "virtual private server" hosting plans where you have your own Linux virtual machine to play with). response to these comments, here is the 2-minute guide to installing a Python package to a custom location as a non-root user using setuptools:

1. Use easy_install with the -d option to indicate the target installation directory for the desired package

[ggheo@concord ggheo]$ easy_install -d ~/Routes Routes
Searching for Routes
Best match: Routes 1.1
Downloading egg#md5=be7fe3368cbeb159591e07fa6cfbf398
Processing Routes-1.1-py2.4.egg
Moving Routes-1.1-py2.4.egg to /home/ggheo/Routes

Installed /home/ggheo/Routes/Routes-1.1-py2.4.egg

Because this distribution was installed --multi-version or --install-dir,
before you can import modules from this package in an application, you
will need to 'import pkg_resources' and then use a 'require()' call
similar to one of these examples, in order to select the desired version:

pkg_resources.require("Routes") # latest installed version
pkg_resources.require("Routes==1.1") # this exact version
pkg_resources.require("Routes>=1.1") # this version or higher

Note also that the installation directory must be on sys.path at runtime for
this to work. (e.g. by being the application's script directory, by being on
PYTHONPATH, or by being added to sys.path by your code.)

Processing dependencies for Routes

[ggheo@concord ggheo]$ cd ~/Routes
[ggheo@concord Routes]$ ls -la
total 40
drwxrwxr-x 2 ggheo ggheo 4096 Feb 9 14:13 .
drwxr-xr-x 90 ggheo ggheo 8192 Feb 9 14:17 ..
-rw-rw-r-- 1 ggheo ggheo 27026 Feb 9 14:13 Routes-1.1-py2.4.egg
[ggheo@concord Routes]$ file Routes-1.1-py2.4.egg
Routes-1.1-py2.4.egg: Zip archive data, at least v2.0 to extract

As you can see, a single egg file was installed in ~/Routes. For other packages, easy_install will create a directory called for example twill-0.8.3-py2.4.egg. This is all dependent on the way the package creator wrote the setup file.

2. Add the egg file (or directory) to your PYTHONPATH. This is one way of letting python know where to find the package; as the output of easy_install -d says above, there are other ways too:

[ggheo@concord ggheo]$ export PYTHONPATH=$PYTHONPATH:/home/ggheo/Routes/Routes-1.1-py2.4.egg

3. Start using the package:

[ggheo@concord ggheo]$ python
Python 2.4 (#1, Nov 30 2004, 16:42:53)
[GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import routes
>>> dir(routes)
['Mapper', '_RequestConfig', '__all__', '__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__path__', 'base', 'redirect_to', 'request_config', 'sys', 'threadinglocal', 'url_for', 'util']
>>> routes.__file__

I printed the __file__ attribute to show that the module routes is imported from the custom location I chose for the install.

All in all, I still maintain this was pretty easy and painless.


Anonymous said...

This is also why it's useful to install a local version of Python on your system if you have compilation tools on the box you're working on.

PJE said...

And it's even easier if you follow the directions for custom installation. (The PYTHONPATH hacking would be unnecessary in that case.)

Anonymous said...

I'm failing to see where the easy part comes in...

distutils is easy; this is not.

Anonymous said...

Peter, what's not easy about it? I'd never used setuptools before, but this article has me convinced. What am I missing?

PJE said...

No, the summary doesn't help because that would *still be doing it the hard way*.

If someone reads the directions I linked to, and follows them *one time*, then "easy_install packagename" will then work for them from then on. *That* is the easy part.

This entire post, ironically, makes it look *hard*, because it is doing things in the hard way. Not just any old hard way, but the hardest way imaginable!

Grig, *please* replace this post with just a link to the actual directions that I posted above. The current post is not helping matters, and in fact it's hurting because it makes easy_install look hard and stupid. It's like a distutils supporter demonstrating how you can run each of the build_foo and install_foo distutils commands, one at a time, while specifying every option explicitly. It will 100% work, but it's also 100% stupid.

This is far worse than the guy who said not to use setuptools. He at least had a point: if you don't have root access, you can't use setuptools unless you RTFM.

But *this* post makes setuptools supporters look careless (i.e. not reading the original person's situation) and foolish (doing things in a way that's harder instead of easier).

Grig Gheorghiu said...

PJE, I guess the saying is true which tells us that no good deed goes unpunished. I'm sorry to have incurred your wrath, and I updated my post accordingly. Please do note that we setuptools users are well intentioned and are trying to get more people to use them. If what we're doing is not the official way, then maybe somewhere along the way the official way got lost or misinterpreted. Maybe this says something about the official documentation....I don't know. You're the boss, you decide how you want to spread (or not) the word about the stuff you write. If I were you, I would definitely clear the matter once and for all by blogging about it.

PJE said...

Sorry, Grig, but I *did* post links to the instructions on your previous post and this one, and I was also responding to the subsequent postings of others. I know you meant well.

The reason I hadn't blogged about this myself yet is because I've been fixing the problems Joe's post highlighted. I now have all his original complaints fixed in the Subversion edition of setuptools, and I also contacted Joe privately to find out more specifics of his issues. In other words, I wanted to make sure that the response would fully address the issue.

Please accept my apologies, and understand that I do appreciate your enthusiasm. Sorry for being a bit harsh; my anger is directed at the situation I felt your post was causing (as evidenced by the other comments here), but it was not actually directed at you.

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...