Thursday, February 09, 2006

Please start (or continue) using setuptools

You might have seen Joe Gregorio's blog post entitled "Please stop using setuptools". Well, I'm here to tell you that you should run, not walk, towards embracing setuptools as your Python package deployment tool of choice. I've been using setuptools for a while now and it makes life amazingly easy. It's the closest thing we have to CPAN in the Python world, and many times it does a better job than CPAN.

I think part, if not all, of the problem Joe Gregorio had, was that instead of bootstrapping the process by downloading and running it in order to get setuptools and easy_install installed on his machine, he painstakingly attempted to install Routes via I do find the two names and easy_install confusingly similar myself, and I sometimes start typing easy_setup when I really want easy_install. So I wish PJE changed the name of at least one of these two utilities to something else, to make it easier to remember. I vote for changing easy_install to something with less characters and that doesn't have an underscore in it, but I don't have any good ideas myself.

If Joe had run "easy_install Routes", he would have seen this:

$ sudo easy_install Routes
Searching for Routes
Best match: Routes 1.1
Processing Routes-1.1-py2.4.egg
Moving Routes-1.1-py2.4.egg to /usr/local/lib/python2.4/site-packages
Adding Routes 1.1 to easy-install.pth file

Installed /usr/local/lib/python2.4/site-packages/Routes-1.1-py2.4.egg
Processing dependencies for Routes

I don't see how it can get easier than this, to be honest. If you've never used setuptools before, here's a quick howto from Titus. EZ indeed :-)


See this other blog post of mine for instructions on installing a Python package via setuptools, to a custom location and as a non-root user.


Ian Bicking said...

I think PJE is planning a "nest" command that will take easy_install's place, and add some functionality. (Or maybe easy_install will become a facet of nest, but remain a command? Not sure)

I expect such a command would also avoid some of the UI problems of easy_install, which it inherits from distutils (like the rather inaccurate help messages). Logging is apparently a little crufty in distutils, which is why it is so verbose (I wish it was more quiet until failure, probably collecting log messages and then showing them on an error).

Maybe would be a good name for

Grig Gheorghiu said...

Ian, that's good news. I like the name 'nest', it's short and sweet, and of course it has to do with eggs. I also think bootstrap_something would work , but bootstrap_easy_install seems like a lot to type :-)....maybe bootstrap_nest?

Anonymous said...

I was thinking of, myself.

Anonymous said...

If you read Joe's original post, you'd notice he doesn't have root - he's running on a host server.

setuptools is a pain.

PJE said...

The whole issue here is one of documentation. Setuptools has good reference documentation now, where "good" means that everything you need to know is in there.

What it doesn't have is a FAQ or quick-reference guide to give end-users the information they need. The information relevant to the original post is the What Your Users Should Know section of the setuptools manual and the Custom Installation Locations section of the EasyInstall manual. The latter would've told Joe exactly what he needed to do to make his setup work.

The right tools are all there, and they're documented; it's more quickstart material that's actually needed. (And, as Joe points out in his post, some more polishing of the tools sure wouldn't hurt for those who just refuse to read *any* docs.)

Anonymous said...

How about




The first does not need to be run often so the length does not matter so much.

Florian said...

The ones who don't read any documentation try.
The ones who fail trying whine.

Any particular reason why anyone is paying attention?

Anonymous said...

Just to repeat what several others have said:

$ sudo easy_install Routes
joe is not in the sudoers file. This incident will be reported.

That said, I think your idea of doing away with the script is a good idea, but I get the impression is not meant as a setuptools installer -- it's meant as a your-package-installer for people who don't have setuptools yet -- at least that's how I've seen it used in places such as Turbogears or Routes. One thing that bothers me about it is that I can't help but assume that problems will arise in the future relating to the fact that the web will be full of locally-hosted and differently-versioned setuptools installer scripts. But my main complaint is that it is hiding the fact that setuptools is a complex subsystem which might need to be configured before being used, and that it is hiding this from both the installers and the package authors choosing to use it as their distribution system. Instead, most websites just act like is some sort of automatic download script, which also adds some stuff to your system "that you'll thank me for later, really"; Routes' download page doesn't link to the manual, or even mention the word setuptools!

Packages really should transition to instructions more along the lines of
1. Download and install the dependency, setuptools (here is PJE's download page) (and PJE can stick a link to the manual there if people need to read it before installing setuptools)
2. Run easy_install Routes (which will install the remaining dependencies)
but of course, some just say "run this magic script I have and everything will work!"

But for projects like Routes (which has no other dependencies), that is a bit more complicated than
1. Download and run install
Which is kind of Joe's point, I think. That anything using setuptools as its sole means of distribution has a large, complex, possibly hard-to-install dependency -- ironicly, a package is often requiring setuptools because it wants to avoid the complexity of dependencies. And requiring people to install large, complex, hard-to-install dependencies sucks and should be avoided when possible. And turning that dependency into an optional feature is as simple as adding a tarball alongside the egg.

So no, package authors shouldn't just read the title of Joe's post and follow it; you and I both like being able to just 'easy_install Routes'. But they should read the title and subtitle of Joe's post and follow that.

Anonymous said...

anonymous wrote:
>That anything using setuptools as its sole means of distribution has a large, complex, possibly hard-to-install dependency

What is so hard about installing setuptools?

1. Download
2. Run it.

After you have done that once, installing a package that suports setuptools, becomes only

$ easy_install package_name

which is IMHO far easier than:

1. Locate the URL of the package
2. Download it.
3. Extract it.
4. Run python install

Chris Arndt

Anonymous said...

I've so far found easy_install and friends to be really, really irritating to use under Debian. The problem I've been having is that there's a Debian version installed, and also a /usr/local version installed, and they fight with each other and there doesn't seem to be any way to remove one of them and still have the system work. Perhaps eventually when this is fully Debianized things will work, but right now it's kind of a disaster.

Anonymous said...

chris: setuptools is easy to install most of the time -- I've never had any problems with it, and I too find a one-time followed by a single run of easy_install any time I want to install any package much easier than manually downloading, and clearly the way of the future. But that isn't the case for all people, and package authors need to stop assuming that just because it worked for them nobody will ever need to do anything different.
Installing setuptools and using it to install a package is not always just python && easy_install pkgname, otherwise the EasyInstall instruction manual would be a lot shorter than it is.
All I'm saying is that package authors need to start admitting that they are asking people to install a seperate, 3rd party dependency with documented configuration requirements and quirks of its own, and start linking to instructions on doing that instead of just asking them to run some random script called And if they also admit that not everyone can make the second of the two assumptions running with no configuration makes:
* You have unrestricted internet access on the computer where you are installing.

than that would be cool too. Currently, Routes (only mentioning them since they are the example that started all this) doesn't even have a link to the egg file on their download page -- what am I supposed to do if I want to install it to a machine that doesn't have an internet connection? I can transfer over the setuptools egg and get easy_install all setup over there, but how do I get the Routes egg transfered over there? It only seems to be discoverable by easy_install itself.

As to the 4-step process you outlined, you have to keep in mind that if I'm installing setuptools just to install your package, it becomes a 5-step process:
1. Locate URL of
2. Download
3. Read ez_setup instructions and configure it
4. Run python
5. Run easy_install pkgname

Of course, all those setup steps can be skipped for future packages, but it sure makes your package look hard to install.
Hopefully in the glorious future this will be avoided though, as setuptools will presumably some day become part of the standard library and configuring and installing it will become part of configuring and installing python itself. Then everyone will be able to just easy_install blah or easy_install -d ~/lib blah and have it work.

Anonymous said...

Haha, I just found out where the egg is -- in the cheeseshop itself, along with the very tarball whose supposed non-existance is what started all this!
So it turns out Joe's point isn't actually about Routes, or rather isn't "please stop using setuptools exclusively", it's "please stop using the cheeseshop exclusively" :), because some people (at least him and me) never think to look at the cheeseshop, at least not when already looking at a package's self-hosted "download" page for files not listed on that download page.

So I guess my complaint, and what I'm assuming is Joe's complaint re:setuptools is really only re:Routes' html. That they should 1) mention you will be installing setuptools and point out that you might need to read the manual before installing, especially if the two assumptions ez_setup makes aren't true in your case, and 2) they should list the files they offer, or at least link to the cheeseshop as the location of that list.

Anonymous said...

setuptools sucks! distutils is much easier!

Modifying EC2 security groups via AWS Lambda functions

One task that comes up again and again is adding, removing or updating source CIDR blocks in various security groups in an EC2 infrastructur...