Via an email from SourceForge, I found out about splunk, a piece of software that indexes and searches log files (actually not only logs, but any "fast-moving IT data", as they put it). I downloaded the free version and installed it on a server I have, then indexed the /var/log/messages file and played with it a bit.
Here is the search results page for "Failed password". A thing to note is that every single word on the results page is clickable, and if you click on it a new search is done on that word. If you want to add the word to the current search words, click Ctrl and the word, or if you want to exclude the work from the search, click Ctrl-Alt and the word.
Pretty impressive. It uses various AJAX techniques to enhance the user experience, and best of all, part of the server software is written in Python! The search interface is based on Twisted:
root 504 1 0 11:26 pts/0 00:00:04 python /opt/splunk/lib/python2.4/site-packages/twisted/scripts/twistd.py --pidfile=/opt/splunk/var/run/splunk/splunkSearch.pid -noy /opt/splunk/lib/python2.4/site-packages/splunk/search/Search.tac
Definitely worth checking it out.