News You Can Bruise for 2011 August 27 (entry 0)

(1) Sat Aug 27 2011 20:29 "No Module Named BeautifulSoup": Since Beautiful Soup 4 is not backwards compatible with Beautiful Soup 3, I put it in a different module: bs4 instead of BeautifulSoup. In a non-ironic twist, the module rename has itself turned out to be the biggest compatibility problem between BS3 and BS4. The new module name has caused problems on several occasions where users thought BS4 worked just like BS3, or didn't even know they were using BS4 (1 2 3).

Why would you be using BS4 without knowing it? It's an unreleased beta. Well, that's happened before. When I made the BS4 alpha release, I put the tarball in /software/BeautifulSoup/download/4.x, and PyPI picked it up because it knows /software/BeautifulSoup/download/ is where I keep my tarballs. PyPI believed the 4.0 alpha to be the latest release of BeautifulSoup and started recommending it it to all and sundry, which was not what I wanted. So I moved the 4.x tarballs into a different directory that PyPI doesn't know about: /software/BeautifulSoup/unreleased/4.x/, and that solved the problem.

But now it was happening again. Some installation process or other was finding my /unreleased/ directory, picking up the beta tarball, and installing it by default as the latest version of Beautiful Soup. Why?

Thanks to the bug Brian Shumate filed today I tracked the problem down. It turns out the pip package-installer program scrapes the Beautiful Soup homepage (using regexes, not Beautiful Soup, ha). It looks for tarballs and picks the one with the biggest version number. So just by linking to the beta tarball and giving it a "4.0" name I was declaring 4.0 ready for prime time.

So I got a couple problems:

• I've changed the link so I link to "BeautifulSoup-beta.tar.gz", which doesn't contain an explicit version number. I'm not terribly happy with this solution. Is there a way to tell pip to ignore certain tarballs? I've scanned the source code and it doesn't look like there is.
• People aren't going to stop writing from BeautifulSoup import BeautifulSoup even once BS4 is released. I don't see a way to make sure that a) both BS3 and BS4 can be installed on the same system, and b) if BS4 is installed but BS3 is not, from BeautifulSoup import BeautifulSoup works. Is there a way? My current plan is to punt, and rely on the fact that if I talk about this enough, a Google search for "No module named BeautifulSoup" will eventually point you to an explanation.

Comments:

Posted by Thad Guidry at Thu Sep 15 2011 11:41

Maybe it should not be called BeautifulSoup at all, since as you say it's clearly a very different bit of software and the functions have changed ? Sticking a core version number into the name might be the simplest fix ? But that breaks into other problems all on it's own, hmm...perhaps no number in the name is better ?

import NewBeautifulSoup from NewBeautifulSoup

import BeautifulSoup4 from BeautifulSoup4

import BeautifulSoupAgain from BeautifulSoupAgain