News You Can Bruise

The CHANGELOG for 4.13.0 is quite large so I'm writing this blog post to highlight just the most important changes, specifically the changes most likely to make you need (or want) to change your code.

Deprecations and backwards-incompatible changes

• DeprecationWarning is issued on use for every deprecated method, attribute and class from the 3.0 and 2.0 major versions of Beautiful Soup. These have been deprecated for at least ten years, but they didn't issue DeprecationWarning when you tried to use them. Now they do, and they're all going away soon.
• This version drops support for Python 3.6, which went EOL in December 2021. The minimum supported major Python version for Beautiful Soup is now Python 3.7, which went EOL in June 2023.
• The storage for a tag's attribute values now modifies incoming values to be consistent with the HTML or XML spec. This means that if you set an attribute value to a number, it will be converted to a string immediately, rather than being converted when you output the document.

  More importantly for backwards compatibility, setting an HTML attribute value to True will set the attribute's value to the appropriate string per the HTML spec. Setting an attribute value to False or None will remove the attribute value from the tag altogether, rather than (effectively, as before) setting the value to the string "False" or the string "None".

  This means that some programs that modify documents will generate different output than they would in earlier versions of Beautiful Soup, but the new documents are more likely to represent the intent behind the modifications. To give a specific example, if you have code that looks something like this:

  checkbox1['checked'] = True
  checkbox2['checked'] = False

  Then a document that used to look like this (with most browsers treating both boxes as checked):

  <input type="checkbox" checked="True"/>
  <input type="checkbox" checked="False"/>
  

  Will now look like this (with browsers treating only the first box as checked):

  <input type="checkbox" checked="checked"/>
  <input type="checkbox"/>
  

  You can get the old behavior back by instantiating a TreeBuilder with attribute_dict_class=dict, or you can customize how Beautiful Soup treats attribute values by passing in a custom subclass of dict.
• If you pass an empty list as the attribute value when searching the tree, you will now find all tags which have that attribute set to a value in the empty list--that is, you will find nothing. This is consistent with other situations where a list of acceptable values is provided. Previously, an empty list was treated the same as None and False, and you would have found the tags which did not have that attribute set at all.
• When using one of the find() methods or creating a SoupStrainer, if you specify the same attribute value in attrs and the keyword arguments, you'll end up with two different ways to match that attribute. Previously the value in keyword arguments would override the value in attrs.
• The 'html5' formatter is now much less aggressive about escaping ampersands, escaping only the ampersands considered "ambiguous" by the HTML5 spec (which is almost none of them). This is the sort of change that might break your unit test suite, but the resulting markup will be much more readable and more HTML5-ish.

  To quickly get the old behavior back, change code like this:

  tag.encode(formatter='html5')
  

  to this:

  tag.encode(formatter='html5-4.12')

  In the future, the 'html5' formatter may be become the default HTML formatter, which will change Beautiful Soup's default output. This will break a lot of test suites so it's not going to happen for a while.

New features

• The online documentation now includes full API documentation generated from Python docstrings.
• The new ElementFilter class encapsulates Beautiful Soup's rules about matching elements and deciding which parts of a document to parse. This gives you direct access to Beautiful Soup's low-level matching API. See the documentation for details.
• The new PageElement.filter() method provides a fully general way of finding elements in a Beautiful Soup parse tree. You can specify a function to iterate over the tree and an ElementFilter to determine what matches.
• The NavigableString class now has a .string property which returns the string itself. This makes it easier to iterate over a mixed list of Tag and NavigableString objects.
• Defined a new warning class, UnusualUsageWarning, which is a superclass for all of the warnings issued when Beautiful Soup notices something unusual but not guaranteed to be wrong, like markup that looks like a URL (MarkupResemblesLocatorWarning) or XML being run through an HTML parser (XMLParsedAsHTMLWarning).

  The text of these warnings has been revamped to explain in more detail what is going on, how to check if you've made a mistake, and how to make the warning go away if you are acting deliberately.

  If these warnings are interfering with your workflow, or simply annoying you, you can filter all of them by filtering UnusualUsageWarning, without worrying about losing the warnings Beautiful Soup issues when there *definitely* is a problem you need to correct, such as use of a deprecated method.

• Emit an UnusualUsageWarning if the user tries to search for an attribute called _class; they probably mean class_.

How to maintain a popular Python library for most of your life without with burning out

The quick takeaway is that strong boundaries are important: both the software boundaries provided by published APIs and packaging dependencies, and the decision as to where your volunteer open source work ends and the rest of your life begins. I have some suggestions for the ways the two interact, and an anecdote about how we mentally rewrite our memories of our struggles to make ourselves more active participants. If you're the maintainer of an open source project, I recommend checking it out!

This makes it easy to do things like change the rules about which attributes are treated as multi-value attributes. If you don't like how Beautiful Soup parses class into a list of CSS classes, this is the release for you. It's not a huge release, but this project's now fifteen years old so I'm relieved at how stable it's been.

Speaking of CSS, although this is a feature release, it's a little smaller than the 4.7.0 release I put out at the end of 2018. That one took out the lackluster implementation of CSS selectors, based on Simon Willison's "soupselect" project from the early 2010s. I replaced it with a dependency on Isaac Muse's SoupSieve project, which has a nearly complete CSS selector implementation. The old implementation was a common cause of complaints, but—like the HTML5 parsing algorithm—it's not something I have a strong interest in and I'm happy to give the whole job to an external dependency.

There was a period of about a year in 2017-2018 when I wasn't interested in doing Beautiful Soup work, but Tidelift changed that. Tidelift gathers subscription money from companies that rely on free software, and distributes the money to the developers in exchange for a level of support that I find sustainable.

Nobody builds an entire product around Beautiful Soup (or at least nobody will admit do doing this), but thousands of people have used Beautiful Soup to save time at their day jobs. Bundling Beautiful Soup together with bigger projects like Flask and numpy is a solution that works really well for me.

I've fixed 17 bugs, added some minor new features, and changed the implementations of __copy__ and __repr__ to work more like you'd expect from Python objects. But in my mind the major new change is this: I've added a warning that displays when you create a BeautifulSoup object without explicitly specifying a parser:

UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently. To get rid of this warning, change this:

BeautifulSoup([your markup])

to this:

BeautifulSoup([your markup], "lxml")

It's a little annoying to get this message, but it's also annoying to have your code silently behave differently because you copied it to a machine that didn't have lxml installed, and it's also annoying when I have to check pretty much every reported bug to see whether this is the problem. Whenever I think I can eliminate a class of support question with a warning, I put in the warning. It saves everybody time.

The other possibility: now that Python's built-in HTMLParser is decent, I could make it so that it's always the default unless you specify another parser. This would cause a big one-time wrench, as even machines which have lxml installed would start using HTMLParser, but once it shook out the problem would be solved. I might still do that, but I think I'll give everyone about a year to get rid of this annoying warning.

Anyway, try out the beta. Unless there's a big problem I'll be releasing 4.4.0 on Friday.

4.3.2 itself is a pretty minor bugfix release. Still left unfixed is a bug I can't reproduce because the federal government is shut down. When you file a bug that happens with a specific web page, please provide the HTML of the web page, not the URL.

Here are the release notes. The main new features are a much more capable CSS selector engine, and a diagnostics module that should help with tech support.

If anyone knows how to convince everyone to use PyPI, I'd appreciate the knowledge. But it's not a big deal right now, and it gives me some visibility into how people are using Beautiful Soup. Visibility which I will share with you.

Yesterday, the 17th, the Beautiful Soup 4.1.3 tarball was downloaded 2223 times. It is by far the most popular thing on crummy.com. The second most popular thing is the Beautiful Soup 3.2.1 tarball, which was downloaded 381 times. The vast majority of the downloads were from installation scripts: distribute or setuptools.

1516 distinct IP addresses were responsible for the 2223 downloads of 4.1.3. I wrote a script to find out how many IP addresses downloaded Beautiful Soup more than once. The results:

Downloads from a single IP	Number of times this happened
55	1
35	1
15	1
13	1
11	1
5	2
4	12
3	43
2	453
1	1001

Naturally my attention was drawn to the outliers at the top of the table. I investigated them individually. The IP address responsible for 55 downloads is a software company of the sort that might be deploying to a bunch of computers behind a proxy. The 35 is an individual on a cable modem who, judging from their other traces on the Internet, is deploying to a bunch of computers using Puppet. The 15, the 13, and the 11 are all from Travis CI, a continuous integration service.

One of the two 5s was an Amazon EC2 instance. Five of the twelve 4s were Amazon EC2 instances. Thirty-seven of the forty-three 3s were Amazon EC2 instances. And 395 of the 453 double-dippers were Amazon EC2 instances. Something's clearly going on with EC2. (There was also one download from within Amazon corporate, among other BigCo downloaders.)

I hypothesized that the overall majority of duplicate requests are from Amazon EC2 instances being wiped and redeployed. To test this hypothesis I went through all the double-dippers and calculated the time between the first request and the second. My results are in this scatter plot. Each point on the plot represents an IP address that downloaded Beautiful Soup twice yesterday.

For EC2 instances, the median time between requests is 11 hours and 45 minutes. So EC2 instances are being automatically redeployed twice a day. For non-EC2 instances, the median time between requests is 51 minutes, and the modal time is about zero. Those people set up a dev environment, discover that something doesn't work, and try it again from scratch.

import urllib2
from bs4 import BeautifulSoup

url = 'http://www.poemhunter.com/best-poems/william-stafford/thinking-for-berky/'
soup = BeautifulSoup(urllib2.urlopen(url))

print soup.find(attrs="title").text
for s in soup.find(attrs="poem").strings:
    print s.strip()

I'll stop beating around the bush: sometimes otherwise UTF-8 documents contain Microsoft smart quotes. This isn't terribly common, but when it happens there's been no easy way to convert that document to Unicode... until now. Beautiful Soup 4.1.0, released today, adds the method UnicodeDammit.detwingle(). This method converts a mixed UTF-8/Windows-1252 document to pure UTF-8, allowing you to run it through BeautifulSoup() or UnicodeDammit() and get Unicode.

I'll let the documentation give the details. In theory I can expand detwingle() to handle other pairs of encodings, but UTF-8/Windows-1252 is the only one currently supported. I'm imagining adding support for other popular encoding pairs, maybe EUC-JP + Shift JIS. But I'm not imagining writing that code, just incorporating patches from other people.

If you're ever in this situation, try it out and let me know how it works.

Beautiful Soup 4.1.0 also includes a bunch of medium-level bug fixes, and a major refactoring of the search code that will hopefully have no effect whatsoever on the way searches work.

I've also updated the docs quite a bit to help people solve common problems. I'm not sure where to stop, because Beautiful Soup is the first Python library a lot of people use, so it gets caught up in questions like "how do I install Python packages on Windows?" (It's not easy.)

If you're just tuning in, Beautiful Soup 4 is nearly a complete rewrite that works on Python 2 and Python 3. Instead of a custom-built parser from 2006, Beautiful Soup 4 sits on top of lxml (for speed) or html5lib (for browser-like parsing) or the built-in HTMLParser (for convenience). Methods and attributes are renamed for PEP 8 compliance, and Beautiful Soup 4 incorporates the soupselect project to provide basic CSS selector support. I completely rewrote the documentation, Beautiful Soup's secret weapon since 3.0, for clarity and completeness.

That's the major stuff. Even though most of the code has changed, my goal was not to add a bunch more features, but to make sure Beautiful Soup will still be usable and useful years into the future.

Beautiful Soup 4 is mostly but not entirely backwards compatible with Beautiful Soup 3. Most users should be able to switch from 3 to 4 just by changing an import line. In the Python tradition of sticking a number on the end of your module name when you break backwards compatibility, I've released it as a separate package, beautifulsoup4.

This release also inaugurates the Beautiful Soup Hall of Fame, featuring the uses of Beautiful Soup that I personally find the coolest or highest-profile.

So, try out Beautiful Soup 4 the next time you need to do some screen-scraping. If you've used Beautiful Soup 3, I think you'll be pleasantly surprised. If not, I'll just say I hope you like it.

I've thanked them before, but special thanks are once again due to Thomas Kluyver and Ezio Melotti for helping me get everything working under Python 3.

[0] The first release is called 4.0.1 instead of 4.0.0 because I've been bitten by clever packagers before and I don't want them thinking "4.0.0" is an earlier version than "4.0.0b10".

Beta 8 adds very basic namespace awareness. By "basic" I mean:

1. Handle documents that include namespaced tags and attributes without crashing or mangling the document on output.
2. If the parser provides namespace information for a tag or attribute, store it for the user's reference instead of discarding it.

That's it. No one responded to my request for namespace-related feature requests, so I'm doing the bare minimum.

The big news is a new release of the 3.x series, Beautiful Soup 3.2.1. This fixes a pretty bad problem that can let through cross-site scripting attacks if you use Beautiful Soup to sanitize HTML. If that's you, you should upgrade ASAP.

That was certainly worth fixing, but I don't do much work on Beautiful Soup 3 anymore. I mean, if I fixed every bug in BS3, I'd have... Beautiful Soup 4, which is now almost done. All the bugs are closed out. There's one more big feature I may add, and some minor cleanup I want to do, but mainly I want to make sure people are comfortable with the new API.

Thanks to Stefano Rivera, BS4 is now in Debian unstable and Ubuntu Pangolin, as beautifulsoup4. So the clock is ticking on freezing the API. This would be a great time to try to port your BS3 scripts to BS4, and let me know how difficult it was and what you had to change.

Big changes:

• If you're using Python 3.2, the built-in html.parser is now reliable enough to use on its own. You don't need to install lxml or html5lib just to parse bad HTML (but lxml is still a lot faster). The forthcoming Python 2.7.3 should also work this way.

  This is of course a feature of Python, but due to a pretty bad bug in html.parser, I wasn't taking advantage of it. I worked with Ezio Melotti to monkeypatch that bug from within BS, and now we're back in the very good situation of not needing any external dependencies.

• new_tag() will follow the rules of whatever tree builder was used to create the original soup. For example, a new <p> tag will look like "<p />" if you're dealing with XML, but it'll look like "<p></p>" if you're dealing with HTML.
• There's now a new_string() method to go along with new_tag().
• There are two new methods for manipulating the tree: PageElement.insert_before() and PageElement.insert_after().
• I replaced the substitute_html_entities argument with the more general formatter argument. You can do all sorts of crazy stuff with this.
• The default formatter converts bare ampersands and angle brackets to XML entities, but doesn't touch HTML entities. I think it's kind of America-centric to convert characters like é to &eacute; by default, but I might make the default a "punctuation" formatter that converts things like curly quotes to HTML entities.

After talking with Ezio today, I was able to monkeypatch BS4 to avoid the bug in 3.2.2. This means on Python 3, BS4 with no external parser installed will give reliability comparable to BS4+lxml (98% versus 99%). It's still about 50% slower, though, parsing about 1300 kb of HTML per second, versus 2100 kb/second for BS4+lxml.

Introduction

When Beautiful Soup was first released in 2004, the state of HTML parsing in Python was appalling. Over the past eight years, things have improved so dramatically that Beautiful Soup's HTML parser is no longer a competitive advantage. I don't want to duplicate other peoples', work, so I'm getting Beautiful Soup out of the parser businesss. Beautiful Soup's job is now to provide a Pythonic screen-scraping API on top of a data structure created by a third-party parser.

This will be Beautiful Soup 4, and I've been planning it for years. With help from Thomas Kluyver and Ezio Melotti, I've now met the three main goals of Beautiful Soup 4:

1. Make a single codebase that works under Python 2 and Python 3.
2. Stop using SGMLParser (removed in Python 3) and make it possible to swap out one parser for another.
3. Support two major Python parsers (lxml and html5lib) as well as Python's (not currently very good) batteries-included parser, html.parser.

On Python 2 or Python 3 you can install the BS4 beta with this command:

easy_install beautifulsoup4

You can also get the source tarball.

The documentation has been completely rewritten. You may find the section on porting BS3 code to BS4 especially interesting.

There are three major things I'd like your feedback on before completing the release.

Hall of Fame

The BS3 documentation lists open-source projects that use Beautiful Soup. I stopped maintaining this list many years ago because there are hundreds of these projects, and since most of them are screen-scrapers, they're pretty ephemeral.

I'd like to bring this feature back as a "hall of fame", featuring applications of Beautiful Soup that grab a reader's attention. People who used Beautiful Soup in a high-profile way or to tackle a big issue. Projects that are interesting to hear about even if the software doesn't work anymore, or uses an old version of Beautiful Soup, or if Beautiful Soup was used internally and the public only saw the results.

My bias is towards projects having to do with space, science, journalism, politics and social justice. Here are some examples so you know the kind of thing I'm thinking of:

• "Movable Type", a work of digital art on display in the lobby of the New York Times building, uses Beautiful Soup to scrape New York Times feeds.
• Alexander Harrowell uses Beautiful Soup to track the business activities of an arms merchant.
• The Lawrence Journal-World used Beautiful Soup in 2006 and 2010 to gather election results.
• The NOAA's Forecast Applications Branch uses Beautiful Soup in TopoGrabber, a script for downloading "high resolution USGS datasets."

If you did anything of this sort, or know of someone who did, I'd like to hear about it.

Do you prefer lxml or html5lib?

Right now, the parser ranking goes lxml, html5lib, html.parser. I like lxml because it's incredibly fast and it can parse anything. But I'd like to see what you think of the trees it generates. Would html5lib, with its web-browser-like heuristics, be a better default?

substitute_html_entities

BS3 had a number of overlapping and inconsistent ways of turning HTML/XML entities into Unicode characters, and possibly turning Microsoft smart quotes into HTML entities at the same time. In BS4, all this stuff is gone. HTML and XML entities are *always* converted into Unicode characters.

This is great but there's one problem: output. If you want to turn those Unicode characters back into entities when outputting as a string, you need to call soup.encode(substitute_html_entities=True), which is a little clunky. I'm thinking of adding an output_html_entities attribute that you can set on a soup or tag to control whether this substitution happens. Do you like this idea?

I think I also need to ensure that characters like "&" and "always converted to XML entities on output, even though this will hurt performance a bit.

Conclusion

What you install with easy_install beautifulsoup4 is a beta release. If I hear of a problem soon, there's still time to fix it, even if it means a major change to the API. So please try it out and give me feedback.

Ezio Melotti pointed me to a list of the top 10,000 domains worldwide, according to some random source. It looked legit, so I wrote a script to download the homepages of the top 200 domains as served to a desktop web browser. My dataset included many pages written in Chinese, Japanese, Russian, Portuguese, Polish, and German.

For every parser I was interested in, I parsed each homepage and timed the parse. This gave me 200 numbers for every parser. To reduce that to a single non-huge number I calculated a mean: how many kilobytes of real-world HTML the parser could process in a second. I also noted each parser's success rate: how many of the 200 homepages it had handled without raising an exception.

Here are the results, ordered by their performance under Python 2.7.

	Python 2.7		Python 3.2
Parser	Speed (KB/s)	Success rate	Speed (KB/s)	Success rate
Beautiful Soup 3.2 (SGMLParser)	211	100%	-	-
html5lib (BS3 treebuilder)	253	99%	-	-
Beautiful Soup 4.0 + lxml	255	100%	2140	96%
html5lib (lxml treebuilder)	270	99%	-	-
Beautiful Soup 4.0 + html5lib	271	98%	-	-
Beautiful Soup 4.0 + HTMLParser	299	59%	1705	57%
html5lib (simpletree treebuilder)	332	100%	-	-
HTMLParser	5194	52%	3918	57%
lxml	17925	100%	14258	96%

Note that the "HTMLParser" tests don't actually produce anything. HTMLParser is an event-based parser, so when the HTML is parsed, nothing comes out because I didn't include any handler code. All the other tests build a parse tree in memory.

Another thing to keep in mind about the html5lib results: html5lib is kind of the opposite of BS4. BS4 always builds a tree of Beautiful Soup objects, but you can tell it to generate that tree using html5lib, lxml, or HTMLParser. Whereas html5lib always uses its own parser, but you can tell it to build a tree of lxml objects, a tree of BS3 objects, etc.

The big surprise for me is that on Python 2.7, lxml is the worst choice for a parser to drive BS4. It's a worse choice than html5lib! How did that happen? I have no idea. I was hoping to cash in on the lxml magic (see below), and it's not working. I need to look into this. Notice that html5lib takes a performance hit from using lxml's treebuilder. If the magic's not in the treebuilder and it's not in the parser, where is it?

Unless I can find that magic and exploit it, it remains the case that if you're paying by the minute for computer time, you should use lxml. It's written in C, and on Python 2.7 it builds a parse tree sixty times faster than BS4, three times faster than a pure-Python parser that does absolutely nothing with the data. Even on Python 3, lxml alone is seven times faster than BS4+lxml. I said stuff like this in the BS3 documentation, but I think I need to be more forceful about it in the BS4 docs.

The good news is that Beautiful Soup is 6-8 times faster on Python 3 than it is on Python 2, and even at its slowest, BS4 is noticeably faster than BS3.

The big caveat is that my definition of "success" is pretty minimal. Just because the parser parsed the file without crashing doesn't mean it will give you a useful parse tree.

Another caveat: on Python 3, I couldn't get HTMLParser to take raw bytes as input, so I ran the data through UnicodeDammit first. I counted this time as part of the parse time. This probably explains HTMLParser's slower speed on Python 3 and its higher success rate.

Update: Argh, I found out about this a year ago. The problem is that Unicode, Dammit is incredibly slow in some cases. Here are the results on 2.7 if I take out the prepare_markup methods in the builders for HTMLParser and lxml, and just assume everything's UTF-8:

	Python 2.7		Python 3.2
Parser	Speed (KB/s)	Success rate	Speed (KB/s)	Success rate
Beautiful Soup 4.0 + lxml	2287	96%	2600	96%
Beautiful Soup 4.0 + HTMLParser	2069	48%	1680	57%

That's more like it! The problem is that reliability suffers. Both parsers crash in the 4% of cases where it's not UTF-8 but the encoding is declared in a <meta> tag. And there's an unknown number of cases where the data's not UTF-8 but the conversion doesn't crash, leading to garbled data. But at least now I remember this problem.

Also note that on Python 3.2, getting rid of Unicode, Dammit doesn't matter nearly as much. (It doesn't matter for HTMLParser at all.) Presumably Python 3.2 has better built-in support for encoding autodetection.

I'm still not sure about the best way to distribute this package, either while it's beta or afterwards. I'll probably end up creating a new project on PyPi, because otherwise people who install programs that easy_install beautifulsoup will crash due to the module's new name. Does that make any sense?

Anyhow, we're almost at the end of this fitfully travelled road. Once I figure out distribution and rewrite the documentation, a) no one should need to use BS3 anymore if they don't want to, and b) it should be possible to get lxml-like performance or html5lib-like flexibility with a Beautiful Soup API, by actually using lxml or html5lib as the underlying parser.

PS: remember, it's now from bs4 import BeautifulSoup.

Why would you be using BS4 without knowing it? It's an unreleased beta. Well, that's happened before. When I made the BS4 alpha release, I put the tarball in /software/BeautifulSoup/download/4.x, and PyPI picked it up because it knows /software/BeautifulSoup/download/ is where I keep my tarballs. PyPI believed the 4.0 alpha to be the latest release of BeautifulSoup and started recommending it it to all and sundry, which was not what I wanted. So I moved the 4.x tarballs into a different directory that PyPI doesn't know about: /software/BeautifulSoup/unreleased/4.x/, and that solved the problem.

But now it was happening again. Some installation process or other was finding my /unreleased/ directory, picking up the beta tarball, and installing it by default as the latest version of Beautiful Soup. Why?

Thanks to the bug Brian Shumate filed today I tracked the problem down. It turns out the pip package-installer program scrapes the Beautiful Soup homepage (using regexes, not Beautiful Soup, ha). It looks for tarballs and picks the one with the biggest version number. So just by linking to the beta tarball and giving it a "4.0" name I was declaring 4.0 ready for prime time.

So I got a couple problems:

• I've changed the link so I link to "BeautifulSoup-beta.tar.gz", which doesn't contain an explicit version number. I'm not terribly happy with this solution. Is there a way to tell pip to ignore certain tarballs? I've scanned the source code and it doesn't look like there is.
• People aren't going to stop writing from BeautifulSoup import BeautifulSoup even once BS4 is released. I don't see a way to make sure that a) both BS3 and BS4 can be installed on the same system, and b) if BS4 is installed but BS3 is not, from BeautifulSoup import BeautifulSoup works. Is there a way? My current plan is to punt, and rely on the fact that if I talk about this enough, a Google search for "No module named BeautifulSoup" will eventually point you to an explanation.

There's still some work to do, and it'll be a while before I get to it, but the work that remains is pretty minor compared to the advantages you get from using BS4 instead of BS3. Try out the beta, and if it gets good reviews I may just make a 4.0 release and deal with the minor things afterwards.

Try it out and file bugs. Check out my weblog archives for a glimpse of what's new in BS4.

But this is just the tip of a general feature. "html" and "xml" are just strings, features for which a tree-builder might or might not advertise support. The tree builders also publish other features, like "fast", "permissive", "html5", and library names like "lxml". So you can make semi-fine distinctions:

BeautifulSoup(markup, ["html", "fast"])
BeautifulSoup(markup, ["html", "permissive"])
BeautifulSoup(markup, ["html", "lxml"])

The BS constructor will try to find the best tree-builder that matches all the features you specify, and will raise an exception if it can't match them all (because you don't have lxml installed or something).

This is overkill right now because there are only three tree-builders (["lxml", "xml"], ["lxml", "html"], and ["html5lib"]). But this gives me an easy way to add tree-builders to the code base, and for you to plug in additional builders, without making end-users learn where the classes are.

This is looking good enough that I can do an alpha release soon. I'm not sure why I've been putting so much work into BS, but I'm sure it has something to do with the fact that my other projects are stalled, blocked, or I want to procrastinate on them.

[0] Those little classes like ICantBelieveItsBeautifulSoup are also gone, because distinguishing between different techniques for parsing markup is now the parser's job. And those classes were kind of silly to begin with.

I also decided this would be a good time to run a performance test. Here's a moderately sized document:

Document is 66409 bytes
BS4 lxml time: 0.06
BS4 html5lib time: 0.25
BS3 time: 0.15

("BS3" here is the latest released version, 3.2.0.)

Pretty good! And here's a huge complicated document:

Document is 1329825 bytes
BS4 lxml time: 12.60
BS4 html5lib time: 2.88
BS3 time: 14.11

Okay, that's kind of random. The problem is in Unicode, Dammit. It takes a long time to figure out the encoding for this particular page. This is ultimately because the document is in ISO-8859-2, but it includes a <meta> tag that claims the document is in UTF-8. I don't yet understand the problem on any deeper level than that. If it's gonna be like that, I may just stop believing anything I see in a <meta> tag.

If you specify the encoding up front, the lxml time drops to 0.90 seconds. The html5lib tree builder doesn't have the problem because it uses html5lib's native Unicode conversion functionality instead of Unicode, Dammit.

Incidentally, BS3 has the same problem. Specify the encoding up front, and BS3 takes about 2 seconds on this page, which makes sense--faster than html5lib but slower than lxml. I find it very annoying that I'm only discovering this problem now—I think this has wasted a lot of cumulative time over the past couple years.

Anyway, now is a good time to start trying out BS4, if you're a fan of new things. I renamed all the major methods to be PEP-8 compliant--details are in the CHANGELOG.

Update: Profiler shows the bottleneck is in the chardet library, sbchardetprober.py, which goes through the file character by character and crunches some numbers for each character. If it can't make a decision until late in a huge file, there's your twelve seconds. So... I have a couple ideas, but it's not a bug in my code that I can just fix. But html5lib uses chardet, so it must be doable.

Fun fact: it doesn't seem to be legal to stick a CDATA section into an HTML document. (By this I mean something like "<![CDATA[foo]]>", not the contents of a <pre> tag.) My knowledge of weird HTML constructs like CDATA comes mostly from studying Python's SGMLParser, which handles CDATA sections just fine since it's an SGML parser. So I had BS3 just create objects for CDATA sections, even when they occurred in HTML documents. But the two parsers I'm using as my testbeds for BS4 basically ignore CDATA sections in HTML documents:

By default, lxml's elementtree implementation replaces CDATA sections with the actual character data, and has an option to leave the CDATA sections alone, but this only works for XML. When parsing HTML, CDATA sections are ignored altogether. The HTMLParser constructor has a "strip_cdata" argument, inherited from XMLParser, but setting it to False does nothing.

BS4 can't be used to parse XML yet (unless you want to parse it by HTML rules), but once I add that, I'll have the lxml elementtree builder preserve CDATA sections.

html5lib treats CDATA sections as broken comments, so "<![CDATA[foo]]>" becomes "<!-- [CDATA[foo]] -->" The latest version of html5lib will replace a CDATA section with the character data if the CDATA section happens within a <svg> or <math> tag (see test), but this is not in any released version.

That took me the morning to figure out, so I hope it saves someone some time. But that person would have to bear a suspicious resemblance to me.

In BS3 you could choose to convert incoming entities into Unicode characters, or to leave them alone. You could also choose to convert Microsoft smart quotes into Unicode characters, XML entities, or HTML entities. In BS4 this will depend on the parser. Both lxml and html5lib convert everything to Unicode. I think this makes more sense--convert absolutely everything to Unicode, use Unicode internally, and optionally convert back to entities when writing the document out. (I'll probably have to write the "convert back to entities" part.)

The story so far: the most recent release of Beautiful Soup (3.2) uses a custom parser based on Python's standard-library SGMLParser. This was a really good parser back in 2005. Here in 2011, html5lib is better at handling bad markup, and lxml and ElementTree are much faster if the markup isn't too bad. Beautiful Soup's parser is no longer a competitive advantage.

What's more, SGMLParser goes away in Python 3, and its replacement is awful at handling bad markup. I tried to switch over in early 2009 and it just didn't work for anyone. So, Beautiful Soup has had the specter of death looming over it for two years.

Beautiful Soup 4 will not be a parser at all. It will be a tree-builder. You will plug a parser into Beautiful Soup, and you'll get an object tree that reflects how that parser sees a document. I have this working reasonably well for lxml and html5lib, which is why I'm comfortable announcing the project now.

Problems this will solve:

• I don't know whether the initial release of BS4 will be ready to run under Python 3, but at least I'll see how to push it in that direction. The software will no longer be doomed.
• If you use Beautiful Soup to scrape valid markup (we do this in the Launchpad test suite), you can plug in lxml and the parser will run much faster.
• You can plug in html5lib instead and scrape really awful markup that no previous version of BS could handle.
• You can write your own code to plug some random parser into Beautiful Soup. I have written a generic SAX event handler, though I don't know if it will make it into the final release.

Problems this will not solve

• I still don't intend to work on Beautiful Soup on any regular basis. I won't be redesigning the API (though I will be renaming the methods to conform with PEP 8) or even necessarily fixing bugs. I'm just trying to stave off death for a library that's used in hundreds of software projects.

I'll be spending alternate Fridays working on Beautiful Soup 4. I'll probably have a beta release in a few months. I'll be pushing my progress to this branch.

(nb. My code-in-progress includes some code from html5lib, and I'm not sure how the licenses interact, but since html5lib is MIT licensed I think it's just a matter of adding some more boilerplate, so I'm not too worried about it.)

Previously, there were a couple versions of Beautiful Soup you might conceivably use. They were 3.0.8.1 and 3.1.0. Version 3.0.8.1 was the stable release, and version 3.1.0 was the development branch. Except 3.1.0 wasn't actually the "development" branch, since a) no development is happening right now, and b) 3.1.0 in particular turned out to be a huge mistake. But for almost two years now, 3.1.0 has been hanging around with a higher version number than 3.0.x, looking like the latest version. And people have been using it, and having problems.

I'm not working on BS right now, but the least I can do is stop actively misleading people, so I added a tiny new feature suggested by Aaron and leapfrogged the 3.1 series with the UNAMBIGUOUSLY THE VERSION YOU SHOULD USE version 3.2.0. I've also set up a new build system which uploads the new releases to Crummy, to PyPI, and to Launchpad. So if my hosting provider randomly shuts down my instance, as happened last night, your craving for the tarball can still be fulfilled.

So I emailed Mark and asked when he would be in New York next. He said he would be in New York starting Thursday! Okay then. Yesterday I went to the Times building (near Times Square, strangely enough) and had lunch with Mark and with Jer Thorp, in the second best corporate cafeteria I've ever eaten in, not that I've eaten in very many.

Great conversation and then a little tour of (selected floors of) the Times building, which you can see a bit of in the pictures. Don't miss the "Page One" room, a room dedicated to a single ritual, where the editors gather every day at 4 PM to decide what goes on the next day's front page.

But, you may ask, what is this picture I've embedded align="right" in this entry, this photo of me standing in front of some screens? Well, I'm looking at Movable Type, a 2007 piece of digital art by Mark Hansen and Ben Rubin. It's on public display in the Times lobby. It shows a constantly changing tableau of sentences, photo captions, crossword puzzle clues, etc. from the latest edition of the NYT. And it turns out that the "custom software" mentioned in the artist's statement includes Beautiful Soup! Yes, the New York Times is using my software to scrape its own data stores for use in a work of art. I did not begin this week expecting to learn that!

[UPDATE] I should be more precise in my language, especially since I work for Canonical. Maverick did not "get rid of" F-Spot. It was moved from "main" to "universe", and replaced by Shotwell. Shotwell is much better, except that when you tell it to export photos to a directory, it exports only the photos themselves and no extra metadata or thumbnails.

I want to briefly mention a weird thing I came home to. Although I don't use Twitter, I do occasionally search for people who link to my site in their Twitter feeds. Usually they're linking to Beautiful Soup. Today I saw a typical example:

Rubyist, pythonista or even php fan. We can all love http://bit.ly/7aRD8T amazing parser library #beautifulsoup

The atypical part was that this was from Paula Deen's Twitter feed. Paula Deen has a show on the Food Network, and I associate her with making food out of butter and hanging out with Jimmy Carter. I could see her tweeting about beautiful soup, but not Beautiful Soup.

I guess what happened is that, like many celebrities, Deen has a tech person who posts on her behalf, and the tech person forgot to sign out as Paula_Deen before posting to what they thought would be their own Twitter feed. The rogue tweet alarmed some people (example), and has been removed, but this follow-up that refers to Deen in the third person bolsters the "tech person" theory:

Paula hasn't gone techy, she's off today but getting ready to head to Alaska for a working vacation.

Paula Deen's tech person, I'm glad you like Beautiful Soup, and everyone makes embarrassing mistakes, so don't feel bad.

I've come up with a Linux kernel-like system where 3.1.x is retconned into the 'development' branch. Hopefully sometime on this vacation I'll decide to finish the 'development' work and release a Beautiful Soup version 4 that everyone can use.

Here's the direct download.

Hopefully this will be useful to the maintainers of libraries like lxml and html5lib who currently jump through hoops to make their parsers generate a Beautiful Soup parse tree. Now you should be able to just maintain a TreeBuilder implementation. The tree builder has a very simple interface, so take a look and make sure it does what you need. I'll be writing an html5lib tree builder and packaging it and the lxml builder in Beautiful Soup for a while, but I think long-term the TreeBuilders should live with their parent projects.

Tomorrow I'll be figuring out how to package this and trying to come up with a compatibility suite that will ensure your tree builder reacts sanely to different trees and different Beautiful Soup setups like SoupStrainers.

But as part of Launchpad Performance Week, everyone's working on improving Launchpad's performance. As it happens, Launchpad uses Beautiful Soup behind the scenes in the test suite, and I was offered the chance to fix the problem on company time. (Fixing the problem will allow us to plug in lxml as the parser, making the tests run faster.)

It's going pretty well. I don't know if I'll be done with the whole thing by Friday, but it's definite progress on something I'd pretty much written off for the near future.

I haven't found a way in Python 2 code to indicate that a string should be converted into a byte string when the conversion script runs. In some cases I can stick .encode() onto the end and it works in both 2 and 3, but some of my tests have random binary data that's not in any particular encoding. And in some cases calling .encode() is just ugly. Kind of frustrating because about 40% of my test failures ultimately boiled down to marking such-and-such a string as a byte string. So I'd appreciate any ideas.

I think a couple people were confused by my earlier statement that you'd be able to "write a plugin for lxml [or] lib5html." I'm talking about using another parser to drive Beautiful Soup tree generation. Turning events generated by some other parser into a generic set of "start tag", "end tag" type events. Thus giving you an alternative to the okay-for-2004-but-not-for-2008 Beautiful Soup rules about parsing bad HTML, and eventually getting rid of those rules altogether, because I don't want to be in that business.

sgmllib doesn't exist in Python 3, so I switched to HTMLParser, which has gotten a lot better at parsing bad HTML. With my hacks in place, only 3 of my unit tests pass under sgmllib but fail under HTMLParser. That's acceptable given that my switching to HTMLParser creates part of the framework I'll use so that you can write a plugin for lxml, html5lib (not as slow as I'd thought), or some other parser. Eventually I'll get rid of the HTMLParser plugin, or at least strip it down so that it doesn't know anything about HTML, making my life easier.

What's left is some minor syntax problems and some huge problems dealing with the way strings work in Python 3 as they go in and out of encodings. At this point I need to stop hacking on BS and do some experiments to get a good understanding of the string changes.

1. Convert the codebase to Python 3; or rather, convert it to Python 2 that can be automatically converted to Python 3.
2. Aaron DeVore has a number of interesting additions to the API. They need to be added.
3. Separate the tree-walker from the tree-builder. Start getting out of the business of writing tree-builders.
4. Simplify the API.

Aaron is doing the integration work. I'm doing the conversion. Once that's done we'll have a big soup that people can use into the distant future if they don't like what I do in steps 3 and 4, or if I give the whole thing up, which is a distant possibility.

Step 3 is still a mystery. Apparently html5lib is even slower than SGMLParser, so generic is probably the way to go. Maybe something with bindings to lxml and html5lib (this is why you may not like what I do in this step and may want to stick with the all-Python, not-as-slow-as-html5lib version).

What's going to happen in step 4? Ian Bicking wrote an appreciation of lxml that ties in with my Beautiful Soup angst. Mostly to do with the tree-builder, but also rightly bashing BS's primitive CSS matcher. Implementing CSS selectors or XPath is another business I don't want to be in, but I wouldn't mind bundling with someone else's strategies for walking the tree according to CSS selectors or XPath.

I'm not terribly motivated to make these changes because I don't really use Beautiful Soup anymore. Partly because I don't do as much non-work programming as I did pre-Canonical, and partly because the sites I used to screen-scrape back in 2004 have wised up and developed web services or syndication feeds. Redoing the library doesn't feel like a fun use of my end-of-year vacation; I'd rather write stories.

But here's where I start when I think about the changes. For me, the core users of Beautiful Soup are the total newbies, people for whom BS is their first or second Python library. I've never made a secret of the fact that BS is slower than other parsers, and although coupling it with a C tree-builder would speed it up a lot, I'm totally serious when I say the point of BS is to save programmer time at the expense of processor time. If you need speed, you've got options. My overriding concern is people who've just realized they can get the information they need off a web site by writing a computer program.

Unfortunately, about 30% of those people have some specific need that goes beyond the simple API you see in Beautiful Soup 1.x, and over time those additional ways of searching got stuck into the API, and you get the Microsoft Word problem. The complexity of the API is now itself costing programmer time. It needs to stop. (But after I get Aaron's additions in so that I'll have more raw data to do my redesign with.)

So what I'd like to try is a stripped-down Beautiful Soup based on list comprehensions, with a little bit of syntactic sugar for total newbies. Matching is done by calling a factory function that returns the equivalent of a SoupStrainer. The factory functions contain most of what's currently in searchTag. Additional factory functions can implement CSS selectors or XPath, except at this point you should be able to just use a parser that has its own support for CSS selectors or XPath.

Don't even get me started on what I need to do to NewsBruiser eventually...

Earlier this year I quietly retired Rubyful Soup because I think _why's hpricot does a better job of being a Rubyish screen-scraping parser than RS can be. But nothing similar has happened in Python, mainly because BS is the market leader. I want to keep that going, but I also want to take advantage of the work that's gone on in this field since 2004.

So, what are the useful features of Beautiful Soup?

1. It can build an object model out of bad HTML.
2. It can build an object model out of bad XML, if you tell it the rules of your XML vocabulary. (This is just the general case of #1.)
3. It can convert almost any encoding into Unicode, usually in the absence of an explicit encoding marker or the presence of an incorrect one.
4. It exposes a useful API. It's easy to learn, more Pythonic than CSS selectors or XPath, and it includes most common ways of traversing the tree.

Of these, the only one I really care about is #4. If I could rid myself of the need to handle all the edge cases in #1-#3, edge cases that in many cases have outstripped my ability to solve them with my current tools and sanity, I'd be happy.

Fortunately, there's html5lib, which is supposed to be as good at parsing HTML as a web browser.

My current plan is to write something that goes on top of html5lib and gives the BS API to whatever DOM you've built. This would take care of #1 and #3. It's not clear to me how you tell html5lib the rules of your XML vocabulary; maybe it only parses valid XML. But BS is relatively rarely used to parse invalid XML, so if I could outsource all the HTML and Unicode crap to html5lib I'd be much more inclined to hack randomly on BS, so I think it's a fair trade.

html5lib already has a "beautifulsoup" tree builder, which creates a tree of Beautiful Soup objects. So in theory I would just need to maintain those objects? I'll find out soon enough.

I bought the Waldrop book as a biography of Licklider (actually I bought it because of the BRIGHT YELLOW COVER that blocked out all other books in the bookstore), but it really strains to fill a whole book with his life and make it interesting. Fortunately the book's method of straining is to bring in all the other characters of American computer science from the 1940s through the 1970s, with a theme of Licklider as the networker between them, and that was very entertaining. Though I admit part of the entertainingness was the celebrity-spotting equivalent of those old Looney Tunes cartoons that feature charicatures of all the Warner Bros. contract stars of the 1940s. Oh look, it's {Peter Lorre,John McCarthy}.

The most interesting new information I found in the book is the history of ARPA, specifically of the Information Processing Techniques Office (Licklider was IPTO head twice). The major theme is the Vietnam-era mission anticreep that slowly pushed ARPA from advanced research projects, to the more circumscribed realm of advanced research projects that we can use on the battlefield within two years without fundamentally changing anything (somewhat parallel to the problems Xerox PARC had). Meanwhile Licklider's trying to scrape together a couple million to fund his crazy "Intergalactic Network" project, and his office is sticking boilerplate defense justifications onto incoming grant proposals so they can get ARPA money.

"All that language about military rationale wasn't in the Stanford version of the proposals," explains Ed Geigenbaum: it was slapped on at the very end by the ARPA funding officers back in Washington. "The only people who ever saw it were the students who would later dig it up under the Freedom of Information Act. Then they'd bring it on campus and say, 'See, McCarthy is working on such and such.' McCarthy would say, 'What do you mean? I never heard of that!'"

But the book also ties together things that I learned about in college with no that they were connected. For instance, ALOHA and Ethernet use the same collision resolution mechanism not because it's an obvious fact about the universe, but because Bob Metcalfe read a paper slamming ALOHA and decided to prove it could work. And multitasking came directly from time-sharing, as cheaper computers made time-sharing less of a selling point.

Fun fact: the book made it sound like ARPA was originally planned to encompass space research, and that NASA was created as a separate agency after a bureaucratic turf war. So the space program could have been part of ARPA.

Understanding the book requires no technical knowledge, but I don't think I would have enjoyed it as much if I didn't already have all these pieces in my mind ready to be connected. The book made me more interested in another item on my wishlist, John Markoff's What the Dormouse Said.

This entry got kind of long, so I'll talk about the Turing book in another entry.

BeautifulSoup, c'est l'un des meilleurs parseurs HTML du monde mondial, c'est ce que j'utilise pour mon script de conversion blueflagz => RSS

"Monde mondial"? Does that mean "real-world"? If so, that's a great idiom.

I don't know when I'm going to be fixing Rubyful Soup to have the features of Beautiful Soup 3.0, but it's probably going to be after a bunch of WADL work.

''.join([e for e in soup.recursiveChildGenerator() if isinstance(e,unicode)])

Where soup is your soup object. I really like the if conditional in list comprehensions; I don't think Ruby has anything like it. I can think of ways to simulate it but they kill its simplicity, which is what I like about it.

The other major new feature is that you can now rip out a chunk from the parse tree with the extract method. You can use the chunk and abandon the rest of the tree, or vice versa. This is especially useful because the data structures you abandoned can now be garbage-collected: in current Beautiful Soup, the whole tree stays in memory forever because every Tag and NavigableText is connected to every other Tag and NavigableText through an intricate web of lies. And by "lies", I mean "instance variables".

There are some more new features, but I have to take a shower now to go and meet Pete Peterson II for dinner. Test it out; I'll be rewriting the documentation over the next month or so, and hopefully by then I'll have gotten enough feedback to release it.

Beautiful Soup needs an update, too, which will come probably this weekend.

Incidentally, when I saw Austin Powers I did not get the Macarena reference at all. I only got it a couple days ago after Sumana painstakingly explained every aspect of the joke. I thought Dr. Evil was walking like a toy robot, making a sound a toy robot might make while walking. I actually like this explanation of his behavior a lot better.

I started writing some documentation for Rubyful Soup. Only about halfway done, but it's a good start. I've learned a lot about technical writing in the past couple months, as you can tell by comparing the documentation files for the two Soups.

I finished my writing quota early and spent most of the rest of the day at the beach with Sumana, which was lots of fun: interesting rocks, and snowy plovers. On the way there Sumana came up with the term "Goohoo" to describe the Oxbridge-like brain sinks where about 80% of my CollabNet ex-coworkers work now.

I made deep-dish pizza in the cast-iron skillet and it was pretty good. I also made brownies with this egg/fat substitute made from prunes and they were quite acceptable, though not as good as other brownies I've made (Sumana disagrees; she says they're great). Also she says I should keep my cell phone on.

New Beautiful Soup, with minor fixes and real Python 2.2 support. Now going to see Hitchhiker's with Sumana and Zack.

Well, I tried, but no Beautiful Soup 2.0 tonight. I'm almost done with the online docs but not quite, and I still have some things on the TODO list I want to put into 2.0 as long as I'm breaking backward compatibility. Tomorrow I get my reviewed copy of the web applications/services chapter back, (contrary to my fears the tech reviewer thinks I wrote a good chapter) so if I can't somehow squeeze it in tomorrow it's probably not going to happen for over a week.