TITLE What's new in Beautiful Soup

<img align="right" src="11.2.jpg" height="250" width="183">

<h1>What's New in <a href="./">Beautiful Soup</a></h1>

<h2>Release 3.0.7a (2008/07/03)</h2>

<ul>
<li>Added an import that makes BS work in Python 2.3.
</ul>

<h2>Release 3.0.7 (2008/06/22)</h2>

<ul>
<li>Fixed a UnicodeDecodeError when unpickling documents that contain
non-ASCII characters.

<li>Fixed a TypeError that occured in some circumstances when a tag
contained no text.

<li>Jump through hoops to avoid the use of chardet, which can be extremely
slow in some circumstances. UTF-8 documents should never trigger the
use of chardet.

<li>Whitespace is preserved inside &lt;pre&gt; and &lt;textarea&gt; tags that contain
nothing but whitespace.

<li>Beautiful Soup can now parse a doctype that's scoped to an XML namespace.
</ul>

<h2>Release 3.0.6 (2008/04/26)</h2>

<ul>

<li>Added a Tag.decompose() method that completely disconnects a tree or a
subset of a tree, breaking it up into bite-sized pieces that are
easy for the garbage collecter to collect.

<li>Got rid of a very old debug line that prevented chardet from working.

<li>Tag.extract() now returns the tag that was extracted.

<li>Tag.findNext() now does something with the keyword arguments you pass
it instead of dropping them on the floor.

<li>Fixed a Unicode conversion bug.

<li>Fixed a bug that garbled some <meta> tags when rewriting them.
</ul>


<h2>Release 3.0.5 (2007/12/12)</h2>

<ul>
<li>Beautiful Soup is now licensed under a BSD-style license.

<li>Soup objects can now be pickled, and copied with copy.deepcopy.

<li>Tag.append now works properly on existing BS objects. (It wasn't
originally intended for outside use, but it can be now.) (Giles
Radford)

<li>Passing in a nonexistent encoding will no longer crash the parser on
Python 2.4 (John Nagle).

<li>Fixed an underlying bug in SGMLParser that thinks ASCII has 255
characters instead of 127 (John Nagle).

<li>Entities are converted more consistently to Unicode characters.

<li>Entity references in attribute values are now converted to Unicode
characters when appropriate. Numeric entities are always converted,
because SGMLParser always converts them outside of attribute values.

<li>ALL_ENTITIES happens to just be the XHTML entities, so I renamed it to
XHTML_ENTITIES.

<li>The regular expression for bare ampersands was too loose. In some
cases ampersands were not being escaped. (Sam Ruby?)

<li>Non-breaking spaces and other special Unicode space characters are no
longer folded to ASCII spaces. (Robert Leftwich)

<li>Information inside a TEXTAREA tag is now parsed literally, not as HTML
tags. TEXTAREA now works exactly the same way as SCRIPT. (Zephyr Fang)
</ul>


<h2>Release 3.0.4 (2007/04/10)</h2>

<ul>

<li>Fixed a bug that crashed Unicode conversion in some cases.

<li>Fixed a bug that prevented UnicodeDammit from being used as a
general-purpose data scrubber.

<li>Fixed some unit test failures when running against Python 2.5.

<li>When considering whether to convert smart quotes, UnicodeDammit now
looks at the original encoding in a case-insensitive way.

</ul>

<h2>Release 3.0.3 (2006/06/06)</h2>

<ul>

<li>Beautiful Soup is now usable as a way to clean up invalid XML/HTML
(be sure to pass in an appropriate value for
<code>convertEntities</code>, or XML/HTML entities might stick around
that aren't valid in HTML/XML). The result may not validate, but it
should be good enough to not choke a real-world XML
parser. Specifically, the output of a properly constructed soup object
should always be valid as <i>part</i> of an XML document, but parts
may be missing if they were missing in the original. As always, if the
input is valid XML, the output will also be valid.

</ul>

<h2>Release 3.0.2 (2006/06/02)</h2>

<ul>

<li>Previously, Beautiful Soup correctly handled attribute values that
contained embedded quotes (sometimes by escaping), but not other kinds
of XML character. Now, it correctly handles or escapes all special XML
characters in attribute values.

<li>I aliased methods to the 2.x names (<code>fetch</code>,
<code>find</code>, <code>findText</code>, etc.) for backwards
compatibility purposes. Those names are deprecated and if I ever do a
4.0 I will remove them. I will, I tell you!

<li>Fixed a bug where the <code>findAll</code> method wasn't passing
along any keyword arguments.

<li>When run from the command line, Beautiful Soup now acts as an HTML
pretty-printer, not an XML pretty-printer.

</ul>

<h2>Release 3.0.1 (2006/05/30)</h2>

<ul>

<li>Reintroduced the "fetch by CSS class" shortcut. I thought keyword
arguments would replace it, but they don't. You can't call
<code>soup('a', class='foo')</code> because <code>class</code> is a
Python keyword.

<li>If Beautiful Soup encounters a meta tag that declares the
encoding, but a <code>SoupStrainer</code> tells it not to parse that
tag, Beautiful Soup will no longer try to rewrite the meta tag to
mention the new encoding. Basically, this makes
<code>SoupStrainer</code>s work in real-world applications instead of
crashing the parser.

</ul>

<h2>Release 3.0.0 (2006/05/28), "Who would not give all else for two p"</h2>

<ul>

<li>This release is not backward-compatible with previous releases. If
you've got code written with a previous version of the library, go
ahead and keep using it, unless one of the features mentioned here
really makes your life easier. Since the library is self-contained,
you can include an old copy of the library in your old applications,
and use the new version for everything else.

<li>The <a href="documentation.html">documentation</a> has been
rewritten and greatly expanded with many more examples.

<li>Beautiful Soup autodetects the encoding of a document (or uses the
one you specify), and converts it from its native encoding to 
Unicode. Internally, it only deals with Unicode strings. When you
print out the document, it converts to UTF-8 (or another encoding
you specify). [<a
href="documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode,%20Dammit">Doc
reference</a>]

<li>It's now easy to make large-scale changes to the parse tree
without screwing up the navigation members. The methods are
<code>extract</code>, <code>replaceWith</code>, and
<code>insert</code>. [<a
href="documentation.html#Modifying%20the%20Parse%20Tree">Doc
reference</a>. See also <a
href="documentation.html#Improving%20Memory%20Usage%20with%20extract">Improving
Memory Usage with <code>extract</code></a>]

<li>Passing <code>True</code> in as an attribute value gives you tags
that have <i>any</i> value for that attribute. You don't have to
create a regular expression. Passing <code>None</code> for an
attribute value gives you tags that don't have that attribute at all.

<li>Tag objects now know whether or not they're self-closing. This
avoids the problem where Beautiful Soup thought that tags like &lt;BR
/&gt; were self-closing even in XML documents. You can customize the
self-closing tags for a parser object by passing them in as a list of
<code>selfClosingTags</code>: you don't have to subclass anymore. [<a
href="documentation.html#selfClosingTags">Doc reference</a>]

<li>There's a new built-in parser, <code>MinimalSoup</code>, which has
most of <code>BeautifulSoup</code>'s HTML-specific rules, but no tag
nesting rules. [<a
href="documentation.html#Other%20Built-In%20Parsers">Doc
reference</a>]

<li>You can use a <code>SoupStrainer</code> to tell Beautiful Soup to
parse only part of a document. This saves time and memory, often
making Beautiful Soup about as fast as a custom-built
<code>SGMLParser</code> subclass. [<a
href="documentation.html#Improving%20Performance%20by%20Parsing%20Only%20Part%20of%20the%20Document">Doc
reference</a>, <a href="documentation.html#Fun%20With%20SoupStrainers"><code>SoupStrainer</code> reference</a>]

<li>You can (usually) use keyword arguments instead of passing a
dictionary of attributes to a search method. That is, you can replace
<code>soup(args={"id" : "5"})</code> with
<code>soup(id="5")</code>. You can still use <code>args</code> if (for
instance) you need to find an attribute whose name clashes with the
name of an argument to <code>findAll</code>. [Doc reference: <a
href="documentation.html#arg-**kwargs">**kwargs</a> <a
href="documentation.html#arg-attrs">attrs</a>]

<li>The method names have changed to the better method names used in
Rubyful Soup. Instead of <code>find</code> methods and
<code>fetch</code> methods, there are only <code>find</code> methods.
Instead of a scheme where you can't remember which method finds one
element and which one finds them all, we have <code>find</code> and
<code>findAll</code>. In general, if the method name mentions
<code>All</code> or a plural noun (eg. <code>findNextSiblings</code>),
then it finds many elements method. Otherwise, it only finds one
element. [<a
href="documentation.html#Searching%20the%20Parse%20Tree">Doc
reference</a>]

<li>Some of the argument names have been renamed for clarity. For
instance <code>avoidParserProblems</code> is now
<code>parserMassage</code>.

<li>Beautiful Soup no longer implements a <code>feed</code>
method. You need to pass a string or a filehandle into the soup
constructor, not with <code>feed</code> after the soup has been
created. There is still a <code>feed</code> method, but it's the
<code>feed</code> method implemented by <code>SGMLParser</code> and
calling it will bypass Beautiful Soup and cause problems.

<li>The <code>NavigableText</code> class has been renamed to
<code>NavigableString</code>. There is no
<code>NavigableUnicodeString</code> anymore, because <i>every</i> string
inside a Beautiful Soup parse tree is a Unicode string.

<li><code>findText</code> and <code>fetchText</code> are gone. Just
pass a <code>text</code> argument into <code>find</code> or
<code>findAll</code>.

<li><code>Null</code> was more trouble than it was worth, so I got rid
of it. Anything that used to return <code>Null</code> now returns
<code>None</code>.

<li>Special XML constructs like comments and CDATA now have their own
<code>NavigableString</code> subclasses, instead of being treated as
oddly-formed data. If you parse a document that contains CDATA and
write it back out, the CDATA will still be there. [<a
href="documentation.html#The%20Parse%20Tree">Doc reference</a>]

<li>When you're parsing a document, you can get Beautiful Soup to
convert XML or HTML entities into the corresponding Unicode
characters. [<a href="documentation.html#Entity%20Conversion">Doc
reference</a>]

</ul>

<h2>Release 2.1.1 (2005/09/18)</h2>

<ul>

<li>Fixed a serious performance bug in <code>BeautifulStoneSoup</code>
which was causing parsing to be incredibly slow.

<li>Corrected several entities that were previously being incorrectly
translated from Microsoft smart-quote-like characters.

<li>Fixed a bug that was breaking text fetch.

<li>Fixed a bug that crashed the parser when text chunks that look
like HTML tag names showed up within a SCRIPT tag.

<li>THEAD, TBODY, and TFOOT tags are now nestable within TABLE
tags. Nested tables should parse more sensibly now.

<li>BASE is now considered a self-closing tag.

</ul>

<h2>Release 2.1.0, "Game, or any other dish?" (2005/05/04)</h2>

<ul>

<li>Added a wide variety of new search methods which, given a starting
point inside the tree, follow a particular navigation member (like
<code>nextSibling</code>) over and over again, looking for
<code>Tag</code> and <code>NavigableText</code> objects that match
certain criteria. The new methods are <code>findNext</code>,
<code>fetchNext</code>, <code>findPrevious</code>,
<code>fetchPrevious</code>, <code>findNextSibling</code>,
<code>fetchNextSiblings</code>, <code>findPreviousSibling</code>,
<code>fetchPreviousSiblings</code>, <code>findParent</code>, and
<code>fetchParents</code>. All of these use the same basic code used
by <code>first</code> and <code>fetch</code>, so you can pass your
weird ways of matching things into these methods.

<li>The <code>fetch</code> method and its derivatives now accept a
<code>limit</code> argument.

<li>You can now pass keyword arguments when calling a <code>Tag</code>
object as though it were a method.

<li>Fixed a bug that caused all hand-created tags to share a single
set of attributes.

</ul>

<h2>Release 2.0.3 (2005/05/01)</h2>

<ul>

<li>Fixed Python 2.2 support for iterators.
<li>Fixed a bug that gave the wrong representation to tags within quote tags like &lt;script&gt;.
<li>Took some code from Mark Pilgrim that treats CDATA declarations as data
instead of ignoring them.
<li>Beautiful Soup's setup.py will now do an install even if the unit tests fail. It won't build a source distribution if the unit tests fail, so I can't release a new version unless they pass.
</ul>

<h2>Release 2.0.2 (2005/04/16)</h2>

<ul>

<li>Added the unit tests in a separate module, and packaged it with
distutils.

<li>Fixed a bug that sometimes caused renderContents() to return a
Unicode string even if there was no Unicode in the original string.

<li>Added the <code>done()</code> method, which closes all of the
parser's open tags. It gets called automatically when you pass in some
text to the constructor of a parser class; otherwise you must call it
yourself.

<li>Reinstated some backwards compatibility with 1.x versions:
referencing the <code>string</code> member of a NavigableText object
returns the NavigableText object instead of throwing an error.

</ul>

<h2>Release 2.0.1 (2005/04/12)</h2>

<ul> 

<li>Fixed a bug that caused bad results when you tried to reference a
tag name shorter than 3 characters as a member of a Tag,
eg. <code>tag.table.td</code>.

<li>Made sure all Tags have the 'hidden' attribute so that an attempt to access
 <code>tag.hidden</code> doesn't spawn an attempt to find a tag named
'hidden'.

<li>Fixed a bug in the comparison operator.

</ul>

<h2>Release 2.0, "Who cares for fish?" (2005/04/10)</h2>

<p>Beautiful Soup version 1 was very useful but also pretty stupid. I
originally wrote it without noticing any of the problems inherent in
trying to build a parse tree out of ambiguous HTML tags. This version
solves all of those problems to my satisfaction. It also adds many new
clever things to make up for the removal of the stupid things.

<h3>Parsing</h3>
 <ul>

 <li>The parser logic has been greatly improved, and the BeautifulSoup
  class should much more reliably yield a parse tree that looks like
  what the page author intended. For a particular class of odd edge
  cases that now causes problems, there is a new class,
  ICantBelieveItsBeautifulSoup.</li>

 <li>By default, Beautiful Soup now performs some cleanup operations
  on text before parsing it. This is to avoid common problems with bad
  definitions and self-closing tags that crash SGMLParser. You can
  provide your own set of cleanup operations, or turn it off
  altogether. The cleanup operations include fixing self-closing tags
  that don't close, and replacing Microsoft smart quotes and similar
  characters with their HTML entity equivalents.

 <li>You can now get a pretty-print version of parsed HTML to get a
  visual picture of how Beautiful Soup parses it, with the
  <code>Tag.prettify()</code> method.  

</ul>

<h3>Strings and Unicode</h3>
 <ul>

 <li>There are separate NavigableText subclasses for ASCII and Unicode
  strings. These classes directly subclass the corresponding base data
  types. This means you can treat NavigableText objects as strings
  instead of having to call methods on them to get the strings.

 <li><code>str()</code> on a Tag always returns a string, and
  <code>unicode()</code> always returns Unicode. Previously it was
  inconsistent.

 </ul>

<h3>Tree traversal</h3>

<ul>

<li>In a <code>first()</code> or <code>fetch()</code> call, the tag
  name or the desired value of an attribute can now be any of the
  following:

<ul>
<li>A string (matches that specific tag or that specific attribute value)

<li>A list of strings (matches any tag or attribute value in the list)

<li>A compiled regular expression object (matches any tag or attribute
    value that matches the regular expression)

<li>A callable object that takes the Tag object or attribute value as a
    string. It returns None/false/empty string if the given string
    doesn't match, and any other value if it does.
</ul>

  <p>This is much easier to use than SQL-style wildcards (see, regular
  expressions are good for something). Because of this, I took out
  SQL-style wildcards. I'll put them back if someone complains, but
  their removal simplifies the code a lot.

<li>You can use <code>fetch()</code> and <code>first()</code> to
search for text in the parse tree, not just tags. There are new alias
methods fetchText() and firstText() designed for this purpose. As with
searching for tags, you can pass in a string, a regular expression
object, or a method to match your text.

<li>If you pass in something besides a map to the <code>attrs</code>
argument of <code>fetch()</code> or <code>first()</code>, Beautiful
Soup will assume you want to match that thing against the "class"
attribute. When you're scraping well-structured HTML, this makes your
code a lot cleaner.

<li>1.x and 2.x both let you call a Tag object as a shorthand for
<code>fetch()</code>. For instance, foo("bar") is a shorthand for
foo.fetch("bar"). In 2.x, you can also access a specially-named member
of a Tag object as a shorthand for <code>first()</code>. For instance,
<code>foo.barTag</code> is a shorthand for
<code>foo.first("bar")</code>. By chaining these shortcuts you
traverse a tree in very little code: <code>for header in
soup.bodyTag.pTag.tableTag('th'):</code>

<li>If an element relationship (like parent or next) doesn't apply to
a tag, it'll now show up Null instead of None. <code>first()</code>
will also return Null if you ask it for a nonexistent tag. Null is an
object that's just like None, except you can do whatever you want to
it and it'll give you Null instead of throwing an error. 

<p>This lets you do tree traversals like
<code>soup.htmlTag.headTag.titleTag</code> without having to worry if
the intermediate stages are actually there. Previously, if there was
no 'head' tag in the document, headTag in that instance would have
been None, and accessing its 'titleTag' member would have thrown an
AttributeError. Now, you can get what you want when it exists, and get
Null when it doesn't, without having to do a lot of conditionals
checking to see if every stage is None.</p></li>

<li>There are two new relations between page elements: previousSibling
and nextSibling. They reference the previous and next element at the
same level of the parse tree. For instance, if you have HTML like
this:

 <blockquote>&lt;p&gt;&lt;ul&gt;&lt;li&gt;Foo&lt;br /&gt;&lt;li&gt;Bar&lt;/ul&gt;</blockquote>

The first 'li' tag has a previousSibling of Null and its nextSibling
is the second 'li' tag. The second 'li' tag has a nextSibling of Null
and its previousSibling is the first 'li' tag. The previousSibling of
the 'ul' tag is the first 'p' tag. The nextSibling of 'Foo' is the
'br' tag. 

<li>I took out the ability to use fetch() to find tags that have a
specific list of contents. See, I can't even explain it well. It was
really difficult to use, I never used it, and I don't think anyone
else ever used it. To the extent anyone did, they can probably use
fetchText() instead. If it turns out someone needs it I'll think of
another solution.

</ul>

<h3>Tree manipulation</h3>

<ul>
<li>You can add new attributes to a tag, and delete attributes from a
tag. In 1.x you could only change a tag's existing attributes.
</ul>

<h3>Porting Considerations</h3>

<p>There are three changes in 2.0 that break old code:

<ul>

<li>In the post-1.2 release you could pass in a function into
fetch(). The function took a string, the tag name. In 2.0, the
function takes the actual Tag object.

<li>It's no longer to pass in SQL-style wildcards to fetch(). Use a
regular expression instead.

<li>The different parsing algorithm means the parse tree may not be
shaped like you expect. This will only actually affect you if your
code uses one of the affected parts. I haven't run into this problem
yet while porting my code.

</ul>

<h2>Between 1.2 and 2.0</h2>

<p>This is the release to get if you want Python 1.5 compatibility.

<ul>
<li> The desired value of an attribute can now be any of the following:
  <ul>
  <li> A string
  <li> A string with SQL-style wildcards
  <li> A compiled RE object
  <li> A callable that returns None/false/empty string if the given value
    doesn't match, and any other value otherwise.
  </ul>
  <p>This is much easier to use than SQL-style wildcards (see, regular
  expressions are good for something). Because of this, I no longer
  recommend you use SQL-style wildcards. They may go away in a future
  release to clean up the code.

<li> Made Beautiful Soup handle processing instructions as text instead of
 ignoring them.

<li> Applied patch from Richie Hindle (richie at entrian dot com) that
 makes tag.string a shorthand for tag.contents[0].string when the tag
 has only one string-owning child.

<li> Added still more nestable tags. The nestable tags thing won't work in
 a lot of cases and needs to be rethought.

<li> Fixed an edge case where searching for "%foo" would match any string
 shorter than "foo".
</ul>

<h2>Release 1.2, "Who for such dainties would not stoop?" (2004/07/08)</h2>

<ul>

<li>Applied
    patch from Ben Last (ben at benlast dot com) that made
    Tag.renderContents() correctly handle Unicode.

<li>Made BeautifulStoneSoup even dumber by making it not implicitly
    close a tag when another tag of the same type is encountered; only
    when an actual closing tag is encountered. This change courtesy of
    Fuzzy (mike at pcblokes dot com). BeautifulSoup still works as
    before.
</ul>

<h2>Release 1.1, "Swimming in a hot tureen"</h2>

<ul>
<li> Added more 'nestable' tags. Changed popping semantics so that
    when a nestable tag is encountered, tags are popped up to the
    previously encountered nestable tag (of whatever kind).  I will
    revert this if enough people complain, but it should make more
    people's lives easier than harder. This enhancement was suggested
    by Anthony Baxter (anthony at interlink dot com dot au).

</ul>

<h2>Release 1.0, "So rich and green" (2005/04/20)</h2>

<p>Initial release.</p>