The Magic Arrow

Hi, I'm Leonard Richardson. I work at the New York Public Library. Here I am [on the right] with some of our team, outside the Mid-Manhattan branch on the day we launched our product. I wore the same clothes today so you'd recognize me.

In 2015 I gave a talk here at RESTFest that previewed the product we were working on: a mobile application for borrowing and reading ebooks.

I mainly talked about OPDS, a hypermedia format for talking about catalogs of books and things you can do to books, like borrowing them from a library.

The title of this year's talk, "The Magic Arrow", comes from an interesting property of OPDS that I hinted at last year. When I draw architectural diagrams for the Library Simplified project, I'm able to use OPDS to connect all sorts of different components to each other, whether it's a client-server relationship with a human on one end, or a machine-to-machine integration with no human in the loop. OPDS is a magic arrow that can connect all sorts of components.

Here's a diagram from Roy Fielding's thesis which shows that HTTP is also a magic arrow. The web browser speaks HTTP to the proxy, which speaks HTTP to the gateway, which speaks HTTP to the origin server. Every arrow in this diagram is labelled "HTTP", except for the connection between an HTTP proxy and a WAIS server, which has to be WAIS.

Over the past year my team has been really busy. We expanded from a team of two people to a team of seven. We released the product you saw in that group photo, SimplyE, which is an OPDS client for NYPL cardholders that gives you integrated access to our ebook collection.

We've created a basic OPDS client that runs inside a web browser.

And we worked on a project with the White House called Open Ebooks. Most libraries serve a limited geographic area, but Open Ebooks is a national collection for children. If you attend a Title I school or you're eligible for free or reduced lunch, you can use the Open Ebooks app to get free access to about four thousand books.

Just after we launched Open Ebooks, the President went to SXSW to talk about tech policy, so we were really excited, waiting for our shout-out.

[plays video from 31:52 to 32:18]

THE PRESIDENT: Private industry has really stepped up. And so part of the task -- you're right that we've got to make sure that, given the power of this space, everybody is plugged in. But one of the great tricks to all this is making sure that whatever government is doing is then supplemented with and enhanced by a private sector and nonprofit sector that are ready to step up. And it's not just, by the way, getting a line in or Wi-Fi there. It's also training teachers. We've set up something called -- well, open book? Somebody out here --

AUDIENCE MEMBER: Open eBooks.

THE PRESIDENT: There you go, Open eBooks. (Laughter.) I knew there was somebody in the audience who'd know about this. (Laughter.) To make sure that kids in places that don't have a lot of books that suddenly they have access to this enormous e-library, and that that becomes folded into the mechanics and the infrastructure that's been set in place.

Thanks, Obama.

Moving on to our current projects. We have a partnership with the Connecticut State Library to bring our ebook service to every library in Connecticut.

I can't name any more states, but there are 14 libraries interested in the Library Simplified project. These libraries cover a big chunk of the population of the United States.

I'm proud of this, but I'm not bragging that we have a lot of customers. What I want to brag to you about is that we have built an ecosystem.

The Internet Archive has a library where you can borrow books they scanned. You can read the books in your browser but they don't have a mobile app. We're helping them publish OPDS feeds and implement the right state transitions, so they can lend books out to anyone with a compliant OPDS reader.

unglue.it and Standard Ebooks provide books that are public domain or licensed under Creative Commons. We helped them publish OPDS feeds of their books, we integrated those feeds into our system, and now the books published on those sites automatically make it into NYPL's collection. We expand our collection with free books without paying anything.

Online bookstores in Norway and Sweden are interested in building their own mobile apps based on Library Simplified code. So we're also helping drive the use of OPDS in a commercial context.

Now for the rest of the talk I'm going to tell about something interesting I discovered while I was working on the Connecticut project. First I want to give a little background about how libraries loan out books.

Here's the part that faces the public: the library lets you borrow books for free. You're supposed to bring the books back. If you lose the books, or you're late bringing them back, we fine you. If you owe a lot of money in fines, you might lose your borrowing privileges until you pay the fines. If you steal DVDs, or attack a librarian, you can lose your borrowing privileges permanently.

Now, internally, a library system has an inventory tracking system called an ILS, Integrated Library System. Loans happen in one particular branch but they're recorded in the master record, in the ILS. If you owe fines, or you're not allowed to borrow books for some other reason, that fact is stored in the ILS. This way you can't evade your fines just by going to a different branch.

How do the branches communicate with the ILS? What is the arrow in this architecture diagram?

Generally they use a protocol called SIP.

This is not Session Initiation Protocol, the Internet telephony protocol. I saw a diagram early on and someone said "this is a SIP connection" and I thought "damn, that's overkill".

I'm talking about a different SIP, Standard Interchange Protocol. SIP was designed by 3M in 1993 to sell these self-service checkout machines to libraries.

It's a plain text client-server protocol. It has a message-response pattern, like HTTP.

A request or response starts with a numeric code. You would send an "11" request to check out a book, and you would expect to get back a "12" response from the server.

Then you have a number of fixed width fields, then a number of variable width fields.

Here's what a request to check out a book looks like:

11[SC renewal policy][no block][transaction date][due date][institution id][patron identifier][item identifier][terminal password][patron password][item properties][fee acknowledged][cancel]

Just as an example, 'no block' here means, are you asking the ILS to check out a book for this patron, or did you already loan out the book and you're telling the ILS that this patron has it now?

Every other variable in SIP has a similarly interesting story.

A SIP response has the same format. Numeric code, fixed width fields, variable width fields. Here's the response you get after sending a checkout request:

12[ok][renewal ok][magnetic media][desensitize][transaction date]<[institution id][patron identifier][item identifier][title identifier][due date][fee type][security inhibit][currency type][fee amount][media type][item properties][transaction id][screen message][print line]

Here's how NYPL uses SIP. I'm going to talk about a print book because ebooks don't actually go through this system, for reasons I'll mention briefly later.

Let's say a patron borrows a book at Jerome Park branch in the Bronx, either by using a self-service machine or by talking to a librarian, who uses a similar machine. The checkout is actually performed by this computer sending a SIP 11 message, i.e. a Checkout message, over the Internet to the ILS.

The ILS sends back a SIP 12 message, which is a "Checkout response" message, and the self-service machine desensitizes the book's RFID so the patron can take it past the security gate.

The patron keeps the book for a couple of weeks. During these weeks the book has no interactions with our system whatsoever. When they're done, they drop the book in a return box at the Todt Hill branch in Staten Island.

Some libraries have an RFID reader in the book slot, and in those libraries there's a SIP message 09 ("Checkin") sent to the ILS as soon as you drop the book in the slot, but we don't do that. Once a empty the return box into a truck and drive all the books to Long Island City, in Queens. We put all the books on a conveyor belt and scan them. At this point we send off the SIP 09 message indicating that the book has been checked in.

Once the SIP 09 message is received, the patron is off the hook! If we lose the book it's our fault, not theirs. Also, we destroy the record of their loan, to protect patron privacy.

Now you might think this book is going back to its home in the Bronx, but in Long Island City we make a SIP 10 ("Item Information") request and we find out that that's not true. Another patron has placed a hold on the book, and they want to pick it up at the Harlem branch in Manhattan.

So we put the book back in a truck and deliver it to Harlem, where it's scanned. That sends a SIP 19 ("Item Status Update") message to the ILS. The book is put on the reserve shelf and it's ready for the next patron to pick up, and because of that SIP 19 message, its status in the ILS reflects this.

There's a reason I'm giving this talk at RESTfest, but it's not because SIP is a RESTful protocol, on any level. It's not based on HTTP, it's a raw socket protocol. It doesn't use URIs. Not only is SIP not RESTful, it's also bad. It's a bad design. If I had to give a one-sentence critique I'd say SIP is stuck in 1993: technically, culturally, and in terms of its model of how libraries work.

This is not a catastrophic problem because libraries haven't actually changed that much since 1993. Obviously some things have changed. Libraries now loan out ebooks. You can now go to the library and use the Internet on a library-provided computer. But these new functions of libraries are handled by other systems. They aren't run through SIP and they tend to bypass the ILS, because ILSes are also stuck in 1993.

And for my work giving ebook service to every library in Connecticut, it doesn't really matter that SIP is bad. SIP isn't "RESTful" in the typical sense that term is used in 2016, but it has two properties of REST that make it really good at what I need it to do.

But before I get to that, here are some funny crowd-pleasing examples of how SIP is stuck in 1993.

We all know how MIME defines media types. Here's how SIP defines media types, and brother, I mean media types. These are the types of media you can check out from a library. There are eleven media types, and each has a three-digit code.

000 - other
001 - book
002 - magazine
003 - bound journal
004 - audio tape
005 - video tape
006 - CD/CDROM
007 - diskette
008 - book with diskette
009 - book with CD
010 - book with audio tape

This is a cutting-edge list for 1993, but it's missing some obvious things, like DVDs, and ebooks.

Earlier my slide said SIP suffers from cultural insensitivity, and here's an example. The default character set for a SIP is Code Page 850, the character set used by American versions of MS-DOS. I love this character set dearly, but if you don't spell your name with these characters, your name won't fit into the library's system. [Note: this is slightly wrong. The character set used by American versions of MS-DOS is Code Page 437. Code Page 850 is the one they used in western Europe. So it's still very Western, but not as America-centric as I thought.]

UTF-8 was announced in 1993, the same year as SIP, and I'll cut 3M some slack for not being on the cutting edge. And it is just a default, a library can change it.

But I can't make any excuses for this one. This is an enumerated list of languages found in the SIP spec. The spec says that patrons can request messages be localized into these twenty-six languages.

Okay, localization is good, but there are a lot of problems in here. First, there are more than twenty-six languages in existence. Hindi is the second most common language in the world, but it's not on this list.

Second, "United Kingdom" isn't a language, it's a locale. "Belgian" isn't a language or a locale, it's a nationality. "North American Spanish" is what they speak in Mexico, but why does this list have that and "Spanish"? Is that European Spanish or South American Spanish? And what does it mean that "Chinese" and "Taiwanese" are separate? I feel like this list is going to cause an international incident.

I have no idea where this list came from. ISO-639, which has the two-letter codes used by the Accept-Language header, became a standard in 1967, and libraries know a thing or two about languages, so you can't plead ignorance.

The only thing I can think of is this list was taken directly from 3M's sales department. This list corresponds to the places they wanted to sell their self-checkout machines.

Fortunately, there's a way out of this mess. NYPL uses an ILS called Sierra, and Sierra just released their brand new REST API.

You've probably seen this sort of document before. It talks about the problem domain: branches, patrons, books. It says you can manipulate the problem domain by getting JSON documents and sending HTTP requests.

It explains how you should construct URLs to get certain things to happen.

This is a perfectly adequate level 2 API.

Except...

This is the Sierra REST API. It's designed by Sierra. Their name is all over it. There is no implication that other ILSes can or should use the same API.

Which is a problem because different libraries use different ILSes. I have to get Library Simplified to work with every ILS in Connecticut, and every ILS in the other thirteen states, and potentially every ILS in the world.

At this point it's a good thing for me that ebook loans bypass the ILS, because it means my interaction with the ILS is limited. I only need to ask each ILS two questions.

* Authentication: Who is this patron? If they provided a password, is their password valid?

* Authorization: Is this patron allowed to borrow books? Or do they have too many fines, or were they banned for stealing DVDs?

Now, I've been doing surveys of the ILSes used by Connecticut libraries and to a first approximation every ILS defines its own custom API.

But they also all support SIP.

They all implement this stupid protocol created by 3M in 1993 to sell self-check machines.

SIP is a magic arrow, just like HTTP and OPDS. SIP is used to connect components from different vendors. SIP has what I need, something that all these new "REST" APIs don't have.

How did this happen?

Well, first, SIP has the property that the Fielding thesis calls extensibility. SIP doesn't know about DVDs, but it gives each media type a three-digit code, and it only uses up eleven codes talking about books and floppy disks. If you want to put DVDs in your library, you extend the protocol. You pick a three-digit number and assign it to DVDs.

Same if you want to use a different language from the 26 that are mentioned in the spec. Same if you want to hook up a whole other system, like an ebook lending system, like the one we're writing.

Now, since 3M is a big company that has defined a protocol for the sole purpose of selling their stuff, you might think they take a dim view of someone going in and modifying their protocol for the purpose of connecting to their competitors' stuff. And there is a paragraph in the spec that says...

Protocol Extensions

If you extend the protocol for the purpose of communicating to a device other than a 3M SelfCheck system, you are requested to send documentation of your extensions to 3M. In an effort to make this protocol a general-purpose standard, 3M will try to incorporate into the 3M Standard Interchange Protocol those extensions that are deemed to be general-purpose, and will also try to avoid re-using Message IDs and Field IDs that are in use in your extensions.

This is probably the nicest thing I've ever read in a specification, and it's the most important paragraph in the SIP spec. It says two things:

1. We will not sue you if if you use SIP to connect components of your library that you didn't buy from us.

This is the architectural property that Fielding calls reusability (section 2.3.4.5). This is the property that almost all of today's "REST" APIs are missing.

2. In fact, if you tell us what you did, we'll try to make sure we don't break your setup in a future release. And if we think what you did is generally useful, we will add it to this spec so everyone can use it.

SIP became a magic arrow because it's extensible and reusable. These properties emerge from the spec, but not just from the technical part of the spec. They come from this paragraph that reads more like a software license. These paragraph that explains the relationship between you and the Minnesota Mining And Manufacturing Company.

Because SIP is extensible, the worst problems can be fixed. You can make SIP work for a library where the patrons speak Hindi, or borrow DVDs.

Because SIP is reusable, you can use SIP to connect one piece of library hardware to another piece of library hardware, even if none of that hardware is provided by 3M. Reusability means that when the protocol falls short, you are legally allowed to improve it.

SIP is not an elegant protocol, but it's really good at being a protocol, because the purpose of a protocol is to connect things together. The Sierra REST API is a much better design, but it's only designed to connect your stuff to the vendor's stuff.

I think a lot of the cargo culty stuff around REST happens because we try to make REST work without these two architectural properties: extensibility and reusability. I see a lot of these architectural diagrams, and usually the API drawn as a box that communicates with another box—the client—via an arrow. The arrow is frequently not labelled, but here I've labelled it "WWW" because the over-the-wire protocol is some combination of the Web technologies: HTTP, URIs, and hypermedia.

This diagram implies that the API is a unique thing that lives on one computer. Most of the time, that's accurate! But that's not a good thing! It means we've given up the benefits of extensibility and reusability. And we don't notice the problem, because when we talk about this setup we can't stop talking about the arrow, about this API's use of the Web technologies, about whether its URLs look right or whether it uses the right link relations.

An API like SIP doesn't use any of the Web technologies. But it has the two features of the Web that are most important when you're orchestrating library tech: extensibility and reusability. Because of reusability, you can't draw SIP as a box. It's got to be an arrow.

Once you draw the API as an arrow, it becomes clear what role the box actually plays in the architecture. It's not an API, it's a server that speaks a certain protocol. It's an architectural component that talks to other components.

And it becomes conceivable that you could have other components that speak this protocol to each other, not just to the central server.

Users are hungry for extensibility and reusability. For those two architectural features they may be willing to give up quite a lot of what we call "REST" in 2016. Possibly all of it.

The good news is that you probably don't have to redesign your spec to get these features, because they pertain to the relationship between component authors. You can probably get these features just by changing your attitude and putting a notice in your spec. Instead of creating an API for your product, create an API for your product category, or your industry.