The Enterprise Media Distribution Platform At The End Of This Book
I gave this talk at RESTFest in September 2015. The opinions expressed in this talk are mine and not those of the New York Public Library or the Library Simplified consortium. I didn't even use the PowerPoint template that has the logo on every page.
This is a cleaned-up version of the talk as prepared for delivery. The original slides are available here.
I also gave two five-minute talks at RESTFest; you can see those slides here.
Hi. I'm Leonard Richardson, co-author with Mike [Amundsen] of RESTful Web APIs, and this is the story of The Enterprise Media Distribution Platform At The End Of This Book.
I love books so much that I work for the New York Public Library. For purposes of this talk you can think of NYPL as a small tech company which has some odd budgeting rules and which happens to be 120 years old.
That's pretty old, it's older than IBM, but most American public libraries are about 100 years old. NYPL is unusual among libraries in that it employs enough developers to act like a tech company.
A public library does more things than you might think, and NYPL is also a research library so we do even more things than that, but my work focuses on what public libraries have done since the beginning. We buy books and let people read them for free..
This includes e-books. The problem I was hired to solve is, we are spending a lot of money on e-books that nobody reads.
Why is this? Patrons go into our branches and check out print editions of the same books. But a lot of our patrons would rather buy an electronic edition of this book from Amazon than read it for free from us.
Well, where do New Yorkers read? They read on the subway. Mostly it's print newspapers and magazines, but among books it's about fifty-fifty between print books and e-books. The audience is there. So there's probably something wrong with our mobile strategy.
And in fact, yes, our mobile strategy is terrible! Checking out ebooks from NYPL can take thirteen steps per book and require that you install three different apps on your phone.
And these apps have an awful user experience. They don't have features you expect from commercial apps. They're designed like bookstore apps, even though bookstores and libraries have totally different inventory strategies. And we can't support these apps properly—93% of our support calls we just can't close.
Our patrons are frequently older, they have young children, they are young children, they're busy, they're not tech savvy, and this is what we give them. We ask them to come to a class at a branch library to teach them how to check out books onto their iPads.
It is unacceptable.
If NYPL was a normal software company we would have written an e-reading app. But ebooks come out of the operating budget, which is what we use to keep the branches open, and you can't do software development with the operating budget. However you can use the operating budget to write a check to a third-party vendor.
And lo and behold, there are all these third-party vendors who want us to make multi-million dollar investments in their library ebook platforms.
Well, we hedged our bets. We did deals with two of the three main library ebook distributors, each of which has their own e-reading app, and we did a deal with a catalog vendor to integrate the other vendors' catalogs into a single online catalog, and we use their app to tell patrons which e-reading app they should use for a given book. That's how we get to three apps on your phone.
Wherever you live in the US, if you look into the situation with your local public library it's probably got a smaller version of this same problem.
But we're fixing the problem! The first thing we did was we got together with other public libraries and formed a pressure group. We issued a requirements document, in the form of a manifesto, that said if you want to license ebooks to public libraries you need to have a documented API.
And it worked! All these vendors did publish some kind of API.
And with the APIs in place, we were able to get a federal grant for some software development to solve this problem for NYPL, nine other partner libraries, and potentially every library in the country.
The project is called Library Simplified and we've developed our own e-reading app, as well as some middleware that sits between that app and those vendor APIs.
Here's a screenshot of the app. It looks like an app for downloading books and reading them. I assure you, that's what it is. You find a book, you scan your library card, you download the book, you read the book. The end.
Now, I'm not a mobile developer. I do server stuff. So why bring me into it? Why not just have the mobile app talk directly to the vendor APIs?
There are a number of reasons but the one that's important to this talk is that although we were able to pressure the vendors into each developing an API, they each designed a completely different API.
There is no standard way of classifying books, or checking out a book, or seeing how many licenses the library has for a book, or a million other things.
Okay, so I'm brought in to take care of this stuff. And I write the middleware that talks to the vendors, and I write the API that talks to the Library Simplified app.
So now you're probably wondering, what does this API look like? Why do I think I can do a better job than the commercial vendors?
I didn't try to do better than the commercial vendors. I found an existing spec that was first released in 2010, called OPDS. That stands for Open Publication Distribution system. It's based on Atom, a technology that was invented in 2003.
An Atom feed is an XML document that describes a list of publications. In an OPDS feed, those publications are books that someone might buy from a bookstore, or borrow from a library, or download for free.
All these transactions are handled with link relations. We also have link relations for things like book covers.
Here's the state diagram. The patron starts off at the home page, and they either go into a search to find a specific book, or they follow links like 'Nonfiction' and then 'Cooking' until they get to a browsable shelf of books. Once the patron finds a book they like, they trigger the 'borrow' link relation, and I either send them an unencryped EPUB file, or I send them an file in an adobe format, which their mobile client can use to fulfill a DRM-encrypted book.
However it works, you start at the front page, you follow some
links, and eventually you get a book. At that point you leave the API
part of the app and go into the reader part.
OPDS had some library-specific features, like a link relation for carrying out a 'borrow' transaction, but OPDS was designed by Europeans so it was based on a European library model. So I joined the OPDS team on behalf of the Library Simplified consortium, to provide the American perspective.
And after a lot of back-and-forth we fleshed out the library use case for OPDS and gave it much better support for American library use cases like placing a hold on a book.
Here's the process. I would start by putting some extensions into my generated OPDS, I'd test it out with my client devs, and then I'd go hash it out with the Europeans. We'd refine it and eventually something which usually looked a lot better would go into the spec.
I don't want it to spend a lot of time on this process but although overall it works very well, I don't want it to sound like it's a perfect solution.
In particular, client devs get cranky because sometimes I change the server to generate legal OPDS that the client software can't handle, because they'd written to the server, not the spec. And sometimes I make the opposite mistake. The client makes a legal request I can't handle.
The OPDS spec is not as general as HTML, but from a mobile developer perspective it's uncomfortably vague.
Now picture us with a working OPDS client implementation for iOS and Android, because this is where things start getting interesting.
I'm going to start you off slow. Remember that there are three main vendors in the library ebook space. We did deals with two of them. Now that we've got the middleware in place, we can do a deal with the third vendor. We can license books from a third source without having to tell our patrons to install app #4 on their phones.
Okay, that's nothing to do with OPDS. Any kind of middleware would allow that sort of integration.
But then we decide we also want to offer Project Gutenberg books to our patrons. Unfortunately Project Gutenberg does not have an API. They have this ugly system where you have to use rsync to mirror the ebooks and then pull the metadata from a big RDF document.
So I write a simple content server, which rsyncs the ebooks and pulls the metadata and then offers a collection that is the equal, in quantity if not in quality, of the commercial collections. But instead of making up a custom API for my collection to talk to the middleware, the way the commercial vendors did, I use the API I already have—OPDS.
So now I'm using OPDS for machine-to-machine integration, not just to talk to the patrons. I can use this protocol whenever I am talking about books or collections of books.
Now other sources of free ebooks want to get in on the action. unglue.it is mostly an aggregator for Creative Commons books and other open-access books that aren't a hundred years old. Standard Ebooks is a little org that makes really nice editions of public domain ebooks, because Project Gutenberg ebooks have really horrible formatting.
So I told those people: you generate OPDS feeds, and I'll slurp them up into my content server, they'll show up in our collection and patrons will be able to download them. And that's what they did. I haven't set up my part of it yet, the part that slurps, because I haven't had time, but it's going to work.
At that point the OPDS protocol is doing machine-to-machine integration across organizational boundaries. It's hypermedia API heaven!
We had to do a mockup for a project we're doing with the White House Office of Science and Technology Policy, to give free ebooks to kids. We had like two days to come up with a specialized mockup. No problem. An OPDS 'API' is just a bunch of documents that link to each other.
I generated some static OPDS documents, we uploaded them to a server, and we pointed the Library Simplified app at the URL. There was no need to spin up a middleware instance. The client will function as long as it gets OPDS documents that link to EPUBs.
I had to do a lot of work on metadata remediation, because the metadata provided by the third-party vendors was incomplete, not standardized, and not suitable for a mobile environment. Like, they give us full-scale book covers, which is great, but those things are a megabyte each and as you saw from the screenshot we're showing twelve covers at once on someone's phone.
So I wrote a piece of middleware called the metadata wrangler, which takes metadata from every source I have access to, using everything from JSON-LD to SOAP. It tries to figure out the best set of metadata for a book, and give it an appropriately scaled cover.
Of course, when the metadata wrangler is discussing books with the rest of the middleware, it uses OPDS as the medium of communication.
And on and on. Maybe the Chattanooga public library wants to organize its collection differently from NYPL. They don't need a custom client, they just need to generate a different set of OPDS feeds.
We got a test suite we had to pass to get Adobe certification, but the test suite was just a bunch of epubs in a zip file. I uploaded those EPUBs to S3 along with an OPDS file that linked to all of them. We could test our compliance just by pointing the client app to that OPDS feed and tapping on all the books to open them up and seeing if they rendered.
And of course we should be able to swap out the client as well. Aldiko is a commercial client produced by one of the European OPDS contributors. It'll present the data differently, but everything should still work.
And this should not be surprising because it's all normal hypermedia stuff. When the client is a web browser we expect this kind of behavior.
Even when the client is not a web browser, we know that a hypermedia-driven approach makes this sort of modularity possible.
So how come, after years of designing hypermedia-based APIs for various organizations, do I feel like this is the first time everything went the way it was supposed to?
Libraries are very conservative technically. We love technology, but we computerized long before the rest of society, and we're still using systems that architecturally belong in the 1970s. I heard one of our core components was just upgraded to a version that features a relational database. Not because we moved to Mongo and moved back, but because now's the time for relational databases.
So how come a public library is the only place where I've said “I think we should use hypermedia” and the response was not “I'm not so sure” or “you're the architect, do whatever you want”, but “we understand this plan and it sounds great.”?
Well, here's something else that happened. We started hearing from libraries that were not part of the Library Simplified consortium. Like, one in Colorado and one in the Netherlands. They did not want my middleware. If there's one thing libraries already have, it's middleware.
Turns out our client app is solid gold! It's just a simple, no-drama app for checking out books, but that's a pretty rare find in the library world.
Well, good news! You can connect your middleware to the Library Simplified client app.You don't need a client dev. You just need to make your middleware serve OPDS feeds and obey the published state transitions for things like checking out a book.
And we heard from our neighbors the Queens Library, which, long story, is a different library system from NYPL. They have their own app, which they built, and their own catalog, which they built, so they do not want Simplified's app or Simplified's middleware.
But they do want the metadata wrangler, the thing that goes all over the Internet and tries to find the best metadata and the best cover for a given book. Sure, no problem. You just have to change your catalog to understand the OPDS that the metadata wrangler serves.
In commercial terms, Library Simplified has four products. The client apps, the circulation manager that handles the loans, the metadata wrangler that finds the best metadata, and the content server that manages Project Gutenberg and the other open-access content.
In a typical commercial environment we would sell this as an integrated platform. How do I know this? Well, that's the pitch from the vendors we work with. We had to buy the whole package, even though we only wanted the books.
But the real value of Library Simplified is not in the boxes in my diagrams. It's in the arrow that connects the boxes, and the fact that every single time the arrow has the same label. It turns out that loveable furry old OPDS is the Enterprise Media Distribution System At The End Of This Book.
And because NYPL and Library Simplified are not for-profit enterprises, when someone says “I want to use your app but not your middleware”, I am allowed to say “that's fine.” There is no pressure on me to say “well, it's all part of an integrated full-service platform solution” blah blah.
I am allowed to hear what they are really saying: “I want to connect my thing to something else, and you have the something else.”
This is why 'normal hypermedia stuff' is so rare. REST's uniform interface constraint, the principle that all these parts are interchangeable, is antithetical to the way tech companies do business today.
Why should I make a server that can serve my competitors' clients? That just helps my competitor! Why should I make a client that drives people away from my property and sends their previous personal data to someone else? The idea is absurd.
How many instances can you think of where two unrelated companies provide compatible APIs? It's not common. Sometimes it happens and the one company sues the other.
Twitter knows this. Their official client app is mediocre. Other people keep coming up with better clients. Organizations like Politwoops use Twitter in ways that interfere with Twitter's business model.
So sometimes Twitter kills clients, either with Terms of Service changes, or by just revoking the client keys. Sometimes the features from those dead clients make it into the official Twitter client, sometimes not.
What if I told Twitter that I wanted to license the official Twitter client for my own social microblogging service? It's ridiculous. Twitter's not in that business.
Netflix were the first major organization to publish a hypermedia-based API. People wrote all kinds of clients for that API, to do discovery and provide alternate views of the data.
Since everything was based on hypermedia, those clients were flexible enough to adapt to changes in the representations. But they couldn't adapt to the biggest change of all: Netflix shutting down the API because their streaming business was big enough they don't need your help anymore.
In a commercial environment there is the drive to own as much of the user experience as possible. The most common model for a “REST” API has one server implementation controlled by one company, and a bunch of little client implementations.
When your product is new and unformed, and you're not sure what to do with it, you want the flexibility to change it, and you want people to try different things with it. Hypermedia is an unconventional choice but it works well for this.
But at some point you're going to figure out what your API is “for”, and at the point, why not own the entire experience? Once you reach a certain level of success you'll probably make the same call Netflix made. You don't need hypermedia anymore.
Okay, well, I'm your customer, I've got my own organization to run, and the history I've seen makes me jumpy.
Like I said, we license books from all the main vendors in the industry. But one of these vendors has 80% of the market. Everyone else is competing for 20%. And all these companies are trying the same full-service platform strategy.
If I want a full-service platform, I'm going to go with the company with 80% market share. It's a no-brainer. So I wish the companies fighting over the 20% that's left would try a different strategy.
Myself, I'm under no pressure to pitch a full platform solution, so I am free to let people from Colorado or the Netherlands take the components they want. Even if you all you take from me is the idea of using OPDS, the OPDS 'web' grows larger and everyone benefits.
Hypermedia was an easy sell to NYPL because libraries are used to having multiple competing authorities and reorganizing things all the time. Libraries have been around so long, and we adopted computers so early, that we constantly have to take twenty-year-old systems and connect them to something brand new.
We didn't set out to compete with commercial offerings, but distributed hypermedia is a very competitive strategy, if you can handle the ideological implications.
Public libraries are in this for the long haul. When it comes to our core operations, like buying books and letting people read them for free—we don't want a full-service platform. We want multiple vendors who follow a spec, and we want input into the spec, because we have to maintain service to our patrons when a vendor goes out of business or decides to cut us off.
And it's not just libraries. I've felt this way before. This is how I feel about Facebook, in my personal life. But NYPL is the first place I've worked where I've had the power to push back against this default relationship.
And I don't think there's an irreconcilable difference in values here between corporate and nonprofit. If you're not the company with 80% of the market, there's an opportunity here for you.
Hammer out the equivalent in your industry of OPDS, or the Web. This will mean talking to your customers. It will mean talking to your competitors and finding common ground. Don't invite the 80% company. Band together with the other 20-percenters.
I don't say this lightly and I know this won't be easy, because I've worked in both ends of the 80/20 split. I stand with the 20%. I'm rooting for you. But I'm not rooting for one of you to take over the market and become the 80%. I'm rooting for the industry as a whole.
Libraries don't compete with each other, and working in a place where coming together was relatively easy made me see how effective it is when you pull it off. This is the spirit that gave us great public goods like Atom and the Web. There's no reason it can't keep happening.
So if you're in an 80-20 industry, give it a try. Unless you work for the 80% company in your industry, in which case please forget this whole talk.
|
This document is part of Crummy, the webspace of Leonard Richardson (contact information). It was last modified on Thursday, January 07 2016, 12:49:46 Nowhere Standard Time and last built on Friday, June 02 2023, 14:00:22 Nowhere Standard Time.
| Document tree: Site Search: |
