Duplicate Authors? Wave your Magic Wand!

[note: the author merge feature is no longer patron-facing]

In your wanderings around Open Library, you may occasionally have seen two records for a person you know to be a single author, like Brooks, Terry & Terry Brooks.

Look for the Magic Wand around the site to start merging!

Today, we’re releasing a new feature to help you merge those two separate Terry entries into one. This, in turn, will update all the Works listed under each Terry and try to reconcile each Work by each Author to try to reconcile a tighter list of Works for the newly merged Terry. Magic!

Try a search for your favourite author now, browse recent author merges, or read on…

A few things bear explaining:

  • The merge feature works on the idea of a Master author and its Duplicates. As you do the merge, it will be up to you to elect the most suitable Master. We select the author record with the most Works as the default, but you can change that
  • Only people with an Open Library account can merge authors
  • Updating the search engine after a merge takes a little while at the moment, up to about 10 minutes, so you won’t see the list of the new Master’s Works updated immediately. We’re looking to speed this up, but are very happy to release this as a “minimum viable product.” As I mentioned, merging an author with either lots of works, lots of editions, or both, takes a long time to update, so please be patient.
  • Duplicate authors’ names will be saved as an alternate on the Master record. For example, the (new) Master record for H. P. Lovecraft now lists alternates like Howard Philips Lovecraft, H. P Lovecraft, Howard P. Lovecraft and H.P Lovecraft. These alternates are often just subtle differences in spacing or capitalization, and we’re hoping they might prove useful later if we begin to stockpile them now.
  • If you’re in any doubt about whether or not to merge an author, don’t. It’s possible you might come across an odd-looking author name like August (re: H. P. Lovecraft) Derleth or H. P. (introduction by Lin Carter) (with Harry Houdini on Pharoahs) Lovecraft in a search for H. P. Lovecraft… these are trickier, because they’re noting contributors in the author name. Ideally, those contributors would be siphoned out into the contributors field per edition, and not merged into the H. P. Lovecraft Master. That would be a loss of information. So, it’s probably easier to just leave those long, odd “authors” alone for now.

I’ve actually found it really fun to test this new feature. I found a useful directory listing of authors on Yahoo of a ton of authors that I began to merge in Open Library. By referring to an external list like this, I could just move from one to the next, rather than trying to come up with authors to search for.

We’ve also bundled another enhancement into this release: Recent Changes V2: There’s a new little bit of navigation to the recent changes page, so you can see things like all the authors merged on 8/16/2010, or all the bot edits made in June 2010. We’re looking forward to adding other bits and pieces to these new filtered views, for example, all the new ebooks made available on a certain day, or all the new covers uploaded in a certain month. Perhaps these could also have feeds available too, so you could subscribe to a feed of changes to keep your version of the Open Library dataset up-to-date.

As well as Recent Changes V2, we’ve introduced the concept of “save_many” for transactions that contain lots of little updates. This is a performance improvement, and entered as a single line in Recent Changes – look for the little “expand” link to open up the contents of the save_many transaction.

So, why not have a shot at merging two duplicate authors? The best place to start is the Author search page.

Anyhoo, we’re excited to show you the first major feature we’ve rolled out since the launch of the redesign back in May, and we’re excited to see what you make of it. Go forth and merge!

Easy permanent links to book page images

We just launched a new image permalinks feature for downloading and linking to page images of books hosted on the Internet Archive. Using a page image permalink makes it easier to references the contents of a book hosted on the Archive without having to know the details of how or where the book is stored. Since a book’s data could be moved around within the multiple petabytes of data in the Archive at any time the permalinks provide a consistent and stable way to access the page images.

Here are a few quick examples. For each of these URLs you would add http://www.archive.org/download/{item identifier} to the beginning (hover over an image to see its full URL).

Referencing the cover image for a book at thumbnail size:
/page/cover_thumb.jpg

Continue reading

Improved Set Up for Developers

Over the last few months, a handful of the developers at the Internet Archive have begun working more closely with Open Library code, where previously, the project was more isolated and had really only been worked on by the core team of, well, two: Anand & Edward. Apart from more fun collaborating with colleagues across the Archive, this increased exposure of the Open Library code base has been profoundly useful for the project. Apart from the very useful fresh perspectives and questions, it’s also led to an improved toolset for getting a developer’s instance of Open Library up and running on your local machine – so important when you’re trying to find your way around a new system.

The cherry on top is an install script for Linux, written by Raj Kumar, on top of the awesome work done by Michael Ang (Mang) to prepare for our recent Lending launch. The updated docs are here:

http://openlibrary.org/dev/docs/setup

This is a bit of a milestone for us – making the codebase more accessible and easier to work with is something we’ve wanted for ages, so it’s nice to see it well on its way.

Open Library Ore: A MySQL data dump is available

A while back, Ben Gimpert - click to visit Ben's websiteBen Gimpert wrote a guest post for us called Open Library Ore, explaining how he had begun to hack on the massive full text corpus on the Internet Archive, practising various Natural Language Processing techniques to begin to teach machines to glean topics of books by sheer letter crunching. Turns out the elements in the ore are beginning to emerge, particularly in the form of a dataset available for download under Attribution-Noncommercial-Share Alike 3.0 CC license… Please, if you know SQL, why not download the dataset and see what you can find out? We’d love to hear any discoveries you make, perhaps in the comments?

Continue reading

Today's Downtime (is over)

We’re going to bring OpenLibrary.org offline at around 3PM Pacific Time to upgrade our database server. If all goes according to plan, this upgrade should take approximately three hours. We’ll be replacing our old spinning platter disks with modern solid state drives as part of a broader effort to improve the overall performance and reliability of the Open Library system.

We’ll post updates if they come to hand.

6PM UPDATE – Looks like it’s going to take a wee bit longer than expected to get everything copied over. And by “wee bit,” we mean another hour or so from now. If the database decides it wants to move slower, we’ll update again. Thanks so much for your patience!

7.45PM UPDATE – And, we’re back!