Back in 2020, we started the tradition of hosting an annual Community Celebration to honor the efforts of volunteers across the globe who help make the Open Library project possible.
Tomorrow, Tuesday, October 31st at 9am Pacific, we warmly invite the public to join us in a small gathering to celebrate the hardworking humans who keep the website going, see demonstrations of their accomplishments, and get a glimpse into our direction for 2024 — Halloween Edition!
Earlier this year, the Internet Archive’s Open Library conducted a brief survey to learn more about patrons’ experiences and preferences when borrowing and reading books. As promised, we’ve anonymized the results and are sharing them with you!
The purpose of this survey was to better understand:
If, how, & why Open Library patrons download books
How patron reading preferences align with our offerings
For one week, starting on Tuesday 2022-02-07, OpenLibrary.org patrons were invited to participate in a brief survey including 7 questions — one of which was a screener to ensure we only included the responses of patrons who have prior experience using the Open Library.
In total, 2,121 patrons participated in the survey and, after screening, 1,118 were included in the results.
Errata: In the original survey, the question asking patrons “When you DON’T DOWNLOAD the books you’ve borrowed from Open Library, what is your primary reason?”, we mistakingly omitted a “N/A – I Don’t typically download” option and we corrected this on day 1 of the survey.
6 Key Learnings
Around half of participants have used adobe content server with DRM to securely download their loaned books
Of participants who download their loans, the top reason (54%) is for offline access
Of participants who download their loans, a quarter do so because they prefer the EPUB text format to the image-based experience of the online bookreader.
Around 42% ofparticipants report difficulty downloading their loans. Of these participants…
69% were unable to locate a download option (or a download option didn’t exist for that book)
31% experienced found a download option but couldn’t get it to work
Around half of participants intentionally opt for BookReader for a variety of reasons:
Its simplicity & convenience; no app installation required
Many teachers can’t download on school computers
Many patrons don’t trust downloads, dislike DRM, or want their reader privacy protected
Some patrons have limited storage space
Around half of participants read for pleasure, the other half for some form of self-learning or research.
What participants said
~150 participants shared their praise, thanks, and personal inspirational stories
~75 participants offered productive critiques for how we could improve our book finding and book reading experiences
Fixing OCR, hiding menu bars while reading, zooming & scrolling, etc
~55 participants expressed concerns about “1 hour” lending duration
Some participants did not like the intrusive, non-dismissable Open Library banner
We heard feedback from the community loud and clear that the implementation of 1-hour loans may not always be ideal for all patrons. The Internet Archive has been exploring and prototyping various tweaks to lending, such as an auto-renewal mechanism, that could extend a loan automatically for a patron if, at the end of the loan period, the book is still actively being read.
Hello, I am Jayden Teoh, a student from Singapore, and this year I participated as a 2023 Google Summer of Code contributor with the Internet Archive’s Open Library project to improve the site’s performance and supercharge subject pages.
If you are an Open Library patron, you have likely encountered times where certain pages seem to take and eternity to load. The Open Library team understands the importance of a smooth browsing experience and empathizes with how degraded site performance affects patrons. This is why we prioritized site performance as a key focus for our 2023 GSoC roadmap. As strongly as we felt about improving the core performance of the current website, we also wanted to push the boundaries of Open Library’s capabilities by releasing community-powered subject pages we hope will help patrons more easily showcase and discover books they’ll love. I’m excited to share more about what we accomplished and next steps in our plans.
Improving Site Performance
According to Browserstack,”40% of visitors will leave a website if it takes longer than three seconds to load”. But how do we measure which pages are slow or fast? How do we determine if a slow load time is an anomaly or a systemic pattern? Do we care about improving the average load time for a page or eliminating the most egregious case where pages load especially slowly?
Sentry, a visual dashboard often used for error monitoring, has a “Performance” mode we were able to use to identify and rank pages according to metrics called P50 and P95 — the upper bound number of seconds at which 50% (P50) and 5% (P95) of transaction took to complete. For example, a P95 score of 5 seconds tells us that 95% of such requests completed within 5 seconds (and perhaps 5% were slower). Once we ranked pages in consideration of these metric, it became clearer just how bad certain pages could be in worst case scenarios. We coupled this information with our own domain expertise about which pages are most important to the average patron’s experience and then embarked on a journey with the aspiration of reducing the average load time of key pages by at least half.
For each row in Sentry’s performance dashboard, one can “drill in” to the page to see stack tracebacks and detailed breakdowns about which functions were participating most to the slow response.
Our research revealed 2 opportunities:
The “Search Inside” page was taking more than 11 seconds and an average of more than 2 seconds because the response was making redundant archive.org metadata request on each search result match on the page to determine each book’s availability, rather than computing the availability of all the books in a single request.
Several of the slow pages had a common slow component — the LoanStatus borrow button — which we could speed up by caching and thus “feed two birds with one scone”.
By the end of the 12 weeks of this program, we manage to reduce the load times of several key pages significantly. One of my proudest achievements was the reduction of the ‘search/inside’ page by over 500%. This feature is important to patrons because it allows them to search for content within books, rather than just searching based on the author and title so I am glad we were able to make this feature faster and thus more accessible.
Editor’s note: We are still collecting metrics and plan to add before-and-after graphs of the search inside page speeds. Our changes to the borrow button are in the process of being staged and tested and we’re excited to update this blog post with metrics in the future. Hopefully you have noticed the improvements since it was launched a few weeks ago!
Unleashing the Power of Subject Tags
Empowering Librarians and Expanding Book Categorization at Open Library
For almost a decade, the Open Library has had basic subject pages that give readers a way to browse or search for books on a given topic, see books with similar subjects, and discover prolific authors of a genre. It may surprise you to learn that the whole page experience is generated based on the name of the subject. For instance, when one visits the “Magic” subject page, one may notice a carousel of books that is populated using a query based on its name: “subject:magic“. This approach gives us a simple formula for creating millions of subject pages on-the-fly, but it also has significant shortcomings.
Namely, subject pages are incapable of storing additional metadata about a given subject and the current subject pages is limited to showcasing a single carousel of books. If the subject is overly vague, like “textbooks“, the reader may often not be shown a useful set of books and there’s no affordance provided that helps the reader narrow their search further, e.g. to design textbooks. If we search for a subject called “design textbooks“, we are informed no matching subjects exist. However, if we do an intersecting search for books that are subject:textbooks AND subject:”industrial design”, there are a few interesting results! The problem is, there’s currently no mechanism which allows librarians to extend Open Library subjects and specify which book collections should show up.
My primary objective through GSoC was to give librarians the ability to enrich and edit any subject page on Open Library so each page may be as beautiful and thoughtfully curated as a library or bookstore showcase. Our solution was to give librarians the ability to create a new “Tag” document for any subject page and load it with custom logic to extend how that subject page should be rendered. Tags serve as a catalyst for librarians to provide more precise categorizations within broad subjects. By leveraging Subject tags, librarians can dive deeper into specific areas of interest, allowing readers to discover a rich array of sub-subjects. For instance, librarians might choose to add new rules into the Tag document for the Cooking subject featuring carousels for vegan and budget cooking, in order to make it more useful for readers. This granularity opens up a world of possibilities, enabling readers to explore their preferred niches and discover hidden gems within subjects they cherish. Just like how a physical library may rotate their bookshelves with new categories every month, Subject Tags grant librarians more freedom to curate interesting subject topics that may suit patrons, allowing for a more personal and humane touch to the book discovery process.
By now, I hope you are able to understand just how pertinent Subject Tags will be to our Open Library and why it is a privilege for me to be working on such an important feature. Although the idea is clear, the implementation certainly is not. Open Library’s database is built using our own niche and complex Wiki engine called Infogami. To create a new class of data, we would have to create a new Infogami type. Here’s the catch: there has not been a new Infogami type created in the last 13 years and there is no existing documentation for doing so. Navigating any new code architecture can be a tedious task for any programmer and now I had to miraculously work with an arcane technology that no one knows how to use? What could go wrong?
Thankfully, I had the support of a wonderful community and amazing mentors like Mek, Jim, and Drini. They provided me with a lot of guidance throughout the process of reverse engineering the creation of an Infogami type. And after months of work, I was able to successfully incorporate a new Subject Tags Infogami type into the Open Library architecture. Especially since Open Library is an open-source project, I decided to write a tutorial and document the unintuitive technical aspects of implementing a new Infogami type, as a gift to help future developers who may wish to extend the functionality of the platform in similar ways. The tutorial can be found here.
Now, let me show you the power of Subject Tags and how they can be used to enrich the Open Library’s Subject pages. Let’s use the ‘Magic’ subject page as an example. This is how it looks right now.
As you can see, currently the subject page is plain with no description about what the subject is about. That’s not very informative is it? Prior to Subject Tags, we are unable to store more information about subjects because they are just strings with no capabilities to store other metadata. However, now with the Subject Tags, we can do that! Let me show you how. First, let’s add a new Subject Tag into the Open Library for the ‘Magic’ subject.
The Subject Tag creation form allows us to store metadata about the ‘Magic’ subject, including its description. After we’ve created the Subject Tag, let’s head back to the ‘Magic’ subject page. Tada, we can now see the newly added description in the subject page.
You are probably still not convinced of the utility of Subject Tags. Let me give you a deeper glimpse into the realm of possibilities that Subject Tags offer. Currently on the Magic page, we are only able to display a carousel with books that have the subject ‘Magic’. What if we want to include a carousel displaying books about ‘Magic Tricks for Kids’? Well, with Subject Tags, now we can! As a librarian, we can edit the ‘Magic’ Subject Tag and use the experimental Plugin to define a new carousel. Right now, the interface is quite advanced, is still being prototyped, and is intended for expert librarians who know how to compose queries, but in the future we aim to make it easy for any librarian to extend the functionality of subject pages using Tags.
Plugins allow subject pages to load custom templates within our system and utilizes them to enrich the subject page. For example, in the Plugins field of the Subject Tag edit form above, we included a new QueryCarousel Plugin that allows the ‘Magic’ subject page to search for all books with the “magic tricks juvenile literature” subject and display them in a template carousel. Let’s take a look at the ‘Magic’ subject page again.
Fascinating isn’t it? Subject Tags have enabled the enhancement of the previously one-dimensional subject pages. Through Subject Tags, librarians are now equipped to curate and display information that can enrich the book discovery experience of patrons.
What happens when librarians want to add a new carousel of books to a subject page but the books haven’t been labeled with subjects? When we developed the Tag feature, adding a subject to books had to be done one book at a time. To aid librarians in the process of curating books and subjects, I also implemented a Bulk Tagging tool that enables librarians to add subjects to multiple books simultaneously.
Subject Tags are still in beta so we can time our time understanding the needs of our patrons and the librarians who will use these new tools. As a next step, we have decided to do research on where this feature can have the most impact and will focus our efforts on enriching a small handful of specific subject pages using Tags. One subject we’re excited to prototype with is ‘Cooking’. The team has been curating the best cooking-related information to showcase using Subject Tags and testing new features to launch alongside Subject Tags. Here is a mockup by Roya, a fellow in our design community, showing one possible UI we have in mind:
When Subject Tags are launched, we hope you can visit the ‘Cooking’ page and provide us with input on what we can improve on and what you would like to explore in a Subject page. With your feedback, librarians will have a better understanding on how to enhance your book exploration process with personally curated topics. Moving forward, we will utilize Subject Tags to enrich other subject pages on the Open Library and slowly phase out the mundane subject pages we have currently.
Thank you to the incredible Open Library community for their unwavering support over these past months. A special shout-out goes to Mek, whose mentorship has been nothing short of exceptional. Not only has Mek dedicatedly guided me through the program, but also gone the extra mile to make sure I’ve had the most enriching learning journey. Lastly, my deepest thanks to the Internet Archive and Google Summer of Code for making it possible for me to be a part of this life-changing experience. This is an experience that I’ll never forget.
In an ongoing series, the Internet Archive hosts renowned authors of the digital humanities and beyond for virtual book talks that are free and open to the public.
For nearly a decade, the Internet Archive has hosted events at its headquarters in San Francisco, occasionally welcoming forward-thinking authors of give presentations about their recent books in the digital humanities.
The COVID pandemic has been a catalyst for change, both in the types of challenges facing our communities and how we may address them. In 2020, many libraries were required to close their doors, leaving many authors without important venues for connecting with their audiences. Many patrons have increasingly turned to their screens for access to trusted voices from the safety and convenience of their homes. Organizations like ours have also adapted by running more digital events. For instance, did you know that since 2020, the Internet Archive has commissioned more than 200 artists to perform live Essential Music Concerts From Home? We invite you to browse their recorded performances.
Best of all, you can access the complete collection of book talks for free though the library at openlibrary.org/booktalks, or by clicking the “Book Talks” tab under the “Browse” drop-down menu on OpenLibrary.org.
Stay Up To Date
Want to learn about upcoming virtual book talks hosted by the Internet Archive?
To set a new reading goal, click the “Set reading goal” button (highlighted above) and enter the number of books that you’d like to read in the input, and press the “Submit” button.
Upon submission, the “Set 2023 reading goal” button is replaced with a progress bar.
Changing or Removing your Reading Goal
Want to change your reading goal? It’s okay to change your goal! Any time before December, you will be able to adjust your reading goal by clicking on the “Edit” link next to your yearly reading progress bar.
Want to change or unset your reading goal? If at any time before December you’d like to stop tracking your progress and remove your yearly reading goal, you can click the “Edit” link and update your reading target to the number 0. You can easily opt back in later, should you choose.
Tracking Your Reading Progress
Progress towards your reading goal is made by submitting a “check-in” with the date that you finished a book. Once you have marked a book as “Already Read”, you will be given the opportunity to also set a progress check-in with a completion date. The check-in prompt will appear below the reading log button:
There are three date options for progress check-in. First, clicking the year option will create a check-in which indicates that you have finished reading the book at some point during the year. Second, clicking “Today” will automatically set the read date to today’s current date. Finally, if you finished reading the book on another day, you can set custom date by clicking the “other” link.
The custom date form allows for both full and partial dates. For instance, if you forget exactly when you’ve finished a book but you have a rough idea, you may simply choose to set the year (or the year and the month). As long as a year is provided, the book will still be counted towards that year’s reading goals.
Viewing & Managing Your Check-ins
Once set, the last read date will be displayed beneath the Reading Log button. These dates can be edited or deleted by clicking the “Edit” link and will appear both on the Book Page as well as your Already Read shelf of your Reading Log.
If a book with a check-in is moved to the “Currently Reading” or “Want to Read” shelf, the check-in can still be seen and edited.
Careful! When a book is removed from your Reading Log, the books check-ins will be deleted! In these cases, you will be warned that the book’s check-ins will be deleted and prompted for confirmation:
As a small team, we’re doing the best we can to roll out value to our patrons, knowing well that the feature won’t work perfectly for everyone’s needs. As you set out to achieve your 2023 reading goals there are some important things one should note. These limitations noted, we hope you enjoy reaching your goals with Open Library!
Q: Can I delete a previous year’s reading goals?
A: There is not yet an interface for deleting previous year’s goals, however we’d like to make this possible. If and when we do add the ability to delete previous yearly reading goals, all of your reading check-ins will stick around — only the yearly goal number will be removed. For now, if you’d like to delete a previous year’s goals, please feel free to email us at firstname.lastname@example.org and we’ll do our best to help!
Q: Are my yearly reading goals public?
As of the initial release, only you will see your yearly reading goals but in the future it may be visible to your friends and followers according to your Reading Log Privacy settings.
Q: What if I read multiple editions of the same book? Can I add progress check-insfor each edition?
The short answer is not yet. We’ve build the Yearly Reading Goals feature to allow check-ins at the edition level but currently the “Already Read” shelf operates at the Work level and so it will take us some time to develop a UI that supports per-edition check-ins. This means currently there’s no UI for checking-in multiple different editions of the same book.
Thank you to Jim Champ, on the core engineering team of Open Library, for leading the development of this feature.