Category Archives: Uncategorized

2022 Review

It seems like just the other day when the Open Library welcomed its 2 millionth registered patron in 2018. This year, we zoomed past 6M registered book lovers who collectively in 2022 have borrowed 4.3M books and counting, and who have added more than 4.7M books to their reading logs. Our book catalog expanded to nearly 38M editions and we cleaned up nearly 230k low quality records.

Together, our team released a flurry of features and improvements to the Open Library service including:

Imminently coming is a game changing smart edition-search upgrade, a Yearly Reading Goals feature, support for Web Books, a significantly more usable 1-stop “My Books” page, and design improvements to the Books Page.

Want to see a full run-down of what we accomplished together?
Check out our 2022 Open Library Community Celebration video!


Important Documents

If you’ve ever wanted to dig down deeper into the Open Library’s transparent processes:


Previous Community Celebrations

For the past 3 years around October, the Open Library has recognized its contributors by hosting an Open Library Community Celebration.


Historical Yearly Highlights

In addition to the yearly community celebration, we’ve tried to make end-of-year review posts to give the community transparency into our victories, changes, and planning. In:


Gratitude. Central to these achievements were my fellow staff on the Open Library’s engineering team: Drini Cami, Jim Champ, & Chris Clauss. Equally indispensable to this year’s achievements was Lisa Seaberg from Internet Archive’s Patron Services team. Lisa is both a voice and champion for our patrons as well as the Open Library’s Lead Community Librarian who helps facilitate our community of 500 librarian contributors and our Super Librarians (Daniel, Travis, Onno, et al) who work tirelessly together to keep our library catalog organized. Charles Horn from the openlibraries.online team has been instrumental in keeping MARC records flowing into the catalog and Cari Spivack on policy support. And this year 6 Open Library Fellows — Hayoon Choi, Sam Grunebaum, Dana Fein-Schaffer, Scott Barnes, Constantina Zouni, and Teo Cheng — who selflessly committed several months of their time to improve the Open Library platform for the world, alongside a team of more than 30 volunteer developers from around the globe. Thank you, of course, to Brewster Kahle and all of our patrons and generous donors for believing in us and keeping us funded for another year. And a special thank you to a sorely missed Aaron Swartz, without whom none of this would be possible.

Happy Thanksgiving!

– The Open Library Community

Search for Books in your Reading Log

by Scott Barnes, 2022 Open Library Fellow

As of last week’s deploy, it’s now possible to search the Open Library for the books in your reading log by navigating to the My Books page, selecting the Currently Reading, Want to Read, or Already Read bookshelf, and typing in a search query.

Keep reading to learn tips and tricks on how to effectively search for books within your reading log.

A Forward by Mek

This year the Open Library has been exceedingly lucky to collaborate with Scott Barnes, a lawyer who has reinvented himself as a very capable software engineer. We had the pleasure of meeting Scott earlier this year while he was scouring the Open Library for old rock climbing guidebooks. Ever since joining one of our community calls, he’s been surmounting challenging technical hurdles as one of our most active 2022 Open Library Fellows. As law professor Lawrence Lessig famously penned in his 1999 book Code: “Code is law”. I guess that’s why we shouldn’t be too surprised how quickly Scott became familiar with the Open Library codebase, at the precision of his work and attention to detail, and his persistence in getting code just right without getting slowed down. We hope you’ll enjoy Scott’s contributions as much as we do and learn at least one new way of using the reading log search to improve your book finding experience.

A Forward by Drini
I’m exceedingly pleased to introduce Scott Barnes to the Open Library Blog. I have had the honour of mentoring Scott throughout some of the projects on his fellowship, and have been floored by his love, passion, and skill in all things programming. Whether it’s working on user facing features (such as this one), improving code architecture, investigating performance issues, setting up infrastructure, or keeping up-to-date with new programming techniques by diving into a new programming book or topic, Scott is always excited to dive in, learn, and make an impact. And his code never fails to meet requirements while being well-architected and robust. I am so excited to see what he does next with his programming super powers! Because as far as I can tell, there’s no stopping him. Now without further ado, I’ll hand it off to Scott to talk about:

👋 Hi, my name is Scott Barnes. This year as a 2022 Open Library Fellow I collaborated with Drini Cami to develop reading log search. In this post, I’ll show different ways of effectively using the new reading log search feature, as well as technical insight into how it was engineered behind the scenes.

Profile photo for scottreidbarnes

3 Ways of Searching your Reading Log using a Web Browser

The most common way to use search your reading log is by entering a natural, free-form search query, just like you might using your favorite search engine. From the Currently Reading, Want to Read, or Already Read page, you can search for your books on that reading log shelf by submitting text describing the book’s title, author name, ISBN, or publisher. An example could be “Lord of the Rings by J.R.R Tolkien”.

If you want greater control, you can also harness the power of Apache Solr, the underlying technology which powers the Open Library search engine.

Let’s say, for example, that you’d like to find books on your reading log by a specific author named “King” but using the Natural Language mode instead returns books with “King” in the title. Using keyword search, you could search for author: king to see only books by authors named “King” (while not seeing books with “King” in the title). On the other hand, if you only want to find titles matching “King” and not the author, you could instead search for the keyword title: king. Want to find your horror books? Try subject:Horror.

For a list of common keywords, you can see the Open Library Search How-To Guide, or peek at the code behind it: worksearch/code.py.

Reading log search, like the main Open Library search, supports boolean operators, specifically AND, OR, and NOT, along with wildcards such as * and ? to match multiple characters and a single character respectively. Therefore, to search for all books matching “climb”, “climber’s”, “climbs”, etc., that were published by Sierra Club Books that you want to read, you could visit Want to Read and search for title: climb* AND publisher: sierra club books.

NOTE: the boolean operators are CaSe sensitive, so AND will work as expected, but and will not. The actual search terms themselves are not CaSe sensitive, however, so “king” and “KiNg” will return the same results.

Searching your reading log via the API

For those looking to perform programmatic searches, it’s possible to search via a web API using the following RESTful pattern: https://openlibrary.org/people/{USERNAME}/books/{SHELF}.json?q={QUERY}.

For example, to search for all titles matching “king” on my Want to Read shelf I’d query: https://openlibrary.org/people/srb36/books/want-to-read.json?q=king.

Or if I wanted to search for all titles matching climb* on that same shelf, I’d search for https://openlibrary.org/people/srb36/books/want-to-read.json?q=title:(climb*)

Behind the scenes

Now for the technical details! Reading log search was added in pull request #7052. In exploring how reading log search might be accomplished, two key things leapt to our attention:

  1. Reading log records are stored in the database, and work and edition data (i.e. “books”) are stored in Solr; and
  2. To work with the split data, we were probably doing more queries than we needed to, both in the back end itself, and within the templates that make up the pages.

The goal then was to add the ability to search the reading log, ideally while reducing the number of queries, or at least not increasing the number.

Changing the back end

Most of the heavy lifting was done in core/bookshelves.py. The challenge here was addressing the reading log records being in one database, and the edition data being in Solr.

The solution was to query the reading log database once, and then use those results to query Solr to get all the information we’d need for the rest of the process. Then we could simply pass the data around in a Python dataclasses, and then ultimately pass the results through to the templates to render for display in patrons’ browsers.

For the super curious, this is found in get_filtered_reading_log_books()

Changing the templates

As mentioned, having the data in two places had led to some excess querying, which manifested itself in the templates where we re-queried Solr to get additional data to properly display books to patrons.

However, because we had gathered all of that information at the outset, we just had to change the templates to render the query results we passed to them, as they no longer had to perform any queries.

How you can help

Volunteers not only help make Open Library special, but they help make it even more awesome. Check out Volunteering @ Open Library.

Improvements to the Main Navigation

Dana, UX/Design Fellow on Open Library

Forward by Mek

Few aspects of a website have greater impact and receive less recognition than good navigation. If done well, a site’s main navigation is almost invisible: it’s there when patrons need it, out of the way when they don’t, and it zips patrons to the right place without making them overthink. Over the years, we’ve attempted improvements to our navigation but we haven’t had the design bandwidth to conduct user research and verify that our changes solved our patrons problem. It turns out, we still had several opportunities for improvement. That’s why this year we were incredible lucky to have collaborated with Open Library UX Design Fellow Dana Fein-Schaffer, who recently transitioned into design from a previous role as a neuropsychological researcher. Dana’s formal education and experience links a trifecta of complimentary fields — computer science, psychology, and design — and has resulted in unique perspective as we’ve endeavored to redesign several of Open Library’s core experiences: the website’s main navigation and the desktop version of our My Books page.

As an Open Library UX Design Fellow, Dana has been in charge of design direction, conducting user interviews, figma mockups, feedback sessions, and communicating decisions to stakeholders. In addition to her skill prototyping and notable problem solving capabilities, Dana’s warmth with the community, affinity for collaboration, and enthusiasm for the project has made teaming up with her a gift. While we’d rather the fellowship not end, we endorse her work with great enthusiasm and highly recommend organizations which share our values to view Dana’s portfolio and engage her for future design opportunities.

The Design Process

IMG_7214.JPG

Hello, I’m Dana Fein-Schaffer, and I’ve been working as a UX Design Fellow with Open Library over the past several months. I’m currently transitioning into UX Design because after gaining professional experience in both psychology research and software engineering, I’ve realized that UX is the perfect blend of my skills and interests. I’ve been enjoying growing my UX skillset, and working with Open Library has been a perfect opportunity for me to gain some formal experience because I love reading and was hoping to work on a book-related project. Moving forward, I’m particularly excited about working as a UX designer for an organization that focuses on social good, especially in the literary, education, or healthcare spaces! If you have a role that may be a good fit, or if you work in one of those industries and are interested in connecting, please feel free to reach out. You can also learn about me from my portfolio.

Problem

At the beginning of my project, the current navigation bar and hamburger menu looked like this: 

I met with members of the Open Library team to identify three key areas of concern: 

  1. The label “more” was not descriptive, and it was unclear to patrons what this meant
  2. The hamburger menu was not consistent with the navigation bar items
  3. The hamburger menu was confusing for patrons, especially new patrons, to navigate

Approach

Before I began my project, the Open Library team decided to implement an interim solution to the first point above. To address the concern with the “more” label, the navigation menu was changed to instead include “My Books” and “Browse.” The website analytics showed that patrons frequent their Loans page most, so for now, the “My Books” page brings patrons to the Loans page. 

To begin redesigning the main site navigation, I first created a prototype in Figma with some potential solutions that built off of this updated navigation menu:

  1. I created a dropdown menu for “My Books” that would allow patrons to select the specific page they would like to go to, rather than automatically going to the Loans page
  1. I reorganized the hamburger menu to be consistent with the navigation menu and to use the subheadings of “Contribute” and “Resources” instead of “More.” I felt that these changes would make the hamburger menu easier to navigate for both new and long-time patrons. 

After creating the prototype, my next goal was to get feedback from patrons, so I scheduled user interviews with volunteers.

User Interviews

I conducted user interviews via Zoom with four patrons to answer the following questions: 

  1. How do users feel about the My Books dropdown? 
  2. Are users using My Books and Browse from the navigation menu or hamburger menu?
  3. Are users able to effectively use the hamburger? Do they find it easier or harder to find what they’re looking for using the reorganized hamburger?

Results & Findings

  1. Three out of the four patrons preferred the dropdown, and the fourth user didn’t have a preference between the versions. The patrons enjoyed having the control to navigate to a specific section of My Books. 
  2. All users used the navigation menu at the top of the page to navigate, rather than the hamburger menu, which supported the switch to “My Books” instead of “More.” This finding also highlighted the need to make sure patrons could access the pages they wanted to access from the top navigation menu. 
  3. Finally, all four patrons found the existing hamburger menu confusing and preferred the reorganized hamburger. Some patrons specifically mentioned that the reorganized hamburger was more compact and that they felt that the headings “contribute” and “resources” were more clear than “more.”

Synthesis and a New Direction

After learning from the user interviews that patrons preferred increased granularity for accessing My Books and a more concise hamburger menu, the Open Library team began discussing the exact implementation. We wanted to keep the navigation and hamburger menus consistent; however, we also wanted to provide many options in the My Books dropdown, which made the hamburger menu less concise. 

At the same time that we were debating the pros and cons of various solutions, another Open Library UX Design fellow, Samuel G, was working on designing a My Books landing page for the mobile site. Inspired by his designs, I created mockups for a desktop version of the My Books page. Having a My Books page that summarizes patrons’ loans, holds, and reading log at a glance allows patrons to still have increased control over My Books navigation while keeping the hamburger menu concise, since all of the individual My Books items can now be condensed into one link. 

Furthermore, having a My Books landing page opens the door for more ways for patrons to interact with their reading through Open Library. For instance, I’ve created a mockup that includes a summary of a patron’s reading stats and yearly reading goal at the top of the page. 

As we work towards implementing this design, I’m looking forward to getting feedback from patrons and brainstorming even more ways to maximize the use of this page. 

Reflections

Working with the Open Library team has been an amazing experience. I’m so grateful that I got to lead a UX project from start to end, beginning with user research and ending with final designs that are ready to be implemented. Working with such a supportive team has allowed me to learn more about the iterative design process, get comfortable with sharing and critiquing my designs, and gain more experience with design tools, such as Figma. It was also a great learning experience that sometimes your projects will take an unexpected turn, but those turns help you eventually come to the best possible design solution. Thank you to everyone who provided feedback and helped me along the way, especially Mek, who was a wonderful mentor, and Sam, who was a great collaborator on our My Books mobile and desktop project!

About the Open Library Fellowship Program

The Internet Archive’s Open Library Fellowship is a flexible, self-designed independent study which pairs volunteers with mentors to lead development of a high impact feature for OpenLibrary.org. Most fellowship programs last one to two months and are flexible, according to the preferences of contributors and availability of mentors. We typically choose fellows based on their exemplary and active participation, conduct, and performance within the Open Library community. The Open Library staff typically only accepts 1 or 2 fellows at a time to ensure participants receive plenty of support and mentor time. Occasionally, funding for fellowships is made possible through Google Summer of Code or Internet Archive Summer of Code & Design. If you’re interested in contributing as an Open Library Fellow and receiving mentorship, you can apply using this form or email openlibrary@archive.org for more information.

It takes a Classroom to build an Open Library

On most days, the Open Library is hard at work improving the experience it offers to students and teachers in classrooms. But for the past few months, Open Library has had the privilege of enjoying contributions from 7 students around the globe who had been assigned by their universities to participate on open source software projects.

First and foremost, the entire Open Library community extends our deep gratitude to AUEB / Athens University of Economics and Business‘s Dr. Diomidis Spinellis (professor of Software Engineering, who taught the course Software Engineering in Practice) and NYU‘s Dr. Joanna Klukowska (Clinical Associate Professor of Computer Science, who taught the course CSCI-UA.0480-061, Open Source Software Development) for incorporating open source contributions into the curriculum of their classrooms. As we hope you’ll see, the decision to promote hands-on development has an outsized impact on supporting open source projects like ours.

In the spring semester of 2022, four students from Greece’s AUEB (Constantina Z., Vassilis B., Dimitris B., and Philippos P. / Φιλιππος Π.) and three students from NYU (Michelle T., Crystal C., Chloe Q.) spent time participating in community calls, problem solving, and improving the Open Library service for the public. In return they received mentorship and first-hand experience learning how to contribute to a platform trusted by millions of international readers.

This year, the foci of Open Library’s roadmap is improving core experiences for patrons. Towards this goal, each of these students exceeded our expectations by contributing meaningful improvements like: Chinese internationalization of the website, google analytics to help inform us on meaningful ways to improve the organization of the website, fixing broken mobile navigation for our Books Page, UI improvements for sharing books on social media, adding APIs for Trending Books, and much more. We’re extremely proud of and grateful for the work these students were able to contribute.

In the past, Open Library has reserved a special honorary title of “Open Library Fellow” for exemplary contributors who have demonstrated exceptional commitment, leadership, and impact with the Open Library project. Our list of previous Fellows include Sabreen Parveen (who designed our onboarding experience), Yash Saravgi (who developed our mobile Progressive Web App), and Bharat Kalluri (who helped standardize our import pipelines). Each dedicated several months implementing features which redefined core behaviors and experiences of the Open Library.

This year, we believe one student in particular, Constantina Zouni, stands out as being especially deserving of this special Fellowship distinction, for her initiative, participation in engineering and design process with stakeholders, and outstanding work ethic.

Please join us in celebrating the work of this 2022 international student cohort, sharing our gratitude, and congratulating Constantina on her inspiring example.

Improving Experiences for Open Library Patrons

By Constantina Zouni

As this semester of my studies is coming to an end, I want to do a retrospect about my experience with the open library project.

My Journey with Open Library

In the beginning of the semester my professor Dr. Diomidis Spinellis for the course “Software Engineering in Practice” announced that in the context of an assignment we had to choose an open source project to make contributions thought out the semester. As a result, I started searching for a project and I was lucky to quickly find open library’s repository. Some of the main reason that made me to choose that project is that the community was very friendly and really open to contributors. The documentation of the project was really detailed and there were videos that helped me understand how the project works. Also, another good thing was that the issues of the project were well organized with labels and the context was explanatory enough. Moreover, the project seemed to be very active with quick responses in the comments section and pull request merges almost every day. After the first communication with the team everything went very smoothly. I was welcomed in the slack channel, and I was invited to participate in the weekly meetings. Mek quickly stepped in and helped me to get started. Because that period was busy and contributors from other universities also chose to contribute to open library the project’s team made effort to create a GitHub project and assign issues to everyone. I started solving minor issues related with text appearing when not needed, adding the subtitle to the search results and some UI improvements. Ultimately, in collaboration with another student from my university Vassilis Bubis we created the twitter social card that enables users to share their book lists. Through out the whole period that I contributed to the project I was impressed that Mek and the other members of the open library team dedicated time answering our messages and even jumping on small zoom meetings.

Book page header in mobile

One of the issues that I think had a big impact in the open library users is the improvement of the book page header in the mobile environment. When users visited a book page from mobile the experience wasn’t that pleasant. The book title and other important information like the author, the subtitle etc didn’t fit in the phone screen and the user had to scroll down to see them. The issue was more significant in the cases where the book covers were ambiguous, and it made difficult for the user to understand if they were in the correct page. Jim Champ recommended to follow a specific layout for the book page in mobile in order to fit all the important information in the mobile page. The challenge was the layout had to be different depending on the device of the user. My first implementation involved some java script code that change the order of the elements and an event listener that was activated when the screen had a specific size. The open library team quickly informed me that this implementation was causing a delay in the loading of the page, and they recommended me to use HTML and CSS. This time with a new implementation and the help of Jim Champ who was reviewing my pull request I managed to solve the issue using an HTML file that included only the title summary and some CSS commands.

Book header in mobile before and after

Dynamic book list preview for sharing

This new feature was a little more challenging than the previous one. This time I collaborated with Vassilis Bubis in order to create a dynamic preview for the book lists of the users that displays the first 5 books of the list. Then this preview is passed to the twitter social card and every time a user wants to share a list with the URL the preview image appears. This is a more interactive way for users to show their book lists to others and makes open library more recognizable across twitter users. The first challenge was to create a mock-up of the preview. To achieve that I used a design tool called Figma to create prototypes with different colour combinations and I let the open library team to decide which on they like more. For the design I used colours from the open library’s webpage, and I added a twist in the preview that represents a self where the books are placed. Alongside with the mock-ups Vassilis worked on retrieving the book covers that we need and place them above the background with the help of a Python library called Pillow. Then I stepped in, and I made sure that every book cover was resized in a way proportional to the original dimensions that it had. We noticed that some covers were stretching so it was important that every time we changed the width of a cover the height was adjusted properly. Another challenge was the text that we wanted to add in the preview. The text had to change dynamically, and we had to change line every time the characters exceeded a specific number to achieve an aesthetically pleasant result. One issue that we faced was that the coordinates of the covers that we had figured out with Figma had to change because in python the coordinates are applied from upper left corner compared to Figma that apply to the center of an image. After solving that Vassilis and I proceeded on storing the image in an in-memory binary array for better performance and finally creating the API for the list page.

Twitter social card for book lists sharing

Book page editing improvement

While working on some issues in the book page I realised that compared to other library webpages open library gives users the ability to edit the details and the information of a book. That feature is very valuable because users can add important details for a book that were missing when it was added, they can update that information and they can add descriptions and subjects that might be useful for other users. Although this feature is really important the editing user interface is not that pleasant. When users click on the edit button, they are directed to another page. My recommendation regarding that is to use a modal that pops up when the button is clicked. In that way users will feel like they have more control because they won’t be directed to another page, and they can still see the book page behind the modal. Another issue with the existing editing form is that users can discard the changes with the cancel button, but they can’t undo a change without deleting all the changes. In the mock-up that I created I added an arrow in the right upper side that symbolizes the undo action. I noticed that the examples for every field were placed next to the field title, and I opted to move them inside the text box for a clearer look. Finally, I added the info symbols beside every field that provides details on how you should fill out that specific field. Overall, the purpose of those recommendations is to make the booking editing more simple, compact and user friendly.

Book editing page now
Edit book page with modal created with Figma

2021 End-of-Year Community Updates

Hi Open Library Community! This is going to be a less formal post detailing some of our recent community meetings and exciting Q3 (quarter 3) opportunities to learn, celebrate, and participate with the Open Library project.

Earlier this Month

Upcoming Events

  1. 📙 Library Leader’s Forum 2021-10-13 & 2021-10-20
  2. 🎉 Open Library Community Celebration (RSVP) 2021-10-26
  3. 📅 2022 Roadmap Community Planning (join) 2021-11-02 @ 10am PT

Open Library Community Celebration 2021

Last year we started the tradition of doing an Open Library Community Celebration to honor the contributions & impact of those in our community. On October 26, 2021 @ 10am Pacific we will be hosting our 2nd annual community celebration. We hope you can join us!

During this online event, you’ll hear from members of the community as we:

  • Announce our latest developments and their impacts
  • Raise awareness about opportunities to participate
  • Show a sneak-peek into our future: 2022

EDIT: The Community Celebration happened and you may watch it here!


5-Year Vision

End of September on 2021-09-28 @ 10am PT, the Open Library community came together to brainstorm Open Library’s possible long-term directions. Anyone in the community is welcome to comment and add their notes and thoughts:

https://docs.google.com/document/d/1q_jAcdEc705H3gsZv_Yt_08c8YFmefdSvMbiljc2O8g/edit#


2021 Year-End Review

On 2021-10-12 @ 10am PT the community met to review what we had accomplished (see review doc) on our 2021 roadmap.


2022 Community Planning

First week of November on 2021-11-02 @ 10am PT the community will meet to brainstorm goals for Open Library’s 2022 roadmap. This community planning call will be open to the public here.

EDIT: Community Planning happened and you can see the results or leave comments using the 2022 roadmap link.

EDIT: Our 2022 Priorities can now be seen here: https://docs.google.com/document/d/1edU3lCTHAjFr1mXUilh8l1_rNek33pRTzFHltBU–fM/edit#