Category Archives: Fellowships

Improving Search, Removing Dead-Ends

Posted on by

Thanks to the work of 2024 Design & Engineering Fellow Meredith White, the Open Library search page now suggests Search Inside results any time a search fails to find books matching by title or author.

Before:

After:

The planning and development of this feature were led by volunteer and 2024 Design & Engineering Fellow, Meredith White who did a fantastic job bringing the idea to fruition.

Meredith writes: Sooner or later, a patron will take a turn that deviates from what a system expects. When this happens, the system might show a patron a dead-end, something like: ‘0 results found’. A good UX though, will recognize the dead-end and present the patron with options to carry on their search. The search upgrade was built with this goal in mind: help patrons seamlessly course correct past disruptive dead-ends.

Many patrons have likely experienced a case where they’ve typed in a specific search term and been shown the dreaded, “0 results found” message. If the system doesn’t provide any next steps to the patron, like a “did you mean […]?” link, then this is a dead-end. When patrons are shown dead-ends, they have the full burden of figuring out what went wrong with their search and what to do next. Is the item the patron is searching for not in the system? Is the wrong search type selected (e.g. are they searching in titles rather than authors)? Is there a typo in their search query? Predicting a patron’s intent and how they veered off course can be challenging, as each case may require a different solution. In order to develop solutions that are grounded in user feedback, it’s important to talk with patrons

In the case of Open Library, interviewing learners and educators revealed many patrons were unaware that the platform has search inside capabilities.

“Several interviewees were unaware of Open Library’s [existing] full-text search, read aloud, or note-taking capabilities, yet expressed interest in these features.”

https://blog.openlibrary.org/2024/06/16/listening-to-learners-and-educators/

Several patrons were also unaware that there’s a way to switch search modes from the default “All” to e.g. “Authors” or “Subjects”. Furthermore, several patrons expected the search box to be type-agnostic.

From our conversations with patrons and reviewing analytics, we learned many dead-end searches were the result of patrons trying to type search inside queries into the default search, which primarily considers titles and authors. What does this experience look like for a patron? An Open Library patron might type into the default book search box, a book quote such as: “All grown-ups were once children… but only a few of them remember it“. Unbeknownst to them, the system only searches for matching book titles and authors and, as it finds no matches, the patron’s search returns an anticlimactic ‘No results found’ message. In red. A dead-end.

As a Comparative Literature major who spent a disproportionate amount of time of my undergrad flipping through book pages while muttering, “where oh where did I read that quote?”, I know I would’ve certainly benefitted from the Search Inside feature, had I known it existed. With a little brainstorming, we knew the default search experience could be improved to show more relevant results for dead-end queries. The idea that emerged is: display Search Inside results as a “did you mean?” type suggestion when a search returns 0 matches. This approach would help reduce dead-ends and increase discoverability of the Search Inside feature. Thus the “Search Inside Suggestion Card” was born.

The design process started out as a series of Figma drawings:

Discussions with the design team helped narrow in on a prototype that would provide the patron with enough links and information to send them on their way to the Search Inside results page, a book page or the text reader itself, with occurrences of the user’s search highlighted. At the same time, the card had to be compact and easy to digest at a glance, so design efforts were made to make the quote stand out first and foremost.

After several revisions, the prototype evolved into this design:

Early Results

The Search Inside Suggestion card went live on August 21st and thanks to link tracks that I rigged up to all the clickable elements of the card, we were able to observe its effectiveness. Some findings:

  • In the first day, 2k people landed on the Search Inside Suggestion card when previously they would have seen nothing. That’s 2,000 dead-end evasion attempts!
  • Of these 2,000 users, 60% clicked on the card to be led to Search Inside results.
  • 40% clicked on one of the suggested books with a matching quote.
  • ~8% clicked on the quote itself to be brought directly into the text.

I would’ve thought more people would click the quote itself but alas, there are only so many Comparative Literature majors in this world.

Follow-up and Next Steps

To complement the efforts of the Search Inside Suggestion card’s redirect to the Search Inside results page, I worked on re-designing the Search Inside results cards. My goal for the redesign was to make the card more compact and match its styling as closely as possible to the Search Inside Suggestion card to create a consistent UI.

Before:

After:

The next step for the Search Inside Suggestion card is to explore weaving it into the search results, regardless of result count. The card will offer an alternate search path in a list of potentially repetitive results. Say you searched ‘to be or not to be’ and there happens to be several books with a matching title. Rather than scrolling through these potentially irrelevant results, the search result card can intervene to anticipate that perhaps it’s a quote inside a text that you’re searching for. With the Search Inside Suggestion card taking the place of a dead-end, I’m proud to report that a search for “All grown-ups were once children…” will now lead Open Library patron’s to Antoine de Saint-Exupéry’s The Little Prince, page 174!

Technical Implementation

For the software engineers in the room who want a peek behind the curtain, working on the “Search Inside Suggestion Card” project was a great opportunity to learn how to asynchronously, lazy load “parts” of webpages, using an approach called partials. Because Search Inside results can take a while to generate, we decided to lazy load the Search Inside Suggestion Card, only after the regular search had completed.

If you’ve never heard of a partial, well I hadn’t either. Rather than waiting to fetch all the Search Inside matches to the user’s search before the user sees anything, a ‘No books directly matched your search’ message and a loading bar appear immediately. The loading bar indicates that Search Inside results are being checked, which is UX speak for this partial html template chunk is loading.

So how does a partial load? There’s a few key players:

  1. The template (html file) – this is the page that initially renders with the ‘No books directly matched your search’ message. It has a placeholder div for where the partial will be inserted.
  2. The partial (html file) – this is the Search Inside Suggestion Card
  3. The Javascript logic – this is the logic that says, “get that placeholder div from the template and attach it to an initialization function and call that function”
  4. More Javascript logic – this logic says, “ok, show that loading indicator while I make a call to the partials endpoint”
  5. A Python class – this is where the partials endpoint lives. When it’s called, it calls a model to send a fulltext search query to the database. This is where the user’s wrong turn is at last “corrected”. Their initial search in the Books tab that found no matching titles is now redirected to perform a Search Inside tab search to find matching quotes.
  6. The data returned from the Python class is sent back up the line and the data-infused partial is inserted in the template from step 1. Ta-da!

About the Open Library Fellowship Program

The Internet Archive’s Open Library Fellowship is a flexible, self-designed independent study which pairs volunteers with mentors to lead development of a high impact feature for OpenLibrary.org. Most fellowship programs last one to two months and are flexible, according to the preferences of contributors and availability of mentors. We typically choose fellows based on their exemplary and active participation, conduct, and performance within the Open Library community. The Open Library staff typically only accepts 1 or 2 fellows at a time to ensure participants receive plenty of support and mentor time. Occasionally, funding for fellowships is made possible through Google Summer of Code or Internet Archive Summer of Code & Design. If you’re interested in contributing as an Open Library Fellow and receiving mentorship, you can apply using this form or email openlibrary@archive.org for more information.

Improving Open Library’s Translation Pipeline

Posted on by

A forward by Drini Cami
Drini Cami here, Open Library staff developer. It’s my pleasure to introduce Rebecca Shoptaw, a 2024 Open Library Engineering Fellow, to the Open Library blog in her first blog post. Rebecca began volunteering with us a few months ago and has already made many great improvements to Open Library. I’ve had the honour of mentoring her during her fellowship, and I’ve been incredibly impressed by her work and her trajectory. Combining her technical competence, work ethic, always-ready positive attitude, and her organization and attention to detail, Rebecca has been an invaluable and rare contributor. I can rely on her to take a project, break it down, learn anything she needs to learn (and fast), and then run it to completion. All while staying positive and providing clear communication of what she’s working on and not dropping any details along the way.

In her short time here, she has also already taken a guidance role with other new contributors, improving our documentation and helping others get started. I don’t know how you found us, Rebecca, but I’m very glad you did!

And with that, I’ll pass it to Rebecca to speak about one of her first projects on Open Library: improving our translation/internationalization pipeline.

Improving Open Library’s Translation Pipeline

Picture this: you’re browsing around on a site, not a care in the world, and suddenly out of nowhere you are told you can “cliquez ici pour en savoir plus.” 

Maybe you know enough French to figure it out, maybe you throw it into Google Translate, maybe you can infer from the context, or maybe you just give up. In any of these cases, your experience of using the site just became that much less straightforward.

This is what the Open Library experience has been here and there for many non-English-speaking readers. All our translation is done by volunteers, so with over 300 site contributors and an average of 40 commits added to the codebase each week, there has typically been some delay between new text getting added to the site and that text being translated.

One major source of this delay was on the developer side of the translation process. To make translation of the site possible, the developers need to provide every translator with a list of all the phrases that will be visible to readers on-screen, such as the names of buttons (“Submit,” “Cancel,” “Log In”), the links in the site menu (“My Books,” “Collections,” “Advanced Search”), and the instructions for adding and editing books, covers, and authors. While updates to the visible text occur very frequently, the translation “template” which lists all the site’s visible phrases was previously only updated manually, a process that would  usually happen every 3-6 months. 

This meant that new text could sit on the site for weeks or months before our volunteer translators were able to access it for translation. There had to be a better way.

And there was! I’m happy to report that the Open Library codebase now automatically generates that template file every time a change is made, so translators no longer have to wait. But how does it work, and how did it all happen? Let’s get into some technical details.

How It Began

Back in February, one of the site’s translators requested an update to the template file so as to begin translating some of the new text. I’d done a little developer-side translation work on the site already, so I was assigned to the issue. 

I ran the script to generate the new file, and right away noticed two things:

  1. The process was very simple to run (a single command), and it ran very quickly.
  2. The update resulted in a 2,132-line change to the template file, which meant it had fallen very, very out of date.

I pointed this out to the issue’s lead, Drini, and he mentioned that there had been talk of finding a way to automate the process, but they hadn’t settled on the best way to do so.

I signed off and went to make some lunch, then ran back and suggested that the most efficient way to automate it would be to check whether each incoming change includes new/changed site text, and to run the script automatically if so. He liked the idea, so I wrote up a proposal for it, but nothing really came of it until:

The Hook

In March, Drini reached back out to me with an idea about a potentially simple way to do the automation. Whenever a developer submits a new change they would like to make to the code, we run a set of automatic tests, called “pre-commit hooks,” mostly to make sure that their submission does not contain any typos and would not cause any problems if integrated into the site. 

Since my automation idea had been to update the translation template each time a relevant change was made, Drini suggested that the most natural way to do that would be to add a quick template re-generation to the series of automated tests we already have.

The method seemed shockingly simple, so I went ahead and drafted an implementation of it. I tested it a few times on my own computer, found that it worked like a charm, and then submitted it, only to encounter:

The Infinite Loop of Failure

Here’s where things got interesting. The first version of the script simply generated a new template file whether or not the site’s text had actually been changed – this made the most sense since the process was so fast and if nothing actually had changed in the template, the developer wouldn’t notice a difference.

But strangely enough, even though my changes to the code didn’t include any new text, I was failing the check that I wrote! I delved into the code, did some more research into how these hooks work, and soon discovered the culprit. 

The process for a simple check and auto-fix usually works as follows:

  1. When the change comes in, the automated checks run; if the program notices that something is wrong (i.e. extra whitespace), it fixes any problems automatically if possible.
  2. If it doesn’t notice anything wrong and/or doesn’t make any changes, it will report a success and stop there. If it notices a problem, even if it already auto-fixed it, it will report a failure and run again to make sure its fix was successful.
  3. On the second run, if the automatic fix was successful, the program should not have to make any further changes, and will report a success. If the program does have to make further changes, or notices that there is still a problem, it will fail again and require human intervention to fix the problem.

This is the typical process for fixing small formatting errors that can easily be handled by an automation tool. But in this case, the script was running twice and reporting a failure both times.

By comparing the versions of the template, I discovered that the problem was very simple: the hook is designed, as described above, to report a failure and re-run if it has made any changes to the code. The template includes a timestamp that automatically lists when it was last updated down to the second. When running online, because more pre-commit checks are run than when running locally, pre-commit takes long enough that by the time it runs again, enough seconds have elapsed that it generates a new timestamp, causing it to notice a one-line difference between the current and previous templates (the timestamp itself), and so it fails again. I.e.:

  1. The changes come in, and the program auto-updates the translation template, including the timestamp.
  2. It notices that it has made a change (the timestamp and any new/changed phrases), so it reports a failure and runs again.
  3. The program auto-updates the translation template again, including the timestamp.
  4. It notices that it has made a change (the timestamp has changed), and reports a second failure.

And so on. An infinite loop of failure!

We could find no way to simply remove the timestamp from the template, so to get out of the infinite loop of failure, I ended up modifying the script so that it actually checks whether the incoming changes would affect the template before updating it. Basically, the script gathers up all the phrases in the current template and compares them to all the incoming phrases. If there is no difference, it does nothing and reports a success. If there is a difference, i.e. if the changes have added or changed the site’s text, it updates the template and reports a failure, so that now:

  1. The changes come in, and the program checks whether an auto-update of the template would have any effect on the phrases. 
  2. If there are no phrase changes, it decides not to update the template and reports a success. If there are phrase changes, it auto-updates the template, reports a failure and runs again.
  3. The program checks again whether an auto-update would have any effect, and this time it will not (since all the new phrases have been added), so it does not update the template or timestamp, and reports a success.

What it looks like locally:

A screen recording of the new translation script in action. A developer adds the word "Your" to the phrase "Delete Your Account" and submits the change. The automated tests run; the translation test fails, and updates the template. The developer submits the updated template change, and the automated tests run again and pass.

I also added a few other options to the script so that developers could run it manually if they chose, and could decide whether or not to see a list of all the files that the script found translatable phrases in.

The Rollout

To ensure we were getting as much of the site’s text translated as possible, I also proposed and oversaw a bulk formatting of a lot of the onscreen text which had previously not been findable by the template-updating function. The project was heroically taken on by Meredith (@merwhite11), who successfully updated the formatting for text across almost 100 separate files. I then did a full rewrite of the instructions for how to format text for translation, using the lessons we learned along the way.

When the translation automation project went live, I also wrote a new guide for developers so they would understand what to expect when the template-updating check ran, and answered various questions from newer developers re: how the process worked.

The next phase of the translation project involved using the same automated process we figured out to update the template to notify developers if their changes include text that isn’t correctly formatted for translation. Stef (@pidgezero-one) did a fantastic job making that a reality, and it has allowed us to properly internationalize upwards of 500 previously untranslatable phrases, which will make internationalization much easier to keep track of for future developers.

When I first updated the template file back in February of this year, it had not been updated since March of the previous year, about 11 months. The automation has now been live since May 1, and since then the template has already been auto-updated 35 times, or approximately every two to three days. 

While the Open Library translation process will never be perfect, I think we can be very hopeful that this automation project will make une grosse différence.

HOWTO: Design a Strategic Communications Plan

Posted on by

Hello, I’m Nick Norman – I volunteer as the Lead Communications Fellow at the Internet Archive’s Open Library.

The Open Library provides readers with free, digital access to millions of library books, but we can only help readers who know we exist. My mission is to cultivate an inclusive communications program that helps even the most remote readers discover and access our free services, learn about new features, and stay informed about the latest library news. In the service of achieving this goal, I’ve spent the past several months designing a strategic communications plan for OpenLibrary.org.

In this post I’ll share my personal story. I’ll also teach why effective communication is essential to helping patrons, explain how we’re structuring our communications strategy, and share what we’ve achieved so far.

For me, volunteering was always in the books

Open Library’s mission of being a free, public, open source catalog of books is personal to me because my parents were both librarians and my mother was a writer. Growing up, my library card was one of my most prized possessions and I seldom left home without it. In 2020, I took my library card digital by participating in an Open Library hackathon – a unique social event where volunteers of all backgrounds joined together for a few intense hours with the shared goal of improving the website together.

During this event, I felt supported by a passionate community of designers, engineers, and librarians. It was the Open Library community’s emphasis on inclusivity that truly resonated with me and motivated me to apply and become an Open Library Fellow. As someone who has felt personally marginalized and overlooked, the community’s commitment to equity and inclusion holds deep significance to me.

Empowered by these experiences, I’m proud to direct my expertise in communications to help bridge the gap between the many overlooked readers who need access to books and the bountiful books lining Open Library’s digital shelves.

The Importance of Communications

The greatest minds may develop the world’s greatest ideas and still fail to reach the minds of others. Some of Nikola Tesla‘s most brilliant inventions ended up being coopted by more effective communicators; his invention of Alternating Current was capitalized by Thomas Edison and his Tesla Coil, which proved the concept of wireless transmission, was popularized by Guglielmo Marconi who applied the technology to produce the iconic radio.

Since connecting with other humans through storytelling is such an important aspect of effective communication, here is a story of a renowned scientist named James Hutton, who in 1785 failed to excite those at the Royal Society of Edinburgh, despite making the groundbreaking (pun intended) discovery of tectonic plates. You can probably guess why based on this excerpt from Hutton’s writing recounted in Bill Bryson’s book, “A Short History of Nearly Everything”:

A practitioner may architect the grandest service with every bell and whistle, but if it doesn’t reach the right audience and get presented in a way that resonates with their needs, it’s unlikely to stick. It’s a sentiment perfect captured by Dr. George Berkeley in his proverbial question, “If a tree falls in a forest and no one is around to hear it, does it make a sound?”

So how might a project go about communicating a message effectively? What does effective communication look like and what does it achieve? At Open Library, we start with a desired outcome, like teaching patrons about a new feature, and then gain additional clarity by applying a framework to answer 5 Ws:

  • For Whom is the message intended (audience)?
  • Why is the message being broadcast (purpose)?
  • What is the strategy, tone, and approach for the message (content)?
  • When will the message be broadcast (timing)?
  • Where & how will the message be broadcast (distribution)?

Within each of these 5 W’s, there are opportunities for experimentation and testing to measure and improve success. Where did readers prefer to receive our messages — on twitter or our email distribution channel? Was broadcasting a message when it was 2pm better for readers than 5pm? Was what we sent out engaging to readers? For those who read the news, were we able to reach librarians or did we reach teachers instead? As we systematically break down communications into smaller problems and a framework for testing and improving success, we begin to see that it’s a lot of work. The more news there is to publish, the more features to announce, and the broader the audience, the more challenging it can be to effectively coordinate answers to each of these questions.

Communications Challenges at Open Library

Open Library’s growth over the past several years, driven both by the challenges of the COVID pandemic as well as a continuous torrent of new features, has led to increased demand by patrons looking for resources to guide their experience. The absence of a formal communications program has made it challenging to produce content at this large a volume and frequency — particularly articles for our blog. No one person could do it well and so our first objective has been to explore streamlined processes for coordinating the production and distribution of high quality content.

The production of each blog post involves multiple stages: ideation, research, graphics, writing, editing, promoting, and evaluating impact. With a small team, one could only produce a few quality articles a month. Scaling this process requires first having a formal process for producing a single blog post and then understanding where the bottlenecks are, which tasks may be parallelized, and developing effective training tools for a volunteer community where contributors are likely to come and go based on their busy schedules. In a scenario where multiple blog posts are being authored at once, we also needed a system for coordinating progress and needs across blog posts.

Solution: Our Communications Program

To address the challenge of not having a communications program in place, we initiated a period of planning and strategizing which has led to some amazing achievements. One notable accomplishment is the successful implementation of a dedicated Slack channel, serving as a space for our communications team to work together on a diverse range of projects. Additionally, our communications channel fosters cross-functional collaboration by allowing leads from within Open Library to join and contribute to the progress of intersecting projects.

Another significant accomplishment is the creation of our communications homebase, providing a central location for our communications efforts. Within the homebase, volunteers can access information about active projects and stay updated on the latest strategies being developed. Additionally, the homebase serves as a valuable resource for understanding our communication guidelines, community rituals that foster inclusion, and policies that shape our collaborative environment.
By establishing a dedicated communications homebase, we have successfully tackled the challenges associated with showcasing active communications projects and onboarding volunteers at varying engagement paces. The homebase also plays an important part in keeping everything organized, streamlining volunteer contributions into our Slack space and ensuring everyone’s impact is truly felt.

Another achievement by our communications team is the implementation of a bi-weekly communications video call. This gives us balance between working on projects in our Slack space and having real time check-ins to connect and share updates, provide support, and engage in discussions around intensive aspects of projects such as our blog program, a project that’s been our center focus.

Through the progress we’ve made with our blog program, we have defined processes and identified key roles to form a team of dedicated volunteers responsible for producing one blog post at a time. Each team will consist of a project lead, content writer & graphics, editor, and promoter. This structured approach not only allows us to scale up with additional teams as new volunteers join but also enables us to accommodate the expanding scope of future blog posts.

The ongoing development of the blog program has enabled us to establish two distinct pathways for volunteers to participate. Individuals can choose to fill a specific role in the production of blog posts or they join a blog testing team, which helps us to test and explore best approaches to expanding the program as needed. Here are a few links to documents pertaining to testing and the development of our blog program.

In the process of obtaining these learnings for our blog, our team of volunteers have also been able to advance Open Library’s communications efforts by creating 15 pilot episodes of an amazing unofficial community podcast. In doing so, we’ve achieved a better understanding of what it takes to run a podcast program where community members are empowered to help produce episodes.

Ways to Contribute

You can learn more about our progress with the podcast by visiting here, as we eagerly anticipate sharing an update in the future.

If you’re inspired by the work we’re doing in our communications program, we’d love for you to join us. You can do so by visiting our volunteer page or you can visit our fellowship page to learn more about contributing that way.

Personal Reflections

The process required for a single person to produce a blog post is very different from the systems and conventions required for a team to coordinate on producing content collaboratively. As a fellow on the communications team, I am quite proud to contribute to the mission of Open Library to bring online books to patrons and to serve as a champion for the idea that every reader should be included.

Reflecting on my journey from the time I first arrived at Open Library to where I am today, it feels like I have traversed two entirely different worlds. Each person I’ve encountered throughout my fellowship has played a significant role in contributing to expanding my professional skillsets and fueling my drive for promoting inclusion and accessibility in my work and daily interactions.

Work with Nick

While I intend to continue volunteering with the Open Library to further its communications program as a fellow, I am seeking career opportunities to support developing communications programs for communities, both online and offline. I love engaging with communities and making people feel welcomed. If your community would benefit from working with a communications specialist, please do not hesitate to contact me. I am also open to being contacted with opportunities to share my expertise through speaking, writing, or mentoring others in implementing similar communications programs.

Shoutouts

In Buddhist culture, “Dana” embodies the spirit of generosity. Mek Karpeles, the leader of the Open Library project, has exemplified this virtue by selflessly dedicating his time to help not only myself but also many others in their personal and professional growth. Just as Buddha teaches, I have gratefully received his gift with open hands and intend to share it freely with others.

I also want to thank Wendy Hanamura, the Director of Partnerships at Internet Archive, for providing me with valuable mentoring and foundational knowledge in communications prior to the start of my fellowship here at Open Library. It is a great privilege for me to have both Mek and Wendy as mentors in the same lifetime.

I would like to also acknowledge the valuable contributions from the members of our communications team over the past several months: Rachel Bayston, Urja Upadhyaya, Debbie San, Samuel Grunebaum, Crystal Mares, Roselle Oswalt, and many other volunteers who have played a crucial role in advancing our communications program from the planning phase to its actualization. Their feedback and contributions have greatly enhanced my learning and professional development.

Finally, I want to express my deepest appreciation to the many staff members and community leads across the Internet Archive and Open Library and volunteers who have provided their support to our communications program. Here are some of those names: Lisa S., Chris Freeland, Drini Cami, Jim Champ. Mark H., Brenton Cheng.

A Brand New My Books Experience

Posted on by

Have you noticed our new My Books page experience?
Please let us know what you think!

Read on to learn more about what’s changed.

Forward by Mek

This year we’ve had the great fortune of collaborating with Samuel Grunebaum, 2022 Open Library Design & Engineering Fellow. Samuel is wonderfully positioned at the cross-section of software engineering, design, and education, making him capable of rapidly prototyping new designs, ensuring these designs are clear and instructional, and bringing these designs to life through engineering. It’s rare I find someone, like Sam, who can so easily and effectively switch contexts between design and engineering while also keeping bigger product pictures in mind. This set of skills has not only made Sam essential to early prototyping stages, we have also benefited greatly from Sam’s ability to uniquely recognize and raise challenges about component accessibility and mobile/desktop compatibility we likely would have otherwise missed. In addition to being a 2022 Open Library Fellow, Sam co-directs a software & design consultancy that is accepting new freelance projects, tutors college, high school, and intro-level computer science and is accepting new students, and is open to the right mission-aligned, full-time role.

Personal Intro

Hello, I’m Samuel Grunebaum and I’ve been working with Open Library as a Design & Engineering Fellow, contributing designs and code to the My Books redesign process. I’m currently transitioning into a career in software design and front end engineering after working as a Computer Science educator at the Horace Mann School in The Bronx, New York and as a freelance designer, developer, and teacher in all things software.

Problem

Open Library patrons and stakeholders alike identified the My Books page as a major pain point in the site’s navigation and information hierarchy. At the beginning of the project the desktop interface loaded by clicking the ‘My Books’ button in the header looked like this:

Perhaps the most confusing issue with this flow, is that the ‘My Books’ button brought patrons to their account’s Loans page. Another problem that was continually observed with the existing design is the mobile navigation on this page:

The mobile design took the desktop sidebar menu and added it directly below the site header, creating three layers of navigation and a very confusing split in the My Books page interface. 

The central problem of the existing design for My Books was that there was no true My Books page, but rather a My Books button driving to the Loans page. This meant that there was no single place for patrons to find their books, whether books they had on loan or books they had added to one of their reading logs. 

The previous menu design also had the unfortunate side effect of burying the Reading Log options, as well as Reading Stats, Notes, Reviews, and Import/Export options deep below the fold in an already confusing mobile menu. 

Digging deeper into this problem and hearing from Open Library stakeholders, patrons, librarians, and the community at large clarified the direction we would take: working towards a mobile-first consolidated My Books interface that gives patrons an understandable and discoverable way to access the books associated with their account, as well as the other account specific sections including Stats, Notes/Reviews, and Lists.

Interim Fix

Before beginning the design process in earnest, we decided an interim fix would be helpful to mitigate the confusing nature of the mobile My Books navigation. The solution we decided on was an extremely quick fix, able to be implemented in just a few minutes. 

The adjustments made used a darker color to differentiate the menu from the rest of the header and the page content, as well as making the menu section smaller with an adjustable height click-and-drag feature:

Approach

After meeting with members of the Open Library team to discuss the main issues, we agreed that the central areas to focus on were:

  1. Creating an interface unique to My Books that would consolidate a patron’s loans and logs into one page accessed by the My Books button
  2. Improving the usability of the My Books page on mobile by moving towards a responsive, mobile-first design for the My Books page and redesigning the menu on mobile

The next step was to iterate multiple potential interfaces for the new My Books page on both mobile and desktop. With the help of the Open Library team and other design fellows, we came up with the following options for mobile and desktop My Books interfaces:

In conjunction with Dana, another 2022 Design Fellow, we continued to iterate on the designs based on feedback received from Open Library stakeholders, librarians, and patrons. 

We settled on the following approach for desktop, which includes new carousel sections for displaying books and creates space for a Reading Stats data visualization widget:

Alongside the new desktop design, the mobile-first redesign that we settled on makes use of the existing sidebar menu to guide the structure of the new mobile interface while making use of an information hierarchy already familiar to patrons. 

This mobile design not only improves usability and accessibility to the key components of My Books, but also decreases engineering overhead by allowing for a responsive design using the original sidebar menu:

You can try the new mobile design prototype here.

First Phase Release: Desktop My Books Interface

The first and most immediate step to improving the overall My Books experience was to create a distinct page on the site for an account’s books. The new My Books page now exists at /account/books for any patron of Open Library. The new page is almost a carbon copy of the original My Books page, but with its own content that is separate from the Loans page, which was originally what came up when clicking My Books in the header. Below is an image of the current desktop release of My Books:

After adding the new page at /account/books, I created a new file for the My Books content and added custom carousels to the page for displaying a patron’s loans and reading logs. Beyond improving the flow of clicking the My Books button, this initial redesign aims to increase user engagement with reading logs and will also populate the My Books page with images of book covers from a patron’s own selections, creating a more welcoming and dynamic account page driving from the My Books header button.

Another change in this release is the addition of Reading Stats and Import/Export Options buttons at the top of the page, as these are currently buried at the bottom of the My Books sidebar. In the next phase of the design, there will be a prominent link to Stats at the top of the My Books page with the addition of the data visualization widget.

The creation of this page not only provides an elegant interim solution to the issue of confusing My Books navigation by adding a novel page to the Open Library but also lays the groundwork for a more comprehensive redesign including mobile-first improvements, multiple new custom components, and more prominently featured a patron’s Reading Stats. 

Next Steps: Mobile Release and Stats

The next steps of the My Books redesign process will begin with improving the mobile usability by overhauling the My Books interface. The engineering approach I will take is to hide the My Books index content on mobile, instead only displaying the sidebar menu as the whole My Books interface. The sidebar will be responsively designed to display custom carousels on mobile, as rendered in this interactive Figma prototype.

The next and final step after completing the implementation of the mobile redesign will be to finish the redesigned desktop My Books interface by building the Reading Stats data visualization widget. Here’s an interactive Figma prototype of this new design.

I am so excited to continue working on this redesign process, which has already been a wonderful introduction to the Open Library design system and code base. Moreover, I’m excited to contribute to what is hopefully a welcome improvement to the Open Library ecosystem, increasing both access and usability to some really wonderful account features. My hope is that patrons will be able to more easily save and read books on Open Library once they have one clearly defined place to look at all of their books, whether checked out or saved in reading logs, as well as their reading stats and account information.

Reflections

Working on this project with the Open Library community has been an amazing experience in UX design, full stack web development, and community collaboration across state and national lines. I am grateful to be able to contribute to a project that is so meaningful to so many people through its unique ability to disseminate knowledge freely to anyone with Internet access. It was also a fun way to expand my web design and development experience. 

I am immensely grateful to the Open Library community as a whole for being so welcoming to me when I joined a few months ago and for continuously supporting my design process through helpful critiques and design input, as well as the general kindness shown in the weekly community meetings. I am especially grateful to Mek, my counter-point on Open Library staff, who has taught me so much about the Internet Archive stack and the Open Library design language, and my main collaborator Dana, who has expertly taken the reins on the Desktop interface designs and navigation for the overall site. I also want to extend my thanks to Drini, Lisa, Jim, Abby, and Hayoon who have all had invaluable contributions to the My Books design and implementation process, as well as the ongoing development of a comprehensive Open Library design system. I’m so excited to continue working with this community and for the completion of the My Books redesign. 

Search for Books in your Reading Log

Posted on by

by Scott Barnes, 2022 Open Library Fellow

As of last week’s deploy, it’s now possible to search the Open Library for the books in your reading log by navigating to the My Books page, selecting the Currently Reading, Want to Read, or Already Read bookshelf, and typing in a search query.

Keep reading to learn tips and tricks on how to effectively search for books within your reading log.

A Forward by Mek

This year the Open Library has been exceedingly lucky to collaborate with Scott Barnes, a lawyer who has reinvented himself as a very capable software engineer. We had the pleasure of meeting Scott earlier this year while he was scouring the Open Library for old rock climbing guidebooks. Ever since joining one of our community calls, he’s been surmounting challenging technical hurdles as one of our most active 2022 Open Library Fellows. As law professor Lawrence Lessig famously penned in his 1999 book Code: “Code is law”. I guess that’s why we shouldn’t be too surprised how quickly Scott became familiar with the Open Library codebase, at the precision of his work and attention to detail, and his persistence in getting code just right without getting slowed down. We hope you’ll enjoy Scott’s contributions as much as we do and learn at least one new way of using the reading log search to improve your book finding experience.

A Forward by Drini
I’m exceedingly pleased to introduce Scott Barnes to the Open Library Blog. I have had the honour of mentoring Scott throughout some of the projects on his fellowship, and have been floored by his love, passion, and skill in all things programming. Whether it’s working on user facing features (such as this one), improving code architecture, investigating performance issues, setting up infrastructure, or keeping up-to-date with new programming techniques by diving into a new programming book or topic, Scott is always excited to dive in, learn, and make an impact. And his code never fails to meet requirements while being well-architected and robust. I am so excited to see what he does next with his programming super powers! Because as far as I can tell, there’s no stopping him. Now without further ado, I’ll hand it off to Scott to talk about:

👋 Hi, my name is Scott Barnes. This year as a 2022 Open Library Fellow I collaborated with Drini Cami to develop reading log search. In this post, I’ll show different ways of effectively using the new reading log search feature, as well as technical insight into how it was engineered behind the scenes.

Profile photo for scottreidbarnes

3 Ways of Searching your Reading Log using a Web Browser

The most common way to use search your reading log is by entering a natural, free-form search query, just like you might using your favorite search engine. From the Currently Reading, Want to Read, or Already Read page, you can search for your books on that reading log shelf by submitting text describing the book’s title, author name, ISBN, or publisher. An example could be “Lord of the Rings by J.R.R Tolkien”.

If you want greater control, you can also harness the power of Apache Solr, the underlying technology which powers the Open Library search engine.

Let’s say, for example, that you’d like to find books on your reading log by a specific author named “King” but using the Natural Language mode instead returns books with “King” in the title. Using keyword search, you could search for author: king to see only books by authors named “King” (while not seeing books with “King” in the title). On the other hand, if you only want to find titles matching “King” and not the author, you could instead search for the keyword title: king. Want to find your horror books? Try subject:Horror.

For a list of common keywords, you can see the Open Library Search How-To Guide, or peek at the code behind it: worksearch/code.py.

Reading log search, like the main Open Library search, supports boolean operators, specifically AND, OR, and NOT, along with wildcards such as * and ? to match multiple characters and a single character respectively. Therefore, to search for all books matching “climb”, “climber’s”, “climbs”, etc., that were published by Sierra Club Books that you want to read, you could visit Want to Read and search for title: climb* AND publisher: sierra club books.

NOTE: the boolean operators are CaSe sensitive, so AND will work as expected, but and will not. The actual search terms themselves are not CaSe sensitive, however, so “king” and “KiNg” will return the same results.

Searching your reading log via the API

For those looking to perform programmatic searches, it’s possible to search via a web API using the following RESTful pattern: https://openlibrary.org/people/{USERNAME}/books/{SHELF}.json?q={QUERY}.

For example, to search for all titles matching “king” on my Want to Read shelf I’d query: https://openlibrary.org/people/srb36/books/want-to-read.json?q=king.

Or if I wanted to search for all titles matching climb* on that same shelf, I’d search for https://openlibrary.org/people/srb36/books/want-to-read.json?q=title:(climb*)

Behind the scenes

Now for the technical details! Reading log search was added in pull request #7052. In exploring how reading log search might be accomplished, two key things leapt to our attention:

  1. Reading log records are stored in the database, and work and edition data (i.e. “books”) are stored in Solr; and
  2. To work with the split data, we were probably doing more queries than we needed to, both in the back end itself, and within the templates that make up the pages.

The goal then was to add the ability to search the reading log, ideally while reducing the number of queries, or at least not increasing the number.

Changing the back end

Most of the heavy lifting was done in core/bookshelves.py. The challenge here was addressing the reading log records being in one database, and the edition data being in Solr.

The solution was to query the reading log database once, and then use those results to query Solr to get all the information we’d need for the rest of the process. Then we could simply pass the data around in a Python dataclasses, and then ultimately pass the results through to the templates to render for display in patrons’ browsers.

For the super curious, this is found in get_filtered_reading_log_books()

Changing the templates

As mentioned, having the data in two places had led to some excess querying, which manifested itself in the templates where we re-queried Solr to get additional data to properly display books to patrons.

However, because we had gathered all of that information at the outset, we just had to change the templates to render the query results we passed to them, as they no longer had to perform any queries.

How you can help

Volunteers not only help make Open Library special, but they help make it even more awesome. Check out Volunteering @ Open Library.