Search for Books in your Reading Log

by Scott Barnes, 2022 Open Library Fellow

As of last week’s deploy, it’s now possible to search the Open Library for the books in your reading log by navigating to the My Books page, selecting the Currently Reading, Want to Read, or Already Read bookshelf, and typing in a search query.

Keep reading to learn tips and tricks on how to effectively search for books within your reading log.

A Forward by Mek

This year the Open Library has been exceedingly lucky to collaborate with Scott Barnes, a lawyer who has reinvented himself as a very capable software engineer. We had the pleasure of meeting Scott earlier this year while he was scouring the Open Library for old rock climbing guidebooks. Ever since joining one of our community calls, he’s been surmounting challenging technical hurdles as one of our most active 2022 Open Library Fellows. As law professor Lawrence Lessig famously penned in his 1999 book Code: “Code is law”. I guess that’s why we shouldn’t be too surprised how quickly Scott became familiar with the Open Library codebase, at the precision of his work and attention to detail, and his persistence in getting code just right without getting slowed down. We hope you’ll enjoy Scott’s contributions as much as we do and learn at least one new way of using the reading log search to improve your book finding experience.

A Forward by Drini
I’m exceedingly pleased to introduce Scott Barnes to the Open Library Blog. I have had the honour of mentoring Scott throughout some of the projects on his fellowship, and have been floored by his love, passion, and skill in all things programming. Whether it’s working on user facing features (such as this one), improving code architecture, investigating performance issues, setting up infrastructure, or keeping up-to-date with new programming techniques by diving into a new programming book or topic, Scott is always excited to dive in, learn, and make an impact. And his code never fails to meet requirements while being well-architected and robust. I am so excited to see what he does next with his programming super powers! Because as far as I can tell, there’s no stopping him. Now without further ado, I’ll hand it off to Scott to talk about:

👋 Hi, my name is Scott Barnes. This year as a 2022 Open Library Fellow I collaborated with Drini Cami to develop reading log search. In this post, I’ll show different ways of effectively using the new reading log search feature, as well as technical insight into how it was engineered behind the scenes.

Profile photo for scottreidbarnes

3 Ways of Searching your Reading Log using a Web Browser

The most common way to use search your reading log is by entering a natural, free-form search query, just like you might using your favorite search engine. From the Currently Reading, Want to Read, or Already Read page, you can search for your books on that reading log shelf by submitting text describing the book’s title, author name, ISBN, or publisher. An example could be “Lord of the Rings by J.R.R Tolkien”.

If you want greater control, you can also harness the power of Apache Solr, the underlying technology which powers the Open Library search engine.

Let’s say, for example, that you’d like to find books on your reading log by a specific author named “King” but using the Natural Language mode instead returns books with “King” in the title. Using keyword search, you could search for author: king to see only books by authors named “King” (while not seeing books with “King” in the title). On the other hand, if you only want to find titles matching “King” and not the author, you could instead search for the keyword title: king. Want to find your horror books? Try subject:Horror.

For a list of common keywords, you can see the Open Library Search How-To Guide, or peek at the code behind it: worksearch/code.py.

Reading log search, like the main Open Library search, supports boolean operators, specifically AND, OR, and NOT, along with wildcards such as * and ? to match multiple characters and a single character respectively. Therefore, to search for all books matching “climb”, “climber’s”, “climbs”, etc., that were published by Sierra Club Books that you want to read, you could visit Want to Read and search for title: climb* AND publisher: sierra club books.

NOTE: the boolean operators are CaSe sensitive, so AND will work as expected, but and will not. The actual search terms themselves are not CaSe sensitive, however, so “king” and “KiNg” will return the same results.

Searching your reading log via the API

For those looking to perform programmatic searches, it’s possible to search via a web API using the following RESTful pattern: https://openlibrary.org/people/{USERNAME}/books/{SHELF}.json?q={QUERY}.

For example, to search for all titles matching “king” on my Want to Read shelf I’d query: https://openlibrary.org/people/srb36/books/want-to-read.json?q=king.

Or if I wanted to search for all titles matching climb* on that same shelf, I’d search for https://openlibrary.org/people/srb36/books/want-to-read.json?q=title:(climb*)

Behind the scenes

Now for the technical details! Reading log search was added in pull request #7052. In exploring how reading log search might be accomplished, two key things leapt to our attention:

  1. Reading log records are stored in the database, and work and edition data (i.e. “books”) are stored in Solr; and
  2. To work with the split data, we were probably doing more queries than we needed to, both in the back end itself, and within the templates that make up the pages.

The goal then was to add the ability to search the reading log, ideally while reducing the number of queries, or at least not increasing the number.

Changing the back end

Most of the heavy lifting was done in core/bookshelves.py. The challenge here was addressing the reading log records being in one database, and the edition data being in Solr.

The solution was to query the reading log database once, and then use those results to query Solr to get all the information we’d need for the rest of the process. Then we could simply pass the data around in a Python dataclasses, and then ultimately pass the results through to the templates to render for display in patrons’ browsers.

For the super curious, this is found in get_filtered_reading_log_books()

Changing the templates

As mentioned, having the data in two places had led to some excess querying, which manifested itself in the templates where we re-queried Solr to get additional data to properly display books to patrons.

However, because we had gathered all of that information at the outset, we just had to change the templates to render the query results we passed to them, as they no longer had to perform any queries.

How you can help

Volunteers not only help make Open Library special, but they help make it even more awesome. Check out Volunteering @ Open Library.

Improvements to the Main Navigation

Dana, UX/Design Fellow on Open Library

Forward by Mek

Few aspects of a website have greater impact and receive less recognition than good navigation. If done well, a site’s main navigation is almost invisible: it’s there when patrons need it, out of the way when they don’t, and it zips patrons to the right place without making them overthink. Over the years, we’ve attempted improvements to our navigation but we haven’t had the design bandwidth to conduct user research and verify that our changes solved our patrons problem. It turns out, we still had several opportunities for improvement. That’s why this year we were incredible lucky to have collaborated with Open Library UX Design Fellow Dana Fein-Schaffer, who recently transitioned into design from a previous role as a neuropsychological researcher. Dana’s formal education and experience links a trifecta of complimentary fields — computer science, psychology, and design — and has resulted in unique perspective as we’ve endeavored to redesign several of Open Library’s core experiences: the website’s main navigation and the desktop version of our My Books page.

As an Open Library UX Design Fellow, Dana has been in charge of design direction, conducting user interviews, figma mockups, feedback sessions, and communicating decisions to stakeholders. In addition to her skill prototyping and notable problem solving capabilities, Dana’s warmth with the community, affinity for collaboration, and enthusiasm for the project has made teaming up with her a gift. While we’d rather the fellowship not end, we endorse her work with great enthusiasm and highly recommend organizations which share our values to view Dana’s portfolio and engage her for future design opportunities.

The Design Process

IMG_7214.JPG

Hello, I’m Dana Fein-Schaffer, and I’ve been working as a UX Design Fellow with Open Library over the past several months. I’m currently transitioning into UX Design because after gaining professional experience in both psychology research and software engineering, I’ve realized that UX is the perfect blend of my skills and interests. I’ve been enjoying growing my UX skillset, and working with Open Library has been a perfect opportunity for me to gain some formal experience because I love reading and was hoping to work on a book-related project. Moving forward, I’m particularly excited about working as a UX designer for an organization that focuses on social good, especially in the literary, education, or healthcare spaces! If you have a role that may be a good fit, or if you work in one of those industries and are interested in connecting, please feel free to reach out. You can also learn about me from my portfolio.

Problem

At the beginning of my project, the current navigation bar and hamburger menu looked like this: 

I met with members of the Open Library team to identify three key areas of concern: 

  1. The label “more” was not descriptive, and it was unclear to patrons what this meant
  2. The hamburger menu was not consistent with the navigation bar items
  3. The hamburger menu was confusing for patrons, especially new patrons, to navigate

Approach

Before I began my project, the Open Library team decided to implement an interim solution to the first point above. To address the concern with the “more” label, the navigation menu was changed to instead include “My Books” and “Browse.” The website analytics showed that patrons frequent their Loans page most, so for now, the “My Books” page brings patrons to the Loans page. 

To begin redesigning the main site navigation, I first created a prototype in Figma with some potential solutions that built off of this updated navigation menu:

  1. I created a dropdown menu for “My Books” that would allow patrons to select the specific page they would like to go to, rather than automatically going to the Loans page
  1. I reorganized the hamburger menu to be consistent with the navigation menu and to use the subheadings of “Contribute” and “Resources” instead of “More.” I felt that these changes would make the hamburger menu easier to navigate for both new and long-time patrons. 

After creating the prototype, my next goal was to get feedback from patrons, so I scheduled user interviews with volunteers.

User Interviews

I conducted user interviews via Zoom with four patrons to answer the following questions: 

  1. How do users feel about the My Books dropdown? 
  2. Are users using My Books and Browse from the navigation menu or hamburger menu?
  3. Are users able to effectively use the hamburger? Do they find it easier or harder to find what they’re looking for using the reorganized hamburger?

Results & Findings

  1. Three out of the four patrons preferred the dropdown, and the fourth user didn’t have a preference between the versions. The patrons enjoyed having the control to navigate to a specific section of My Books. 
  2. All users used the navigation menu at the top of the page to navigate, rather than the hamburger menu, which supported the switch to “My Books” instead of “More.” This finding also highlighted the need to make sure patrons could access the pages they wanted to access from the top navigation menu. 
  3. Finally, all four patrons found the existing hamburger menu confusing and preferred the reorganized hamburger. Some patrons specifically mentioned that the reorganized hamburger was more compact and that they felt that the headings “contribute” and “resources” were more clear than “more.”

Synthesis and a New Direction

After learning from the user interviews that patrons preferred increased granularity for accessing My Books and a more concise hamburger menu, the Open Library team began discussing the exact implementation. We wanted to keep the navigation and hamburger menus consistent; however, we also wanted to provide many options in the My Books dropdown, which made the hamburger menu less concise. 

At the same time that we were debating the pros and cons of various solutions, another Open Library UX Design fellow, Samuel G, was working on designing a My Books landing page for the mobile site. Inspired by his designs, I created mockups for a desktop version of the My Books page. Having a My Books page that summarizes patrons’ loans, holds, and reading log at a glance allows patrons to still have increased control over My Books navigation while keeping the hamburger menu concise, since all of the individual My Books items can now be condensed into one link. 

Furthermore, having a My Books landing page opens the door for more ways for patrons to interact with their reading through Open Library. For instance, I’ve created a mockup that includes a summary of a patron’s reading stats and yearly reading goal at the top of the page. 

As we work towards implementing this design, I’m looking forward to getting feedback from patrons and brainstorming even more ways to maximize the use of this page. 

Reflections

Working with the Open Library team has been an amazing experience. I’m so grateful that I got to lead a UX project from start to end, beginning with user research and ending with final designs that are ready to be implemented. Working with such a supportive team has allowed me to learn more about the iterative design process, get comfortable with sharing and critiquing my designs, and gain more experience with design tools, such as Figma. It was also a great learning experience that sometimes your projects will take an unexpected turn, but those turns help you eventually come to the best possible design solution. Thank you to everyone who provided feedback and helped me along the way, especially Mek, who was a wonderful mentor, and Sam, who was a great collaborator on our My Books mobile and desktop project!

About the Open Library Fellowship Program

The Internet Archive’s Open Library Fellowship is a flexible, self-designed independent study which pairs volunteers with mentors to lead development of a high impact feature for OpenLibrary.org. Most fellowship programs last one to two months and are flexible, according to the preferences of contributors and availability of mentors. We typically choose fellows based on their exemplary and active participation, conduct, and performance within the Open Library community. The Open Library staff typically only accepts 1 or 2 fellows at a time to ensure participants receive plenty of support and mentor time. Occasionally, funding for fellowships is made possible through Google Summer of Code or Internet Archive Summer of Code & Design. If you’re interested in contributing as an Open Library Fellow and receiving mentorship, you can apply using this form or email openlibrary@archive.org for more information.

Internet Archive Summer of Design 2022

Forward by Mek

For several years and with much gratitude, the Internet Archive has participated in Google’s Summer of Code (GSoC). GSoC is a program run by Google that supports select open source projects, like the Open Library, by generously funding students to intern with them for a summer. Participation in GSoC is as selective of organizations as it is for students and so in years when GSoC is not available to the Internet Archive, we try to fund our own Internet Archive Summer of Code (IASoC) paid fellowship opportunity.

GSoC and IASoC are traditionally limited to software engineering candidates which has meant that engineering contributions on Open Library have often outpaced its design. This year, to help us take steps towards righting this balance, an exceedingly generous donor (who wishes to remain anonymous but who is no less greatly appreciated) funded our first ever Internet Archive Summer of Design fellowship which was awarded to Hayoon Choi, a senior design student at CMU. In this post, we’re so excited to introduce you to Hayoon, showcase the impact she’s made with the Open Library team through her design work this summer, and show how her contributions are helping lay the groundwork to enable future designers to make impact on the Open Library project!

Introducing Hayoon Choi

Profile photo for hayoonc

Hello, my name is Hayoon Choi and this summer I worked as a UX designer with Open Library as part of the Internet Archive Summer of Code & Design fellowship program. I am a senior attending Carnegie Mellon University, majoring in Communication Design and minoring in HCI. I’m interested in learning more about creative storytelling and finding ways to incorporate motion design and interactions into digital designs. 

Problem

When I first joined the Open Library team, the team was facing three design challenges:

  1. There was no precedent or environment for rapidly prototyping designs
  2. There wasn’t a living design system, just an outdated & static design pattern library
  3. The website didn’t display well on mobile devices, which represents and important contingency of patrons.

Approach

In order to solve these challenges, I was asked to lead two important tasks:

  1. Create a digital mockup of the existing book page (deskop and mobile) to enable rapid prototyping
  2. A propose a redesign of the book page optimized for mobile.

To achieve the first task, I studied the current design of the Open Library Book Page and prototyped the current layout for both mobile and desktop using Figma. In the process, I made sure every element of that Figma file is easily editable so that in the future, designers and developers can explore with the design without having to code.

For the second task, we first scoped our work by setting our focus to be the set of content which appears above the fold — that is, the content which first loads and appears within the limited viewport of a mobile device. We wanted to make sure that when the page initially loads, our patrons are satisfied with the experience they receive.

Even before conducting interviews with patrons, there were easily identifiable design issues with the current mobile presentation:

  • Information hierarchy: some texts were too big; certain information took up too much space; placement of the book information were hard to discover
  • Not mobile friendly: Some images were shown too small on images; it was hard to scroll through the related books; one feature included hovering, which is not available on mobile devices

To address these concerns, I worked with the Open Library community to receive feedback and designed dozens of iterations of the mobile book page using Figma. Based on feedback I learned about the most necessary information to be presented above-the-fold, I choose to experiment with 6 elements:

  1. The primary Call To Action (CTA) buttons: how do I make them more highlighted?
  2. The Navigation Bar: which placement and styling are most convenient and effective?
  3. The Editions Table: how might we make it easier for patrons to discover which other book editions and languages may be available?
  4. Ratings & reviews: how do I encourage users to rate more and help them understand the book effectively with the review system?
  5. Sharing: how do I make it easier for users to share the book?
  6. The Information Hierarchy: how can we reorder content to better meet the diverse needs of our audience?

From these questions and feedback from the Open Library team, I was able to settle on five designs which seemed like effective possibilities for showcasing differences in book cover size, sharing buttons, information display, and rating and reviewing system which we wanted to test:

User Interviews & Mazes

With these five designs selected, I planned on running multivariate user testings to get feedback from actual users and to understand how I can more effectively make improvements to the design. 

I believed that I would gather more participants if the user testing was done remotely since it would put less pressure on them. However, I wasn’t sure how I would do this until I discovered a tool called Maze.

Maze provides a way for patrons to interact with Figma mockups, complete certain tasks, answer questions, and leave feedback. While this is happening, Maze can record video sessions, keep track of where patrons are clicking, and provide valuable data about success rates on different tasks. I felt this service could be extremely useful and fitting for this project; therefore I went ahead and introduced Maze to the Open Library’s team. Thanks to a generous 3-month free partner coupon offered by Maze, I was able to create six Maze projects — one for each of our five new designs, as well as our current design as a control for our experiment. Each of these six links were connected to a banner that appeared across the Open Library website for a week. Each time the website was reloaded, the banner randomized the presented link so participants would be evenly distributed among the six Maze projects.

Although the Maze projects showed patrons different mobile screens, they enabled comparisons of functionality by asking patrons to answer the same pool of 7 questions and tasks:

  1. What was the color of the borrow button (after showing them the screen for five seconds)
  2. What key information is missing from this screen (while showing the above-the-fold screen)
  3. Share and rate this book
  4. Borrow the Spanish edition for this book
  5. Try to open a Spanish edition
  6. Review this book
  7. Try to open the preview of this book

In between these tasks, the participants were asked to rate how challenging these tasks were and to write their feelings or opinions.

In addition to Maze, which we hoped would help us scale our survey to reach a high volume of diverse participants, we also conducted two digital person-to-person user interviews over Zoom to get more in depth understanding about how patrons approach challenging tasks. Because Maze can only encode flows we program directly, these “in person” interviews gave us the ability to intervene and learn more when patrons became confused.

Results & Findings

After around a week of releasing the Maze links on the website, we were able to get a total of 760 participants providing feedback on our existing and proposed designs. Maze provided us with useful synthesis about how long it took participants to complete tasks and showed a heat map of where patrons were clicking (correctly or incorrectly) on their screens. These features were helpful when evaluating which designs would better serve our patrons. Here’s a list of findings I gathered from Maze:

The Sharing Feature:

Results suggest that the V1 design was most clear to patrons for the task of sharing the book. It was surprising to learn patrons, on average, spent the most time completing the task on this same design. Some patrons provided feedback which challenged our initial expectations about what they wanted to accomplish, reporting that they were opposed to sharing a book or that their preferred social network was not included in the list of options.

Giving a book a Star Rating:

One common reaction for all designs was that people expected that clicking on the book’s star ratings summary would take them to a screen where they could rate the book. It was surprising and revealing to learn that many patrons didn’t know how to rate books on our current book page design!

Leaving a Community Review

When participants were asked to leave a community review, some scrolled all the way down the screen instead of using the review navigation link which was placed above the fold. In design V4, using a Tag 🏷️ icon for a review button confused many people who didn’t understand the relationship between book reviews and topic tags. In addition, the designs which tested combining community review tags and star ratings under a single “review” button were not effective at supporting patrons in the tasks of rating or reviewing books.

Borrowing Other Editions

Many of our new designs featured a new Read button with a not-yet-implemented drop down button. While it was not our intention, we found many people clicked the unimplemented borrow drop down with the expectation that this would let them switch between other available book editions, such as those in different languages. This task also taught us that a book page navigation bar at the top of the design was most effective at supporting patrons through this task. However, after successfully clicking the correct navigation button, patrons had a difficult time using the provided experience to find an borrow a Spanish edition within the editions table. Some patrons expected more obvious visual cues or a filtering system to more easily distinguish between available editions in different languages.

Synthesis

By synthesizing feedback across internal stakeholders, user interviews, and results from our six mazes, we arrived at a design proposal which provides patrons with several advantages over today’s existing design:

  • First and foremost, redesigned navigation at the very top of the book page
  • A prominent title & author section which showcases the book’s star ratings and invites the patron to share the book.
  • A large, clear book cover to orient patrons.
  • An actionable section which features a primary call to action of “Borrow”, a “Preview” link, and a visually de-emphasized “Want to Read” button. Tertiary options are provided for reviewing the book and jotting notes.
  • Below the fold, proposals for a re-designed experience for leaving reviews and browsing other editions.
(Before)
(After)

Reflections

I had a great time working with Open Library and learning more about the UX field. I enjoyed the process of identifying problems, iterating, and familiarizing myself with new tools. Throughout my fellowship, I got great feedback and support from everyone from the team, especially my mentor Mek. He helped me plan an efficient schedule while creating a comfortable working environment. Overall, I truly enjoyed my working experience here and I hope my design works will get to help patrons in the future!

About the Open Library Fellowship Program

The Internet Archive’s Open Library Fellowship is a flexible, self-designed independent study which pairs volunteers with mentors to lead development of a high impact feature for OpenLibrary.org. Most fellowship programs last one to two months and are flexible, according to the preferences of contributors and availability of mentors. We typically choose fellows based on their exemplary and active participation, conduct, and performance within the Open Library community. The Open Library staff typically only accepts 1 or 2 fellows at a time to ensure participants receive plenty of support and mentor time. Occasionally, funding for fellowships is made possible through Google Summer of Code or Internet Archive Summer of Code & Design. If you’re interested in contributing as an Open Library Fellow and receiving mentorship, you can apply using this form or email openlibrary@archive.org for more information.

It takes a Classroom to build an Open Library

On most days, the Open Library is hard at work improving the experience it offers to students and teachers in classrooms. But for the past few months, Open Library has had the privilege of enjoying contributions from 7 students around the globe who had been assigned by their universities to participate on open source software projects.

First and foremost, the entire Open Library community extends our deep gratitude to AUEB / Athens University of Economics and Business‘s Dr. Diomidis Spinellis (professor of Software Engineering, who taught the course Software Engineering in Practice) and NYU‘s Dr. Joanna Klukowska (Clinical Associate Professor of Computer Science, who taught the course CSCI-UA.0480-061, Open Source Software Development) for incorporating open source contributions into the curriculum of their classrooms. As we hope you’ll see, the decision to promote hands-on development has an outsized impact on supporting open source projects like ours.

In the spring semester of 2022, four students from Greece’s AUEB (Constantina Z., Vassilis B., Dimitris B., and Philippos P. / Φιλιππος Π.) and three students from NYU (Michelle T., Crystal C., Chloe Q.) spent time participating in community calls, problem solving, and improving the Open Library service for the public. In return they received mentorship and first-hand experience learning how to contribute to a platform trusted by millions of international readers.

This year, the foci of Open Library’s roadmap is improving core experiences for patrons. Towards this goal, each of these students exceeded our expectations by contributing meaningful improvements like: Chinese internationalization of the website, google analytics to help inform us on meaningful ways to improve the organization of the website, fixing broken mobile navigation for our Books Page, UI improvements for sharing books on social media, adding APIs for Trending Books, and much more. We’re extremely proud of and grateful for the work these students were able to contribute.

In the past, Open Library has reserved a special honorary title of “Open Library Fellow” for exemplary contributors who have demonstrated exceptional commitment, leadership, and impact with the Open Library project. Our list of previous Fellows include Sabreen Parveen (who designed our onboarding experience), Yash Saravgi (who developed our mobile Progressive Web App), and Bharat Kalluri (who helped standardize our import pipelines). Each dedicated several months implementing features which redefined core behaviors and experiences of the Open Library.

This year, we believe one student in particular, Constantina Zouni, stands out as being especially deserving of this special Fellowship distinction, for her initiative, participation in engineering and design process with stakeholders, and outstanding work ethic.

Please join us in celebrating the work of this 2022 international student cohort, sharing our gratitude, and congratulating Constantina on her inspiring example.

Improving Experiences for Open Library Patrons

By Constantina Zouni

As this semester of my studies is coming to an end, I want to do a retrospect about my experience with the open library project.

My Journey with Open Library

In the beginning of the semester my professor Dr. Diomidis Spinellis for the course “Software Engineering in Practice” announced that in the context of an assignment we had to choose an open source project to make contributions thought out the semester. As a result, I started searching for a project and I was lucky to quickly find open library’s repository. Some of the main reason that made me to choose that project is that the community was very friendly and really open to contributors. The documentation of the project was really detailed and there were videos that helped me understand how the project works. Also, another good thing was that the issues of the project were well organized with labels and the context was explanatory enough. Moreover, the project seemed to be very active with quick responses in the comments section and pull request merges almost every day. After the first communication with the team everything went very smoothly. I was welcomed in the slack channel, and I was invited to participate in the weekly meetings. Mek quickly stepped in and helped me to get started. Because that period was busy and contributors from other universities also chose to contribute to open library the project’s team made effort to create a GitHub project and assign issues to everyone. I started solving minor issues related with text appearing when not needed, adding the subtitle to the search results and some UI improvements. Ultimately, in collaboration with another student from my university Vassilis Bubis we created the twitter social card that enables users to share their book lists. Through out the whole period that I contributed to the project I was impressed that Mek and the other members of the open library team dedicated time answering our messages and even jumping on small zoom meetings.

Book page header in mobile

One of the issues that I think had a big impact in the open library users is the improvement of the book page header in the mobile environment. When users visited a book page from mobile the experience wasn’t that pleasant. The book title and other important information like the author, the subtitle etc didn’t fit in the phone screen and the user had to scroll down to see them. The issue was more significant in the cases where the book covers were ambiguous, and it made difficult for the user to understand if they were in the correct page. Jim Champ recommended to follow a specific layout for the book page in mobile in order to fit all the important information in the mobile page. The challenge was the layout had to be different depending on the device of the user. My first implementation involved some java script code that change the order of the elements and an event listener that was activated when the screen had a specific size. The open library team quickly informed me that this implementation was causing a delay in the loading of the page, and they recommended me to use HTML and CSS. This time with a new implementation and the help of Jim Champ who was reviewing my pull request I managed to solve the issue using an HTML file that included only the title summary and some CSS commands.

Book header in mobile before and after

Dynamic book list preview for sharing

This new feature was a little more challenging than the previous one. This time I collaborated with Vassilis Bubis in order to create a dynamic preview for the book lists of the users that displays the first 5 books of the list. Then this preview is passed to the twitter social card and every time a user wants to share a list with the URL the preview image appears. This is a more interactive way for users to show their book lists to others and makes open library more recognizable across twitter users. The first challenge was to create a mock-up of the preview. To achieve that I used a design tool called Figma to create prototypes with different colour combinations and I let the open library team to decide which on they like more. For the design I used colours from the open library’s webpage, and I added a twist in the preview that represents a self where the books are placed. Alongside with the mock-ups Vassilis worked on retrieving the book covers that we need and place them above the background with the help of a Python library called Pillow. Then I stepped in, and I made sure that every book cover was resized in a way proportional to the original dimensions that it had. We noticed that some covers were stretching so it was important that every time we changed the width of a cover the height was adjusted properly. Another challenge was the text that we wanted to add in the preview. The text had to change dynamically, and we had to change line every time the characters exceeded a specific number to achieve an aesthetically pleasant result. One issue that we faced was that the coordinates of the covers that we had figured out with Figma had to change because in python the coordinates are applied from upper left corner compared to Figma that apply to the center of an image. After solving that Vassilis and I proceeded on storing the image in an in-memory binary array for better performance and finally creating the API for the list page.

Twitter social card for book lists sharing

Book page editing improvement

While working on some issues in the book page I realised that compared to other library webpages open library gives users the ability to edit the details and the information of a book. That feature is very valuable because users can add important details for a book that were missing when it was added, they can update that information and they can add descriptions and subjects that might be useful for other users. Although this feature is really important the editing user interface is not that pleasant. When users click on the edit button, they are directed to another page. My recommendation regarding that is to use a modal that pops up when the button is clicked. In that way users will feel like they have more control because they won’t be directed to another page, and they can still see the book page behind the modal. Another issue with the existing editing form is that users can discard the changes with the cancel button, but they can’t undo a change without deleting all the changes. In the mock-up that I created I added an arrow in the right upper side that symbolizes the undo action. I noticed that the examples for every field were placed next to the field title, and I opted to move them inside the text box for a clearer look. Finally, I added the info symbols beside every field that provides details on how you should fill out that specific field. Overall, the purpose of those recommendations is to make the booking editing more simple, compact and user friendly.

Book editing page now
Edit book page with modal created with Figma

A High Schooler’s Experience Contributing to the Open Book Genome Project

Meet Teo Cheng, a high school student who has been volunteering to lead development on the Open Book Genome Project. In 2021, Teo took Harvard’s online CS50 Intro to Computer Science to prepare for his AP Computer Science Principles exam. The following summer, he took MIT’s Introduction to Computer Science and Programming course. To put these learnings into practice and gain more hands-on experience, he searched for impactful opportunities within the non-profit Internet Archive, where his father Brenton Cheng runs the UX team. For the past year, Teo has been working closely with Mek, making improvements to the Open Book Genome Project Sequencer — a software robot that reads the Internet Archive’s publicly available books and derives public insights to enable greater access to their themes and data. Meet Teo!

Goals

This year, by joining the Open Book Genome Project team, I hoped to understand a piece of production software well enough to make meaningful contributions. Also, because this project may someday be run on every book digitized by the Internet Archive, I wanted to gain experience contributing to something which needs to have a high level of accuracy and runtime performance. When I joined the project, I learned of several problems. For example, the book sequencer module, which is responsible for deriving ngrams, was noisy and wasn’t honoring the defined stop words. Also, the page type detection would frequently break because it was too strict and wasn’t robust against OCR errors, punctuation, and variety in syntax. Furthermore, because I already have experience programming, I was interested in learning more about the engineering development process, such as using tools like git, writing tests, and running pipelines.

What I’ve Learned

So far while working on the Open Book Genome Project (OBGP) I’ve gained experience with the following 10 things: I learned how to use docker to install a project in a contained way without having to mess up my computer’s file system. I used ssh to run the OBGP pipeline on a more powerful remote computer. Because the internet connection could be disrupted, we did our work using a program called tmux which ensured our processes would continue running even if the connection between the client and server died. This remote computer ran Linux and so I needed to learn basic BASH commands. I also needed to learn about XML and JSON formats, and how those are used in the results of our pipeline. We used bash commands and regex (e.g. grep) to analyze the pipeline results, such as to extract URL counts from books. Some bash commands I used to discover link counts are: for loop, grep, variables, cat, wc. I worked on improving the existing OBGP Sequencer, so I had to learn how to read through and understand a new codebase. To submit our code changes, we used the git protocol and managed our tasks on GitHub.

Accomplishments

In addition to learning a lot throughout working on the Open Book Genome Project, I’ve accomplished a few different things. I noticed the issue with the Page Type Detector, which I solved. My improvement to the detector involved allowing regex patterns in addition to exact text matches. I also improved the ISBN detector to reduce false positives, which were happening pretty commonly. Lastly, I solved the bug with the stop words that get removed from the ngrams to make them less noisy and more useful. I also added more stop words to decrease the amount of clutter in the ngram results.

How it Works

As a developer on the Open Book Genome Project, here’s an inside look at what it’s like when staff members run the Sequencer on Internet Archive’s books:

  1. Set up the project using the Docker instructions
  2. On Archive.org, identify a search query which returns the books we want to sequence
  3. Create an AdvancedSearch query which returns identifiers for these books in JSON
  4. Reformat the results from this query and feed it into the Sequencer pipeline

Here’s an example of a completed book_genome.json created by this process.

Want to try it yourself?

You can add your own processing modules too! If you’d like to try out the Open Book Genome Project Sequencer using just your browser, you can try it using the OBGP google colab.

Learn More

Want to learn more about the Open Book Genome Project? Check out the official bookgenomeproject.org website, Open Library’s announcement of the project, and learn about the work of Nolan Windham who previously led development on the project as a high school senior and incoming college freshman as part of Google Summer of Code.

Want to contribute?

Come volunteer to be an Open Library or Open Book Genome Project fellow!