A forward by Mek, Open Library’s program lead:
This year, the Open Library team and I were fortunate to collaborate with Chisom, as part of Google Summer of Code, to make millions of books more discoverable to readers. Chisom entered this year’s Google Summer of Code program a motivated and capable software developer and continued to impress us with her focus, proactivity, and problem solving. It was a joy working with Chisom and rewarding to witness her make consistent forward progress, rise to growth opportunities, and — as a result of hard work and initiative — achieve an excellent outcome that I believe will benefit millions of Open Library patrons. I encourage you explore how she strategically approached this challenge and created a general purpose tool and patterns that will allow others to continue her work into the future.
My name is Chisom Nnamani, and this summer I had the opportunity to join the Internet Archive’s Open Library team as a Google Summer of Code (GSoC) contributor. This was my first experience contributing to a large open-source project, and I could not have asked for a better place to start. As someone who cares deeply about making books accessible to everyone, I was drawn to Open Library’s mission of providing a free webpage where anyone, anywhere can discover and access published works. You can view my initial proposal here.
My GSoC project focused on a problem that sounds simple at first: helping readers find books by genre.
Millions of books available, the challenge is discovery
Over the years, Open Library has accumulated millions of free-form book labels that have never gone through a standardization process. As a result, a book about science fiction might be labeled “science fiction,” “science-fiction,” “sci-fi,” “scifi,” or dozens of other variations. Without a way to recognize and merge these synonyms, books become scattered across hundreds of different labels like a needle in a hundred haystacks.
| Search term | Results returned | Books missed vs. best |
| “Science Fiction” | 17,900 hits | — (baseline) |
| “science-fiction” | 16,497 hits | ~1,421 books |
| “sci-Fi” | 2,721 hits | ~15,179 books |
The result shows how a reader can miss hundreds or even thousands of books simply by using a different term to describe the same genre. The books are there. The problem is that the catalogue does not always connect them.
Messy labeling means messy recommendations
The problem goes deeper than the search box. For years, Open Library has used a subjects field to describe what a book is about. These subjects are stored as plain text strings in a flat list, with no consistent rules about how they should be written or organized.
As a result, different kinds of information can end up sitting side by side. A book can have its genre, characters, places, themes, and other descriptions all represented as separate subject strings. There is no structure telling the catalogue how these descriptions relate to one another.
The screenshots below show what this looks like on real book pages. The Hobbit has “Fantasy,” “Fantasy fiction,” and “Juvenile fantasy fiction” as separate tags on the same page:
Figure 1: The Hobbit’s subjects include multiple variants of “fantasy” with no major classification.
The screenshot below shows another example: And Then Were None has over 30 subject tags with no structure distinguishing genre, subgenre, language, audience, character, and other types of information. You can also see multiple variations of “mystery,” including “Mystery fiction,” “Mystery & Detective,” and “Fiction, mystery & detective, general.”
Figure 2: And Then Were None — 30+ subjects with no type distinction between genre, subgenre, language, audience, character.
Together, these examples revealed the larger problem I wanted to address. Open Library had a huge amount of useful information about its books, but it lacked a consistent structure for connecting related genres and descriptions.
That became the starting point for my GSoC project: building a more structured way for Open Library to describe books by genre and subgenre, and eventually using that structure to make browsing and discovery better for readers.
The recipe for organizing 860K books
When I began GSoC, the Open Library team had already identified several high-impact label categories (tag types) that could benefit from this kind of cleanup, including genres and subgenres, audiences, content warnings, and formats. What we didn’t yet have were mappings from our existing messy labels to these new, cleaner categories, or a common software framework that contributors could use to define these mappings and perform the cleanup.
I began by working with genres and subgenres. The first step was to define the categories we wanted to recognize. We then needed to connect the many ways these concepts already appeared in Open Library’s catalogue to a consistent set of canonical labels – the standardized labels we want those variations to map to. For example, different subject descriptions might refer to the same genre using slightly different wording or formatting.
From there, I worked on expanding and refining the mappings so that more of the catalogue’s existing subject descriptions could be connected to the appropriate genres and subgenres.
But the goal was not to build something that only worked for genres. As the project evolved, we built a common core that could support different types of labels through the same process: define a vocabulary, create mappings, analyze existing data, and eventually migrate the cleaned information back into Open Library.
This separation between the tag type and the shared tooling became an important part of the project. Genres and subgenres were the first categories I worked on, but the same framework can be used for other categories as contributors begin cleaning and structuring them.
Here’s what that mapping looks like for a few genres:
| Existing subject strings on Open Library | Canonical genre |
| “Fantasy”, “fantasy fiction”, “Juvenile fantasy fiction” | Fantasy |
| “Mystery fiction,” “Mystery & Detective,” “Fiction, mystery & detective, general” | Mystery |
| “Science fiction,” “science-fiction,” “sci-fi,” “Science Fiction Literature” | Science Fiction |
Several different subject strings can now point to the same canonical genre.
Once these mappings were in place, the next step was to give each canonical label a structured representation in Open Library. In the Tags project, a Tag is an Open Library data object representing a defined label, such as a genre or subgenre. This gives the canonical concept its own consistent identity instead of treating every variation of a subject string as a separate concept.
I then created the canonical Tags for the genres and subgenres we had defined.
Fantasy, now represented as a genre Tag in Open Library, rather a plaintext subject string.
Steampunk, a subgenre represented as a Tag.
With the genre and subgenre labels defined and represented as Tags in Open Library, the next challenge was connecting them to the millions of existing works in the catalogue.
Open Library already had millions of works with years of existing metadata. I could not simply assign these new Tags manually to every work. The next challenge was figuring out how to connect the information that was already there to this new structure, and then safely apply those connections across the catalogue.
So I built migration tooling that could analyze existing subjects, identify matches using the genre and subgenre mappings, and connect those matches to the appropriate Tag keys on each work.
Before thinking about millions of records, I first needed to understand what the migration would actually find. I ran the matching process against Open Library’s April data dump, a monthly snapshot of the catalogue’s data, and found 869,461 works with genre matches and 51,526 works with subgenre matches.
Those numbers changed the way I thought about the project. This was no longer just about creating a better vocabulary. It was about applying that vocabulary across millions of works while making sure the information already there was not accidentally changed or lost.
I worked on the migration scripts, the shared utilities behind them, and the changes to Open Library’s work schema needed to store the new genre information. I also validated the migration on a smaller pilot before moving toward the production run.
Safely running a large-scale migration
One of my biggest lessons from this project was that writing the code is only part of the job. When you are changing a large, live system, you have to think about what happens when the code actually runs.
- What happens if something fails halfway through?
- How do you know the migration did what you expected?
- How do you avoid changing records that should not be changed?
- How do you test an operation that will eventually touch hundreds of thousands of works?
I used dry runs and small pilots before larger operations. I added ways to track progress and designed the migration so that it could be run in controlled batches. Along the way, I also encountered some of the less glamorous parts of working with production systems, from authentication and request limits to unexpected differences in the data itself.
One of my favourite lessons from the project is that production engineering requires trust.
Before you can make a change at scale, you have to earn the right to trust your own tools.
Translating better data into better discovery
In our GSoC project, fixing book labels was always a means to an end: improving how readers discover books. Many patrons come to the Open Library looking for a specific book. But not every reader arrives knowing exactly what they want to read next. With consistent genre data in place, we could begin to ask a different question:
What if readers could browse and discover books by genre instead of having to already have a book in mind?
Search and filtering can help when you already know what you are looking for. But discovery is different. Sometimes you just want to browse.
That question became the idea behind Genre Explorer.
Taking inspiration from Drini Cami’s Library Explorer – a system that uses Dewy Decimal classification numbers to digitally emulate the organized bookshelves of a physical library – we imagined a more visual way for readers to explore the Open Library’s book catalogue, using the same genre and subgenre structure I was building for the tagging project. Instead of presenting genres as another long list of links, we imagined something closer to the experience of walking into a bookstore.
Genres could act as bookcases.
Subgenres could become shelves.
A reader could choose a genre, step inside it, explore its subgenres, and discover books along the way. The idea was to make genre browsing feel less like searching through metadata and more like browsing a library. I developed the initial concept and built a clickable prototype to explore how this experience could work.
From there, Mek and I continued developing the idea together. We reviewed the experience, explored how it could fit with Open Library’s existing components, and refined the concept into an interactive version now available on the testing site.The interactive version follows the same idea: genres act as bookcases, and entering a genre reveals its subgenres as shelves.
What I find most exciting about Genre Explorer is that it grew out of the original tagging problem, but takes the idea one step further. The canonical Tags give Open Library a consistent way to describe books. That structure can support better search and filtering, while Genre Explorer explores what it could look like when the same information is used to help readers browse.
It was an unexpected direction for my GSoC project. I came in focused on the data and infrastructure behind genre information. Along the way, I started thinking beyond how books are described to how that work could become something a reader actually experiences.
Takeaways
When I started this project, I expected to learn more about software engineering. I did, but not always in the ways I expected.
One of my biggest lessons was learning to slow down and understand a system before trying to change it. I learned to look at messy data and find the patterns hidden inside it, to test my assumptions against real examples, and to treat small experiments as part of the engineering process rather than as steps before the “real” work begins.
I also learned to think beyond the implementation. Throughout the project, I kept coming back to a simple question: Does this actually make the experience better for the person using it? That question shaped how I thought about the tagging system, and eventually led to the idea of Genre Explorer. It reminded me that good engineering is not only about building something that works. It’s also about understanding if and why something should exist in the first place.
Working with Open Library also gave me my first real experience contributing to a large open-source project. I had to learn how to navigate an unfamiliar codebase, communicate ideas clearly, ask questions when I was unsure, respond to feedback, and make decisions when there was no obvious answer. I was not doing this work in isolation. My mentor, Mek, pushed me to think beyond the code and focus on the larger problem we were trying to solve. Open Library contributors and maintainers, including Jim, Drini, Liz, and Katrina helped me understand different parts of the systems I was working with. Every review, discussion, and debugging session became part of the learning process.
Looking back, I think that may be one of the most valuable things I am taking away from GSoC: learning how to become useful in a system that existed long before I arrived.
Next steps
By the end of GSoC, we were able to add genre tags to more than 50,000 works. We also built common infrastructure to standardize the tag migration process and enable others to contribute to the greater cleanup process.
The next stage is to extend this process to add subgenre tags to works and to index these genre and subgenre tags in Open Library’s search engine, so readers can find and explore books by genre.
Genres and subgenres are only the beginning. Open Library has other high-impact label categories that will benefit from the same approach, including audiences, moods, content warnings, and content formats. Because we built this project to have a shared core for defining vocabularies, creating mappings, analyzing existing data, and migrating cleaned information, future contributors can use the same tooling to work on these categories rather than building a new system from scratch.
This shared core is the legacy I hope will last beyond this GSoC project: not just cleaner genre and subgenre data, but a reusable foundation that makes it easier for Open Library and its contributors to continue turning messy catalogue labels into structured information that can improve how readers discover books.
