This summer, continuing a years-long tradition, Open Library and the Internet Archive took part in Google Summer of Code (GSoC), a Google initiative focused on bringing new contributors into open source software development. This year, I was lucky enough to mentor Sandy, a long-time Open Library volunteer, on an exciting project to increase the accessibility of our books with real-time translations. We have invited Sandy to speak about her experience here as we reach the culmination of the GSoC period. It was a pleasure getting to work on this exciting project with you Sandy! – Drini

My name is Sandy Chu and I am a 2025 Google Summer of Code (GSoC) candidate who had the opportunity to work with the amazing Internet Archive engineering team. Prior to participating in the GSoC program, I had contributed as a volunteer software engineer for the Open Library open source repo. As someone who grew up using local libraries as a place to supplement my education and read books that my school could not afford, I was drawn to the Open Library’s mission to empower book lovers and provide a free, valuable resource to all. You can view my initial proposal here.

Coming soon in September, the Open Library will be able to better serve its global audience to access books that were previously not available due to a lack of localization. With the help of open source projects such as the Mozilla Firefox Translation Models and Bergamot Translator library, a new BookReader plugin will have the ability to leverage a user’s browser and hardware resources to toggle translations from a book’s original language to a translation in their language. Additionally, the translated text will also work with the ReadAloud feature to read books in the translated language.

The “Real-Time In-Browser Book Translation w/ Read Aloud (TTS)” project closely aligns with the Open Library’s 2025 goal of providing more with less. Although the Internet Archive hosts and provides its patrons with hundreds of thousands of publicly available works, patrons are limited to a subset of works that were published in their native language. Due to the unique image based implementation of the BookReader application, default browser translator options are not viable for many readers, so this project presents an opportunity to make a big impact for international audiences.

Currently in internal beta, the translation plugin allows patrons to quickly initiate a local translator on-their device and translate the book’s text in just a few seconds per page. With nine distinct languages available for translation from English (and potentially over 40 as we update to Mozilla’s latest models), this project will make countless works more accessible for patrons.

The primary goals of this project were:

• Translating a book’s original text content to the patron’s desired language with minimal delay or disruption
• Creating a visually seamless experience to maintain the immersive experience of reading a book without having to go back and forth between a translator and the book
• Redirecting the existing TTS plugin to use the translated text when the BookReader is in translation mode

Retrieved from Wikipedia, which references Ethnologue as its source. Chinese and Arabic dialects are grouped together since they both have a unified written system.

Translations

At the center of the translation plugin are the Neural Machine Translation (NMTs) models provided by Mozilla Foundation’s Firefox Translation Models project. These files contain the lexical and vocabulary conversions from the original language to the target language; these compact models are essential to the real-time, browser-side aspect of this project. Since we are currently using an older subset of models, the translation feature is still considered in the “alpha” stage of maturity and accuracy.

When the translation plugin is enabled by the user, the language registry and model files are fetched from a server within the Internet Archive. After the models have successfully loaded into the user’s browser, we are able to use the Bergamot Translator project scripts to create a dedicated Web Worker, which is initialized to handle the translation tasks in a separate background thread. The Web Worker immediately retrieves the text content within the text selection layer for the currently visible page(s) for translation. Pages that have been rendered but not visible in the BookReader are given a lower priority and translated after the queue of visible content is completed. 

The translation plugin script feeds the text within the text selection layer into the model for processing and prepares the stage for the translated output by covering the original image and text selection layer with a beige background. [Pull Request #1410]

Once the translation is completed by the dedicated Web Worker, the output is then used as the text content for its respective paragraph chunk and appears as if the work is actually written in the target translation language.

Images with captions are also carefully handled so that the translated text box occupies nearly the same space as the text selection layer itself. 

Each translated paragraph is stored in a cache with a unique key to prevent the browser from re-translating recently viewed content [Pull Request #1410/commit]. To prevent readers from having to wait for the translation when “flipping” to the previous/next page, the translation plugin targets the visible pages on the screen then works to complete the translations for the non-visible but loaded pages. If a user decides to flip far from their current page in the work, the translation plugin will detect the newly rendered page and translate / populate the translated text layer while adjusting to a new page on the fly.

Fine-tuning the visual presentation and behind-the-scenes functionality of the plugin were the main challenges for this portion of the project. Ensuring that the translations for each text chunk were done without depending on a previous chunk was an essential behavior we identified in the early stages of the project. Both asynchronous and synchronous behavior is implemented within the code to ensure that users do not have to wait for longer than needed for paragraphs to complete their translations. The translation plugin utilizes event listeners within the BookReader to detect when a newly rendered text layer is created, which then triggers a translation call to the text content from the upcoming page.

Styling the translated text layer also proved to be difficult. Although it is possible to reuse the style properties on the existing (and invisible) selection text layer, additional adjustments were needed to ensure that the visible translation text would not overlap or go beyond the bounds of the original paragraph. In the early phases of the translation plugin development, there were many instances of text chunks exceeding the boundaries set for the translation layer, which resulted in scrollbars appearing within paragraph elements or not aligning properly with the text on the page.

Another styling issue that caught us off guard was a pre-existing bug that was only visible in the Chrome browser. Since Drini and I were both using Firefox as our default browser, we later learned during a demo that there was an element scaling issue that was immediately visible when the translation plugin was activated. [Pull Request #1421/commit]

ReadAloud

The next major piece of this project was to connect the translation plugin to the ReadAloud feature and allow users to hear the translated text read aloud. 

The normal flow of the TTS (Text-to-Speech) plugin calls a server-side API to retrieve chunks of text and bounding rectangles based on the page and paragraph index. However, since we have the translated text available within the BookReader locally, the extra network calls to the server were dropped in favor of feeding the translated text lines into the TTS engine directly. Tweaking the pre-existing functionality of the TTS plugin to interact with the content generated by the translate plugin required a substantial amount of investigation to figure out where the adjustments needed to be made for the translation plugin to gracefully take over. 

When the TTS plugin is activated, it checks whether or not the translation plugin is enabled within the BookReader. If the translation plugin is active, the TTS plugin retrieves the translated text on the page to use as its text input for the voice engine. 

Voice overs are also automatically adjusted as soon as the TTS begins to streamline the reading process. By checking the source language from the work’s metadata, the default voice of the TTS reader is automatically adjusted to the target language that was set within the translate plugin. The voice menu is also re-rendered to allow users to more easily switch between the source, target, or other languages for the TTS reader. [Pull Request #1430/commit]

Visual parity between the original TTS and translated TTS was maintained as well by highlighting the entire translated paragraph section. Since network info containing the bounding rectangles for a text chunk were no longer available, I was able to use a paragraph element’s offset properties to highlight the text being actively read by the TTS reader [Pull Request #1431/commit].

Although this stage of the project did not require as much new code, we encountered a relatively complex issue that would cause the TTS reader to not progress if the translation plugin is activated in a part of a book that contained one or more blank pages. The translation adjusted implementation of the TTS plugin would wait for a new page to be loaded and rendered within the browser but remain stuck on a page due to a synchronization issue. After two weeks of extensive investigation and testing, we were able to resolve the issue by utilizing an existing method that returns all pages that have been loaded but not rendered in the DOM yet [Pull Request #1431/commit] and consolidating the asynchronous translation call with Promise.all().

Next Steps

For now, this feature is currently scheduled to be released for internal testing before being released for full public use. While the majority of goals were completed within the project timespan, there are many additional improvements and expansions that are planned in the future as the BookReader’s translate plugin becomes more mature. The next major steps for this project involve expanding the number of available translation pairs by integrating the latest models from Mozilla’s Translation Model project, receiving and implementing feedback from a round of internal testing, and continually improving the UI of the plugin. Unit tests and offline testing environments are also part of the project’s future goals to help improve the troubleshooting process for developers.

Conclusion

I would like to express my thanks once again to my GSoC mentor Drini for his guidance. The first few weeks of this project felt especially daunting, but the patience and advice that I was given throughout this program helped me realize that this big intimidating project was easier to manage as a number of small tasks were taken step-by-step. I am very glad that I had the chance to be challenged in new ways while being able to leverage my existing JavaScript skills. 

I am extremely grateful that I was able to participate in the GSoC program and to help contribute to a high-impact feature for both the Internet Archive and Open Library. Though my time as a GSoC contributor has officially ended, I intend to continue my work as a contributor with the Open Library team to expand on the functionality of this feature and help increase the availability of published works to a wider global community.