How Duke Students Are Teaching Computers to Read Four-Hundred-Year-Old Handwriting
It started with a computer screen left open at the right time.
Adriane Fresh, Assistant Research Professor at Duke’s Social Science Research Institute, had been talking with colleague Nick Eubank, Associate Research Professor at the same institute, about a collection of documents she had spent years gathering: port books from sixteenth and seventeenth century England. These handwritten ledgers recorded the ships, cargoes, taxes, and merchants that moved through English ports during one of the most turbulent economic periods in the country’s history. Fresh had been describing how difficult the records were to work with when Eubank happened to have one of them pulled up on his computer.
Astrid Giugni, a computational humanist, walked by and saw it on the screen. According to Fresh, Giugni was immediately curious about what she was looking at, and Eubank explained that the documents were part of a larger collection Fresh had been gathering and trying to make sense of.
That exchange turned into a meeting, and the meeting turned into a partnership. As Fresh described it, Giugni suggested that the documents “seem like they would be great fodder for a Data+ project, because it’s something really concrete and part of something bigger.”
That idea became a summer project now underway with the Data+ program, which brings together history, economics, and artificial intelligence to answer a question that has occupied historians for generations: when the world changes rapidly around them, how do those in power hold on to it?
A Puzzle Written in Ink
Fresh has spent years studying how economic upheaval during this period affected who held political power and who didn’t, but the port books that could help answer those questions have remained largely untouched by researchers. Why? They are handwritten, inconsistent, and scattered across archives. Until recently, there was no practical way to read them at scale.
That’s where machine vision comes in. Over the summer, a small team of students has been building a process that trains computer models to recognize centuries old handwriting, extract the text, and turn it into usable data. The goal isn’t just to answer questions about English trade, but to build tools that other historians can use on their own collections of old, handwritten records.
Three undergraduates, Faisal Alghamdi, Sebastian Estrada, and Alan Lu, are doing most of the work, guided by Lauren Hoaglund, a graduate student in the English Department who serves as the team’s mentor, along with regular input from Fresh and Giugni. Fresh described a kind of layered structure, with the undergraduates driving the daily work, Giugni meeting with them weekly and staying available in between, and Fresh checking in on the bigger picture.
“The three undergrads are doing ninety percent of the work, and they are incredibly self-directed,” Fresh said. “Each day, they know exactly what they want to do and accomplish, and they are setting out to do it… When something doesn’t go well, they try something different. They come back with the right level of question, where we’re like, oh yeah, you’ve hit a snag that you really need our help with, let’s talk through it, let’s jump on a Zoom, let’s put that on our agenda for the next meeting.”
Harder Than It Looks
Reading typed historical text with a computer is, at this point, fairly routine. Once the characters are recognized, the text can be fed into a language model and analyzed in all sorts of ways. But handwriting is a different problem entirely.
“That first step of how we get the computer to recognize that these are characters, this is text, is still really difficult,” Fresh said. “People say, oh, machine learning can do it all. Not quite.”
One challenge the team didn’t expect going in was teaching the models where the text was located on the page. A modern printed document is white with text running down the center in tidy lines. A four-hundred-year-old ledger is nothing like that. Margins wander, ink fades unevenly, and the aging of the paper itself creates shadows that can look like writing where there is none.
“A standard, modern sheet of paper is white. It’s text almost always right down the middle in lines and paragraphs,” Fresh said. “And these old books, there’s random indentations, and there’s also the aging of the paper itself, which can throw shadows, making it look like it could be text. So that’s part of their pipeline too, is to segment the document to find the text. Then, once the text areas are identified, to run the model to identify what the text is.”
Giugni told Fresh at the outset not to expect the students to reach amazing performance on these documents by the end of the summer since it was such a tight timeframe for a difficult project. The handwriting hasn’t been transcribed with near perfect accuracy. That said, Fresh said the students have well exceeded what she thought they would achieve. They built a public repository that is, in her words, “really thorough.” They have not only compared performance across models but have refined and engineered the prompts used to run them, working to iterate and improve the results along the way.
Speeding Up a Slow Process
Astrid Giugni, Lecturing Fellow in the Social Science Research Institute, has spent seven years working with economic and colonial documents, and has run into the same problem again and again: reading manuscript sources takes an enormous amount of time, with no guarantee of a payoff.
“I can spend months working through an archive only to find out the information I have been looking for is not there,” Giugni said. She sees this project as a way to speed that process up. Using what researchers call multi modal large language models, tools that can take in both images and text, the team can assess the usefulness of thousands of pages at once. The models are still far from perfect and require extensive human supervision, she said, but the results so far have been promising. The team has built a process that quickly produces a rough first transcription for a test set of economic documents, and they plan to keep refining it and begin analyzing those AI aided transcriptions soon.
Giugni explains how the role of AI in this kind of research is usually described. The common standard, she said, is a “human in the loop” process, where human oversight is built into the AI workflow. In her view, that framing sells human judgment short.
“For research in the humanities and interpretive social sciences, it’s more realistic to talk about AI in the loop, where AI is carefully directed to time consuming tasks by researchers,” Giugni said. Current models, she explained, lack judgment when it comes to research tasks. Even with careful prompting, they cannot decide what is relevant or useful in work that is simple but slow. What they are good at, she said, is performing tasks at scale once a person has pointed them in the right direction. Rather than replacing a researcher’s evaluation and assessment, the models can become efficient aides for specialized but repetitive work, such as transcribing thousands of documents.
Finding the Right Combination
The students began the summer by testing several commercial language models alongside open-source alternatives. None of them could produce transcriptions accurate enough to be useful for analysis. It was through that testing process that the team landed on something more promising: a workflow that combines several tools rather than relying on just one.
The approach starts with the images themselves. Undergraduate Alan Lu focused on this part of the work, processing the scanned port book pages to improve their readability and cutting down on visual noise before any transcription was attempted. From there, the team used an open-source tool called Kraken to segment each manuscript page, identifying exactly where the text sits and setting aside parts of the image that are not text at all, such as the spine of the book or a stray ink blot. That step breaks a page down into a clean set of lines, which can then be handed off to a commercial model, in this case Gemini, for the actual transcription.
To bring these steps together, undergraduate Sebastian Estrada built the workflow and code behind a tool the team calls ScriptBench, which lets a user combine each of these steps, regardless of which underlying model is doing the work, with very little manual effort. The project is publicly available on GitHub.
Testing any of this required something to test against, which is where Faisal Alghamdi came in. Alghamdi, a student from Purdue University and KAUST Academy, worked on creating the ground truth transcriptions the team used to train and test their models. The team also credited the Folger Shakespeare Library for giving them access to its own collection of manually transcribed manuscripts, which the students used to check how well their process was performing.
What the Students Are Learning
For Sebastian Estrada the summer has reshaped how he thinks about the relationship between artificial intelligence and historical research and pushed it in a direction he didn’t fully anticipate.
“The overarching takeaway has been that human expertise matters more, not less,” Estrada said. “This project has made it clear that any model’s effectiveness ultimately depends on the quality of the ground truth data it is trained on. Without painstaking efforts from experts with immense domain knowledge, AI has little to learn from.” He added that while the technology allows the team to process far more material than would otherwise be possible, historians are still the ones who have to decide what the goal is and whether a given transcription is actually useful for later research.
That division of labor, between what AI can do and what still requires a person, shaped how the team worked day to day. According to Estrada, coding tools built with AI assistance were useful for constructing the image processing and transcription pipelines at the center of the project, so AI played a real role during that implementation phase. But when it came to planning and making sense of results, the team largely set AI aside or used it only for early brainstorming.
“We had accumulated so much project specific knowledge throughout the summer that relying on AI for decision making would have been difficult and potentially misleading,” Estrada said.
The project also taught Estrada skills he hadn’t expected to walk away with. Explaining the work to people with wildly different levels of background knowledge, something he found himself doing almost daily, sharpened his sense of how to read an audience and choose what needed to be said. “I noticed that I was becoming more intentional about probing an audience’s existing knowledge and selecting only the most important threads to communicate,” he said. The reverse was also true. Taking in a steady stream of information, from the ideas of the project leads to presentations from other teams, pushed him to get better at recognizing what he did not yet understand and asking sharper questions in the moment.
Working alongside teammates from different academic backgrounds left its own mark. Estrada said the differences in how people approached problems created a kind of productive friction that made the team’s overall approach stronger. Where his own instinct often ran toward a narrower technical question, whether something could be built, conversations with the rest of the team pushed him to think harder about weaknesses hidden in the design and whether the results would mean anything once they were finished.
Alan Lu, another of the undergraduates on the team, came away from the summer with a different set of takeaways, shaped in part by a career path that doesn’t run through history or computer science at all. Lu plans to pursue finance or business after graduation, but he said the project has put problem solving and teamwork at the center of how he thinks about that future.
“The interdisciplinary skills I gained from this project are very transferable,” Lu said. “Specifically, I am now better at analyzing and then communicating quantitative information in a digestible manner.”
Asked how he would describe the project to a friend or family member in a sentence or two, Lu kept it simple. “Teaching LLMs how to read historical documents that are in bad shape, in not only an accurate, but efficient way,” he said.
Lu also sees the underlying technology reaching well past this one project. He pointed to the broader potential of computer vision and image recognition to assist people who are visually impaired and said he is especially excited about what the work could mean for humanities research more generally. “There are so many under discovered and under researched historical documents in libraries and archives all around the world that have yet to be analyzed,” he said.
If there is one thing Lu hopes readers take from the story, it is a case for spending a summer on research even when it does not point directly at a career goal. “Doing research in the summer is a worthwhile experience even if it doesn’t align with your future goal,” he said. “I know that I will be a better team player for future internships and jobs.”
Beyond One Summer
For Fresh, the project sits at the intersection of two things she cares about equally: the technical work of building tools that can read difficult historical sources, and the substantive questions those sources can help answer. She sees the growth of global trade during this period as one of the most transformative economic shifts in modern history, rivaled only by the Industrial Revolution in the scale of its impact on Europe and, eventually, on much of the world.
Getting reliable data out of the port books, she said, will make it possible to describe that transformation in far more detail than historians have managed before, and to connect it to bigger questions about who gains and who loses power when an economy is remade.
The Data+ portion of the project runs through the summer, but the tools the students are building are meant to outlast it. A public repository will let other researchers apply the same methods to their own archives, extending the project’s reach well beyond Duke and well beyond this one collection of old English ledgers.
—– —– —–
Project: Machine Vision and AI to Uncover Early Modern Economic Upheaval
Team Members: Alan Lu, Sebastian Estrada, Faisal Bander Alghamdi
Project Leads: Astrid Giugni and Adriane Fresh
Project Manager: Lauren Hoaglund