Undergraduates Raphael Kim and Liuyi Zhu aren’t art history majors at Duke, and yet, they’ve spent their summer laying the groundwork for enhancing museumgoers’ experiences.
The two are participants on a team that’s exploring the feasibility of building an app that museum visitors could use to virtually restore paintings in museums.
Led by James B. Duke Professor of Mathematics and Electrical and Computer Engineering Ingrid Daubechies, the team also includes Andreas Badea, Robert Ravier, a graduate student in the Mathematics Department, and Gilad Amitai, a graduate student in Statistics. Bruno Cornelis, of Vrije Universiteit Brussel, and multiple experts at the North Carolina Museum of Art (NCMA) have also provided input and helped guide the group’s work.
The project is part of Data+, an immersive summer program supported by the Social Science Research Institute (SSRI) in which students of all levels work on interdisciplinary teams working with data. And while the connection between big data and art restoration may not seem obvious at first, the team is not the first to take on a project of this kind.
Their work is inspired by a 2016–17 Bass Connections team, also led by Daubechies, that participated in the recreation process for the missing ninth panel from the St. John Altarpiece by Francescuccio Ghissi.
A 14th-century Italian altarpiece depicting Jesus’s crucifixion and scenes in the life of St. John the Evangelist, it was dismantled and sawed apart during the 19th century. The panels were then sold to collectors, with the ninth and final panel missing ever since.
Artist conservator Charlotte Caspers recently reconstructed the missing ninth panel for an NCMA exhibition displaying the entire piece as a whole for the first time in over 100 years.
Caspers used input from NCMA experts and Ghissi’s own painting techniques to replicate the panel, but it stood out from the authentic panels due to the other panels’ age and wear. That’s where Daubechies’ Bass Connections team stepped in. Using imaging tools, they were able to digitally age the reconstructed panel so it would look consistent with the other eight authentic panels.
Daubechies’ team used open source computer graphics tools and machine learning to add cracks to the surface, darken the gold leaf background, and fade the colors of the paint. The result is a reconstructed panel that doesn’t detract from the beauty and experience of the others.
“It turns out many conservators in museums don’t tend to have recent, updated versions of Photoshop because the money they spend is typically for physical conservation,” Daubechies said.
Using open source tools was therefore paramount for providing a tool museum partners could access.
The exhibition Reunited: The Ghissi Altarpiece displayed the work at the North Carolina Museum of Art and the Portland Art Museum, where it closed at the start of July. For the exhibit, a print of the digitally aged panel was shown with the eight other authentic panels. Caspers’ panel was shown separately so museumgoers could appreciate how the work would have appeared to Francesco Ghissi without detracting from the experience of the aged piece.
Viewers interested in seeing the Altarpiece and how the panels once looked can view them at the project’s site using a sliding tool that shows the difference between the aged and non-aged panels.
This difference in museumgoers’ experiences inspired Daubechies to continue her team’s work over the summer as a Data+ team. The idea was to use the imaging techniques learned with the Ghissi piece and automate them so they could work in an app for museumgoers.
With the app, visitors could then use their smartphones to experience the artwork both as it appears today in the museum and as it appeared to the artist hundreds of years ago.
But before an app could be developed, they needed to know if it was even feasible. Months of manual work went into the Ghissi piece, so engineering an app that could replicate that work wasn’t necessarily practical.
“As a computer scientist, I was thinking of ways to improve the workflow,” Kim said of his time on the Bass Connections team. “I just didn’t have the chance to do it. And this [Data+ project] offered the chance to actually try carrying that out.”
With team members tasked with different aspects of rejuvenation and some necessary overlap as tasks blended into one another, the Data+ team was able to come up with a feasible workflow for the app.
Their goal is to hit the sweet spot between user-friendly and engaging. The app they envision involves photographs taken by the user in the museum and high-resolution images of the work provided by the museum. The app would then allow user-assisted image manipulation, letting users interact with the artwork through the app.
Daubechies is scheduled to meet with individuals at the Getty Foundation to discuss the next stage of the project, which would require collaborating with professional developers who can write the app. The Foundation is interested in supporting digital humanities projects, Daubechies said, so supporting the continuation of this project seems a good fit.
The project is a unique take on data in the Data+ program. Rather than cleaning and analyzing a large CSV file of data, Daubechies’ team cleaned and analyzed the data on the paintings: signs of wear, cracks in the paint, and faded punch marks in gold leaf all required a precise, mathematical approach to discover, manipulate, and automate.
Using tools like Python common in data science work, her team developed an approach to data that highlights how open and interdisciplinary the data science can be.
“It shows that data is much more than what people think of, and not just statistical and data mining,” Daubechies said. “These are data as well. These are the data on the paintings, and they contain together with the human knowledge of the conservators, all of the ingredients in order to make this rejuvenation possible. If the application does take off, it will engage people who like working with computers and image and data files with the art, which they might not have otherwise engaged with.”
Amitai, Kim, and Zhu echoed this sentiment. With backgrounds in computer science statistics, computer science, and mathematics, they never imagined an opportunity to apply their knowledge in such a creative way.
“It was very different work than I’ve ever done,” Amitai said. “We got to learn about digital imaging and other things I hadn’t learned about. It makes going to the museum a lot more interesting now that I have a little bit of background knowledge about how these paintings age.”
“Every museum you go to now, you’ll be scrutinizing those punch marks and the colors’ way of fading,” Daubechies responded. “It will be a totally different experience.”
Once the app launches, the team’s work will allow a transformed museum experience for all visitors, even those without a data imaging background.