This AI can discover the hidden links between great works of art

When MIT CSAIL PhD student Mark Hamilton saw the “Rembrandt and Velazquez” exhibit in Amsterdam’s Rijksmuseum last year, he was surprised to see that some works of art that have no connection on paper, can look eerily similar in reality.

The show’s curators had paired Francisco de Zurbarán’s The Martyrdom of Saint Serapion, a 17th century Spanish religious painting, with Jan Asselijn’s The Threatened Swan, a Dutch canvass from a similar age. While the artists never met each other during their lives, the two works show some clear visual resemblance. 


The researchers were inspired by an unlikely, yet similar pairing: Francisco de Zurbarán’s, The Martyrdom of Saint Serapion (left) and Jan Asselijn’s The Threatened Swan (right).


It got Hamilton thinking about the other hidden links that could be uncovered in the history of art. The researcher and his team, in partnership with Microsoft, have now unveiled a new algorithm that takes image retrieval technology a step further, to run through millions of paintings across thousands of years and find unexpected parallels in themes, motifs, and visual styles.

Dubbed “MosAIc”, the system is currently running on the databases of works from the Metropolitan Museum of Art and the Rijksmuseum. From a single image, the tool can uncover connections in whatever culture or media the user is interested in, and quickly reach a number of closest possible works that match the original query.

MosAIc, for instance, was presented with the Dutch Double Face Banyan, an anonymous item of clothing from the late 18th century, and found similarities with a Chinese ceramic figurine. The connection can be traced to the flow of porcelain and iconography from Chinese to Dutch markets between the 16th and 20th centuries.



MosAIc was presented with the Dutch Double Face Banyan, and found similarities with a Chinese ceramic figurine.  


To develop MosAIc, the research team used an image retrieval system and the well-known “k-nearest neighbors” (KNN) algorithm, which is widely used to find objects based on similarity, for product recommendation for example. 

Typically, however, image retrieval systems that are enabled by the KNN algorithm present some limitations. The scope of a query is effectively limited: in the case of paintings, users could only ask for similar artwork from a specific artist. Or, they could run so-called “unconditional” queries, and gradually filter their way through results until they got an accurate answer, a process that is costly and time consuming. 

Hamilton and his team, instead, created a conditional image retrieval system (CIR), which delegates the filtering to the algorithm. The researchers still used the KNN algorithm, but enabled it to add “conditions”, like texture, content, color or pose, while the program is running, until it reaches the closest match for the original query.

The process is called a conditional KNN tree: the algorithm groups similar images together in a tree-like structure, and starting from the trunk, applies new filters as it climbs up, following the most promising branch until it finds the most accurate image.

Hamilton said: “Restricting an image retrieval system to particular subsets of images can yield new insights into relationships in the visual world. We aim to encourage a new level of engagement with creative artifacts.”

While recognizing that the technology does not break speed records, the team of researchers said that CIR can improve result diversity in a simple and efficient way.

And the new technology is not limited to artwork queries. Hamilton and his colleagues anticipate a number of applications for the new algorithm, including using MosAIc to better study deepfakes, and particularly where deepfakes most struggle to model reality. 

The algorithm, while working its way to the top of the tree to find an image that best matches a real picture, at the same time leaves behind – on its branches – the pictures that it believes fail to represent the original input. 

By going back to those branches, the researchers could visualize which images are deepfakes, as well as which conditions, or filters, convinced the algorithm to leave them behind – typically, because the deepfake failed to accurately represent a certain element of reality, like a microphone or a hat.

Although sometimes invisible to the human eye, those “blind spots” are what distinguish a sophisticated deepfake from a genuine image.

Hamilton hopes that MosAIc will be used in many other fields ranging from social science to medicine. “These fields are rich with information that has never been processed with these techniques and can be a source for great inspiration for both computer scientists and domain experts,” he said.



By going back to those branches, the researchers could visualize which images are deepfakes, as well as which conditions, or filters, convinced the algorithm to leave them behind. 


Artificial Intelligence

About the author

E-Crypto News was developed to assist all cryptocurrency investors in developing profitable cryptocurrency portfolios through the provision of timely and much-needed information. Investments in cryptocurrency require a level of detail, sensitivity, and accuracy that isn’t required in any other market and as such, we’ve developed our databases to help fill in information gaps.

Related Posts

For a Hedge Against Inflation - Click the Rocket!

E-Crypto News Executive Interviews

Blockchain/Cryptocurrency Questions and Answers

Stressed about crypto
The Worst Places in the World to Buy and Hold Crypto
January 27, 2023
What Are Crypto Index Funds?
What Are Crypto Index Funds?
January 19, 2023
Can You Make Money Anymore With Crypto in 2023?
January 13, 2023
What Is Stagflation And How Does It Affect The Crypto Markets?
What Is Stagflation And How Does It Affect The Crypto Markets?
January 12, 2023
passive crypto
How To Minimize Risks When Investing in Crypto
December 28, 2022

Automated trading with HaasBot Crypto Trading Bots

CryptoCurrencyUSDChange 1hChange 24hChange 7d
Bitcoin23,171 0.90 % 0.27 % 1.76 %
Ethereum1,595.1 0.84 % 0.18 % 1.90 %
Tether1.001 0.04 % 0.08 % 0.08 %
USD Coin1.000 0.44 % 0.25 % 0.18 %
BNB283.64 0.09 % 0.32 % 2.79 %
XRP0.4687 0.46 % 4.02 % 4.38 %
Binance USD1.000 0.15 % 0.14 % 0.18 %
Cardano0.3874 0.91 % 0.77 % 4.90 %
Dogecoin0.08884 0.60 % 1.13 % 4.87 %
Polygon1.140 1.24 % 0.61 % 15.07 %

Bitcoin (BTC) $ 23,110.19
Ethereum (ETH) $ 1,592.28
Tether (USDT) $ 0.999513
USD Coin (USDC) $ 1.00
BNB (BNB) $ 307.27
XRP (XRP) $ 0.410973
Binance USD (BUSD) $ 0.994098
Cardano (ADA) $ 0.385455
Dogecoin (DOGE) $ 0.088693
Polygon (MATIC) $ 1.14