We have developed a research focus aimed at establishing iconographic connections between objects present in a scene. These connections can involve actions (such as looking at or holding an object), spatial relationships (one object placed on top of another), or even implicit interactions between elements. Among these objects, the face occupies a central place, as it often constitutes a valuable source of information in the analysis of a work. Just as two works can be compared based on the co-occurrence of objects, it is possible to establish a connection based on the pose of the characters, particularly the face. This is why we focused our attention on this specific element, detecting it and then estimating its pose.
Figure 1 Pipeline for iconographic analysis based on gaze direction
However, facial pose alone does not allow us to accurately infer gaze direction, which is essential information for establishing connections between the face and other objects in the artwork. To overcome this limitation, we integrated a 2D visual interest point estimator. This not only provides a 2D gaze direction vector, but also a binary classification indicating whether the gaze is directed inside or outside the scene (in/out). By combining this data with a depth estimate, we were able to generate a map of the field of view. Depth here adds a perspective dimension, essential for distinguishing between planes (background, foreground) and refining spatial analysis. Finally, we detected the objects present in this field of view and then modeled this information in vector form to enable similarity searches based on the objects actually viewed in the artworks.
Figure 2 Comparison between classic iconographic research and research based on gaze analysis
Left: The query image, whose character is looking at a crucifix in the upper right of the scene, accompanied by its field of view map highlighting the objects located in its line of sight. First row: Search results based on Lgaze gaze analysis. Below, the associated field of view maps. Second row: Results from traditional visual search. And below, the associated field of view maps.
To illustrate the value of gaze analysis in iconographic research, we compared two types of visual queries based on the same source image depicting Mary Magdalene contemplating a crucifix located in the upper right-hand corner of the painting. Her field of vision also encompasses a book and a skull, two elements often associated with the theme of vanitas. In contrast, in the second search, based on a detailed analysis of gaze direction, which we will refer to here as Lgaze, the results highlight works, sometimes very different stylistically, depicting both women and men, but all united by a common motif: a figure gazing at a crucifix. This approach thus allows for the emergence of a deeper thematic and iconographic coherence, beyond superficial visual similarities. In the first search, based solely on overall visual similarities (co-occurrence of objects, colors, composition), which we will refer to here as Lvisual, the works found predominantly depict female figures sharing certain iconographic attributes with the original scene. However, these characters generally present an evasive gaze, directed offstage, without explicit interaction with the objects present.
Figure 3 Combining visual similarity and gaze analysis with Lgaze_visual and Lvisual_gaze
To further refine iconographic research and increase the relevance of results, we propose combining the two previous approaches: classic visual similarity and gaze analysis. This fusion leverages both object co-occurrences and the attentional dynamics of the depicted figures, selecting works that share not only iconographic elements but also a common visual intention. We illustrate this hybrid approach with a concrete example, using the same query image of Mary Magdalene. The results thus obtained show scenes where the characters look at a similar central object (the crucifix), while maintaining strong visual consistency with the original work (presence of objects such as the skull or the book, posture, atmosphere) – illustrated with Lgaze_visual and Lvisual_gaze. This combination makes it possible to filter false positives and enrich the semantic interpretation of iconographic connections.
This work was published for the VISAPP 2025 conference.