Augmented Artwork Analysis

The project

Framework

The aim of the project is to produce a system for analysing a corpus of works of art from three partner museums: the Musée des Beaux-Arts in Lyon, the Palais des Beaux-Arts in Lille and the Musée National d’Histoire et d’Art in Luxembourg.

The project will be carried out by teams from the Lyon laboratories ICAR (Interaction, Corpus, Apprentissages, Représentations) and LIRIS (Laboratoire d’Informatique en Image et Système d’Information), the Luxembourg institute IRAM (Institut d’Etudes Romanes, Arts et de Médias) and two partner research centres (Center for Contemporary and Digital History and Centre d’Analyse Culturelle de la Première Modernité). It will bring together a range of multi-disciplinary skills, the main disciplines represented being Pragmatics, Semiotics (ICAR), Computer Science (LIRIS) and Art History (IRAM and associated research centres), thus forming a multi-faceted skills base capable of supporting the current trend in Visual Studies.

Objectives

What’s more, the analytical device we are seeking to create is at the root of at least three challenges facing modernity:

  1. Recognising the skills of artificial intelligence and providing it with a hermeneutical testing ground for investigating its scientific potential;
  2. Building theoretical and practical bridges between the disciplines of Art History, Semiotics, Computer Science and Visual Studies;
  3. Offering an innovative analysis of works of art that is both enhanced and personalised for a varied range of visitors.

The system will enable the different levels of organisation of a work of art to be studied: plastic, figurative and figural. To do this, it will employ two modes of observation: an on-site mode (in situ observation) and a remote mode through a freely accessible corpus of images. In this way, the project will provide several ways of accessing the meaning and practice of the work of art, seen as a stratified or multilayered object of expertise (Greimas, 1986).

These challenges will involve, in particular

  1. Elucidating genealogies and intertextual dialogues, the results of which will form the basis of our augmented system;
  2. Making these results accessible by thinking about the distance between what users are looking for and what they find thanks to the device;
  3. To support a new history of art renewed by, among other things, an approach to new forms inspired by the work of Henri Focillon and by new methods of visual analysis that recognise the importance of the plastic dimension of images, i.e. a new semiotic conception of a history of artistic forms.

Concepts used

Several key concepts stem from these objectives:

  1. The notion of genealogy, at the crossroads between genetics and dialogism, refers in the context of our project to the notion of form taken as a visual identity resisting over time within a given field and listed according to genealogical criteria. Genealogy is therefore linked, alongside other iconographic classes, to the idea of recognising the filiation between several families of transformations; form – in this context extended to the complexity of symbolic forms – constituting a relevant index of filiation.
  2. The notion of augmentation applied firstly to works of art, and secondly to technological devices, represents an important digital turning point in the renewal of strategies for accompanying the museum experience. As such, the augmentation of works of art appears to be a capacity for images to hold together, opening up the possibility of constituting cross-disciplinary paths of consultation and enjoyment. In our project, then, augmentation is intended to play a role in optimising interpretative practices that are already attested and respectful of all the facets of the image’s identity;
  3. The notion of the complementary museum: the museum, as the place where the system is implemented and tested beforehand, provides knowledge that complements that developed in the research institutes. Indeed, the partnership with three different museums provides the ideal framework for our project to gain a better understanding of the figure of the user and to study these skills, knowledge and objectives by means of in situ observation sessions. The museum, as a complementary field for parameterising the user, provides an opportunity to fine-tune and test the usability of our prototype.

Perspectives

In short, the scientific challenge of the project as a whole is to combine (i) recent advances in visual studies, (ii) augmented reality affordances and (iii) object detection and recognition methods based on deep learning in order to produce a new semiotic conception of a history of artistic forms.

For the history of digital art, the project of a history of forms is linked to the tradition established by Henri Focillon. Focillon conceived of forms as (i) the seeds of a structural dynamic within a work of art and (ii) the origins of a family of transformations that run through works of art. This research programme has not been widely exploited, despite important predecessors and revivals. This is due to two technical obstacles:

  • handling large corpora of images
  • detecting significant patterns that cross heterogeneous corpora.

To overcome these obstacles, we propose to study the diachronic changes in forms and their multiply-mediated genealogical ramifications using a range of hermeneutic tools: internal (for metavisual aspects) and external (for the study of paratexts). In this way, a digital device will provide a homogeneous representation in which it is possible to:

  1. observe whether images structure their own meaning by drawing the viewer’s attention to the way in which they are constructed and the way in which they ask to be seen;
  2. check whether the hermeneutic process can draw on other sources (literature, music, theatre);
  3. assess whether the explanation of images should involve their sensory/perceptive/experiential dimension.

Finally, the AAA project aims to understand any given image simultaneously at (1) intratextual and (2) intertextual levels. Automatic processing is needed at both levels, especially for archive consultation. The most innovative aspect is the experience that an image can be explained by a multitude of other images (augmented device) and not by an integrative and reductive verbal discourse.

Far removed from an augmented reality built on existing encyclopaedic knowledge, the reading of images cannot be reduced to the mere extraction of information. Extending these epistemological aspects would provide a rich pedagogical base capable of filling the gaps in primary and secondary education, where the curriculum is almost devoid of programmes devoted to visual culture.