Periodic Reporting for period 4 - Machine Vision (Machine Vision in Everyday Life: Playful Interactions with Visual Technologies in Digital Art, Games, Narratives and Social Media)
Berichtszeitraum: 2023-02-01 bis 2024-07-31
Machine vision has become part of the everyday life of ordinary people. Smartphones have advanced image manipulation capabilities, social media use image recognition algorithms to sort and filter visual content, and games, narratives and art increasingly represent and use machine vision techniques such as facial recognition algorithms, eye-tracking and virtual reality. The ubiquity of machine vision in ordinary peoples’ lives marks a qualitative shift where once theoretical questions are now immediately relevant to the lived experience of ordinary people.
MACHINE VISION has developed a theory of how everyday machine vision affects the way ordinary people understand themselves and their world through 1) analyses of digital art, games and narratives that use machine vision as theme or interface, and 2) ethnographic studies of users of consumer-grade machine vision apps in social media and personal communication. Three main research questions were addressed 1) new kinds of agency and subjectivity; 2) visual data as malleable; 3) values and biases.
The project filled a research gap on the cultural, aesthetic and ethical effects of machine vision. Previous research on machine vision was skewed, with extensive computer science research and rapid development and adaptation of new technologies. Cultural research primarily focused on systemic issues (e.g. surveillance) and professional use (e.g. scientific imaging). Aesthetic theories (e.g. in cinema theory) had mostly addressed 20th century technologies. The MACHINE VISION project thus provided much needed humanities research (theories and methods) just in time for the major technological shift we saw with the rapid spread of generative AI from late 2023 onwards.
MACHINE VISION has developed a theory of how everyday machine vision affects the way ordinary people understand themselves and their world through 1) analyses of digital art, games and narratives that use machine vision as theme or interface, and 2) ethnographic studies of users of consumer-grade machine vision apps in social media and personal communication. Three main research questions were addressed 1) new kinds of agency and subjectivity; 2) visual data as malleable; 3) values and biases.
The project filled a research gap on the cultural, aesthetic and ethical effects of machine vision. Previous research on machine vision was skewed, with extensive computer science research and rapid development and adaptation of new technologies. Cultural research primarily focused on systemic issues (e.g. surveillance) and professional use (e.g. scientific imaging). Aesthetic theories (e.g. in cinema theory) had mostly addressed 20th century technologies. The MACHINE VISION project thus provided much needed humanities research (theories and methods) just in time for the major technological shift we saw with the rapid spread of generative AI from late 2023 onwards.
Our research shows that machine vision is more than a technology. It is actively imagined, contextually situated, and historically complex. Machine vision is not only about seeing – it does things, and it is biased at every level. Despite the prevalence of dystopic stories where machine vision is used to oppress humans, we also see that machine vision is constantly negotiated, opening up a potential for change.
Key outputs include:
- The curation of a publicly available dataset with structured interpretations of how humans and other entities interact with machine vision technologies in 500 movies, novels, digital artworks and video games. Software for analysing the dataset has been published, and there is a data paper describing the dataset and how it can be used by other researchers.
- The development of an analysis model, "machine vision situations", that allows both close and distant readings of how agency is represented or functions when humans and technologies interact. The model was developed for analysing our dataset, but will also be useful for other studies of technology, both in media studies and literary studies and in ethnographic analyses of real-world interactions with technology. A feature of this model is that it enables both large scale analysis and close readings.
- A series of analyses of specific digital artworks, films, television series and video games with a focus on how machine vision is represented and used in cultural modes of discourse.
- Cross-cultural anthropological analyses of machine vision in a Chinese context, in Europe and in the USA.
- Development of concepts such as "cyborg vision", which helps understand the double way of seeing the world through both our human eyes and technical systems, and which we are trained in through video games and other forms of entertainment.
- An understanding of how digital artworks function as artistic audits that test and critique existing AI systems and how this can impact technological development and public discourse
- A method for using "algorithmic failure" or machine learning's failure to predict the correct answer to identify rich cases for qualitative analysis in a large dataset.
- A monograph titled Machine Vision: How Algorithms are Changing the Way We See the World (Polity Press 2023) and many public lectures on the topic
- We used creative methods in addition to critical theory and analysis. Team member Linda Kronman developed the artwork Suspicious Behaviour (2020) with collaborator Andreas Zingerle, allowing users to experience how a clickworker is required to code datasets for AI-based surveillance. The work was shown at several exhibitions. Kronman and Zingerle won the 2022 Austrian Outstanding Artist award in the media arts category. (https://kairus.org/suspicious/)
- We developed a series of larps (live action roleplaying games) exploring visual surveillance in 2020-2021, and in 2024 a larp exploring social and cultural impacts on new visual and algorithmic technologies in the 1840s centred around Charles Babbage's Saturday night soirées.
- Three PhD students were trained in the project, Ragnhild Solberg, Linda Kronman and Marianne Gunderson, with the last defending her dissertation in Nov 2024. Each now has extensive expertise in critical theories and technical knowledge related to generative AI and AI-based image recognition, and specialised knowledge in their field: game studies for Solberg, digital art for Kronman and narratives and social media for Gunderson.
The research is summarised in a visually presented report that can be downloaded at https://hdl.handle.net/11250/3158329
Key outputs include:
- The curation of a publicly available dataset with structured interpretations of how humans and other entities interact with machine vision technologies in 500 movies, novels, digital artworks and video games. Software for analysing the dataset has been published, and there is a data paper describing the dataset and how it can be used by other researchers.
- The development of an analysis model, "machine vision situations", that allows both close and distant readings of how agency is represented or functions when humans and technologies interact. The model was developed for analysing our dataset, but will also be useful for other studies of technology, both in media studies and literary studies and in ethnographic analyses of real-world interactions with technology. A feature of this model is that it enables both large scale analysis and close readings.
- A series of analyses of specific digital artworks, films, television series and video games with a focus on how machine vision is represented and used in cultural modes of discourse.
- Cross-cultural anthropological analyses of machine vision in a Chinese context, in Europe and in the USA.
- Development of concepts such as "cyborg vision", which helps understand the double way of seeing the world through both our human eyes and technical systems, and which we are trained in through video games and other forms of entertainment.
- An understanding of how digital artworks function as artistic audits that test and critique existing AI systems and how this can impact technological development and public discourse
- A method for using "algorithmic failure" or machine learning's failure to predict the correct answer to identify rich cases for qualitative analysis in a large dataset.
- A monograph titled Machine Vision: How Algorithms are Changing the Way We See the World (Polity Press 2023) and many public lectures on the topic
- We used creative methods in addition to critical theory and analysis. Team member Linda Kronman developed the artwork Suspicious Behaviour (2020) with collaborator Andreas Zingerle, allowing users to experience how a clickworker is required to code datasets for AI-based surveillance. The work was shown at several exhibitions. Kronman and Zingerle won the 2022 Austrian Outstanding Artist award in the media arts category. (https://kairus.org/suspicious/)
- We developed a series of larps (live action roleplaying games) exploring visual surveillance in 2020-2021, and in 2024 a larp exploring social and cultural impacts on new visual and algorithmic technologies in the 1840s centred around Charles Babbage's Saturday night soirées.
- Three PhD students were trained in the project, Ragnhild Solberg, Linda Kronman and Marianne Gunderson, with the last defending her dissertation in Nov 2024. Each now has extensive expertise in critical theories and technical knowledge related to generative AI and AI-based image recognition, and specialised knowledge in their field: game studies for Solberg, digital art for Kronman and narratives and social media for Gunderson.
The research is summarised in a visually presented report that can be downloaded at https://hdl.handle.net/11250/3158329
Our most important innovations are methodological and include a model for analysing agency in technological situations and a range of methods for analysing generative AI from a humanities perspective. We developed an analysis model, machine vision situations, that enables structured analysis across hundreds of works as well as providing a framework for qualitative
and interpretative analysis of individual situations, and we developed methods for using machine learning in qualitative analysis.
The project has also functioned as a launch pad for further research. It was an extremely useful time period to be developing research on AI-power vision, and this meant that the project team was ready to contribute when generative AI suddenly went mainstream with DALL-E and then ChatGPT in 2023, receiving three major grants all related to AI from a humanities perspective:
- The PI, Jill Walker Rettberg is now co-director of a Norwegian Centre of Research Excellence, the Center for Digital Narrative (funded for 10 years, approximately €15 million), where AI is an important aspect of our research. (http://uib.no/en/cdn)
- The postdoc in the project, Gabriele de Seta, has received a prestigious starter grant (approximately €2 million including confounding from the University of Bergen) from the Trond Mohn Research Foundation: Algorithmic folklore: The mutual shaping of vernacular creativity and automation. (https://www.uib.no/en/cdn/171826/algofolk)
- Jill Walker Rettberg also received an ERC Advanced grant, AI STORIES: Narrative Archetypes for Artificial Intelligence (2024-2029, €2.5 million - https://www.uib.no/en/cdn/169711/ai-stories)
We have also contributed extensively to policy development in response to the sudden mainstreaming of generative AI, both by participating in expert committees for the Norwegian government and research council, and by giving many public talks to industry, policy makers and stakeholders in libraries, schools, universities, science museums, etc. For a full list of public talks see https://app.cristin.no/projects/show.jsf?id=577988
and interpretative analysis of individual situations, and we developed methods for using machine learning in qualitative analysis.
The project has also functioned as a launch pad for further research. It was an extremely useful time period to be developing research on AI-power vision, and this meant that the project team was ready to contribute when generative AI suddenly went mainstream with DALL-E and then ChatGPT in 2023, receiving three major grants all related to AI from a humanities perspective:
- The PI, Jill Walker Rettberg is now co-director of a Norwegian Centre of Research Excellence, the Center for Digital Narrative (funded for 10 years, approximately €15 million), where AI is an important aspect of our research. (http://uib.no/en/cdn)
- The postdoc in the project, Gabriele de Seta, has received a prestigious starter grant (approximately €2 million including confounding from the University of Bergen) from the Trond Mohn Research Foundation: Algorithmic folklore: The mutual shaping of vernacular creativity and automation. (https://www.uib.no/en/cdn/171826/algofolk)
- Jill Walker Rettberg also received an ERC Advanced grant, AI STORIES: Narrative Archetypes for Artificial Intelligence (2024-2029, €2.5 million - https://www.uib.no/en/cdn/169711/ai-stories)
We have also contributed extensively to policy development in response to the sudden mainstreaming of generative AI, both by participating in expert committees for the Norwegian government and research council, and by giving many public talks to industry, policy makers and stakeholders in libraries, schools, universities, science museums, etc. For a full list of public talks see https://app.cristin.no/projects/show.jsf?id=577988