prompting the past

Frédéric Clavert
(C²DH, University of Luxembourg)

inactinique.net / github.com/inactinique
@inactinique@hcommons.social

“a portrait of Mr Bean as Napoléon Bonaparte” – obviously a more accurate representation of Napoléon than the Ridley Scott movie. Source: lexica.art

digital history and digital memory studies litterature

(Kansteiner (2022)): what would be a specifically trained generative AI for historians
(Makhortykh (2023)), (Walden and Marrison (2023)): possibilities and risks of AI, particularly generative AI, in Holocaust studies
(Hutchinson (2022)): online software to understand what generative AI systems “know” about history

Digital history and digital memory studies litterature on AI has expanded. For instance, Wulf Kansteiner investigated what a specifically trained generative AI for historians could be but without really looking at primary sources produced by these systems.

The possibilities and risks of AI, particularly generative AI and especially in Holocaust studies, were analysed in an “open forum” of Eastern European Holocaust Studies (Makhortykh (2023)). In particular, the (low) adequacy of the responses of some of the generative AIs to Holocaust research was pointed out (Makhortykh, Vziatysheva, and Sydorova (2023)).
Online software can be tested to understand what generative AI systems “know” about history (Hutchinson (2022)).

Changes – positive or not – in the memorialisation processes have also been analysed (Makhortykh et al. (2023)), rather from the infrastructure angle. Most of those articles are also dealing with ethics, privacy and biases (including biased training dataset). Though research on AI and the past becomes more dense, as far as I know, there is little research on prompts as a source for history or for the study of collective memory.

questionning the past…

…as historian’s basic epistemological operation

…with ‘stochastic parrots’? (Bender et al. (2021))

‘“A historian facing massive data” by Caspar David Friedrich’

prompts as

primary sources
artefacts that tells us a lot about the societies of the (near) past
open doors to users’ imagination about the past
- BUT result of a user-machine negotiation

Harvesting data

strong barrier to that kind of research
ethics challenges (D/ARC community, AoIR Ethics Guidelines.)
different paths to get data

The first methodological barrier to a research project on prompts related to the past is the making of a corpus. There are several ways we could assemble a database of prompts.

A first path would be to use dedicated search engines, such as Lexica. Those search engines poses several problems: either they are basic keywords based search engine and collecting prompts would imply a database of past-related keywords, or they are “semantic” (the case of lexica) and becomes hence black boxes. A further problem with those search engines is that, in our experience, their developers have not kept a crucial metadata: date and time.
A second path would be to directly do what those search engines are doing: collecting data on Discord, an instant messaging and VoIP social platform. Discord has indeed become a keypoint for communities to share their uses of large langguage models or image-generation systems. Companies such as midjourney have even used Discord as an interface to their system. Data could then be scrapped from discord public servers – after consultation of an ethics review panel – through a bot to respect Discord terms of use.
Third path is to use existing datasets, including krea open prompts (that I am currently investigating)
other solutions: setting up a website designed for prompts about the past, or a mix of all those solutions.

No perfect solutions, probably important to mix all of them.$$

a balanced corpus?

Analyzing data

Analysis of the corpus of prompts with IRaMuTeQ

In this analysis, I have collected data from lexica.art‘s search engine. This small corpus (with no personal data, using the lexica API that existed at this time) – almost a sample corpus – is made of 1908 prompts (in other experiments? i deal with much more prompts), collected in March 2023 with the keyword ’european union’.

I chose this keyword to see in which ways users are relating to the European Union’s and Europe’s historical past.

The preliminary results show that, here using iramuteq, references to the past are of different nature in those prompts.

Styles (the style users want for an image) are sometimes ‘historical’ (‘soviet propaganda’ for instance).
References to the past are mixed with elements of the current or recent news (‘marine’ for the far-right politician Marine Le Pen in France, ‘nigel farage’ for the Brexit activist, or references to the Russian agression against Ukraine for instance).
The notion of ‘Europe’ is linked to a precise period of time, such as the Middle Ages (‘heraldic’).
Some historical concepts linked to the European history are also quite visible (‘empire’ for instance). War is very present, but our experience shows that there is a general link that is made between history and war in that kind of corpus.

I have now moved to the krea prompt corpus (10 millions prompts, of course only some smal part with historical references), and am experimenting with the more traditional LDA style topic modelling, and to word vectors for now. Using a LLM couls of course be a supplementary option.

Conclusion

Code

Short paper DH2024. Available on Github, with code notebook

References

- extended bibliography on AI and collective memory: https://www.zotero.org/groups/4874770/ai__collective_memory

Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610–23. FAccT ’21. New York, NY, USA: Association for Computing Machinery. https://doi.org/10.1145/3442188.3445922.

Hutchinson, Daniel. 2022. “What Do AIs ’Know’ About History? A Digital History Experiment.”

Kansteiner, Wulf. 2022. “Digital Doping for Historians: Can History, Memory, and Historical Theory Be Rendered Artificially Intelligent?” History and Theory 61 (4): 119–33. https://doi.org/10.1111/hith.12282.

Makhortykh, Mykola. 2023. “Open Forum: Possibilities and Risks of Artificial Intelligence for Holocaust Memory.” Eastern European Holocaust Studies 0 (0). https://doi.org/10.1515/eehs-2023-0053.

Makhortykh, Mykola, Victoria Vziatysheva, and Maryna Sydorova. 2023. “Generative AI and Contestation and Instrumentalization of Memory About the Holocaust in Ukraine.” Eastern European Holocaust Studies, November. https://doi.org/10.1515/eehs-2023-0054.

Makhortykh, Mykola, Eve M. Zucker, David J. Simon, Daniel Bultmann, and Roberto Ulloa. 2023. “Shall Androids Dream of Genocides? How Generative AI Can Change the Future of Memorialization of Mass Atrocities.” Discover Artificial Intelligence 3 (1): 28. https://doi.org/10.1007/s44163-023-00072-6.

Walden, Victoria Grace, and Kate Marrison. 2023. “Recommendations for Using Artificial Intelligence and Machine Learning for Holocaust Memory and Education.” REFRAME. https://doi.org/10.20919/ELVH8804.