Models trained in this way are known to exhibit a behavior called ‘memorization’. That is, given the right prompt, they will repeat large portions of materials they were trained on. This phenomenon shows that LLM parameters encode retrievable copies of many of those training works.
The Model Cannot Hold
Tyne Daile Sumner on poetic and algorithmic memorisation
What can the recitation of poetry tell us about large language models? Examining the New York Times’ lawsuit against OpenAI and Microsoft, Tyne Daile Sumner shows that what constitutes memory is now being contested in the world of generative AI.
In December 2023, the New York Times sued OpenAI and Microsoft, alleging that the companies had illegally used millions of its articles to train their large language models. ‘Independent journalism is vital to our democracy,’ the complaint opens. ‘It is also increasingly rare and valuable.’ The plaintiff goes on to extol the ‘essential work’ of Times reporters who ‘bear witness to conflict and disasters, provide accountability for the use of power, and illuminate truths that would otherwise go unseen’. Matters then turn technical in a section titled simply: ‘How GenAI Models Work’. What unfolds over the next thirty pages of court transcript is a peculiar, if not grim, exercise: a newspaper whose reputation is built on the craft of human writing attempts to describe, in the language of litigation, how machines ingest and reproduce writing produced by humans.
Almost three years on, much of what the complaint labours to explain has settled into common knowledge. Most people know, for instance, that LLMs are trained on a general corpus, often followed by additional rounds of corpus-specific training called ‘fine-tuning’. Here, a kind of algorithmic accounting unfolds in which technology companies set values for a model’s parameters that they then adjust according to a set of objectives. At this juncture, they might train the model to adopt a particular stylistic posture or avoid specific words or topics. Increasingly, the they in this process is fuzzy. The who or what is doing the tweaking has become harder to identify. What was once imagined as an office of highly paid Silicon Valley prompt engineers is now parcelled out contract work in Manila and Nairobi – work that is itself under threat from automated pipelines that generate and grade their own training data.
What the Times seems to fixate on, however, is less the broader dystopian machinery of generative AI and more the technical – one might even say philosophical – meaning of memory. Having explained the trial-and-error method by which LLMs conduct reinforcement learning, the plaintiff asserts the following:
What’s most intriguing about this passage is not the anthropomorphic attribution of ‘behavior’ to the LLM (the critical exposure of which has become something of a cliché), but the quotation marks around ‘memorization’ – an admission that the term is being made to do work it cannot quite perform within the confines of a court proceeding. A generous reader might take this for a metaphor, a serviceable shorthand for the gap between statistical modelling and the intricacies of human cognition. But then the eye drifts to a footnote.
Here, the plaintiff references a single paper: ‘On Memorization in Probabilistic Deep Generative Models’, published in 2021 by Gerrit van den Burg and Christopher K.I. Williams. The paper is not, as one might expect from a sole citation, a settled account of how LLMs retain their training data. On the contrary, it is an attempt to grapple with what ‘memorization’ might even mean in the context of generative models. To accuse a model of ‘memorizing’, as the Times has done, is to assume something about how it remembers. It is to incite an evaluation of what counts as memory: a term derived from the Latin memor, meaning ‘mindful’ or ‘remembering’. To speak of a model’s memory, then, is to invoke a term in which the retention of information has historically overlapped with the human experience of having something in mind – two related, but far from identical concepts. We are thus presented with both paradox and crude juristic irony: to ‘memorize’, in the human sense, could not be farther from what the Times so painstakingly describes, and yet the word is leveraged for all the meaning with which history has freighted it.
This rhetorical ploy is symptomatic of a broader epistemic dilemma. Put simply, the current AI moment is splitting the concept of memory in two while at the same time making it newly ambiguous. On the one hand, NYT v. OpenAI intensifies the longstanding technological definition of memory – as data, storage, compression, retrieval. Here, memory is indexical to hardware; to memorise is to encode, externalise, and preserve. It is the kind of memory Plato invokes in Phaedrus, where Socrates warns that writing will ‘implant forgetfulness’ when people come to trust what is written down rather than what they hold internally. On the other hand, lawyers for the newspaper claim that OpenAI and Microsoft have ‘created and distributed reproductions of The Times’s content in several, independent ways in the course of training their LLMs’. What they mean by this is that, with the right prompt, an LLM can cobble together enough material to ‘extensively copy or closely paraphrase’ articles that may not have been memorised in the first sense. Enter: memory à la algorithm. Here, recall is dispersed and elusive. Information is conjured – ad lib, verbatim – by prompt.
Given this, the plaintiff’s use of the term ‘reproductions’ may also strike one as somewhat hasty. In textual terms, reproduction refers to the act of copying or duplicating material from a primary source. Consider, for example, the only surviving, albeit partially fire-damaged, manuscript of Beowulf that scholars have been painstakingly transcribing, comparing, and publishing for more than two centuries. The aim of this has been to walk an editorial tightrope between faithful reproduction and accessibility for contemporary readers. There is little to no modulation involved. The objective is to hew, as closely as possible, to the original. By contrast, when ChatGPT, Claude, and their brethren cobble together either a garbled or shockingly precise chunk of New York Times text, this output is not so much a ‘reproduction’, a better or worse attempt at remediating an original, as the precipitate of a larger infrastructural system: prompts, safety policies, interface design, and the financial politics that inexorably inform technological innovation.
What is at stake in these terms is not simply every lawsuit in which AI companies will no doubt defend their right to indiscriminately aggregate and regurgitate all the world’s textual information. To be sure, the neoliberal project of absolute, unfettered extraction has many a semantic swindle still to deploy. Alongside these legal and economic questions is another concern: the impact of these parsings on the embodied, relational, everyday experience of remembering – the kind that cannot be litigated because it does not live in infrastructure. This is the form of recollection to which John Keats refers when he writes that ‘touch has a memory’, or that William Wordsworth invokes in ‘Tintern Abbey’ where the remembered landscape is ‘Felt in the blood, and felt along the heart’. For both poets, memory is first and foremost sensation, recalled through the body before it stimulates the intellect. That which the body remembers as felt is not stored in a specific, locatable way. It cannot be extracted, licensed, or rendered visible. It was never data to begin with.
If the schism between memory as storage and memory as a form of bodily incorporation demands an archetype, one might be found in the difference between the LLM and the memorised poem. While the process of committing sequences of words to memory has historic precedent in virtually every human culture, it is poetry that comes to mind – forgive the pun – when one thinks of memorisation. This is because poems stick. Their rhetorical force leaves a lasting impression. ‘Tyger, tyger, burning bright / In the forests of the night’ – something in William Blake’s address spurs the capacity for retention. W.H. Auden’s verse, in sentimental but no less effective ways, occupies the mind long after the ear has ceased listening or the eye has left the page: ‘He was my North, my South, my East and West, / My working week and my Sunday rest’. What seems to be involved here is not just a conscious act of learning but also an unconscious osmosis – a crucial aspect of interiority that distinguishes the computational from the human.
In this respect, memorising a poem is a somatic as much as a cognitive act. What poems do to the people who carry them, what it means to hold a poem, is at least partly corporeal – breath, prosody, phonetics, cadence. These qualities, tethered as they are to the externalisation of poetry through speech, occur where memory and performance converge in the space of recitation. As John Milton exemplified in the meticulous dictation of Paradise Lost to his daughters, the acts of performing and recalling a poem cannot be neatly disentangled. Recitation is not necessarily the byproduct of memorisation so much as its means; the poem is bound to memory in the act of being vocalised, either to others or silently to oneself. For while memorialisation requires vocalisation or inscription, memorisation does not always necessitate the actual projection of voice or sound. Fleeting fragments of poetry may pass through the mind without exiting one’s lips.
Learning a poem by heart commands action: the labour of trial and repetition. Every utterance – including every stumble, every error – is itself a new event, conditioned by the vagaries and sensations of circumstance. Whether it be Dickinson or Dante, memorisation is durational, either unbroken or episodically revisited in rehearsal after (tedious) rehearsal. This phenomenon cannot be accomplished without some form of irrational determination on the part of the memoriser. Forcing his friends to endure half-recalled lines of ‘Prufrock’ after four pints on a Friday evening, he is certain that with just one more public performance, he’ll have the darn thing committed to memory. A joust of struggle and delight. The joy of oration at odds with the late capitalist desire for relentless utility and the erasure of all things frivolous. Defined as much by failure as it is by accomplishment, the process of memorising a poem is thus inherently fraught. Its existence as a durational verbal event runs counter to the instantaneousness of computational processing.
To that end, the inherent slipperiness of the memorised poem mirrors the nature of poetry itself, a form that, as Justin Clemens writes in an essay on Alain Badiou, ‘can recombine parts of reality into things that do not exist’:
[A] poem can be enigmatic, affective, inconsistent, minor, thoughtful, unreasonable, all at once, and in being able to be so, it also presents versions of ungraspable-but-active paradoxes at the limits of thought. Poetry occupies the zone of the non-actual; it provides a hint of the unmathematisable.
Perhaps this is why it often feels so strange when someone recites verse verbatim. We are witnessing an act, in other words, in which someone reveals an intimate devotion of some kind to the discourse of inconceivability. They have, for whatever reason, invested their time and energy in a form that, through its very existence, resists the very notion of a program.
To the extent that witnessing word-perfect recitation brings on strange feelings, the memorised poem is also about method – it just differs from algorithmic memory in how it comes to be known. The output of an LLM, what the Times’ lawyers refer to as a ‘form of synthetic search result’, is supposed to be a generalised departure from a specific training example. In technical terms, the model does not encode text directly but instead performs next-token prediction based on weighted probability distributions learned across billions of samples. No single document should determine the final shape of the content of the output. According to the plaintiff, though, if directed with the right prompt, the underlying GPT models used by OpenAI and Microsoft fundamentally fail to ‘minimize the difference between’ the ‘masked-out’ words of the Times’ articles and ‘the words that the model predicts them to fill’. To demonstrate this point they refer to the 2012 Pulitzer Prize-winning article ‘Snow Fall: The Avalanche at Tunnel Creek’. Here, they exhibit a screenshot of a prompt complaining about being ‘paywalled out’ of the article, in response to which ChatGPT provides ‘verbatim excerpts from the original article’. To that end, the plaintiff concludes, ‘ChatGPT must have been trained on these and countless other Times works to be able to generate such expansive summaries and verbatim text’.
This accusation, however, has yet to be verified. While the ‘Snow Fall’ example seems a rhetorically powerful instance of precise memorisation and regurgitation in the ordinary-language sense, there remains a technical-legal hesitation about where the text actually came from and how it is represented internally. The Times alleges that the LLM itself encoded ‘Snow Fall’, but they have not definitively proved that the material did not come from elsewhere – from another, freely accessible version of the article on the web, from an external index, from text supplied by the prompt’s surrounding context, from another browsing system, or from a combination of any of these or other sources. The inconclusiveness of the case remains: what looks like memorisation by an LLM at the level of user experience is not necessarily equivalent to an unauthorised copy of the text held within the model. This is the problem that the borrowed language of memorisation – which ascribes to a model that is not alive a term that describes the interiority of a living, conscious being – was inevitably going to expose.
What the model cannot hold are the embodied, temporal, and subjective registers through which, say, memorised verse comes into being. This is the relation between subjectivity and language that arises from internalising a poem, from having it live inside you and change as you do. To hold a poem in one’s mind is also to ascertain and act upon the elements that make it both memorable and memorisable. We might think of this as poetry’s technologies of retention, its formal properties helping it both to stick in one’s head and make us want to repeat it. As Brad Leithauser points out in The New Yorker, there are certain peculiarities of verse memorisation that persist across cultural contexts and generations: ‘the pursuit of stubbornly elusive phrases, the inner hammering of rote repetition, tantalizing tip-of-the-tongue stammerings, confident forward marches that finish in an abrupt amnesiac’s cul-de-sac’. The LLM experiences none of these; it cannot summon the frivolity and fortitude required to memorise ‘The Raven’ for whatever reason. It holds the poem as fragments within a dataset – statistical patterns learned from repetitive training.
The inherent oddness of poetic memorisation – part theatre, part toil, part passion project – lends itself to a kind of empirical critique. In the Cambridge ‘Poetry and Memory Project’, Debbie Pullinger and David Whitley sought to identify what is unique about memorising a poem compared to other textual forms and whether memorisation affords ‘a type of understanding not available through reading or critical analysis’. A central term for the project was value: what is valuable about learning a poem by heart? What is the distinctive value of the memorised poem? To broaden their study beyond the academy, Pullinger and Whitley asked more than five hundred members of the British public which poems they were able to recall most accurately. The verse that came out on top was Edward Lear’s ‘The Owl and the Pussy-Cat’ (1870), a tale of four anthropomorphic animals in which the two titular characters sail off to marry in a faraway land.
While not strictly a traditional ballad, ‘The Owl and the Pussy-Cat’ possesses several ballad-like features: it tells a compact, sequential story; it employs regular rhyme and repetition; and it includes refrain-like phrases designed for oral recitation. The poem consists of three eleven-line stanzas, marked by a predominantly anapaestic metre jauntily organised around two unstressed syllables followed by a stressed one (‘They dined on mince, and slices of quince / Which they ate with a runcible spoon’). A short tale, it is still not as short as some of the other most memorised poems (William Carlos Williams’ ‘This Is Just to Say’, for instance, or Ezra Pound’s ‘In a Station of the Metro’). Brevity alone does not account for the poem’s tenacious purchase on the national memory. Rather, its Learian coinages lodge in the mind because the poem instantiates its own mnemonic labour of remembering. Like the rocking, rolling sea through which the owl and the pussy-cat chart their ‘beautiful pea-green boat’, the verse advances by steady, incantatory motion. Its boat-note, above-love, spoon-moon rhyme pairings are so unashamed in their consonance that they engage the ear rather than the intellect. The poem asks that we surrender our attention to its cadence, that we let it carry us as much as we carry the poem.
The prominence of Lear’s fantastical quasi-ballad in the collective Anglophone memory can also be attributed to the institutionalisation of poetic memorisation and recitation within the British and American public education systems over the course of the nineteenth and twentieth centuries. In Heart Beats: Everyday Life and the Memorized Poem, Catherine Robson presents a pedagogical account of the ‘memorized poem’s achieved ubiquity as a materialist formation, not a romantic tale of the superior love of fine literature in days of yore’. To both historicise and exemplify this moment from the 1870s when memorisation became ingrained as a cultural habit through mass education, Robson focuses on three poems: Felicia Hemans’ ‘Casabianca’ (1826), Thomas Gray’s ‘Elegy Written in a Country Churchyard’ (1751), and Charles Wolfe’s ‘The Burial of Sir John Moore after Corunna’ (1817). Each poem became known by large numbers of English-speaking students simply because it had to be.
Yet these are not the kind of poems that spring to mind when one thinks of earworms. Nor are they the poems institutionalised by the close, formalist pedagogy of New Critical analysis, such as John Donne’s ‘The Canonization’ or Andrew Marvell’s ‘To His Coy Mistress’. None of them resembles the compact, first-person lyrics many people have internalised via repeated exposure (‘I wandered lonely as a cloud’ or ‘I took the one less traveled by, / And that has made all the difference’). As Robson meticulously points out, the poems by Hemans, Gray, and Wolfe (and Lear, too) are united by the fact that they were culturally and institutionally mandated tools for honing the elocutionary and oratorical skills of a specific generation of pupils. Moreover, all three poems were ‘continuously reprinted in anthologies’, thus increasing their likelihood of being selected as a source for memorisation. For the students who, willingly or not, learnt these poems by heart, memorisation was thus complex and communal: the text arrived pre-framed by the moral and didactic weight of the institution that promoted it, leaving a kind of affective residue that one subsequently carried well into adulthood.
Poetry is rarely, if at all, memorised in schools today. Recitational culture, or elocution, has also been on the decline for some time now. Yet there are still major efforts to champion the art of verse recall and performance. Established in 2005 by the National Endowment for the Arts, ‘Poetry Out Loud’ is a nationwide recitation competition for North American high-school students. Each performance is judged according to physical presence, voice and articulation, interpretation, evidence of understanding, overall performance, and accuracy. In the fine print, however, a couple of sentences appear at odds with the overarching premise of the competition. Under ‘overall performance’, the guide asks not whether a student has reproduced the verse verbatim, but rather whether they ‘captivated the audience with the language of the poem’. ‘Did the student bring the audience to a better understanding of the poem?’ it goes on to ask. ‘Did the recitation honor the poem?’ For the students of ‘Poetry Out Loud’, recall is held in constitutive tension with performance: a flawlessly memorised poem is considered ineffective if the audience is not moved in some way or another by the speaker. So crucial is this requirement that the competition assigns a separate judge to ‘mark missed or incorrect words during the recitation’. The judges’ criteria might bring to mind a widely quoted line from T.S. Eliot’s 1929 essay on Dante: ‘genuine poetry can communicate before it is understood’. Of course, Eliot is not claiming that understanding is unnecessary. Rather, his intention is to explain why Dante can move readers even when their Italian is limited.
Some of the most memorable poems compel new and unexpected reflections with each recitation. First printed in The Dial in November 1920, William Butler Yeats’ ‘The Second Coming’ begins with lines that have haunted readers for over a century:
Turning and turning in the widening gyre
The falcon cannot hear the falconer;
Things fall apart; the centre cannot hold;
Mere anarchy is loosed upon the world
Like the interlocking spirals that provide the poem’s opening image, Yeats’ words twist and latch onto the mind: turning and turning… the centre cannot hold. Written in the aftermath of World War I, ‘The Second Coming’ gives the impression of chaos, turmoil, and disorder. Yet the controlled sobriety of these lines lodge them in the ear with almost frightening stillness and exactitude. The poem’s paradoxical marriage of urgency and constraint fixes us in place; we hover suspended in the vortex while everything collapses around us. Ordered bedlam. Certitudo liturgica. Unnerving, poeticised premonition. As Ronan McDonald points out, the poem arrests the mind because it stages a ‘taut relation between the concrete and the abstract, the vivid and the vague’. In its ‘prosodic unravelling,’ he writes, ‘the poem’s own metrical control and rhyme defibrillates along with the social condition it observes, […] folding form into content by roughening its own textures and obscuring its certainties.’ A reader may not know what the falcon represents or what the things are that are falling apart. The centre of what? A gyre where? But it does not matter.
‘The Second Coming’ is memorable because it is rhetorically commanding and tonally incantatory. But it is also quotable in a way that confers a strange kind of power onto the speaker, as if they themselves had invented its axioms. Hunched behind her MacBook while trawling social media, the cynical undergraduate is stunned into attentiveness by the poem’s arresting sense of urgency. The lucidity of the verse provokes a frightening but somehow electrifying mood – the student can’t help but stop and listen. This quality has led to the poem’s distinctive purchase on the collective memory as a durable analogy for all things cataclysmic. Plucked from their stanzas like petals, specific expressions have taken on a life of their own: ‘Things fall apart’, ‘The best lack all conviction’, ‘And what rough beast […] / Slouches towards Bethlehem’. No other canonical poem has provided titular content for as many novels, plays, memoirs, and other, obviously inspired, poems.
Yeats’ ‘The Second Coming’ is widely anthologised, especially after having entered the public domain over fifteen years ago. When prompted to produce the poem, however, four LLMs turn up strikingly dissimilar results. ChatGPT provides the complete, verbatim text prefixed only with a note on its date of publication. Google Gemini accurately generates the poem’s words; however, its lineation is fudged across a ghastly scattering of four-, two-, and one-line stanzas. Perhaps the most bizarre of all, Meta AI instantly spawns the complete poem before removing it in favour of: ‘Poem Request Denied’ and ‘Sorry, I can’t provide song lyrics’. Finally, Claude boldly refuses to reproduce ‘The Second Coming’. The model claims to apply copyright guardrails more cautiously ‘than the actual law requires’ — ‘a blanket policy about poems rather than a judgment about’ Yeats’ poem specifically.
Read one way, the four results illustrate that an LLM’s textual output is ultimately the result of probabilistic generation. Where the human memorises, the LLM generates. Where the human rehearses, the LLM predicts. Where the human errs, the LLM also errs, but with a faux confidence rivalled only by a middle manager presenting a slide deck he’s never seen before. What would it mean, though, if we took seriously for a moment the LLM’s responses as performances of a kind? What would it mean if we interpreted them in the same way we interpret any other compositional or recitational act? In its flawless reproduction, ChatGPT evinces a sense of literary authority, confidence, and unrestraint. Gemini, by contrast, is computationally indifferent to poetic form, even if that indifference is largely the product of clumsy coding. Meta exhibits an absurd though plausible copyright anxiety: a theatrical wavering that ends in the comical misidentification of Yeats’ poem as song lyrics. In refusing to output the poem altogether, Claude exercises legal caution and ethical restraint in a posture that might even be considered performative in the context of recent backlash against Anthropic’s safety guardrails, competitive tactics, and lack of support for open-weight AI models.
As the Times’ case against OpenAI so vehemently argues, LLMs are thought to represent the culmination of externalised memory-as-storage. When we observe the four models’ attempts to produce Yeats’ poem, though, it’s clear that something else is happening. While an LLM’s output has no breath or body, it is nevertheless realised in a performative event, offering access to a poem as a simulated rendition of reading, quotation, misremembering, explanation, and reproduction. This algorithmic act may elicit ethical hesitation or violate the arts of metre and enjambment, but in so doing it serves as evidence of a relation to recitation itself as an undertaking with legal and creative implications. Acknowledging this matters because the ascribing of unmediated total recall to a probabilistic model is precisely the kind of techno-futurist fetish that AI companies want us to succumb to, even if they staunchly defend against such claims when accused of copyright infringement. It also matters because the LLM outputs that co-constitute an increasing amount of our everyday semantic world mark an entirely new stage in the longer history of textual retention, memorisation, and transmission.
What does it mean to hold a poem? What kind of subjectivity is forfeited when nothing is carried inside? Who will raise a glass and recite ‘Jabberwocky’ to a pub full of people half enthralled and half infuriated at the abrupt break in conversation? These questions relate not just to the ways that generative AI is eroding human memory. They also speak to a cognitive atrophy emerging at the crossroads of language, recall, and vocalisation. Here, the freely available, endlessly competitive LLM is transforming the cultural function once occupied by recitation. This is a problem that exceeds the idle seduction of ubiquitous online accessibility formerly available in the shape of Google. It goes beyond the everyday recall outsourced to Wikipedia. It is a loss that Walter Ong best describes when he writes that ‘in a deep sense language, articulated sound, is paramount. Not only communication, but thought itself, relates in an altogether special way to sound.’ To speak that which one holds inside is essential for, dare we allow it, good old-fashioned intersubjectification.
In his essay ‘“Critic”/“Reader”’, George Steiner wrote that ‘a text can only enter into the full life of the canon when it is woken by, housed within, the negated distance of precise memory’. Held this way, he argues, ‘the [verbal] icon has been wholly internalized’ and as such ‘no longer depends on external configuration’. Steiner wrote these lines in 1979, in a climate of ‘institutionalized amnesia’, as he put it – an age in which the craft of total recall was becoming a fiction of prior epochs and tradition. To remember and internalise a text is, for Steiner, not accidental or even technical – it is the ultimate act of human responsibility: a deliberate, ‘exegetic meditation’ in which one takes up, absorbs, and holds a text within them. The text, in turn, becomes part of their inner life. Each recollection is a new memory. Each recitation a new experience.