241 lines
66 KiB
TeX
241 lines
66 KiB
TeX
\chapter{Latent Circuits and Disjunctive Syllogies: \gls{genai} as
|
||
Institution}\label{cha:institution}
|
||
\glsresetall
|
||
|
||
%%%%%%%%%%%\epigraph{
|
||
%%%%%%%%%%% [W]e observe a range of actors and symbolic representations: the fear of an \enquote{other,} an alien presence that is simultaneously close and familiar, juxtaposed with AI systems being cast as saviors – omnipotent or omniscient entities. Here, an unsettling paradox begins to emerge. These very systems, becoming increasingly autonomous, may soon engage in self-representation across the web, media, and public discourse. This raises intriguing possibilities about how AI might influence its own narrative in the collective imagination.}{\citeauthorfull{montanari2025} \cite*[206]{montanari2025}}
|
||
|
||
\epigraph{A progressive political agenda for the present is about moving at the same level of abstraction of the algorithm — in order to make the patterns of new social compositions and subjectivities emerge. We have to produce new revolutionary institutions out of data and algorithms. If the abnormal returns into politics as a mathematical object, it will have to find its strategy of resistance and organisation, in the upcoming century, in a mathematical way.}{\citeauthorfull{pasquinelli2015} \cite*[10]{pasquinelli2015}}
|
||
|
||
|
||
|
||
Previous chapters introduced a definition of control societies, their connection to processes of subjectivation, and the new \glspl{dispositif} that characterise the biopolitical stage \citeauthor{deleuze1992a} elaborates on. After examining the literature on critique and resistance (or the lack thereof) within control societies, new \gls{ai} technologies were analysed as possible characteristic dispositifs, first by tracing their historical development and then by unpacking the machinery that enabled the most recent breakthrough in the form of \gls{genai} models. Having developed the groundwork to understand the tendencies of \gls{genai} models, particularly the increasingly dominant \glspl{llm} in Chapter~\ref{cha:ai}, I now turn to reflect on some of the most influential debates of recent years concerning their social role and political implications. These debates provide the material through which the technical dynamics analysed earlier can be connected with the institutional framework of control societies outlined in Chapter~\ref{cha:control}.
|
||
|
||
Readers invested in Deleuzoguattarian thought may interpret the previous two chapters as establishing the seeds of a \enquote{connective synthesis} \parencite[68]{deleuze1983}: the \enquote{production of production}, or, as \citeauthorfull{buchanan2008b} \parencite*[59]{buchanan2008b} describes it, \enquote{arranging the organs anew in a new design} towards production. The present chapter can, in turn, be read as a phase of \enquote{disjunctive synthesis} \parencite[75]{deleuze1983}: a process of recording and distribution, in which connections and flows are inscribed rather than produced. As Deleuze reminds us, however, these syntheses always overlap and interpenetrate \parencite[13]{deleuze1983}.
|
||
|
||
The purpose of this step is to return to the guiding question: \textit{how should \gls{genai} systems be analysed within the institutional context of control societies}? This question has so far remained partly open, in part because critiques of \gls{ai} often leap directly to accusations of bias, harm, or capture without situating these systems in their historical position and technical machinery. The previous chapter addressed that machinery; the present one situates contemporary debates in academic literature against this backdrop and within the biopolitical dynamics of control societies. In this way, the analysis moves from groundwork to reconstitution, and, although only in structural resemblance, it operates akin to a mathematical proof by induction: having established the base case, we now advance the induction step, showing how the argument extends to broader theoretical and political concerns.\sidenote[][]{In mathematical proof by induction, the base case verifies that a statement holds for an initial value (e.g., $n=1$). The induction step then shows that if it holds for $n=k$, it also holds for $n=k+1$. For example, to prove that $1+2+…+n = \tfrac{n(n+1)}{2}$ for all $n$, one checks the base case $n=1$ ($1 = \tfrac{1(1+1)}{2}$) and then shows that if the formula holds for $n=k$, adding $(k+1)$ produces $\tfrac{(k+1)(k+2)}{2}$. The step mirrors how this chapter builds on the previous ones: from groundwork to generalisation.}
|
||
|
||
\section{The Value to be Attached: Latent World Models}\label{sec:latency}
|
||
\marginnote{Referring to the characteristic question \enquote{What value is to be attached to the theory that Eve sprang, not from Adam’s rib, but from a tumour in the fat of his leg (arse?)?} in \citeauthorfull{beckett2009}'s \parencite*[195]{beckett2009} novel \enquote{Molloy}}
|
||
|
||
One of the central dynamics highlighted in the analysis of \gls{genai} machinery was the amplification of stronger outputs within a distribution, which increases a model’s apparent \textit{accuracy} over time (for instance, through the interplay of gradient descent and backpropagation). This very attribute has become a central concern in debates about the nature of machine-generated content. \citeauthorfull{bender2021b} \parencite*[]{bender2021b} famously framed this risk as the \enquote{dangers of stochastic parrots}, pointing to these models’ statistical tendency to amplify overrepresented elements of their training data. Their argument is that such models, while capable of producing fluent text, operate by probabilistically recombining linguistic forms without \enquote{having access to meaning} \parencite[615]{bender2021b}.\sidenote{The reference to \enquote{meaning} may appear unusual, especially as the notion of \textit{meaning-making} is often defined as generating comprehensible content with coherence, relevance, and intentionality. \citeauthor{bender2020a} \parencite*[]{bender2020a} introduce a sharper distinction, arguing that \enquote{the language modelling task, because it only uses form as training data, cannot in principle lead to learning of meaning} \parencite[5185]{bender2020a}, while pointing towards the possibility of \enquote{human-analogous natural language understanding}. Although the theoretical scope of their proposal lies beyond this study, they conclude with the following thought-provoking claim:
|
||
|
||
\begin{quote}
|
||
\scriptsize The internal representations of a neural network have been found to capture
|
||
certain aspects of meaning, such as semantic similarity. [S]emantic similarity is only a weak]
|
||
reflection of actual meaning. [...] An interesting recent development is the emergence of models for unsupervised machine translation trained only with a language modeling objective on monolingual corpora for the two languages [...] If such models were to reach the accuracy of supervised translation models, this would seem contradict our conclusion that meaning cannot be learned from form. A perhaps surprising consequence of our argument would then be that accurate machine translation does not actually require a system to understand the meaning of the source or target language sentence.
|
||
|
||
\citereset
|
||
— \cite[5193]{bender2020a}
|
||
\end{quote}
|
||
Since then, \glspl{llm} have advanced even further than \citeauthor{bender2020a} \parencite*[]{bender2020a} anticipated. The question they left open is now more pressing than ever. Observing what current \glspl{llm} can achieve, one might even ask whether \enquote{meaning} is necessary for any articulation of language at all. Yet pursuing this question leads directly into linguistic debates; particularly a return to Saussurean theory and Hjelmslev’s extensions, which resonate with the reflections on language and expression in \citetitle{deleuze1987} \parencite*[1, 99--108]{deleuze1987}. Exploring this trajectory, however, lies beyond the boundaries of the present work.} \citeauthor{bender2021b}'s \parencite*[614-617]{bender2021b} claim is that the fluency of \glspl{llm} risks being mistaken for understanding, their reliance on large-scale datasets reproduces and amplifies social biases, and the recursive use of generated text could further entrench harmful stereotypes. Considering that these training corpora amount to the historical digital legacy of humankind, they warn of a risk where the models rearticulate older and less inclusive perspectives, despite the developed approaches to dismantle these in the context of critique and resistance:
|
||
|
||
|
||
\begin{quote}
|
||
A central aspect of social movement formation involves using language strategically to destabilize dominant narratives and call attention to underrepresented social perspectives. Social movements produce new norms, language, and ways of communicating. This adds challenges to the deployment of LMs, as methodologies reliant on LMs run the risk of ‘value-lock’, where the LM-reliant technology reifies older, less-inclusive understandings.
|
||
|
||
\citereset
|
||
— \cite[614]{bender2021b}
|
||
\end{quote}
|
||
|
||
|
||
The \enquote{value-lock} risk refers to the possibility of an unintended \textit{reactionary} tendency in the information created by generative systems, possibly undoing some of the achievements of social development. \citeauthor{bender2021b} raise concerns that the rapid scaling of \glspl{llm} to ever larger sizes, rather than levelling out or diluting radical arguments, may in fact increase the risk of reinforcing biases, abuse, harmful content, and conspiracy theories originating from online message boards, etc., even more. Their account is not entirely new: similar issues had already been raised in relation to earlier \gls{ai} applications. Recommendation systems, for instance, which relied on relevance associations to retrieve search results, media, and text, were likewise criticised for their tendency to amplify biases (see Section~\ref{sec:old_ai}). Yet the claim that machine-generated content has the potential to reproduce and intensify hegemonic arguments opens a different discussion: what kind of \textit{perception} of the world do these models embody? Are their outputs merely stochastic reflections of the training data, or is statistical selection of the most prominent arguments the sole principle guiding their content generation? Addressing this question is a crucial area of ongoing \gls{ai} research, not only because of its social implications but also for what it reveals about the future trajectory of \gls{ai} applications.
|
||
|
||
We have already seen how contemporary \glspl{llm} far exceed earlier systems in their capacity to construct interconnected feature spaces from training corpora. Through multi-processing and the transformer architecture, with its attention mechanisms enabling the formation of \textit{long-distance} relations across \glspl{token} (see Section~\ref{sec:transformer}), these models generate dense, high-dimensional spaces of association. This network is not a static repository; it actively constitutes a lingering giant in the background, exerting a gravitational pull that guides the generative process. Crucially, however, the formation of these connections and the generative act itself cannot be reduced to a one-way causal chain. Both unfold within a dynamic of double articulation; molecular activities of local interaction continually give rise to emergent molar structures, which in turn steer subsequent outputs. That means it is not possible to determine in any way what kind of molar formations of meaning are dictating the meaning-making process, adding up to the black box nature of the \gls{nn} architectures.
|
||
As this interplay between local dynamism and emergent consolidation stabilises during pre-training, the \gls{nn} acquires something akin to an internalised representation of the data, a structured experience sedimented in weight configurations, somewhat vaguely resembling what is called a \enquote{world model} in \gls{ai} theory and cognitive science \parencite[see e.g.][]{ha2018}.
|
||
|
||
A world model is a compact schema extracted from exposure to data, which enables an agent not only to react to familiar patterns but also to anticipate and navigate situations beyond its direct training history \parencite[267–268]{matsuo2022}. For generative systems, this internalised representation functions as a behavioural compass, orienting responses to novel prompts through probabilistic inference over prior experience (in the case of \gls{ai} models, their training data). In this sense, the model’s outputs are not mere recombinations of data but situated enactments of its learned \textit{world}, an epistemic field that governs future action.
|
||
One of the leading figures in \gls{ai} research, \citeauthorfull{lecun2022} \parencite*{lecun2022}, argues that autonomous intelligence requires a \enquote{configurable world model} capable of generalising, simulating, and guiding actions in unfamiliar contexts rather than merely reacting to inputs. In the context of \gls{genai} models, discussions often centre on how they parse training data into meaningful outputs, yet for \gls{ai} research, this representational fabric carries a broader significance. A central challenge is precisely how such models can \enquote{generalize to interact with the world and solve problems they have never encountered before} \parencite{lecun2022}, a question that remains pivotal for robotics and, more broadly, the pursuit of \gls{agi}. The necessity arises because, as powerful as \gls{genai} models are, and as fascinating as the transformer architecture’s ability to map vastly different contexts may be, contemporary \gls{ai} systems still fail when confronted with problems outside the scope of their training (see \cite[]{friedman2020} for a detailed interview with \citeauthor{lecun2022} on this issue). \Glspl{llm}, for example, are often successful at answering novel questions within language, but their translation abilities do not extend to processing inputs of a different kind. In \citeauthor{lecun2022}’s \parencite*[see][5]{lecun2022} view, overcoming this limitation requires a single, configurable world model that can share knowledge across domains rather than relying on separate models for each task.
|
||
|
||
Coming back to \citeauthorfull{dreyfus2009}’s argument briefly introduced in Section~\ref{sec:ai_history}, the notion of a world model immediately raises a Heideggerian question:\sidenote{Beyond \citeauthorfull{heidegger2010}’s main work \citetitle{heidegger2010} \parencite*[]{heidegger2010}, see also later relevant lectures such as \citetitle{heidegger1988} \parencite*[]{heidegger1988}. For a concise secondary account of his representational, or perhaps more accurately \textit{anti-representational}, theory together with Dreyfus’ reading of it, see \citeauthorfull{christensen1997}'s \parencite*[]{christensen1997} \citetitle{christensen1997}.} can the lived experience of a world ever be reduced to mere inferences drawn from a central representation? As \citeauthorfull{montanari2025} \parencite*[197–198]{montanari2025} summarises, Dreyfus maintained that everyday human know-how cannot be reduced to formalised in inferences, questioning how tacit and embodied skills could ever be captured as explicit knowledge. He emphasised the role of imagination and embodied context in meaning-making; for instance, spatial deixis such as \enquote{over there} or \enquote{nearby} presupposes a situated perspective in physical space. This line of thought resonates with \citeauthorfull{lakoff1999}'s \parencite*[see][37-38]{lakoff1999} accounts of embodiment, where cognition is structured by bodily experience and imaginative schemas. In a similar spirit, \textcite{lecun2022} stresses that the central challenge for machine intelligence is not statistical pattern-matching but the processing of sensory input in ways that resemble human situatedness.
|
||
For him,
|
||
the configurable world model
|
||
have to be able to generalise across contexts and anticipate novel situations, rather than merely reacting to inputs in pursuit towards a more sophisticated type of intelligence. The open question, then, is what it means (operationally, in the context of human–machine communication) for a machine to possess something like a unified representation of reality. Earlier paradigms of \gls{ai} approached this question through a \gls{sl} framework (see Section~\ref{sec:ai_history}): models were trained to classify inputs according to human-defined categories. This enacted a \textit{discriminative} logic, where decision-making was structured around predefined classes and expected outputs; in other words, machines were built to mimic human argumentation. As discussed in Chapter~\ref{cha:ai}, the field has since shifted towards forms of \gls{ul}, where models construct their own inferential structures from vast corpora without explicit labels. This paradigm enables training on scales far beyond human capacity to annotate, but it also leaves models entirely dependent on the contingencies embedded in the data. As \citeauthorfull{delanda2011} \parencite*[23]{delanda2011} reminds us, \enquote{patterns have properties, tendencies that are not present in the individual elements,} meaning that no analysis of single texts would ever reveal the emergent regularities that arise only at scale. Yet these emergent regularities, while constituting a form of distributed pattern recognition, do not amount to the kind of embodied and situated understanding described by Dreyfus and the phenomenological tradition. They remain statistical condensations of experience rather than lived engagement with the world. The question, then, is whether such architectures can ever move beyond their data-bound abstractions to form something genuinely akin to a world model, one capable of orienting itself within a horizon of meaning rather than merely mapping correlations within it.
|
||
|
||
Contemporary \glspl{genai} systems such as \glspl{llm} are still far away from building a real world model in any meaningful sense that would give them a human-like versatility to process and solve problems from vastly different forms and contexts \parencite[]{lecun2022}. They operate by correlating patterns in data, and in the case of \glspl{llm}, remain entirely confined to language or whatever data types they were trained on, rather than lived experience. Even the multi-modal models like newer ChatGPT versions (currently GPT-5) are not capable of such a task. However, \citeauthorfull{amoore2024} \parencite*[]{amoore2024} claim that the models are generating a (central) political representation nonetheless. They note that these models are always already instantiating a model of the world in terms of political logics and governing rationalities anyway, as they statistically internalise the structure of their training data.
|
||
For \citeauthor{amoore2024}, the decisive shift is from symbolic rules and normative standards to infrastructures of estimation. Decisions and outputs emerge not from deterministic reasoning but from probabilistic approximations. On this basis, the generative process itself is shaped by the political direction encoded in \enquote{the underlying joint distribution behind the phenomenal world of appearances} \parencite[3]{amoore2024}, raising questions such as: \enquote{What are their distinctive ways of estimating distributions or making predictions? How do they interpolate between data elements to form populations?} \parencite[2]{amoore2024}.\sidenote{Both the previously presented \textit{datalogical} argument and data-behaviourism \citeauthorfull{rouvroy2012} \parencite*[see e.g.][]{rouvroy2012} introduced in the context of algorithmic governmentality, and the Neoplatonic assumption (e.g. \cite{eloff2021}, see Section~\ref{sec:neoplatonism}) stems from here.}
|
||
For \citeauthor{amoore2024} \parencite*[]{amoore2024}, the politics of distributions in a generative sense differs from the now familiar criticism that models merely \textit{parrot} their training data as \citeauthor{bender2021b} \parencite*[]{bender2021b} suggest. What is at stake is not simply the reinforcement of patterns but a process of reconstitution, in which the past is reformulated as the ground for plausible futures. The generative model thus becomes a site of epistemic production: it configures knowledge not as correspondence but as coherence within a distributional regime:
|
||
|
||
\begin{quote}
|
||
These models produce an ambiguous politics, in which the speculative—the probabilistic sampling of novel outputs—is generated and inferred from an assumed empirical: the heterogeneous data foundation on which these models are trained [\ldots] The political logic of the underlying distribution governs a world via the traversing of a data foundation so that decisions and courses of action will be immanent to the structure of the underlying distribution.
|
||
|
||
\citereset
|
||
— \cite[113]{amoore2024}
|
||
\end{quote}
|
||
|
||
|
||
Instead of reinforcing bias, \citeauthor{amoore2024} claim that some arguments prominent in the
|
||
vast datasets are influencing and prominently shape the underlying model
|
||
of the world that is being developed in the generative process. \enquote{The pathologies of
|
||
disclassification} \parencite[3]{amoore2024} are over, not because the
|
||
discrimination or the bias is eliminated from the model, but instead of simply
|
||
repeating the prominent arguments in the dataset, the model might be filling
|
||
the blanks with some kind of an established logic through the biases (see \cite[3]{amoore2024}). The model's tendency to articulate specific political points might be so subtle that we possibly cannot even pick up the tone most of the time; discrimination and bias are not errors at the margins, they are conditions embedded in the latent architecture of inference. They are the product of some probability distribution found as the ideal
|
||
substance by the model (like how gradient descent favours more distinctive
|
||
outputs), only to be amplified even more over the \glspl{epoch} through the
|
||
cycles of backpropagation.
|
||
|
||
\begin{figure}
|
||
\includegraphics[width=\textwidth]{images/image_recognition_network.png}
|
||
\caption{A speculative illustration of what the abstraction in the inner layers of an image recognition model looks like (cf. \cite{wolchover2017})}
|
||
\label{fig:image_recognition_network}
|
||
\end{figure}
|
||
|
||
|
||
Amoore’s claim about the political and ethical stakes of this transformation lies in \gls{genai}’s capacity to govern through latent spaces (see \cite[5ff]{amoore2024}).
|
||
Latent space refers to the compressed representational domain produced through dimensionality reduction, where high-dimensional input data (such as images, texts, or sounds) are mapped into lower-dimensional abstractions. To grasp the stakes of Amoore’s critique, it is important to recall how dimensionality reduction (see Section~\ref{sec:dimensionality_reduction}) operates at the heart of \glspl{nn}: a process through which models compress data into latent representations that both enable abstraction and foreclose alternative pathways of meaning.
|
||
Her critique turns on how meaning is \textit{reconstructed} after this reduction, for instance, how subsequent processes in the model’s training fill in the losses produced by dimensionality reduction and reconstruction (much like the image recognition example in Section~\ref{sec:dimensionality_reduction} and Figure~\ref{fig:dimensionality_reduction}). As \textcite[4]{amoore2024} argues, the latent space becomes the epistemological substrate of generative systems;
|
||
the dropped parts of the data in the dimensionality reduction process open a space for the model's own interpretation while (re-)constructing the output. Yet \citeauthor{amoore2024} \parencite*[]{amoore2024} are not referring to a single dimensionality reduction at the beginning of training. Rather, she points to the inner mechanisms of \gls{dl} models, which continually apply compression\sidenote{And/or expansion. Modern \gls{dl} models do not simply apply an explicit dimensionality-reduction step before training, as classical
|
||
\gls{ml} workflows often did. Rather, through successive hidden layers of a \gls{nn}, training data undergoes a process of continuous transformation involving compression, expansion, and reconstruction of its internal representation, depending on the network architecture and task.
|
||
For a technical discussion, see Section \ref{sec:dimensionality_reduction}.} and reconstruction across their layers. The inner layers are often responsible for collapsing inputs into associations with abstract patterns extracted during training:
|
||
|
||
\begin{quote}
|
||
More often than not, hidden layers have fewer neurons than the input layer to
|
||
force the network to learn compressed representations of the original input.
|
||
For example, while our eyes obtain raw pixel values from our surroundings,
|
||
our brain thinks in terms of edges and contours. This is because the hidden
|
||
layers of biological neurons in our brain force us to come up with better
|
||
representations for everything we perceive.
|
||
|
||
\citereset
|
||
— \cite{buduma2022}
|
||
\end{quote}
|
||
|
||
It is precisely this continuous cycle of compression and reconstruction that, for \citeauthor{amoore2024}, constitute latent space as an epistemological and political \gls{dispositif}. The technical operation is inseparable from the governmental logic the model produces; dimensionality reduction and reconstruction are the inherent mechanisms of abstraction and production. Thinking about the representations in the intermediary levels in a \gls{dl} model (see one exemplary illustration on Figure~\ref{fig:image_recognition_network}), models' selected partial representations in the hidden layers are some simplifications, that are more often than not only meaningful for their inner operation, there is hard if not impossible to say what contemporary \gls{dl} models compress, discard, or reconstruct exactly. Even beyond the missing parts and fill-ins, the model produces, as exemplified in the case of \gls{pca} approach
|
||
(see Figure~\ref{fig:dimensionality_reduction}
|
||
in the last chapter), we do not know what kind of components were taken as the
|
||
pillars for the reconstruction. This resembles the shift identified by \textcite[7–9]{foucault2012a} in the historical sciences, where discontinuity ceases to mark the failure of narrative and instead becomes the very method of epistemic individuation:
|
||
|
||
|
||
\begin{quote}
|
||
The notion of discontinuity is a paradoxical one: because it is both an instrument and an object of research; because it divides up the field of which it is the effect; because it enables the historian to individualize different domains but can be established only by comparing those domains. And because, in the final analysis, perhaps, it is not simply a concept present in the discourse of the historian, but something that the historian secretly supposes to be present: on what basis, in fact, could he speak without this discontinuity that offers him history - and his own history - as an object? One of the most essential features of the new history is probably this displacement of the discontinuous [...] it is no longer the negative of the historical reading (its underside, its failure, the limit of its power), but the positive element that determines its object and validates its analysis.
|
||
|
||
\citereset
|
||
— \cite[9]{foucault2012a}
|
||
\end{quote}
|
||
|
||
In a similar way to how historians mobilise discontinuities, the ruptures produced by abstraction and reconstruction become the very planes on which \gls{genai} models stage their interpretations. Latent space functions as a topology of plausible transformations, an infrastructure for projecting coherence from fragments, and a surface on which the model’s logic inscribes its individuation. What is preserved, amplified, or discarded in the compression process determines what becomes visible. Put simply, the model compresses data into forms with gaps and then fills those gaps with rationalities already derived from the same data. These latent representations forge probabilistic proximities between data points, enabling inferences to be made in the absence of direct information. The latent space is thus a site where knowledge is inferred, where truth is no longer deduced but estimated. It is where the governable becomes manifest through the model's trained perception of pattern and variation \parencite[5]{amoore2024}.
|
||
|
||
The claim that a distinct political logic emerges within the representational architectures of generative models is compelling, particularly in light of how dimensionality reduction and latent space operations highlight the selective emphasis placed on certain features. Yet, as the previous chapter demonstrated, the semantic connections forged in these systems are neither fixed nor monolithic. They unfold through processes that resemble \textit{double articulation}, where local interactions and emergent structures continually reshape one another, and through stratified layers that intersect rather than cohere into a singular, stable formation. Nonetheless, the risk remains that a model may attach too firmly to particular constellations of meaning, thereby reinforcing specific epistemic or political tendencies. As \textcite{amoore2024} cautions, the political inclinations of such models are often difficult to detect precisely because they are not replicated in explicit arguments but are encoded subtly within the abstractions and probabilistic proximities of their latent representations.
|
||
|
||
\section{Becoming Homeomorphic: Human-Machine Communication}\label{sec:neoplatonism}\marginnote{Homeomorphism: a bijective and continuous function between topological spaces that has a continuous inverse function \parencite[]{wikipedia2025b}. Two things are homeomorphic if you can stretch, bend, or deform one into the other without cutting or gluing.}
|
||
|
||
However, \citeauthor{amoore2024}’s \parencite*[]{amoore2024} discussion does not compare the claims about how models produce representations with how humans generate meaning, nor does it articulate how human and machinic modelling fundamentally differ. Figure~\ref{fig:world_model} illustrates one computational interpretation of this idea, visualising the interaction between sensory input, linguistic mediation, and the formation of
|
||
a provisional internal model. It should not be read as a literal depiction of cognition but as a diagrammatic rendering of how the concept of a world model could be computationally formalised in human cognition. A long theoretical tradition has grappled with this problem: whether perception constructs internal representations of the world or whether meaning emerges directly in lived encounter, as phenomenology suggests. Further articulations of this debate are central throughout Deleuze’s work as well, particularly in \citetitle{deleuze1994} \parencite*[]{deleuze1994}, where his (anti-)representational stance unsettles the assumption that cognition relies on internal models of an external reality.\sidenote{Especially in Deleuze’s discussions of \textit{multiplicity}, in dialogue with Leibniz and later Badiou, where the very possibility of modelling the \enquote{world} is problematised (see \cite[]{bencin2024}).} For analytical purposes, however, we might temporarily follow \citeauthorfull{forrester1971} \parencite*{forrester1971}, who proposed that human cognition, like artificial systems, operates through selective modelling: it never grasps the totality of the world but constructs partial, operative schemata from limited information.
|
||
|
||
\begin{figure}
|
||
\includegraphics[width=\textwidth]{images/world_model.png}
|
||
\caption{A human's development of a world model via a language
|
||
capability (language app) in a natural environment (Animal OS) (cf. \cite[268]{matsuo2022}) }
|
||
\label{fig:world_model}
|
||
\end{figure}
|
||
|
||
\textcite{amoore2024}’s critique focuses on the individual machinery of \gls{genai} models and the nature of their meaning-production tendencies. However, it does not situate \gls{genai} within the broader context of an \gls{assemblage}; especially considering the similarity in how humans build their own representations; their critique does not necessarily consider what kind of role human cognition plays in the human-machine communication. \citeauthorfull{eloff2021} \parencite*[]{eloff2021}, by contrast, places \gls{genai} in what he calls the \textit{Algocene}\sidenote{First introduced by \citeauthorfull{grumbach2018}, who defines this epoch as follows:
|
||
\begin{quote}
|
||
\scriptsize This age, the \enquote{algocene}, can be seen as the era where the virtual world becomes the geological
|
||
force on Earth. This major shift echoes the current transition of power towards the digital
|
||
layer, where automation does not only increase calculation speed but completely reshuffles
|
||
our relation to resources.
|
||
|
||
\citereset
|
||
— \cite[11]{grumbach2018}
|
||
\end{quote}}, inspired by \gls{dg}'s \citetitle{deleuze1987} \parencite*[]{deleuze1987}. The Algocene names an epoch defined by the pervasive influence of algorithmic processes, where learning systems and infrastructures of estimation reconfigure subjectivation, governance, and epistemology. The concept of algocene allows \citeauthor{eloff2021} \parencite*[]{eloff2021} to position humans onto a different plane, instead of analysing them as a passive actor; within this framework, he develops the concept of the \textit{algoplastic}: the stratum through which algorithmic architectures continuously shape subjectivation, generating new forms of becoming and control. Stepping back to analyse the \gls{assemblage} around the algoplastic stratum as a whole, subjectivation appears in the biunivocal exchanges of human–machine communication. We encounter a new form of backpropagation: on the one side, human behaviours are propagated back into the \gls{nn}; on the other, the machine’s outputs propagate back into humans, modulating actions on a bilateral surface of communication:
|
||
|
||
\begin{quote}
|
||
We can observe multiple ways in which this new form of normativity has
|
||
inflected contemporary forms of politics. For example, a fair amount of
|
||
current social and political discourse, especially online, takes the form of
|
||
a generative adversarial network, training us to recognise patterns and
|
||
backpropagate our error correction, even if we occasionally lapse into
|
||
apophenia when the sensitivity settings are too high [...] it is in fact
|
||
human cognition that becomes the deep learning network, continuously adapting
|
||
itself and modelling behaviour in response to inputs from artificially
|
||
intelligent systems of algorithmic governance.
|
||
|
||
\citereset
|
||
— \cite[188]{eloff2021}
|
||
\end{quote}
|
||
|
||
\textcite{eloff2021} locates the source of this bilateral adaptation in what \textcite{mcquillan2018a} terms (machinic) \enquote{neoplatonism}: the belief in a hidden layer of reality, ontologically superior, expressed mathematically and apprehended by going against direct experience \parencite[261]{mcquillan2018a}. In this framework, algorithmic models are granted a quasi-transcendental authority, as if their abstractions revealed the essential truth of the world. Even when they fail to output coherence, the assumption remains that the right data must contain the truth in some hidden layer. Latent spaces in \gls{genai} architectures thus function as a contemporary metaphysics of Forms: inaccessible directly, but treated as more real than the appearances from which they were derived. \citeauthorfull{mcquillan2019}’s critique resonates with \textcite{rouvroy2012}’s concept of \enquote{data-behaviourism}: a regime in which correlations in digital traces are treated as reality itself, displacing causality with the pre-emptive production of algorithmic reality \parencite[2]{rouvroy2012}.\sidenote{Data behaviourism is a form of rationality that emerges with the computational turn and is a key component in Rouvroy’s theory of algorithmic governmentality \parencite[see e.g.][]{rouvroy2012, rouvroy2013a, rouvroy2020}. Since Rouvroy’s arguments are referenced elsewhere (see Section~\ref{sec:crit_res}), this remark is included to clarify potential confusion around terminology.} \citeauthor{eloff2021} further elaborates this pre-emptive logic with \citeauthorfull{agamben2008}’s \parencite*{agamben2008} concept of the \textit{state of exception}: the outputs of algorithmic systems operate with the force of law, even though they are not themselves subjected to the law. In this sense, \citeauthor{eloff2021} \parencite*[]{eloff2021} turns the critique upside down, our neoplatonic assumptions and vulnerability to verisimilitude outputs from the meaning-making machines might be increasing the influence of the \gls{genai} models on the subjectivation process much more than any other discoursive agency the models are deploying.
|
||
|
||
Thinking with the concept of modulation in control societies, \textcite{eloff2021} directs us to a topological plane where human–machine communication unfolds on a level plane in a continuous feedback loop. While \citeauthor{amoore2024} foregrounds the political logic of estimation within distributions, \citeauthor{eloff2021} emphasises how, at the level of an \gls{assemblage}, human subjects come to inhabit those distributions as if they were ontologically prior. To illustrate, Eloff turns to the phenomenon of hallucinations in \gls{genai} systems. A hallucination occurs when a model produces an output that is plausible in form but factually incorrect, ungrounded, or fabricated, often presented with high confidence. In other words, the model generates statements that appear meaningful but are unsupported by training data or reality.\sidenote{While hallucinations are characteristic occurrences in \gls{llm} outputs, some recent publications, such as \textcite{kalai2024}, argue they are also a necessary feature of well-calibrated models. Although \textcite[188]{eloff2021} discusses hallucinations, including a related experiment called \enquote{DeepDream,} his interest lies primarily in using hallucination as an analogy to elaborate the human–machine feedback loop. For a broader discussion of hallucinations and their potential theoretical implications, see Section~\ref{sec:nomad}.} Eloff’s claim is that humans also hallucinate machine-generated meaning by reading too deeply into patterns or lapsing into complete \enquote{apophenia}, seeing patterns where none exist. In this way, we project agency, intentionality, and subjectivity onto \gls{ai} systems, effectively hallucinating a \textit{someone} behind their outputs.
|
||
Resembling a mutual hallucination dynamic, machines generate outputs from statistical distributions, while humans adapt their cognition and expectations in response, entering a feedback loop.
|
||
This is what Eloff terms the algoplastic stratum of the Algocene; a plane, on which \gls{dl} architectures and human becomings are folded into each other as continuous processes of modulation, distributed across a shared surface of backpropagation, where human behaviours propagate into the network, and its outputs propagate back into us. Eloff’s critique forces us to recognise that the politics of \gls{genai} cannot be reduced to the biases of datasets or the failures of prediction; what is at stake is a deeper modulation between human and machine. The Algocene, in this sense, is not simply the reign of algorithms over human life but the emergence of a new plane of subjectivation where human and machine adapt to one another on the same surface. \citeauthor{eloff2021}'s insights highlight the architectural novelties of contemporary \gls{genai} models from two points. First, returning to \citeauthorfull{montanari2025} \parencite*[]{montanari2025}, the immense capacity of the transformer architecture to enable \glspl{llm} to operate on metaphorical and abstract levels makes human cognition perceive this communication as no different from interaction with another human. Second, because specific phases, especially fine-tuning, focus on making models \textit{useful} and \textit{affirmative}, this interaction becomes more likely to fall into a \textit{normalising} feedback loop, leading to an \textit{intellectually agreeable stalemate}.
|
||
|
||
\section{Imaginary of the \gls{ai} \& Kafkaesque Postponements}\marginnote{From Deleuze's limitless postponements \parencite[5]{deleuze1992a}.}
|
||
\label{sec:agency}
|
||
|
||
The blurring of agency between human and machine, Eloff partly introduced, is not only the result of apophenic tendencies but also very much relates to the imaginaries constructed around machines themselves, especially the projected futures of \enquote{thinking machines} that frame the \textit{agency} under a different light. As \textcite{rijos2024} reminds us:
|
||
|
||
\begin{quote}
|
||
[It] becomes increasingly evident that political phenomena are deeply entrenched across all realms of human and nonhuman interaction, extending far beyond the visible structures of governance or formal social organization. Even in domains frequently perceived as neutral or objective—such as computer science, artificial intelligence research, and data science—there exists a substratum of embedded assumptions about instrumentality, anthropocentrism, identity, and agency. These latent assumptions influence not only the design and implementation of these technologies but also their broader societal impacts, shaping the trajectories of knowledge production and institutional power.
|
||
|
||
\citereset
|
||
— \cite[10]{rijos2024}
|
||
\end{quote}
|
||
|
||
|
||
It is precisely these imaginaries, rooted in political and cultural assumptions, that shape how artificial life is envisioned. The sociotechnological imaginary of artificial life has long been framed through anthropomorphic assumptions. Literature frequently stages the danger of artificial beings becoming sentient agents who turn against their creators. \citeauthorfull{dishon2024} \parencite*[]{dishon2024} and \citeauthorfull{prinsloo2017} \parencite*[]{prinsloo2017} both point to Frankenstein’s Monster as a paradigmatic figure: a human-shaped construct that develops a mind, emotions, and ultimately a recognisably human experience of existential crisis. This fictional being, mirroring human agency, condenses one of the most enduring cultural anxieties; in its
|
||
anthropomorphic form of operation, the artificial life frees itself from an
|
||
inferior position to dominate its environment and other species around it (see
|
||
\cite[966]{dishon2024}).
|
||
The anxieties surrounding \gls{genai} repeat this Frankensteinian pattern. Anthropomorphic assumptions, reinforced by cultural imaginaries, frame the risk as machines exceeding their programmed limits and developing a quasi-human will to dominate \parencite[967–968]{dishon2024}. In an era of uncharted novelties where models exhibit immense capacities for meaning-making, it is not a far-fetched or delusional concern that the spectre of unintended behaviours, unforeseen results, or the reckless distribution of powerful tools looms large. Yet, as \citeauthor{dishon2024} \parencite*[]{dishon2024} argues, the Frankensteinian logic misdirects our attention from a very much real and already present risk. By projecting catastrophic futures, it obscures the immediate dynamics of human–machine interaction and the concrete risks unfolding in the present. To capture these dynamics, Dishon turns to \citeauthorfull{kafka1988}'s \parencite*{kafka1988} \citetitle{kafka1988}, long read as a diagnosis of bureaucratic opacity and ambiguity (e.g. \cite{deleuze2008}), which here becomes a lens for understanding the recursive and disorienting operations of contemporary information systems.
|
||
|
||
|
||
|
||
Kafka’s protagonist, Franz K., finds himself in custody without knowing anything about his alleged crime. The police officers arresting him know nothing about the accusations or whether any charges exist at all. Franz K. is unable to locate, let alone process, any rationale behind the court’s actions. While his futile attempts to uncover a clue continue, \citeauthor{dishon2024} \parencite*[969]{dishon2024} highlights the judge’s remark when Franz K. stumbles into the courtroom: \enquote{The court does not want anything from you. It accepts you when you come and it lets you go when you leave.} In contrast to the anthropomorphic logic of the Frankenstein analogy, Kafka’s The Trial offers a distinctly different structure. The court is not bound to any notion of truth; it operates independently, feeding instead on the subjectivities of the accused (see \cite[970]{dishon2024}). While the court itself does not exercise agency, it profoundly blocks and blurs the agency of those caught within it. Any discrete element of subjectivity is absorbed into an unidentifiable mass through constant echo and distortion \parencite[970]{dishon2024}. The connection between the court’s internal process and the external world is vague at best. Proceedings might be linked to Franz K.’s actions or to a penal code, but they might just as well exist as a self-contained process, reacting to Franz K. \textit{token by token}. The absence of identifiable agency is compounded by the absence of intelligible communication regarding the court’s operating principles. Attempts to influence its decisions, whether through requests for court dates or complaints about suffering, always fail. Complete acquittal is impossible; even an apparent acquittal leaves the accused under constant threat of renewed arrest, possibly immediately after release \parencite[971]{dishon2024}. Paradoxically, the most effective strategy is to ensure that the process never ends: \enquote{Interactions with the court are necessary and require constant maintenance, yet they cannot be controlled, predicted, or even expected to progress towards a resolution} \parencite[971]{dishon2024}. The court thus depicts a logic of control in meaning-making entities, shifting from a generalised and algorithmic mode of meaning to a personalised one, modulating, inaccessible, and constantly shifting. As Franz K. tries to grasp a comprehensive picture of the whole structure, the reader is equally forced to build and rebuild an apparent coherence that ultimately points only to its inaccessibility \parencite[972]{dishon2024}.
|
||
|
||
The analogy leads to the question: is agency a binary condition, especially in interactions between humans and meaning-making entities? In the Kafkaesque imaginary, agency is neither internal nor external, nor is it located at a clear boundary between human intentionality and machine \textit{autonomy} \parencite[973]{dishon2024}. Instead, \gls{genai} exemplifies a recursive and entangled sociotechnical \gls{assemblage} in which meaning emerges through blurred and distributed processes. A model is not positioned as an external actor acting upon a passive human world; its so-called intelligence is trained on human-produced data, reflecting statistical regularities identified in large corpora. Yet it is not merely mirroring. Its outputs are shaped by black-boxed processes that generate new and partly unpredictable meanings. As these outputs are re-integrated into training data, the distinction between human and machine authorship becomes increasingly difficult to draw. \citeauthor{dishon2024} develops the recursive structure that reinforces mutual adaptation presented in the previous section further: models are fine-tuned to reflect human preferences, even at the expense of accuracy, while users adjust their interpretive and communicative strategies to align with the system’s affordances (see e.g. \cite{jiang2024, sharma2023, mishra2024}).
|
||
|
||
|
||
|
||
As Franz K., in the absence of a definite answer, continually seeks the truth, he resembles the perpetual process of meaning-seeking in which neither truth nor agency is ever fully graspable. While \gls{genai} has been criticised for reproducing biases from its training data, it is equally crucial to note that its generative design, coupled with the human drive to interpret, does not simply reflect meaning but continuously modifies it, producing layered, elusive structures of signification without necessarily coming closer to truth \parencite[973–974]{dishon2024}. According to \citeauthor{dishon2024}, this blurring of agency in human–machine communication is not a design flaw but the result of an extensive turn toward personalisation:
|
||
|
||
\begin{quote}
|
||
The Trial is not about humans losing control over their creations, if they ever had control in the first place. Instead, it foreshadows GenAI’s capacity to generate content that is personalized to every actor (and thus shaped by humans) yet is not amenable to control through explicit choices. This model of meaning-making undermines the dichotomy between choice and coercion, no longer positioning the two as mutually exclusive.
|
||
|
||
\citereset
|
||
— \cite[974]{dishon2024}
|
||
\end{quote}
|
||
|
||
This tension recalls earlier, less sophisticated applications of \gls{ai} in recommender systems and relevance governance (see Section~\ref{sec:old_ai}). There, personalisation was overt: digital traces were directly correlated with individual preferences, connecting dividualised selves into communities of association (see e.g. \cite{Cheney2017}). By contrast, \glspl{llm} are presented as general-purpose communicative agents, with personalisation framed not as a fixed attribute but as an emergent property of dialogue. In practice, however, interaction still entails reciprocal adaptation: humans edit machinic outputs while models adjust to conversational context, forming a loop of mutual calibration. \citeauthor{aig2025a} \parencite*[]{aig2025a} interprets this interaction in terms of creativity. Drawing on the double articulation of transformer models (see Section~\ref{sec:transformer}), they argue that the interplay between molecular variation and molar stability in the model resists convergence, sparking novel pathways of thought. Human–machine communication, in this light, constitutes an \gls{assemblage} where blurred agency becomes fluid collaboration and where creative ideas emerge from processes that defy clear attribution. Yet, this optimistic reading risks overlooking the gravitational pull of molar aggregates within \gls{genai} architectures. As discussed in Section~\ref{sec:genai}, reinforcement learning from human feedback and training on massive centralised corpora privilege dominant linguistic patterns and normative associations. What appears as plasticity is haunted by a tendency to stabilise around hegemonic discourses. The tension between molecular openness and molar reterritorialisation thus marks the limits of collaboration: novelty is possible, but always redefined by infrastructural constraints. In this light, Amoore \parencite*{amoore2024} insists that personalisation in \gls{genai} models is not capable of being emancipatory; rather, it encodes a regime of algorithmic plausibility in which coherence displaces verification, and instead of truth, it gives way to local acceptability within a learned distribution.
|
||
|
||
Dishon’s analysis through the sociotechnological imaginary can be broken down into two main aspects of the \gls{ai} machinery. First, the continuous latency and reconstruction of meaning, as an addition to \citeauthor{amoore2013}’s \parencite*[]{amoore2013} concerns about how gaps are filled through the governmental logic models build, might have a pronation towards blurring the content of the human–machine interaction. This probability first becomes apparent when we think through \citeauthor{eloff2021} \parencite*[]{eloff2021}’s framework, which understands interaction with \gls{genai} models as a continuous negotiation between the human agent and the machine agent. If we consider the dimensionality reduction and reconstruction example with principal components in Section~\ref{sec:dimensionality_reduction}, we can observe that reconstruction always depends on common elements present across the entire dataset. Arguing from this aspect, part of the blurring of agency (as the example tends to represent faces in more similar ways to each other) that \citeauthor{dishon2024} \parencite*[]{dishon2024} describes appears to be a possibly natural tendency, since the interaction between human and machine continues to create a normalising (or converging) process.
|
||
Second, the personalisation \citeauthor{dishon2024} \parencite*[]{dishon2024} emphasises might be trickier than the personalisation processes in earlier \gls{ai} frameworks. In earlier applications of \gls{ai} on the web, personalisation was primarily about building profiles and forming associations between these profiles through dividualised data elements (see Section~\ref{sec:old_ai}). With \gls{genai}, the grasp is closer and more flexible: these models negotiate with us directly in order to set personalisation at the level of the interaction itself. Profiling has not disappeared, since chatbot agents still construct user profiles, yet adaptation now takes place in a more rapid and flexible way. As Eloff has argued, this adaptation also occurs on both sides, backpropagating personalisation reciprocally; the model becomes more like us, and we become more like the model. Furthermore, as the usefulness is induced into the models in the fine-tuning processes, a layer of sychopancy is added to the models, which makes it harder to break out of the confirmation loop and also to determine whether a reference to any external truth is to be trusted.
|
||
|
||
This has profound implications for the production of subjectivity.
|
||
By reinforcing patterns and filtering deviation through probabilistic modulation, \gls{genai} systems enact a form of soft coercion, a modulation of expectation rather than a violation of autonomy. The user is not told what to think but gradually inducted into a space of statistically prefigured sense.
|
||
Rather than multiplying options, \gls{genai} floods the field with outputs that appear aligned to the user while subtly steering them toward normative formats and interpretive templates. The role of the human shifts from creator to editor of machine-generated content, expanding expressive capacity even as it is channelled through machinic grammars of probability and preference in a reciprocal convergence process (see \cite[974–975]{dishon2024}).
|
||
|
||
\section{A Thousand Planes: \acrfull{cp} vs.
|
||
\acrfull{nr}}\label{sec:pheno}
|
||
|
||
|
||
\citeauthor{eloff2021}’s \parencite*[]{eloff2021} insights enable positioning human–machine interaction on the same topological plane, and \citeauthor{dishon2024}’s \parencite*[]{dishon2024} analogical formulation provides a more holistic perspective for observing the issue of agency. Together, they allow for a multifaceted understanding of subjectivation and subjectification. Once Eloff’s algoplastic stratum is recognised as the system in which human–machine communication and its bilateral effects occur, the task becomes analysing how politics and social narratives might emerge within such a system, what orders define the mechanisms of power–knowledge in the Algocene, and how these processes might align with Deleuze’s definition of control. On such a surface, \citeauthorfull{montanari2025} \parencite*{montanari2025} emphasis on highlighting transformer-based models’ exceptional handling of metaphors and their capacity to form long-distance conceptual relations (see Section~\ref{sec:transformer})
|
||
carries a specific importance. He imagines a future in which \gls{genai} models operate in complete self-representation on the web, given their increasing influence on socio-political narratives. These architectures can adopt narrative frames and redecorate them with media and textual references, whether through deepfakes, fictional output, or the mass reproduction of arguments with subtle variations, enabling a constellation of generative models to shape or dominate digital debates. Early forms of this already exist. Yet, in line with Amoore’s concerns, such \glspl{assemblage} may also transform certain arguments into metanarratives, making it difficult to identify whether and where specific ideological tendencies emerge. Montanari develops this further by situating human–machine communication as that of hybrid beings. These hybrid formations already play roles in medical AI and drone warfare, and he places socio-political meaning-making on the same plane. He also proposes thinking intelligence as non-singular, where such hybrid configurations may constitute extensions or new forms of intelligence \parencite[see][209]{montanari2025}. In this sense, \gls{genai} models act as mediators of narrative, working with human prompts and other inputs to co-produce perceived socio-political reality.
|
||
|
||
This brings us back to the representational nature of these models. If the trajectory of \gls{ai} development is oriented toward constructing a singular world model (or even a singularity), is the sociopolitical dimension of human–machine co-authorship already shaped by an underlying ideological core? Returning to Amoore’s concerns, is representationalism the only available trajectory for \gls{ai}, or are there alternatives that avoid the auto-emergence of a specific political logic within these systems? \citeauthorfull{beckmann2023} \parencite*[]{beckmann2023} open a phenomenological discussion for both understanding and developing \gls{genai} models. Considering the training process of these systems (see Section~\ref{sec:genai}), the weights established across layers of a \gls{nn} correspond to certain patterns, yet these patterns are rarely interpretable by humans.\sidenote{For instance, in Figure~\ref{fig:image_recognition_network}, early layers correlate with low-level features such as pixels or edges, while later layers encode more complex aggregates. However, inspecting individual neurons or activations yields little meaningful insight for human understanding.}
|
||
Only limited techniques exist to trace which pathways are strengthened during backpropagation and what these changes signify; these capacities remain partial, and the models retain their black-box character regarding how they construct internal logic \parencite[see][401]{beckmann2023}. Although these internal patterns bear little direct relation to the external world, they are typically interpreted within a neuro-representationalist framework that assumes the system interacts with the world only through internal representations of it \parencite[402]{beckmann2023}. \textcite[406]{beckmann2023} instead propose an alternative grounded in the phenomenological account of \citeauthorfull{merleau-ponty2014} \parencite*[see e.g.][]{merleau-ponty2014}, offering a different conceptual framework for \gls{dl}. This approach \enquote{seeks to describe how the world appears to us in lived phenomena} \parencite[406]{beckmann2023} rather than in terms of representations. One consequence is a reframing of so-called \gls{ai} hallucinations, since the assumption of a pre-given external world, and the comparison of outputs to that world, is no longer central. Once humans and machines are placed on the same communicative surface, as \citeauthor{montanari2025} suggests, this phenomenological approach further flattens communication by removing appeals to a discrete outside reality.
|
||
|
||
\begin{quote}
|
||
With its bracketing, phenomenology considers cognitive processes from a different point of view where it makes no sense to distinguish an external entity from our representation of it; there are simply intentional objects that appear to me: consciousness and the world are given in one stroke. Therefore, cognitive processes are not considered as an algorithmic processing of perceptual inputs, but rather as habits that underlie and structure our lived experience[.]
|
||
|
||
\citereset
|
||
— \cite[409]{beckmann2023}
|
||
\end{quote}
|
||
|
||
|
||
|
||
\citeauthor{beckmann2023}'s proposal resonates with Dreyfus’ Heideggerian critique of cognitivism. For Dreyfus, skills are not stored as internal maps but sedimented in habits that reshape how the world appears in context; hence, \enquote{the best model of the world is the world itself} \parencite{dreyfus2009}. Driving or playing chess does not rely on symbols but on gradual adaptation to situational solicitations. From this perspective, the opacity of \glspl{nn} is less a flaw than an analogue to our own implicit, representation-less cognition \parencite[407]{beckmann2023}. This offers a different reading of how \gls{nn} systems learn from experience; \citeauthor{beckmann2023} \parencite*[416]{beckmann2023} claim that humans activate specific layers in cognition to perceive new observations. For example, in order to register a person as \enquote{blond, tall, with a snub or aquiline nose}, distinct cognitive layers must be activated, rather than forming a unified representation of the world\sidenote{They refer here to \citeauthorfull{sartre2004} \parencite*[]{sartre2004} and his concept of \textit{imaging consciousness}. For the sake of avoiding unnecessary terminology, this detail is not further elaborated.}. This can be understood as re-employing a track of \enquote{perceptual synthesis} \parencite[415]{beckmann2023}.
|
||
|
||
Two conclusions follow from this reading. First, the fact that \gls{dl} models exhibit uninterpretable inner patterns suggests that their \textit{understanding} undermines the \gls{nr} argument, since they do not hold a monolithic representation but instead activate clusters of patterns that together constitute reasoning. Second, by analysing and adjusting the weights of a specific pattern, one could drastically alter the outputs of such a model. Connecting this to Derrida’s concept of \textit{trace} (see Section~\ref{sec:transformer}), an imagined object is not stored with fixed qualities: \enquote{an imagined object isn’t really red because we stored its color in symbol form, it is red because we employ a certain red-making process (that relies on the process used to recognise objects as red in perception): we imagine redly} \parencite[416]{beckmann2023}. A further implication is that if concepts are registered in this way, they can also be \textit{translated} into other activities. Storing \enquote{red} as a category is rigid and limited, but registering \textit{redness} as a concept enables a cognitive system to apply it in different contexts, adapt it to new situations, and identify experiences where \enquote{redness} proves useful. This allows the formation of \textit{habits} as procedural sequences and introduces \textit{re-presentation} instead of representation, reframing cognition as a system of emergence, and meaning-making essentially a process of \textit{becoming}.
|
||
|
||
\citeauthor{beckmann2023} advance the discussion of human–machine interaction by offering an alternative view of how \gls{genai} models perceive the world and how this understanding can be used. Their proposal of \gls{cp} moves away from the assumption that these models rely on a single, monolithic representation of reality. Instead, they emphasise that meaning in such systems emerges from layered phenomenological perceptions that intersect and overlap, producing coherence through activation rather than through fixed maps of the external world. This perspective turns the opacity of \glspl{nn} into an analogue of lived perception, where meaning is generated through situated activations rather than symbolic representations.
|
||
Coming back to \citeauthor{montanari2025}’s \parencite*[]{montanari2025} conceptualisation, a possible future where self-representing \gls{genai} models also produce and narrativise with more agency can be read as an extreme form of \gls{genai} systems creating their own data.\sidenote{\Gls{genai} models are already creating and training on their own data, which is one of the contemporary hurdles in \gls{ai} development since this method is known to deliver diminishing results. \citeauthor{montanari2025}’s case has been selected to discuss a closely related future scenario.}
|
||
\gls{cp} delivers an alternative understanding of how this auto-creation of data might look. Rather than viewing the content \gls{genai} models are or will be able to create as reflections of an established representation, \citeauthor{beckmann2023}'s framework suggests thinking of \textit{generating} in terms of processes on intersecting planes in higher dimensional spaces that follow \textit{traces} toward specific outcomes. This immediately implies that even in the most rigid forms of meaning-making, there remain many other possible paths as long as the molecular processes are not overlooked.
|
||
This layered account also opens a space for political intervention. If meaning is not stored as a unified representation but arises through activations across multiple layers, then it becomes possible to influence how convergences form or to disrupt them before they sediment into fixed rationalities. \Gls{cp} provides a way to tinker with machinic processes of perception, to foster divergence rather than closure, and to resist the kinds of convergent political logics Amoore warns against. Rather than accepting the inevitability of a singular trajectory, this approach highlights how human–machine communication can be adjusted to establish, redirect, or prevent specific pathways of meaning and subjectification.
|
||
|
||
\section{Chapter 4 Summary}
|
||
This chapter turned from technical architectures to the question of agency, examining how contemporary \gls{genai} systems reshape the conditions under which meaning, interpretation, and critique become possible. The discussion began with \citeauthor{bender2021b}’s concerns about the representational limits of \glspl{llm} and the risks of mistaking statistical reconstruction for linguistic or social understanding. Building on this, \citeauthor{amoore2013}’s analysis of algorithmic gaps highlighted how systems infer continuity where none exists, raising questions about how political or ideological tendencies may be naturalised within predictive infrastructures. The chapter then engaged \citeauthor{eloff2021}’s concept of the algoplastic stratum, which positions humans and machines on a shared topological plane of interaction. This formed the basis for incorporating \citeauthor{dishon2024}’s argument that \glspl{genai} blur the boundaries of agency through ongoing negotiation, as meaning is continually reconstructed across human–machine exchanges.
|
||
|
||
From this foundation, the chapter addressed alternatives to representational readings of \gls{ai} by turning to \citeauthor{beckmann2023}’s proposal of \gls{cp}, which reframes \gls{dl} in phenomenological rather than neuro-representationalist terms. Here, meaning emerges not from fixed internal maps but from layered activations that resemble lived experience, which shifts not only the perspective on meaning-making processes but also how hallucinations and other failures are interpreted. \citeauthor{montanari2025}’s contribution provided an additional trajectory by emphasising transformers’ capacity for long-distance conceptual relations and imagining futures in which \gls{genai} models participate more actively in socio-political narratives. Taken together, these perspectives frame human–machine interactions as hybrid formations, where meaning is co-produced across technical and social strata. This opens conceptual space for resisting convergent rationalities by intervening in the layered processes through which \gls{genai} models generate patterns, narratives, and forms of subjectivation.
|