Exhibiting Procedural Generation
Generative art toys are software applications that create aesthetically pleasing visual patterns in response to the users toying with various input devices, from keyboard and mouse to more intuitive and tactile devices for motion tracking. The “art” part of these toy objects might relate to the fact that they are often installed in art galleries or festivals as a spectacle for non-players that exhibits the unlimited generation of new patterns from a limited source code. However, the features that used to characterise generative arts as a new meditative genre, such as the autonomy of the algorithmic system and its self-organisation (Galanter 151), do not explain the pleasure of fiddling with these playthings, which feel sticky like their toy relatives, slime, rather than meditative, like mathematical sublime.
Generative algorithms are more than software tools to serve human purposes now. While humans are still responsible for the algorithmically generated content, this is either to the extent of the simple generation rules the artists design for their artworks or only to the extent that our everyday conversations and behaviours serve as raw material to train machine learning-powered generation algorithms, such as ChatGPT, to interpret the world they explore stochastically, extrapolating it in an equivalently statistical way. Yet, as the algorithms become more responsive to the contingency of human behaviours, and so the trained generation rules become too complex, it becomes almost impossible for humans to understand how they translate all contingencies in the real world into machine-learnable correlations. In turn, the way we are entangled with the generated content comes to far exceed our responsibility.
One disturbing future scenario of this hyper-responsiveness of the algorithms, for which we could never be fully responsible, is when machine-generated content replaces the ground truth sampled from the real world, leading to the other machine learning-powered software tools that govern human behaviour being trained on these “synthetic data” (Steinhoff). The multiplicities of human worlds are substituted for their algorithmically generated proxies, and the AIs trained instead on the proxies’ stochastic complexities would tell us how to design our future behaviours.
As one aesthetic way to demonstrate the creativity of the machines, generative arts have exhibited generative algorithms in a somewhat decontextualised and thus less threatening manner by “emphasizing the circularity of autopoietic processes” of content generation (Hayles 156). Their current toy conversion playfully re-contextualises how these algorithms in real life, incarnated into toy-like gadgets, both enact and are enacted by human users. These interactions not only form random seeds for content generation but also constantly re-entangle generated contents with contingent human behaviors. The toy-being of generative algorithms I conceptualise here is illustrative of this changed mode of their exhibition. They move from displaying generative algorithms as speculative objects at a distance to sticky toy objects up close and personal: from emphasising their autopoietic closure to “more open-ended and transformative” engagement with their surroundings (Hayles 156). (Katherine Hayles says this changed focus in the research of artificial life/intelligence from the systems’ auto-poietic self-closure to their active engagement with environments characterises “the transition from the second to the third wave” of cybernetics; 17.) Their toy-being also reflects how the current software industry repurposes these algorithms, once developed for automation of content creation with no human intervention, as machines that enact commercially promising entanglements between contingent human behaviors and a mixed-reality that is algorithmically generated.
Tool-Being and Toy-Being of Generative Algorithms
What I mean by toy-being is a certain mode of existence in which a thing appears when our habitual sensorimotor relations with it are temporarily suspended. It is comparable to what Graham Harman calls a thing’s tool-being in his object-oriented rereading of Heidegger’s tool analysis. In that case, this thing’s becoming either a toy or tool pertains to how our hands are entangled with its ungraspable aspects. According to Heidegger a hammer, for instance, is ready-to-hand when its reactions to our grip, and swinging, and to the response from the nail, are fully integrated into our habitual action of hammering to the extent that its stand-alone existence is almost unnoticeable (Tool-Being). On the other hand, it is when the hammer breaks down, or slips out of our grasp, that it begins to feel present-at-hand. For Harman, this is the moment the hammer reveals its own way to be in the world, outside of our instrumentalist concern. It is the hint of the hammer’s “subterranean reality”, which is inexhaustible by any practical and theoretical concerns we have of it (“Well-Wrought” 186). It is unconstrained by the pragmatic maxim that any conception of an object should be grounded in the consequences of what it does or what can be done with it (Peirce). In Harman’s object-oriented ontology, neither the hammer’s being ready to serve any purpose of human and nonhuman others – nor its being present as an object with its own social, economic, and material histories – explicate its tool-being exhaustively. Instead, it always preserves more than the sum of the relations it has ever built with others throughout its lifetime. So, the mode of existence that describes best this elusive tool-being for him is withdrawing-from-hand.
Generative art toys are noteworthy regarding this ever-switching and withdrawing mode of things on which Harman and other speculative realists focus. In the Procedural Content Generation (PCG) community, the current epicentre of generative art toys, which consists of videogame developers and researchers, these software applications are repurposed from the development tools they aim to popularise through this toy conversion. More importantly, procedural algorithms are not ordinary tools ready to be an extension of a developer’s hands, just as traditional level design tools follow Ivan Suntherland’s 1963 Sketchpad archetype. Rather, procedural generation is an autopoietic process through which the algorithm organises its own representation of the world from recursively generated geographies, characters, events, and other stuff. And this representation does not need to be a truthful interpretation of its environments, which are no other than generation parameters and other input data from the developer. Indeed, they “have only a triggering role in the release of the internally-determined activity” of content generation. The representation it generates suffices to be just “structurally coupled” with these developer-generated data (Hayles 136, 138). In other words, procedural algorithms do not break down to be felt present-at-hand because they always feel as though their operations are closed against their environments-developers.
Furthermore, considered as the solution to the ever-increasing demand for the more expansive and interactive sandbox design of videogames, they not only promise developers unlimited regeneration of content for another project but promise players a virtual reality, which constantly changes its shape while always appearing perfectly coupled with different decisions made by avatars, and thus promise unlimited replayability of the videogame. So, it is a common feeling of playing a videogame with procedurally generated content or a story that evolves in real time that something is constantly withdrawing from the things the player just grasped. (The most vicious way to exploit this gamer feeling would be the in-game sale of procedurally generated items, such as weapons with many re-combinable parts, instead of the notorious loot-box that sells a random item from the box, but with the same effect of leading gamers to a gambling addiction by letting them believe there is still something more.) In this respect, it is not surprising that Harman terms his object-oriented ontology after object-oriented programming in computer science. Both look for an inexhaustible resource for the creative generation of the universe and algorithmic systems from the objects infinitely relatable to one another thanks ironically to the secret inner realities they enclose against each other.
Fig. 1: Kate Compton, Idle Hands. http://galaxykate.com/apps/idlehands/
However, the toy-being of the algorithms, which I rediscover from the PCG community’s playful conversion of their development tools and which Harman could not pay due attention to while holding on to the self-identical tool-being, is another mode of existence that all tools, or all things before they were instrumentalised, including even the hammer, had used to be in children’s hands. For instance, in Kate Compton’s generative art toy Idle Hands (fig. 1), what a player experiences is her hand avatar, every finger and joint of which is infinitely extended into the space, even as they also serve as lines into which the space is infinitely folded.
So, as the player clenches and unclenches her physical hands, scanned in real-time by the motion tracking device Leapmotion, and interpreted into linear input for the generation algorithm, the space is constantly folded and refolded everywhere even by the tiniest movement of a single joint. There is nothing for her hands to grasp onto because nothing is ready to respond consistently to her repeated hand gestures. It is almost impossible to replicate the exact same gesture but, even if she does, the way the surrounding area is folded by this would be always unpredictable.
Put differently, in this generative art toy, the player cannot functionally close her sensorimotor activity. This is not so much because of the lack of response, but because it is Compton’s intention to render the whole “fields of the performer” as hyperresponsive to “a body in motion” as if “the dancer wades through water or smoke or tall grass, if they disturb [the] curtain as they move” (Compton and Mateas). At the same time, the constant re-generation of the space as a manifold is no longer felt like an autonomous self-creation of the machine but arouses the feeling that “all of these phenomena ‘listen’ to the movement of the [hands] and respond in some way” (Compton and Mateas).
Let me call this fourth mode of things, neither ready-to-hand nor present-at-hand, nor withdrawing-from-hand, but sticky-to-hand: describing a thing’s toy-being. This is so entangled with the hands that its response to our grasp is felt immediately, on every surface and joint, so that it is impossible to anticipate exactly how it would respond to further grasping or releasing. It is a typical feeling of the hand toying with a chunk of clay or slime. It characterises the hypersensitivity of the autistic perception that some neurodiverse people may have, even to ordinary tools, not because they have closed their minds against the world as the common misunderstanding says, but because even the tiniest pulsations that things exert to their moving bodies are too overwhelming to be functionally integrated into their habitual sensorimotor activities let alone to be unentangled as present-at-hand (Manning).
In other words, whereas Heideggerian tool-being, for Harman, draws our attention to the things outside of our instrumentalist concern, their toyfication puts the things that were once under our grip back into our somewhat animistic interests of childhood. If our agency as tool-users presupposes our body’s optimal grip on the world that Hubert Dreyfus defines as “the body’s tendency to refine its responses so as to bring the current situation closer to an optimal gestalt” (367), our becoming toy-players is when we feel everything is responsive to each other until that responsiveness is trivialised as the functional inputs for habitual activities. We all once felt things like these animistic others, before we were trained to be tool-users, and we may consequently recall a forgotten genealogy of toy-being in the humanities.
This genealogy may begin with a cotton reel in Freud’s fort-da game, while also including such things as jubilant mirror doubles and their toy projections in Lacanian psychoanalysis, various playthings in Piaget’s development theory, and all non-tool-beings in Merleau-Ponty’s phenomenology. To trace this genealogy is not this article’s goal but the family resemblance that groups these things under the term toy-being is noteworthy. First, they all pertain to a person’s individuation processes at different stages, whether it be for the symbolic and tactile re-staging of a baby’s separation from her mother, her formation of a unified self-image from the movements of different body parts, the child’s organisation of object concepts from tactile and visual feedbacks of touching and manipulating hands, the subsequent “projection of such ‘symbolic schemas’” as social norms, as Barbie’s and Ken’s, onto these objects (Piaget 165-166), or a re-doing of all these developmental processes through aesthetic assimilation of objects as the flesh of the worlds (Merleau-Ponty). And these individuations through toys seem to approach the zero-degree of human cognition in which a body (either human or nonhuman) is no other than a set of loosely interconnected sensors and motors. In this zero-degree, the body’s perception or optimal grip on things is achieved as the ways each thing responds to the body’s motor activities are registered on its sensors as something retraceable, repeatable, and thus graspable. In other words, there is no predefined subject/object boundary here but just multiplicities of actions and sensations until a group of sensors and motors are folded together to assemble a reflex arc, or what Merleau-Ponty calls intention arc (Dreyfus), or what I term sensor-actuator arc in current smart spaces (Ahn). And it is when some groups of sensations are distinguished as those consistently correlated with and thus retraceable by certain operations of the body that this fold creates an optimal grip on the rest of the field.
Let me call this enfolding of the multiplicities whereby “the marking of the ‘measuring agencies’ by the ‘measured object’” emerges prior to the interaction between two, following Karen Barad, intra-action (177). Contrary to the experience of tool-being present-at-hand as no longer consistently contributing to our habitually formed reflex arc of hammering or to any socially constructed measuring agencies for normative behaviors of things, what we experience with this toy-being sticky-to-hand is our bodies’ folding into the multiplicities of actions and sensations, to discover yet unexplored boundaries and grasping between our bodies and the flesh of the world.
Generative Art Toys as the Machine Learning’s Daydream
Then, can I say even the feeling I have on my hands while I am folding and refolding the slime is intra-action? I truly think so, but the multiplicities in this case are so sticky. They join to every surface of my hands whereas the motility under my conscious control is restricted only to several joints of my fingers. The real-life multiplicities unfolded from toying with the slime are too overwhelming to be relatable to my actions with the restricted degree of freedom. On the other hand, in Compton’s Idle Hands, thanks to the manifold generated procedurally in virtual reality, a player experiences these multiplicities so neatly entangled with all the joints on the avatar hands. Rather than simulating a meaty body enfolded within “water or smoke or tall grass,” or the flesh of the world, the physical hands scanned by Leapmotion and abstracted into “3D vector positions for all finger joints” are embedded in the paper-like virtual space of Idle Hands (Compton and Mateas). And rather than delineating a boundary of the controlling hands, they are just the joints on this immanent plane, through which it is folded into itself in so many fantastic ways impossible on a sheet of paper in Euclidean geometry.
Another toy relative which Idle Hands reminds us of is, in this respect, Cat’s Cradle (fig. 2). This play of folding a string entangled around the fingers into itself over and over again to unfold each new pattern is, for Donna Haraway, a metaphor for our creative cohabitation of the world with nonhuman others. Feeling the tension the fingers exchange with each other across the string is thus, for her, compared to “our task” in the Anthropocene “to make trouble, to stir up potent response to devastating events, as well as to settle troubled waters and rebuild quiet places” (Haraway 1).
Fig. 2: Nasser Mufti, Multispecies Cat's Cradle, 2011. https://www.kit.ntnu.no/sites/www.kit.ntnu.no/files/39a8af529d52b3c35ded2aa1b5b0cb0013806720.jpg
In the alternative, in Idle Hands, each new pattern is easily unfolded even from idle and careless finger movements without any troubled feeling, because its procedural generation is to guarantee that every second of the player’s engagement is productive and wasteless relation-making. In Compton’s terms, the pleasure of generative art toys is relevant to the players’ decision to trade the control they once enjoyed as tool users for power. And this tricky kind of power that the players are supposed to experience is not because of their strong grip, but because they give up this strong grip. It is explicable as the experience of being re-embedded as a fold within this intra-active field of procedural generation: the feeling that even seemingly purposeless activities can make new agential cuts as the triggers for some artistic creations (“Generative Art Toys” 164-165), even though none of these creations are graspable or traceable by the players.
The procedural algorithm as the new toy-being is, therefore, distinguishable from its non-digital toy relatives by this easy feeling of engagement that all generated patterns are wastelessly correlated with the players’ sensorimotor activities in some ungraspable ways. And given the machine learning community’s current interest in procedural generation as the method to “create more training data or training situations” and “to facilitate the transfer of policies trained in a simulator to the real world” (Risi and Togelius 428, 430), the pleasure of generative art toys can be interpreted as revealing the ideal picture of the mixed-reality dreamed of by machine learning algorithms. As the solution to circumvent the issue of data privacy in surveillance capitalism, and to augment the lack of diversity in existing training data, the procedurally generated synthetic data are now considered as the new benchmarks for machine learning instead of those sampled from the real world. This is not just about a game-like object for a robot to handle, or geographies of fictional terrains for a smart vehicle to navigate (Risi and Togelius), but is more about “little procedural people” (“Little Procedural People”), “synthetic data for banking, insurance, and telecommunications companies” (Steinhoff 8).
In the near future, as the AIs trained solely on these synthetic data begin to guide our everyday decision-making, the mixed-reality will thus be more than just a virtual layer of the Internet superimposed on the real world but haunted by so many procedurally generated places, things, and people. Compared to the real world, still too sticky like slime, machine learning could achieve an optimal grip on this virtual layer because things are already generated there under the assumption that they are all entangled with one another by some as yet unknown correlations that machine learning is supposed to unfold. Then the question recalled by this future scenario of machine learning would be again Philip K. Dick’s: Do the machines dream of (procedurally generated) electronic sheep? Do they rather dream of this easy wish fulfillment in place of playing an arduous Cat’s Cradle with humans to discover more patterns to commodify between what our eyes attend to and what our fingers drag and click?
Incarnated into toy-like gadgets on mobile devices, machine learning algorithms relocate their users to the zero-degree of social profiles, which is no other than yet-unstructured personal data supposedly responsive to (and responsible for regenerating) invisible arcs, or correlations, between things they watch and things they click. In the meanwhile, what the generative art toys really generate might be the self-fulfilling hope of the software industry that machines could generate their mixed-reality, so neatly and wastelessly engaged with the idle hands of human users, the dream of electronic sheep under the maximal grip of Android (as well as iOS).
References
Ahn, Sungyong. “Stream Your Brain! Speculative Economy of the IoT and Its Pan-Kinetic Dataveillance.” Big Data & Society 8.2 (2021).
Barad, Karen. Meeting the Universe Halfway: Quantum Physics and the Entanglement of Matter and Meaning. Durham: Duke UP, 2007.
Compton, Kate. “Generative Art Toys.” Procedural Generation in Game Design, eds. Tanya Short and Tarn Adams. New York: CRC Press, 2017. 161-173.
Compton, Kate. “Little Procedural People: Playing Politics with Generators.” Proceedings of the 12th International Conference on the Foundations of Digital Games, eds. Alessandro Canossa, Casper Harteveld, Jichen Zhu, Miguel Sicart, and Sebastian Deterding. New York: ACM, 2017.
Compton, Kate, and Michael Mateas. “Freedom of Movement: Generative Responses to Motion Control.” CEUR Workshop Proceedings, 2282, ed. Jichen Zhu. Aachen: CEUR-WS, 2018.
Dreyfus, Hubert L. “Intelligence without Representation: Merleau-Ponty’s Critique of Mental Representation.” Phenomenology and the Cognitive Sciences 1 (2002): 367-383.
Galanter, Philip. “Generative Art Theory.” A Companion to Digital Art, ed. Christiane Paul. Hoboken, NJ: Wiley-Blackwell, 2016. 146-180.
Haraway, Donna J. Staying with the Trouble: Making Kin in the Chthulucene. Durham: Duke UP, 2016.
Harman, Graham. Tool-Being: Heidegger and the Metaphysics of Objects. Chicago: Open Court, 2002.
———. “The Well-Wrought Broken Hammer: Object-Oriented Literary Criticism.” New Literary History 43 (2012): 183-203.
Hayles, Katherine N. How We Become Posthuman: Virtual Bodies in Cybernetics, Literatures, and Informatics. Chicago: U of Chicago P, 1999.
Manning, Erin. The Minor Gesture. Durham: Duke UP, 2016.
Merleau-Ponty, Maurice. The Visible and the Invisible. Ed. Claude Lefort. Trans. Alphonso Lingis. Evanston, IL: Northwestern UP, 1968.
Peirce, Charles S. “How to Make Our Ideas Clear.” Popular Science Monthly 12 (1878): 286-302.
Piaget, Jean. Play, Dreams and Imitation in Childhood. Trans. C. Gattegno and F.M. Hodgson. New York: W.W. Norton, 1962.
Risi, Sebastian, and Julian Togelius. “Increasing Generality in Machine Learning through Procedural Content Generation.” Nature Machine Intelligence 2 (2020): 428-436.
Steinhoff, James. “Toward a Political Economy of Synthetic Data: A Data-Intensive Capitalism That Is Not a Surveillance Capitalism?” New Media and Society, 2022.
