In this article, I explore the phenomenon of augmentation by questioning its representational nature and analyzing aesthetic modes of our interrelationship with the environment. How can senses be augmented and how do they serve as mechanisms of enhancing the feeling of presence? Media art practices offer particularly valuable scenarios of activating such mechanisms, as the employment of digital technology allows them to operate on a more subtle level of perception. Given that these practices are continuously evolving, this analysis cannot claim to be a comprehensive one, but rather aims to introduce aspects of the specific relations between augmentation, sense of proprioception, technology, and art.
Proprioception is one of the least detectable and trackable human senses because it involves our intuitive sense of positionality, which suggests a subtle equilibrium between a center (our individual bodies) and the periphery (our immediate environments). Yet, as any sense, proprioception implies a communicational chain, a network of signals traveling and exchanging information within the body-mind complex. The technological augmentation of this dynamic process produces an interference in our understanding of the structure and elements, the information sent/received. One way to understand the operations of the senses is to think about them as images that the mind creates for itself. Artistic intervention (usually) builds upon exactly this logic: representation of images generated in mind, supplementing or even supplanting the existing collection of inner images with new, created ones. Yet, in case of proprioception the only means to interfere with and augment these inner images is on bodily level. Hence, the question of communication through images (or representations) should be extended towards a more complex theory of embodied perception.
Drawing on phenomenology, cognitive science, and techno-cultural studies, I focus on the potential of biofeedback technologies to challenge and transform our self-perception by conditioning new pathways of apprehension (sometimes by creating mechanisms of direct stimulation of neural activity). I am particularly interested in how the awareness of the self (grounded in the felt relationality of our body parts) is most significantly activated at the moments of disturbance of balance, in situations of perplexity and disorientation. Projects by Marco Donnarumma, Sean Montgomery, and other artists working with biofeedback aesthetically validate and instantiate current research about neuro-plasticity, with technologically mediated sensory augmentation as one catalyst of this process.
Augmentation as Representation: Proprioception and Proprioceptive Media
Representation has been one of the key ways to comprehend reality. But representation also constitutes a spatial relation of distancing and separation: the spectator encounters an object placed in front of him, external to him. Thus, representation is associated more with an analytical, rather than synthetic, methodology because it implies detachment and division into parts. Both methods involve relation, yet in the case of representation there is a more distinct element of distance between the representing subject and represented object. Representation is always a form of augmentation: it extends our abilities to see the "other", otherwise invisible sides and qualities of the objects of reality. Representation is key to both science and art, yet in case of the latter, what is represented is not a (claimed) "objective" scheme of reality, but rather images of the imaginary, inner reality (even figurative painting always presents a particular optical and psychological perspective, to say nothing about forms of abstract art).
There are certain kinds of art (visual arts, music, dance, etc.) that deal with different senses and thus, build their specific representational structures. Proprioception is one of the senses that occupies relatively marginal position in artistic production (which is exactly because of the specificity of its representational nature and because it does not create a sense of an external object.
The term "proprioception" comes from Latin propius, or "one's own", "individual", and capio, cepi – "to receive", "to perceive". It implies a sense of one's self felt as a relational unity of parts of the body most vividly discovered in movement and in effort employed in it. The loss of proprioception usually means loss of bodily orientation and a feeling of one's body (Sacks 43-54). On the other hand, in case of additional stimulation and training of this sense (not only via classical cyber-devices, like cyber-helmets, gloves, etc. that set a different optics, but also techniques of different kinds of altered states of mind, e.g. through psychotropics, but also through architecture of virtual space and acoustics) a sense of disorientation that appears at first changes towards some analogue of reactions of enthusiasm, excitement discovery, and emotion of approaching new horizons. What changes is not only perception of external reality, but a sense of one's self: the self is felt as fluid, flexible, with penetrable borders.
Proprioception implies initial co-existence of the inner and outer space on the basis of originary difference and individuality/specificity of the occupied position. Yet, because they are related, the "external" and "other" already feels as "one's own", and this is exactly what causes the sense of presence. Among the many possible connections that the body, in its sense of proprioception, is always already ready for, only a certain amount gets activated. The result of proprioception is a special kind of meta-stable internal image. This image may not coincide with the optical, auditory, or haptic image. According to Brian Massumi,
proprioception translates the exertions and ease of the body's encounters with objects into a muscular memory of relationality. This is the cumulative memory of skill, habit, posture. At the same time as proprioception folds tactility in, it draws out the subject's reactions to the qualities of the objects it perceives through all five senses, bringing them into the motor realm of externalizable response. (59)
This internal image is not mediated by anything, though it depends directly on the relations between the parts. It cannot be grasped because it is by definition fluid and dynamic. The position in one point is replaced here by a position-in-movement (point-in-movement). "Movement is not indexed by position. Rather, the position is born in movement, from the relation of movement towards itself" (Massumi 179). Philosopher of "extended mind" Andy Clark notes that we should distinguish between a real body schema (non-conscious configuration) and a body image (conscious construct) (Clark). It is the former that is important to understand, and yet is the most challenging.
Due to its fluidity and self-referentiality, proprioception is not presentable to consciousness (the unstable internal image that it creates resides in consciousness but cannot be grasped and thus re-presented). A feeling/sense, it is not bound by sensible forms that would serve as means of objectification and externalization. As Barbara Montero observes, while the objects of vision and hearing, i.e. the most popular senses involved in the arts, are beyond one's body, sense of proprioception relates directly to the bodily sensation, it does not represent any external objects, but the sensory itself (231). These characteristics of proprioception help to reframe the question of augmentation as mediation: in the case of proprioception, the medium of sensation is the very relational structure of the body itself, irrespective of the "exteroceptive" (tactile) or "interoceptive" (visceral) dimensions of sensibility. The body is understood, then, as the "body without image,” and its proprioceptive effect can then be described as "the sensibility proper to the muscles and ligaments" (Massumi 58).
Proprioception in (Media) Art
One of the most convincing ways of externalization and (re)presentation of the data of proprioception is through re-production of its structure and its artificial enhancement with the help of technology. This can be achieved in at least two ways: by setting up situations and environments that emphasize self-perspective and awareness of perception, and by presenting measurements of bio-data and inviting into dialogue with them.
The first strategy may be connected to disorientation and shifted perspective that are created in immersive virtual environments that make the role of otherwise un-trackable, fluid sense of proprioception actually felt and cognized. These effects are closely related to the nuances of perception of space, for instance, to spatial illusion. Practice of spatial illusion in the arts traces its history as far back as Roman frescos, trompe l’oeil, as well as phantasmagorias, like magic lantern. Geometrically, the system of the 360º image is still the most effective in producing a sense of full immersion—either in spaces from panoramas, Stereopticon, Cinéorama to CAVE (Computer Augmented Virtual Environments), or in devices for an individual spectator’s usage, like a stereoscope, Sensorama and more recent Head Mounted Displays (HMD). All these devices provide a sense of hermetic enclosure and bodily engagement with its scenes (realistic or often fantastical). Their images are frameless and thus immeasurable (lack of the sense of proportion provokes feeling of disorientation), image apparatus and the image itself converge here into an almost inseparable total unity: field of vision is filled, and the medium becomes invisible (Grau 198-202; 248-255).
Yet, the constructed image is even more frameless and more peculiarly ‘mental’ in environments created on the basis of objectless or "immaterial" media, like light or sound; or in installations prioritizing haptic sensation and in responsive architectures, i.e. environments that transform physically in reaction to their inhabitants. The examples may include works by Olafur Eliasson that are centered around the issues of conscious perception and employ various optical and other apparata (mirrors, curved surfaces, coloured glass, water systems) to shift the habitual perspective and make one conscious of the subtle changes in the environment depending on one's position in space (there have been instances of spectators in Eliasson's installations falling down after trying to lean against an apparent wall that turned out to be a mere optical construct.).
Figure 1: Olafur Eliasson, Take Your Time, 2008. © Olafur Eliasson Studio.
In his classic H2OExpo project for Delta Expo in 1997, the Dutch architect Lars Spuybroek experimented with the perception of instability. There is no horizontal surface in the pavilion; floors, composed of interconnected elliptical volumes, transform into walls and walls into ceilings, promoting a sense of fluidity and making people respond by falling, leaning, tilting and "experiencing the vector of one’s own weight, and becoming sensitized to the effects of gravity" (Schwartzman 63). Along the way, specially installed sensors detect the behaviour of the ‘walker’ and send signals to the system to contribute further to the agenda of imbalance and confusion by changing light, image projection, and sound.
Figure 2: Lars Spuybroek, H2OExpo, 1994-1997. © NOX/ Lars Spuybroek.
Philip Beesley’s Hylozoic Ground (2010) is also a responsive environment filled by a dense organic network of delicate illuminated acrylic tendrils that can extend out to touch the visitor, triggering an uncanny mixture of delight and discomfort. The motif of pulsating movement was inspired by fluctuations in coral reefs and recreated via the system of precise sensors and microprocessors. This reference to an unfamiliar and unpredictable natural environment, which often makes us feel cautious and ultra-attentive, is a reminder of our innate ability of proprioception (a deeply ingrained survival instinct) and its potential for a more nuanced, intimate, emphatic and bodily rooted communication.
Figure 3: Philip Beesley, Hylozoic Ground, 2010. © Philip Beesley Architect Inc.
Works of this kind stimulate awareness of both the environment and one's own response to it. Inviting participants to actively engage with the space, they evoke reactions of self-reflexivity, i.e. the self becomes the object of its own exploration and (potentially) transformation.
Another strategy of revealing the processes of the "body without image" is through representing various kinds of bio-data, bodily affective reactions to certain stimuli. Biosignal monitoring technologies most often employed include EEG (Electroencephalogram), EMG (Electromyogram), GSR (Galvanic Skin Response), ECG (Electrocardiogram), HRV (Heart Rate Variability) and others. Previously available only in medical settings and research labs, many types of sensors (bio and environmental) now become increasingly available (bio-enabled products ranging from cardio watches—an instance of the "quantified self" trend—to brain wave-controlled video games). As the representatives of the DIY makers community put it: "By monitoring some phenomena (biofeedback) you can train yourself to modulate them, possibly improving your emotional state. Biosensing lets you interact more naturally with digital systems, creating cyborg-like extensions of your body that overcome disabilities or provide new abilities. You can also share your bio-signals, if you choose, to participate in new forms of communication" (Montgomery). What is it about these technologies besides understanding more accurately the unconscious and invisible signals?
The critical question in relation to biofeedback data is about the adequacy of the transference of the initial signal, about the "new" brought by the medium, as well as the ontological status of the resulting representation. These data are reflections of something real, yet themselves have a different weight, also providing the ground for all sorts of simulative methods and creation of mixed realities.
External representations, unlike internal, are often attributed a prosthetic nature that is treated as extensions of existing skills. Besides serving their direct purpose (for instance, maps give detailed picture of a distant location), these extensions provide certain psychological effects, such as disorientation, displacement, a shift in a sense of self and enhancement of the sense of presence. Artistic experiments with bio-data started in the 1960s most famously with employing the method of sonification. Among the pioneers were the composers Alvin Lucier, Richard Teitelbaum, David Rosenblum, Erkki Kurenemi, Pierre Henry, and others. Today's versions of biophysical performance may include not only acoustic, but also visual interpretation, as well as subtle narrative scenarios. An example can be Marco Donnarumma's Hypo Chrysos, a piece that translates visceral strain in sound and moving images. The title refers to the type of a punishing trial in one of the circles of hell in Dante's Divine Comedy: the eternal task of carrying heavy rocks is imitated by the artist-performer, while the audience can feel the bodily tension enhanced by sound and imagery. The state of the inner body is, thus, amplified, or augmented. The sense of proprioception experienced by the performer is translated into media perceivable by others. In this externalized form it can also be shared, i.e. released into a space of inter-subjectivity, where it receives other, collective qualities and is not perceived negatively, in terms of pressure.
Figure 4: Marco Donnarumma, Hypo Chrysos, 2011. © Marco Donnarumma.
Another example can be an installation Telephone Rewired by the artist-neuroscientist Sean Montgomery. Brainwave signals are measured from each visitor upon the entrance to the installation site. These individual data then become part of the collective archive of the brainwaves of all the participants. In the second room, the viewer is engulfed by pulsing light and sound that mimic endogenous brain waveforms of the previous viewers. As in the experience of Donnarumma's performance, this process encourages tuning in to the inner state of the other and finding resonating states in one's own body. It becomes a tool for self-exploration, self-knowledge, and self-control, as well as for developing skills of collective being, of shared body-mind topologies. Synchronization of mental and bodily states of multiple people serves here a broader and deeper goal of training collaborative and empathic abilities. An immersive experience, it triggers deep embodied neural circuits, reaching towards the most authentic reactions not mediated by conscious procedures and judgment.
Figure 5: Sean Montgomery, Telephone Rewired, 2013. © Sean Montgomery.
The potential of biofeedback as a strategy for art projects is a rich area that artists have only begun to explore. The layer of the imaginary and the fictional (which makes art special and different from, for instance, science) can add a critical dimension to understanding the processes of augmentation and mediation. As the described examples demonstrate, art is an investigative journey that can be engaging, surprising, and awakening towards the more subtle and acute forms of thinking and feeling. This astuteness and percipience are especially needed as media and technologies penetrate and affect our very abilities to apprehend reality. We need new tools to make independent and individual judgment. The sense of proprioception establishes a productive challenge not only for science, but also for the arts, inviting a search for new mechanisms of representing the un-presentable and making shareable and communicable what is, by definition, individual, fluid, and ungraspable. Collaborative cognition emerging from the augmentation of proprioception that is enabled by biofeedback technologies holds distinct promise for exploration of not only subjective, but also inter-subjective states and aesthetic strategies of inducing them.
Beesley, Philip. Hylozoic Ground. 2010. Venice Biennale, Venice.
Clark, Andy, and David J. Chalmers. “The Extended Mind.” Analysis 58.1 (1998):7-19.
Donnarumma, Marco. Hypo Chrysos: Action Art for Vexed Body and Biophysical Media. 2011. Xth Sense Biosensing Wearable Technology. MADATAC Festival, Madrid.
Eliasson, Olafur. Take Your Time, 2008. P.S.1 Contemporary Art Centre; Museum of Modern Art, New York.
Grau, Oliver. Virtual Art: From Illusion to Immersion. Cambridge, Mass.: MIT Press, 2003.
Massumi, Brian. Parables of the Virtual: Movement, Affect, Sensation. Durham: Duke University Press, 2002.
Montero, Barbara. "Proprioception as an Aesthetic Sense." Journal of Aesthetics and Art Criticism 64.2 (2006): 231-242.
Montgomery, Sean, and Ira Laefsky. "Biosensing: Track Your Body's Signals and Brain Waves and Use Them to Control Things." Make 26. 1 Oct. 2013 ‹http://www.make-digital.com/make/vol26?pg=104#pg104›.
Sacks, Oliver. "The Disembodied Lady". The Man Who Mistook His Wife for a Hat and Other Clinical Tales. Philippines: Summit Books, 1985.
Schwartzman, Madeline, See Yourself Sensing. Redefining Human Perception. London: Black Dog Publishing, 2011.
Spuybroek, Lars. Waterland. 1994-1997. H2O Expo, Zeeland, NL.