Sound is the first factor of immersion

We associate immersion with the image that surrounds us, with the spherical screen, with the headset. I defend here the opposite idea, and I unfold it through a history of immersion taken from the angle of sound, from the panoramas of the eighteenth century to Dolby, by way of thinkers of sound and of my own workshops. It is sound that immerses us, and it is with sound that we must begin, on the side of the audience as much as on the side of those who create.

In front of the image, inside the sound

When we enter a movie theatre, we sit facing a screen that stands at a certain distance, within a certain frame, and however large it may be, we remain in front of it. Sound, for its part, does not stand in front of us. It comes from everywhere, it fills the volume of the room, it takes hold of us from all sides without leaving us any outside position. We can turn our eyes away from the screen, we cannot turn our ears away. It is sound that makes us be inside, and not in front.

This idea is nourished by my practice as a filmmaker and educator, where I observe, workshop after workshop, that the work on sound largely decides whether a piece takes us in or leaves us outside. The film industry has known this for a long time, since the whole technical history of movie theatres, to which I will return, has had no other purpose than to build this immersion through the ears, far more effectively than anyone has ever managed to do through the eyes in an ordinary theatre.

Entering the room with closed eyes

That is why, when I lead a workshop on immersion, I often begin with sound, and sometimes literally with closed eyes. During a session with teachers, before letting them enter the darkened room, I asked them to close their eyes and guided them in by the hand. “Immersion is first of all sound,” I explained to them, “because sound is something we are always immersed in.” Entering immersion through the ear, and not through the eye, shifts from the outset the way we understand what it is about.

This approach joins Michel Chion’s work on audio-vision, where sound is not an accompaniment to the image but takes full part in the construction of space. What we believe we see, we hear as much as we see it, and sound shapes the depth, the texture, the presence of what is on screen. Letting people experience sound immersion before any image means taking this idea seriously, and making it lived rather than stated.

Filming through one’s window while telling a memory

The distinction I am drawing here between image and sound is in fact rather technical, and even somewhat false with regard to what happens in the brain. We receive on one side light, which the eye converts into electrical impulses along the optic nerve, and on the other side variations in air pressure, which the inner ear also converts into electrical signals. But the two domains never stop communicating. We see an image and it makes a sound in our head, we listen to someone telling a story and we see what they are telling. Images are born from sound, sounds are born from images, and what we perceive is always a blend of the two.

I have experienced this with thousands of films. Since 2009, I have been inviting people to film what they see through a window of their home, a more or less still image where almost nothing happens, while telling an important memory they wish to share. From this instruction was born in 2020, during the first lockdown, the platform « Par ma fenêtre » (“Through my window”). Since there is not much to see on screen, the images that form in our head as we watch these films come from the voice, not from the image. We see many images while watching them, but they are not the ones on the screen. And what never ceases to astonish me is that small things do happen in the image, a bird crossing, a tree moving, at moments that match the story with unsettling accuracy. What immerses us, in these films, has less to do with the image than with what the voice brings to life in us, and with the meeting of the two.

The panorama was a century ahead of the cinematograph

The history of spectacle sheds light on this inversion of priorities, because immersion did not wait for cinema. In 1787, the painter Robert Barker filed a patent for the panorama, a rotunda where spectators, placed at the centre on a platform, are surrounded by an immense circular painted canvas whose edges are hidden from them, so that they no longer see anything but the bay of Naples or the battle depicted. In 1798, Étienne-Gaspard Robert, known as Robertson, set up his phantasmagoria shows in a deconsecrated chapel in Paris, where magic lantern projections onto screens of smoke, accompanied by noises in total darkness, made ghosts loom up in the middle of the audience. In 1822, Daguerre opened his diorama, where gigantic canvases changed appearance under plays of light. The whole nineteenth century thus built devices of visual immersion, which were mass entertainment.

When the cinematograph arrived in 1895, it offered something quite different, a frame, a flat screen, a frontal image watched from one’s chair, and most often without any sound tied to the images, just a little music, or a lecturer’s commentary beside the screen. From the point of view of immersion, this was a clear step backwards compared with the panorama. And yet cinema buried those older immersive devices, for a reason that has to do with what André Bazin named, in 1945, the ontology of the photographic image. The cinematographic image is a mechanical recording of the real, the trace of something that really stood before the lens, and that presents itself as such. One could, on that account, have called the cinematograph virtual reality, since it puts us in the presence of a trace of the world. There is something in it that defies death, the movement of life whose trace can be repeated indefinitely, and this fascination was enough to set immersion aside for a long time. The envelopment of the audience, cinema only reconquered much later, and it reconquered it through sound.

Fairground screenings that were never silent

This reconquest begins with a fact that historiography long masked, namely that so-called silent cinema was never silent. Martin Barnier, in « Bruits, cris, musiques de films. Les projections avant 1914 » (Presses universitaires de Rennes, 2010), reconstructs the sound environment of the first film screenings, and it is an uproar. On the fairgrounds, the steam engines producing the electricity rumble next to the tent, the barkers shout to draw the crowd, and inside, the sound-effects makers, the lecturers, the singers, the orchestras of all sizes compete with the applause, the booing and the comments of the audience. The audience of the Belle Époque was bathed in sound, and every screening had its own soundscape, unique every time. The arrival of talking pictures in 1927 fixed this profusion onto the film strip, but in mono, on a single loudspeaker behind the screen, and the history of sound in theatres then became that of a slow reconquest of space.

The strategic coup of Dolby Stereo

This reconquest accelerated under pressure from television. In the 1950s, as the small screen entered households and emptied the theatres, the industry sought to offer what television could not give, and invented the wide screen combined with multichannel sound, Cinerama in 1952, CinemaScope in 1953 with its magnetic tracks. But magnetic sound was expensive to install, and most theatres around the world gave it up and went back to mono optical sound. This is where Ray Dolby came in, whose first invention was the reduction of background noise on magnetic recordings. Applied to cinema, his ingenuity comes down to one decisive point, Dolby Stereo fits four channels, left, centre, right and surround, into the place of the single mono optical track, through an encoding that a simple decoder mounted on the projector unfolds. The same print is projected in mono in a theatre without the equipment, and opens out into enveloping sound as soon as a theatre has installed the decoder and the loudspeakers. Exhibitors could therefore move towards immersive spectacle gradually, without massive investment and without changing prints.

« A Star Is Born » inaugurated the process in 1976, but it was « Star Wars », in 1977, that created the demand. For the first time in an ordinary theatre, the spaceship passed over people’s heads, sound moved through the room and enveloped the audience, and exhibitors converted their theatres in order to screen the film in those conditions. A handful of theatres were equipped when the film came out, and six thousand worldwide less than ten years later. Digital Dolby 5.1 extended the movement from 1992 onwards, up to the dozens of channels of today’s systems. What brought audiences back, and still does, is largely this immersion through the ears, and the same logic can be found in the success of today’s immersive experiences, where the promise of being enveloped is often enough to draw people in, regardless of the artistic value of what they live there.

Being inside music, with Günther Anders

Philosophers of sound give words to this experience. At the very beginning of the 1930s, the young Günther Anders, trained by Husserl and shaped by music, wrote investigations into “musical situations” that remained unpublished in his lifetime and were published in French under the title « Phénoménologie de l’écoute » (Éditions de la Philharmonie de Paris, 2020). Anders describes what distinguishes the musical experience from the contemplation of a painting. In front of a painting, I remain a subject facing an object, I can come closer or step back, take my time. In music, this face-to-face dissolves, the one who listens finds themselves inside the music, carried by its time, transformed in their very state. Anders is interested in this transformation of being within listening, and it is, it seems to me, the most accurate description of what I am calling immersion here. We are not in front of sound as we are in front of an image, we are inside it, and this inside changes us.

Schaeffer’s sound object and Adorno’s colour of sound

Pierre Schaeffer, in his « Traité des objets musicaux » (1966), provides another tool, acousmatic listening. The word comes from the disciples of Pythagoras, who listened to the teaching of the master hidden behind a curtain, and Schaeffer takes it up to name the situation that became ordinary with radio and recording, hearing a sound without seeing what produces it. Detached from its source, sound becomes a sound object that can be listened to for itself, for its grain and its texture, what Schaeffer calls reduced listening. Immersive works live almost entirely within this acousmatic regime, their sources are invisible, and that is one of the reasons, I believe, for their power, a sound with no visible origin is located nowhere, and so it is everywhere.

Theodor W. Adorno, for his part, devoted his last Darmstadt lectures, in 1966, to what German calls Klangfarbe, the colour of sound, that is to say timbre. These texts, collected in French in « La fonction de la couleur dans la musique » (Contrechamps, 2021), follow the progressive emancipation of timbre, long held to be a mere dressing of pitches and harmonies, until it became in the twentieth century a building material in its own right. This thinking of timbre connects directly with my practice. When I put together my box of instruments for a workshop, kalimbas, guiro, triangle, metallophones, I am not choosing notes, I am choosing colours of sound, and it is through their timbres that participants build the climate of a scene, joyful or unsettling.

Jonathan Sterne’s warning

We should be careful, though, not to turn this into a metaphysics of the senses. Jonathan Sterne, in « The Audible Past » (2003, published in French as « Une histoire de la modernité sonore », La Découverte, 2015), showed that the opposition between a hearing that would be by nature immersive and subjective, and a vision that would be by nature distant and objective, is a litany inherited from sources that are partly theological, recited from book to book without ever having been seriously questioned. I take this warning seriously. What I defend here does not bear on an eternal essence of the ear, but on concrete devices and on practices, a screen frames and keeps at a distance, loudspeakers surround and envelop, and in the present state of our technologies and our uses, it is sound that accomplishes the immersion the image promises. My observation is situated within this state of technology, and it may evolve with it.

Muzak, or the political face of sound immersion

This power of sound also has a political history, which it would be unwise to ignore. Juliette Volcler, in « Contrôle. Comment s’inventa l’art de la manipulation sonore » (La Découverte, 2017), retraces the career of Harold Burris-Meyer, an almost forgotten American sound engineer. Burris-Meyer began in the 1930s in the theatre, where he experimented with sound devices designed to intensify the emotions of the audience, going as far as trying to provoke collective reactions, sometimes with the help of infrasound that put spectators in turmoil without their being able to guess the cause. Industry then took over, and Burris-Meyer became vice-president of the Muzak company, which broadcast in factories musical programmes designed to sustain production rates, then in shops ambiences meant to steer purchasing behaviour, a music made to be heard without being listened to. War finally requisitioned this expertise, sound decoys and acoustic harassment.

If sound lends itself so well to manipulation, it is for the very reason that makes it the first factor of immersion, we cannot stand apart from it, and it acts on the body before reaching consciousness. This history gives the work on sound a stake of critical education, which seems to me inseparable from the creative stake. Learning to build a sound ambience is also learning to recognize the ones being built around us, in shops as in films.

The music-loving brain and bodies that synchronize

Neuroscience today describes what the body does with sound. The neurologist Pierre Lemarquis, in « Les pouvoirs de la musique sur le cerveau des enfants et des adultes » (Odile Jacob, 2021), gathers the studies establishing how music sculpts the child’s brain, supporting language acquisition, motor skills, memory and self-confidence, and continues to act in adulthood. The neuroscientist Stefan Koelsch has shown for his part that making music in a group synchronizes physiological processes between the members of the group, heart rates draw closer, breathing patterns fall into step with one another, levels of attention align. I have seen this synchronization at work in a workshop, the tempo of a demon character’s cries settling on that of the percussion, the breathing of the actress finding the rhythm of the kalimba.

Sound is a vibration, a pressure wave that touches the whole body and not only the eardrum, something we feel physically in front of a subwoofer or a gong. Sound therapy practices, with their bowls and gongs, work directly on this vibratory dimension and on the emotional states it induces, and without taking a position on their therapeutic protocols, I note that they take seriously a physical fact that the audiovisual field neglects, sound is touch at a distance. In my own practice with instruments, I know that these shared vibrations bind the group together, and this cohesion matters enormously in moments of creation. We have been speaking here of immersion on the audience’s side, but it can be worked on just as much on the side of those who create.

The sound ambience choir, building immersion together

I have described in detail in « Le chœur d’ambiance sonore » (“The sound ambience choir”) a method that rests on this idea. In a recent virtual reality film project with secondary school students, the 360-degree camera filmed the whole space, which occupied a number of students in front of the lens, but not all of them. I suggested that the others take simple instruments, kalimbas, percussion, guiro, triangle, hide out of shot, behind a piece of furniture or a folding screen, and build the scene’s sound ambience live, adjusting their playing to what they saw, and lowering the volume as soon as someone spoke. The music ends up in the film without having been added in editing, with the slightly imperfect accuracy of live performance, and it reinvents itself with every take.

This shift changes many things. Instead of having active participants in front of the camera and others who wait, everyone holds a real share of the work, and this share is precisely the one that builds the future immersion of the audience. By placing themselves around the microphones, at different distances, the students learn concretely how a sound immersion is made, and they do it with their bodies, inscribing them in space. Building through sound the place of every element in the scene also means, for each of them, mentally building their own place in space, and therefore their place altogether. The psychoanalyst Didier Anzieu speaks, in « Le moi-peau » (1985), of a primary sound envelope, prior to language, which gives the baby its first feeling of being held within an inhabitable world. The sound ambience choir consciously builds an envelope of this kind, and the students feel, in their hands and in their bodies, that immersion is not a magical property of headsets, that it is something made, and that they know how to make it.

Begin with sound

Audio too often remains the poor relation of the audiovisual field. We take care of the image, the framing, the editing, and sound comes afterwards, as a finishing touch, whereas the immersion experienced by the audience is decided upstream, in the very way the work is made. Sergio Leone had understood this. For « Once Upon a Time in the West » (1968), he had the music composed by Ennio Morricone before the shoot, and he played it on set during the takes. Italian cinema of that era post-synchronized dialogue and sound effects, which left filmmakers free to speak and to play music while shooting, and Leone went as far as filming long sequences that he then adapted to the score. When we see these images, they are as if fused with the music, because they were made with it and sometimes by it. The music immerses us in the image, all the more so since this image was born while the music was playing. It is one of the strengths of these films, and a power that seems almost inexplicable as long as one does not know how it was produced. For the opening scene, Leone even gave up the composed music and kept only the noises, the drop of water, the windmill, the fly, the approaching train, and this sound ambience made, on its own, a great piece of music.

This is what I experiment with in my workshops, where the sound is made entirely during the shoot, and where nothing can therefore be fixed afterwards. This constraint, which one might think a handicap, is precisely what gives the result its accuracy and its strength, because immersion is built there in the same gesture as the image, by people who are present to what is taking place. Working on sound in an embodied way, letting it be heard before the image, placing it in the participants’ hands while we create, is giving it back the place that is its own. One must dare to do it. If we truly care about immersion, that is where we must begin, because that is how we get inside.