Audio Vision, Part 1

A big part of my research thesis – the core conceit if you will – was one part a belief in the power of the relationship between audio and visuals, and an equal belief that videogames present a new direction for that relationship to be explored.

Images and music (even just sound generally) have a particular way of working together – when combined in certain ways they inform and affect each other to change the meaning we derive from them. When viewed together they have the potential to change what you would perceive, and understand them to mean, than if you examined them independently. Think of any popular song you have seen a music video for –after viewing the attendant visuals created to accompany that song, from then on you may have trouble thinking of the song without taking into account what the visuals add to the music.

In any serious theory based discussion of how and why music and visuals go together as well as the effect it has, one quickly runs into audiovisual theorist Michael Chion as he had some quite a bit to say about the subject. Chion was focused specifically on the music of cinema, and how music and visuals inform each other in that medium. He notes the because of the inherent difficulty in pinning down descriptions and categories of sounds,

“…there remains the risk of seeing the audio-visual relationship as a repertoire of illusions, even tricks- all the more contemptible for being so. Audiovisual analysis does not involve clear entities or essences like the shot, but only “effects”, something considerably less noble.”

Does that mean, then, that we should give up trying to better our understanding and ability to control these audio-visual relationships? Certainly not, and Chion advocates a number of activities we can do to improve our understanding.

He recommends one activity in particular which he says reveals the power of something he calls “forced marriage” – one aspect of potential audio-visual relationships. Forced marriage demonstrates that even music composed with no regard for the visual elements can present a synchronisation with the image onscreen. As we shall see, in some cases no effort even needs be made to try and match the audio to the visuals, they can just ‘work’ almost as if it were intended.

He describes an experiment we can do to look at the effect of “forced marriage”, saying…

“Take a sequence of film and also gather together a selection of diverse kinds of music that will serve as accompaniment. Taking care to cut out the original sound (which your participants must not hear at first or know from prior experience) show them the sequence several times, accompanied by these various musical pieces played over the images in an aleatory [random] manner. Success assured: in ten or so versions there will always be a few that create amazing points of synchronisations and moving or comical juxtapositions, which always come as a surprise.”

So that is exactly what I have done in the following seven videos. I have replaced the music from a piece of footage that (hopefully) you will have never seen before (or forgotten the specifics of if you have) and overlaid some other random pieces of music over them. The video is taken from a pre-launch trailer for the videogame Army of Two, it is arranged in a playlist with all seven ‘forced marriage’ examples from A to G followed by the original. The videos are approximately one minute in length and the only thing different between them is the sound. The very last video is the original footage.

A video playlist with all seven videos plus the original at the end.

Chion designed this experiment to highlight how, even accidentally, music and visuals can be made to work together. He says,

“Changing music over the same image drastically illustrates the phenomena of added value, synchresis, sound-image association and so forth. By observing the kinds of music the image “resists” and the kinds of music cues it yields to, we begin to see the image in all it’s potential signification and expression…”

He says that the effect of revealing the original videos sound after all the other alternatives,

“…never fails to be staggering. Whatever it is, no one would have ever imagined it that way beforehand; we conceived of it differently, and we always discover some sound element that never would have occurred to us. For a few seconds, then, we become aware of the fundamental strangeness of the audio-visual relationship.”

What did you notice about the sound and music in the videos? Was the "real" music and sound surprising? Did you notice "the fundamental strangeness" of the relationship between sound and image? Even having heard the final clip several times, after watching all the videos again followed by the original, I can't help but feel certain things - particularly some of the weapon sounds - sounded strage or mis-matched.

Understanding and utilising the Audio Visual relationships is particularly applicable for games since, as Marty O'Donnell noted in the very first part of my interview with him, there is no "real" sound to go with a particular image - it's all virtual. It is entirely created, designed, sculpted, engineered and controlled.

I hope that I’ve gone some way towards convincing you that audio-visual relationships are worth examining. As we have seen, even accidental synchronisation can prove startlingly interesting. In the second part of this series I will examine the issue further by asking the question 'What would be the result of a more deliberate synchronisation of sound and image?' and hopefully have some interesting points to make.

In the mean time - I would like to encourage you to discuss the different effects you felt particular pieces of music had on the video, as well as any salient points of synchronisation you noticed. I'd love to have a conversation on which pieces of music people felt were "better" and which ones they felt were "resisted" by the images, as well as why.

Hocking's Masterpiece

I went to bed last night thinking about the ending of Far Cry 2. My response, entirely subjective, was a mix of dismay and real, genuine sadness. The feeling was comparable to the one I get at the very conclusion of the Harry Potter series – a kind of melancholy sadness at having to say goodbye to this mind-blowing world, compounded by the tragic events of the closing pages. Similarly, I know there are many people who don’t like the Harry Potter series, or the ending, and can’t help but think that some people will react similarly to Far Cry 2. The ending is imperfect. It’s flawed. But for those few of us who now care about this game, it’s also deeply emotional.

I can forgive the flaws of the ending, however, and now rate it as one of my all time favourite videogame conclusions ever. Partly it's because it is going to be somewhat unpopular, and I don’t want it written off or forgotten. But in it’s depths I can see the glimmer of the potential that it was trying to achieve. Hocking’s Masterpiece.

Far Cry 2 is about the individual; death; nihilism. The contentious design decisions, even the whole game, only starts to make sense when viewed through this lens. Like a David Lean epic from the golden era of cinema it deals with many concepts and issues all viewed through the prism of a central concern. Doctor Zhivago’s conceit was viewing the Russian revolution through the personal story of one man. Similarly Lawrence of Arabia also possessed the same focus on viewing history as it orbited around an individual. Far Cry 2 is about you and death. Of course every single person you meet wants to kill you. Of course you spend about as much time fighting the environment as other persons. Of course you are clinging to the barest scrap of health and well-being; Even the malaria is trying to kill you.

Some people will probably want to throw around that clichéd phrase “You’re a survivor” and the sentiment that goes with it, but the truth is, you’re not. Take from that what you will, but Far Cry 2 reminds at every possible opportunity that life is fatal. No one gets out of this place alive. It makes for what I believe to be truly the first game that goes beyond, or outright rejects, the mental switch-off performed by pathological mass-murderers and videogame players. That switch you flick in your brain when playing a shooter - the small part up the back that empathises with your victims, and which is incompatible with what most games narcissistically tell us – that you should survive, you deserve it, you’re the hero. Here instead, nihilism is the name of the game.

As the man you have been trying to kill says, “No one is going to die today that didn’t have it coming.” Far Cry 2 screams that we all have it coming.

Videogames will save the world!

And I don't mean that in a 'figurative sense' or as a playfully ironic commentary on the the over-abundance of superhero tropes and 'hero complex' tales. I mean it instead that I think videogames are teaching our kids and young adults to think through, what is to many older persons, potentially opaque and incomprehensible systems.

Let me explain.

I was watching the fantastic Michael Moore style documentary The Corporation. You've probably seen the promo's for it - it's the one where the film makers compare the corporation as a legal "person" to a psychopath. It makes sense, after all it has no conscience and it's main driving motivation is the bottom line.

One of the big things that I got out of the documentary was that whenever people saw through what was a patently exploitative system (as was the case in the Bolivian water riots and a number of other examples) people got up and did something about it.

Videogames, according to Ian Bogost, can speak to players through what he terms Procedural Rhetoric, and as games get more complicated and convoluted, gamers are only getting more and more used to the idea of procedures and systems. Gonzalo Frasca in his essay for The Videogame Theory Reader (which I have summarised elsewhere) likens simulation to the act of creating government legislation and I think he's quite right when he says that

"Video games imply an enormous paradigm shift for our culture because they represent the first complex simulational media for the masses".

Videogames can and are giving players the experiences necessary and the cognitive tools to be able to start to come to grips with large systems and procedures. In the film The Corporation, a company appearing 'socially responsible' becomes a market driven reaction to consumers being turned off by irresponsible plunderers of the environment. This is a product of the market, which in itself is a system. When more people are taught (by videogames) the ability to think through processes in general, it is my hope more people will start to take positive action and make themselves heard about a number of these inherently bad systems that have cropped up in society.

That's enough of my ranting. Go watch The Corporation on YouTube or DVD and get excited about the prospect of a gamer revolution! Viva La Gamer!