Monday, 18 May 2009

Audio Vision, Part 1

A big part of my research thesis – the core conceit if you will – was one part a belief in the power of the relationship between audio and visuals, and an equal belief that videogames present a new direction for that relationship to be explored.

Images and music (even just sound generally) have a particular way of working together – when combined in certain ways they inform and affect each other to change the meaning we derive from them. When viewed together they have the potential to change what you would perceive, and understand them to mean, than if you examined them independently. Think of any popular song you have seen a music video for –after viewing the attendant visuals created to accompany that song, from then on you may have trouble thinking of the song without taking into account what the visuals add to the music.

In any serious theory based discussion of how and why music and visuals go together as well as the effect it has, one quickly runs into audiovisual theorist Michael Chion as he had some quite a bit to say about the subject. Chion was focused specifically on the music of cinema, and how music and visuals inform each other in that medium. He notes the because of the inherent difficulty in pinning down descriptions and categories of sounds,

“…there remains the risk of seeing the audio-visual relationship as a repertoire of illusions, even tricks- all the more contemptible for being so. Audiovisual analysis does not involve clear entities or essences like the shot, but only “effects”, something considerably less noble.”

Does that mean, then, that we should give up trying to better our understanding and ability to control these audio-visual relationships? Certainly not, and Chion advocates a number of activities we can do to improve our understanding.

He recommends one activity in particular which he says reveals the power of something he calls “forced marriage” – one aspect of potential audio-visual relationships. Forced marriage demonstrates that even music composed with no regard for the visual elements can present a synchronisation with the image onscreen. As we shall see, in some cases no effort even needs be made to try and match the audio to the visuals, they can just ‘work’ almost as if it were intended.

He describes an experiment we can do to look at the effect of “forced marriage”, saying…

“Take a sequence of film and also gather together a selection of diverse kinds of music that will serve as accompaniment. Taking care to cut out the original sound (which your participants must not hear at first or know from prior experience) show them the sequence several times, accompanied by these various musical pieces played over the images in an aleatory [random] manner. Success assured: in ten or so versions there will always be a few that create amazing points of synchronisations and moving or comical juxtapositions, which always come as a surprise.”

So that is exactly what I have done in the following seven videos. I have replaced the music from a piece of footage that (hopefully) you will have never seen before (or forgotten the specifics of if you have) and overlaid some other random pieces of music over them. The video is taken from a pre-launch trailer for the videogame Army of Two, it is arranged in a playlist with all seven ‘forced marriage’ examples from A to G followed by the original. The videos are approximately one minute in length and the only thing different between them is the sound. The very last video is the original footage.

A video playlist with all seven videos plus the original at the end.

Chion designed this experiment to highlight how, even accidentally, music and visuals can be made to work together. He says,

“Changing music over the same image drastically illustrates the phenomena of added value, synchresis, sound-image association and so forth. By observing the kinds of music the image “resists” and the kinds of music cues it yields to, we begin to see the image in all it’s potential signification and expression…”

He says that the effect of revealing the original videos sound after all the other alternatives,

“…never fails to be staggering. Whatever it is, no one would have ever imagined it that way beforehand; we conceived of it differently, and we always discover some sound element that never would have occurred to us. For a few seconds, then, we become aware of the fundamental strangeness of the audio-visual relationship.”

What did you notice about the sound and music in the videos? Was the "real" music and sound surprising? Did you notice "the fundamental strangeness" of the relationship between sound and image? Even having heard the final clip several times, after watching all the videos again followed by the original, I can't help but feel certain things - particularly some of the weapon sounds - sounded strage or mis-matched.

Understanding and utilising the Audio Visual relationships is particularly applicable for games since, as Marty O'Donnell noted in the very first part of my interview with him, there is no "real" sound to go with a particular image - it's all virtual. It is entirely created, designed, sculpted, engineered and controlled.

I hope that I’ve gone some way towards convincing you that audio-visual relationships are worth examining. As we have seen, even accidental synchronisation can prove startlingly interesting. In the second part of this series I will examine the issue further by asking the question 'What would be the result of a more deliberate synchronisation of sound and image?' and hopefully have some interesting points to make.

In the mean time - I would like to encourage you to discuss the different effects you felt particular pieces of music had on the video, as well as any salient points of synchronisation you noticed. I'd love to have a conversation on which pieces of music people felt were "better" and which ones they felt were "resisted" by the images, as well as why.


Simon Ferrari said...

Yay Chion! I had to read a bit of this for my 3D design class last semester, which was good because our project ended up being very sound-heavy (though I didn't do that part of the work, unfortunately).

Some of the videos wouldn't load on my crappy Internet connection, but I got in three or four. I don't really have any valuations to give, because I don't really know if it's useful. Some of the genres you used are so varying that it's hard to make a fine-tuned analysis of which one would be "better" objectively. You know, like if you wanted to scare people watching then there's a particularly genre (the second example with the heavy, high-pitch strings) you'd want to get ten clips from and then compare. Right now you're just showing the beauty of contrapuntal sound, which I don't think is what Chion is talking about exactly (though I may be remembering incorrectly).

What it is reminding me of is how "hot" a medium music is for me and how cold and blank images really are. I know if you're a classically trained musician or critic, then the process can get very intellectual. But most music creation is emotional manipulation. The strange thing is that emotionally manipulative imagery is usually called tactless, tacky, etc (you know, explosions or dead bodies all cut up with maggots), whereas emotionally manipulative music is kind of universally accepted as something positive.

Simon Ferrari said...

"because I don't really know if it's useful."

Note: I didn't mean your analysis, I meant my valuations wouldn't be useful because I don't know jack about how to make music.

Michael Abbott said...

This reminds me (though obviously a different medium) of the Kuleshov Workshop in the pre-montage era of cinema. The subjectivity of images juxtaposed with other images, and the incredible degree to which all this can be manipulated, obviously connects with your subject here.

As Simon points out, we tend to receive visual manipulations differently, though I would suggest that the average viewer remains fairly unaware or unreflective about the ways these juxtapositions affect them. I find with my students that they only really see or consider this process when it's unwrapped and isolated for them to analyze.

Otherwise, despite the hundreds or thousands of film experiences they've had, they remain nearly as vulnerable to manipulation as Kuleshov's subjects were 90 years ago.

Ben Abraham said...

I think it's interesting that you thought the genre of music was important for the effect it had, Simon.

I am personally inclined to think that "genre" doesn't really tell us much about what the music is doing or what effect it's going to have on music (mostly because "genre" is way too broad a description to be useful).

For example - Beethoven's Fifth is "classical" or "romantic" and so is Mahlers' First Symphony. And yet the two could not be further apart in terms of feel or the effect they would have on a piece of music. The Beethoven piece is tense and the familiar rhythmic stabs at the start (the 'ba-ba-ba-BUUM')is completely different to the first movement of Mahler's which is airly, light, non-rhythmic and starts so quietly, building up so slowly that it's hard to even pinpoint when it actually begins.

I'm inclined to look at things like how particular musical phrases or gestures affect the image and how we can match the two up. Hopefully I'll explain this a bit better in Part Two. =)

Simon Ferrari said...

Mary Ann Doane has some nice stuff about the audio equivalent of the Kuleshov effect in her "The Body in the Cinema," about the ideological effects of sync sound.

I definitely didn't mean genre in the way you construed it :) Change the word to "style" if you like, I simply meant that most of these examples show contrapuntal sound (mis-matching an image and sound track for a certain effect) and not necessarily the gestural sound weaving that you're talking about. Looking forward to part two, though!

Simon Ferrari said...

Oh and the reason I say Chion isn't talking about contrapuntal sound is that it was already quite old hat by the time this book was written. His ideas of audio/visual synchresis and "sound objects" are more nuanced, kind of an attempt to go back and explain why the original examples of contrapuntal sound in Surrealist and Dada film worked so well.

joshua said...

Mr. Abraham, do you think you could give a list of those pieces you overlaid on the trailer?

I think the idea is superb—the forced marriage phenomenon that one already intuitively knows, but rarely gets to enunciate explicitly. I do wonder how forced marriage may be applied between other concepts too—not just switching whole music or a moving image, but perhaps musical key signature (shifting keys, switching major<->minor, shifting octaves), visual color, and visual/audio tempo. This doesn't invoke the accidental synchronization idea so much as the general concept of the relationship between two senses, as well as the range of emotion or thought that may be invoked simply by changing one thing.

How would one apply it to games? Maybe—if there's a game with gameplay governed by parameters (such as character running traction), and changing those parameters changes the "feel" of the, I don't know where I'm going with this.

Ben Abraham said...

Here's a list of the pieces used in the videos:

Video a) 'Courage' by The Whitest Boy Alive
Video b) 'Black Angels - 1. Departure' composed by George Crumb, performed by the Kronos Quarter
Video c) ‘So What’ by Miles Davis
Video d) ‘Invisble Connections’ by Vangelis
Video e) ‘Main Title’ (from the new series of Battlestar Galactica) by Bear McCreary
Video f) ‘The Prince of Parties’ by Flight of the Conchords
Video g) ‘We Are Rockstars’ by Does It Offend You, Yeah?

In part 2 of this series I'm going to try and suggest how a game could make use of 'deliberate' synchronization. Now if only I could find the time to write it... =)

Thanks for commenting, Joshua.