For the RSS readers: SLRC Does Pinball

Because I know there are some readers of SLRC that don't catch all I mention on Twitter, I wanted to let you know of a piece I wrote for Kotaku Australia on the first Annual Pinball Expo in Sydney. I wrote about it from a videogame angle because I expected there to be more "gamers" at the expo than there were - in hindsight, of course, Pinball was most popular with a generation older than my own, but I somehow overlooked that going in.

Anyway, there were too many videogame parallels to pass up commenting on and the editor-at-large of Kotaku AU, David Wildgoose, was kind enough to give me a place to talk about them. It starts out like this,

I guess the website should have tipped me off about the expo in advance – it was done in crude HTML and featured the kind of layout scheme that would have been the height of excellence on a Geocities site circa 1999. That is, it had fluoro coloured text and a visitor counter at the bottom currently sitting pretty at around 7,000 visitors.

And continues on in a fashion that even my mother found compelling enough to read. Head on over for the full text and some better pictures (but not much better).

Audio Vision, Part 1

A big part of my research thesis – the core conceit if you will – was one part a belief in the power of the relationship between audio and visuals, and an equal belief that videogames present a new direction for that relationship to be explored.

Images and music (even just sound generally) have a particular way of working together – when combined in certain ways they inform and affect each other to change the meaning we derive from them. When viewed together they have the potential to change what you would perceive, and understand them to mean, than if you examined them independently. Think of any popular song you have seen a music video for –after viewing the attendant visuals created to accompany that song, from then on you may have trouble thinking of the song without taking into account what the visuals add to the music.

In any serious theory based discussion of how and why music and visuals go together as well as the effect it has, one quickly runs into audiovisual theorist Michael Chion as he had some quite a bit to say about the subject. Chion was focused specifically on the music of cinema, and how music and visuals inform each other in that medium. He notes the because of the inherent difficulty in pinning down descriptions and categories of sounds,

“…there remains the risk of seeing the audio-visual relationship as a repertoire of illusions, even tricks- all the more contemptible for being so. Audiovisual analysis does not involve clear entities or essences like the shot, but only “effects”, something considerably less noble.”

Does that mean, then, that we should give up trying to better our understanding and ability to control these audio-visual relationships? Certainly not, and Chion advocates a number of activities we can do to improve our understanding.

He recommends one activity in particular which he says reveals the power of something he calls “forced marriage” – one aspect of potential audio-visual relationships. Forced marriage demonstrates that even music composed with no regard for the visual elements can present a synchronisation with the image onscreen. As we shall see, in some cases no effort even needs be made to try and match the audio to the visuals, they can just ‘work’ almost as if it were intended.

He describes an experiment we can do to look at the effect of “forced marriage”, saying…

“Take a sequence of film and also gather together a selection of diverse kinds of music that will serve as accompaniment. Taking care to cut out the original sound (which your participants must not hear at first or know from prior experience) show them the sequence several times, accompanied by these various musical pieces played over the images in an aleatory [random] manner. Success assured: in ten or so versions there will always be a few that create amazing points of synchronisations and moving or comical juxtapositions, which always come as a surprise.”

So that is exactly what I have done in the following seven videos. I have replaced the music from a piece of footage that (hopefully) you will have never seen before (or forgotten the specifics of if you have) and overlaid some other random pieces of music over them. The video is taken from a pre-launch trailer for the videogame Army of Two, it is arranged in a playlist with all seven ‘forced marriage’ examples from A to G followed by the original. The videos are approximately one minute in length and the only thing different between them is the sound. The very last video is the original footage.

A video playlist with all seven videos plus the original at the end.

Chion designed this experiment to highlight how, even accidentally, music and visuals can be made to work together. He says,

“Changing music over the same image drastically illustrates the phenomena of added value, synchresis, sound-image association and so forth. By observing the kinds of music the image “resists” and the kinds of music cues it yields to, we begin to see the image in all it’s potential signification and expression…”

He says that the effect of revealing the original videos sound after all the other alternatives,

“…never fails to be staggering. Whatever it is, no one would have ever imagined it that way beforehand; we conceived of it differently, and we always discover some sound element that never would have occurred to us. For a few seconds, then, we become aware of the fundamental strangeness of the audio-visual relationship.”

What did you notice about the sound and music in the videos? Was the "real" music and sound surprising? Did you notice "the fundamental strangeness" of the relationship between sound and image? Even having heard the final clip several times, after watching all the videos again followed by the original, I can't help but feel certain things - particularly some of the weapon sounds - sounded strage or mis-matched.

Understanding and utilising the Audio Visual relationships is particularly applicable for games since, as Marty O'Donnell noted in the very first part of my interview with him, there is no "real" sound to go with a particular image - it's all virtual. It is entirely created, designed, sculpted, engineered and controlled.

I hope that I’ve gone some way towards convincing you that audio-visual relationships are worth examining. As we have seen, even accidental synchronisation can prove startlingly interesting. In the second part of this series I will examine the issue further by asking the question 'What would be the result of a more deliberate synchronisation of sound and image?' and hopefully have some interesting points to make.

In the mean time - I would like to encourage you to discuss the different effects you felt particular pieces of music had on the video, as well as any salient points of synchronisation you noticed. I'd love to have a conversation on which pieces of music people felt were "better" and which ones they felt were "resisted" by the images, as well as why.

Stay a while and listen

One of the ways that valve tells a story in their games is by scattering little ‘micro stories’ around the game world, often in slightly out of the way places. If you’ve played any of their games you’ll know what I’m talking about – graffiti on the walls in Left 4 Dead that hint at survivors who have gone before are a great example. In Half-Life 2 there is a small corner in the level “Water Hazard” that I wish to highlight as an excellent example of using sound in a particularly videogame way.

This place, which has no name as far as I know, appears at first glance to have once been another outpost on the Underground Railroad and occurs a few hours into the game. Looking so much like an overnight camp, there are a few scattered supply boxes, a mattress, resistance graffiti and a clutter of other detritus; obvious evidence of human habitation. There is no-one there when you encounter it, but a curious feature remains, giving the location a very unique feel.

When I encountered it the other day, I greedily lapped up the atmosphere. I actually sat and listened for a number of minutes, becoming utterly absorbed in the sound of the place – and more than that, in the sense of place that the unique sonic signature gave to this tiny corner of City 17.

There are a few things I want to note about the sound in this scene, and most obviously anyone who comes across this place in the game will probably be struck first by the ‘wind chimes’. They stand out from the sound of the rest of the game because of their volume in the mix and their distinct timbre. The sound contains a variety of both pitched and unpitched sounds that work to blur the line between musical elements and sound effects. Obviously, a sound doesn’t have to be pitched to be ‘musical’ or to have beauty – the timbre of a sound plays as much a part in our response to sounds as does the fundamental frequency of the sound.

Another feature that adds beauty and a unique sonic signature to this place is the background sound of a constant strong wind. What we call the sound of the wind doesn’t really have a fundamental note that we can identify as a pitch as it is really a complex combination of sounds. Perhaps most importantly, wind doesn’t even have a sound until it blows past an object, and so what we associate with a “wind” sound is actually a series of sounds that will often change and fluctuate in unison in accordance with the wind speed. However, even this ‘sound’ of the wind blowing has a musicality to it – that is, it has musical properties that I find incredibly attractive. In this corner of Half-Life 2, it fluctuates seemingly as if at random, increasing and decreasing in volume and pitch as the speed of the wind changes. There is, of course, no real wind in the game, but the sound is so convincing a recording that I half-expect to feel the breeze on my face while I stood listening to it.

The ‘aural scene’ of the location is reinforced by the visual, that is, the sounds make sense with the visual. The windmill above the player reinforces the notion that there is a strong wind blowing. The audible and the visual combine to reinforce a single impression and it’s all the more powerful and impacting for it.

While the wind chimes themselves arguably become the aural focus in the place because of their prominence in the mix (i.e. they are quite loud comparative to other sounds), they actually have no visible propagating source. Which is an important point to note, as sounds don’t have to have a visible source and leaving their location hidden or up to player interpretation is a completely valid artistic technique. Cinema has used this trick for years, and will often use a sound to tell the viewer about something happening off-screen. With games, however, a developer can rarely ever tell where a player is looking, that is, what is “off screen” at any one time is up to the player, making it difficult to use “off screen sound” in a deliberately artistic way.

What often happens is that the game will spawn an object behind the player or in a location above and out of view of the player and attach a sound to it to attract the player’s attention. For example, in Half-Life 2 there are many occasions in the area around this location where soldiers spawn on bridges above the player.

You might argue that the sound of a gun cocking behind you in a game happens “off screen”, which is true, but strictly speaking, from the point of view of the game engine that place in game exists whether the player is looking at it or not. I’m not entirely sure about this analogy, but to me the equivalent of actual “off screen” in a game would be a sound coming from a place that doesn’t even exist in engine. That is kind of what is happening here, with the sound of the wind chimes and the wind coming from an invisible, virtual location.

The end result of all of the above points is that while technically being sound effects, the wind chimes, and the wind, take on a similar role to what music would – giving the location a very real and specific atmosphere. This blurring of the lines between more pure sound and music is something that I am very interested in, and was a core finding of my research thesis last year. It seems to me to be an somewhat unexplored area ripe with potential.

I hope you’ve enjoyed taking a moment to listen to the sound in the one corner of Half-Life 2 as much as I have.