Audiosurf - Breakfast of Champions

I’ve been playing Audiosurf recently, and it struck me that buried deep within my thesis was a nice little bit of theorising about the game. So I've chosen to reprint it here, slightly edited, for the convenience of anyone who can’t be arsed to wade through my multiple thousands-of-words thesis and pick out the good bits (probably most people).

____________________________________________________________________

Audiosurf was the work of primarily by one person, Dylan Fitterer, and was released on the Steam digital distribution platform in February 2008. Audiosurf requires music to play – it takes your music collection, and creates a 3D track based upon features of the music which is then navigated by the player who, depending on the game-mode, collects coloured blocks that visually correspond to the music. The game ostensibly provides a way to ‘ride your music’ as the game’s tag-line suggests[1] - a feat of musical gameplay that is operating on a rather different level to a game like Guitar Hero. It’s also a great step towards overcoming some of the widely acknowledged problems with games like Guitar Hero - many critics have noted that the strength of a music game is largely subject to how good its track listings are[2]. Alec Meer says,

…we were all playing Guitar Hero and wishing we could stick our favourite music into it. Audiosurf says “fuck it, why not?” and provides the scaffolding of a game around it[3]

Audiosurf’s particular implementation of representing and performing music in a game does however come with a number of its own disadvantages. Firstly, the way the three dimensional track is generated by the program is fixed and determined by a set algorithm[4]. In an interview with Ars Technica, the developer Dylan Fitterer commented on the way that the algorithm turns the song into a three dimensional track, saying;

…when the music is at its most intense, that's when you're on a really steep downward slope, like you're flying down a rollercoaster in a tunnel. When the music is calmer, that's when you're chugging your way up the hill, watching that peak in the distance you're going to reach.[5]

The experience of playing the game itself is where I personally find the major innovations of Audiosurf as well as its major problems. When surfing a song the game’s analysis algorithm has pre-determined the majority of the course’s parameters from the musical elements contained within the recording. Some aspects of the course are determined from relatively transparent musical parameters – the track’s length corresponds directly to the length of the song and the contours of the course are derived from reasonably straightforward aspects such as volume and dynamics. In music with a strong steady beat, the track will often appear to undulate along beneath the player’s ship character in time with the rhythm of the song. The comprehensible translation of the music into visuals, or lack thereof, is where I encounter the main problem of Audiosurf.

In the examples outlined above, the relationship between music and the visuals (the track environment) is clear and direct, making sense to the player and allowing for a pleasurable and organic merging of knowledge of the song with knowledge of the corresponding Audiosurf track. This is a significant aspect of the appeal of the game as much community discussion goes on about the suitability of tracks for surfing[6]. Indeed the process works effectively on the macro structural scale, however a core component of Audiosurf is a ‘match 3’ type block collection game, where the block placement – called ‘traffic’ by the game – is generated from the rather more musically ambiguous parameter of “volume spikes”. The developer, Dylan Fitterer, describes the process saying

…whenever there's a spike in the music, the intensity of that spike determines the block's color. So the most distinct spikes, like a snare drum, that would tend to be a red block, a really hot block. If something is a little more subtle, like a quiet high hat, that would be a purple block, which is worth less points.[7]

This kind of relationship between music and visuals or environment becomes, musically at least, increasingly murky on this micro level as a sheer ‘spike’ in volume is no guarantee that a listener would make the corresponding connection to what they are hearing. Indeed the issue of what a listener actually perceives about a song when listening to it is much, much more complicated. Albert S. Bregman, author of the comprehensive text ‘Auditory Scene Analysis: The perceptual organisation of sound’ coined the term “stream” for what he identified as an audible cognitive process which was lacking adequate terminology. Bregman’s research noted a significant distinction between the cognitive process of the grouping of sounds that ‘go together’[8] from what might be distinguished as pure ‘sounds’. He notes that, ‘A series of footsteps, for instance, can form a single experienced event, despite the fact that each footstep is a separate sound.’ He also makes a musical comparison, saying that,

A soprano singing with a piano accompaniment is also heard as a coherent happening, despite being composed of distinct sounds (notes). Furthermore, the singer and piano together form a perceptual entity – the “performance” – that is distinct from other sounds that are occurring.[9]

Kieron Gillen writing for Rock, Paper, Shotgun says that

The problem with Audiosurf is that the concentration you take to really make the block game work is entirely the opposite of what you need to do to feel the music. The two parts of the game can tug at each other a little...On one hand, a zone game. On the other, a high-speed sorting puzzle.[10]

What I believe that Gillen has identified here is the inherent disjunction between what the musical listener focuses on when listening to the song, and what the game makes the player focus on. I suggest that this phenomenon is somewhat analogous to Ian Bogost’s term ‘simulation fever’. The concentration Gillen identifies as being necessary for successful play means that the player is acutely aware of block placement, largely determined by the volume spikes mentioned earlier.

I would argue that simply focussing on volume spikes is not adequately representative of the music to withstand the scrutiny that a player applies to it. I propose that, in a situation of high concentration on music, a more complex system is needed, one which addresses the issue of how a listener perceives a song. Admittedly, this is a daunting prospect and one inevitably encounters certain apparently insurmountable barriers to rendering onscreen what any one particular person is most likely to concentrate on within a song at any one time, needing as it would to take into account personal differences and background as well as individual musical training. However, the fact remains that this process is undertaken by humans themselves leads me to believe that a more accurate model is possible. When listening we can (and do) lock onto a number of particular elements of a song – the melody, a catchy lead rhythm or hook – and this is not always represented visually on screen. While Audiosurf often wonderfully represents the underlying kick-drum rhythm, especially if it is prominent, it will rarely pick up and single out an element like the aforementioned melody or hook unless it stands out in a particular way – namely through sheer volume.

Guitar Hero, in contrast, sidesteps some of these problems through both its position as a guitar game (with the player’s concentration largely limited to being focussed on the guitar) and by having a human pre-define the on screen actions the player has to undertake to ‘perform’ the song. However it does not yet allow for any meaningful input of a players own music library, and for that I am continually thankful for Audiosurf’s existence – imperfect though it may be.

____________________________________________________________________

---

[1] Wikipedia contributors, "Audiosurf," Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Audiosurf&oldid=241996378, accessed October 7, 2008.

[2] See for example, Mitch Krpata, ‘Rock Band 2: Why now?’, Insult Swordfighting, http://insultswordfighting.blogspot.com/2008/07/rock-band-2-why-now.html, accessed October 7^th, 2008.

[3] Alec Meer in ‘The RPS Verdict: Audiosurf’, Rock, Paper, Shotgun, http://www.rockpapershotgun.com/2008/03/03/the-rps-verdict-audiosurf/, accessed March 3, 2008.

[4] Thomas Wilburn, ‘Catching Waveforms: Audiosurf Creator Dylan Fitterer speaks’, Ars Technica, http://arstechnica.com/journals/thumbs.ars/2008/03/11/catching-waveforms-audiosurf-creator-dylan-speaks, accessed

[5] Dylan Fitterer in Thomas Wilburn, ‘Catching Waveforms: Audiosurf Creator Dylan Fitterer speaks’.

[6] See the comments section of any Rock, Paper, Shotgun Post tagged ‘Audiosurf’ – every single one involves readers suggesting songs that others should try: http://www.rockpapershotgun.com/tag/audiosurf/

[7] Dylan Fitterer in Wilburn, ‘Catching Waveforms: Audiosurf Creator Dylan Fitterer speaks’.

[8] Albert S Bregman, Auditory Scene Analysis : The Perceptual Organization of Sound, 2nd MIT Press paperback ed., Cambridge, Mass.: MIT Press, 1999, p.9

[9] Ibid, p.10

[10] Kieron Gillen in ‘‘The RPS Verdict: Audiosurf’, Rock, Paper, Shotgun, http://www.rockpapershotgun.com/2008/03/03/the-rps-verdict-audiosurf/, accessed March 3, 2008.

Marty O'Donnell in Interview - Part 2

Marty O’Donnell is a man on the bleeding edge, being responsible for the music and audio vision for the stratospheric Halo trilogy from Bungie studios. I wanted to try and find out what this leader in the field of music and sound for games had to say about the subject and get some input for my then to be completed thesis. Around August of this year I embarked on a campaign of multiple emails to multiple addresses to somehow get in contact with Marty for an interview. After what seemed like a longer period than it probably was, finally in October a scant week before I had to hand in my thesis I spoke to the man himself via phone. Even though I approached him essentially out of the blue, Marty held no qualms about chewing my ear off for close to 90 minutes, to my own delight of course, and it is clear that music for videogames is a subject about which he is fiercely passionate.

In this, the second of the series, I ask Marty about his views on using a ‘granular’ live assembled music approach for videogame music, and he explains both why he doesn’t like that approach (for the time being) and what his own approach looks like.

Ben: I guess what’s most deceptive is, because Halo’s levels are designed so that there are multiple ways through it, but you go from point A to point B, you end up hearing all the music in the same order and maybe that contributes to the sense that you hear the same thing every time through. But I never got bored of the music in Halo either, so it’s obviously doing its job.

Marty: Halo 2 was a while back for me now, but I think the first level is the space station, right? If you got to the last encounter in the space station just before the end cinematic, pay attention to when the music actually starts, what the music does, how the music changes based on how you perform and then how the music actually seamlessly goes into the cinematic music, which is a more linear piece because I’m scoring something that’s cinematic… If you were to play through that sequence several times, try to go really fast, try to go slow, try to see what happens when the music in the level and the music in the cinematic go together, you’ll actually get several different kinds of recordings out of that.

Now it’s not like an entirely different experience, that’s not the point… it is still like, ‘that is the piece of music that plays there’, but it’s not a linear piece because different sections don’t happen in the same order and the way the music leads into the cinematic… it doesn’t [just] cross-fade it actually plays simultaneously with the cinematic music but it plays in sync with it in a way that changes depending on how you get into it.

B: So what would be your reaction to someone that wanted to do away with that sort of authorial control you have by mixing it up, and says ‘well lets set some parameters, and if the player health gets to this, do this’ (obviously that’s a simple way of doing it) do you think you do lose something if you take that step back?

M: Well you know it’s interesting because, as someone who’s worked with computer generated music or computer controlled music… and as you know in the music business and creating music, we use digital for everything… I remember back when there were music programs that were… basically lets generate music algorithmically. And I remember hearing a lot of that stuff and playing with all those things and… because I think I’m probably more of a traditionalist when it comes to what music I like, and what music I think actually evokes emotions and what music actually speaks to me, I think the composer still makes better choices than just a combination of random events.

Even when I was using algorithmic music programs, out of an hour’s worth of stuff that was generated I would find maybe 30 seconds of it that was actually interesting. So that’s my problem with that. I think that you might get some fun stuff, but I don’t think you get stuff that really speaks. Most of that stuff doesn’t end up telling a story musically, and I still think that the power of music is a storytelling power, and I might be wrong about this, but I still think that a composer tells stories better than computers do.”

However, if something is repetitive, even if it’s really, really great and you keep hearing it over and over again, and it plays back the same way every time – that is to me where boredom set in or it just starts getting annoying. Which is why when I hear games that use linear music that always plays back identically and starts to get repetitive, and loops... when I notice where the loop is I get really annoyed. That’s something I’m trying to eliminate. I really don’t want people to hear where the loops are.

B: Michael Chion wrote in the 90’s about audiovisual relationships in film and he talked about how if you strip the music off a piece of image, and you just place a selection of random other songs… some pieces of music work better with others. He said that in a few pieces… you will get a few moments of almost serendipitous synchronization between the audio and the video, so I guess where I see the potential for the live generated, granular, building the music up from individual notes, is the potential to pick out those points of synchronization and specifically hit them with the moment that you really want.

M: I would say… I have not been one of the guys who advocates the tiniest granular approach. There are some music engines and music for games approaches that’s very, very granular and right down to the individual sample level. ‘This will interact in this way’ and ‘this will interact this way’, it’s all possibly midi controlled, note generated everything.

There’s a couple reasons I don’t like that, number 1, I lose some of the fidelity that I like, I lose some of the live performance that I think is still essential… A midi flute performance just does not compare to a live flute player performing a melody. I don’t like walking away from something that has a giant history of success, so I’ve never really been an advocate of the high granular approach to writing music for games.

That doesn’t mean it’s the wrong way, it’s just that it’s my preference. The biggest problem I have with scoring a game is that if there is a sequence or an encounter or a moment or whatever it is… the most important moment is how it begins and then how it ends. Because it’s a game and there’s a human being interacting with it, what I don’t have any perfect knowledge of is exactly how long the entire experience is going to last.

So what I decided early on was, I can control when it begins, I can control when something ends, musically, what I need to do is keep the middle section malleable. And if I can do that without people knowing that it’s being malleable – so if they have a 2 minute experience some place then they get a 2 minute piece of music and they’re happy, but if somebody else play it for 5 minutes they get a 5 minute piece of music and they’re still happy because the beginning, middle and the end all correspond with what they wanted the experience to be, then I feel successful.

So it’s all about variations on the beginnings, on the ends, and being malleable with the middles. I keep trying to advance the way that stuff is manipulated musically and it’s sort of an interesting puzzle for me. I really enjoy it, I like doing it, I like composing music that I can kind of dissemble and say “what are different ways of telling the same musical story but making the middle something that is sorta indeterminate.” I don’t know if that makes sense but that’s the system.

In the next post, Marty talks about how close he’s coming with his own musical approach to realising a high level of ‘granularity’ in the music. I ask whether you can get away with more if you’re aiming for an ‘electronic’ sound, as opposed to orchestral and Marty talks about Rez & the potential for more ‘synaesthesia’ in games.

An investigation of new musical potential in videogames; A Thesis

So the day has finally arrived and the chickens can now be counted. First the raw mark:

For my thesis, which I spent all year writing, I received a mark of 86 which is safely within the High Distinction band. Accordingly, the full text of my thesis is now available here. Go read it, print it, bind it, critique it, lambast it, or just put it on your coffee table and let it look pretty.

So here's the inside scoop on what to expect:

- Chapter 1 is me raving, perhaps somewhat inadvisably, about a loony Indie game for 1000 of my 17,000 word limit before getting around to talking about music and what I'm going to say in this wordy monstrosity.

- Chapter 2 is my literature review (highly skippable if you aren't interested in either a) Ian Bogost or b) Gonzalo Frasca)

- Chapter 3 is all about how I think the current musical paradigm in videogames is, erm... how you say? COMPLETELY RUBBISH (okay, not quite, but almost) and then I talk about games like Guitar Hero, Audiosurf and Everyday Shooter and how wicked awesoe they are....

- Chapter 4 is the thesis, really, and it's where I interview the awesome Marty O'Donnell in an attempt to glean some insights about music for videogames from him. If you only read one chapter, it should really be this one.

Which also reminds me that I've got the full text of the interview ready to put up, so I'll kick start that series later in the week - keep an eye out for it. I think Marty has some genuinely interesting and important things to say about sound and music in games - and about game design more generally.

What are you still doing here? Go download it already!

Also: props to my man in Melbourne, Dan Golding, who recently posted his own thesis for which he got an even better mark (90) so go download and read his when you're done with mine.

Edit: Matthew Gallant from The Quixotic Engineer has graciously provided hosting for my thesis so I don't have to use Mediafire. Isn't he lovely? =)

Thesis is complete!

And thus ends another chapter of my life; that of being an undergraduate student. I printed, bound and submitted my thesis last Friday and I have been Walking on a Dream since.

I plan to turn the more compelling parts of it into a blog series, either here or elsewhere, and elaborate a bit on my findings. I'll also make it publicly available at the same time, which should be around mid-November when I get it back from being marked. In the intervening time, you can read the abstract of my Thesis:

Abstract

Unlike traditional artistic endeavours such as literature, painting or sculpture, videogames and their creation, according to Janet Murray, are still in an incunabular period. Various efforts have been made to view videogames in light of other media such as film and narrative while few have yet to address, specifically, ways in which videogames present unique opportunities for expression. This thesis draws upon a number of authors to identify areas unique to videogames, and examines the implications for the employment of music within them. After examining the case for videogame uniqueness, the thesis looks to the current musical paradigm within videogames and, finding it somewhat lacking, offers a critique of the paradigm. A number of games that do, however, break from tradition and utilise music in exceptional ways are then discussed and their potential for adoption in future games is assessed. The final component of the thesis is an investigation into the use of music within the Xbox videogame Halo 2 (2004) through discussion with the composer, Martin O’Donnell, and an analysis of the music and sound of the game. In the process I discover that the game uses music in a way similar to the dominant paradigm, while also exhibiting a musicality within the in-game sound effects and level ambience. The result is a ‘soundscape’ style approach well suited to attaining both the emotive power of linear compositions as well as a closer relationship between music and visuals, seemingly a ‘best of both worlds’ videogame musical approach.

Thesis Update

[The guy on the far left is me :D]


It's been a while since I last posted and although apologising for not posting more frequently smacks of disingenuousness (if you were really worried about it, you'd do something about it) I'm going to do so anyway. To anyone who cares, Sorry- but I've been busy writing my thesis.

Yes, that's right I'm actually up to the writing stage, despite the fact that I have 8 weeks and 3 days to go. Also I've just passed the rough 10,000 word mark which is pretty spiffy if I do say so myself, as it's a milestone on the way to completion and an order of magnitude larger than anything I've ever written on one subject before.

I would really like to share some parts of it with my readers, but I'm also aware that I maybe shouldn't, as potentially my final marker could end up reading it and getting the wrong impression. It will probably undergo some serious revision before it reaches its final incarnation.

Suffice to say, I've been writing about the games Guitar Hero, Audiosurf and Everyday Shooter which are all must-play games for people interested in what the future of music in videogames looks like. [Or could look like it everyone reads my thesis ;)]

So I'll leave it at that for a while and hope it satisfies the hunger of those desperately desiring an update on my work.

Cheerio!