From Eight Bits to Infinity: Music Composition and Video Games
The interaction between music and technology is often discussed in terms of expanding capabilities. As technology improves—data storage becomes less costly, computer processors become faster, manufacturing methods become more consistent—musicians might also expect improved audio fidelity or more varied synthesis techniques. But it might be that, just as often, technical limitations propel innovation, and many of the cutting-edge technologies available to musicians were created to imitate, albeit expeditiously, much lower-tech sounds. Sampling computers and sequencers, for instance, have stylistic roots in Jamaican dub music, which made heavy use of re-arranging music from spliced tapes and other forms of clever manipulation. 1
When Yoko Shimomura told the BBC about her early days as a composer for Nintendo’s Famicom Disk System, she defined her work largely in terms of its technical limitations. “There was very little depth. I could have two sounds—melody and bass,” she said. “The space I had to create all the music and effects was about 20 megabits … that meant the music had to be very short. I thought it was impossible, but everyone before me was doing it, so I had no choice.” Video game systems of the time were severely limited in terms of audio capabilities, “limited to a few simple tones and one ‘noise channel’ that could be used for percussion and sound effects.” 2
Even a decade later, with game soundtracks that could “adapt” as a player moved about the game’s space, composers and programmers had to resort to clever tricks to create sonic diversity. Grant Kirkhope, who scored the Nintendo 64 game GoldenEye explains:
In a MIDI file, you get 16 channels, so you can have 16 instruments. So when I’d write the tunes, I’d find out what areas the design guys would want to have in each level, and I’d allocate a certain amount of instruments to each area. When the player crosses that line, it signals to the audio engine to fade down, say, channels one, three and six and turn up two, four and eight. 3
Technical challenges and music makers’ responses to them often lead to new genres or paradigm shifts in music. And while the historical importance of various genres should not be understated, a discussion of what constitutes a new kind of music is difficult. Genuine shifts in a cultural understanding of what music is—the appearance of truly new kinds of music—might be astonishingly rare.
If striking out—consciously or subconsciously—for a new definition or expansion of the difference between unsystematic sounds and genuine music is the defining characteristic of new music, few places might offer more promising terrain for exploration than the worlds of video games. They are a form of entertainment like no other: simultaneously constrained by technological limitations while facing the demands of supplementing a world and being listenable. As their interactive medium develops, we find game music in a unique space that might represent a truly new shift in our understanding of music.
Understanding game soundtracks as a new form of music
It might initially seem counter-intuitive to discuss video game music as a major departure from previous schools of thought about the nature of music. The scores and soundscapes of video games are designed to be listenable and straightforward, traits which might often belie repetition and blandness. However, video game music and sounds are separate from previous forms of music because of the open-ended, interactive nature of the medium and, by extension, the constraint that they communicate information about interaction.
If we want to understand why a video game soundtrack might be fundamentally different from the soundtrack for other audio-visual media, we should first consider the relationship between the audience for video games and TV or movies. Iain Hart notes that, while both video game and audio-visual dramas have soundtracks that communicate information about what is happening on screen, the requirements placed on players and viewers are quite different. “[Players] require physical engagement … Rather than being simply a method of control, physical engagement enables a duplex flow of information between the player and the game.” Audiences, however, only observe and interpret a set flow of information.
A game’s bidirectional exchange is important for keeping generating interest, and it also alters the semiotic understanding of what a player is experiencing. Hart suggests that because contemporary games are filled with cascading choices that affect a player’s strategy, the meanings conveyed by various audio cues within the game’s score or sound effects are not static. 4
Critics might take issue with Hart’s claim that video game music is essentially performative on the parts of both the player and the composer (and software developers, artists, etc.). However, his analysis of the differences between the semiotics of both stand-alone musical compositions, film scores and video game music is quite compelling.
The meaning of a purely auditory composition is solely derived from the preexisting notions brought by a listener. A film’s score often provides information about on-screen events by using understood modalities of music—a bold melody in a minor key communicates a villain, for instance. But as a player’s choices through a long-form game compound into more varied experiences through an individual’s play of a game, the response of a player to game music will change. The auditory cue for the entrance of a certain kind of enemy or element of game play might cause wildly different reactions based on the a player’s prior choices.
The bidirectional nature of video game music is not limited to the complex, open-ended adventures of contemporary console and PC games. Even much more limited musical interactions within games can manipulate how players respond. The Nintendo DS game Electroplankton is plotless and scoreless, instead inviting players to compose sounds and music by manipulating on-screen avatars reminiscent of plankton under a microscope. Games like Electroplankton and even more straightforward puzzles such as Tetris use music in a way that might induce what Mihaly Csikszentmihalyi has referred to as “the flow”: a pleasant state of focus derived from an interesting challenge. 5 As music accelerates or grows more complex—and challenge increases—the growing pressure might also heighten a player’s enjoyment of the tension between success and failure.
Furthermore, the forces of narrative meaning and a player’s physiological response to music might combine in some instances. Tim Summers analyzed a scene in the 1994 Super Nintendo game Final Fantasy VI, in which a player controls a character performing an opera as part of the game’s story. In order to advance through the game, a player must successfully choose the upcoming sequences of the opera’s aria. He writes:
The sequence … is particularly notable for the effusive reports of emotional responses that typically accompany its discussion in casual and journalistic discourse. Even though the sequence after the aria is more obviously comic, players nevertheless highlight the main aria as a profoundly affecting experience. 6
Although this sequence is significantly more linear than the previously discussed examples, it still points to Hart’s notion that the incorporation of music in games is necessarily performative. Through the game’s opera, Summers writes, “the player is brought into engagement with the opera through the necessity of paying close attention to operatic materials … there is a musical process that makes the case for opera, despite its overt artifice, as somehow speaking an emotional truth.”
Video game scores are a new space for music that has elements of semiotic meaning and performance in two directions—and this is perhaps the medium’s greatest departure from other forms of music.
Expanding worlds, expanding compositions
While early examples of video games scores point to the limitations of digital storage space or audio processing, newer games create challenges for composers by pushing the limitations of creativity. Procedurally generated games such as No Man’s Sky for the PlayStation 4 offer players the chance to explore a world that is, for all intents and purposes, boundless. How does a composer write for a game that might be infinite—and do so in a way that is still engaging?
Principles of machine learning might be useful for composers to utilize as video game environments expand beyond what is creatively feasible. Spencer Lee Churchill offers some context for the use of recurrent neural networks for composing game scores.
“Designing music for various locations, such as a cave or underwater regions, has become complicated since the game map is consistently variable. Having the map and music generate as the user plays would lead to a completely unique gaming experience,” he writes. The use of procedural generation in games has been largely limited to maps, but the opportunities presented by tools such as Google’s Magenta neural network are promising. 7
In one experiment, Churchill was able to train a neural network to generate interesting musical themes based on the data from existing video game MIDI files. However, he also points to the broader understanding of context that is necessary to score a video game effectively. He calls video game music essential in “creating an emotional road map for context, and when done well, the effect is synergistic. Video game music is driven by the events taking place in the story, and its sound is an attempt to enhance these events.” 8
Churchill faults current techniques for generating music that is not truly supplementary to the narrative or events of a prodecurally generated game. He also notes the music’s lack of complexity. Complexity, however, might not be as easy to define as a listener might believe intuitively. Nicholas Hudson applied compression techniques from information theory to classical music (as a proxy for “complex” music) and popular radio hits (representing more “simple” compositions). While the application of these labels can be debated, he found that classical music often had a significantly larger ratio between the uncompressed and compressed audio files than pop music. Hudson writes, “One might say—at least from an information theoretic perspective—that classical music is apparently complex but really simple, while popular music is apparently simple but really complex.”
He argues that the real simplicity underlying apparent complexity “may reflect an intrinsic appreciation for successful information compression that is held below our conscious awareness.” 9 This line of thinking might be instrumental for future composers of video game sounds and music. To create an engaging composition, it might be more meaningful to convey emotional or semiotic meaning efficiently, in a way that listeners “decompress” over time.
Replay value
The influence that video game composition has had on popular culture borders on undeniable. Game themes and sound effects, from the 8-bit tunes of Super Mario Bros. to the symphonic scores of the Halo franchise, are widely known—even outside of gamer circles. Much like our favorite songs, we often come back to the games we enjoy and discover new things within their worlds.
But video games have important differences from static artistic compositions. Each play is different because of the nature of interaction. While the earliest game soundtracks might be understood as only supplementing the information on screen or intensifying an existing emotional cue, a thorough look at the field allows an appreciation for the diversity of meanings and functions that music in games serves.
As composers are asked to create around the constraints of meaning and world-building for ever more complex software, we might become more aware of their craft as a discipline distinct from other forms of composition or sound design. The application of machine learning seems somewhat lacking now, but insights from information theory and deep learning could yield interesting results in artificial creativity that are useful for other fields in computing. As the range of experiences within games grows, so too do the tests of composers’ creativity.
1 Veal, Michael. Dub: Soundscapes & Shattered Songs in Jamaican Reggae. Wesleyan University Press (2000): Middletown, Connecticut.
2 Savage, Mark. “Top Scores, From Pong to Red Dead: Can video game music change the way you play?” Accessed 24 March 2019.
4 Hart, Iain. “Meaningful Play: Performativity, interactivity and semiotics in video game music.” Musicology Australia 36 no. 2 (2014): 273–290.
5 Abduhamdeh, Sami; Csikszentmihalyi, Mihaly. “The Importance of Challenge for the Enjoyment of Intrinsically Motivated, Goal-Directed Activities.” Personality and Social Psychology Bulletin 38, no. 3 (2012): 317–330.
6 Summers, Tim. “Opera Scenes in Video Games: Hitmen, Divas and Wagner’s Werewolves.” Cambridge Opera Journal 29, no. 3 (2018): 253–286.
7 Churchill, Spencer Lee. “RNN Composition of Thematically Diverse Video Game Melodies.” The Computer Games Journal, no. 8 (2019): 41–58.
9 Hudson, Nicholas. “Musical beauty and information compression: Complex to the ear but simple to the mind?” BMC Research Notes, no. 4 (2011):