Table of Contents

I. Abstract
II. Keywords
III. Introduction
IV. Significance and Audience
V. Domains and Precedents
VI. Methodology
VII. Findings, Reflections, and Next Steps
VIII. Acknowledgements

Here is a link to the final Thesis paper in PDF format.

I. Abstract

Originating as an investigation into the relationships between rhythm and technology, RhythmSynthesis uses color, shape, and sound to demonstrate how our understanding of visual music, computation, and tangible, audio-visual interactions can be applied as considerations in musical compositions. As an electronic musical instrument, the piece allows for experimentation, rewards for mastery, and is a vehicle for expression.

This paper outlines the RhythmSynthesis Thesis project, as completed by Ryan Raffa, a Master’s Candidate in the program of Design and Technology at Parsons the New School for Design, 2011.

By asking questions about how visual music can be used to perceive rhythm, what ways visual rhythms can be used for composition, and what ways composition can be intertwined with improvised performance and notation, this project illustrates that sound is a reliable and effective way to provide users feedback for making visual composition decisions, judgements, and actions. From amateur to professional musicians, the instrument allows for unique, personal interactions and expressive choice.

II. Keywords

Rhythm, Sound, Technology, Composition, Musical Performance, Design Research, Tangible User Interfaces, Computation

III. Introduction

“We could use sound for much more communication than we do”
– Donald Norman, The Design of Everyday Things (1990)

“Teach the listeners how to interpret what they hear.”
– Matt Wright, Research Director at Allosphere (2011)

“you know exactly as I do what art is: it is nothing more than a rhythm. But if that is so, I shan’t bother myself with imitation or the soul, but purely and simply produce rhythms with whatever takes my fancy.”
– Kurt Schwitters (date unknown)

CONCEPT
New musical interfaces are necessary to further explore the complexities of rhythm. RhythmSynthesis proposes a new instrument for composition and performance to continue such exploration. Originating as an investigation into the relationships between rhythm and technology, RhythmSynthesis applies color, shape, and sound to demonstrate how our understanding of visual music, computation, and tangible, audio-visual interactions can be applied as considerations in musical expression.

There is a long history of investigation into rhythm (rhythm and sound, language, writing, etc) as well as the connections between the visual, sonic, and computational. The goals of this thesis investigation are:

• Use technology as a way to reveal alternative models and interactions with rhythm using color, shape, and sound as compositional tools.
• Provide an interaction that proposes multiple strategies to groups or individuals in order to reveal positive outcomes.
• As an electronic musical instrument, the piece should allow for experimentation, reward for mastery, and be a vehicle for expression.

DESIGN QUESTIONS
This project began by asking: in what ways can visual music be used to perceive rhythm? As the research progressed, additional questions surfaced, such as how is rhythm embedded within our visually-dominated experiences, how is the visual encoded with sound, and what are ways visual rhythms can be used for composition.

The building and prototyping process was key to exploring these questions, and in doing so, the final question of what are ways in which composition can be intertwined with performance and experimental notation guided the completion of this Thesis.

What is rhythm?
In the first presentation of this Thesis concept (September 22, 2010), rhythm was defined as, “movement or procedure with uniform or patterned recurrence of a beat, accent, or differing conditions.” This definition was a combination of common wisdom, institutionalized knowledge, and my personal relationship with the term. Keywords from this initial definition that were carried through this project were “movement,” “process/procedure,” and “recurrence.”

A term that is missing from this definition is “perception.” Whether it is the light reflecting off a city wall, the sound of the breeze blowing through a tree, or the computer on the table top, the world that surrounds us is constantly giving off some form of rhythmic pulse. Unintelligible vibrations, such as radio transmissions and infrared light, is where technology can be exploited to map those frequencies to a space where they can be perceived.

Concepts such as the Gestalt principles¹ and the multiplicity of speech cadence, the notion that we are perpetually in the process of attempting to perceive and create meaning out the rhythms that surround us, is paramount to this project.

IV. Significance and Audience

SIGNIFICANCE
In light of the constructivist view of learning,² the mental models and templates that allow us to understand our experiences are based on what we hear, see, feel, and smell. We now have the opportunity to use technology, not as a singular lens, but as an addition to our current senses to perceive and apply meaning to rhythms.

Much has been written about how we perceive and apply meaning to all that surrounds us. The French philosopher and music critic, Gabriel Marcel, wrote:

Meaning arises when an individual becomes aware, either affectively or intellectually, of the implications of a stimulus in a particular context. As long as behavior is habitual and ‘unthinking’ the stimuli presented to the mind are neither meaningful nor meaningless…[O]ur experience of such stimuli stands in the same relationship to the meaningful-meaningless axis as the concept of ‘amoral’ stands in relation to the moral-immoral axies.³

Similar to the process of viewing the color coding of a resistor (ex. a 4-band resistor with the color values red, red, brown has a resistance value of 220 Ω), the colors of an object are often used to understand the physical substance and characteristics of that object. This project proposes the use of technology to introduce, rehearse, and create dialogue around the associations of color and sound.

The previous decade’s worth of efficiency-based technological advancements, such as the fax machine, cell phone, and word processing software, have been predominantly focused on the improvement of systems that we currently had in place. In Why the Computer Revolution Hasn’t Happened Yet, Alan Kay writes:

When I was working on personal computers thirty years ago, I realized that as long as a computer sat on a desk and looked like a time-sharing terminal, it wasn’t participating in the actual revolution. It was participating in the automation of the old.⁴

We are now in a time where technology can and should be designed to explore new aspects of our understanding of and expression of our perceptual experiences.

During his talk at SXSW 2010 titled “Program or be Programmed,” Douglas Rushkoff stated, “We are attempting to operate our society on obsolete code…Our social software are basically legacy systems that we don’t even remember.”⁵ While we cycle through a series of technological advancements, we are repeatedly embedding our previous understandings and biases into the new systems of knowing and operating.

In Hertzian Tales, Anthony Dunne wrote, “In a world where practicality and functionality can be taken for grated, the aesthetics of the post-optimal object could provide new experiences of everyday life, new poetic dimensions.”⁶ With the post-optimal object, designed electronics that introduce culturally significant interactions and enriching experiences, taking computers and other technology out from behind the screen, keyboard, spreadsheet, or “black-box” and introducing new interactions, will move the computer away from the printing press and improve it as an instrument in learning and understanding.

In an era where media (specifically sound) consumption has become codified, standardized, and prescribed, there is a large space for play within the realm of media and sound experiences and is an essential understanding for why this project should be done now.

Sound interactions will provide the opportunity to familiarize an individual or group with new perspectives on existing realities. By developing this ability to express human-condition based feelings or processes through sound, the hope is that new views and ways of thinking will permeate and people will be, as psychologist Mihaly Csikszentmihalyi writes, “partially freed from reliance on the objectified consciousness.”⁷ The current applications of electronics and sound, although portrayed as worthwhile and meaningful experiences, are basically staged events for purchasing and consuming. In turning “consumer” into “maker” through simple interactions and computational-based systems, individuals and groups will provide themselves sonic and visual experiences that are original, creative, and locally-produced, technically mediated interactions.

There have been many that warn against the adoption of using technology as a method for perceiving, especially as it relates to sound. In the book Philosophy of Modern Music, Theodor Adorno writes, “The adaptation to machine music implies a renunciation of one’s own human feelings.”⁸ In an era where the majority of all popular music is recorded and performed with the drums toeing the line of a click-track, the voices auto-tuned, the guitars processed through digital foot-pedal interfaces, are we all a little less human for hearing (and possibly enjoying) this metronomic machine music? Is it Adorno’s feeling that the click-track beat falls where a human hand would have placed it? Or is it that we have relinquished (or surrendered) our autonomy and placed a computer behind the wheel of our culture and our music?

Adorno overlooks one of the most exciting areas of the exploration of beats and rhythm; that is the area that resides between the spaces left by the computer’s methodical measurements of time. Having spent hours upon hours hearing a metronome measuring out hurdles of time, the drummer will find the deeper, more human elements of rhythm that exist at or around the automated time-keeping.

In addition, as opposed to focusing on musician as performer, Adorno could also look to dancer as performer. In Urban Media and the Politics of Sound Space, Peter Shapiro writes, “music organises time with its rhythm, and dancing to it is one of the few ways we have to suspend and stretch time. Even more than solar and lunar cycles, rhythm machines are presently the arbiters of time.”⁹ Using the steady pulse of the machine, our human motions warp its measurements with movement.

Jonathan Kramer, in Postmodern Music / Postmodern Thought, lists out 16 characteristics which he deems representative of postmodern music. Two that are supremely relevant to this project are that postmodern music “considers music not as autonomous but as relevant to cultural, social, and political contexts,”¹⁰ and postmodern music “considers technology not only as a way to preserve and transmit music but also as deeply implicated in the production and essence of music.”¹¹

In reflecting on these two statements, similar to games and the playing of games, music is not something that resides outside of what it is to be human or social beings. Rather it is a direct comment on, a clear reflection of, and a contributor to our nature and our culture. Does that explain our natural ability and desire to understand and physically respond to rhythm?

AUDIENCE
The objective is to provide an experience which allows the novice a low enough barrier of use so that they are able to participate and create, while at the same time providing the expert a balanced level of feedback so there is a reward for experimentation, mastery, and exploration. In addition, the process of using the instrument is a method for practicing and developing mental connections between the visual and sound.

There has been much work in the area of speech and learning (from Lev Vygotsky’s Thought and Language to Noam Chomsky’s Syntactic Structures), but research methods and practices need to be developed to map and document human’s natural links between sound, perception, and understanding. There must be a set of standards put forth that can be debated and improved. “If they are to be useful, sounds must be generated intelligently, with an understanding of the natural relationship between the sounds and the information to be conveyed,” Donald Norman writes¹². We are in the experimental phase of this field and much work must be done to share, critique, and document the methods, processes, and connections between data and sound.

As an example, JoAnn Kuchera-Morin and her team of engineers, scientists, musicians, and artists have been collaborating on a 3-story “dynamically, varying digital microscope,” called the Allosphere, at UC Santa Barbara. In her TED talk describing this project, Kuchera-Morin asks the audience to, “imagine if a team of physicists could stand inside of an atom and watch and hear electrons spin…[I]magine if a team of surgeons could fly into the brain as though it was a world and see tissues as landscape and hear blood density levels as music.” At UC Santa Barbara, Kuchera-Morin and her team represent the early explorers of this new field.

In his NPR article “Particle Pings: Sounds Of The Large Hadron Collider,” Andrew Prince writes about Lily Asquith, a physicist who worked with the LHC at CERN, the European Organization for Nuclear Research, who used LHC’s data to create sound pieces. In Prince’s article, Asquith states, “You tend to personify things that you think about a lot…So we’ll map, for example, the first column of numbers, which may be a distance, to time…And we may map the second column of numbers to pitch, and the third, perhaps, to volume.”

Understanding the difference between viewing mountains of data and providing an auditory version of the same information, Asquith states, “You could certainly have an alarm system which told you when, for example, you have an event which looks ridiculous according to what you’ve expected,” she says. “And that’s quite difficult to do using your eyes.” Prince notes that Asquith assigns particles with personalities, colors, and sounds; according to Asquith, “it’s much easier to use your ears than your eyes, particularly with something that’s changing over time.” What we are seeing in sound is similar to what we have seen with developments in data visualization, the attempt to interpret the story or narrative contained within the data.

At the beginning of “Journalism in the Age of Data”, IBM researcher Fernanda Viégas states, “Half of our brain is wired for vision.” Viégas goes on and says, “Vision is the biggest bandwidth that we have in terms of sensory information to the outside world.”¹³ Our understanding of what surrounds us, and thus the meaning we apply to it, is driven primarily through what we see. Our bias for specific senses needs to be acknowledged. Will a more complete understanding of information be provided through multiple-sensory data experiences?

Responding to the combination of sensory information to perceive data, Matt Wright, Research Director at UC Santa Barbara Center for Research in Electronic Art Technology (CREATE), wrote, “Different sensory modalities have different affordances. Hearing provides the most accurate temporal resolution. Sight provides the best spatial resolution. Touch provides the shortest reaction time.”¹⁴ Using our receptors as the constraint, these interactions with information can be designed in a better way.

Each person approaching this instrument will bring their own experiences and archetypes of rhythm (whether grounded in color, shape, sound, or any variation of the three) as well as his/her native ability in composition; thus, there needs to be a simplicity in rules and an increased complexity in how colors, shapes, and sounds interact. With Csikszentmihalyi’s understanding of Flow as well as the unpredictable nature of musical instruments, I plan to address these challenges with multiple levels of play.

V. Domains and Precedents

• rhythm
• sound
• technology
• tangible user interfaces
• visual music
• composition

PRECEDENTS
Visual Music
Many have looked at the connections between the semiotics of the visual and the perception of sound over the past three centuries. Musical notation and written language represent traditional approaches while the work of Paul Klee and Wassily Kandinsky at the Bauhaus, Len Lye and Oskar Fischinger’s moving images, as well as John Cage and Merce Cunningham’s collaborations can be seen as more experimental approaches to understanding these connections.

While discussing my plans for this project, fellow classmate, Manuel Rueda Iragorri, suggested I read the chapter “Bit by Bit by Bit” from the book Code. In the chapter, Charles Petzold dissects the underpinnings of bar codes (and other visual codes that can be seen by computers and by humans) and how their fundamentals are based on the binary. The majority of imagery we encounter outside of these codes are not only analog but represent the breadth of complexity in nature. By using a palette of specific hue values, the goal here is to take the next step with the technology that is available to push beyond a binary sense of sound and imagery.

Experimental notation techniques in works such as Ivan Vyshnegradsky’s Temple of Light (1943) and John Cage’s Concert for Piano and Orchestra (1960) illustrate the deinstitutionalizing of traditional notation in favor of exploration and experimentation. Sheet music was originally used by performers as a simple reference for music that may have presented and developed into elaborate forms and systems of performance structures. In classical music, notation fostered the idea of composer as maker/designer and the musician as performer. In Temple of Light, Vyshnegradsky designed specific color assignments for portions of prepared scores. This project takes this a step further and re-aligns composer and musician with non-traditional notation and performance.

Paul Klee’s Variations (1927) and Sol LeWitt’s Wall Drawing #273: Lines to points on a grid (1975) provide a framework for how the combination of variation in condensed line and color can be used to provide rhythmically complex compositions.

František Kupka’s The Cathedral (1913), Paul Klee’s Fugue in Red (1921), and Armin Martinmuller’s Hommage A Froberger Opus 225 – Lamentation (1981-1982) had considerable influence over how I envisioned the colored pieces that would be used within the interaction. The shapes painted within Kupka’s magnificent visual music piece was the original inspiration for using a light-based platform. The idea of the sun shining through stained glass, coloring the inside of a large cathedral provided me with the emotional and aspirational backdrop. I wanted to emulate how patterns of shape and color can resemble music as the sun moves the images across the floors, walls, and wooden pews, providing a brief, synesthesic-based experience.

Christian Marclay’s Chalkboard (2010) provided museum goers with a collaborative score that not only transformed the gallery walls at the Whitney Museum here in New York but allowed for a playful and direct interaction with sound creation. Instead of posing the question, “Can you compose music?”, I am presenting a similar scenario with the question, “Do you know what happens next?”

Toshio Iwai, the Japanese media artist and designer of the iconic, electronic musical instrument Tenori-On (2005), commented on how presently electronic instruments are “imitating traditional instruments,”¹⁵ and goes on to say, “the visual world and sounds are in an inseparable relationship…[I]t is more natural to experience sound and vision at the same time.”¹⁶ Iwai further supports the notion that new musical instruments that address the performer/instrument relationship are needed to explore the interesting interactions electronic instruments can provide.

With an understanding of the different styles of music, this thesis proposes what a collaboration between Larry Levan and John Cage would sound like if they had this tangible instrument to use.

Tangible User Interfaces (TUI)
When reviewing music educator James Mursell’s views on rhythm and music, Reimer and Wright wrote, “rhythm in music must literally be felt in order to be experienced…Listeners perceive rhythmic patterns in sensory media (even media that seem nonrhythmic) and invent structures if needed.”¹⁷ The mental structures that we create to understand and rationalize are based on personal collective experiences up to this point. What needs to change is not specifically the methods in which we learn, but the structures we use to understand our internalized archetypes.

With the theory of learning that Seymour Papert and constructionist theory puts forth¹⁸ and in light of James Mursell’s views on rhythm, I seek to provide a tangible experience where users are free to experiment and build out their own compositions is key to the success of this project.

Projects such as James Patten’s Sensetable (2001), Golan Levin’s Scrapple (2005), and Reactable Systems Reactable (2009) all address how TUI’s can be used in creative sound experiences. This project takes the next step by freeing the user from precomposed rhythmic structures and looks for the interface to allow for new patterns to emerge that address the inevitable shift of expressional needs. Although there will always remain specific constraints for each unique interface, the goal was to address the desire to explore and discover through visual and sonic rhythms and resist reinforcing the familiar.

Other recent projects, such as s David Merrill, Tim Hankins, and Jocelyn Robert’s Circular Optical Object Locator (2002), Atelier Hauert Reichmuth’s Instant City (2006) and James George and Daniel Peterson’s Temporary Cities (2010) show how visual music can blur the distinction of object, sound, and composition.

Rhythm Theory
Rhythm is part of the daily experience but rarely does an individual’s relationship to these patterns and pulses enter into a conscious and understood action. Our full connection to these movements and motions is relatively unknown. Henri Lefebvre, the French sociologist and philosopher, and Catherine Régulier wrote extensively about rhythm and the everyday life and noted:

Everyone thinks he knows what this word means. In fact, everyone perceives it in an empirical way that is very different from knowledge; rhythm is part of the ‘lived’, but that does not mean that it is part of the ‘known’. There’s a big gap between an observation and a definition, and an even bigger one between grasping a rhythm — the rhythm of a tune, of breathing, or the beating of the heart — and being able to conceive of the simultaneous intertwining of several rhythms, their unity in diversity.¹⁹

In discussing rhythm, music, and our minds, Oliver Sacks, professor of neurology and psychiatry at Columbia University Medical Center, wrote, “We anticipate the beat, we get rhythmic patterns as soon as we hear them, and we establish internal models or templates of them.”²⁰ Similar to Mursell’s view of rhythm, we cannot help but create models and templates that allow us to understand; however, Sacks brings up a valuable point patterns don’t necessarily have to be felt but can also be heard and seen.

Matt Wright, responding to Sacks’s comments, wrote, “Our minds try to make sense of the world via our senses, and there are many examples of cases in which we infer a continuous world to ‘fill in the blanks’ when sensory input is incomplete. So if we hear a tiger behind us we turn and expect to see one.”²¹ Similar to the gaps in rhythmic understanding discussed by Lefebvre and rhythmic models by Sacks, despite having incomplete information, we formulate the missing pieces.

Sacks discusses further the implication of rhythm and the group mind, and writes:

[t]he binding is accomplished by rhythm–not only heard but internalized, identically, in all who are present. Rhythm turns listeners into participants, makes listening active and motoric, and synchronizes the brains and minds (and, since emotion is always intertwined with music, the “hearts”) of all who participate.²²

The rhythmic patterns and models that we have are not just constructed by individuals but are a component to group understanding and learning.

VI. Methodology

PROCESS
The design process I propose is not strictly a problem-solver. It is rather a continual model for exploration. Thus, is it inappropriate to think of it as a never-ending cycle of prototypes within the cycle of creating? The idea of continuous prototyping on a set of processes, principles, or design concepts is being used extensively in this thesis project. Each work is a part of a much larger work (our life’s work), and these works describe micro-cycles of the much larger life cycle.

Throughout the process this semester, I used a set of initial questions and constraints (theoretical, material, and/or interaction-based) to create a series of sketches. From there I developed a strategy for how best to execute these prototypes based on my current skill set and available resources (ex. what materials are available in the left-over bin in the Physical Computing room). Often times, I would take on more challenging prototypes that were outside of my comfort zone in order to learn a new skill in the process.

Michael Hensel and Achim Menges, in their article “Patterns in Performance-Orientated Design,” investigate patterns that “arises out of the interaction of man-made interventions with the natural environment”²³ (as opposed to aesthetically-driven or natural-model driven pattern making) within architectural design. Hensel and Menges write:

From flocking patterns to fractal geometry, the reduction of complexity to a single set of abstract rules has shifted attention away from the messy, embodied realities of existence towards the very real need to create working methodologies that are inclusive, process based and physical.²⁴

The goal of RhythmSynthesis is not to create complex, visual structures of sound representations, but instead to provide a set of simple rules that allow users to create visual music pieces that transmit the visual and sound rhythms embedded within.

David Carroll, who participated in the final Fall critique of MFA DT Major Studio Interface (2009), said, “look to make the experience more real than real.” This phrase is very influential in how to approach the design process. Now that optimally designed objects have reached the point of saturation (ex. as Daniel Braithwaite of the London College of Communication wrote, “How can an alarm clock wake us up any better than one that is currently available?”²⁵), moving beyond efficiency and providing experiences for the brain and body that contextualize and alter how people view their daily lives should be a primary focus.

John Cage consistently emphasized that he was not interested in objects; instead he was interested in processes.²⁶ Through these processes, Cage was retelling or creating new versions of a similar story, the story of sobering and quieting the mind to become susceptible to divine influences. For this thesis, the process is to realize rhythm through the interaction with technology which is key to exploring and illustrating why certain decisions are made from prototype to prototype.

Cage, in The Tiger’s Eye, wrote “any attempt to exclude the irrational is irrational. Any composing strategy which is wholly ‘rational’ is irrational in the extreme.”²⁷ This is a powerful statement, especially when applied to the design process. Does the final form need to be “rational”, or is the “why?” and “who cares?” capable of holding a set of irrational outcomes together to say something meaningful?

Cage goes on and writes, “Structure–that is, the division of a work into successive sections and phrases–is controlled by the composer’s intellect. The method of composition and the composer’s materials may either be controlled through rational decision or may be the spontaneous result of improvisation and inspiration. Form, the final result, is the actual content of a composition.”²⁸ The final form of this Thesis was a direct result of rigorous conceptual research and development, continuous prototyping, and a balance of genuine improvisation and purposefully, designed selection.

Prototype Overview

For the first series of prototypes, I focused on 2-dimensional, screen-based projects. After reflecting on the experience and receiving valuable feedback, I began creating tangible, object-oriented scenarios.

PROXIMITY INSTRUMENT
For this prototype, I used three specific aspects of the user and object interaction as constraints to better develop these ideas. The aspects were:

• proximity
• user input via physical interaction
• user relationship to the installation piece

Using an Ardweeny, small bread board, MaxBotix Ultra Range Finder, a small plastic encasing, hardboard, and a RadioShack speaker, I created a proximity-based sound box that is mounted on the edge of a bicycle wheel. The mount (made of leftover scraps of wood from the Physical Computing room here at Parsons) is attached to the wheel with zip ties.

I wanted to provide some form of physical input for the person interacting with the piece as well as recognize that if multiple people were involved, each person should have a unique sound experience. By including the bicycle wheel as the method for activating the installation piece as well as PVC pipe that can easily be moved, the user has a direct impact on the sounds that they hear.

As the sound box rotates around the hardboard platform, the PVC pipes that are placed on the platform trigger tones based on their distance to the sound box. The tone rises and falls as the pipe comes into and out of range.

I conducted initial prototyping of possible patterns and designs that could be projected or otherwise provided on the platform. I initially played with simple patterns of circles, but based on a suggestion from a classmate, I used patterns based on the golden ratio and the fibonacci sequence as well.

Having addressed location and proximity in this prototype, the next prototype was focused on the consideration of color. Using the understanding of subtractive color, the prototype uses the color of the object to determine the tone that is played.

TANGIBLE INSTRUMENT MOCK-UP
For this prototype, the goal was to demonstrate the possible changes in the interaction when I moved from a 2-dimensional on-screen space to a 3-D physical space. The color, size, and location of the objects placed within the radar field all have an impact on the sounds produced.

TANGIBLE COLOR MUSIC INSTRUMENT
This prototype moves my “radar player” out into physical space and allows users to manipulate and move translucent, colored plastic pieces over a lit background. A camera above the lit platform provides information to my openFrameworks application that is looking for specific data points of hue, saturation, and value. As a virtual “radar” passes over combinations of colors, users are able to compose through exploring sound and the visual.

I focused on three specific aspects of the object and user interaction, which were:

• color
• look and feel
• collaboration

I made 3 different versions of this prototype in 3 days to move quickly through a set of possible components and interactions. The components that were used consistently throughout were several pieces of translucent, colored-plastic (1″ X 1″ and 3″ X 3″), light, and a PS3 camera.

Initially I used a light table which provided a quick set-up but poor image quality (hallogen light) and too small a surface to allow users to work well together. The second and third iteration made use of a milk crate, cardboard spray painted white, and a translucent piece of plexiglass (18″ X 24″).

I set these prototypes up in the 10th floor lab here at Parsons and had a series of Design and Technology students (both MFA and BFA) try each of them. A total of 5 groups of students tested this instrument.

Within each of the scenario’s, the sets of user groups attempted different approaches to the plastic pieces. I feel these approaches differed based on the user’s own creative style. Some stacked them, others bound them with tape, and one even tossed the plastic pieces in the air and let them fall randomly on the lit plastic platform of the instrument.

In relation to the collaborative element, I feel the actual set-up of the prototypes influenced the users. The first prototype only allowed people to approach it from two sides, so there was a limited amount of space. Comments from users such as, “Oh, I’ll wait until you’re finished,” or, “Sorry,” were prevalent.

The second and third prototypes allowed users to approach the instrument from 3 sides, and the level of collaboration, such as combining colors and adding to a single piece together, was much more evident. The feedback that I received was directed more towards possible next steps and additional elements to include, which I felt reflected a successful prototype that provided an accurate representation of the project goals.

SOUND TESTS
For the next prototype, I reserved a classroom on the 10th floor of 2 W 13th Street (room 1013) and set up the prototype along with a large guitar amp, various effects pedals (delay, distortion, etc), and a drum machine. The group of people who tested the project included fellow students, friends, musicians, and Parsons Faculty.

I included a few updates to the program, which included a full 12 tone scale matched to specific hue values within the full visible light color spectrum, tempo/speed changes, and the use of object area to determine appropriate scale (meaning the larger the piece, the lower the octave; the smaller the piece, the higher the octave). I also updated the light platform to provide a better color for camera detection. The PS3 Eyesight is biased a little toward cooler color values.

The reactions and results were great, and it provided me plenty of feedback for taking the next steps with the interface and the software portion of the project. In general, users exhibited was a wide array of approaches when creating musical compositions. Some were methodical. Other’s used the space between the platform and the camera to manipulate the sounds. Other’s just went crazy.

I finished the hardware portion improvements which includes a new aluminum arm for the camera mount and an enclosure for the camera. I removed the PS3 eye from its original encasing and reduced the size of weight and space the camera took up.

The main goals for this original experiment was to rigorously test the software to see if it was stable enough for long performances and to see how musicians and artists dealt with the interface. I truly feel that the audio is enough feedback for performers to know where the playhead is, but I needed to put the new hardware in front of people and see. Also, up to this point, the majority of the people testing my project were from the MFA Design and Technology program, and I wanted a fresh set of eyes and ears on the project.

I received a lot of feedback (and some encouragement) that will help me with the next steps of this project. The primary comment dealt with how best to provide the musician feedback as to what is actually happening and to help them make decisions on how to use the pieces. Some people preferred more audio feedback, such as more drastic changes between the sounds when pieces are rotated or differ in shape. Others preferred to have more visual feedback so the user would know the location of the rotating playhead.

INTERFACE TESTS
I invited Brett Tieman, J. Nicholas Moy, and Jeremy Gough (who I played with in the band, The Mugs) as well as fellow MFA DT students Haeyoung Kim and Matt Ruby to test the interface of my final project.

I incorporated a number of updates to both the software and hardware, which included:

• play head movement tied to computer clock for more accurate timing
• “sliders” along the right and left edges to control speed and volume
• updated oscillators that generate sounds within openFrameworks application
• new acrylic top that diffuses light more evenly
• purposefully designed shapes and colors for acrylic pieces used to make sounds

The comments centered specifically on the sounds that were generated by the application, and they also reaffirmed my belief that when the sounds provide more distinctive characteristics (such as range of octaves and speed of attack), users are able to compose without obvious or deliberate visual feedback.

VII. Findings, Reflections, and Next Steps

FINDINGS AND REFLECTIONS
Sound is a reliable and effective way to provide users feedback for making visual composition decisions, judgements, and actions. Early feedback from test sessions centered around providing users more visual feedback of the position of the rotating, play-head, but once the sounds became more distinct, represented a greater sonic range (specifically the mid-to-higher range), and had a pronounced attack, users relied less on a strictly visual association process and worked with combined senses to make expressive choices.

Both amateur and professional musicians, along with visual artists, engineers, programmers, and designers, found the experience enjoyable. Over sixty people (as individuals and groups) tested the instrument, and each testing session yielded an array of different strategies, approaches, and executions. From scientific approaches to more adventurous tactics, the testing sessions demonstrated unique and personal interactions.

There is more work needed to align the full spectrum of color to the full spectrum of sound. As shown in the chart below, only a fragment of the possible hue and frequency values have been associated and performed. The spaces between the indicated values is an interesting space to play for future projects.

NEXT STEPS
There are a number of challenges and exciting updates that need to be addressed in the coming months, which can be broken down into three specific areas:

• Different user needs as they relate to Flow
• Variety of experience
• Extensibility

Matt Leacock, Yahoo user experience designer and the game designer behind Pandemic, spoke of his challenges during his “What Can Board Games Teach Us?” lecture at Google. Leacock illustrated these exact same challenges when reviewing his approach to Pandemic. I see many parallels between game design and instrument design as my project develops.

The chief parallel between Leacock’s game and this music instrument is the creation of a system that responds to many types of users. As a musical instrument, the interaction must have a simple and transparent rule set implied by its affordances and respond to the skill level of the participants.

In addition to that, the goal is also to provide ease of composition and variation. By providing multiple instrument states (ex. simple composition mode and complexity mode) as well as reward for exploration, my goal is to encourage users to try multiple strategies when interacting with the instrument.

Embracing the core beliefs of open-source as referred to by Steve Lambert in his talk “Free” at the New Museum, the system should be available to be modified, updated, adjusted, and reinvented by the users. As opposed to a black box that provides many barriers to reinterpretation, this project should imbue and embody open-source theoretic aesthetics.

In the book dadaism by Dietmar Elger, Kurt Schwitters was quoted as saying, “you know exactly as I do what art is: it is nothing more than a rhythm. But if that is so, I shan’t bother myself with imitation or the soul, but purely and simply produce rhythms with whatever takes my fancy.”²⁹ This approach to the purpose and simplicity of rhythm should be one of the many voices that can be heard through the instrument.

Near the end of his TED talk about Flow, Csikszentmihalyi states, “the question we are trying to address is how to put more and more of everyday life in the flow channel and that is the kind of challenge that we are trying to understand.”³⁰ I feel that by providing an exploratory instrument that provides simple interactions with rhythm and technology, we will find new ways of seeing and understanding the rhythms we encounter in our daily lives.

VIII. Acknowledgements

I would like to thank the following individuals for their support and guidance through the entire process:

Ivy Raffa, Todd Raffa, Ernesto Klar, Nora Krug, Katherine Moriwaki, Scott Pobiner, Louisa Campbell, Andrew Zornoza, Matt Ruby, Manuel Rueda Iragorri, Bryant Davis, Brett Burton, Jeff Crouse, Zach Lieberman, Zach Gage, Robert Ramirez, Kyle McDonald, James George, Matt Wright, Ann Dewitt, David Carroll, Cynthia Lawson, Jonah Brucker-Cohen, and Anezka Sebek

Thanks to all of the students in my Thesis class, Carebear Bootcamp Section, and AV Systems course.

This thesis is dedicated to my wife, Ivy Raffa. Ivy encouraged me to attend the MFA D+T program, and her constant love and support are the reasons I had the tireless energy to pursue this project and degree.

REFERENCES
¹ Soegaard.
² Piaget.
³ Marcel, 9.
^4 Kay, 8.
^5 Rushkoff.
⁶ Dunne, 20.
^7 Csikszentmihaly, 28.
^8 Shapiro, 132.
^9 Ibid, 139.
^10 Auner and Lochhead, 16.
¹¹ Ibid.
^12 Norman, 249.
^13 Journalism in the Age of Data.
^14 Wright, Interview.
¹⁵ Wiltshire, Interview.
¹⁶ Ibid.
¹⁷ Reimer and Wright, 130.
¹⁸ Papert.
¹⁹ Lefebvre, 193.
²⁰ Sacks, 261.
²¹ Ibid.
²² Ibid, 262.
²³ Michael Hensel and Achim Menges, 89.
²⁴ Ibid, 96.
²⁵ Braithwaite, 4.
²⁶ Cott and Cage.
²⁷ Bernstein, 118.
²⁸ Ibid.
²⁹ Elger, 60.
³⁰ Csikszentmihaly, TED Talk.

BIBLIOGRAPHY
Adorno, Theodor W., Wesley V. Blomster, and Anne G. Mitchell. Philosophy of Modern Music (Athlone Contemporary European Thinkers Series). New York: Continuum International Publishing Group, 2003.

Auner, Judy, and Joseph Henry Lochhead. Postmodern Music/Postmodern Thought. New York: Routledge, 2001.

Bernstein, David W.. “John Cage and the “Aesthetic of Indifference”.” In The New York Schools of Music and the Visual Arts (Studies in Contemporary Music and Culture, V. 5.). 1 ed. New York: Routledge, 2001. 113-133.

Braithwaite, Daniel. “To what extent is the relationship between innovation in academia and industry preventing Anthony Dunne’s ‘post-optimal object’ from entering the mainstream of consumer culture?.” http://www.danielbraithwaite.co.uk/db_diss.pdf. www.danielbraithwaite.co.uk/db_diss.pdf (accessed April 28, 2011).

Brougher, Kerry, Jeremy Strick, Ari Wiseman, and Judith Zilczer. Visual music: synaesthesia in art and music since 1900. New York, N.Y.: Thames & Hudson, 2005.

Carroll, David. “Design Fictions Critique.” Design Fictions Critique. Parsons. Room 1013, New York. October 6, 2009.

“Christiaan Huygens – Wikipedia, the free encyclopedia.” Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Christiaan_Huygens (accessed September 9, 2010).

Cott, Johnathan, and John Cage. “John Cage Interviewed by Jonathan Cott.” Internet Archive. www.archive.org/details/CottInterviews/ (accessed September 5, 2010).

Csikszentmihalyi, Mihaly. Finding flow: the psychology of engagement with everyday life. New York: BasicBooks, 1997.

Csikszentmihalyi, Mihaly. Flow: the psychology of optimal experience. New York: Harper & Row, 1990.

Csikszentmihalyi, Mihaly. “Mihaly Csikszentmihalyi on flow | Video on TED.com.” TED: Ideas worth spreading. http://www.ted.com/talks/mihaly_csikszentmihalyi_on_flow.html (accessed December 12, 2010).

Csikszentmihalyi, Mihaly. “Why we need Things.” S. Lubar and W.D. Kingery. History from Things. Washington, DC: Smithsonian, 1993.

Dunne, Anthony. Hertzian Tales: Electronic Products, Aesthetic Experience, and Critical Design. London: The Mit Press, 2008.

Dunne, Anthony, and Fiona Raby. Design noir: the secret life of electronic objects. Basel: Birkhäuser, 2001.

“Egalitarianism.” Wikipedia. http://en.wikipedia.org/wiki/Egalitarianism (accessed September 26, 2010).

Elden, Stuart, Elizabeth Lebas, and Eleonore Kofman. Henri Lefebvre: Key Writings (Continuum Collection). New Ed ed. New York: Continuum International Publishing Group, 2006.

Elger, Dietmar. Dadaism. Köln: Taschen, 2004.

“Gestalt psychology – Wikipedia, the free encyclopedia.” Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Gestalt_psychology (accessed April 28, 2011).

Harel, Idit, and Seymour Papert. Constructionism (Cognition and Computing). Norwood: Ablex Publishing, 1991.

Hensel, Michael, and Achim Menges. “Patterns in Performance-Orientated Design: An Approach towards Pattern Recognition, Generation and Instrumentalisation.” Architectural Design 79, no. 6 (2009): 88 – 96.

Hewett, Ivan, Carey Lovelace, Sharon Kenach, and Makhi Xenakis. Iannis Xenakis Composer, Architect, Visionary. New York: The Drawing Center, 2010.

Kay, Alan. “The Computer Revolution Hasn’t Happened Yet.” Technology: Blessing or Curse? Franz Rosenzweigh Lehrhaus Series. Jewish Theological Seminary, New York. Fall 1997.

Kluitenberg, Eric, Ari Altena, Kristina Andersen, Elizabeth Sikiardi, Daniel Van Der Velden, Frans Vogelaar, Koen Brams, Max Bruinsma, Marion Hamm, Klaas Kuitenbrouwer, Noortje Marres, Dirk Pultau, Howard Rheingold, and Saskia Sassen. Open 11: Hybrid Space. Pap/Cdr ed. Rotterdam: NAi Publishers, 2007.

Lambert, Steve. “Free as in Freedom and Free as in Free Beer: A Talk and Walking Tour with Jokes.” Lecture, Free from The New Museum, New York, November 6, 2010.

Leacock, Matt. “Cooperation and Engagement: What Can Board Games Teach Us?.” Lecture, Google TechTalks from Google, Mountain View, April 25, 2008.

Lust, Chandler McWilliams, and Casey Reas. Form+Code in Design, Art, and Architecture. 1st ed. New York: Princeton Architectural Press, 2010.

Marcel, Gabriel, Maddux, J. Stephen, and Robert E. Wood. Music And Philosophy (Marquette Studies in Philosophy). Milwaukee, WI: Marquette University Press, 2005.

Maur, Karin Von. The Sound of Painting: Music in Modern Art. New York: Prestel, 1999.

Meyer, Leonard B.. “Meaning in Music and Information Theory.” The Journal of Aesthetics and Art Criticism Vol. 15, no. No. 4 (1957): 412-424 . http://www.jstor.org/pss/427154 (accessed October 3, 2010).

“Modulor.” Wikipedia. http://en.wikipedia.org/wiki/Modulor (accessed September 26, 2010).

Munari, Bruno. Design as art. Harmondsworth, Middlesex, England: Penguin, 2008.

Norman, Donald A. The Design of Everyday Things. Cambridge, Mass.: MIT Press, 2002.

“Now Watch This: “Journalism in the Age of Data” | Motherboard.” Exploring the Culture of Technology — Post Something | Motherboard. http://motherboard.tv/2010/10/14/now-watch-this-journalism-in-the-age-of-data–2 (accessed October 16, 2010).

Papert, Seymour, and Idit Harel. “Situating Constructionism.” Professor Seymour Papert. http://www.papert.org/articles/SituatingConstructionism.html (accessed October 16, 2010).

Petzold, Charles. Code: The Hidden Language of Computer Hardware and Software. Redmond: Microsoft Press, 2000.

Piaget, Jean. The Psychology of Intelligence. London: Routledge & Paul, 1950.

“Postmodern music.” Wikipedia. http://en.wikipedia.org/wiki/Postmodern_music (accessed September 26, 2010).

Reimer, Bennett, and Jeffrey E. Wright. On the Nature of Musical Experience. New York: On The Nature Of Musical Experience, 1992.

“Rhythm – Wikipedia, the free encyclopedia.” Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Rhythm (accessed April 28, 2011).

“Rhythm – Definition and More from the Free Merriam-Webster Dictionary.” Dictionary and Thesaurus – Merriam-Webster Online. http://www.merriam-webster.com/dictionary/rhythm (accessed April 28, 2011).

“Rhythm | Definition.” Dictionary.com. http://dictionary.reference.com/browse/rhythm (accessed December 7, 2010).

Rushkoff, Douglas. “SXSW 2010: Program or be programmed.” Youtube. www.youtube.com/watch?v=imV3pPIUy1k (accessed April 12, 2011).

Sacks, Oliver. Musicophilia: Tales of Music and the Brain. 1st ed. New York: Knopf, 2007.

Shapiro, Peter. “Automating The Beat.” In Undercurrents: The Hidden Wiring of Modern Music. New York: Continuum International Publishing Group, 2003. 131-139.

Soegaard, Mads. “Gestalt principles of form perception by Mads Soegaard.” Interaction-Design.org. http://www.interaction-design.org/encyclopedia/gestalt_principles_of_form_perception.html (accessed May 1, 2011).

Sterne, Jonathan. “Urban Media and the Politics of Sound Space.” In Open 9: Sound. Pap/MP3 ed. Rotterdam: NAi Publishers, 2005. 6-14.

Taylor, Patrick. Interview by author. Personal interview. Parsons The New School for Design, October 7, 2010.

Wiltshire, Alex. “PIXELSURGEON | Interviews | Design | Toshio Iwai.” Pixelsurgeon is Dead.. http://www.pixelsurgeon.com/interviews/interview.php?id=239 (accessed May 4, 2011).

Wright, Matt. Interview by author. Email interview. Email, March 18, 2011.