Sound and Action Analysis Summary

1. Introduction

In observing interaction with everyday objects, we realize that the complexity of real experiences brings forward always new relations between action and sound, that can not always be addressed with classical design approaches such as linking data to sound. In this study, we focus on understanding the elements which compose an action, and the ways in which the sound feedback shapes the performance in different stages of the interactive experience. Through videotaped ethnographic field study and an adapted version of hierarchical task analysis we investigated everyday interactions in a kitchen scenario. The following report describes the aim, the main questions we tried to investigate, our method of analysis and the findings. Furthermore the analysis is illustrated with an example.

2. Aims

The aims of this investigation were the following:

- to concretize the relationship between sound and action, using objects and processes in the kitchen.
A central aim of the action analysis was to investigate general concepts about action and sound in a concrete, everyday scenario of use, applying theories related to action and sound analysis & description. As setting we chose the kitchen, because it is a location in which variety of actions are performed, with a variety of tools, which can be very distinct, be it in form, material or function. The kitchen tools we chose range from simple manual tools without any mechanics, such as knifes or spoons; to simple mechanical tools, such as garlic squeezers, and finally to the vast array of electromechanical kitchen appliances, such as toasters, coffee grinders and blenders. Moreover there are electronic tools such as the stove, or the freezer, which produce heat or cold, and evoke a series of processes involving movement of physical matter, thus generating sound.

- to explore and experience directly the importance of everyday sound for action
As designers it was our goal to "learn by experiencing and observing" in a structured and methodological way. Therefore we chose to use as material of investigations our own ordinary cooking activities. By analyzing the video recordings afterwards we had a means to reflect upon our own experiences and phenomena we encountered. This was an important inspiration for future designs and scenarios described below.

- to understand the relation between function, task, action and sound
A goal was to investigate, how the device's function, the intended task and the resulting action correlated. Also it was important to find out, what kind of action primitives could be identified in a specific scenario, in this case the kitchen. But most importantly, we wanted to find out, how sounds relate to the functions, tasks and actions, in order to discover possible approaches for design.

- to get insights into the importance of the context (in this case domestic), and different social situations (cooking alone, with friends, family).

- to produce a critical mass of documentary material on sonic actions, which can be used for further research and reference.

3. Work process and method

In what follows, we describe the method we used for the experimental investigation of sound and action in the kitchen setting. We began with recording a number common cooking activities. Our selection criteria was the high amount of manual interaction with the tools. Heating up a stew in the oven, for example, was of rather minor interest for our research as it did not involve performative engagement from the side of the user. A total amount of 48 video clips, ranging in length from about 20 seconds to a few minutes, were created and documented on a WiKi. Once the video shooting, editing, and organizing was completed we began with a process of analysis.

The analysis process consisted of three main parts: Action Analysis, Sound Description and Relevance of Sound for Action. In order to find the appropriate way to describe an action, we did a background research on action and task analysis. Our Action Analysis consists of a process of decomposition of the task into smaller task and finally action primitives, and a detailed description of the ways the action is performed. Action primitives are those actions which repeatedly appeared in a number of studied examples, and to which no specific meaning can be assigned when isolated from each other or a task context. Sound Descriptions are based on common categories from psychoacoustics, the exercises conducted in the CLOSED workshop at HGKZ, everyday sound categorization by Gaver (Gaver, 1993), and the categories developed by our project partners. Judgment of relevance of sound for action was based on a specific phenomenological observation. We developed hypotheses on what sound meant for our perfomance in a specific situation. Relevance of Sound for Action was an important step towards showing the existing value of sound in relation to action, as well as revealing new design opportunities for example where sound produced in interaction is of low volume.

Action Analysis Description

- Force : What force is (subjectively) required to execute the action? The human body serves as a reference for deciding, weather a force was relatively small or big. It also includes the sense of speed: was it rather a short energetic movement or a long powerful one?

- Duration: Also time plays a role: the force can be used very shortly, repeatedly or steadily

- Type of manipulation : how is the object handled? (eg. holding the object with one or two hands, holding the object with lips and hand)

- Form and configuration of object: Is the object heavy? Does it have a handle? Is it big or small? What are it's surface properties (slippery, hot, rough ..)

As for describing aspects of force and energy, we also investigated the use of musical notation, such as sings for piano (p) or forte (f), diminuendo (>) or crescendo (<). This seems to make a lot of sense, because actually what the musical notation describes is a kind of physical and mental energy put into a note or a melody. This was not elaborated further, but could be interesting to investigate in the context of notation systems for action-sound relationships. Also for the question of sound - action relevance approaches for notations were discussed, eg. relating envelopes for pitch, loudness, amount of noise, rhythmic patterns etc. to manipulation / process "envelopes", eg. based on parameters like force.

Sound Description

- Basic descriptive level (pitch, perceived loudness (pp, f, ff, etc.), timbre, roughness, brightness

- Aspects of time: duration (very short, short, a few seconds, longer), development over time, envelope (attack, decay, sustain, release) speed, ...

- Rhythm: of vibration, rhythm of repetition of sound events, eg. bouncing)

- Sound source: aerodynamics sounds, liquid sounds, vibrating solids, fire sounds, specific movements of the source causing a sound (eg. vibration, bouncing...)

- Material (of sound emitter / of stimulator (if applicable)): elasticity, density...

- Configuration : shape, size, structure (ex. resonatig cavities), support, weight ...

- Aspects of space : delay (reverb, echo), dampening, perspective, distance, resonance, echo ...

- how does it sound like? What does it remind of? Onomatopoetics (squeek, whoosh, clack)

It's important to be aware of the fact, that not only the sounds are important to notice, but also the absence of sound, or silence.

Relevance of Sound for Action

- direct sound (eg. when grating a carrot, the sound is directly linked to the movement of the hand) or indirectly sound(eg. when the action triggers another movement, that produces sound, like closing a cupboard). A sound was considered relevant to the action, if it was associated with executing the action, either by being tighly linked to it or as a sonic sign which ones communicated would affect a certain action.

- continuously vs descreet sound (as, eg. in putting a glass on a table). The occurence of the sound influences the way the action is excerted. This could be on the level of feedback, either continuous or discrete.

- manipulative action and triggering action (which starts a self running process). In both cases the sound can be related either to the action itself, or to the process, or to both.

- What would happen if we remove sound feedback?

4. Examples

Following are excerpts of three action analysis transcriptions. In the transcriptions we used the following abbreviations:

Abbreviations used to describe actions:
AD: duration of action in seconds (approximate value)
AE: Energy excreted during manipulation

Abbreviations used to describe sounds:
MS - manipulative sound (sound as a result of direct manipulation and processes directly linked to it. can be produced by the interface itself or the medium being influenced through the interface)
AS - automatic (machine) sound - (sounds from events or processes not linked to a direct manipulation)
NS: No sound

Abbreviations used to describe relevance to action:
NSf - not very relevant or strong sound feedback for action (like wobbling of the coffee maker top while moving it from one place to another or)
RSf - sound feedback relevant for action

Example 1: Preparing Coffee
Chosen excerpt: Filling the coffe maker with water

{image: preparing_coffee_1.tiff}

Process Transcript

1. unscrewing the coffee maker
1.1 taking the coffee maker with two hands
1.1.1 wobbling of the coffee maker top
1.2. exerting a lot of pressure until the top part starts turning
1.2.1. pushing 2 parts in the opposite directions (AD: few sec. AE: high)
- NS
1.2.2. release of the screwed parts (AD: less then a sec. AE: dropping)
- MS : short sound at the end
- Relevance for action: the sound communicates the release of the energy that is felt though action

1.3. turning the top part three times with one hand (grasp, turn, release, grasp in a different place, turn, release ...), while keeping the bottom with the other hand
- MS : unscrewing sound. longer, repetitive, friction between 2 metal parts
- Relevance for action: the looseness of the two parts can be heard (At the beginning I keep the bottom part strongly and the top part I turn, release and grasp in a different position, then repeat. As this process comes to an end I can hear when I need to keep the two parts strongly because they are separating). The continuity and speed of action are audible.
1.4. separating the two parts (AD: short. AE: medium)
- MS : opening sound, continuous sound interrupted by silence
- Relevance for action: the action is finished, the task accomplished

Example 2: Peeling Carrots
Chosen task excerpt: Scrape the surface of the carrot

{image: peeling_carrots.tiff}

Process Transcript

1. push onto the bottom of the carrot with the peeler
2. pull/slide toward the top end of the carrot until the peeler touches the thumb.
- MS: sound of cutting into the carrot and scraping its surface
- relevance for action: provides feedback about the speed of action, depth of the cut, eventual irregularities of the carrot surfaces
- MS: sound of wobbling metal
- relevance for action: provides feedback of how controlled the movement and the contact between the carrot and the peeler is.

Example 3: Grating Carrots
Chosen task excerpt: Grate

{image: grating1a.tiff}

Process Transcript

1. push the carrot onto the surface of the grater
2. slide the carrot down while pushing strongly onto the surface of the grater (AD: few sec. AE: high)
- MS : the sound of grating? the sound is louder when pushed in this direction (top-bottom).
- relevance for action: The sound accentuates the haptic feedback of the carrot encountering the bumps of the grater and passing through them while being cut.
3. slide the carrot in the opposite direction to come back (AD: few sec. AE: low)
- MS: smoother sliding sound. Little faster from the previous sound.
4. repeat 1-4
- MS: the sound changes and becomes "thinner"
- relevance for action: as we repeat the action and the carrot piece becomes smaller, we can perceive its size though sound

Identified Action Primitives (is this required/useful? I think it's not really worked out yet enough to be presented as a result. Maybe just give a few examples?)

Based on the action analysis we could identify the following action primitives:

Basic action primitives
Directional movement /pressure : Push, Hit, Slide
Embracing pressure: Squeeze, Grasp
Displace (while holding) : Elevate, Put down, Remove,
Rotation: Tilt, Turn, Spin

Composed action primitives
Pull : squeeze and move in a certain direction (Stretch)
Pick-up (embrace and squeeze keeping constant pressure, so it doesn't fall out + displace)
Take out of the container (bag, box etc)
Unravel the knot
Pour: aim and tilt
Close : (place the cover back, screw the top etc)
Shake : squeeze and move back and forth
Aim / shoot / insert / pick / drop into
Swing (swinging a rope)
Catch (something falling)
Pick up


Conclusion, Relevance to Closed (is this required for 3.1? Better if you provide this, as you have better insight. Some points:)
- Beginning of design process - search for design opportunities
- First step to building a design methodology (because we didn't really refine the method, and also not apply it consequently because of the complexity)
- Understanding design for action (that's of course too general, but maybe you have a concrete suggestion?)
- Action primitives - how do they map to sound categorization by eg. Gaver, IRCAM, Verona? (this is a potentially important question, which need to be addressed in further work)
- Knowledge on abstracting sounds from their "natural" conditions (eg. material & mechanics) to more symbolic or metaphoric levels (this in my opinion is not a result from this work, but definitely an issue for questions related to sounddesign for interactive objects...)