Exploring expressive complexity in music controller hardware interaction.

Thesis advisors: Ralph Ammer and Massimo Banzi

Thesis report download (screen) (730kb pdf)
Thesis report download (print) (2.7mb pdf)

Media

Video:
String Thing prototype testing (wmv, 32mb)

Sound:
stringthing_drone_01.mp3 (1.01mb)
stringthing_drone_02.mp3 (1.01mb)
stringthing_drone_03.mp3 (1.63mb)
stringthing_drone_04.mp3 (1.03mb)

Image
High resolution press photograph (9.80mb)

Idea/problem/context

The sight of a musician playing a traditional instrument, perhaps particularly a stringed one, is a satisfying and intelligible visual counterpart of the sound. Computer-generated and computer-enhanced sounds have enriched musical language and expression. But computer-based music performances still mostly involve people sitting rigidly behind a laptop, their fingers hidden by the screen.

What it is

String Thing is a cello-like electronic instrument played by stroking or beating metal rods with the hands. The use of bodily gestures, infinitely variable and visible to the audience, avoids the ‘robotic’ and visually uncommunicative quality typical of computer music.

How it works

Four roughly parallel metal rods are each divided into a long and a short section by a bridge element. The pressure and position of the fingers on the longer rod section are continuously sensed to control pitch and expression; pressure on the shorter sections controls velocity, attack and volume. MIDI software converts this information into sound and, through magnets under each rod, vibrates the rod according to the pressure it senses, thus returning haptic feedback to the player.

Like a fretless instrument, String Thing produces continuous pitch: notes bend seamlessly into each other. Light from a laser pointer above each rod, reflected as a dot on the player’s finger, is detected by a small webcam in the bridge; this movement controls the pitch.

Value/potential

The instrument and its interface is cheap to produce. Though designed for musicians of varying ability, it offers those who have mastered the subtleties of traditional stringed instruments not only a new range of sounds but a familiar richness of aural and haptic feedback. Audiences, similarly, enjoy the concord of sight and sound associated with traditional music-making.

See a video clip of the prototype testing (wmv, 32mb).

Process Overview

1. Introduction, general thoughts etc...
2. Getting started
3. My first Controller
4. Second Prototype
5. Third Prototype
6. Prototype Ingredients

Introduction, general thoughts etc...

This subject area is problematic for a variety of reasons - the most obvious being that it is and has been very well explored for many years, by many talented people. Bringing something interesting to this area of research and design is a big challenge.

For all the seemingly revolutionary and crazy controller devices and gesture recognition systems constantly being created around the world, the main tools most commonly seen to be used by musicians incorporating computers into performance still seem to consist of the same old midi controllers and keyboards that we have had for the last 20 years or so.

As a 'traditional' musician, i.e. playing cellos, guitars, flutes, drums etc, I want to explore the combination of elements involving the physical, visual, aesthetic, feedback and control qualities that make an exciting acoustic experience, and how these elements can be successfully integrated into an interactive/computer system to enhance some aspects of computer music performance.

I believe that there is still a healthy appetite in this area of computer interaction, for more innovation combined with a serious understanding and passion for playing and performing with musical instruments.

Getting started

My ideas so far have all been based on 'string-based' interaction, with the intention of exploring the immediacy and directness of manipulating a vibrating string to create a musical sound, and how this powerful tangibility can be used in new computer music interfaces.

I am interested in interfaces that offer a degree of complexity (you could say 'difficulty of use'), offering a rewarding experience closer to that of playing an acoustic instrument, in comparison to many simplified MIDI controllers, requiring an investment of time and effort on the part of the musician to explore and master, and to find unique expressions and sounds.

One notable product of great inspiration to this project is the 'haken continuum', a high resolution MIDI controller that requires a totally new method of interaction to play, somewhere between a keyboard and a fretless stringed instrument, the continuum suggests a potential new breed of controller interface with a degree of sensitivity that can equal that of traditional, acoustic instruments.

My initial idea is a kind of alternative controller (as in alternative to the MIDI keyboard), intended to offer continuous note and expression control like that of a stringed instrument. The design uses steel rods arranged in a cello-like structure, each rod separated into two specific parts and suspended in a floating, spring loaded fashion that can allow a variable degree of haptic feedback, in the form of vibration. Of these two parts, one longer rod acts as a pitch/expression controller, sensing the user's finger pressure and position along its length, the smaller part being responsible for sensing pressure to generate velocity/attack/volume values.

I anticipate these steel rods to have an appealing tangibility and musical resonance, without the complications and actual sound output involved in using existing dedicated stringed instruments strings.

Of these two parts, one longer rod acts as a pitch/expression controller, sensing the user's finger pressure and position along its length, the smaller part being responsible for sensing pressure to generate velocity/attack/volume values.

Another idea to be explored in this design, is to have the supporting cross section structure contain an acoustic chamber, intended to receive the outgoing audio signal generated by the computer and generate more haptic feedback in the form of acoustic resonance channeled throughout the instrument, to provide a really real feeling sense of acoustic activity and feedback, allowing the musician to feel the sound as with traditional acoustic instruments.

The intention with the above idea is that the interface is stripped down to basic elements, with just enough suggestion of familiarity to still enable it to be considered a new instrument and yet facilitate some pre-learnt understandings and techniques.

My first Controller (attempt)

Given the emphasis is on the physical interface side of control and interaction, it was important to start learning and experimenting with the relevant electronic and software aspects as soon as possible, these aspects most likely to be interaction between microcontroller driven hardware and software interpretation.

I am currently working with Max/MSP, as it seems a powerful and popular platform for working in this area, though I am always looking at other alternatives. One software I have been a fan of for a long time now is audiomulch, which is a very nice to use tool, one for which I am constantly wishing for a suitably powerful and expressive hardware interface.

The wiring microcontroller platform is used for all electronics prototyping, for all hardware sensor use.

The first technical aspect I wanted to work on was how to set up a steel rod to sense a person's finger position along its length, in order to generate a continuous signal that the wiring board can feed into Max/MSP, which can then be turned into useful musical information.

Experiments with different conductive (available!) materials, copper paper, silver foil and different types of magnetic tape. Eventually I used guitar strings and miniDV tape.
The first attempt, it worked well but the copper paper lost its effect/variable conductivity as it was used, the more it was touched.

My initial controller building exercise was basically a pressure sensitive continuous controller (like a ribbon controller or a kind of touch-theremin), eventually with two independent pitch controls, the idea being that you have two pitch controls, controlled by running your finger along the length of the device (the 'strings'), while the whole thing is sensitive to pressure exerted by the users fingers, controlling volume, attack or velocity of a note etc.

Working! - using guitar strings suspended over magnetic/conductive tape (covered with clear plastic to protect the tape) - it is supported on blocks of conductive foam to provide a variable resistance as pressure is applied.

See a short video clip of me using the modified trouser hanger to try some music controller application ideas (wmv, 13mb).

While this exercise has not resulted in the most amazing of music interface controller designs, it has been a useful learning exercise and now I feel more prepared to push the exploration further and start to realise some of the more interesting design ideas mentioned previously.

Second Prototype

For the next prototype I explored the use of a linear touch sensor, combined with a sense of physical feedback. I decided to experiment with ways of making a suspended metal rod vibrate, like a bowed string.

I used a solenoid as a magnet to switch on and off at a rate determined by finger pressure. This attracts and releases the metal rod, creating the impression of vibration. The idea here is that the interface can give a synthetic sense of acoustic response to the sound being manipulated.

See a short video clip of the solenoid being used as a 'string vibration generator'. (wmv, 4.7mb).

Third Prototype

Having found a satisfactory method of generating physical feedback (directly related to a sound), I proceeded to find a successful way to track finger position along a floating/suspended metal rod - something suprisingly difficult to do!

Eventually I found that a combination of webcam and video tracking (using eyesweb software), with laser pointers, provided a reasonable way of capturing finger movement for use as continous pitch manipulation - in fact this technique results in a very 'analogue' feel. Using this method with independent touch sensing for each rod provides a convincing sense of physical interaction with the metal rod(s).

At this point in the project, I had been working on a single rod/string setup, the plan being that this could be a flexible modular setup to allow the rapid building of a multi-rod prototype. So the next step was to design and prototype the general form required to support four rods, as well as some controls required for some of the settings developed in the software side of the project (see thesis report for more detail).

Many thanks to Edoardo Brambilla for all the prototyping work!

Prototype Ingredients

The latest prototype consists of a board hosting the components for four individual rod sensors and switches for activating solenoids/vibration feedback. This board is then connected to the wiring board which handles all input and output from the computer, as well as running a program to vary the physical feedback depending on input (vibration frequency responding to finger pressure sensed).

An eyesweb patch running on a second, networked computer sends finger position data through an open sound control connection.

Max MSP then receives this collected serial input and video processing data to control MIDI note output.

Code examples used
(if you want to have a look! - its a bit messy...)

Wiring code used in the latest prototype (includes code to control a multiplexer switch, used to increase the number of analogue inputs for the wiring board).

Eyesweb patch

Max MSP patch

String thing guts laid out! (all boards mounted to a plexiglass tray to slide into the tube-body)

The wiring board and sensor board connected

Sensor board

Assembling the individual 'mode' switches and LEDs - connected to the digital inputs on the wiring board

Assembing all connected parts within the instrument body (laser diodes powered from the wiring board)