A peak inside my PD workspace

Two things to show today: A sampler which switches between beat-matched samples (of the same length) and the comparison engine for the wiimote movement.

The sample switcher

This is very prototype-y at the moment. Essentially, the sample switcher toggles in between samples based on messages sent from the top (0, 1, 2). This is done so later a router can be used to sort which sample is playing at what time. This can be done live, and is quite fun to toggle in-between samples as a pretend DJ.

The comparison engine

The comparison engine is at the heart of the sample switching, and core to the design. Basically, it determines and upper and lower threshold for sample switches. If lower than a certain threshold, it moves back to a lower intensity sample. If higher, it switches to a higher intensity sample. It may also stay the same if there isn’t enough movement to qualify one extreme or another.

Data is captured using an adder throughout the dance. When the end of a sample is reached, the comparison engine is asked where in the threshold the dancer’s movements are. This will then trigger the next sample set, be it higher or lower intensity. Once this happens, the comparison engine is turned back on to “quiet mode” and continues to observe the dancer’s movements for the next sample switch. In this way, the song could go on infinitely.

The Prototype Design

I have not updated in some time because I’ve been very busy redesigning the system.

How the design has changed

The original design relied on entirely user-generated sound from stomps, kicks and arm movements. It proved too technically difficult to prototype such a system, and so a more interpretive design was explored. The results of the new design are a result of many hours of prototyping and testing.

Using completely generative sound with the WiiMotes is somewhat difficult because the dancer concentrates on the sound and is not free to dance. This indicates that some form of guidance or underlying beat is needed; the dancer adding or morphing the music. This is the basis for the new design.

I have switched entirely to PureData (“PD”) as a prototyping medium. I am using MusicController to hook into puredata using OSC messages. This is a replacement to using strictly C# code, which takes too long to make change.

The new design

The new design is currently being developed and will be demoed in the upcoming weeks. It features four WiiMotes connected to PD. The WiiMote controls alter or add to the music in some way.

WiiStyler Prototype Design Sketch

The dancers amount of movement also alters the music without direct input from the user. As the user begins to dance harder (with more movement), the song samples shift to a higher peak in the song. If the dancer slows down or stops (less movement) the song samples down-shift. This provides a two-fold control of music: both direct and indirect.

More to come soon.

Intuitive manipulation design is probably too much

The original design for the WiiStyler called for an underlying beat detection which would drive the rest of the song/music. In this way, the dancer would not have worry about direct manipulation of music and let their dancing drive the music.

After much work on an initial prototype using a hacked version of DarwiinRemote, this task may be too great. The beat detection alone is quite buggy; this has to do with the variance in data that a stomp can create. False positives (and negatives) abound as the software tries to interpret movements from the accelerometer gracefully.

Foot stomp detection prototype from Andrew McKinney on Vimeo.

Having a functional prototype by the end of one semester using this method seems too great. Informaticians have suggested learning/natural algorithms to train the software to detect beats, but this alone could take a semester to properly implement. In addition, that would require the dancer to train the machine for specific movements, and is ultimately inconvenient for a new user.

Stomp identification challenges and opportunities

I have come quite a way with my research since the first post with preliminary WiiMote data. Following this, I began to attempt to identify, through raw WiiMote data, how a stomp vs. a shuffle vs. a kick might work. The results are mixed:

Flat stomp

This is a baseline stomp for reference. Please note there is no “wide-up” here; simply a wiimote connected to a shoe slammed on the ground. The idea here is to capture what the impact with the floor looks like.

Shuffle stomp

This is a simple shuffle-stomp wearing the WiiMote in the same way. You can see a similar “pop” here as in the first graph – the Y and Z-axises bouncing off of the charts suddenly.

This last graph is the running man performed in a full cycle. We can see that the data becomes rather skewed here because of the high degree of movement. Despite all of this, we can still identify those “pops” seen in earlier graphs.

What does this all mean? Basically, false positives will be a challenge here. Since stomping and stepping is so fundamental to the dance, incorrect beats may be identified prematurely. Ultimately, the prototype (under development now) needs to have a varied degree amplitude.

As an additional challenge, my Xcode development is hampered by the DarwiinRemote project using the 10.4 OSX SDK. This causes undue warnings in my 10.6 build, so I’m currently exploring (and looking for) the 10.4 SDK to leverage. Wish me luck!

Understanding accelerometer peaks in Melbourne Shuffle moves

Accelerometer peaks using DarwiinRemote

To kick off the the project I began by attaching a WiiMote to my body and perform some basic Melbourne Shuffle moves – the running man into some shuffle/kicks and back to running man.

I found the best results were achieved by attaching the wiimote to my feet, latched down by my shoe laces. Attaching the wiimote to other parts of my legs (calves, ankles, etc.) causes too much jostling.

The results were promising. The running man “stomps” of the foot produce easily recognizable peaks. The different accelerometer axii converge and compress. The stomping creates a sudden, distinct waveform on the graph.

On the other hand, the kicks, which do not have a sudden stopping of motion, produce a more delicate, elongated waves.

We can see here an opportunity for both beats (stomps). This can be accomplished with software-based peak identification. These must be fast and effective, however, as misreading of the beat (or slow identification) could lead to strange audio output. The dancer needs to trust the output.

Kicks on the otherhand may need to create more interpretive output. Variable record scratches might be an option, as here we are associating the feet with the beat/rhythm and the arms with melody or addative sound. Further research will be required, but this is just the start.