Colour your sound using Snap!

A computer program that transforms sound into colour and then colour back into sound would be sure to blow a few minds.

What happens then: Fun! Learning! Unpredictable but interesting pathways!

I encountered this briefly through Jay Silver / Eric Rosenbuam (who I think of as the very clever hippy digital innovators) and then later through Jens Monig (Snap! developer). I'll write up the Jens version for now and add some references from the creative hippie pair to write up later. I can see this blossoming into a fun, innovative course.

If you prefer go straight to the Snap! program (Whistle Draw) and explanatory video by Jens

Jen is a master of Snap! and I learnt a lot as well as having much fun in emulating his program. I'll document it here since I'm planning to use it at school.

The Synesthetic pair here is sound and colour. At first sound produces colour and then the process is reversed so that colour produces sound

I'm including some preliminaries and extra explanation which Jen leaves out. With Snap! colours I find it necessary to check the defaults so that the colours work. I set the pen saturation and brightness to 100 and transparency to 0. Then if hue is set to 0 it shows red and changing hue through 0 to 100 goes through a ROYGBIV spectrum.

The colours are working as they should be so now we investigate and calibrate the microphone:

I whistled a tune into the microphone and observed the range of frequencies. Since they started at around 700 Hz I subtracted 700 to obtain a starting point of zero. Then I divided by 10 so as to roughly bring the values into a 0 to 100 range. This range coincides with the full range of colour values in Snap! Here is the calibrated microphone:

Next up, write a procedure that will produce variations in colour depending on the sound frequency:

I inexpertly whistled "Twinkle twinkle little star" and recorded the colours by moving the mouse. It looked like this:

You have to untick the sprite's draggable box. This held me up for a while so I'll offer an explanation.

In everyday usage when the pen is down we want to be able to drag the sprite without drawing. That could be inconvenient. When the draggable box is unticked the sprite disappears. You can show it by right click > show on the sprite but can't drag it. When you run the procedure forever [go to mouse pointer] the sprite does show and follows the mouse pointer (not the same as being dragged)

When the pen is down and you run forever [go to mouse pointer]:

• When draggable box is ticked drawing happens with mouse up but not mouse down.
• When draggable box is unticked drawing happens with both mouse up and mouse down (the behaviour we want for this program)

Then I wrote a procedure that could play it back. This is the inverse of the earlier procedure, this time converting colour to sound:

Then when I ran my mouse over the image (with mouse down) my inexpert tune played back!

Finally, Jens shows how to save the image so you can try new ones and return to the saved ones later. Click on the Stage and run this code:

You can see the image is saved under the Costumes tab!

Jens goes onto make a piano by whistling the correct notes. But even if you can't whistle well (like me) all you have to do is put blobs of ROYGBIV on the screen to make a piano that might work. Try it and see.

Here is the Snap! Colour your Sound program I ended up with.

When I read up on the earlier work by Jay and Eric (references below) I can envisage lots of new themes emerging from Jen's program. I'll try to get back to this later.

REFERENCE
LENS X BLOCK: World as Construction Kit Jay Silver PhD thesis
Eric Rosenbaum: Designing for Creative Play
Singing-fingers
NB their explanatory notes
Twinkle
colour code
(and there are more Jay Silver videos on the sidebar)