Colour your sound using Snap!

A computer program that transforms sound into colour and then colour back into sound would be sure to blow a few minds.

What happens then: Fun! Learning! Unpredictable but interesting pathways!

I encountered this briefly through Jay Silver / Eric Rosenbuam (who I think of as the very clever hippy digital innovators) and then later through Jens Monig (Snap! developer). I'll write up the Jens version for now and add some references from the creative hippie pair to write up later. I can see this blossoming into a fun, innovative course.

If you prefer go straight to the Snap! program (Whistle Draw) and explanatory video by Jens

Jen is a master of Snap! and I learnt a lot as well as having much fun in emulating his program. I'll document it here since I'm planning to use it at school.

The Synesthetic pair here is sound and colour. At first sound produces colour and then the process is reversed so that colour produces sound

I'm including some preliminaries and extra explanation which Jen leaves out. With Snap! colours I find it necessary to check the defaults so that the colours work. I set the pen saturation and brightness to 100 and transparency to 0. Then if hue is set to 0 it shows red and changing hue through 0 to 100 goes through a ROYGBIV spectrum.

The colours are working as they should be so now we investigate and calibrate the microphone:

I whistled a tune into the microphone and observed the range of frequencies. Since they started at around 700 Hz I subtracted 700 to obtain a starting point of zero. Then I divided by 10 so as to roughly bring the values into a 0 to 100 range. This range coincides with the full range of colour values in Snap! Here is the calibrated microphone:

Next up, write a procedure that will produce variations in colour depending on the sound frequency:

I inexpertly whistled "Twinkle twinkle little star" and recorded the colours by moving the mouse. It looked like this:

You have to untick the sprite's draggable box. This held me up for a while so I'll offer an explanation.

In everyday usage when the pen is down we want to be able to drag the sprite without drawing. That could be inconvenient. When the draggable box is unticked the sprite disappears. You can show it by right click > show on the sprite but can't drag it. When you run the procedure forever [go to mouse pointer] the sprite does show and follows the mouse pointer (not the same as being dragged)

When the pen is down and you run forever [go to mouse pointer]:

• When draggable box is ticked drawing happens with mouse up but not mouse down.
• When draggable box is unticked drawing happens with both mouse up and mouse down (the behaviour we want for this program)

Then I wrote a procedure that could play it back. This is the inverse of the earlier procedure, this time converting colour to sound:

Then when I ran my mouse over the image (with mouse down) my inexpert tune played back!

Finally, Jens shows how to save the image so you can try new ones and return to the saved ones later. Click on the Stage and run this code:

You can see the image is saved under the Costumes tab!

Jens goes onto make a piano by whistling the correct notes. But even if you can't whistle well (like me) all you have to do is put blobs of ROYGBIV on the screen to make a piano that might work. Try it and see.

Here is the Snap! Colour your Sound program I ended up with.

When I read up on the earlier work by Jay and Eric (references below) I can envisage lots of new themes emerging from Jen's program. I'll try to get back to this later.

REFERENCE
LENS X BLOCK: World as Construction Kit Jay Silver PhD thesis
Eric Rosenbaum: Designing for Creative Play
Singing-fingers
NB their explanatory notes
Twinkle
colour code
(and there are more Jay Silver videos on the sidebar)

the wider walls

In commmemoration of the 50th anniversary of "Twenty Things to do with a Computer" by Seymour Papert and Cynthia Solomon

THE WIDER WALLS

“the room was humming harder
as the ceiling flew away”
- Whiter Shade of Pale by Procol Harem (1967)

Seymour was very good at finding a great pithy expression to describe a learning event. For example, he described the turtle as “an object to think with”. Another such expression is “wide walls”, to express a diversity of doing, but I’m less certain about who coined that one.

I wasn’t aware that the concept of wide walls (as distinct from the expression) was there from the start. I thought there was a transition as the baton passed from Seymour to Mitch Resnick. Not only did the turtle become a cat but, in my mind, the philosophy also changed from “low floor, high ceiling” (Seymour) to “low floor, wide walls” (Mitch).

Being a long way away, in Australia, perhaps I missed some nuance. Maybe someone closer to MIT can fill me in. In reading “Twenty Things ...” I can see the wide walls were there from the beginning: floor turtle, screen turtle, turtle graphics, game making, movies, music, robots, light displays, poetry, physics, curiosity about self, even making fun of Computer Aided Instruction.

Nevertheless, we owe a lot to the Scratch team for continuing to lower the floor (block coding), opening the windows (remix and a web site where it is so easy to share) as well as wider walls (easy to code multimedia and extensions to music, the micro:bit, Makey Makey and others). The user interface, with its colour coding of blocks into categories has always been brilliant.

What attracted me to computers in the first place was Seymour’s book “Mindstorms” with its intention of making maths more accessible to those who struggle with textbook maths or maths in general. Start by drawing a square using your body. Build further from that simple starting point by changing the angles, changing the number of repeats and introduce variables. This approach was both more interesting and could bring maths to more students.

Tinkering: Seymour was also there early on in a collaborative article with Sherry Turkle, “Epistemological Pluralism”. Ostensibly, this was directed at the needs of girls but in reality, it was about those who tinker or the bricoleurs. It’s better to initially allow sloppy code (spaghetti) and for design to iterate through stages. If a teacher insists on getting it perfect from the start then that is a sure way to kill motivation in many students. If only the standards based curriculum designers, remote from the classroom, who separate the what from the how, understood that. This was pointed out by Mitch Resnick in a recent exchange with Mark Guzdial in the comments at Mark’s blog. There are those who understand how the wider walls can work and those who, through not understanding, put up barriers to them working.

In my research, I came across a thesis by Jennifer Cross where she designed a course called Arts & Bots. Taking this cue, I wrote a submission for a new course in my current school and called it Artbotics. Rather than robotics, we can have Artbotics, with the Hummingbird Bit. Let us create a provocative, tangible sculpture and then add robotic actuation and sensing. This reframing of robotics makes a difference. The artbotics word hit a nerve since there is an ongoing desire for school administrations to introduce more creative middle schooling curricula. I see Artbotics as another expression of wider walls, integrating different subjects in the curriculum into a whole which students find more meaningful.

As Seymour pointed out, the computer has a protean nature and can be used as a multimedia hard fun machine. It is also true that the division of Knowledge into different subject domains, although useful in some ways, has always created artificial distinctions too. Maths can be Arty. Art can be Mathy.

Look up Wassily Kandinsky for more like this

In Central Australia, where I live, there is a famous indigenous art movement called Papunya Tula. One of its core motifs is dotted circles. I had a go at simulating parts of that work using SNAP and the local gallery in Alice Springs agreed it was worthwhile. I invite you to see the simulations at my blog, with a link to the SNAP program where you can do your own.

By making art a real part of the plan we soften the traditionally hard cultures of the STEM subjects and broaden the appeal of the whole mix. It has to be real and it can be real because digital by its nature does embrace everything. A program such as Scratch is multimedia with built in icons, backgrounds, sounds, music, speech with all of them editable. The coding is there too, of course, and the purpose of the coding is to bring the multimedia to life.

As well as art, storytelling needs to be in there too. I managed to rewrite all my introductory Scratch projects as short stories rather than lessons in technique, eg weird animals where the dog meows and the cat barks. This serves as a good model for later when I ask students to create their own stories.

One of the best expressions of the wider walls concept is a 2005 article by Mitch Resnick and Brian Silverman about how to evaluate construction kits. We need those design principles now with all the new construction kits (and this also applies to the new microcontrollers and new software) that have come on stream.

A key principle here is that a little bit (of programming) goes a long way. By using the KISS principle, new users will make a fast start and the possibility is there to transition rapidly to projects they really want to do, not just can do.

When it comes to learning principles the value of tinkering and iteration (iterate, iterate, iterate …) as well as curriculum integration also fit under the rubric of wider walls.

You can see the same KISS principle designed into Turtle Art (by Brian Silverman, Paula Bonta and Artemis Papert). Its variety of coding blocks is restricted yet you can make beautiful art readily. See some beautiful Turtle Art here

Here's a sample from my article How to create a great background in Turtle Art:

What I’ve really been looking for is a pithy expression to describe the ongoing evolution of the creative use of computers in education. In reading “Invent to Learn” (thanks to Gary and Sylvia) I was struck by the three game changers assertion, the game changers being coding, physical computing and fab labs. Could there be a pithy expression to describe these?

Possibly “wide walls”, although a good try, lacks a bit of oomph, requires too much explanation and can only be stretched so far. Can we set the world on fire with wider walls? Perhaps. As my students explore more I do feel the room humming harder ... but I am still not satisfied that it is an adequate descriptor.

Jay Silver takes this a step further when he situates his invention, Makey Makey, within a larger context. He designs tools which enable a sensual re-experiencing, a re-seeing, of the everyday world. With Makey Makey you can make an orchestra which is played by pieces of fruit, rather than a keyboard. Digital meets the banana. It began with Seymour’s insight that the turtle could be “an object to think with” and has now further developed, in Jay Silver’s words, into conceiving the whole “world as a construction kit”.

The world as construction kit has a very long history. Indigenous Australians used to live off the land and made all their tools, as well as their food and medicine, directly from nature. When they looked at the natural environment they saw a construction kit.

The part of this argument that I like is that our perception is a variable: we can design new tools, like Makey Makey, which changes the way we perceive the computer / keyboard. Insofar as modern consumerism can kill off self reliant productivity this new way of perceiving is a good thing.

Wouldn’t it be desirable for many more of us to become more maker orientated, particularly if the developments in modern technology lower the entry barriers? From STEM to STEAM then moves on to STEAM for the 99%.

STEAM for the 99% means bringing all of the subjects to a broader audience. This may be achieved through diverse cross curricular subjects which go under names like Artbotics, Digital Wearables, Culturally Situated Design Tools and Unruly Splats.

When the computer is used as a dynamic (programmable) multimedia fun machine it becomes the best tool available for wholesale curriculum integration. In other words it’s time to merge the computer into the world with all its junk. If that isn’t wider walls, then what is it?

REFERENCE
Cross, Jennifer. Creative Robotic Systems for Talent-Based Learning (2017)
Culturally Situated Design Tools
Makey Makey
Papert, Seymour. Mindstorms: Children, Computers and Powerful Ideas (1980)
Papert, Seymour and Solomon, Cynthia. Twenty Things to do with a Computer (1971)
Resnick, Mitchel and Silverman, Brian. Some Reflections on Designing Construction Kits for Kids (2005)
Scratch
Silver, Jay LENS x BLOCK: World as Construction Kit (2014)
SNAP
Stager, Gary and Martinez, Sylvia. Invent to Learn (2nd Edition, 2019)
“The goal of a first CS course should be to promote confidence … ”
(see comments 2, 3, 11, 13, 14, 15 and 16 for discussion between Mitch Resnick and Mark Guzdial)
Turkle, Sherry and Papert, Seymour. Epistemological Pluralism and the Revaluation of the Concrete (1991)
Unruly Splats

digital smashes the banana

Jay Silver and Eric Rosenbaum on the origins of Makey Makey

I downloaded Jay Silver’s PhD thesis, Lens x Block, a while back and took some notes. When I came to the bit about “world as construction kit”, I thought, at last, a modern rationale for the Maker Movement. This feeling has persisted, that Jay explains it deeply, whilst others who are great practitioners haven’t found the right words. They use more mundane rationales, such as “we are all makers”.

I won’t try to summarise all my notes to Jay’s PhD here but was reminded of them when I read the Forward to the book by Colleen and Aaron Graves about the Makey Makey.

It took me a while to buy my own Makey Makey because initially I thought it could be just a gimmick without much real depth. But once you realise that the concept is for the digital to smash the banana then that all changes.

I’ll just extract a few words, from the above mentioned Forward, from Jay Silver and Eric Rosenbaum, the inventors of Makey Makey below to illustrate.

We were super obsessed with the everyday world, skin to skin contact, and touching things in nature ...

Scratch has this powerful flexibility and open endedness. We also wanted that kind of flexibility for physical stuff and how to hook things up in a super flexible way ...

Computation is the greatest practical power but it is tied up in coding. And coding has barriers to access no matter how simple you make it ...

We thought a lot about making things tinkerable. One way to do that is to make them really immediate, so you could try something and see its effects right away…. So the idea of the keyboard hack and making a device that thinks it’s a keyboard partly came from that...

In Lifelong Kindergarten (LLK) we’re obsessed with creating the building blocks and fingerpaints for people of all ages...
The reason we were in LLK was because we want to live in a world created by everybody. That can only be possible if you create tools that help people create the world and those tools are very easy to use.

I think this video presents the best rationale for the modern maker movement that I’ve seen Hack a banana, make a keyboard (NY 2013)

"what kind of tools can we give adults who know too much ..."
“Let us smash computers with everyday objects…”
“the world as construction kit”

Reference:

Graves, Colleen and Aaron. 20 Makey Makey Projects for the Evil Genius (2017), Foreward.

Lens x Block: World as Construction Kit (Jay Silver’s PhD Thesis)