In Designing for Learning by Creating taught by Karen Brennan at Harvard, our final assignment was to (as you might suspect) design something for learning...by creating! The entire semester I had been experimenting with different interaction and AR concepts, and was really fascinated by the idea of modularity. I was thinking a lot about color, music, legos, building, and making things compositional and modular. One of my first projects in the semester was making an AR game with Scratch called Air Control, which you can play with here. It's a "catch the falling thing" kind of game that you controlled with your hand and a webcam.
It made me think a lot about how spatial reasoning and how UX will change as AR/VR overtake the market in a few years. I also think that allowing people to break down their ideas into distinct components, then mixing and matching these components with each other is a key to novel UX. I took a bit more inspiration from a project I made the previous summer, an app called Notette that converted the color palette of your environment into a scale of your choosing, allowing you to "play the world" like an instrument.
It wasn't long before I realized I needed to make some sort of AR instrument that naturally felt spatial and explorative. Although Notette was cool (and barely worked), it still felt 2D and trapped inside of the screen. Once you allow people to walk through, touch, and move about something invisible-- like sound and music-- they'll be more involved, engaged, and able to understand abstract concepts more quickly (and is part of what my lab researches!). Giving users high degrees of freedom-- being able to place notes in 3D space, for example-- takes away the rigidity of a fretboard on a guitar or the linearity of a keyboard. This freedom could potentially place users closer into “conversation with the material” instead of getting bogged down by barriers placed by traditional instruments.
For the longest time I've been fascinated by the theremin, an instrument which you play by moving your hand closer and farther away from an atenna. It was invented in 1928. Clearly Leon was far ahead of his time.
In 2011, Moog Music, a company that makes amazing electronic instruments, posted an April Fool's joke that actually summarizes everything I really want out of the theremin: the ability to play chords and truly move with your music.
While it's physically impossible to have multiple antennas in one theremin, this is totally possible virtually! After seeing this comic masterpiece, I was super fascinated by the idea of music you could literally walk around.