An AR instrument inspired by the theremin


Project brief

This past fall, I took an amazing course called Designing for Learning by Creating with Karen Brennan at Harvard. For my final project, I created an AR musical instrument inspired by the theremin that lets you "walk around" music.

Sole Designer and Engineer
October 2018 - December 2018 (2 months)



In Designing for Learning by Creating taught by Karen Brennan at Harvard, our final assignment was to (as you might suspect) design something for learning...by creating! The entire semester I had been experimenting with different interaction and AR concepts, and was really fascinated by the idea of modularity. I was thinking a lot about color, music, legos, building, and making things compositional and modular. One of my first projects in the semester was making an AR game with Scratch called Air Control, which you can play with here. It's a "catch the falling thing" kind of game that you controlled with your hand and a webcam.

It made me think a lot about how spatial reasoning and how UX will change as AR/VR overtake the market in a few years. I also think that allowing people to break down their ideas into distinct components, then mixing and matching these components with each other is a key to novel UX. I took a bit more inspiration from a project I made the previous summer, an app called Notette that converted the color palette of your environment into a scale of your choosing, allowing you to "play the world" like an instrument.

It wasn't long before I realized I needed to make some sort of AR instrument that naturally felt spatial and explorative. Although Notette was cool (and barely worked), it still felt 2D and trapped inside of the screen. Once you allow people to walk through, touch, and move about something invisible-- like sound and music-- they'll be more involved, engaged, and able to understand abstract concepts more quickly (and is part of what my lab researches!). Giving users high degrees of freedom-- being able to place notes in 3D space, for example-- takes away the rigidity of a fretboard on a guitar or the linearity of a keyboard. This freedom could potentially place users closer into “conversation with the material” instead of getting bogged down by barriers placed by traditional instruments.

The theremin

For the longest time I've been fascinated by the theremin, an instrument which you play by moving your hand closer and farther away from an atenna. It was invented in 1928. Clearly Leon was far ahead of his time.


In 2011, Moog Music, a company that makes amazing electronic instruments, posted an April Fool's joke that actually summarizes everything I really want out of the theremin: the ability to play chords and truly move with your music.


While it's physically impossible to have multiple antennas in one theremin, this is totally possible virtually! After seeing this comic masterpiece, I was super fascinated by the idea of music you could literally walk around.

One of my first low-fidelity designs of the basic idea.


Early designs and mathematical visualizations

One of the most exciting parts about this project was the underlying math and music theory. I really loved the idea of relative music systems and took a lot of inspiration from geometrical visualizations of harmony like Tonnetz, which are lattice diagrams that show harmonic relationships between notes.

Every step to the right or left is a 5th. Diagonal-right is a major third, and diagonal-left is a minor third. Chords are formed by the notes at the vertices of each triangle.

I was fascinated by relative harmony and thought about how you could walk around groups of notes (I'm using "note" very liberally here) and have their pitches change purely because of the position of the user.

For example, when a user is parallel to all three notes in space, they may have the same pitch. But the moment the user walks over to the side and is perfectly inline with all the notes, the distance between each note and the user is different, changing the pitch of each.

This all gets freakier when you start thinking about having multiple users walkingn around the same notes in space. What does music composition look like when harmony is created by people coordinating their positions in space rather than the keys they're playing?

Placing notes in space

After I understood a bit more about how I wanted the math to operate, I prototyped a small AR app to get spheres placed in space and have them connect to a central point.

Once that was done, it was simply a matter of getting these spheres' central reference point to follow the camera.

Getting sound to play

Once the note placement was figured out, all I had to do was produce sound based on distance. To keep things simple I divided the distance between the camera and each sphere (in meters) by 50 to obtain a frequency.

DISCLAIMER FOR HEADPHONES USERS: watch out for your ears


From here, most of the work of Polymin 0.1 was done: a "note" you could walk around in space and have its pitch change based on distance.

I tested the prototype and after getting casual feedback from musicians and kids, I took away two primary pain points:

  1. Kids wanted an easy way to stay "in tune" and not have to worry about "sounding good"
  2. Musicians wanted an easy way to turn notes on/off to experiment with different note combinations

In the end, I addressed these problems by implementing two major features: Polymin was select/deselect whichever notes you want dynamically and could snap all of the notes to scales (major, minor, pentatonic blues, chromatic, and spanish gypsy).

This led Polymin to feel like an inclusive and fun musical experience for whoever was using it.

Final submission

Our final presentation ceremony was dubbed T550 madness because all 100-or-so people in class had exactly a minute to present in a rolling powerpoint. Below is the one minute clip I edited for madness.


Q: What's the point of all this?

A: My final memo

For part of our final submission, we wrote a final memo describing the creative process behind our final projects. Hopefully this can give some insight into why I made this and why I find musical tinkering so fascinating.


To a curious reader,

You and I probably started making art with blocks, paint, wires, pipe cleaners, and paper-- the kinds of materials that you can mix together in odd combinations without worrying about things making sense. But when I picked up the guitar as an eager and naive 10-year-old, suddenly music teachers were dictating that being creative with music involved learning all these silly “rules” and “scales.” What bullshit!

I learned pretty quickly that, lo and behold, scales were important. Understanding the constraints of an instrument was important. But even as my playing matured, I struggled to visualize the structure of music and how its different components together. I would lose motivation and slip on the learning curves of music theory, leading me to wonder: why and how does curiosity kill the cat?

This question made me realize that I prefer to paint without worrying about the material of the brush. To this day, the fretboard can feel constraining. I needed something that would let me move around music in the same way I could discover hidden entrances to buildings and peek around mysterious corridors.

My project, Polymin, is my first draft of a tool to address these problems. It's an AR instrument designed to make music feel spatial, letting you arrange notes in space. The closer you are to each note, the higher the pitch, similar to a Theremin. This allows you to create sounds and music that you can literally move around in all directions.

Creating Polymin was complex in surprising ways. Specifically, programming it was way less frustrating than designing non-intimidating and exploratory interactions. Especially because today's most popular creative tools often have bloated and intimidating interfaces, I simplified the UX by testing with musicians, kids, and adults alike with each design iteration.

Interacting with Polymin starts with playing a single note and is designed to encourage stacking and organizing these musical units into harmonies, just like playing with a single Lego brick naturally leads to creating a small tower or car. In a way, the modular design of Polymin’s “building blocks” is my homage to everything I’ve learned about constructionism this semester. At the core, I want it to be a low-floor, high-ceiling instrument that rewards being curious about the intersection of music, art, and mathematics.

Regardless if Polymin is treated as a real instrument, I hope that it can change the way some people perceive music, even slightly. I really believe in those bits of inspiration when you notice the slightest quirks about the world that make you stop and think for a second-- the kind of inspiration that comes to you in questions and clues. And through my creative process, I’ve realized that it’s not just my goal to investigate these clues but to develop tools that inspire people to become fearless detectives.

With love and music,


Up next: 
Harvard LIT Lab