This is a project through which I am exploring the possibilities at the intersection of image recognition, interactive music and AR. Music cARds is a system that recognizes different pictures to tell a computer what to do next in decomposed musical performance. The current stage of the app allows for synchronized vertical stems to play on cue but future iterations will include the ability to jump from section to section through "structure cards" and will offer superimposed information on the screen. 

​

Music cARds was born with the intention of teaching kids and adults alike about music in an entertaining and interactive way. Through the isolation of smaller fragments, understanding and appreciating more complex pieces becomes an easier task. Also, once these fragments have been assimilated, the system becomes a rewarding real time remixing/DJ game.

​

This concept will also be useful for interactive multi-media performances and literacy/language acquisition. Instead of having instruments and musical terms, the cards can also feature letters, words or Japanese kanji. This way, the user can hear and see realtime what certain characters mean and start building sentences right away. This replicates the way in which children learn much better than most systems we utilize today. Information is assimilated as it is, without older biases, translations or reinterpretations.

​

For this prototype I used Unity, C# and Vuforia. No audio middleware was used but the future iterations could benefit from it.