The idea of using a phone or tablet to scan Carolan’s decoration to read its history has always felt somewhat incomplete. While this kind of visual interaction makes sense in some cases, when all is said and done Carolan is a musical instrument – made to be be played. Shouldn’t it also be possible to conjure up its story by actually playing it?

This nagging thought kept returning to the research team and eventually inspired us to pursue a new idea called Muzicodes. The Artcodes technology that is used to create Carolan’s interactive decorations encodes information into visual patterns. Why not do the same for music? Why not embed hidden codes into music so that digital interactions can be triggered whenever a musician plays them?

There are already several technologies for recognising musical phrases and triggering interactions such as the Shazam music discovery service that attempts to recognise songs from live sound or the Chirps technology that encodes and transmits digital assets as either audible or inaudible audio signals. Our approach is different. We provide composers with ability to embed hidden codes as phrases in a musical score. They can compose a series of notes – combinations of pitches and rhythms  – that are then recognised by a computer whenever they occur in the stream of notes generated by someone’s playing. You can read full details in this paper,

An important feature of the approach is that the Muzicodes can be triggered from a variety of different instruments providing that a stream of notes can be generated. Successfully performing the codes might trigger various actions from introducing musical effects and additional instrumental parts to displaying video and potentially even controlling lighting and effects on stage.

The video below shows a banjo, keyboard and of course, the Carolan guitar triggering various Muzicodes. In the case of Carolan, the codes are being used to retrieve posts from this blog so that a musician can explore its lifestory by playing various different chords. In the case of the banjo, playing a key phrase from an Irish tune launches a video of Carolan playing a backing track to which they can play along, showing how Muzicodes might be used to trigger additional parts during a live performance.

An important feature of Muzicodes is the use of ‘wildcard’ notes and phrases within a code giving flexibility over how it can be played, for example with the player being able to add various embellishments and improvisations. The video shows how the same code can be played in quite different ways and yet still trigger a result. This makes Muzicodes very different from existing music recognition systems.  Put in Computer Science terms, we are encoding information at a higher level of abstraction – as composed musical phrases rather than recognising finally rendered sounds Not that we are the first to do this. Various composers have hidden codes in their compositions, most notably J S Bach. The difference is that ours are intended to be recognisable by computers and to trigger interactions.