Music generation
Computers can record sounds as a sequence of numbers and can reproduce them back with a remarkable precision. However, they do not listen and understand sound and music the same way humans do. Musical intelligence involves the ability to compress the signal and represent it in a compact and meaningful way. This allows identifying stylistic traits allowing for creative re-use of the learnt material.
Research in music generation lays at the intersection of MIR, music cognition, music representation, human computer interaction and artificial intelligence. It is particularly interesting to study systems that generate music as they push the research knowledge on all of those research fields at the same time.
Research in music generation lays at the intersection of MIR, music cognition, music representation, human computer interaction and artificial intelligence. It is particularly interesting to study systems that generate music as they push the research knowledge on all of those research fields at the same time.
Unsupervised generation of percussive sequences from an audio example
In order to understand the chain of phenomena that link music cognition to music creation I developed, in a collaboration with Dr. Hendrik Purwins, a system that is able to learn short percussion samples from an audio and generate musical variations by triggering re-shuffled sound atoms. This system applied techniques of unsupervised clustering of sounds, multiple viewpoint transcription of the audio to a symbolic representation, estimation of the metrical levels, variable length Markov model training and generation.
The following video demonstrates the system showing the multiple viewpoint transcription of the original signal, the sound unit clustering (with the colors), the metrical level detection based on the most regular sub-sequence, and finally the generation based on variable length Markov chains (the past context length is highlighted at each instant). More details about the system can be found in our SMC paper.
The following video demonstrates the system showing the multiple viewpoint transcription of the original signal, the sound unit clustering (with the colors), the metrical level detection based on the most regular sub-sequence, and finally the generation based on variable length Markov chains (the past context length is highlighted at each instant). More details about the system can be found in our SMC paper.
Music Interaction
Human computer interaction represents a powerful tool for testing the capabilities of music generation systems. In my work at Sony CSL, I am interested in studying interaction systems that span a variety of techniques from MIR, artificial intelligence and music representation. I have worked on extending a system based on style modeling called VirtualBand which was developed at Sony CSL. The idea behind VirtualBand is to capture essential elements of the style of a musician - either solo or comping styles - and reuse these elements for musical improvisation. During the interaction a corpus of recordings becomes a live object that reacts to a musician who improvises and instantiates music dialogues with the machine.
At its core, the system translates the corpus of audio recordings into a map encoding both temporal and structural relations among the sound units in the corpus.
On top of this, intelligent agents autonomously trigger new music sequences by navigating the space of combinatorial possibilities.
The musician can interact with the agents in various ways: by guiding the generation directly on the sound map or by providing musical inputs thus using the system an accompaniment system.
Finally, by using the very same live performance input as a growing corpus, the musician can instantiate dialogues among the agents and himself where he plays together with multiple copies of himself. The system becomes an intelligent loop pedal that can respect harmonic and melodic constraints provided by a score.
At its core, the system translates the corpus of audio recordings into a map encoding both temporal and structural relations among the sound units in the corpus.
On top of this, intelligent agents autonomously trigger new music sequences by navigating the space of combinatorial possibilities.
The musician can interact with the agents in various ways: by guiding the generation directly on the sound map or by providing musical inputs thus using the system an accompaniment system.
Finally, by using the very same live performance input as a growing corpus, the musician can instantiate dialogues among the agents and himself where he plays together with multiple copies of himself. The system becomes an intelligent loop pedal that can respect harmonic and melodic constraints provided by a score.