VOICE OVER: Earlier this year, Celemony introduced the new Direct Note Access technology followed by a huge echo in the public. Now we visited Melodyne inventor Peter Neubacker in his laboratory to get some insight to get some insight to the development of this groundbreaking technology.

PETER NEUBACKER: I have been thinking about how to best make use of polyphonic audio material for quite some time, but my thoughts were more like some general pondering about algorithms, about listening really closely to what's inside the material and about ways to translate this.

The first impulse to actually become concrete was provided by a Melodyne user. This guy from LA who had already been a long-time Melodyne user at this time did a production using a marimbaphone. This instrument was readied for the session, and when they realized that there had been one note written wrong in the score, the instrument had already left the studio so they had no chance to record it over.

And yeah, that's how it is with the marimbaphone: There's a lot of decay and the notes blend into each other. Let's just play this for you.

[PETER NEUBACKER PLAYS BACK A MARIMBA TRACK]

You see, it was a pretty short phrase, which was good luck for experimenting with it. Now the question is how to get inside of this phrase. With monophonic material, this is easy. The signal is periodic, allowing you to read a pitch at every point in time. Now, when you have polyphonic material, you need to transform this, like in this typical view of a spectrogram. Here, you might already anticipate what's inside the sound as you can see single notes, some being higher, others being low. Let me play it again.

[PETER NEUBACKER PLAYS BACK A MARIMBA TRACK]

Okay. There is a kind of progression, but the fundamental tones down here are all really close to each other whereas the harmonics are a bit easier to see although they still blend and mix. You can't really judge what belongs to each other.

Well, my first attempt to get in there in order to find dedicated notes was to do it just the other way around than you normally would. I simply listened and measured it. Let's say this tone here is A = 110 Hz and it starts at 1.35 seconds and sustains until 3.5 seconds or whatever. Well, I wrote these values directly into the program, so the program already knew what it was supposed to look for. Then I started to calculate and diversify pitches in order to find out which spectral parts belong to which fundamental tones, and this turned out to work better than what I expected. I had never tried this before. I've been only thinking about it in theory, and this was the result of this first approach...

[PETER NEUBACKER PLAYS BACK A MARIMBA TRACK]

...which of course sounds exactly the same as the original. The only difference is that you can grab every note individually now, and you can move it up and down, changing its pitch at will.

[PETER NEUBACKER PLAYS BACK A MARIMBA TRACK WITH ONE NOTE MOVED UP A FEW PITCHES]

So, I did send this to the Melodyne user. I gave him separate sound files, laid them out in a Melodyne arrangement, and he was really happy with it, surprisingly thinking this was a pretty normal thing. Yeah, and so this was the first step in really getting a clue as to this form of note separation. Later on, I made the next step, which normally would have been the first one to let the program find the notes on its own and to separate them content-wise before separating them sound-wise.

Okay, let's see how this works practically. What can you do with the polyphonic detection and what kind of results do you get from it? You see, this is a normal sound file being shown in a pretty usual style. Let's listen to it.

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK]

And so on, so this is a Spanish guitar played live, and what you could already do with Melodyne before was to slowly ride to the sound, listening precisely to what's going on.

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK TIME-STRETCHED SLOWER]

As you can hear, you have to deal with chords here, which are blending into each other, but if you wanted to know what is exactly going on at this point, what is happening inside the chord, you would need to spread it up to look inside just like viewing a piano roll or any kind of abstract notation or so, and this is exactly what happens when you decide to separate the notes.

Now, a detection process starts, and when it's done you have this result. Every note can be seen at exactly the place where it was played. Let's play it once again.

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK]

You see, every note is actually there. Each of these things we call "blobs" stands for a note, and I can grab it and move it or I can turn its volume up or down. I'll just let this passage play for a while...

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK WHILE MODIFYING THE AMPLITUDE OF ONE NOTE]

...and change the pitch...

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK WHILE MODIFYING THE PITCH OF ONE NOTE]

...or I can move it back or forth in time.

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK WHILE MODIFYING THE POSITION OF ONE NOTE]

In this way, I can totally redesign the melody or I can retune the whole thing by using special macro functions. Here the system analyzes the harmony of this passage as being A minor. Let's take a longer package, and while it plays we switch on the fly to another key.

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK AND SWITCHES KEY FROM MINOR TO MAJOR]

Major...

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK]

...or a Spanish scale...

[PETER NEUBACKER PLAYS BACK A NYLON-STRING GUITAR TRACK AND SWITCHES MODE/SACLE FROM MAJOR TO SPANISH SCALE]

...and so on. Now, since this part has been analyzed, the sound of every note can be extracted and I can freely manipulate any single note inside.

Some years ago I was asked if it could be possible to separate notes out of a chord once the computing power was strong enough. At that time, I was already convinced that this is no matter of pure computing speed. I mean, you would have been happy if you managed to extract notes out of a chord even if this took the whole day. But in developing a software like this, the interaction with a fast computer turns out to be of great importance. You know, it's not much fun to develop something when you have to wait half the day before you can check your results and see if your code works in real life. From this point of view, computing power was indeed crucial for Direct Note Access. It helped to have more fun in developing this technology, but the usage of DNA isn't really influenced by computing power. The analysis is faster than real time and the playback is fast anyway, so for musical usage of DNA the computing power isn't such a big issue.

Let's try a real life test then. I'm going to record a guitar loop.

[PETER NEUBACKER RECORDS AN ACOUSTIC GUITAR TRACK]

Okay. Let's see what we've got.

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK]

Well, the guitar. Now we are separating notes again, and in this case, this is split up into exactly six voices because the chord was built out of six strings. You see, each note can be associated to one string. This is coming from the high E string, this is the B string, and so on. Now we can do what we always could do with Melodyne: make it play back faster...,

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK FASTER]

...or slower...,

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK SLOWER]

...or pitch it up.

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK AND TRANSPOSES PITCH HIGHER]

But you notice that at this point, everything is simply transposed, the same as if I had placed a capo on the guitar and not like I had changed the positions of each finger individually, so that's not what I actually want.

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK AND REVERTS IT TO NORMAL PITCH]

Instead, I would rather want to grab single notes. Let's say this one here.

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK AND RAISES THE PITCH OF ONE NOTE]

Or I'd like to recompose the whole chord. I'll let it roll now and play different chords on the keyboard, forcing each note to be moved individually on every string.

[PETER NEUBACKER PLAYS BACK RECORDED ACOUSTIC GUITAR TRACK AND USES KEYBOARD TO PLAY CHORD PROGRESSIONS USING SAME GUITAR LOOP]

Well, you can process all kinds of material with this, especially all kinds of polyphonic stuff, but it's mainly meant for single instruments which you can play chords with like a piano or guitar, a harp and so on. In the studio, you normally record these instruments on separate tracks and just like with monophonic instruments in Melodyne before, it is now also possible with polyphonic instruments to individually edit the separated notes in each track, and you may also take a group of instruments like a brass or a string ensemble where the sounds are pretty homogenous and record this group to one track in which you can still separate and edit every single note. But what the program doesn't do is to identify voices or voice progressions so what you get is more of a kind of piano score of the whole piece rather than the score with individual instruments. On the other hand, why not take a whole mix with drums and everything? It can well be possible to change the whole song from major to minor, only the more complex the signal gets, the more complicated it will be to separate and process everything properly enough but no one keeps you from at least trying any material.

The situations for using this technology are endless. The first thing obviously would be the correction. That's what Melodyne is used for already in the studio. When somebody played a wrong note, you now don't have to stop the take. You just keep on rolling and repair the track later by retuning this wrong note or deleting it. This can now also be done with polyphonic instruments like piano, guitar, or others. One other thing would be to retune a guitar after the recording has been done; no problem as long as you recorded this to a single track.

So, these are the typical situations where you can insert the Melodyne plugin on multiple tracks in any digital audio workstation to individually correct several instruments after the recording has been done.

The next thing would be to use it for composing or arranging. Just take any pre-recorded material, any guitar loops or piano accompaniment, and bring it into a complete different context by completely rebuilding and adjusting its harmonic structure. Think of all your useless old libraries. Suddenly, all these sound can gain new life and can lead you to new compositions.

These scenarios are showing completely new fields of application, and we don't know yet what else will happen with it so I'm pretty excited and curious to see what the users will do with it and what feedback we're going to receive. Let's see what comes out from this.