RA: Where does the definition of "sampler" fit in?
EL: In sampling you have a model that involves recordings of individual notes. There is then a simple mapping between keys on the keyboard and recordings: when you play this key you get a certain recording. This is a bit simplified, but that's the basic idea—you have very simple and predictable behavior with little of the interesting interaction from one note to the next that occurs in a real instrument. So that's where I really try with Synful's RPM [Reconstructive Phrase Modeling] engine to move out ahead of where sampling is today.
RA: Hang on—is Synful Orchestra a software synthesizer plugin, or a "reconstructive phrase-modeling synthesizer"?
EL: The implementation is in the form of a soft synth plugin, but the underlying technology is the interesting part with RPM. The idea of RPM is that you're trying to figure out from the incoming MIDI note events what kind of phrase you're playing. Specifically, let's say you have a series of four notes, for instance, C-D-E-G [sings the "Tennessee Waltz" intro melody]. Or let's make it a little more interesting with more character to be played as [stretches the same phrase out] "Da-da-daaaa-daaaa," so you have some articulation in there. Now, let's make this phrase go a little faster. We now have a C and a D that are separated by a very short little silence, a D and E separated by the same silence, and then finally an E-to-G slur at the end. All in this little group are one quick phrase gesture.
When a real instrument plays this phrase, all these notes influence each other—especially the short silences between the first three notes of the phrase and when there's a slur between the E and the G. The way those notes sound and are played on the instrument is affected by their context in our example phrase. In speech technology, we call this "co-articulation," meaning the syllables and vowel sounds and their pronunciations are affected by the sound that comes before and after each note or event. It's the same thing for the notes in our phrase here, especially when they're in such close proximity.
Synful Reconstructive Phrase Modeling maps incoming MIDI data to the phrase database to shape the output of the synthesizer in real time. (Click to see the complete flow chart.)
RA: How do you build and play an RPM instrument?
EL: I begin by recording a bunch of musical phrases from an instrument and storing them in a phrase database. When MIDI comes in [from a subsequent performer or sequencer], Synful Orchestra analyzes it in terms of separation or overlap between notes, note duration, velocity, expression control, etc. The idea is to make as clear a picture as possible of the phrase being played from the incoming MIDI.
Then, in real time, the database searches for little pieces of phrases that correspond to the incoming MIDI. This could be just a transition between two notes or a series of several fast notes. For example, if someone plays C-D-E-G on the keyboard with a little separation between C and D and between D and E and a little overlap between E and G, then in real time the database is searched for phrase examples like that. Somewhere in the database we might find a C#-D-E phrase with the right kind of separation and a legato transition F to A.
Now, if we transpose C# to C and transpose F/A to E/G, adjust the timing a bit, and splice the pieces together, we have our desired phrase. That's the way Synful Orchestra and RPM work. There's a lot of searching for little phrase fragments, a lot of pitch shifting and time adjustment—or let's just say "morphing"—and a lot of splicing of phrase fragments. That's something that you cannot do with a sampler with sounds stored as PCM [pulse code modulation] samples.
RA: Why? Because the resulting database would be too large?
EL: Not only that, but ignoring the size problem, it's just not a flexible or malleable way to store sound. In PCM it is very difficult to change the pitch without changing the timbre. It's also difficult to change the length of a note without changing the speed of the vibrato, and it's difficult or impossible to splice phrases without generating a noticeable timbral discontinuity. That's why these sample libraries are getting so huge in trying to cover just the most simple note transitions.
Here's an analogy from the graphics world: a bitmapped versus a vector representation of an image. In Photoshop you're for the most part manipulating bitmaps, and in Illustrator you're manipulating vector graphics. So, in Illustrator you're manipulating objects that are represented by formulas, such as a circle, a square, a rectangle, and so on. In Photoshop, there are some filters that try to get smart about a bitmapped object's form, but essentially there's no knowledge of what the objects "are" in the picture. It's all just color represented by bits. So, in a sense, the traditional sampled representation of an instrumental sound is like a bitmap. It's a dumb image in which you have no knowledge of the contents. Let's say there was some circular object in your picture, a light bulb, for instance—
RA: Perfect example! There's a dim light bulb over my head beginning to glow brighter.
EL: [Laughs.] Right! So let's say you want to move that light bulb to the right, or make it bigger or smaller. In Photoshop it is very difficult to do that because there's no knowledge that there's a vector object within all those colors that are the light bulb. In Illustrator it'd be easy because it's represented by an intelligent circle object that you can grab, move, enlarge, and even recolor.
In a sense what I'm doing with Synful is starting with a bitmapped image of sound, a PCM recording of a musical phrase, and trying to convert that bitmap into a smarter vector sound image—the RPM-additive representation—that knows about the objects but still keeps it sounding like the original PCM recording, except now with the flexibility of moving objects around, the notes, as you do with vector objects in Illustrator. But it's still difficult to do with sound. For me, Synful Orchestra is technology [for] getting far more expressive musical synth performances and compositions.
Lindemann (R) jams with guitarist and drum machine inventor Roger Linn.