O'Reilly    
 Published on O'Reilly (http://oreilly.com/)
 See this if you're having trouble printing code examples


Hit "Play" and a bass guitar shaped like a pogo stick plucks itself with robotic arms, then gallops down a 3D path alongside fellow computer-generated bandmates. Skip ahead and streams of shiny balls bounce in perfect tempo across a room of spinning drum heads, chimes, and plumbing as 5.1-channel surround sound wraps around your head.

Click again and you're breathlessly floating above dozens of crisscrossing strings on an impossibly multi-necked "acoustic" instrument. Strangely graceful robotic fingers pick and strum, producing the sounds you'd expect from an actual guitar, lute, cello, or harp. Click once more and you're front and center at a majestic computerized pipe organ as it bellows a MIDI-powered, high-definition version of Mussorgsky's Pictures at an Exhibition.

This is Animusic 2, the astonishing new DVD from Animusic, a wildly creative video production company.

"Watching computer animation perfectly synchronized to MIDI music does seem to stimulate something very positive in our brains," says Wayne Lytle, Animusic's chief creative visionary. "Someone once gave me a scientific explanation of the process, and it only half made sense to me," he laughs. "But apparently watching our music animations lights up more of your brain and sparks creativity."

Scientists say they've gotten new ideas as they watched Lytle's music animations, while elementary school children who had never shown an interest in drawing suddenly fill walls with Animusic pictures. My 76-year-old mother, who recoils from all things digital, was mesmerized by the "Pogo Sticks" and "Resonant Chamber" animations on the Animusic 2 DVD.

Pogo Sticks Movie

Drum and bass robots bounce in perfect sync to a musical score in "Pogo Sticks." Animusic's proprietary MIDI-to-3D-graphics software transformed each note and drum hit into motion. (Click here to play a 4.5MB QuickTime movie excerpt.)

Surely Lytle knew these would be the eventual fruits of his labor when, 15 years ago, he wrote his first music animation software after completing graduate work in computer graphics at Cornell.

"Not at all," he admits. "We had no idea. It was about making Animusic for people like us: music fans and computer geeks with similar tastes. It was more about building something we wished we could buy off the shelf but couldn't, so we created it. We had no idea our DVDs would appeal to babies and great-grandparents alike."

Cows, Farmers, and Computer Music

Dave Crognale and Wayne Lytle

Dave Crognale and Wayne Lytle

Somewhere in the pastoral hills of upstate New York, not far from Cornell, Lytle lives with his wife/office manager, Patricia, and their kids. Animusic's lead digital artist David Crognale and his family live just down the road. It's a million miles from the high-tech Meccas of L.A., Silicon Valley, and New York City.

"Our neighbors are cows and farmers," says Lytle. "I'm wound pretty tight as it is, so my brain would probably blow a fuse in a major city. This laid-back environment is a perfect balance."

Lytle envisioned the first Animusic disc in 1982, when MIDI synthesizer technology was forming. But it wasn't until the late '80s that computers caught up enough for him to begin plying his art. But what was his original inspiration for Animusic?

"That's difficult to answer because this is pretty much all I think about for 16 hours a day," he says. "Writing software for the new Animusic production pipeline, sequencing music, designing virtual instruments—it all just floats my boat. I do know that what I'm doing is what I was meant to do."

Lytle punched out his first computer program in the third grade—literally. Back then, computers filled large rooms and programmers entered code on punch cards. When Lytle's father introduced him to the mainframe computer in his office, it was love at first sight, and programming has been nothing but fun for Lytle ever since. Along the way he's also embraced the synth stylings and odd time signatures of early ELP, Yes, Genesis, Kansas, and Rush; studied classical piano; and played keyboards in progressive rock bands. But he says that, as much as he desired to do so, he just wasn't cut out to be a music performer.

"Honestly, I wasn't that good. I just wanted to be a rock star," Lytle laughs again. "But that was just a phase I went through leading me to where I am today. At Animusic, the real rock stars are the little computer graphics guys on the covers of our DVDs. I'm more than happy that everyone is watching them while I'm tucked away in the back room."

Starship Groove

Neon lights flash to each note as a band of robots rock on top of their spaceship in "Starship Groove." (Click here to play a 4.3MB QuickTime movie excerpt.)

MIDI In = Animusic Out

And what an amazing back room it is at Animusic. The company is currently in preproduction for Animusic 3, and there's even talk of releasing Animusic|studio, their proprietary animation and music sequencing software, to the rest of us. Imagine composing a piece of MIDI music, importing those tracks into Animusic|studio (or sequencing directly in the program), and leaning back in your desk chair to watch as your music magically morphs into fully rendered Animusic animations.

"It's been discussed," Lytle confirms. "But that is still a ways off. There are many ways Animusic|studio could find its way into the hands of users—from simple tools that kids could goof around with to a fully professional tool not unlike what we use here every day to create the Animusic discs. It's a matter of deciding what makes sense for the company. I'd like to see us release this to the world someday. But when a creative content company like ours releases its software, suddenly things change a lot—it's no longer just a few people working in predictable ways, sitting next to the developers who wrote it. Users will try all kinds of crazy things and," he laughs, "the software would probably go up in flames within minutes."

Animation Integration

Wayne Lytle at PCs

Animusic founder Wayne Lytle wanted to make music-driven computer animation back in 1982, but the technology wasn't available. Over the years, he developed his own software to make it happen.

Lytle and his team have plenty of other projects to keep them busy. They're currently switching from their longtime rendering tool, Autodesk's 3ds Max, to Pixar's RenderMan. They rendered Animusic and Animusic 2 with 3ds Max on networked dual-processor Windows machines, but things are different now out amongst the cows. There's a lot of music synthesis going on, as well.

"We run a ton of software synths in parallel to the realtime graphics animation," Lytle explains. "I have one dual-processor machine fully dedicated to 3D animation while two other dual-processor machines are dedicated to the audio. That way they all have enough cycles. We keep them synced together through MIDI.

"There is a lot of setup involved in Animusic|studio, but once the setup is in place, it's completely automatic; as the music changes, so does the animation—automatically. The heart of Animusic|studio is a set of algorithms called MIDImotion that analyze the MIDI music fed in and look forward and backwards in time. And there are secondary motions, such as stage platforms moving up and down, which are keyframed manually, as well as other things such as camera moves and lighting changes."

Pipe Dream 2

In "Pipe Dream 2," balls shoot out of pipes and smack into drums and chimes. Each flying ball represents an individual note. (Click here to play a 5.4MB QuickTime movie excerpt.)

Lytle has progressed through a variety of MIDI sequencers over the years, from Roger Powell's Texture 3.0 to MOTU Performer and Digital Performer, then to Steinberg Cubase and Nuendo. But eventually he couldn't resist the temptation he'd had for years to write his own music sequencer. The result is now built into Animusic|studio.

"I've integrated my favorite features from all the sequencers I ever worked with into Animusic|studio," Lytle says. "It's now a very customized MIDI sequencer tuned just for me. I would never claim it to be a general sequencer that everybody would enjoy using, but for the way I work, it's very fast and fully integrated with the animation generation processes. It makes a huge difference when it comes to constantly tweaking the music and seeing the resulting animation immediately. I actually used to have to 'sneakernet' every sequence change from my MIDI studio to the animation studio by car; it took 15 minutes. Now we see the effect of every single note change reflected immediately in the realtime wireframe previews."

For instance, Lytle says he could never have completed the complex "Gyro Drums" on the Animusic 2 DVD without integrating his MIDI sequencing module into the Animusic|studio suite.

"Animusic|studio is all written in C++ using OpenGL for the realtime graphics and Trolltech's Qt for developing the GUI stuff," Lytle continues. "I can't say enough good stuff about Qt. It rocks, I'm a huge fan. In the '70s I was a huge fan of various bands; now I'm a huge fan of software. I've gone from gazing at record covers to gazing at online help files, which is a little sick, now that I think about it. Maybe I need some professional help!" he laughs.

Gyro Drums

"Gyro Drums" was one of the most challenging pieces to animate, consisting of 150-odd drums and three spinning robot drummers. (Click here to play a 4.5MB QuickTime movie excerpt.)

MIDI and Sample Synth Tools

The multi-necked string instrument on Animusic 2's "Resonant Chamber" is a gem. The harmonic string vibrations and intonations are absolutely stunning in their accuracy. Sound sources in "Gyro Drums" and "Resonant Chamber" include Larry Seyer's Acoustic Drum and Acoustic Bass libraries, the Vienna Concert Guitar library, Post Organ Toolkit, and various TASCAM GigaStudio sample discs.

"'Gyro Drums' was actually my first adventure with GigaStudio, and 'Resonant Chamber' was entirely done with GigaStudio," Lytle reveals. "As far as soft synths are concerned, I probably have way too many! I got obsessive with collecting them to the point that I couldn't even keep track of them anymore. The ones I use most heavily, though, are Spectrasonics' Trilogy and Atmosphere; Novation's V-Station, which is pretty cool; and Native Instruments' Battery 2, which, with the exception of the song 'Gyro Drums,' is the main percussion tool for everything else we've done so far. Animusic 2 was 100 percent soft synths."

Lytle has also been a big fan of Propellerhead's Reason since Version 1.0. He says he'll likely get more into Reason as his company develops Animusic 3 in 2006, and he plans to drive Reason from Animusic|studio.

Guitar-Lute

Spidery robot fingers fret and pluck the strings in "Resonant Chamber." (Click here to play a 4.2MB QuickTime movie excerpt.)

Render unto Crognale

Parallel to Lytle's music composition, instrument rigging, and animation-parameter setup is the work of digital artist David Crognale. The latter spent most of his time during the production of Animusic and Animusic 2 modeling instrument details and tweaking lights, shaders, and texture maps. Every subtle scratch and dent on a drum head, stage floor, and cymbal face is created within Dave's realm.

"The things that differentiate us from most other animation studios such as Pixar is that our animation is driven by music rather than a story, and that most of the animation here is generated procedurally" rather than through keyframing, Lytle says.

Dave Crognale

David Crognale works on "Resonant Chamber." In the background is "Fiber Bundles."

All the gorgeous graphics of Animusic and Animusic 2 were modeled and rendered in 3ds Max, a program Lytle and Crognale started using with Version 1.0. Crognale also uses Adobe Photoshop, Adobe After Effects, Fractal Painter, and a host of others.

"The rest is Animusic|studio, which is really the hub between the graphics and music worlds," says Lytle. "Dave drives Max, and I drive Animusic|studio. I build the skeletons and do the engineering parts of all the complex mechanisms and how they work together and Dave fleshes it all out. If you look at my screen, everything is spheres and cubes," Lytle laughs. "You'd probably recognize which animation it was, but there's no detail, just all these white plastic primitives and wireframe. Meanwhile, Dave is in Max, without any concern about any of the structure or mechanics or hierarchy of the objects, and he focuses entirely on the aesthetics of how to make it look beautiful. As I update the music and animation, the Max objects come to life."

Two animations from the Animusic 2 DVD feature the lush backgrounding of renowned digital matte painter Eric Chauvin, whose BlackPool Studios created digital backgrounds for Star Wars: Episode V and Contact, and for the TV shows Alias, Babylon 5, and Lost.

"We had just decided we needed to find someone to do backgrounds for us when, out of the blue, Eric called," Lytle remarks. "It was one of those synchronicity things—'Hey, do you guys need any help with your backgrounds?' 'Uh, yeah, we do.' He's very, very good. My son and I were watching Star Wars: Episode V the other day and there was Eric's name in the credits."

Chauvin's background work shines behind the "Pogo Sticks" and "Heavy Light" animations on Animusic 2—but not too much. "This is CG, after all, and we're not trying to look too photorealistic," Lytle explains. "If you don't have the same degree of realism and surrealism between your animated models and backgrounds, it will look incongruous and fake."

Cathedral Pictures

"Cathedral Pictures" recreates Mussorgsky's Pictures at an Exhibition on a pulsating organ and insectoid chimes. (Click here to play a 4.2MB QuickTime movie excerpt.)

For "Pogo Sticks," Chauvin took a series of digital photographs to cover a 360° section of real sky, then flew those files into Photoshop to blend the seams into one expansive, giant background. There he added painterly effects to match the look of the Animusic scenes.

Lytle notes, "To blend Eric's sky with Dave's modeled 'Pogo Sticks' world, Dave used a lens-flare plugin in After Effects." He made the flare track the position of the virtual sun, which positioned the apparent light source in the animation—behind the third stage in this case. That psychological cue "really helped blend the digital background with the rendered models," Lytle says.

Animusic's two DVDs are available through the company's website for $19.95 each. In addition to the main movies, each disc contains numerous behind-the-scenes features, such as multiple viewing angles and close-ups on the otherworldly instruments.

Animusic DVDs

If Animusic does decide to release its proprietary music animation software, perhaps the next DVD in the series could be yours.

Animusic QuickTip: Get Real…Dirty

Map Overview

Rendering a realistic drum requires getting down and dirty.

"In order for something like a drum head to look realistic, it needs to look less than perfect," reveals David Crognale, Animusic's digital artist. "It should look like it has been hit repeatedly. It needs to be given the appearance of being aged, used, and worn.

"First, it is helpful to look at the real thing. This may seem obvious but is something easy to forget: find references of the thing that you are texturing. It can be the actual object, if that is practical, or photos or footage. Anything that shows how a thing looks in real life."

The key concept, Crognale continues, is to mix things up, combining or layering a number of different texture maps to create a more realistic surface. He typically uses two or more different texture maps for the various characteristics of his intended visual result.

"The main effect is the hit marks, usually centered on the drum head," he explains."The location and number of hit marks are directly related to where the sticks or balls impact the drum. If you really want to go wild, you can also look at how many times a particular drum is hit and adjust the intensity of the hit marks in proportion. The hit-mark map is sometimes hand-painted in a program like Photoshop or Painter using a pressure-sensitive tablet. Another approach I may use is to make a procedural map with radial gradients and some built-in noise." (See Figure 1.)

Fig. 1: Hit Map

Figure 1: Hit-mark image

Another characteristic element Crognale layers on an Animusic drum head is overall wear and tear. He sometimes calls this element a dirt and grime map. (See Figure 2.) To his localized hit-mark map, he will add a subtle texture map containing random smudges and scratches. These marks are either hand-painted or taken from a texture map collection. Crognale says this can also be accomplished by combining some procedural noise maps, but warns that can look fake if not done carefully.

Fig. 2: Dirt Map

Figure 2: Hand-painted smudges and scratches increase realism.

"An important concept here is to use restraint," he continues. "You don't want the hit and smudge maps to draw too much attention to themselves, but to blend in with the rest of the scene and level of detail and other intentional imperfections. Another concept is to not limit the maps to the color channel of the material. I tend to apply some variation of the color map to the specular—that is, how much shine there is—and to the glossiness, or in other words, how sharp the highlights are."

Fig. 3: Mixed Map

Figure 3: The mixed image, mapped to a 3D surface

Fig. 4: Rendered Drum

Figure 4: Finally, the drum head is rendered.

Crognale sometimes applies a bit of digital dirt to the bump-map channel as well. "The idea here is to add variations to the highlights on the drum head's surface," he explains. "Again, this is to break up that computer graphics perfection."

Animusic Production Team

Randy Alberts is an author, musician, and photographer who lives on Lummi Island, Washington.


Return to the digitalmedia.oreilly.com

Copyright © 2009 O'Reilly Media, Inc.