With the exception of the final user interface, which runs in Flash, Pandora is based primarily on open source software. The foundation is the PostgresSQL database running in Debian Linux, with the client tier developed in OpenLaszlo. The server-side infrastructure is mostly Java running in a J2SE servlet container. Database access, the playlist generator, and station and feedback management all live in this Java tier.
Pandora creates playlists with a "matching engine," written in C and Python, for each listener station. This engine builds the low-level linkage to the "source" music (the music that listeners indicate they like) and the music that actually gets played (a mixture of what the listener explicitly indicated, mixed with music that the Pandora service believes listeners will like). The replication system is Slony.
The listener runs the Pandora service in a web browser, seeing a mixture of HTML and Ajax techniques for ad delivery and Flash for music delivery. The player, which Pandora calls the "tuner," runs in Flash; the user interface was written with OpenLaszlo. Because Laszlo has been around for six years, it has a long track record in Internet time--certainly longer than its closest competitor, Adobe Flex.
OpenLaszlo has been open source for about a year, inspiring a vibrant developer community to rise up and extend the platform beyond Flash. This is a very interesting development in comparison to the closed-source universe of Flash. OpenLaszlo allows developers to create in the OpenLaszlo environment and deploy to either the Flash runtime or to DHTML.
Because web browsers don't necessarily provide native mechanisms for playing audio, Pandora required some type of plug-in. Windows Media Player, QuickTime, and Flash were the obvious choices. However, Flash appealed to the developers because it didn't come squarely from the two main camps, Apple and Microsoft. Flash does a nice job of providing audio as well as graphics across the three main platforms--Windows, Macintosh, and Linux.
I asked Conrad if there were any special magic about the way Pandora delivers web audio. He said the most interesting item is that the company uses a highly tuned implementation of the Apache web server. "If you were starting an Internet radio service as little as three years ago, it wouldn't be a workable choice," he explained. "Apache wasn't able to deal well with large numbers of connections. The Internet transport, especially with people trying to access with dial-up connections, was such that a connection-oriented protocol like HTTP wouldn't have been a workable choice. So you'd have to go with something like RTSP."
The twisted terms of online radio licensing occasionally intrude on the listening experience.
Of course, it's one thing to develop a slick web app and another to ensure it continues delivering fast, dependable results as the number of users increases. Conrad said there are two kinds of scaling challenges with Pandora. "The first is the need to scale audio delivery to a really large number of simultaneous listeners," he says. "We tried to create an architecture that would allow us to scale our audio capacity horizontally. We have a system that lets us decouple the audio delivery from the computation of playlists. The basic architecture is that our player asks the playlist-generation servers for a series of songs. The player then stitches those songs into a playlist stream. That allows our player to make audio requests from any audio server on the Internet."
Conrad then walked me through the sequence from the point when listeners log on to the point when they start hearing a song:
Also interesting is how the Pandora service balances requests. When a listener makes a request, the request is sent to Pandora's data center, a load balancer distributes that load across an arbitrary number of Playlist Servers, and those Playlist Servers interact with an arbitrary number of replicated music databases that hold all of the music logic information. The playlist information is then returned to the listener.
"Similarly," Conrad said, "When an account-oriented request comes through (e.g., 'Hey, give me my station,' or 'I want to give feedback for the definition of this station,' and so on) that work can also be distributed across an arbitrary number of back-end database servers."
We experienced occasional outages, but they didn't last long.
The second challenge is scaling the computer power required to deliver a truly personal set of playlists for each listener. "From the beginning, many of our architecture decisions were driven by the desire to scale the service truly horizontally," says Conrad. "Much of the work we've done was filtered through the lens of: How do we achieve the amount of horizontal scaling flexibility that's required so that adding capacity means only racking up more boxes and turning them on?"
"Pandora has grown extremely quickly; in just 10 months more than 2.5 million listeners have signed up. That growth rate makes accurate forecasting crucial," Conrad said. "If it takes three weeks to get a server built, burned in, and deployed, and it takes four weeks to add a gigabit bandwidth capacity, being able to make accurate predictions about demands so that those lead times are accommodated is one of the biggest challenges. The business challenge is accurately forecasting and the technical challenge is about the logistics management of the hardware provider, the bandwidth provider, the co-location provider.... It's a technical operations challenge as much as a software challenge." (For more on the scaling challenge, see the "Rack It Up" sidebar.)
"Another interesting challenge around services that are growing really fast is the new generation of servers that pack a lot of storage or CPU capacity into a small package," notes Pandora CTO Tom Conrad. "That means you can spend your co-lo power budget in half the number of racks that they planned for. While you can easily get the space, you can easily run your co-lo provider out of electrical power, which is a massive infrastructure challenge for them.
"High-density deployments also have thermal characteristics that challenge the cooling capabilities at data centers," Conrad continues. "I've talked to people who run some of the largest photo-sharing sites in the world and I know they struggle with this problem: lots and lots of storage, lots and lots of power needed, lots and lots of heat generated. They have sufficient physical space, but they don't have sufficient cooling and sufficient power at their data center. A lot of these data centers were built in the bubble and they were built on a set a assumptions that are no longer true.
"One of the reasons that server people are excited by the dual-core chips is that it lets you have more capacity per watt so you can scale up inside a box without dramatically changing the thermal and power picture."
Sampling CDs in the Pandora library.
Pandora hears a recurring theme from its listeners every day: You're introducing music to me that I might never have heard before.
"When we talk to our listenership about what we could do to enhance the product," says Conrad, "the thing that stands head and shoulders above everything else is making the playlist better. I'm proud of the playlist today, but there is no question that the number-one indicator that consumers will fall in love with the service is how great they think the music is. It isn't, 'Gosh, if you just had tagging, or discussion forums, or artist bios, or digital downloads.' People are asking, 'What percentage of the time do the stations I create play the music I love?'
"We really have a deep belief in building the Genome Project one song at a time, and our listener base one person at a time," Conrad continues. "In a lot of other companies, that would be calculated from a listener acquisition goal. It's really not like that here. It's about a deep understanding that the right thing to do is to get out there and find the people who are making the great music, and meet the people who love the service--or hate it--and bring that back to the company to make the product and music selection better.
"We think of the Music Genome as a collection, something that we are building for prosperity," he concludes, "and we want all great music, where 'great' is defined as 'there's an audience of any size out there that's hungry for music like this.'"
Return to digitalmedia.oreilly.com