Yahoo! Pipes: "a milestone in the history of the internet"

When Tim O’Reilly calls a product “a milestone in the history of the internet,” something really, really cool is happening. Here’s what the coolness is all about:

Pipes is a hosted service that lets you remix feeds and create new data mashups in a visual programming environment. The name of the service pays tribute to Unix pipes, which let programmers do astonishingly clever things by making it easy to chain simple utilities together on the command line.

Yahoo! Pipes was built by Pasha Sadri, Ed Ho, Jonathan Trevor, Kevin Cheng, and Daniel Raffel. I work in the row of cubes right beside the team and it’s been inspiring to watch Pipes go from idea to reality (I was lucky enough to work closely with Ed and Jonathan last year when we released the Checkmates prototype at eTech).

Jeremy nails the big picture as usual, so I’ll just point to a mashup that I built. The most exciting thing is that I didn’t really have to know how any of the APIs worked that I used — they are rolled into the product and all I have to do is feed data into the mix from my choice of RSS feeds, set up the pipeline with various parameters, and the data I want transforms and flows out of the other end like magic.

Here’s basically what is happening behind the scenes. I took the Upcoming.org RSS feeds for the Fillmore and the Warfield in San Francisco (two music venues) and joined them into one feed, then piped the unified feed through the Content Analysis Term Extraction API to pull out the keywords in the RSS feeds. Then I looped through the Flickr API to run queries for photos on those keywords. My logic wasn’t perfect, but all I know is that I saw a photo of George Clinton and sure enough, he’s playing at the Fillmore on March 9. It’s 1:20 am as I write this, and although I have been meaning to go to bed for about three hours now, I’ve just been having too much fun reconfiguring these pipes. Yes, fun.

A big congratulations to the team and Bradley and Caterina for carving out the space for the team to do their magic. This is a proud moment for Yahoo!