Continuous Deployment and Basketball

Anyone who knows me well knows that I love basketball and I love documentaries, so basketball documentaries are the best. Today, I watched one from ESPN’s excellent 30 for 30 series, “The Guru of Go,” produced by Oscar-winning director Bill Couturié. Here’s the synopsis:

By the mid-1980s Paul Westhead had worn out his welcome in the NBA. The best offer he could find came from an obscure small college with little history of basketball. In the same city where he had won an NBA championship with Magic and Kareem, Westhead was determined to perfect his non-stop run-and-gun offensive system at Loyola Marymount. His shoot-first offense appeared doomed to fail until Hank Gathers and Bo Kimble, two talented players from Westhead’s hometown of Philadelphia, arrived gift-wrapped at his doorstep. With Gathers and Kimble leading a record scoring charge, Westhead’s system suddenly dazzled the world of college basketball and turned conventional thinking on its head. . . .

Wessthead (who was also a Shakespeare scholar in addition to a basketball head coach) built a system that he called simply “The System.” It’s a beautifully-told story (with deep tragedy), and highly recommended (you can buy it on iTunes). What struck me about “The System” that Westhead created was how similar the language could be to how we at Etsy and others talk about our own system of continuous deployment and “The Etsy Way.” In one segment (starting at 17:47 in the iTunes video), the coach and his players talk about how their seemingly chaotic run-and-gun system (disparagingly called “roller derby in shorts” by ESPN announcer Dick Vitale) was actually quite well-thought-out:

A lot of people say, well, that wasn’t basketball, that was like street ball. Well, it might be street ball to you, but to us, it was orchestrated. (Paul Westhead, coach)

It looked like absolute chaos (Tom Peabody, player)

. . . but it’s very much a structured way to play. (Bo Kimball, player)

The “some people think it’s crazy but this is actually carefully thought out” attitude resonated with me, and reminded of discussions I’ve had about the structured chaos of Etsy’s deployment process, which I detailed in my Optimizing for Developer Happiness talk from Railsconf last year (see the bit about the “push train” starting at 17:58 in this video).

Like I’ve learned at Etsy, talent and culture mattered most in “The System,” too. Without that a great system doesn’t work — see this from the director’s personal statement about the film:
Paul’s system was magic (and big fun to watch) – if you had the right players. Without them, the system looked like crap.

Since Westhead is also a Shakespeare scholar, Shakespeare quotes are sprinkled throughout the film. At one point, Westhead says, “People ask me my favorite Shakespeare quote. It’s a simple one-liner about basketball and maybe your life, from Hamlet: ‘The Readiness is All.’” In Westhead’s system, players were expected to push the ball down the court and be ready to shoot quickly. Readiness is a key aspect of continuous deployment, too. In slide 43 of my talk at SXSW about Continuous Deployment at Etsy, I quote Etsy engineer Lacy Rhoades: “Not being in a state to deploy is a matter of liability. It’s like having the only fire exit blocked. You ignore it at everyone’s peril.” Yes, readiness is all.

When I see connections between basketball, leadership, technology, and Shakespeare, I just can’t resist writing them down. Thanks for indulging me.

Here’s a clip from the film:

The 20 Percent Doctrine

In March of 2010, I got an email from Ryan Tate saying he was writing a book about skunkworks and experimental projects inside large companies, and he wanted to talk to me about Yahoo! Hack Day. Over the next year and a half, I did periodic interviews with Ryan. Ryan eventually asked me to write the foreword for the book. By then it had a title: The 20% Doctrine: How Tinkering, Goofing Off, and Breaking the Rules at Work Drive Success in Business. I loved the topic so much, I agreed to write it during the holiday season (a busy time at Etsy!) I’m happy with how it turned out.20% Doctrine

The book shipped yesterday, and Ryan’s chapter on Hack Day captures the beautifully chaotic spirit of Hack Day better than anything else I’ve read. (The big surprise when I read the chapter for the first time yesterday was the description of me as a “laid-back, freedom-loving, rabble-rousing Yahoo programmer, a sort of cross between Bill Gates and cult movie character Jeff ‘The Dude’ Lebowski.” I’m not that smart or that cool but it definitely made me laugh!)

Below is a copy of the foreword — a big thanks to Ryan for letting me write it, and to all the Yahoo! hackers and leaders who helped make it all possible. Years later, I am still inspired by the spirit we captured with Yahoo! Hack Day, and feel really honored that I played a role in it. I hope you enjoy the foreword, and the whole book.


On December 12, 2005, the phone at my desk rang. It was the Yahoo! HR department. The phone at my desk almost never rang, so this was a bit of a surprise. I had just helped put together something really awesome, though, so I smiled and braced myself for a hearty congratulations.

The Friday before, I had organized the first internal Hack Day at Yahoo! with the help of a loosely-organized band of people around the company. The “hack” designation for the day was a tip of the hat to hacker culture, but also a nod to the fact that we were trying to fix a system that didn’t work particularly well. The idea was really simple: all the engineers in our division were given the day off to build anything they wanted to build. The only rules were to build something in 24 hours and then show it at the end of the period. The basic structure of the event itself was inspired by what we had seen at small startups, but no one had attempted such an event at a large scale at an established company.

The first Yahoo! Hack Day was clearly a success. In a company that was struggling to innovate, about seventy prototypes appeared out of nowhere in a single 24-hour period and they were presented in a joyfully enthusiastic environment where people whooped and yelled and cheered. Sleep-deprived, t-shirt-clad developers stayed late at work on a Friday night to show prototypes they had built for no other reason than they wanted to build something. In his seminal book about open source software, The Cathedral and the Bazaar, Eric Raymond wrote: “Every good work of software starts by scratching a developer’s personal itch.” There clearly had been a lot of developer itching around Yahoo! but it took Hack Day to let them issue a collective cathartic scratch.

But back to that call from HR. I grabbed the phone, prepared to be gracious, then the HR person on the other end told me we needed to take down one of the hacks, a hack by Cal Henderson that created an API to our company directory (available on Yahoo’s intranet, known as “Backyard”) and built a hot-or-not-style “Backyard War” app on top of it. No one was ever really sure what these wars were about, but it was viscerally fun to place random co-workers in battle with each other over unknown stakes. The impromptu judging committee I had put together had given Cal a trophy for his work.

The HR person who called me had made a critical error in reasoning. While I had organized Hack Day, I by no means had any actual control over the event itself or any of the participants. I had designed it that way. There were no sign-ups in advance, no proscribed projects or areas of focus, and no central servers where the projects lived. I couldn’t have taken Backyard War down if I had a gun to my head. In the end, I think HR may have eventually gotten to Cal, but it didn’t matter. At future Hack Days, there was always a feeling of danger and although no one ever really said it, there was an ongoing secret competition to see who would get the call from HR this time. When you’re trying to get things done and change a system, expect to upset a few people along the way.

With that first event, the die was cast and the completely improvised format from that first Hack Day became somewhat of a standard. At that first Hack Day, I didn’t expect seventy presentations (remember, I had no idea who was presenting until they got started). I had originally planned to give each team 5-10 minutes to present, but with 70 hacks, I called an audible — each presentation would be two minutes or less. Years later, I found myself in an elevator at the site of a hack day and heard one hacker explaining the rules to another, “Dude, demos are always two minutes. It’s a rule.” I chuckled to myself that these “rules” had become so solidified over time.

After that, we did Hack Days all over the world, on three continents. We continued doing internal ones for employees and did our first open one for the public nine months after the very first internal one. The basic “rules” remained. People built huge numbers of prototypes to solve a wide range of problems, and the only thing that really changed from place to place was the food. We had pizza in California, samosas in Bangalore, and bad English pizza in London.

In London, we did a joint hack day with the BBC and held it at Alexandra Palace. Just after the event started, lightning struck and the power went out, triggering a fire suppression system that opened up large sections of the roof, causing indoor rain. Hackers pulled out umbrellas and simply started drawing on paper until the power came back. Once the creative spirit reaches liftoff, an unexpected indoor rainstorm just isn’t enough to stop it.

Since that first Hack Day, there have been hack days at companies like IBM. GroupMe was born at the Techcrunch Disrupt Hack Day in May 2010, funded three months later, then sold for tens of millions of dollars within a year. Hack Days are being organized by government agencies to help citizens improve government. Just recently, LinkedIn organized a Hack Day to help veterans.
People ask me all the time why Hack Days work so well. The secret of Hack Day is pretty simple: doing something is the only thing that matters at a Hack Day. You can have the best idea in the world, but if you can’t put some meat on it, no one cares. When I organized the first one, Yahoo! had something internally called “Idea Factory,” a sophisticated online suggestion box to capture ideas from people around the company. Capturing ideas in such a way sounds perfectly innocuous, but such a system has a key ideological flaw: it anticipates that someone else is going to take your idea and and do something with it, relieving you of all responsibility (except, as I learned, complaining that no one had used your awesome idea yet). Hack Day solves that problem. You’re responsible for idea and execution, and your two minutes better have a demo or you’re toast.

Hack Days separate doers from talkers. In the communications around that first Hack Day, I had thrown in a “no PowerPoint” directive to protect the event from the pernicious scourge of corporate slide decks, and that became a rallying cry. There simply was no place for the dull corporate drone of bland PowerPoints. Occasionally, someone would try to present a PowerPoint without a prototype. Without fail, that person would be roundly booed and ruthlessly cut off if he didn’t step away willingly. The cultural norms of Hack Day simply did not allow for vacuous grandstanding. Stop talking and show me what you built. We’ve only got two minutes.

In this book, you will learn about many ways different organizations have tried to innovate, and you can bet they all share this: they trust that people will do awesome things when given room to do it, and they take great pains to create that room.

Happy hacking, and remember that you don’t really have to answer the phone when HR calls.