Continuous Deployment and Basketball

Anyone who knows me well knows that I love basketball and I love documentaries, so basketball documentaries are the best. Today, I watched one from ESPN’s excellent 30 for 30 series, “The Guru of Go,” produced by Oscar-winning director Bill Couturié. Here’s the synopsis:

By the mid-1980s Paul Westhead had worn out his welcome in the NBA. The best offer he could find came from an obscure small college with little history of basketball. In the same city where he had won an NBA championship with Magic and Kareem, Westhead was determined to perfect his non-stop run-and-gun offensive system at Loyola Marymount. His shoot-first offense appeared doomed to fail until Hank Gathers and Bo Kimble, two talented players from Westhead’s hometown of Philadelphia, arrived gift-wrapped at his doorstep. With Gathers and Kimble leading a record scoring charge, Westhead’s system suddenly dazzled the world of college basketball and turned conventional thinking on its head. . . .

Wessthead (who was also a Shakespeare scholar in addition to a basketball head coach) built a system that he called simply “The System.” It’s a beautifully-told story (with deep tragedy), and highly recommended (you can buy it on iTunes). What struck me about “The System” that Westhead created was how similar the language could be to how we at Etsy and others talk about our own system of continuous deployment and “The Etsy Way.” In one segment (starting at 17:47 in the iTunes video), the coach and his players talk about how their seemingly chaotic run-and-gun system (disparagingly called “roller derby in shorts” by ESPN announcer Dick Vitale) was actually quite well-thought-out:

A lot of people say, well, that wasn’t basketball, that was like street ball. Well, it might be street ball to you, but to us, it was orchestrated. (Paul Westhead, coach)

It looked like absolute chaos (Tom Peabody, player)

. . . but it’s very much a structured way to play. (Bo Kimball, player)

The “some people think it’s crazy but this is actually carefully thought out” attitude resonated with me, and reminded of discussions I’ve had about the structured chaos of Etsy’s deployment process, which I detailed in my Optimizing for Developer Happiness talk from Railsconf last year (see the bit about the “push train” starting at 17:58 in this video).

Like I’ve learned at Etsy, talent and culture mattered most in “The System,” too. Without that a great system doesn’t work — see this from the director’s personal statement about the film:
Paul’s system was magic (and big fun to watch) – if you had the right players. Without them, the system looked like crap.

Since Westhead is also a Shakespeare scholar, Shakespeare quotes are sprinkled throughout the film. At one point, Westhead says, “People ask me my favorite Shakespeare quote. It’s a simple one-liner about basketball and maybe your life, from Hamlet: ‘The Readiness is All.’” In Westhead’s system, players were expected to push the ball down the court and be ready to shoot quickly. Readiness is a key aspect of continuous deployment, too. In slide 43 of my talk at SXSW about Continuous Deployment at Etsy, I quote Etsy engineer Lacy Rhoades: “Not being in a state to deploy is a matter of liability. It’s like having the only fire exit blocked. You ignore it at everyone’s peril.” Yes, readiness is all.

When I see connections between basketball, leadership, technology, and Shakespeare, I just can’t resist writing them down. Thanks for indulging me.

Here’s a clip from the film:

The 20 Percent Doctrine

In March of 2010, I got an email from Ryan Tate saying he was writing a book about skunkworks and experimental projects inside large companies, and he wanted to talk to me about Yahoo! Hack Day. Over the next year and a half, I did periodic interviews with Ryan. Ryan eventually asked me to write the foreword for the book. By then it had a title: The 20% Doctrine: How Tinkering, Goofing Off, and Breaking the Rules at Work Drive Success in Business. I loved the topic so much, I agreed to write it during the holiday season (a busy time at Etsy!) I’m happy with how it turned out.20% Doctrine

The book shipped yesterday, and Ryan’s chapter on Hack Day captures the beautifully chaotic spirit of Hack Day better than anything else I’ve read. (The big surprise when I read the chapter for the first time yesterday was the description of me as a “laid-back, freedom-loving, rabble-rousing Yahoo programmer, a sort of cross between Bill Gates and cult movie character Jeff ‘The Dude’ Lebowski.” I’m not that smart or that cool but it definitely made me laugh!)

Below is a copy of the foreword — a big thanks to Ryan for letting me write it, and to all the Yahoo! hackers and leaders who helped make it all possible. Years later, I am still inspired by the spirit we captured with Yahoo! Hack Day, and feel really honored that I played a role in it. I hope you enjoy the foreword, and the whole book.


On December 12, 2005, the phone at my desk rang. It was the Yahoo! HR department. The phone at my desk almost never rang, so this was a bit of a surprise. I had just helped put together something really awesome, though, so I smiled and braced myself for a hearty congratulations.

The Friday before, I had organized the first internal Hack Day at Yahoo! with the help of a loosely-organized band of people around the company. The “hack” designation for the day was a tip of the hat to hacker culture, but also a nod to the fact that we were trying to fix a system that didn’t work particularly well. The idea was really simple: all the engineers in our division were given the day off to build anything they wanted to build. The only rules were to build something in 24 hours and then show it at the end of the period. The basic structure of the event itself was inspired by what we had seen at small startups, but no one had attempted such an event at a large scale at an established company.

The first Yahoo! Hack Day was clearly a success. In a company that was struggling to innovate, about seventy prototypes appeared out of nowhere in a single 24-hour period and they were presented in a joyfully enthusiastic environment where people whooped and yelled and cheered. Sleep-deprived, t-shirt-clad developers stayed late at work on a Friday night to show prototypes they had built for no other reason than they wanted to build something. In his seminal book about open source software, The Cathedral and the Bazaar, Eric Raymond wrote: “Every good work of software starts by scratching a developer’s personal itch.” There clearly had been a lot of developer itching around Yahoo! but it took Hack Day to let them issue a collective cathartic scratch.

But back to that call from HR. I grabbed the phone, prepared to be gracious, then the HR person on the other end told me we needed to take down one of the hacks, a hack by Cal Henderson that created an API to our company directory (available on Yahoo’s intranet, known as “Backyard”) and built a hot-or-not-style “Backyard War” app on top of it. No one was ever really sure what these wars were about, but it was viscerally fun to place random co-workers in battle with each other over unknown stakes. The impromptu judging committee I had put together had given Cal a trophy for his work.

The HR person who called me had made a critical error in reasoning. While I had organized Hack Day, I by no means had any actual control over the event itself or any of the participants. I had designed it that way. There were no sign-ups in advance, no proscribed projects or areas of focus, and no central servers where the projects lived. I couldn’t have taken Backyard War down if I had a gun to my head. In the end, I think HR may have eventually gotten to Cal, but it didn’t matter. At future Hack Days, there was always a feeling of danger and although no one ever really said it, there was an ongoing secret competition to see who would get the call from HR this time. When you’re trying to get things done and change a system, expect to upset a few people along the way.

With that first event, the die was cast and the completely improvised format from that first Hack Day became somewhat of a standard. At that first Hack Day, I didn’t expect seventy presentations (remember, I had no idea who was presenting until they got started). I had originally planned to give each team 5-10 minutes to present, but with 70 hacks, I called an audible — each presentation would be two minutes or less. Years later, I found myself in an elevator at the site of a hack day and heard one hacker explaining the rules to another, “Dude, demos are always two minutes. It’s a rule.” I chuckled to myself that these “rules” had become so solidified over time.

After that, we did Hack Days all over the world, on three continents. We continued doing internal ones for employees and did our first open one for the public nine months after the very first internal one. The basic “rules” remained. People built huge numbers of prototypes to solve a wide range of problems, and the only thing that really changed from place to place was the food. We had pizza in California, samosas in Bangalore, and bad English pizza in London.

In London, we did a joint hack day with the BBC and held it at Alexandra Palace. Just after the event started, lightning struck and the power went out, triggering a fire suppression system that opened up large sections of the roof, causing indoor rain. Hackers pulled out umbrellas and simply started drawing on paper until the power came back. Once the creative spirit reaches liftoff, an unexpected indoor rainstorm just isn’t enough to stop it.

Since that first Hack Day, there have been hack days at companies like IBM. GroupMe was born at the Techcrunch Disrupt Hack Day in May 2010, funded three months later, then sold for tens of millions of dollars within a year. Hack Days are being organized by government agencies to help citizens improve government. Just recently, LinkedIn organized a Hack Day to help veterans.
People ask me all the time why Hack Days work so well. The secret of Hack Day is pretty simple: doing something is the only thing that matters at a Hack Day. You can have the best idea in the world, but if you can’t put some meat on it, no one cares. When I organized the first one, Yahoo! had something internally called “Idea Factory,” a sophisticated online suggestion box to capture ideas from people around the company. Capturing ideas in such a way sounds perfectly innocuous, but such a system has a key ideological flaw: it anticipates that someone else is going to take your idea and and do something with it, relieving you of all responsibility (except, as I learned, complaining that no one had used your awesome idea yet). Hack Day solves that problem. You’re responsible for idea and execution, and your two minutes better have a demo or you’re toast.

Hack Days separate doers from talkers. In the communications around that first Hack Day, I had thrown in a “no PowerPoint” directive to protect the event from the pernicious scourge of corporate slide decks, and that became a rallying cry. There simply was no place for the dull corporate drone of bland PowerPoints. Occasionally, someone would try to present a PowerPoint without a prototype. Without fail, that person would be roundly booed and ruthlessly cut off if he didn’t step away willingly. The cultural norms of Hack Day simply did not allow for vacuous grandstanding. Stop talking and show me what you built. We’ve only got two minutes.

In this book, you will learn about many ways different organizations have tried to innovate, and you can bet they all share this: they trust that people will do awesome things when given room to do it, and they take great pains to create that room.

Happy hacking, and remember that you don’t really have to answer the phone when HR calls.

My field notes

I spend quite a lot of time reading and linking disparate ideas together in the general course of my work, often when I’m trying to put something together — a blog post, a presentation, or an email about something I’m working on. Not all of the material is available on the Internet, so I have to manually input or scan quotes. I do my best to go to the best source for whatever I’m working on, not just the easiest one to get digitally. In the course of putting my “optimizing for developer happiness” keynote at Railsconf, for example, I:

I’m always looking for new ways to share the information I pull together, and I’ve tried all sorts of approaches over the years: Moleskine notebooks, Devonthink (one of Steven Johnson’s favorites), Evernote, assiduous note-taking on a Kindle (if you do the same, be sure to read this from Fred), and pinboard, just to name a few. Most of that ends up in my own private archives, and even when you share something via a service like pinboard, you’re assuming there’s a URL to link to (which there wasn’t in the case of Drucker’s Concept of the Corporation, which blew my mind in how prescient it was.)

I use Twitter a lot, but don’t find it very useful as a place to keep snippets while I’m working on something. On my blog, I’m a little old-fashioned and mostly like to write fully-formed pieces when I do write, so it’s more of a place for introducing relatively complete ideas. I felt like I could use something in the middle to keep the dribs and drabs along the way and get some (hopefully) interesting things on the Internet in the process, so I’m giving Tumblr a try.

I’m calling my tumblog my “field notes.” If you look at the wikipedia page for “fieldnotes” (one word), you’ll see one definition of the term:

Emerson (1995) defines fieldnotes in ethnography (a term referring generally to descriptive writing in anthropology, and also to subfield of sociology) as ‘accounts describing experiences and observations the researcher has made while participating in an intense and involved manner’.

That describes the spirit pretty well. So, we’ll see how this goes (some music might also slip in there now and then). Follow me if you’re interested. (here’s an RSS feed)

Finding your courage

At Hello Etsy in Berlin a few weeks ago, I gave a talk titled “Finding Your Courage.” It was a different subject than what I’m accustomed to talking about and is probably the most personal talk I’ve given, with little glimpses into my background that I don’t talk about much. I really enjoyed giving it, and I hope those of you who watch like it, too (there’s a baby photo of me in there, as you can see in the photo below).

One of the coolest things about the process of putting the talk together was how the Etsy community helped me along the way by telling their stories in the Etsy forums. The whole process was really satisfying all-around, and I thank the community for their help.

Here’s a link to the video from Livestream (and the slides are here.)

Chad Dickerson, HELLO ETSY Berlin, September 17 & 18, 2011

Making things, infinite games, and Fred Brooks

Caterina’s post “Make Things” is so awesome and spot-on. You should read it. Here’s a taste:

There are so many conferences these days, so many voluble, charismatic leaders, and so much noise. I talk to a lot of entrepreneurs in their 20s who are knowledgeable about the valuations various Y Combinator startups have attained, know the names of all the angel investors in the Valley, have in-depth knowledge of the Facebook diaspora and their doings, have opinions on various Zynga acquisitions, and know exactly how to get Andrew Mason on the line…it boggles the mind. These are good things to have in your tool kit. But I want to hear about things out there that they love. About loving the thing they’re building. There’s less of that.

I think part of what Caterina is saying is that pure focus on networking and IPOs and fundraising and valuations and acquisitions neglects the core sense of play that got most of us into this world in the first place.

This is something I was thinking about earlier this week, when I was talking with a team at Etsy about our mission and values and the importance of a sense of play came up. On that theme, I suggested that the folks in the room read James Carse’s Finite and Infinite Games: A Vision of Life as Play and Possibility (a book Caterina recommended to me years ago). I won’t summarize it here (that’s your second reading assignment of this post!) but you can cheat a little with Wikipedia:

In short, a finite game is played with the purpose of winning (thus ending the game), while an infinite game is played with the purpose of continuing the play.

The inside cover of the book gives you a little more flavor:

The rules of the finite game may not change; the rules of an infinite game must change.

Finite players play within boundaries; infinite players play with boundaries.

Finite players are serious; infinite games are playful.

A finite player plays to be powerful; an infinite player plays with strength.

A finite player consumes time; an infinite player generates time.

The finite player aims for eternal life; the infinite player aims for eternal birth.

(I see Etsy and the dynamics of the community that drive it as having strong characteristics of Carse’s “infinite game.” Running any good company is like an infinite game, which is why this Onion story is funny: “Corporation Reaches Goal, Shuts Down.” This may be the subject of another post.)

When I think about what motivates me and attracts me to companies and people in my life, it’s always been the same sense of joy in creation that Fred Brooks described so eloquently in The Mythical Man-Month, the joy that Caterina wants to hear from more entrepreneurs. Brooks’ book is primarily known for identifying and explaining the basic fallacies in project planning that give the book its name and a law named after Brooks. It’s rarely (if ever) credited as such, but I think it’s really a paean to creativity and the inherent problems encountered when trying to break what is essentially creative work and creative people into interchangeable functions and parts. It makes it all the more powerful that he wrote the excerpt below in 1974 (and led with it on page 7), long before most regular folks had even thought about computers, much less that making them work was even remotely creative (note the pinball machine and jukebox references!):

Why is programming fun? What delights may its practitioner expect as his reward?

First is the sheer joy of making things. As the child delights in his mud pie, so the adult enjoys building things, especially things of his own design. I think this delight must be an image of God’s delight in making things, a delight shown in the distinctness and newness of each leaf and each snowflake.

Second is the pleasure of making things that are useful to other people. Deep within, we want others to use our work and to find it helpful. In this respect the programming system is not essentially different from the child’s first clay pencil holder “for Daddy’s office.”

Third is the fascination of fashioning complex puzzle-like objects of interlocking moving parts and watching them work in subtle cycles, playing out the consequences of principles built in from the beginning. The programmed computer has all the fascination of the pinball machine or the jukebox mechanism, carried to the ultimate.

Fourth is the joy of always learning, which springs from the nonrepeating nature of the task. In one way or another the problem is ever new, and its solver learns something: sometimes practical, sometimes theoretical, and sometimes both.

Finally, there is the delight of working in such a tractable medium. The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures…

Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separately from the construct itself. It prints results, draws pictures, produces sounds, moves arms. The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

Programming then is fun because it gratifies creative longings built deep within us and delights sensibilities we have in common with all men.

I, along with Anil, am optimistic that we can make this mindset the default.

Keynote at Railsconf

I did a keynote at Railsconf yesterday entitled “Optimizing for developer happiness.” Huge thanks to Ben Scofield and Chad Fowler for the invite. It was a blast!

Below is the video and here are the slides on Slideshare.

I’ve got a longer post in me that builds on the themes in the talk — hoping to get that up in the next couple of weeks!

Update: just found a talk called “Optimize for Happiness” by Tom Preston-Werner (Github co-founder) about optimizing for happiness vs. money. Tom’s talk definitely pre-dated mine and looks at happiness from a somewhat different point-of-view. Definitely worth reading/watching.

The scourge of PowerPoint

Anyone who worked with me and others on putting the early hack days together at Yahoo! knows that one of the rallying cries was “No PowerPoint!” I’m pretty sure that the first invitation sent around inside Yahoo! back in 2005 said that explicitly, and presenters who started out with PowerPoints at those early hack days were enthusiastically booed. This theme continue to be reflected in future hack days, like the one we put together with Techcrunch just last year. The “No PowerPoint!” stance was a reflection of what I had seen or heard about in a number companies (not just Yahoo!) — seemingly endless twiddling with slide decks, with a disproportionate amount of energy devoted to aligning squares and choosing clip art. At its most pernicious, entire teams become obsessed with “the deck” and lose all sense about what they are actually trying to accomplish.

(And don’t get me wrong, I really love artfully-done PowerPoint or Keynote presentations. It’s absolutely possible and the best slide decks inspire and motivate.)

So, I particularly enjoyed this bit in Nokia CEO Stephen Elop’s remarkable “burning platform” memo:

At the lower-end price range, Chinese OEMs are cranking out a device much faster than, as one Nokia employee said only partially in jest, “the time that it takes us to polish a PowerPoint presentation.” They are fast, they are cheap, and they are challenging us.

“Only partially in jest.” Ouch.

(Thanks to @finitor for his tweet!)