Scaling startups

This blog post was also syndicated to Fast Company.

The Scaling Startups panel I was on last week at Supernova generated a little coverage, but I wanted to go into a lot more detail than what I saw there, so it seems like a good time to jump back into blogging full-force. The panel abstract read:

How can startups develop a distinctive culture, and sustain it as they grow? Are the characteristics that define today’s high potential technology startups the same as five or ten years ago? And does the current environment of low-cost infrastructure and open information sharing give new companies an advantage, or make it harder for them to maintain a competitive edge and reach critical mass?

Coming from the engineering side, I’m very much a pragmatist about all of this since most of my time is spent on shipping products and the systems to support shipping products. Before Etsy, I had done the “startup within the big company” thing and the Web 1.0 startup dot-com IPO thing, both of which have influenced my thinking on the subject considerably. In the panel, I focused mainly on what it takes to institutionalize a distinctive culture focused on shipping and how to sustain that when you’re growing fast. What you read below is based on the notes I made before the panel along with what I remember from the panel itself. I tend to take bad notes during panels because I’m focused on participating, so I don’t have much about the remarks of my co-panelists, who were great (Ethan Mollick of Wharton, our moderator, Steve Cohen of Morgan Lewis, Steve Barsh of PackLate, and Andreas Weigend, advisor to a number of companies and former Chief Scientist at Amazon). A video of the session will be available soon, though, and Jonny Goldstein did an incredible job drawing a visualization of the talk in real time! (see photo on the right)

At Etsy, we’ve quadrupled our engineering team in the past year, so everything I write below is based on that experience. Much of what I write about below was put in place in the last six months, and I think it’s working.

First, to scale a startup, you should always remind yourself why people work at startups in the first place:

• they like to move fast
• they want to make impactful decisions
• they want to DO things and BUILD things (not sit in meetings awash in PowerPoint)
• they want to be able to take risks

For various reasons, startups and the people who inhabit them as they grow tend to become more risk-averse over time. There’s a great quote from Peter Drucker about risks — laminate this quote, frame it at your desk, make it your screensaver, and put a copy of it in your wallet:

People who don’t take risks generally make about two big mistakes a year. People who do take risks generally make about two big mistakes a year.

In talking about culture earlier, I use the word “institutionalize” very deliberately. You have to actively develop a culture of speed and informed risk-taking and put real mechanisms in place to continually reinforce that culture, especially when you’re growing quickly. Pontificating about wanting to move fast isn’t enough and is often just counter-productive. You have to actually move fast.

You have to have the right amount of process, too, but not too much. In most companies, process can be a little like a trash heap that only grows larger and larger over time. You have to be vigilant about keeping that heap from growing. One of my favorite quotes on process comes from Clay Shirky:

Process is an embedded reaction to prior stupidity.

Process is sometimes put in place because one person did one stupid thing once, sometimes literally YEARS ago. What about the other 999 times that nothing went wrong and progress was made while ignoring the process? Leaders should question process at every turn (though tend to your “good” processes, which are important. The larger your team gets, the more you need some clear rules of engagement or folks just bump into each other.) I recall when Fred Brooks spoke at Etsy, he described some of his career experiences and punctuated one by saying (I paraphrase):

If you follow process religiously, you’ll never get anything done!

(I implicitly trust Fred Brooks.)

To scale as you grow, you have to put processes and structure in place that encourage everyone in your company to take more risks (not fewer), keep the right people informed of the risks you are taking, and protect individuals from silly mistakes that would tempt your company to institute unnecessary embedded processes. Just as many mistakes are committed from people being tentative than being aggressive anyway (the point of Peter Drucker’s quote).

So, how do you do it? Here’s what we do to institutionalize speed and risk-taking at Etsy (your mileage may vary):

Hire well

This goes without saying, and I didn’t mention it in the panel. It’s a big topic probably best left for another post. Hiring great people makes everything else below easier.

Communication

Everyone in the company uses IRC, not just engineers. Everyone, all the time, from the CEO on down. Sure, sometimes you can miss things if you’re not in IRC at the time, but the benefits far outweigh the costs, and you have a lot fewer meetings about day-to-day mundane issues.

Deploy code early and continuously

At Etsy, most new engineers deploy code before they are done with their health insurance paperwork. We practice roughly what has become known as “continuous deployment.” This means one-button deploy, period. We’ve invested in the tooling to make this possible, so it doesn’t come for free. Engineers don’t have to wait to ride on a 2-week release: you can do it now. This is all supported by thousands of real-time monitoring checks, business metrics, and automated tests, so it actually requires MORE discipline and ongoing attention than many shops who have multi-week (or month – yikes!) release cycles. A lot of people say, “that sounds risky!” Well, not really. The team at Etsy once reported a serious XSS bug to a site (which will remain nameless) and they said they were going to deploy a fix — in six weeks, because they only deploy every two months. Talk about risky! We did 204 production deployments in July, about 30% more than June, and we are ready to deploy at all times. Deploy, deploy, deploy. (Doing this requires close dev/ops cooperation. The canonical presentation on that subject is 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr by John Allspaw and Paul Hammond. John has been at Etsy since January.)

Oh, and remember IRC? We have IRC bots that automatically announce code deploys in real-time. Those messages look like this:

PRODUCTION deployed by cdickerson build: 32445-trunk-20100804-033748-UTC took: 17 seconds diff: http://awesomeness.internal.etsy.com/diff/web/32441/32445

Since everyone is in IRC, everyone knows who deployed, when we deployed, and what is being deployed (this also goes out in email to the whole engineering team). This level of transparency is what distinguishes how we operate from “cowboy coding,” which often works in early-stage startups but becomes destructive and distracting as a team grows.

Encourage experimentation

With minimal review (“minimal” meaning making sure key privacy, legal, and other issues are taken into consideration), any engineer at Etsy can push a test application out to our members to get feedback, and it could totally fail. We announced this to the community with all appropriate caveats. Allowing experiments eliminates the “no one will let me do my idea” problem that starts to break down startup culture. This is all about reducing the “idea approval index.” We’re just getting started on this one and I’m really excited about it.

Hack Days

One of our teams did a hack day last week where the engineers on the team went off-site and defined and built a single feature and released it on the same day. Always remember: “Every good work of software starts by scratching a developer’s personal itch.” It almost doesn’t matter what the feature is: it’s the cultural idea that you can JUST DO IT that matters most. The Hack Days we’re doing at Etsy now have one key distinction from the ones I organized at Yahoo (though they share the same inspirations) — you build and ship the same day. Our continual investment in tools and culture makes this possible.

Build a culture that loves engineers

I see folks mess this up all the time, and unfortunately it is a difficult thing to teach since it’s so intuitive. If you’re building a technology-driven company, you better have a culture that loves engineers, and I mean love. All too often, I see entrepreneurs who say they “just need engineers” to “bang out the code” for this great idea of theirs. If you view engineers as interchangeable factory workers instead of partners and creative people, you’re in for a tough time getting huge in a world driven by technology. To scale, you need to tend to engineering culture in your company and build a company where engineers love to work. The headlines about tech companies are always about the comings and goings of CEOs, but the rise and fall of tech companies is usually correlated to where the great engineers are going (of course, great CEOs know this and draw great engineers). I tried to express our view of engineering at Etsy on the about page of our engineering blog, Code as Craft (more on that below). Engineering is creative work, and often joyous work at that. No one describes the inherent joy of building software more eloquently than Fred Brooks when he answers the question “why is programming fun?” Recognizing and encouraging that sense of joy is core to any great engineering culture.

External transparency

We talk about all of this stuff in our engineering blog, Code as Craft, which makes our interview and hiring process more productive since folks come in knowing how we think and operate, and even some of the people on the team. We also get help and advice from the community with particular challenges like batch resizing 135 million photos.

Embracing failure

Moving fast often means making mistakes, though still not as many as you would think (again, the Peter Drucker quote). In the panel, I mentioned the award that Flickr gives annually “to the individual who breaks flickr.com in the most spectacular way.” Of course, this is a little tongue-in-cheek, but the point about accepting mistakes is not. Spectacular “failure” means you were probably working on something big and had a spectacular learning experience. (Side note: my remarks on the panel were construed to mean that we give such an award at Etsy — as reported at CNET — and I don’t think I actually said that; however, I immediately emailed the staff back at Etsy to start brainstorming a name for such an award at Etsy. Stay tuned.) “Rewarding” failure is not some crazy “those dot-com kids!” kind of thing. Companies like pharmaceutical giant Merck have institutionalized practices to deal with failure:

[R&D head Peter S.] Kim is promising stock options to scientists who bail out on losing projects. It’s not the loss per se that’s being rewarded but the decision to accept failure and move on.

Of course, catastrophic mistakes are not to be celebrated, but those sorts of mistakes are exceedingly rare and you should have processes in place to deal with them (including a defined post-mortem process). Most of the tension in companies around mistakes is about fairly mundane issues and is relatively unproductive (again, look at the Drucker quote you have in your wallet).

In the Q&A for the panel, someone asked: “This sounds great for engineering and product — how do you scale everything else at a startup?” While I focused primarily on engineering because it’s what I know, I have a huge appreciation for well-run HR, finance, and other operational functions inside a company because they are critical. They can often be the unsung heroes. You can’t add server capacity quickly if the finance team doesn’t help you move quickly.

To illustrate the point, I asked the crowd how many of them knew who Peter Grant was (only one person did). Peter Grant was Led Zeppelin’s manager and was famously quoted as saying: “Led Zeppelin looks after the music and I do everything else.” As you grow, make sure your startup has Peter Grants to keep the band focused on creating awesome music.