Author Archives: Jason Evanish

I'm a coach and consultant for early stage SaaS startups. I help VPs of Product, Heads of Product, and 1st PMs with their toughest challenges like churn, team turnover, stakeholder challenges, issues with consistently shipping great features consistently, and more. I also started Lighthouse, which helps you learn to be a better manager through our blog, podcast, and training. When not working, I'm an avid reader, athlete, and enjoy lake life living in Austin, Texas.

Lessons Learned in Customer Development

18 January 2012

Note: This post originally appeared on GreenhornConnect.com on September 9, 2010. I’m organizing all my customer development posts from GHC on here for easy reference (see the Lean Startups tab).

“Customers live outside the building.” Every startup is well served to remember this and make sure they’re reaching out to their customers/users to understand them.  As customer development manager at oneforty, I’m on the front lines of that effort and our overall goal of implementing Lean Startups methodologies.  I’d like to share a few things I’ve learned along the way.

Lessons Learned in Customer Development

1) Ask the right questions the right way.

One of the key tools in customer development is the user interview.  At its heart, you’re trying to understand what problems your users have and how your startup may solve some of them.  The greatest challenge in these interviews is keeping the discussion focused on user problems and frustrations instead of features and their ideas for solutions; users are notoriously bad at suggesting features they need or would actually use, so you absolutely need to focus on what problems are behind those feature requests.

There are some great contributions in the blogosphere to help guide you in preparing your interview questions:

2) You manage what you measure.

In addition to interviews, metrics are the other big piece of customer development. You have to understand what’s happening on your site and decide what aspects you will work to improve and in what order.

The best way we’ve found to track this is to consider everything under Dave McClure’s AARRR model.  There are all kinds of numbers we can measure on oneforty, but by grouping them into categories of Acquisition, Activation, Retention, Referral and Revenue, a clearer picture of the meaning of all the numbers is formed.  We then take those numbers and focus our engineering sprints on improvements in one of those categories at a time.  This helps engineering understand what they’re building while also making it easier for us to measure for improvements based on those efforts.

3) Communication is key…inside and out!

While Jeremy and I dive into the metrics on oneforty each week in great detail and I interview many users to understand their problems, that’s just the beginning of the work for customer development.  It is essential that what is discovered is shared with the team in a concise and clear fashion; they need the information to act on what we’re seeing and understand the reasoning behind development decisions.  At oneforty, we go over key metrics results with the entire team for 5-10 minutes each week and have a monitor on the wall in the office that displays 3 core metrics we’re focused on right now.

In addition to communicating within the team, it’s important to also communicate outside.  First, you always want to be tweaking your interview questions based on how your site or product is changing; if you’re considering adding a new feature it is important to understand if it solves a significant problem or is particularly compelling for users before you devote significant engineering time.  The best way to do this is through full interviews, but with tools like KissInsights and SurveyMonkey available, you have easy ways to ask large groups of people questions as well.

The other form of outward communication that’s important is with others doing customer development.  Often, it is hard to tell what is a good number for a metric and if you don’t know, you may be trying to improve something that’s already well optimized.

Recently, I discovered two key figures from asking others:

  • A 10% response rate on email requests to do a customer development interview is actually good.
  • For email newsletters, a 30% open rate and a 20-30% click through are solid and standard across most industries.

To learn these numbers I didn’t do anything special; I simply talked to one of Laura’s mentors, Tweeted a question, tried AskDart and asked some friends.  It didn’t take very long, but saved me a lot of time in the long run.

4) Pattern recognition is your most important skill

In the end, I think customer development is all about recognizing patterns.  You don’t memorize everything a user says in an interview; you’re merely looking for commonalities across many interviews. You should be asking yourself, “Are similar users mentioning the same frustrations or questions?”   These patterns are how you identify your “earlyvangelists” and identify the most important issues to work on.

You can also look at metrics the same way.  The goal is not to have to measure every single activity on your site. Rather, you should be searching for a few specific activities that can lead to the needle moving on many things on your site.

—

Customer development is an essential part of an early startup’s life as they search for product market fit. It’s a challenging job, but fortunately there are tons of great resources out there and an open community sharing their knowledge.  If you’re looking to learn more, search for content from Steve Blank and Eric Ries, look for the hashtag #LeanStartups on Twitter and look for Meetups around you on this topic.

What have you learned by doing your startup’s customer development?

5 Lessons Learned in a Lean Startup

18 January 2012

1 Comment

Note: This post originally appeared on GreenhornConnect.com on February 23, 2011. I’m organizing all my customer development posts from GHC on here for easy reference (see the Lean Startups tab).

In honor of both the Lean Startup Circle Meetup on Thursday and the Lean Startup Machine coming this weekend, I’d like to share a few lessons I’ve learned in the past year as I’ve served as Customer Development Manager at oneforty and been actively learning the Lean Startups methodology.

5 Lessons Learned in a Lean Startup

1) Don’t ask what people would pay for. They lie.

Yes that’s right. Even Honest Abe wouldn’t have told you what he’d really pay for if you showed him a web app and asked what he’d pay for on top of what you showed.

Customers are terrible at explaining their problems and understanding the root issue. As the saying goes, the one certainty for a patient seeing a psychiatrist is that what the patient says is cause of their problems is never the actual problem.

Even I have lied. I was talking with Chris Keller about his awesome tool Followup.cc, which is an email reminder system, and I said, “I would pay to be able export all of my reminders to my Google calendar.”  Chris wisely also looked at engagement and based much of his pricing system on number of reminders, but he also put the calendar export in his paid version.  I am now a paying customer of Followup.cc but have yet to actually use the Google calendar system.

Did I intend to lie? Of course not. But it just goes to show the mindset you can get in and how far from the truth it can be. Don’t ask people what they’d pay for.

2) Nothing beats getting a customer to actually pay for something.

There is no better validation for your business than getting a user to actually pay for something.  Despite the true value of your time, few people actually account for this. That means they are actually quite likely to be willing to use your free product, while having no intention of ever paying for it.

Paying also moves you beyond the realm of being a favor; friends, acquaintances or just nice guys, no one will pay for your product unless it really solves a pain or strongly interests them.

As an added bonus, once someone pays for something, they have expectations. That means that their feedback will be stronger, because they gave you their money and now expect that you will deliver on what they hoped they were buying. This feedback is priceless, as building something that satisfies them can be built into the repeatable process that goes into a sales funnel.

3) You have to learn the customer’s language.

You may call your product anything you want but if it and the language you use to describe it doesn’t resonate with your customer, you’re unlikely to move forward.  You need to understand your customer’s language and make your product speak that language.

The best way to do this is to look at what words your customer uses to search for the problem you’re solving and, of course, the customer development interview. Remember, your customers should be doing 70-80% of the talking in your interviews.

4) A customer’s stating a problem is more valuable than a customer agreeing with a problem you present.

One of the key tenants of Lean Startups is that you’re solving a customer’s pain.  Often the question is if what you are building is a so called “Vitamin,” which is nice to have, or you’ve created a “Pain Pill,” which they definitely need.

One indicator of which side of the Vitamin/Pain Pill coin your product is on is how the problem that your product solves is surfaced. If in the middle of your interview (before you pitch your solution) the customer talks specifically about the problem you’re trying to solve, you have a much stronger indicator of pain than if you ask them if they are experiencing pain in the area you believe is a problem.

Now, this does not mean that someone who doesn’t come right out and say your problem isn’t a potentially valuable customer. However, when you’re looking for those early adopters (aka- earlyvangelists) you should be thinking about who most desperately needs your solution and that is likely people who have the problem you’re trying to solve on top of mind.

5) Be precise in your Hypotheses.

So you think you have a product solution that is the greatest thing since sliced bread? Great. But who is it for and how does it solve a key pain for them?

One of the risks in creating your lean startup is that you forget to get specific on your hypothesis. Don’t say “this is for doctors”; say “this is for pediatricians in suburban environments that have small, private practices.”

The reason you need to be specific is to avoid false indicators. If you don’t make your hypothesis specific, than you are very likely to talk to a wide range of people. When talking to this wide range, you may not find the interest you would otherwise if you specifically went after a specific group.  You could end up talking to a bunch of chiropractors and surgeons and never realize that the reason there was no interest in your product wasn’t that it’s not a great idea solving a problem; no one was interested because you weren’t talking to the right people.

—

There’s a seemingly endless amount to learn about Lean Startups, so no matter what stage you’re at just remember: Stick with it, Be patient and Don’t be afraid to ask for help.

There’s a great community on Twitter (look for hashtags #LeanStartups and #CustDev), many awesome blogs (Eric Ries, Ash Maruya, Dan Martell, David Cancel and many others) and of course great events (Lean Startups Circle) to help.

Don’t build that feature! How to use the 5 Why’s to learn what your customer really is saying

6 January 2012

1 Comment

Are you sure we should build that?

Whether you interview customers regularly, watch support requests on HelpScout or use a feedback app like UserVoice, your company is inundated with feature requests from your customers.  

It’s great to talk to your customers, but can lead to the dreaded “feature creep” that creates bloated, unusable products.

What’s a product manager to do?

The answer may surprise you in its simplicity: use the 5 Why’s.

Don’t build that feature! How to use the 5 Why’s to learn what your customer really wants

In case you’re not familiar, the concept of 5 Why’s is that by asking a person a series of “why” questions, you can get to the real root cause of an issue.

Commonly, the issue can be used to do analysis of technical failures as outlined in a great post by Eric Ries found here. 

However, this can be just as powerful when you apply it to discussions with customers about features.  The power comes from saving you from building things customers actually don’t want or need and surfacing what is really important.

This concept really crystallized for me when I had an experience talking to Chris Keller, the founder of email reminder tool, Followup.cc. I was an active user, but Chris hadn’t decided how to monetize yet.

I told him I’d pay for Followup if he provided a calendar to see all my Followup reminders.

There’s just one problem – despite the fact that Chris built that feature not long after our discussion, I’ve never used the calendar even though I was a paying customer because of the core product.

Chris could have avoided building the feature if he had used the 5 Why’s. Here’s how it may have gone.

Me: I’d pay for Followup if you provided a calendar to see all my Followup reminders.

WHY #1: Chris: Why would that make you pay for it?

Me: Because I could then see all the Reminders I’ve set up.

WHY #2: Chris: Why do you want to see all the reminders you’ve set up?

Me: So that I know when I have reminders coming up.

WHY #3: Chris: Why do you want to know when you have reminders coming up?

Me: So I’m sure I know when I should expect a bunch of reminders.

WHY #4: Chris: Why do you need to know when you should expect reminders?

Me: (long pause) …So I’m sure I get all my reminders.

WHY #5: Chris: Why wouldn’t you get all your reminders?

Me: Good question. I guess I’m worried about it not always working.

Chris: Don’t worry. I’ve built Followup to be robust and reliable. Are you sure you need that calendar?

Me: Maybe not.

Your results may vary, but the point is, customers are terrible at coming up with solutions.

What they are good at is sharing their problems, which you can solve.

The key is to interact with them in a methodical way to get to the heart of what they really need. That comes from having live, real conversations with them (not using feature voting tools), and taking the time to really dig into the underlying reasons for their request.

If you do that, you may be surprised what you learned the real problem they have is, and you might just build a better, less-bloated product.

Next time you hear from a customer that you need to build a new feature…ask them the 5 Why’s first!

West Coast Differences – Non Startup Edition

13 December 2011

10 Comments

I just took a trip to the Valley for the first time. I’ve had a lot to say about it from the perspective of an entrepreneur (see on Greenhorn Connect here and at OnStartups here). I also noticed quite a few things that have nothing to do with startups that I found culturally interesting.

1) Everyone is nice

Boston can be a cold place, and no I’m not talking about the weather. In general, you just don’t find people being friendly walking down the street, and you definitely don’t see it on the road.

One event really tipified this for me: I had just made it out of a parking garage before close. Because of this I didn’t have time to set my GPS before hitting the road. There was no where to park so I pulled off blocking a driveway. As I was engrossed in entering my destination address into my Garmin, an SUV started honking at me; they needed in the driveway. I of course complied.

What happened next shocked me. The woman parked her car, got out and walked over to where I had pulled off slightly up the street. When I put down my window, she apologized for honking her horn at me. 

2) Everyone weighs 15 pounds less

It’s hard to believe until you see it. Everyone is just in slightly better shape than I see them in Boston. It’s a visual average I noticed after a few days.

I think the cause of this is pretty simple. Nice weather = more time outdoors = more exercise. What kills us (even a gym rat like me) are these brutal winters. It’s really hard to get enough cardio in under those circumstances which means every winter you’re putting on a little weight. Add that up over a winter or two and you quickly get those 15 pounds.

3) The Valley has safer drivers

I’ll be totally honest: after not driving for 7.5 years, I’m a pretty terrible driver. Luckily, people in the Valley drive slower on the highways (around 65 instead of 80) and well, they aren’t MassHoles. They actually use things like turn signals and let people over when they do signal. It was refreshing and the only reason I got back to Boston in one piece.

4) Parking is a breeze outside SF

There’s easy parking in Mountain View and Palo Alto. Even downtown. And it’s free. I was terrified when I forgot to bring quarters with me on my trip and was pleasantly surprised to find that I didn’t need them.

5) Their public transportation is good, but flawed too

We all have our gripes with the MBTA but it gets you where you need to go….usually. The SF system is the same. Their buses are slightly unreliable, but have some drawbacks: the stops often smell of urine and the back part of every bus is covered in graffiti. Meanwhile, the CalTrain is incredible. Like a well oiled machine, the trains fly through the Valley right on schedule.

The best part of their system is they’ve tied it all together on one master card (their version of the Charlie Card). The Clipper Card, as they call it, was the only thing I needed and was easy to pick up at a station.

6) They have a serious homeless population

I found out while I was there that SF has the largest homeless population in the country and for some ridiculous reason they give each of them $400 a year (as if the year round good weather wasn’t attractive enough for the homeless).  I think this is a likely contributor to the bus stop urine smell.

—

These are just a few of the random differences I noticed in comparing the cities of Boston and SF and the Valley vs. New England.  Have you been both places? What have you noticed as a difference?

Visiting the Valley Next Week

27 November 2011

4 Comments

Late in the summer, I wrote about how I thought startups should be “Tricoastal.” What I meant was that you can benefit greatly from having a presence in Boston, New York City and Silicon Valley.   I’ve made a number of visits to New York City in the past 6 months, but have yet to make a real visit to the Valley. It’s finally time to change that.

December 3rd through 8th I’m going to be visiting Silicon Valley.

In order to make the most of it, I plan to spend a little time in 3 of the major hubs: San Francisco, Palo Alto and Mountain View.

If you’re reading this, I’d love to meet up and/or hear your advice on things I can’t miss during this visit. I really want to get a feel for what it’s like out here, what makes the Valley tick and as many of the places that make the Valley special as possible.  I’m also working on a lean product management tool, so if you know any product people I should meet, let me know!

I’ll be in San Francisco: Saturday evening, 12/3 to midday Monday 12/5

I’ll be in Palo Alto: midday Monday 12/5 to midday Wednesday 12/7

I’ll be in Mountain View: midday Wednesday 12/7 through Thursday 12/8

I have places to crash in Palo Alto and Mountain View, but I’m still looking for a place to crash in SF on Saturday and Sunday nights.  If you’ve got a couch, please let me know!

So, what do I need to check out while in town?

The Lean Product Life Cycle

27 November 2011

6 Comments

See my previous post to understand how I got here. The hypothesis is that product development is messy except for the most disciplined.

After talking to a number of great product people, I have a theory for how great product can be developed while being customer-centric.

The Virtuous Cycle of Lean Product Development

The Product Management Funnel

I think of product management as a funnel. At the top are all of the ideas your team generates. If you’re a lean startup, that’s hopefully driven by customer interviews, website and landing page behaviors and support interactions and of course the occasional green space wild idea.

As product manager, this is the most dangerous step. Once you have more than a few employees, it’s easy to have these ideas overflowing and interfering with the whole product process. Multiple teams I spoke with have been using their project management tools to capture these ideas which leads to a huge mess (one such company had over 4,000 stories in their icebox!) and a major time suck (one vp of engineering is spending an hour + each day managing these ideas).

The problem is project management != product management. The only thing that should be in your project management tool are key bugs to fix and what you’re building now or in the very near future (ie next sprint). Everything should be clearly defined and in a language and structure preferred by your engineers. All the evolution of the product from idea to customer validation to final prioritization should be done outside the project management tool.

The Rest of the Funnel

The Full Product Funnel

Going back to the beginning, you need discipline at the top of the funnel; the best product people I spoke with are requiring a shred of data tied to an idea to make it into their coveted ideas list. They also often expect a disciplined approach to what goes in by having people state, “Feature X will move Metric Y by Amount Z” so it’s clear why the feature needs added. In earlier stages that may be in a “Hypothesis vs. Metric to Invalidate” format instead.

The key of this step is effectively managing the signal to noise ratio; for every 5-10 ideas that come in, you may only make 1 or 2. Even the less disciplined product managers I spoke with have some type of hot-lukewarm-cold system for trying to rank ideas. To avoid the 4,000 stories in your icebox (or anywhere else) you have to be disciplined on dropping stale ideas and focusing on what matters now. When you’re a growing startup, where you were 3 months ago is dramatically different from where you are now so why keep those stories cluttering your system now?

After wrangling the initial feeder of ideas under control, you need to effectively refine the ideas that make it into your system and allow the most important ones to rise to the top of your list. As planning for your next sprint begins, you need to prioritize these ideas and balance with other existing projects, bugs and other demands.

Once you’ve settled on what you’re going to have engineering build next, you need to engage your team on the ideas you want to implement; you may need copy from marketing, stories from support to describe customer complaints, links or screenshots from analytics to show present activity and relevant customer development notes. All of this feeds into the project management tool you have, but now it’s uncluttered and your engineers are much happier and efficient. Unless you’re github or heroku, chances are your engineers don’t understand the customer they’re building for perfectly so all this structure makes it easier for them to see what’s going on and then adapt it to fit the project management tool they’re using to track building.

As an added bonus, there’s transparency for your other employees as they engage in more than just the initial ideation process. It also means sprint planning no longer needs to have all hands at it as they’re already well aware of what’s going on and have had opportunities to contribute as needed to the stories now being generated in the project management tool.

Completing the Virtuous Cycle

We all know this image above so well as the core of what lean startups is all about, but how often is this cycle cleanly implemented? The project management tools out today all do a great job on the build aspect, but what about measure and learn?

The funnel process I described above captures the learn aspect as you engage your team for ideas and validation. All that’s left now is to measure.

After you build something new in your project management tool, your engineers will submit it for some form of approval. Once approved, it universally ends up disappearing into the ether, because the project management tool is built to track what you are building or about to build. After building isn’t part of their process.

To close the lean loop, you need to look back and see if that feature actually moved the needle. This needs to happen within 2-4 weeks of building it; sooner wouldn’t be enough data and longer would be distorted due to other things you’ve likely built by then. This last step of learning should help you refine your instincts and feed back into what you build. This is one of the biggest challenges as Eric discusses his principle of Innovation Accounting in his book.

The amazing thing I discovered in so many of my interviews was how rare it was to do this measure step; how can you improve your accuracy without seeing if you hit the target?

—

The Product – Lean Opportunity

What started as an investigation of lean startup opportunities has broadened to helping teams close the loop in managing a product lifecycle. I believe there is tremendous opportunity to build a platform to aide on both sides of the project management tool and much wider than those adopting lean concepts officially (as we all know, being customer centric is not a new concept).

—

Want to learn more about building customer centric products? Sign up to receive my latest posts & insights below:

 

Using Customer Development on the Customer Development Process

27 November 2011

3 Comments

Earlier in November, I started out on a journey to see if there was an opportunity to build a product that would help companies in being a lean startup. The thought was that with Eric Ries’s book flying off the shelves and the lean startup community exploding, there would be new problems to be solved. Thanks to a nudge from Sim Simeonov and a few consulting opportunities brought my way, I felt like there was definitely something there. Like any startup, once I got outside the building, I learned some harsh realities:

1) Most people only pay lip service to lean.

Painful, but true. In both casual conversations in our community and full blown custdev interviews I found the full spectrum from strict implementers like John Prendergast and Matt Mamet (coincidentally co organizers of the local Lean Startup Circle Boston) to people who claim to be lean and think it has anything to do with how little money they spend (names withheld to protect the innocent).

2) Lean is a hands on activity.

The key to lean really is the human interaction; knowing when to ask an interviewee “tell me more” or deciding what analytics to deep dive to match up against your anecdotal discussions can’t be productized directly. It’s also risky to outsource since it’s so core to the future of your business; no amount of meetings, notes and summaries can compare to being there and doing it yourself. There’s huge opportunities to teach companies how to become lean, but I’m passionate about building a product, not a services business and I’m not sure most companies are ready to invest in lean (see 1).

3) Small startups are more likely to be lean and less likely to need process.

Part of the challenge of the adoption curve of lean is that it has most strongly permeated the zero stage entrepreneurs; they’re the ones with time and motivation to watch the videos, read the books and stay up on all the blogs (which has come a long way in the last year or two since all we had was the Four Steps to the Epiphany to go on).

Unfortunately, those people have little to no budget and because of only having a few people on their team, there isn’t much of a need for tools. Everything can be done ad hoc when you’re that small. The most common behavior when you have 5 or less people is to remove your headphones during the work day and share your findings immediately with everyone on your team.  This is effective until you grow to the point that everyone doesn’t know everything that everyone is doing (usually around 8-15 people).

So all that meant to me was time to pivot:

I’ve known all along that customer development feeds into product management. Fittingly, this is where I did find pain:

  • Managing product while being a growing, lean startup requires significant discipline. Not all companies have product leaders with this strength.
  • Many companies build up product management debt as they grow and fail to adapt to the demands of a team that passes the “take off your headphones and talk” phase.
  • Multiple challenges arise in engaging all members of a company at the right times in the right way.

My next post explains what I’ve found. Read on to see the big picture potential for a lean focused product life cycle.

Continuous Deployment: Possibility or Pipe Dream?

23 November 2011

For those in the Lean Startup world, the utopian version of product development is continuous deployment.  It means every engineer is deploying code multiple times a day, often even on someone’s first day of work.  It’s also exceptionally test driven, reducing risks of bugs taking down the entire system.  While many startups aspire to this, few have succeeded, which is why it was so exciting to have Brett Durrett of IMVU come and speak to the Lean Startup Circle Boston Thursday night.  Brett is VP of Engineering at IMVU, which coincidentally happens to be Lean Startup Guru, Eric Ries’s startup he spent many years helping build before becoming the movement’s biggest evangelist.

{Note: Brett’s presentation was awesome but hasn’t been posted yet. I’m embedding his Lean LA version as a reference until it can be posted.}

Continuous Deployment at Lean LA

View more presentations from Brett Durrett

In a nutshell, continuous deployment breaks down into 3 steps:

  1. Develop a feature
  2. Test it
  3. Deploy it

But why would you do continuous deployment?

The reason for using continuous deployment hits the core of lean startups: more iterations. Whether funded or bootstrapped, there’s a limited amount of time to iterate and nothing speeds iteration like getting new features, site tweaks and updates out faster. Continuous deployment forces you to break down all your features into bite size chunks which can save you building massive features when you can confirm it with much smaller steps. It also minimizes version control issues if no one is working on a long term project based on old code.

It also makes your engineers more efficient. Is it easier to find the problem with freshly deployed code that has 10,000 lines in it or 10 lines? Is it easier to get engineers up to speed on a system that expects them to write a micro feature or build a major piece of the system?  If everyone writes their own test code, you have greater accountability across the entire engineering team (you have to fix what you break) and you don’t have to hire a QA team stuck cleaning up everyone else’s mess.

The Continuous Deployment Process for Engineers at IMVU

After convincing us why it matters, Brett walked us through the process for an IMVU engineer.  Once an engineer has finished building their bite size feature, they walk through all of the following in less than 15 minutes!

1) Engineer runs the test in their sandbox.

To keep from clogging the deployment and testing systems everyone shares, engineers first run some basic PHP tests on their own system to ensure the code is ready.  (Note: they don’t do branches in the repository; the brand in the code instead.)

2) Engineer runs testing on main system

After passing the tests on their computers, they enter the queue for the main testing system for all over IMVU called Buildbot. They have tons of tests so you tag your code based on what parts of the system it affects and what it’s for. This optimizes the right tests to run and which can be avoided. (Running every test would take over 8 hours.)

On average, it takes about 8 minutes to run all the tests needed.  They’ve achieved this speed because they have 40-50 instances running just for testing.  They’ve also discovered that 12 minutes or less is the optimal time to have testing take and keep your engineers happy.

The most common slip at this stage is a missing tag, which means a needed test isn’t run.

3A) If all tests pass, the engineer now deploys the code. 

3B) If any of the tests fail, the engineer reverses their commit.

With the rate the whole team is deploying and testing, their’s no time to have people fixing while they’re in the system. Therefore, regardless of the issue, the engineer will reverse their commit and go back to fixing the problem on their machine and starting the test and deploy process from the beginning.

4) Deployment occurs in pieces

Currently, IMVU (with over 50 Million registered members) has 800 servers in use.  When they deploy new code, it starts out on just 35 servers. This ensures that if something goes wrong, it doesn’t take the entire site down.

5) Testing continues after deployment thanks to the Cluster Immune System

Even after deployment, they’re still testing, just in a different fashion. They’ve developed their own tool called the “Cluster Immune System” which monitors key site (speed, system performance, etc) and customer metrics (revenue, registrations, etc) to make sure there hasn’t been a dramatic change.  Even the best tests won’t notice an engineer accidentally made a blue button on a blue page; the tests will see the button is still there and works, but won’t realize a user can’t see the critical sign up button.

This system runs on those 35 servers they use as a live test bed. If anything goes wrong there, they prevent it from deploying to the rest of the system. If not, it’s deployed system wide to all 800 servers.

An audience member asked about “what if you don’t have massive traffic you can segment to test a new deploy?” Brett said it’s an advantage when you’re bigger, but until then, you may just run at Cluster Immune System to monitor a system-wide deploy.

The best news of all of this?  IMVU plans to open source the Cluster Immune System (CIS) soon.

6) If all CIS testing is passed, deploy to all servers, but continue monitoring

Even after deploying to all servers, they still monitor for anything unexpected. If they see anything alarming, they’ll roll back and remove the feature.

—

This entire process takes only 10-12 minutes.  Only one engineer can be in the Buildbot testing phase at a time, but as soon as you enter deployment (step 4, above) someone else can enter buildbot.

—

This process sounds great, right? But it seems so sophisticated…how do you get started? Brett covered that too..

Getting Started – How do you actually do this?!?

Getting started is a different process depending on if you’re an established company or just an infant startup with limited traffic, but either way, there’s great ways to get started:

  1. If you’re a small startup – Start with a sandbox for each of your engineers and just focus on pushing code quickly and in small chunks.  You can develop your testing as things break; that’s how IMVU built their system.
  2. If you’re an established company – Start with production (ie- the last step before deploying code) and automate that process. Start building tests for whether something should be deployed or not.  Work to get the automated tests as good as the human part of the process. Once you’ve accomplished this, keep working backward to continue to remove humans from the deployment flow.  At first, you should err on the side of preventing problems then clean up your tests to be efficient.

Whether big or small, the same key rule applies: Anything can break once. Then you have to make it so the same thing can’t happen again by writing a test for your mistake. This builds both accountability and builds only the tests you really need…one step at a time.

—

Pitfalls

Like any system, this isn’t perfect. There are challenges both with getting personnel buy in and in scaling this:

1) Philosophy- Blameless systems

It’s not “Ned broke this!” it’s “How did we let that fail get through our system.”  Brett really emphasized this important difference. It’s a philosophical buy in required to really make this work best.

At IMVU, they hold regular “Blameless Post Mortems” to discuss issues that slipped through.

2) Optimize your testing

As you grow, more tests will be required. You should optimize for which tests actually need to be run for a specific line of code (as they did with tagging) and purchase sufficient hardward to make tests fast. IMVU also found they could save a tremendous amount of time simply by optimizing the order the tests run, by running the slower ones first (a 22% time saving).  They also then built in dependency in the testing (ie- if Test B requires you pass Test A to work…make sure Test B runs second).  Finally, sandbox testing of high level issues kept a lot of code from entering their “1 engineer at a time” test system by having everyone be able to test it on their own machines first.

3) Outsourcing doesn’t work well

This sort of system requires a cultural buy in that IMVU found couldn’t be instilled well remotely. It also proved difficult to manage the testing system whenever some engineers were in a different time zone.

4) Complex fails are harder to find

Brett still occasionally finds issues don’t fail during the work day…and being VP of engineering it means he gets the 3am calls about such issues.  They also struggle with MySQL and memcached issues, which has led to separate systems being developed to deal with them.

—

The key principle in all of this is that continuous deployment is a constantly evolving system. It is not perfect. However, taking it one step at a time will help you build a formidable engineering process that allows your company to move faster than you ever thought possible.

Ready to try to bring continuous deployment to your startup?

Lessons learned in getting press

27 October 2011

3 Comments

This post originally appeared on Greenhorn Connect here: https://greenhornconnect.com/blog/lessons-learned

I’ve been very fortunate over the past few months to get some really awesome press coverage for myself and Greenhorn Connect.  I also worked at a startup where the CEO was extremely adept at garnering press as well. From these combined experiences, I’ve learned a bit about getting press. I’d like to share some of those lessons here and hope you’ll leave any lessons you have in the comments.

Lessons learned in getting press

1) Press comes in waves

When I joined oneforty in April 2010, Laura was making waves as she helped cool the Twitter developer community that was upset by the acquisition of Tweetie, the first of many ecosystem upsetting acquisitions. The coverage around the meeting Laura helped organize for the major apps and Twitter led to a ton of additional stories across the blogosphere and press. As I stood amazed at the press coverage, Laura told me the wise words, “Press comes in waves.”

Now that I’ve seen it first hand, it makes complete sense. My part in the Scott Kirsner Sunday Globe article on keeping young talent in Boston led to Boston Magazine deciding to cover me as a Person of Interest.  Curt Nickish, who wrote the Boston Magazine article was then able to spin the piece into a second entry, this time with a slightly edgier tone and a wider audience in Fast Company.  I expect this is the end of it, but who knows.

From talking to friends with PR experience, I’ve learned that you can also try to intentionally create waves.  By strategically pitching different journalists different aspects of your story at varied times, you can create a similar wave of stories about you and your business.  Do this a few times and the relationships you develop will allow you to make serious waves around big news for your business.

2) Everyone has an angle

In many ways the two stories I had were the same story, just told very differently.  It speaks to the different audiences of Boston Magazine and Fast Company; the former is affluent New Englanders, while the latter is a widely distributed, national publication focused on business with a slight edge. To affluent New Englanders, the story is interesting because I’m a young guy making it in Boston. To the wider population of Fast Company, the story became more about the rise to “power” the article implied I have over our community (Obviously no one has “power” over our community, certainly not me).

On the flip side, you can use this to your advantage when trying to be written about by doing your homework. Research the journalist and pitch the story in a way that you believe would be most compelling to them. You can also check sites like Help a Reporter Out (HARO) for opportunities.

3) Control how you’re perceived

If you’ve been following the TechStars TV drama, you’ll notice that what was pitched as a documentary has actually been a by the book reality TV show complete with jocks, overconfident teams and many perceived winners and losers. Hacker News was ablaze the other day over a blog post by Melanie of ToVieFor.  The companies on the show are lucky, as according to Jason Baptiste, TechStars fought hard to tone down the reality-tv bent of the show.

When they called me for the photoshoot for the Fast Company article, they suggested they photograph me wearing a crown and handing out crowns to other people walking by. I rejected this idea flatly, as I think that image combined with their original proposed headline, “King Maker” would have made me look pretty silly at best and a total ego maniac at worst.  Fortunately, I was able to get them to go with a more conservative photo as you see me shaking hands in the article.  If I hadn’t spoken up, no one else would have for me and I’d be the one looking like a fool.

Don’t be afraid to be proactive. In the future, I plan to request the right to read the article before print when it’s more than a simple news story .  If you get major press coverage, what is written can be a the first impression of you to many people.  Press is cool, but you want to make sure it’s something you and your friends/coworkers/staff/family can be proud of.

4) The truth always wins out

If you only knew Jason Baptiste from TechStarsTV, you’d think he’s a cocky, overconfident startup kid.  If you’ve ever read his blog, asked a question of him on Formspring or met him in person, you’ll know he’s a very down to earth, smart guy that really cares about building a great company and helping others when he can…and happens to have a flair for theatrics in presentations.

If you think you have a communication crisis, lean on your friends in PR; they’ll know what to do. People appreciate honesty and transparency. Just look at how Grasshopper handled their Chargify issues or how Theo Epstein made the most of an ugly Red Sox breakup.

5) Journalism is about relationships

Many in Boston were in awe at how easy Laura made it look to get major press for oneforty on a regular basis.  The answer was not that we had a killer PR firm (we had none). It was that Laura built relationships with tons of journalists. During those trips to the Valley Laura periodically took especially early in the days of oneforty, she wasn’t just raising money; she was also building relationships with the tech press out there. Not surprisingly, the same person she was having beers, coffee or meals with were then more likely to write about her.

When I met with Curt for the Boston Magazine article, we ended up meeting much longer than expected and having a conversation that went well beyond an interview and allowed me to learn a bit about Curt as well.  I believe it was this reporte and the mutual respect we developed that Sunday afternoon that contributed to his motivation to try to get wider publication of my story in Fast Company.

Like so many things in startup life, I feel like I’m just learning the tip of the iceberg of what I need to know and master.  If you’ve learned any lessons in PR and media, please share them for all of us to learn!

Does Boston Have Too Many Startups? A response to Kirsner’s Sunday Globe Article

18 October 2011

This post originally appeared on Greenhorn Connect and has a boatload of comments. See here: https://greenhornconnect.com/blog/does-boston-have-too-many-startups-response-kirsner-s-sunday-globe-article

In the Sunday Globe this week, Scott Kirsner posed the question, “Does Boston Have Too Many Startups?”  The article seemed to try to make the argument that all our little startups should just be employees at bigger startups (disregarding how bigger startups, start out…).

The article is really best summed up in the quote in the article by Craig Driscoll, “companies that hope to grow need to do more than complain about how tight the talent market is.” I find it fitting that coincidentally, Ryan Durkin, COO of CampusLive (and mentee of Mr. Driscoll as a Highland Capital portfolio company) writes about attracting talent today:  http://www.ryandurkin.com/blog/2011/10/how-to-hire-dudes-showers-kitchens/

I’ve spoken with a number of friends about the article and had some interesting Twitter conversations as well and wanted to highlight some of the key points that came from them.  (Note: Kirsner sought out some thoughts which you can see on his Globe blog here.)

1) We don’t talk about logical career paths enough

Quick: explain, with examples, a logical career path for someone to evolve to be a successful entrepreneur.  Stumped? I know I am. Most examples I think of fit in the “Zuckerberg” files of folks who didn’t have much of a startup pedigree before launching their monster success (think Matt Lauzon, the many TechStars companies, etc).  I look at the titans of our community like David Cancel, Dharmesh Shah, Jeff Bussgang and know very little about “how they got here.”

It’s easy to tell people “go work for a startup first,” but you need to show them examples of people that have succeeded in doing that, and if you didn’t as a founder, then you have to acknowledge you have some hypocrisy on your hands.

2) We don’t have enough serial entrepreneurs and mafias

We are all familiar with the Paypal Mafia and some of the many startups the former employees spawned, but where are Boston’s Mafia’s? There was a great Bostinnovation series covering some of them, but it seems like there are fewer and certainly less celebrated.

These mafias are exactly what would convince someone to *join* a startup instead of start their own: join a team and have a great exit and then have peers and valuable experience to start another.  Maybe an offshoot of Eric Paley‘s Founder Dialogues needs to be “Mafia Dialogues” and bring in a few people from a successful team to share their combined story. We also need to think about whether it’s good for a CEO of a billion dollar company to be proud that all of his first 10 employees are still working at his now billion dollar company.

I think Rob Go was also on the right track highlighting the power team of Brian Balfour, Aaron White and Ariel Diaz teaming up for Boundless Learning (and also happens to talk about the importance of “more shots on goal,” not less startups as Kirsner suggests).

3) We don’t take enough chances on Greenhorns

I am very lucky. What few of you remember is that no one was interested in hiring me when I came into the tech scene. John Prendergast was the first to give me a small shot doing some work for him, which then led to the opportunity to pitch Laura and the oneforty team on joining them (John was on their board).  That was a 6 month journey to get that full time offer from oneforty.  If I didn’t live as lean as possible and have the luxury of a little savings, I may have never made it.

Just like our investors are often criticized for wanting too much traction before they invest, many of our companies only want to hire people that have done a role before.  I know too many young, eager people who want to work at startups, and yet there doesn’t seem to be many roles available to them.  I get asked about Janet Aronica, who I hired at oneforty, a few times a month it seems and the irony is, I doubt anyone else would have taken a chance on a young, eager talent coming from the low rungs of a PR firm.  I also look at Kristin Dziadul, who once made videos trying to get HubSpot’s attention and was smartly hired by Rob May and Backupify.

This is not to lump everyone together. Matt Lauzon and the Gemvara folks have taken chances, and I know Diane Hessan has been an awesome advocate for some of the truly hungry Gen Y folks. Unfortunately, the rest of the community hasn’t caught on yet.

4) We don’t take recruiting seriously

It bears repeating via Craig Driscoll, “companies that hope to grow need to do more than complain about how tight the talent market is.”  Vinod Khosla went as far as to say “New CEOs should spend 50% of their time recruiting.”  I’ve seen the aggressiveness of Sequoia firsthand as they held an event the night before Startup Bootcamp for their founders who were speaking to tell a bit of their stories, answer questions and meet people. I know Dropbox has spoken at at least 3 Boston area schools in the last 6 months.  How many schools have you spoken at in your own back yard?

Maybe we need an event or two to talk more about recruiting talent (One of my favorite events ever was a fireside chat with Akhil of MassChallenge and Paul English talking about recruiting).  I think it’s a competitive advantage for some companies while others throw money at it, but it certainly seems like it might help.  More awesome posts on the subject like Brian Balfour’s and David Cancel’s would help too.

5) Complaining about funded startups is an insult to entrepreneurs

The funding climate in Boston has improved, but it’s still hard to raise money.  As the opening to the Bloomberg TV show states, TechStars is more selective than Harvard. If you manage to raise money, that’s quite an accomplishment, as David Friend says in his message to Kirsner.  If you have an idea that goes far enough to funding, you have an at bat and need to go try to execute. The amount you’ll learn with your small team will be tremendous and put you in a great position to contribute to another company.

Now, living off of savings, is obviously a big risk and so if someone is risking financial ruin to keep their fledgling startup alive, it most likely makes sense to go work somewhere. Then again, the Valley wouldn’t have Airbnb if those guys weren’t relentless.

6) We need to clarify what a startup is

Wayfair is a $350Mn+ company. HubSpot has over 300 employees.  Are both of them still startups? Neither? I worked at E Ink when they went from about 90 employees to 165.  From my experience, the Dunbar number is very real.  The culture really started to shift at that point and more HR rules and regulations hit, not by choice, but out of necessity.  Few companies are really like Google and provide the independence and opportunities similar to startups, and even there, I don’t think you get the same thing.

I’ve heard about HubSpot’s education classes that try to teach some business and entrepreneurial lessons, but I have a feeling they’re the exception, not the rule. I also wonder how they’ll be able to maintain it as they grow further.

7) What motivates an entrepreneur or early stage employee?

Another great quote from Craig Driscoll on advising companies was, “They need to figure out how to recruit and create jobs that are attractive for entrepreneurial people.”  An entrepreneurial person cares much more about working with other smart people, having flexibility and independence in their role and a feeling of true contribution and influence in the big picture of what they do.

When I worked at E Ink, everyone looked forward to their quarterly meetings where Russ Wilcox explained the state of the company.  It lifted the covers on how the company was doing and especially during the wild ride that was the exploding popularity of the Kindle, you could feel everyone having a renewed sense of purpose after work as they were a part of the amazing opportunity to replace paper books with E Ink technology.  I think that same sentiment is happening now with Gemvara as Matt hammers home the vision to change jewelry and e commerce on the web.  Entrepreneurial employees want to be a part of something bigger than themselves, while feeling like they’re really making a contribution.

—-

I’m glad Scott brought this conversation out for discussion, but feel like it missed the mark on what really matters in this ecosystem developing.