The Lean Product Life Cycle

See my previous post to understand how I got here. The hypothesis is that product development is messy except for the most disciplined. After talking to a number of great product people, I have a theory for how great product can be developed while being customer centric (aka lean).

The Virtuous Cycle of Lean Product Development

The Product Management Funnel

I think of product management as a funnel. At the top are all of the ideas your team generates. If you’re a lean startup, that’s hopefully driven by customer interviews, website and landing page behaviors and support interactions and of course the occasional green space wild idea.

As product manager, this is the most dangerous step. Once you have more than a few employees, it’s easy to have these ideas overflowing and interfering with the whole product process. Multiple teams I spoke with have been using their project management tools to capture these ideas which leads to a huge mess (one such company had over 4,000 stories in their icebox!) and a major time suck (one vp of engineering is spending an hour + each day managing these ideas).

The problem is project management != product management. The only thing that should be in your project management tool are key bugs to fix and what you’re building now or in the very near future (ie next sprint). Everything should be clearly defined and in a language and structure preferred by your engineers. All the evolution of the product from idea to customer validation to final prioritization should be done outside the project management tool.

The Rest of the Funnel

The Full Product Funnel

Going back to the beginning, you need discipline at the top of the funnel; the best product people I spoke with are requiring a shred of data tied to an idea to make it into their coveted ideas list. They also often expect a disciplined approach to what goes in by having people state, “Feature X will move Metric Y by Amount Z” so it’s clear why the feature needs added. In earlier stages that may be in a “Hypothesis vs. Metric to Invalidate” format instead.

The key of this step is effectively managing the signal to noise ratio; for every 5-10 ideas that come in, you may only make 1 or 2. Even the less disciplined product managers I spoke with have some type of hot-lukewarm-cold system for trying to rank ideas. To avoid the 4,000 stories in your icebox (or anywhere else) you have to be disciplined on dropping stale ideas and focusing on what matters now. When you’re a growing startup, where you were 3 months ago is dramatically different from where you are now so why keep those stories cluttering your system now?

After wrangling the initial feeder of ideas under control, you need to effectively refine the ideas that make it into your system and allow the most important ones to rise to the top of your list. As planning for your next sprint begins, you need to prioritize these ideas and balance with other existing projects, bugs and other demands.

Once you’ve settled on what you’re going to have engineering build next, you need to engage your team on the ideas you want to implement; you may need copy from marketing, stories from support to describe customer complaints, links or screenshots from analytics to show present activity and relevant customer development notes. All of this feeds into the project management tool you have, but now it’s uncluttered and your engineers are much happier and efficient. Unless you’re github or heroku, chances are your engineers don’t understand the customer they’re building for perfectly so all this structure makes it easier for them to see what’s going on and then adapt it to fit the project management tool they’re using to track building.

As an added bonus, there’s transparency for your other employees as they engage in more than just the initial ideation process. It also means sprint planning no longer needs to have all hands at it as they’re already well aware of what’s going on and have had opportunities to contribute as needed to the stories now being generated in the project management tool.

Completing the Virtuous Cycle

We all know this image so well as the core of what lean startups is all about, but how often is this cycle cleanly implemented? The project management tools out today all do a great job on the build aspect, but what about measure and learn?

The funnel process I described above captures the learn aspect as you engage your team for ideas and validation. All that’s left now is to measure.

After you build something new in your project management tool, your engineers will submit it for some form of approval. Once approved, it universally ends up disappearing into the ether, because the project management tool is built to track what you are building or about to build. After building isn’t part of their process.

To close the lean loop, you need to look back and see if that feature actually moved the needle. This needs to happen within 2-4 weeks of building it; sooner wouldn’t be enough data and longer would be distorted due to other things you’ve likely built by then. This last step of learning should help you refine your instincts and feed back into what you build. This is one of the biggest challenges as Eric discusses his principle of Innovation Accounting in his book.

The amazing thing I discovered in so many of my interviews was how rare it was to do this measure step; how can you improve your accuracy without seeing if you hit the target?

The Product – Lean Opportunity

What started as an investigation of lean startup opportunities has broadened to helping teams close the loop in managing a product lifecycle. I believe there is tremendous opportunity to build a platform to aide on both sides of the project management tool and much wider than those adopting lean concepts officially (as we all know, being customer centric is not a new concept).

 

Using Customer Development on the Customer Development Process

Earlier in November, I started out on a journey to see if there was an opportunity to build a product that would help companies in being a lean startup. The thought was that with Eric Ries’s book flying off the shelves and the lean startup community exploding, there would be new problems to be solved. Thanks to a nudge from Sim Simeonov and a few consulting opportunities brought my way, I felt like there was definitely something there. Like any startup, once I got outside the building, I learned some harsh realities:

1) Most people only pay lip service to lean.

Painful, but true. In both casual conversations in our community and full blown custdev interviews I found the full spectrum from strict implementers like John Prendergast and Matt Mamet (coincidentally co organizers of the local Lean Startup Circle Boston) to people who claim to be lean and think it has anything to do with how little money they spend (names withheld to protect the innocent).

2) Lean is a hands on activity.

The key to lean really is the human interaction; knowing when to ask an interviewee “tell me more” or deciding what analytics to deep dive to match up against your anecdotal discussions can’t be productized directly. It’s also risky to outsource since it’s so core to the future of your business; no amount of meetings, notes and summaries can compare to being there and doing it yourself. There’s huge opportunities to teach companies how to become lean, but I’m passionate about building a product, not a services business and I’m not sure most companies are ready to invest in lean (see 1).

3) Small startups are more likely to be lean and less likely to need process.

Part of the challenge of the adoption curve of lean is that it has most strongly permeated the zero stage entrepreneurs; they’re the ones with time and motivation to watch the videos, read the books and stay up on all the blogs (which has come a long way in the last year or two since all we had was the Four Steps to the Epiphany to go on).

Unfortunately, those people have little to no budget and because of only having a few people on their team, there isn’t much of a need for tools. Everything can be done ad hoc when you’re that small. The most common behavior when you have 5 or less people is to remove your headphones during the work day and share your findings immediately with everyone on your team.  This is effective until you grow to the point that everyone doesn’t know everything that everyone is doing (usually around 8-15 people).

So all that meant to me was time to pivot:

I’ve known all along that customer development feeds into product management. Fittingly, this is where I did find pain:

  • Managing product while being a growing, lean startup requires significant discipline. Not all companies have product leaders with this strength.
  • Many companies build up product management debt as they grow and fail to adapt to the demands of a team that passes the “take off your headphones and talk” phase.
  • Multiple challenges arise in engaging all members of a company at the right times in the right way.

My next post explains what I’ve found. Read on to see the big picture potential for a lean focused product life cycle.

Continuous Deployment: Possibility or Pipe Dream?

For those in the Lean Startup world, the utopian version of product development is continuous deployment.  It means every engineer is deploying code multiple times a day, often even on someone’s first day of work.  It’s also exceptionally test driven, reducing risks of bugs taking down the entire system.  While many startups aspire to this, few have succeeded, which is why it was so exciting to have Brett Durrett of IMVU come and speak to the Lean Startup Circle Boston Thursday night.  Brett is VP of Engineering at IMVU, which coincidentally happens to be Lean Startup Guru, Eric Ries’s startup he spent many years helping build before becoming the movement’s biggest evangelist.

{Note: Brett’s presentation was awesome but hasn’t been posted yet. I’m embedding his Lean LA version as a reference until it can be posted.}

Continuous Deployment at Lean LA

View more presentations from Brett Durrett

In a nutshell, continuous deployment breaks down into 3 steps:

  1. Develop a feature
  2. Test it
  3. Deploy it

But why would you do continuous deployment?

The reason for using continuous deployment hits the core of lean startups: more iterations. Whether funded or bootstrapped, there’s a limited amount of time to iterate and nothing speeds iteration like getting new features, site tweaks and updates out faster. Continuous deployment forces you to break down all your features into bite size chunks which can save you building massive features when you can confirm it with much smaller steps. It also minimizes version control issues if no one is working on a long term project based on old code.

It also makes your engineers more efficient. Is it easier to find the problem with freshly deployed code that has 10,000 lines in it or 10 lines? Is it easier to get engineers up to speed on a system that expects them to write a micro feature or build a major piece of the system?  If everyone writes their own test code, you have greater accountability across the entire engineering team (you have to fix what you break) and you don’t have to hire a QA team stuck cleaning up everyone else’s mess.

The Continuous Deployment Process for Engineers at IMVU

After convincing us why it matters, Brett walked us through the process for an IMVU engineer.  Once an engineer has finished building their bite size feature, they walk through all of the following in less than 15 minutes!

1) Engineer runs the test in their sandbox.

To keep from clogging the deployment and testing systems everyone shares, engineers first run some basic PHP tests on their own system to ensure the code is ready.  (Note: they don’t do branches in the repository; the brand in the code instead.)

2) Engineer runs testing on main system

After passing the tests on their computers, they enter the queue for the main testing system for all over IMVU called Buildbot. They have tons of tests so you tag your code based on what parts of the system it affects and what it’s for. This optimizes the right tests to run and which can be avoided. (Running every test would take over 8 hours.)

On average, it takes about 8 minutes to run all the tests needed.  They’ve achieved this speed because they have 40-50 instances running just for testing.  They’ve also discovered that 12 minutes or less is the optimal time to have testing take and keep your engineers happy.

The most common slip at this stage is a missing tag, which means a needed test isn’t run.

3A) If all tests pass, the engineer now deploys the code. 

3B) If any of the tests fail, the engineer reverses their commit.

With the rate the whole team is deploying and testing, their’s no time to have people fixing while they’re in the system. Therefore, regardless of the issue, the engineer will reverse their commit and go back to fixing the problem on their machine and starting the test and deploy process from the beginning.

4) Deployment occurs in pieces

Currently, IMVU (with over 50 Million registered members) has 800 servers in use.  When they deploy new code, it starts out on just 35 servers. This ensures that if something goes wrong, it doesn’t take the entire site down.

5) Testing continues after deployment thanks to the Cluster Immune System

Even after deployment, they’re still testing, just in a different fashion. They’ve developed their own tool called the “Cluster Immune System” which monitors key site (speed, system performance, etc) and customer metrics (revenue, registrations, etc) to make sure there hasn’t been a dramatic change.  Even the best tests won’t notice an engineer accidentally made a blue button on a blue page; the tests will see the button is still there and works, but won’t realize a user can’t see the critical sign up button.

This system runs on those 35 servers they use as a live test bed. If anything goes wrong there, they prevent it from deploying to the rest of the system. If not, it’s deployed system wide to all 800 servers.

An audience member asked about “what if you don’t have massive traffic you can segment to test a new deploy?” Brett said it’s an advantage when you’re bigger, but until then, you may just run at Cluster Immune System to monitor a system-wide deploy.

The best news of all of this?  IMVU plans to open source the Cluster Immune System (CIS) soon.

6) If all CIS testing is passed, deploy to all servers, but continue monitoring

Even after deploying to all servers, they still monitor for anything unexpected. If they see anything alarming, they’ll roll back and remove the feature.

This entire process takes only 10-12 minutes.  Only one engineer can be in the Buildbot testing phase at a time, but as soon as you enter deployment (step 4, above) someone else can enter buildbot.

This process sounds great, right? But it seems so sophisticated…how do you get started? Brett covered that too..

Getting Started – How do you actually do this?!?

Getting started is a different process depending on if you’re an established company or just an infant startup with limited traffic, but either way, there’s great ways to get started:

  1. If you’re a small startup - Start with a sandbox for each of your engineers and just focus on pushing code quickly and in small chunks.  You can develop your testing as things break; that’s how IMVU built their system.
  2. If you’re an established company - Start with production (ie- the last step before deploying code) and automate that process. Start building tests for whether something should be deployed or not.  Work to get the automated tests as good as the human part of the process. Once you’ve accomplished this, keep working backward to continue to remove humans from the deployment flow.  At first, you should err on the side of preventing problems then clean up your tests to be efficient.

Whether big or small, the same key rule applies: Anything can break once. Then you have to make it so the same thing can’t happen again by writing a test for your mistake. This builds both accountability and builds only the tests you really need…one step at a time.

Pitfalls

Like any system, this isn’t perfect. There are challenges both with getting personnel buy in and in scaling this:

1) Philosophy- Blameless systems

It’s not “Ned broke this!” it’s “How did we let that fail get through our system.”  Brett really emphasized this important difference. It’s a philosophical buy in required to really make this work best.

At IMVU, they hold regular “Blameless Post Mortems” to discuss issues that slipped through.

2) Optimize your testing

As you grow, more tests will be required. You should optimize for which tests actually need to be run for a specific line of code (as they did with tagging) and purchase sufficient hardward to make tests fast. IMVU also found they could save a tremendous amount of time simply by optimizing the order the tests run, by running the slower ones first (a 22% time saving).  They also then built in dependency in the testing (ie- if Test B requires you pass Test A to work…make sure Test B runs second).  Finally, sandbox testing of high level issues kept a lot of code from entering their “1 engineer at a time” test system by having everyone be able to test it on their own machines first.

3) Outsourcing doesn’t work well

This sort of system requires a cultural buy in that IMVU found couldn’t be instilled well remotely. It also proved difficult to manage the testing system whenever some engineers were in a different time zone.

4) Complex fails are harder to find

Brett still occasionally finds issues don’t fail during the work day…and being VP of engineering it means he gets the 3am calls about such issues.  They also struggle with MySQL and memcached issues, which has led to separate systems being developed to deal with them.

The key principle in all of this is that continuous deployment is a constantly evolving system. It is not perfect. However, taking it one step at a time will help you build a formidable engineering process that allows your company to move faster than you ever thought possible.

Ready to try to bring continuous deployment to your startup?