Experiment, Learn, Repeat, Scale

Plan and Operate Your Experiments for Growth & Innovation Success

The Digital Leader Newsletter — Strategies and Techniques for Change Agents, Strategists, and Innovators.

---

I have not failed. I’ve just found 10,000 ways that won’t work.

—Thomas Edison

I love the history of products that overreach with glorious names. Like an apartment complex named “The Lakes” situated on a barren section of land with one tepid, artificial pond. Products that aimed for the stars but imploded on the tarmac, such as the Apple Newton, GoogleGlass, Microsoft Bob, New Coke, and the Amazon Fire Phone.

My current favorite buzzword is agile methodology: an approach for iterative, incremental solution design and delivery. I’ve worked with and managed several technology programs using an agile methodology. Key participants in the initiatives apparently define agile as “the methodology of no accountability.” Scope, time, cost? You can’t hold us accountable because we are agile (wink, wink). They are wholly noncommittal when it comes to doing what they say they’re going to do. Yet the greatest semantic travesty occurs when agile is used as an excuse for not getting real results—the kind that actually matter for customers and the business.

THINKING BIG, NOT BETTING BIG

I led the launch of the third-party business at Amazon in 2002. This platform allowed retailers to sell their products through Amazon’s website. Today it supports 3 million sellers, and it is responsible for over 50 percent of all units shipped and sold at Amazon. Having and holding a big vision for creating an outstanding customer and seller experience was critical in laying the foundation for success.

As we built the third-party business, we focused on three core principles that would allow it to grow. First, the customer experience with these third-party sellers needed to be indistinguishable from their experience buying directly from Amazon, the retailer. Second, the experience of selling on Amazon’s third-party platform had to be intuitive and easy, even though the process was relatively complex. Third, we had to design the platform and strategize around supporting hundreds of thousands—not tens or hundreds—of sellers. Managing your business at Amazon had to be self-service.

Because of our strategy, the data and transaction choreography between the sellers and Amazon had to be more complex. We built tools, examples, test environments, and support to make selling as intuitive and efficient for sellers as possible. We also had to think through topics of scale. Amazon had parity agreements with its third-party sellers, requiring them to list items on Amazon at the same price and availability as any other sales channels. How could we keep track of whether sellers were living up to this obligation?

The most obvious but least scalable and effective option would have been to rely on manual audits or reviews. That would have been an expensive process that only allowed us to review a subset of items. Instead, we built an automated system to verify that sellers were living up to their parity obligations. Using the item information sent to Amazon, we were able to crawl their website and any other sales channels to verify the consistency of their pricing and availability.

We were thinking big, really big. But we did not confuse “thinking big” with “betting big,” and neither should you.

While my summary of how we built Amazon’s third-party business platform sounds like it was smooth sailing, the fact is that the voyage was littered with failures. However, none of these sharp reefs succeeded in sinking the ship. The business and art of innovation lie in the many failures—the learning, adjusting, and moving ahead—that come along the way. The principle you’ll find below the waterline is about how to scale failure.

STARTING SMALL

The better you become at creating ways to test repeatedly in small ways, the more likely you are to achieve big success.

In other words, think big, but bet small.

It took years before the third-party business became the force it is today. Before I even joined the Marketplace team, Amazon had already tried—and failed—to build two other third-party seller platforms. Before Amazon Marketplace, there was Amazon Auctions and then the zSHOP. The first two were failures; the third is now a huge success.

And although Amazon certainly invested in the Marketplace business, it was not a high-risk investment. Instead, leadership invested in relatively small individual experiments that would improve Amazon’s long-term understanding of the winning formula.

What do those smaller experiments look like? At Marketplace, we thought customers would want to shop through seller-specific storefronts, so we built the infrastructure to enable sellers to create branded online storefronts. Once we launched storefronts, though, we found customers were more likely to shop by category across Amazon’s entire site. As a result, we wound up deemphasizing merchant-specific storefronts and focusing instead on improving Amazon’s core browsing and searching capabilities.

It wasn’t until Marketplace eventually found the right long-term formula—through this and many other experiments— that the company shifted its focus to growth.

Successful growth executives execute lots of small experiments, and they have a patient, long-term approach to product and business success. It is rare, though not impossible that innovation and short-term profits go together. Executed well, these kinds of small experiments help you understand your customers’ needs and how your product might fit the market. Executed poorly, they can be worse than not experimenting at all.

TACTICS FOR BETTING SMALL

A failed experiment could just as easily be the result of bad execution as it could be a valid test of your hypothesis about a product. Luckily, Amazon and others have developed tactics for acting small and moving iteratively to help you avoid this trap.

Low-Fidelity Prototyping

If you’ve ever created something that only partially works just to test a few critical components, you’ve created a low-fidelity prototype. Google’s Cardboard is an example of a low-fidelity virtual reality (VR) prototype: sticking their phones inside a cardboard VR viewer, users can test and experiment with VR.

(Wikipedia on the Google Cardboard)

The first version of this was likely hacked together with a few off-the-shelf parts and a cardboard box. Today, Google used Cardboard to build its developer community and to test the popularity of virtual reality without spending valuable time and money creating a more complex VR product. Consider ways you can test the effectiveness or viability of your experiment without actually producing it. Low-fidelity prototyping can be useful as a visual demonstration of how a product might work and as a way to get buy-in for full prototype development.

Minimum Viable Product

The idea of a minimum viable product (MVP) was popularized by Eric Ries’s 2011 book, The Lean Startup. In it, Ries encouraged business owners to identify and test the critical assumptions behind their businesses and solutions.

Inspired by the work of his mentor, Steve Blank, Ries popularized the idea of using a minimum viable version of your product to help you prove or disprove assumptions about your business and customers through carefully constructed trials.

The key for the MVP is articulating, as succinctly as possible, what part or feature of your bigger vision needs to be validated or tested first (or next) and making the scope as closely limited to that feature as possible. This is the process of defining and measuring to hypothesis and quickly getting customer or real-world feedback on that test, then proceeding in as incremental or agile a manner as possible to the next test.

At a conceptual level, this sounds easy to do. But in practice, there are many forces and realities that come into play and work against the small-as-possible MVP. Among these challenges are accurately understanding and defining what the correct hypotheses are and in what order they should be; finding ways to build just to the capabilities needed versus the many surrounding capabilities needed for a market-ready product; finding ways to let real customers use a minimal product without undercutting your brand; and finally, avoiding the likely impact to other enterprise applications and processes (including potentially avoiding the centralized information technology group) until the right time.

The key suggestion is to avoid committees and group decision-making as much as possible. Trusting one strong voice helps cut through the layers, shrink time to market, and minimize scope. But that one voice does need to be mostly right.

THE FIRE PHONE AND THE ECHO

“One area where I think we are especially distinctive,” Bezos wrote in Amazon’s 2015 letter to shareholders, “is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention but are not willing to suffer the string of failed experiments necessary to get there.”¹

Amazon’s most prolific failure was the Fire Phone, launched in July 2014. It was a short-lived market miss that resulted in a $170 million inventory write-off.

“What the hell happened with the Fire Phone?” asked stock analyst Henry Blodget in a Business Insider discussion with Bezos.²

The Fire Phone, like all of Amazon’s projects, was an experiment, Bezos coolly replied. In his mind, its failure was a learning experience—another chance to iterate or pivot. The phone, Bezos explained, was just one more entry in Amazon’s “device portfolio.” The operative word in that phrase was “portfolio.”

“It’s early,” he told Blodget. “And we’ve had a lot of things we’ve had to iterate on at Amazon. One of my jobs as the leader at Amazon is to encourage people to be bold, . . . to create a shield around the team innovating so they can focus on the hard things they are accomplishing and minimize the noise and concerns of detractors within the company.”³

Amazon launched its next round of smart devices according to the standard playbook: launch a product or service quickly, don’t make too big a deal of it, and spend almost nothing on marketing. Instead, get customer feedback and make adjustments or cut quickly. Launch more experiments. After all, Bezos views every project as an investment in Amazon’s portfolio.

The Amazon Echo was an invitation-only product for Amazon Prime customers. It was advertised as a beta product, and it was available in limited quantities. This kept expectations low and company learning high. Amazon followed the same playbook for the Dash Button. Only when feedback and reviews of the Echo and Dash Button were fantastic did Amazon make them available to everyone. It’s the same strategy that theater producers use. Tour the production around the country in select markets as a workshop production. If the reviews are good enough, open on Broadway. If the audiences hate it, pretend it never happened and move on to the next project.

YOU STILL HAVE TO BE RIGHT; A LOT

Amazon’s Leadership Principle 4 is “Leaders Are Right, a Lot.” It reads, “Leaders are right a lot. They have strong judgment and good instincts. They seek diverse perspectives and work to disconfirm their beliefs.”

Creating and adopting a test-measure-adjust mentality is not without risk. Using it as a crutch for either bad judgment or poor execution can spell the death of your enterprise. Just as “agile” has been allowed to become the development methodology of no accountability, “fail fast” can become the excuse for more failing than needed. Just because many opportunities to become digital exist in the business, that doesn’t mean you can afford repeated attempts at something that you should get right the first time. “Failure” has become an excuse and almost an expectation; this is the risk of the current innovation models.

The difference between a test failure in which you learn and an execution failure is sometimes obvious, sometimes really subtle. The more involvement you have, the better your odds of knowing which one it is.

Marc Andreessen noted,

“We are biased toward people who never give up. That’s something you can’t find on a résumé. We look for courage, and we look for genius. There’s all this talk about how important failure is, which I call the failure fetish. ‘Failure is wonderful. It teaches you so much. It is great to fail a lot,’ people say. But we think failure sucks. Success is wonderful.”⁴

To win in the digital era, you have to have a strong product or business vision and the ability to listen to others, but you must also take command and clearly articulate what is needed. This creates the “think big but bet small” strategy to drive innovation.

About The Digital Leader Newsletter

This is a newsletter for change agents, strategists, and innovators.  The Digital Leader Newsletter is a weekly coaching session focusing on customer-centricity, innovation, and strategy. We deliver practical theory, examples, tools, and techniques to help you build better strategies, better plans, and better solutions — but most of all, to think and communicate better.

1. https://s2.q4cdn.com/299287126/files/doc_financials/annual/2015-Letter-to-Shareholders.PDF

2. https://www.businessinsider.in/The-Amazon-Fire-Phone-Shows-What-Happens-When-The-CEO-Gets-Too-Involved-In-Product-Design/articleshow/45783982.cms

3. https://www.businessinsider.in/The-Amazon-Fire-Phone-Shows-What-Happens-When-The-CEO-Gets-Too-Involved-In-Product-Design/articleshow/45783982.cms

4. https://www.gsb.stanford.edu/insights/marc-andreessen-we-are-biased-toward-people-who-never-give