top of page

Creating Experiments

The biggest challenge a startup or an engineer inside one faces is not building something that works. It's throwing away something he loves. Yet, the reality is nearly all of the code you write is throw-away until the customer falls in love with what you've made - meaning they are using it and paying for it. At the point, you can pause and say the code is good enough; for now. That happens pretty late in the game, usually.

Meanwhile, the more code you add, the more emotional commitment you have to what you've written and the harder it is for you to let go of the code when its time to throw it away.

I feel ya. The heart of the problem is neither the annoying customer nor the apologetic product manager. It's simply the nature of the learning cycle you're in. It's harder and harder to predict what customers will like, given the choices they have, and the increased level of sophistication in their behavior. Not even Paul Graham thought AirBnB was a good idea. So committing a lot of code up-front is destined for heartbreak.

What do you do instead? You run small experiments cheaply first like bait on a hook. If you see your customers consistently biting, then you commit to code. Here's what these experiments look like.

It's a funnel of experiments with the narrowest, most precise one on the right and the widest, least precise one on the left. Turns out, the least precise ones are also the cheapest and quickest to get out of the way. You don't have to go all the way back to the start of a funnel, but it helps to go a few steps back since the effort you put in dramatically decreases.

Assuming you're creating a prototype with working code, one step back is a click-through prototype. This is all of the interaction on the interface mocked up in high-fidelity using a tool like Framer, Adobe XD, or Invision with realistic data mocked up as well, but no calls behind it. You get a realistic feeling of what the application would do if it was actually working but is exponentially less expensive to build compared to a working prototype. With this, you can see whether your customers are truly interested in what you're offering.

One more step back and you get a first-click experiment. Given a goal, say shop for an item, this checks whether your customer places the first click to do this in the right place. From measurements, we know a participant who clicks down the right path on the first click will complete their task successfully 87% of the time! This is simpler to mock-up compared to a click-through since you're only testing the home page and the first click. From this, you can see whether you've captured your user's intent clearly or did it get lost in translation.

One more step back and you have a survey as an experiment. Even before you make a single screen, if you know your customers you can send them a simple survey presenting a few scenarios that make them choose between two options.

For example, given an application that lets you set a timer to force you to walk away from your computer periodically, would you prefer A: the system gives you a countdown to save work and locks you out, or B: Saves all your work for you and locks you out without warning?

You can put several of these choices into a survey to test how they expect an interface to respond to them and reveal unstated biases that may not emerge any other way. In this example, it may be the fact that users do not trust the computer enough to let it do an auto-save. Forcing choices, also known as sacrificial concepts, forces your customer to make choices about what they prefer, and in the process reveal what they care about.

One final step back and you arrive at the top of the funnel with an advertisement. If you do not know your customer and want to see whether or not there is even a market for your intended offering, say sky diving for your pets ( hey, why not !), you post an advert on Craigslist or buy a keyword and post an ad on Google and see what you get. If you get overwhelmed with responses, you have a reason to go on. If not, aren't you glad you figured out it was a goofy idea before you invested time and emotions on building it out?

57 views0 comments

Recent Posts

See All


bottom of page