What makes a successful personal experiment?

As I continue trying to stretch the concept of experiment so that a wide audience understands applying a scientific method to life, I struggle with defining success. While the trite "You can always learn something" is true, I think we need more detail. At heart is the tension between the nature of experimentation's trial-and-error process (I prefer the term Edisonian approach) - which means outcomes are unpredictable - and our need to feel satisfaction with our work. Here are a few thoughts.

Skillful discovery. Rather than being attached to a particular outcome, which we have limited control over, I've found it's better to focus on becoming an expert discoverer and mastering the process of experimentation. Because you have complete control over what you observe and what you make of it, you are guaranteed success. Fortunately, there's always room to develop your investigatory skills.

Fixing the game. At first it might seem contrived, but carefully choosing what you measure can help implement a scientific perspective on success. For example, instead of framing a diet experiment as "Did I lose weight?," it is more productive to ask "How did my weight change?" The former is a binary measure (losing weight = success, not losing = failure) and one that you don't necessarily have control over. After all, you are trying an experiment for the very reason that you don't know how it will work out. The latter phrasing is better because it activates your curiosity and gives you some objectivity, what I call a "healthy sense of detachment."

Improving models. As essentially irrational creatures, we run the risk of not questioning what we know. Updating our mental models of people, situations, and the world helps us to be more open to improvements. And the leading edge of that is the conflict between expectation (predicted outcome) and reality (actual results, AKA data). The quantified way to work that is by explicitly capturing our assumptions, testing them, taking in the results, and adjusting our thinking as necessary. This also leads to better predictions; from The Differences Between Innovation and Cooking Chili:

Of course, all of the experimental rigor imaginable cannot guarantee success. But it does guarantee that innovators learn as quickly as possible. Here, "learn" means something specific. It means making better predictions. As predictions get better, decisions get better, and you either fail early and cheap (a good outcome!) or you zero in quickly on something that works.

Getting answers. Another way to guarantee success is by going into an experiment with clearly formulated questions that your results will answer. Structured correctly, you know you will get answers to them. I think of it as regardless of what happens, you have found something out. (Hmm - maybe thinking of the process as active discovery is a richer concept than the generic "you learned something.")

Designing for surprise. If the product of your experiment was not very surprising, then maybe you should question your choice of what you tried. Exciting experiments probe the unknown, which ideally means novelty is in store. Fill in the blank: "If you're not surprised at the end of your experiment, then __."

Zeroing in. Because we usually dream up experiments with a goal in mind, chances are we come out the other end having moved some amount in the direction of attaining that goal. Progress is a success, so give yourself a pat on the back.

Taking action. Finally, each experiment is a manifestation of personal empowerment, which is a major success factor in life. While health comes to mind (do difficult patients have better results?), I think generally the more we take charge of our lives, the closer we get to happiness.

What do you think?

[Cross-posted to The Quantified Self]

Comments

[Jim Stone]

Hi Matthew.

You wrote: "I've found it's better to focus on becoming an expert discoverer and mastering the process of experimentation."

I agree. Possibly our greatest challenge in self-experimentation is our brain's ability to keep us from seeing that which we have no interest in seeing. When we test an idea in which we have a personal interest (say losing weight), and we get some data consistent with the proposition that our intervention has led to weight loss, we can find ourselves suddenly unable to imagine any other reason for the observed weight loss.

We are especially bad at crediting random variation, even though that often explains most of the data we encounter as we experiment -- especially early on. We often start an intervention when at the negative extreme of our range, and will often see improvement just from regression to the mean -- regardless of which intervention we try.

To become master experimenters we need to cultivate good imaginations, humility, some detachment from outcome (at least the ability to go there when needed), some knowledge of statistics, and some experience actually watching time series to get a gut feel for how random data look.

[Andy]

"I think generally the more we take charge of our lives, the closer we get to happiness. What do you think?"

I couldn't agree more. It's almost like when you get a week off from your priorities... and you spend that day doing ABSOLUTELY NOTHING... and feel kinda gloomy afterwards.

On the other hand, even if it's your day off, and you are creating action, even if it's laying around doing nothing, but you do it CONSCIOUSLY, it takes you a bit closer to happiness.

[Matthew Cornell]

@Jim Excellent, excellent points. I think that moment (or moments) of conflict between our internal models and the external reality is very special. It's where we adapt or ignore. There's a 2x2 matrix there: Trying new things that create conflicts (y/n) vs. open to seeing the conflicts (y/n). We need some names for the four quadrants ;-) Also, I agree with the challenge of seeing patterns. We are pattern-recognition machines, but we are configured to be aware of patterns that, in the past, have been useful to survival. They are not necessarily accurate, though. I wrote a bit about it here: http://quantifiedself.com/2010/11/patterns/

@Andy Good point re: we are taking action in all moments, whether we realize it or not. Best that the choices are conscious, I agree.

Thanks to both of you for your comments.

[Arlene Marie Daniels]

This is my favorite line from the post "each experiment is a manifestation of personal empowerment". I love doing personal experiments and you're correct, they're absolutely empowering. Not only do I learn from them, most of the time, I enjoy myself too and get a better sense of pride for myself.

Speaking of experiments and empowerment, have you ever read http://www.howtostopselfsabotage.com? It's a pretty nice book and I think it will be a good read for moments when we lack inspiration and drive to be more experimental. :)

[Matthew Cornell]

Arlene, thanks very much for your comment. It's really good to have a fellow experimenter on board. Taking a bit of pride is completely well-founded. I'll check out the book - thanks for the pointer. If you've read it and have some resulting experiments, I'd love to hear them.

[Michael]

This is a great article...I've always considered life to be an experiment. I love the concept of having a "healthy sense of detachment." All to often we take our self and life too seriously and we don't allow ourselves to simply observe and adjust based on observation. Instead, we see a result, make a story about it and conclude that it's some how the truth.

[Matthew Cornell]

Extremely well put, Michael. Thanks so much for commenting.

[Motivation]

Nice blog. Keep up the good work.

[Joe Lee]

I learned recently about success in business. In encouraging innovation, we have to allow experiments that may lead to failures. In this experiment phase, the variation is huge. But when the experiment found successes, we have to reduce the variations to improve quality.