The Experiment-Driven Life
Thursday, June 24, 2010 at 6:01PM 
In his seminal article The Data-Driven Life, Gary Wolf [1] introduces us to the field of personal informatics and summarizes the self-tracking movement to date [2]. I was excited to discover this piece because it touches upon concepts I've been experiencing, collecting, and brewing for the past five years.
Wolf approaches his main point - that self-tracking is a tool of self-discovery - by covering what I've extracted as four main areas: General points, Science, Personal development, and Social aspects. Below you'll find my bulleted summary of them (read the full write-up if you have time) followed by how we see our Think, Try, Learn (TTL) work neatly binding together his data-centric ideas into a new way of looking at the world, that of a scientifically-oriented life as experiment. My thinking here is a work in progress, so please, share your thoughts.
Central ideas
General points
- Humans have blind spots and make decisions with partial information. A better alternative is to use data.
- Early adopters have started measuring in earnest and using experiments. This "pathology of quantification" is (currently) considered abnormal.
- Data has value: It facilitates tests, comparisons, and experiments; it increases awareness and the feeling of self-mastery; and can lead to insights and change.
- Quantitative data is best, rather than a journal or therapeutic talk.
- The process is to start with a question, gather data, "live by it," and repeat, possibly with newly-discovered questions.
With respect to science
- Most self-exploration is not principled. Better is to gather data, record dates, toggle conditions, and keep careful records of outcomes.
- Personal experiments are not clinical trials. Instead the goal is understanding and improving yourself, not the entire species.
- Measuring and then changing conditions is different from anecdotes. The former is more like science and goes to testing a theory.
- General knowledge never applies perfectly, so self-tracking is valuable for individual applications.
Applied to personal development
- Human behavior is mysterious, and our reasons for doing things are opaque.
- Applying self-tracking to personal development is new, but becoming popular.
- Our lives generate much data.
- Goals are often unknown. Some people start with questions, but most trust that data has value and might lead to unexpected questions.
- Data tracking reduces emotional qualities (e.g., shame) and increases intellectual/rational ones.
- A central hypothesis is that many problems come from lacking instruments to understand who we are.
- Technology (especially mobile phones) makes numeric tracking easier and more attractive.
- Beware bad science (biases), but intentionally ignoring data is useful (e.g., can lead to forgiveness).
- Beware tying self worth to data because it can lead to judgement, etc.
Social aspects
- Social media makes data sharing encouraged and accepted.
- We like to share, and having data makes sharing natural.
- Sharing affords opportunities to help.
Toward a cohesive framework
Wolf covers a lot of ground, and all of his points are important. At the same time, we think they could benefit from some unifying. To address that, we see two main areas missing from the discussion: a true scientific perspective that encompasses personal tracking, and a comprehensive social platform that supports meta conversations as a central part of our experiments.
Let's look at these in turn.
A scientific method for personal development
While recording and tracking personal data is an important and natural first step, it's only one part of what needs to be a more scientific process. Our Think, Try, Learn work aims to do just that: pull current data-centric approaches together into a new philosophy of life - a kind of personal science that organizes the act of observation (the true role of collecting data) into the larger life-as-experiment perspective. In other words, to move us from a data-driven life to a wider experiment-driven one. We've started putting these elements together into our book [3], but let me share a few of the concepts as they relate to Wolf's piece.
- Method: As pointed out, we need a personal scientific method, but we've found that applying a rigorous interpretation [4] to individual improvement presents significant challenges, such as forming concrete theories, repeatability, and the "N = 1" problem [5] to name a few. While studying this we've discovered a simpler version that works well - what we call the Think, Try, Learn cycle (see the process diagram above). It encapsulates the spirit of the method without turning away people who have little scientific background but could still benefit from the perspective.
- Mindset: Looking at everything you do as an experiment takes a shift in thinking, such as the definition of success, what failure means, and roles mistakes can take. Important concepts include asking questions (vs. having answers), keeping our eyes fresh (vs. biased filters and "expert mind"), and cultivating a healthy sense of detachment (vs. being attached to particular outcomes). "Baking" this into a culture is an opportunity and a challenge. To that end, in addition to the book, something we're currently trying in Edison (our TTL experimenter's journal) is to get our users thinking about this approach from the start. We do so by asking leading questions that people answer when creating new experiments:
- What will you do?
- How will you test your idea and measure success?
- How will you know you are done?
- How will you enjoy the journey?
- Practices: Knowing a method and putting it into practice are different dimensions, and they must be integrated to be effective. We are looking at things like how to work the TTL cycle, keeping the process agile ("test often and fail fast" as Pam puts it [6]), and of course rigorous observation and capture using techniques and tools like Wolf describes.
- Process: Many tools focus on the data itself (product), rather than the act of collecting it (process). However, important growth and character development happens during experimentation, in some cases overshadowing the result. This explains why we think personal informatics sites without user commentary don't capture the data's greater context. After all, this is where our stories [7] are created, encouraged, celebrated, and shared.
- Enjoying the ride [8]: As Wolf points out, there's a risk of getting too caught up in self-measurement. For balance, it's ideal to take pleasure from what we try. No philosophy is complete without addressing this, so how do we enjoy the experimental process itself? TTL practitioners do so via the acts of paying attention, staying curious, relishing discovery, being playful, cultivating a sense of humor, and celebrating. Lest you think science is all seriousness, I like Lewis Thomas's observation that you can tell when something important is going on in an experimental lab by the laughter.
- Collaboration: The best learning takes place with the help and wisdom of others, which makes it crucial to create a network of fellow experimenters. More on this next.
- Lifelong learning: Finally, science should be a lifelong discipline - something we integrate deeply into our lives and work to master. Even when you fail to reach a goal you can always be successful at "being excellent at discovery" (as one TTL practitioner put it). This is the essence of what Wolf calls "our quest to figure ourselves out." How rich that is!
A comprehensive social media for experimentation
Along with a scientific method for personal development, we need a social platform that captures not just data, but something possibly more important: the conversations about the experimentation process itself. It can be argued that this is where insight actually takes place - where lessons are learned and character is built.
Regarding existing self-tracking sites (of which Wolf mentions many), it helps to think of three dimensions (see the diagram to the right):
- Domain: A range of topics from the highly specific (such as your sex life or mood) to virtually anything in your life (such as DAYTUM or FlowingData's your.flowingdata).
- Philosophy: A scientific range from pure observation (Nicholas Felton is a master) through personal experiments (I have no good examples; maybe CureTogether?) all the way up to clinical trials (PatientsLikeMe).
- Context: A social range from individual to group.
We think the sweet spot, which hasn't yet emerged, is the intersection of all three dimensions. That is, a platform that is broad, social, and scientific. As mentioned above, Edison and the eventual TTL Platform are our first baby steps in that direction. (Yes, there's a lot of work ahead, but we're trying to live the TTL principle of start small.)
Summary
To wrap up, we believe the exploding self-tracking movement is an exciting step in the broader direction of a personal scientific philosophy - one that leverages the set of techniques developed over the last 400 years that have given us the most powerful way to approach the unknown: science. The technology that supports this should be a software platform that tightly integrates the this experimental perspective into a social tool that spans collaborative investigations across all domains. This, we argue, could move us from a data-driven life to an experiment-driven one. And how amazing would that be?
References
- [1] Wikipedia entry, his company AETHER, and his Quantified Self work.
- [2] Other influential articles on the personal informatics movement include The New Examined Life by Jamin Brophy-Warren, Wolf's Wired article Know Thyself: Tracking Every Facet of Life, from Sleep to Mood to Pain, 24/7/365, and Matt Jones's presentation Polite, Pertinent, and... Pretty: Designing for the New-wave of Personal Informatics.
- [3] Tentative title: "Think, Try, Learn: A scientific method for discovering happiness."
- [4] The steps typically mentioned when talking about the method are something like the following. Typically not all steps occur in every case, and sometimes they are out of order.
- Make an observation.
- Ask a question.
- Formulate a hypothesis.
- Make a prediction, which will explain our results.
- Conduct an experiment.
- Analyze data and draw a conclusion.
- Repeat the work.
- [5] Small sample size. It's too complicated for me, but my academic colleagues know the details, and ways around this. These will be crucial when we start running trials on our platform.
- [6] This is from Pamela Slim's fine book, Escape from Cubicle Nation: From Corporate Prisoner to Thriving Entrepreneur.
- [7] I wonder if a useful measure of a full life is the number of stories we create and participate in. Fortunately, we believe that all experiments generate stories. For example, who wouldn't want to know about implanting a magnet in your finger or going into a job interview cold turkey?
- [8] I first encountered the phrase in Patricia Ryan Madson's book Improv Wisdom: Don't Prepare, Just Show Up (maxim #13).
Reader Comments (9)
As always, Matt, lots to comment on here, but I'm going to limit myself to one observation from the Edison site, which has to do with the difference between measuring results anecdotal-ly and measuring them objectively through double-blind tests. Those, I think, represent the two ends of the "evidence spectrum."
I have a friend who's a medical doctor, and one of our ongoing discussions has to do with the effectiveness of alternative therapies: we're both well past the point of saying either "this stuff is crap" or saying "there may be something to this." It's tempting to dismiss anything short of a conclusion reached through a double-blind test, but on the other hand we all live in the real world of anecdotal evidence, by which I mean to say that, consciously or unconsciously, we make decisions about how to live based on anecdotal evidence. If you think St. John's Wort reduces your depression, it doesn't matter very much whether a double-blind test says it's useless. A scientist would say that "a scientific method for discovering happiness" (a tag line I love, by the way) that relies on anecdotal "proof" is in no way scientific. But as a practical matter, the anecdotal outcome is all that matters to people in their everyday lives.
I'm also struck by the ways in which TTL dovetails with Cognitive Behavioral Therapy. CBT is emerging as a way to teach people practical techniques for addressing all kinds of problems in their lives. Among other things, it's a way of separating the emotional component of everyday problems from the practical, skill-based, practice-oriented behaviors that give rise to both problems and effective solutions. That's TTL.
> measuring results anecdotal-ly and measuring them objectively
> ...
> consciously or unconsciously, we make decisions about how to live based on anecdotal evidence
This is an excellent point, and one that I also think about - starting from the TTL beginnings. On the measurement side, we started with simple text-based observations, which support the meta conversation as I described above. We'll be supporting quantitative ones in the future, which are essential for the third component of our platform, Einstein, which I've not yet written about. Briefly, it'll be the module that does actual trials, using statistical analyses to form valid results that will go beyond anecdotal observations. That said, as you point out, there is an important aspect of this that is completely personal - "Does this work for *me*?" Edison and Einstein will together help answer the question "Where do I start?" and crucially, "What works for people like me?" so that users can ultimately get help on what they care about, i.e., to answer the first question. This could be big, as it will span anything that people are trying. Full spectrum lighting, Feng Shui, and the rhythm method.
> TTL dovetails with Cognitive Behavioral Therapy
I've been hit on the head with this enough times that I need to pay serious attention to CBT. Thank you. Some links from a person who created an iPhone CBT app (http://www.cbtreferee.com/about/):
* http://en.wikipedia.org/wiki/Cognitive_behavioral_therapy
With the help of a therapist, patients may even devise and carry out behavioral experiments that can help them to learn how to improve their quality of life. [# ^ a b Kingdon D and Price J (April 17, 2009). "Cognitive-behavioral Therapy in Severe Mental Illness". Psychiatric Times 26 (5). http://www.psychiatrictimes.com/paranoia/article/10168/1406055?pageNumber=2. ] -> A collaborative examination of the evidence offered in support of beliefs can be followed by inquisitive planning of simple experiments.
* http://nacbt.org/whatiscbt.htm, http://nacbt.org/historyofcbt.htm
[Feeling Good: The New Mood Therapy Revised and Updated | http://www.amazon.com/gp/product/0380810336?ie=UTF8&tag=masidbl-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0380810336 ]
* http://www.beckinstitute.org/InfoID/220/RedirectPath/Add1/FolderID/237/SessionID/%7B1DF92A00-D062-42A7-99F5-2CB8ABDFE80F%7D/InfoGroup/Main/InfoType/Article/PageVars/Library/InfoManage/Zoom.htm
So it is not a situation which directly affects how a person feels emotionally, but rather, his or her thoughts in that situation.
> Quantitative data is best, rather than a journal or therapeutic talk.
Best for what? While I certainly am of the opinion that quantitative data is useful and lends itself to relatively easier analysis and interpretation, this does not rule out the importance of qualitative data. And certain measures are difficult, if not impossible, to quantify. Tracking the effect of a certain treatment on scar healing, for example, would be qualitative. Additionally, in certain situations, qualitative data can be considered more important than the quantitative. Like in fat loss - of course it's important to track your body weight, body fat percentage and so on... but taking regular progress pictures just puts the quantitative data into perspective. A picture is worth a thousand numbers, so to speak.
As to the importance of journaling: while not ideal for self-experimentation, it can serve as an important reference point over time, and it's possible to quantify the data in them at a later time. Like (lucid) dreaming experiments - if you log your dreams every night, you can later analyze those logs and quantify common themes by frequency of occurrence, and so on.
> Measuring and then changing conditions is different from anecdotes. The former is more like science and goes to testing a theory.
Well, by my definition, the former IS science - science being the discovery of cause-effect relationships by formulating testable hypotheses. (And testing them, of course.)
[follow-up] BTW you might want to check out "How To Measure Anything" by Douglas W. Hubbard. Alex wrote about it here: http://www.kk.org/quantifiedself/2010/08/how-to-measure-anything-even-i.php . I found it at http://halfchai.files.wordpress.com/2009/07/how-to-measure-anything-finding-the-value-of-intangibles-in-business-douglas-hubbard.pdf
Well, smell and taste seem to be qualities that can't be measured to any degree of meaningfulness. Even then, you could rate foods on a subjective scale and get numbers - but this is not true quantitative data in the sense that weighing the food would be. The tool of measurement in the case of such pseudo-quantitative data would be ourselves, and thus, there is a certain degree of subjectivity inherently present in the numbers. This is classically present in all sorts of psychometric and mood tests.
>Re: fat loss taking pictures is a nice idea - a good motivator of tangible progress. Then again, assuming you calibrated the process, you *could* use the photos to measure - intelligence agencies do this.
Sure, it's possible... if you have access to government programmers and the rich resources of intelligence funding. The question is, would the calibrated analysis of photos provide any additional value beyond the measurements one can take in a more simple manner (weight, body fat percentage, circumference of body parts)? And that would be missing the very point - the point of taking progress pictures does not lie in their potential for gathering quantitative data!
Thanks for the interesting discussion and links. I've put "How To Measure Anything" on my reading list, it looks promising.
Definitely. Even if it's not perfectly objective data, it can be useful data. We work with what he have, right? Thanks for the thoughtful discussion, Syler.
Perhaps I've misunderstood the notion of "clinical trials"
I adamantly hold the perspective that all our life experiments not only improve each ourselves but also be a contribution to humanity.
This gives powerful context for action & perserverance
In effect, we are all making our experiments & conclusions public - thus leaving footprints & a legacy for others (even beyond our time) to follow.
If something else was meant by the above statement, please clarify.
Great job on the manifesto!
http://autobiography.chinarut.com
it's public :)