What's a Good Experimental Design to Test "Oil Pulling"?
Sunday, October 24, 2010 at 9:08AM 
In the Edison experiment oil pulling, experimental enthusiast Diane is testing the controversial idea that swishing oil in the mount for ~20 minutes/day lessens the pain of an tooth abscess. As I wrote in Insights, stories, and surprises from Edison, as the TTL leader I'm working to keep my opinions to myself, and focus on encouraging experimentation and moderate if necessary. I've not had to do the latter until passionate comments put the experiment at risk of being "hijacked" as one Edison user put it.
This brought up the question of whether having a control and, doing "real" science is necessary to get valid results, i.e., to have believable evidence that the technique works.
What I'd like to do here is strengthen my understanding of the scientific method by analyzing this experiment's design. Note that in applying Think, Try, Learn we are limited in how rigorously we apply the method. In general we adopt the spirit of science (hence the simplified "Think, Try, Learn" name) but in this case let's play with how to make the experiment as valid as possible given our limitations, such as having a sample of one. (Note: A forthcoming Edison feature is being able to set up group experiments where more than one person can participate in the same experiment, with the system calculating statistically-valid results. This will be extremely cool.)
Following are some possibilities I thought of. I'm new at this and I'd love to hear your thoughts and suggestions. (Note: In talking about variables I'm using the Science Buddies guide to variables).
In all cases, I'm told we are using an "interrupted time-series" design. From A Short Course on Evaluation Basics:
In situations where a control group is not possible, if (1) data on the treatment group can be obtained for several periods both before and after the participants are exposed to the program, and (2) if there is a sudden change in scores, and (3) if there is a continuation of the change, that is considered to be good evidence of the program's impact.
Jumping right in, our question is always:
Will oil pulling Decrease abscess sensitivity?
To get a good run and to give the swishing time to work, how about we test for four weeks? Two weeks doing the swishing followed by two not doing it.
Second, our dependent variable (What we observe) is abscess sensitivity. In researching oral sensitivity I found Measuring Tooth Sensitivity from The American Dental Hygienists' Association, where they recommend stimulating the area then using a Verbal Rating Scale (VRS) to measure results. The scale ranges from 0 (No discomfort/pain, but aware of stimulus) to 3 (Marked discomfort/pain that lasted more than 10 seconds). How we measure this is a good question given that Diane probably does not want to visit her dentist every week for an unpleasant professional measurement session! How about we decide the same area to stimulate each time, and then once a day (or week) press a rubber tipped gum stimulator like this to the area and measure the result using the VRS scale.
Finally, for our independent variable (What to change) will be whether the swishing is done or not. Note: A clever variation suggested is to make the liquid itself the independent variable, say by swishing two weeks with oil followed by two weeks with water. Our controlled variables (What I keep the same) would be the swishing activity itself, the amount and type of oil used, any oral care she's currently doing, etc.
This still doesn't account for the possibility that during the intervening two weeks there was something else going on in the mouth, which is always changing, but I'm shooting for a good start. What do you think?
A variation: Measure toxins absorbed by the oil
The Oil pulling discussion at The Skeptics Society Forum has plenty of other design and variable suggestions, including this one where instead of measuring pain, you would spit the used oil out onto grass and see if it dies from the supposed toxins that the oil absorbs and removes.
Reader Comments (5)
Dr. F. Karach, M.D., presented a paper before the All-Ukrainian Association. The meeting was attended by oncologists and bacteriologists belonging to the Academy of Science of the USSR. Dr. Karach explained an unusual simple healing process using cold-pressed oils. I quote from online. "When Dr Karach examined the gargled milk-white oil under a microscope with 600 magnification, he saw live organisms.
Testing it on grass to see if it kills the grass is something anyone can do. Be sure to use it against a control though. Do you think a control is necessary to prove an experiment? Are all Edison experiments expected to be done with controls? I heard of someone who implanted a magnet in his/her finger. Did they do a control? Were they expected to do a control?
Hello Matt: (if this is really Matt) What the heck happened to the old Matt's Idea Blog? Gee I don't visit for a few months and this is what is left??
Anyhow, glad to be back here and visit the new site. Does this mean a change in direction for you? What are the goals for this site and why the change? Are you no longer interested in productivity per se? Oh and what about the old blog can I still access it?
As for oil pulling; gee I am kind of at a loss. What about a placebo, instead of oil you try say milk or lemon juice or something? Or is this sort of experiment out of the question owing to the nature of the experiment, one man operation and all that? Maybe I need more informatin on the whole Edison thing...
Anyhow, I am glad to be back for a while after a personal hiatus.
Your friend;
JP in Maryland
> I don't care what others think. I do not feel I need to defend my experiment.
I feel your passion. Hear hear. And your not caring what others think made *me* think about what role your fellow Edison users could play that would be helpful. A non-skeptical support network, say.
> Do you think a control is necessary to prove an experiment?
I think the short answer is yes. To do a more scientific experiment (not that one is required to do so in Think, Try, Learn - not at all) you'd need to do one thing for a while, make a single change (such as *not* doing it for a while) and measure something specific. This is what I tried to sketch out above. Positive results might give a little evidence for or against. A full-fledged experiment would have be done with lots of people, over an appropriate amount of time, using good practices like double blind testing.
> Are all Edison experiments expected to be done with controls?
Great question. I'm still learning about the relationship between rigorous experimentation and Edison. I think there is a range, from "What the hell, I'll just give __ a try." to "I'm out to create a strong scientific case for __." There is room for both and everything in between, but I wonder if we need some way to indicate what kind of feedback is welcome. I suppose for now I will just trust people to say what they want. Suggestions very welcome!
> I heard of someone who implanted a magnet in his/her finger. Did they do a control? Were they expected to do a control?
Yes, that was "Have magnet implanted in finger" ( http://edison.thinktrylearn.com/experiments/show/147 ). I think I see what you mean. That experiment is similar to your point about the oil pulling one: That it's more "Just trying it." than "Trying to find evidence for it".
In both cases, as in all experiments, it starts with a question. In my outline above, the question was "Will oil pulling Decrease abscess sensitivity?" Maybe the more general one is "What will happen if I __." I.e., "if I do oil pulling" or "if I implant a magnet." If the point is the experience of the thing, then any results are valid, right? You did it and you learned something, so you succeeded. With the oil pulling, I think the experimenter really wanted to know whether it works for her or not. Without controls, this is called anecdotal evidence ( http://en.wikipedia.org/wiki/Anecdotal_evidence ) and it is untrustworthy for many reasons. That is why the scientific method was created. Here's a quote from http://www.skepdic.com/testimon.html : "Anecdotes are unreliable for various reasons. Stories are prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on." There's a deeper write-up here: http://www.scientificamerican.com/article.cfm?id=how-anecdotal-evidence-can-undermine-scientific-results
> Dr. F. Karach, M.D., presented a paper ...
Yes, I came across that blurb elsewhere on the net. I believe it's the only study that's been done. I found an analysis of that paper that was helpful to me from the scientific perspective. I've included it below [1].
Thanks again for your comment, Diane, and for your questions about the method. I encourage you to keep trying stuff!
[1] http://www.skepticforum.com/viewtopic.php?f=54&t=5098&start=160#p119564
An interesting read. This is only a preliminary study, with 10 participants over 45 days. Each had to swish refined sunflower oil in their mouths for 8-10 minutes a day, early in the morning light, on an empty stomach. A significant reduction in "plaque score" and "gingivitis score" was found.
The major problems I see with this study are:
1. No control group. The participants were limited to those who did not perform any other type of oral rinse. It's not clear whether the same effects could have been achieved by swishing water around. There is a discussion at the end comparing their reductions to those in other studies, but it is not at all clear how their scoring method compares.
2. Not blinded. The participants were supposedly blinded to the purpose of the study, but this seems unlikely to have been effective, given all that the patients had to be instructed on. The experimenters were completely unblinded.
3. Single, unblinded subjective evaluation of the results. A single person evaluated the results for all subjects on flexible, subjective criteria. The evaluation of repeatability showed that the initial/subsequent evaluation of the same tooth on the same day had a correlation of .81 for plaque and .75 for gingevitis. That doesn't seem very reliable. (But I'm no expert.)
All in all, I'd say it's not a meaningful study. Maybe they can use it to get funding for a real study on these effects.
----
JP! Long time no see.
> Oh and what about the old blog can I still access it?
It's all still here. You can find the index by year here: http://www.matthewcornell.org/blog-index/
> Does this mean a change in direction for you? What are the goals for this site and why the change? Are you no longer interested in productivity per se?
Yes, a big new direction for me. I said a little about it here: Welcome to The Experiment-Driven Life blog - http://www.matthewcornell.org/blog/2010/10/4/welcome-to-the-experiment-driven-life-blog.html
There were a number of reasons for this shift:
o The terrible economy means no one is hiring productivity workshops or consulting.
o I've studied most of the time mangement topics, and felt I have a solid handle on them. By now I know my pattern: Once I run out of stuff to learn, I lose interest unless there's something else happening. Having business, for example.
o I don't think I'd ever be able to make a major contribution in productivity that I'd feel happy with. THe value is more in perspective and packaging than it is in content, I've come to conclude. (I will say that I am proud of my daily planning guide, which I think is novel partly because it applies my experimentation ideas to the practice.)
o I don't think I'd ever be able to stand out in the time management realm. Again, packaging and marketing vs. having a significantly different approach.
My Think, Try, Learn work on the other hand feels like the right thing at the right time, and seems to have far more prospects than the above. I'm super excited about it. Initial data coming back from the experiment are highly positive. :-)
I haven't shared this publicly, but I have a three-part plan to spread the life-as-experiment meme:
o Write the Think, Try, Learn book that teaches the self-experimentation mindset and practices.
o Develop a software platform to support living life this way, starting with Edison, the TTL Experimenter's Journal ( http://edison.thinktrylearn.com/ ).
o Become The Guy in the self-experimentation world by thinking, writing, and presenting.
We'll see - it's an experiment!
had a difficult summer, my dad passed away at age 89. We were back and forth from hospital to rehab, etc. for four months. He had cancer but I really thought he could pull through because he was so rugged and very positive right to the end. His father my grandpa was 97 and i thought dad might be able to reach it. Still we gave it our best shot.
I really wanted to ask you more questions about productivity, I see myself applying this stuff all the time in my life. Added with a touch of eastern religion, cognitive therapy, marcus aurelius and such. It all starts to make a bit of sense.
Well, I have to admit I am somewhat at a loss. I am actually thinking of geting into productivity at least part time. But I appreciate your honesty and insight into the current state.
talk to you soon. JP
I do love science and have a degree so I might be able to contribute.