When we discussed the difference in class of “biological significance” versus “statistical significance” I have to admit that I sleepily accepted this statement without really thinking about it. Sure, just because a p-value is 0.06 doesn’t mean there is no importance to what you’ve found and just because a p-value is 0.05 doesn’t mean that what you found is meaningful. I had thought about the first part of this statement many times before. For example, whenever I get a p-value for something that was just above 0.05 I have a strong desire to talk about the result despite the statistical non-significance. Unfortunately, previous statistics teachers and advisors have put their foot down on a strict greater than or less than 0.05 rule. After reading the Vicker’s chapters for this week, I feel somewhat validated in my urge to discuss these “insignificant” results.
The flip side of this, that just because something is statistically significant doesn’t mean it’s meaningful, is not something I had considered. Having had the p<0.05 rule hammered into my head, I have fallen into the habit of rejoicing whenever I see a low p-value in the results. The examples in the Vicker’s chapters made me realize that rejecting statistically significant findings when they were not meaningful was something I already practiced in my everyday life even though I had never considered it in my scientific endeavors. For example, taking the orange line subway to and from school will get me here faster and I would bet that if I timed it for a few weeks the difference would be statistically significant. But the actual implication for me is insignificant – I would arrive at most 10 minutes earlier than I otherwise would have and the ride would be significantly less pleasant compared to the green line (dirty hipsters from JP versus polite old people going to the symphony…)
Another example is my grocery shopping. My friends criticize me for shopping at Whole Foods on the grounds that it is “sooooo much more expensive.” If I were to run some statistics on that I wonder if it would be true? My response is typically, no it’s really not when you buy things on sale and eat mostly produce! So if I were to do the statistics I bet that the average Joe Shmoe who eats hamburgers and potato chips every day probably would spend a statistically significant larger amount of money at Whole Foods compared to Shaws or Stop and Shop, which would make those friends right. But I could just as easily do the statistics for my shopping and show that I may spend a couple dollars less on non-organic produce at another grocery store, but I’m betting that p-value would be greater than 0.05. Of course, these are both biased samples so who knows what you would find if you did a random sample of shoppers…
But this all brings up another point discussed in the chapters – the p-value shouldn’t determine your actions, there are other considerations when deciding what results mean. If my friends put cost as the most important factor in their grocery decisions then a statistically significant difference in cost is pretty meaningful to them. In contrast, I value the (supposed) health, nutritive, and environmental benefits of organic produce so whether the p-value for c0st is significant or not, I’m still going to buy organic produce.
I should try to keep this in mind as I continue with my research. There have certainly been times when I rejoiced in a low p-value for differences between two groups but when you looked at means and the distributions, the differences were slight and perhaps not biologically meaningful. It all goes back to needing to constantly reminding oneself the question: “are these really different in a meaningful way.”
So what do other people think? Should we as scientists give more weight to near-significant results in our discussions rather than adhering to a strict p<0.05 rule? Are examples of non-meaningful results that are statistically significant as prevalent in biological research as they seem to be in my every day life? Am I crazy for believing that Whole Foods isn’t that expensive?