I used to do research on fertilizers, specifically this new fancy organic fertilizer that contained amino acids as an N source. (Yes, plants can assimilate and use amino acids as part of their N nutrition). My experimental design was textbook: plant a bunch of seedlings, create some plots, and randomly assign them treatments. My treatments were of course different rates of the fancy fertilizer, a positive inorganic control, and my negative control, which received zero nutrient input. Then I’d spend an entire grueling hot Michigan summer fertilizing, weeding, measuring things like photosynthesis, threatening to break the machine used to measure photosynthesis, measuring trees, and collecting samples like foliage and leachate. (Don’t get me wrong, I flipping loved doing this minus the endless technology issues…we all know I’m so good with those)
So after doing all of this for four months, then I’d live in my lab for two months doing the “chemical s*$#” as my fellow grad student called it. And after that, I’d spend a month sorting through my data and running statistics. Then the day came (much later than my adviser wanted) that I had my end product and almost nothing was significant in the statistical sense. At first, I was disappointed because this fertilizer was “supposed” to be “soooooo much better” and because I was freaking out about how I was going to write a thesis when all I could say is that the seedlings that got some N did better than the seedlings that didn’t get any. But then I realized something—the fact that I had no differences between the seedlings fertilized with the fancy fertilizer and the inorganic fertilizer was still a result. I just couldn’t reject my null that the seedling performance under amino acid fertilization was similar to that of inorganic fertilization and that wasn’t necessarily a “bad” thing.
I still had results, I still successfully wrote my thesis, and some of those chapters are published. One of my reviewers even said that the statistics were sound (boo yah!). So why do we put such an emphasis on the p-value? It doesn’t always mean that your results are meaningless. If anything publishing your non-significant results and sharing them with the scientific community is good because it lets them know that you found “nothing interesting” as Vickers would say and it gives someone else the information they’d need to keep building, and that’s what science is really about—building. And really, I shouldn’t have quoted Vickers there because my results were interesting and I got to spend months in the library trying to figure out probable physiological explanations as to why I didn’t observe differences. There is still valuable information rooted in non-significant results and many interesting hypotheses that can stem from them as well. I like to think of my experiences as proof of this fact.
Hurlbert and Lombardi point out that all of these super high impact factor journals favor publishing papers with the most significant of results, and I want to scream WHY? I have already brought up the point that non-significant results still can be informative and new ideas can be generated from them (THAT IS WHAT SCIENCE IS ABOUT, NOT A DAMN NUMBER!!). There may be a point to the fact that the findings tend to be less conclusive and novel in the absence of significance, but I seriously don’t believe that this would always be the case. The fact that they shun us who cannot produce significant results could be dangerous. It could lead statisticians to use their knowledge for evil instead of good by manipulating their data to tell a story of significance. It could redirect the path of science so people just focus on conducting experiments with a more assured chance of significance. A p-value is a p-value, it should not have the capacity to undermine the value of results. And at the end of the day, the absence of significant differences is still a result.