Frozen in indecision

I wanted to talk a little about designing experiments, given the Hurlbert paper and our class discussion. The problem I have been coming up with is that the experimental methodology I lay down in the next few weeks is the one that has to be replicated over the next 3-4 years. I find myself frozen with inaction – asking questions at every turn (Is this a good reference site to compare with? How far apart do the sites need to be to constitute independence? Is my sampling design adequate to cover the entire population? Is this all just psuedoreplication?) There is also the goal of the statistical ideal – which isn’t always apparent ahead of time – and even when it is, it runs head-long into the real world of time/money/bureaucratic regulations. It doesn’t help to know that the examination of a time based phenomenon (in this case, an intensive system-wide restoration) is a one-shot deal – screw it up and you have nothing worth comparing at all.

On the other hand, there’s a weirdly comforting thought that accompanies our discussion of ANCOVA. The notion that even if things aren’t ideal, even if your sites arent completely independent, statistics is a living field which can help you to work with what you have and get the most out of the data you collected. While this isn’t a license to half-ass a design, it does help you to breathe and put the pencil back to the paper – start with your question, describe your conceptual model, and design for the best case scenario. Most importantly, be candid about the aspects in which your design is good and in which it is weak.

Methodology to me seems terrifying because in published papers, it is presented as an extremely polished, clear cut design that seems to have been self evident to the authors. In reality, it was probably the result of significant thought and recognition of real world limitations. Thinking that a bad design can dash your hopes and dreams can be paralyzing to the point of inaction, but the counter point is trusting that good practices and good faith can result in some form of a result, even if its not as powerful as you might have hoped when starting out.

This entry was posted in Uncategorized. Bookmark the permalink.

4 Responses to Frozen in indecision

  1. Genevieve Davis says:

    This is a really great point and one I think many of us don’t get to until after data collection- once we realize how great it would have been if we had done something differently, or collected more information on one aspect. The good thing is with experience, we gain a better understanding and more input on how we want to design our experiment. One thing I find amazing is that not all published papers seem valid in their experimental design. There have been publications about populations of species, where they have collected data for only one, or less than 10 individuals. It’s amazing to me that one subject is enough to infer any information, but like you said, beautiful statistic presentations help paint a picture, and let us understand how we can interpret extremely small sample sizes.

  2. jebyrnes says:

    As the old Fisher quote goes, “To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of.” That you’re thinking about these things is excellent. I can only offer two pieces of advice. First, a solid experimental design, created to cope with the issues you specify to the best of your biological knowledge is never going to go wrong. Unless it does. In which case, if you’ve designed it well, you will hopefully be able to salvage something useful from the wreckage. Sometimes, better than useful – something new.

    Second, power power power. If you create a design with an eye towards how you will analyze it, then, great! You should be able to fake some data up right quick, and do a little power analysis to see if your design is at all good. This kind of planning will save you mo’ money, mo’ effort, and mo’ stress than you’d care to think about. And it will lighten the load on that indecision considerably!

    Because, while we haven’t been talking about power and simulation with some of the latest techniques, it hasn’t gone away. All of these parameteric analytical models can be simulated and evaluated for power based on your design! Heck, you can even fold in varying degrees of non-independence, and try out both ‘proper’ and ‘improper’ model structures. How bad will non-independent data foul up your lm, if you assumption of independence turns out to be wrong?

    It’s all there, right at your fingertips!

  3. martinew says:

    I’ve been very grateful recently for having a solid experimental design in the face of tricky data. My data is unbalanced and some of the response variables are not normally distributed and can’t be transformed. The analysis has been somewhat of a headache but I keep on wondering how much worse my situation would be without a good framework to test my data under. That said, what I call a solid experimental design, others may not. I think as long as you make an honest attempt and are able to quantify your choices, as well as plan out your analyses before hand, you should be on the way.

  4. lauraganley says:

    This blog has been strangely therapeutic. I have been frozen myself recently when dealing with my project. For days I was reading and reading and reading trying to figure out exactly what to do. I used to love that whales are individuals, all with a history of their own; animals that I “get to know” over time. Now, the fact that they’re individuals over time makes them temporal pseudoreplicates, and maybe random effects too? Ugh, all this now just makes them another thorn in my side. It wasn’t until today, while reading what you said about putting pencil to paper, that I realized it had been days since I actually opened R studio. So, I decided to break my streak and start throwing some variables around. There was definitely some reckless programming going on, never a good thing, but I feel like I’ve made more progress in the last hour than I have in the last three days. Here’s to thawing out a little….

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s