AIC Insecurity (Plus a Black Hole Simulation)

Following the AIC readings, I can’t but help feel the same level of insecurity regarding just how much our models can tell us about the world around us. Even though AIC allows to choose the best model from a set of candidates, this measure is still limited by our a priori understanding, and the old adage “garbage in, garbage out” applies once more. AIC can only tell us the best relative model, but if we fail to include any good models, it will only be able to tell which of our bad models comes closest to being analyze the data effectively. This is sci-existentially terrifying. Incredibly, even with this fancy new tool, we can only be so certain that the particular model and/or method that we use to analyze our given data is correct. I’m sure that my thoughts on the subject will change over the course of next week, although I suppose my insecurity over choosing the right models to use with AIC will remain.

On a  completely unrelated note, I came across an interesting example of the usefulness of simulations for revealing new insights regarding scientific phenomena. I’ll leave a link to the article below, but in short a group of astrophysicists and special effects artists were able to generate a physically/mathematically correct simulated black hole that turned out to be the most accurate visualization of a black hole to date. In fact, some aspects of the visualization that were initially thought of as bugs in the program turned out to make sense in a physics context, and led to new insights into the appearance of black holes. Overall, it’s definitely an interesting article that I would recommend checking out.

This entry was posted in Uncategorized. Bookmark the permalink.

One Response to AIC Insecurity (Plus a Black Hole Simulation)

  1. lficarra says:

    I find that the more I learn about how a particular instrument works, what the theory is behind a new statistical tool, or what the methods are for a particular research experiment, the less confident I am about the result it produces. I often hear researchers (and even myself) state some iteration of: the more you learn, the more you realize you don’t know. How can anyone get anything done when scientific literature is so full of caveats? Research is not simply to answer questions, but to discover new questions. A big part of it is realizing how little we actually know. This can be very difficult to grapple with, but uncertainty is everywhere.

    Statistical methods come with their own uncertainties, yet help create some order out of our chaotic data. The AIC can help us choose appropriate models to use, but with all statistical tests, biological relevance must be taken into account. The AIC is no exception to this rule. Providing candidate models for AIC reminds me of setting a prior for bayesian analysis. Prior knowledge of the system is used to narrow down potential models. Then, the AIC estimates the appropriate model (just like bayesian analysis incorporates the prior to explore the probability surface).

    While this requirement for a priori knowledge of the system may be necessary for a good model pick, the amount of a priori knowledge available may serve as a measure of uncertainty for that choice. So what I mean is if you know little about your study system, you know there will probably be high uncertainty in the AIC model choice. If your candidate models are backed by strong a priori knowledge, there will probably be much less uncertainty in the model pick.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s