The AstroStat Slog

Over the few years, at the heart of astronomical researches, I see astronomers treat statistics like a magic crystal.

Like a magic crystal, statistics

• is a easy way to tell (future) results. However, astronomers do not need to know how it works.
• can give a wrong prediction. Well, what can we expect from a magic crystal (statistics)? One is rather not trust it with 100% confidence. Even if I used to for this unexpected inconsistent result, this magic crystal has been performing decent jobs previously. As one expects, a magic crystal can say wrong stuffs from time to time.
• A magic crystal does not take a long time to tell the future. Waiting in a line for the fortune teller or the decision making process of which fortune teller to visit requires more time.
• Instead of modifying and redesigning the magic crystal in order to get statistically valid results, astronomers often attempt testing various crystals and with different questions (brute force Monte Carlo and various types of binning for the chi-square method, for example).

Some astronomers know statistics is not a magic crystal but a science covering broad needs. They understand statistics requires data matching assumptions to be fully utilized. As astronomers spend years to build telescope and to process data, statisticians also need time to brood on mathematical theories and modeling procedures. I hope the other astronomers, who think statistics as a magic crystal, consider consulting statisticians and literature of other science fields with statistical methodology (like meteorology, where spatial temporal models and stochastic processes are common). Often times statisticians have to develop new methodology because astronomical data are quite different from those of bio and medical science. It takes long until the mutual understanding between statisticians and astronomers on the data is firmly built. As if astronomers are not astrologers, although I met people who cannot tell the difference, statisticians are not fortune tellers. Think about economists. I hope that astronomers listen statisticians’ ready built never used crystal although protocols of design experiments do not match astronomers’ data and their requests at the beginning (time series data that I mentioned, for example. I believe the data were presented different ways for more statistically rigorous analysis).

I don’t want to fall in any kind of generalization fallacies such as a comment by an astronomer Bayesian is robust but frequentist is not^[1] by saying that “statistics is not a magic crystal. ” Nonetheless, I want astronomers to investigate what is really in those magic crystals they often use. I want them to attempt to break the crystal to know what it is and to change the composition of the magic crystals to please clients (astronomers yourself). Gladly, there are some who put extra efforts for that purpose. Otherwise, be patient with your fortune teller/psychologist to design the right crystal/tools for you instead of treating statistics like a magic crystal.

1. Bayesian can be robust than frequentist methods because of its flexibility of hierarchical modeling but it is not a sufficient condition for being robust. Such statement hurt me a lot and I suddenly felt a sympathy to fortune teller who relies on his/her crystal and all of a sudden the client expresses a great deal of mistrust because the magic crystal is so mystic and uninspectable[↩]