The AstroStat Slog

Astronomers confront with various censored and truncated data. Often these types of data are called after famous scientists who generalized them, like Eddington bias. When these censored or truncated data become the subject of study in statistics, instead of naming them, statisticians try to model them so that the uncertainty can be quantified. This area is called survival analysis. If your library has The American Statistician subscription and you are an astronomer handles censored or truncated data sets, this primer would be useful for briefly conceptualizing statistics jargon in survival analysis and for characterizing uncertainties residing in your data.

Survival Analysis: A Primer by David A. Freedman
The American Statistician, May 2008, Vol. 62, No.2, pp. 110-119

This article explains the basics of survival analysis and adds criticisms on previously conducted studies. Since the given examples are from medical studies, astronomers may not be interested in reading the whole article. Nonetheless, Freedman offers the definitions in survival analysis such as survival function, hazard rate, the Kaplan-Meier estimator, the proportional hazard model with clarity and conciseness. For example, if τ (a positive random variable indicating the waiting time for failure) is Weibull, the hazard rate takes an exact form of the celebrated power law in astronomy (I think modification of pdfs reflecting censoring and truncation may lead more robust results compared to fitting power laws unless parameters in power laws have astrophysical implications and survival analysis approaches cannot perform the same parametrization).

Commonality between power laws and Pareto distributions and frequent appearance of power laws in astronomical journals drives some anticipation of frequent applications of survival analysis to astronomical data; on the contrary, there are not many.

Though there are more, here are a few references relevant to survival analysis, that utilized examples from astronomy or appeared astronomical journals:

• Nonparametric Methods for Doubly Truncated Data by B Efron and V Petrosian. (subscription required)
  Journal of the American Statistical Association, Vol. 94, pp. 824-834 (1999)
• Survival Analysis of the Gamma-Ray Burst Data by B Efron and V Petrosian. (subscription required)
  Journal of the American Statistical Association, Vol. 89, pp. 452-464 (1994)
• A simple test of independence for truncated data with applications to redshift surveys by B Efron and V Petrosian
  ApJ, Vol. 399, pp.345-352 (1992)
• Statistical methods for astronomical data with upper limits. I – Univariate distributions by Feigelson and Nelson
  ApJ, Vol. 293, pp.192-206 (1985)
• Nonparametric Estimation of the Slope of a Truncated Regression by Bhattacharya, Chernoff, and Yang (subscription required)
  The Annals of Statistics, Vol. 11(2), pp. 505-514 (1983)

Note that these papers only dealt particular statistical interests with an general introduction about survival analysis and definitions of estimators based on relatively small sample size data sets. Facing massive survey data with truncation and heterogeneity in measurement errors in astronomy could open a new era of survival analysis.

Lastly, there are studies regarding Pareto distribution some of which are presented in the slog. (Use “search” with Pareto. More statistical papers on survival analysis in astronomy are welcome to be added; please, inform me.)