HST Primer for Cycle 22

7.1	Routine Science Data Processing

7.1.1 Space Telescope Science Data Analysis System (STSDAS)

Science data obtained with HST are sent to the TDRSS satellite system, from there to the TDRSS ground station at White Sands, New Mexico, then to the Sensor Data Processing Facility at Goddard Space Flight Center in Greenbelt, Maryland, and then finally to STScI.

At STScI, the production pipeline provides standard processing for data editing, calibration, and product generation. These functions, performed automatically, include the following:

•	Reformatting and editing of data from the spacecraft packet format to images or spectra.

•	Performing standard calibrations (flat fields, wavelength calibrations, background subtraction, etc.) with best available calibration files.

•	Producing standard data output products (FITS format files of raw and calibrated images, OMS [jitter and performance flags] files).

Standard calibrations performed on HST data, and the resulting output data products, are described in detail in the HST Data Handbook.

In 2015, major upgrades to the Science Data Processing Pipelines (DP) and Data Distribution System (DADS) were installed. Archive users will now find HST data stored on disks for immediate download. Data will be reprocessed, as needed, when updated calibration reference files or improved calibration algorithms are received to ensure the freshest data are available for users. Data from all non-active instruments (HSP, GHRS, FOC, FOS, WF/PC, WFPC-2 and NICMOS) continue to be available for direct download. In the event a dataset is requested when those data are queued for reprocessing, the user will experience a delay as those data are processed through the system at a higher priority before being delivered.

7.1.1 Space Telescope Science Data Analysis System (STSDAS)

The Space Telescope Science Data Analysis System (STSDAS), and its accompanying package, TABLES, provides access to applications for the analysis of HST data as well as various utilities for manipulating and plotting data. The TABLES package facilitates the manipulation of FITS table data. STSDAS and TABLES were originally layered onto the Image Reduction and Analysis Facility (IRAF) software from the National Optical Astronomy Observatories (NOAO). Both packages run from within IRAF and are supported on a variety of platforms, although not all of the platforms that IRAF supports.

All the existing calibration pipeline programs used by STScI to process all HST data can found in the HSTCAL package, a package which allows users to compile and run the calibration software originally written in C without using or running IRAF. The latest versions of all the active instruments’ calibration software no longer relies on IRAF and has been installed as the HSTCAL package for Archive and pipeline operations. HST observers can use the programs in HSTCAL to recalibrate their data, to examine intermediate calibration steps, and to re-run the pipeline using different calibration switch settings and reference data as appropriate.

These HSTCAL software versions replace the calibration software available under STSDAS; specifically, CALACS for ACS, CALWF3 for WFC3, and CALSTIS for STIS.

Much of the newer calibration and analysis software is written in Python, including CALCOS for COS data. It does not require IRAF, and is available as part of the STScI Python library included and installed as part of Ureka releases.

PyRAF is an alternate Python-based command-line environment that enables users to run IRAF tasks and allows the new Python-based software to be used, along with IRAF tasks, with IRAF command-line syntax. Ureka includes astropy, a set of packages that provides Python programs the ability to read and write FITS files and read and manipulate the WCS information in image headers. The new Python environment allows users to manipulate and display data in a way not possible with IRAF that is more akin to how data can be manipulated and displayed by IDL (Interactive Data Language). The STScI Python environment described above is contained within the stsci_python package. Detailed information on STSDAS, TABLES, PyRAF, Ureka, and other Python-based software, including the actual software, is available from the STScI Software webpage. Information about IRAF is available from the IRAF webpage.