Next: On the Need for Input Data Control in Pipeline Reductions
Up: Dataflow and Scheduling
Previous: Enhanced HST Pointing and Calibration Accuracy: Generating HST Jitter Files at ST-ECF
Table of Contents -- Index -- PS reprint -- PDF reprint
Astronomical Data Analysis Software and Systems VII
ASP Conference Series, Vol. 145, 1998
Editors: R. Albrecht, R. N. Hook and H. A. Bushouse
W. Hack
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD, 21218, Email: hack@stsci.edu
J.-C. Hsu
Space Telescope Science Institute, 3700 San Martin Dr., Baltimore, MD, 21218, Email: hsu@stsci.edu
Abstract:
Two new instruments, STIS and NICMOS, were added to HST during the SM97 servicing mission, each with new types of data that need to be processed. As a result, a completely new system for generating paper products was created which provides a detailed summary of the observations and of the calibrations applied to the observations in the HST pipeline. This new version of the paper products generating software produces a totally redesigned summary for the older instruments (WFPC2, FOS, GHRS, and FOC) as well as creating paper products for the newly installed STIS and NICMOS. In this paper, we discuss the design and algorithms used to handle the extremely large amounts of data produced by the instruments on board HST, as well as the new task's limitations. This new version of the HST paper product software is now available in STSDAS Release V2.0.
The Hubble Space Telescope (HST) not only generates a great deal of data in a single day, but a wide variety of data as well - from 1-D and echelle spectra to images and time-tag data. All of this data is processed on the ground to apply basic instrumental calibrations and prepare the data to be ingested into the HST archive and then analyzed by the observer. One final step in this processing is to create a summary for initially judging the quality of the observations and for determining the status of the calibrations applied to the data.
This summary, however, must be efficient in its reporting and yet
comprehensive enough in its results to be useful. Generating such a
report for each instrument's data (WFPC2, FOC, FOS, GHRS, STIS and NICMOS)
while retaining a similar enough format for easy comprehension was
the primary goal for the software. In addition, the software must run
quickly enough to prevent backlogs when ingesting into the archive.
These two primary problems guided our effort to create a single software
system to provide summaries known as `paper products' for all HST data.
This paper will summarize the major features of this software system.
The paper products software performs the following steps in creating the summaries for each set of observations:
- Sorts input data:
Input data can be a combination of one or
more instrument's data from any number of proposals. The input data can
either be in FITS format, as if obtained from the HST DADS system, or
GEIS format. All of the observations, or only selected observations,
can be processed at one time depending on the user's input. A separate
output file with its own cover page and summary will then be created
for each logical unit defined by data from each instrument used in each
proposal.
- Summarizes each observation:
The basic exposure information for all observations in the proposal is
provided in a tabular form. It includes target data, instrument
configuration (such as filters used), and basic data statistics.
Observations for each instrument are handled by their own function,
allowing the summary to reflect the unique nature of each instrument.
- Displays the data for each observation or observation set:
Grey scale images or plots of the data are provided for each exposure.
These displays are created using SPP functions or CL scripts tailored
specifically for the type of data produced by each instrument. In
addition to the graphical display of the data, a summary of the
exposure information is also provided. The basic forms of displaying
the data are given for each instrument:
| FOS |
Target acquisition position plots, spectral plots, total flux plots |
| GHRS |
spectral plots, jitter plots |
| FOC |
image display |
| WFPC2 |
mosaic of CCD images, single chip display |
| NICMOS |
thumbnail index of images, mosaiced images,
target acquisition images |
| STIS |
image display, spectral plots, jitter images,
target acquisition images |
- Provides a summary of the observations data quality:
The last page(s) of the paper products is a summary of the
calibration(s). A summary of pointing information
(jitter data), explanations
regarding quality flags and some basic image
statistics are all reported here with an exhaustive list of
calibration files and the origin of each file.
- Process the IGI scripts:
The results from all the processing are generated in the SPP procedures
and CL scripts as IGI commands, then processed by IGI to produce the
final output.
This processing will produce a report tailored to each instruments data and
contain the following estimated number of pages:
| Cover Page and Help Page |
1 Page each |
| Help Page |
1 Page |
| Exposure Summary |
1+ Pages |
| Data Display |
1 Page/observation (FOC) |
| |
1- 2 Pages/external observation (WFPC2) |
| |
1-3 Pages/observation (FOS) |
| |
1+ Page/observation (STIS) |
| |
32 dithered obs/page + 1 page/image (NICMOS) |
| Observation Summary |
1-2 pages |
HST archive operations currently used Helvetica narrow fonts for all
paper products. Use of different fonts may cause overlapping of text in
the report.
The paper products software was designed to be run within the IRAF CL environment. The package consists of one primary task, pp_dads, and many hidden tasks to control each instrument's output. The pp_dads task only uses a small number of parameters; namely, input, device, fits_conv.
The parameter input controls which files are read in for processing,
including using wildcards, list files or simply one image name.
The parameter device can be set to a PostScript device to create
PostScript files or a printer queue to send the output directly to a
printer. This relies on using IRAF's printer queues, graphics kernels,
and STSDAS's PostScript kernel.
All HST data is routinely delivered to observers on tape in FITS
format, and archived in the DADS system in FITS format as well. The
paper products software, therefore, relies on the latest FITS kernel in
IRAF to work with STIS and NICMOS data whilst also
working directly with FOC, WFPC2, FOS, and GHRS data that are stored in
the GEIS format.
Most data processing is handled by routines written specifically for each
instrument's data. These functions are written in SPP for NICMOS, FOC, WFPC2, and FOS, while CL scripts are used to process STIS and GHRS observations. Modular design can be readily modified for each instrument or even have new code added to support future new instruments.
The use of SPP provides efficient access to image headers, the ability
to perform statistical calculations while the image remains open, and
great control of the output as IGI commands. The CL scripts used for
STIS and GHRS are readily edited and can be used without compiling,
however, they lack the efficient file I/O that SPP offers.
The results of the calculations and instructions for displaying the
data are sent to an output file to be
interpreted by IGI, the Interactive Graphics Interface. This
provides several advantages for creating the paper products:
- manages image display: graphics and images can be displayed with
image resizing being done as needed on the fly, enabling creation of
thumbnail indices and annotated image
- machine independent and flexible output: IGI uses IRAF's graphics
kernels and PostScript kernel to create PostScript output for
any platform.
The paper products software must run fast enough to keep up with the
data stream coming from HST in order to avoid backlog. This led to the
use of SPP code to generate the paper products for most instruments. In
general, the computation of the paper products takes from 30-60 seconds
per observation on a Sparc 5 workstation with only one user, and about
2-3 minutes per observation on a DEC Alpha workstation with several
users. This is quick enough to avoid most backlog situations. The
real bottleneck is the printing of the paper products. A local
Printserver 17 printer can print about 15-20 Mb of PostScript output in
an hour, however, the paper products code can generate 24Mb of
PostScript in about 6 minutes on a Sparc 5, or about 20 minutes on an
Alpha workstation. Therefore, a great deal of emphasis has been placed
on setting up ways to print these jobs in parallel on multiple printers
to keep up with the stream of data coming from HST. A drawback to this
is the need for large temporary disk space for block averaged
WFPC2/STIS/NICMOS images, PostScript files, and temporary IGI files and
an effort is being made to reduce this need.
The new paper products package within STSDAS can process observations from a mix of instruments in one pass resulting in clear and concise documentation. It relies on many of IRAF's built-in functionality as well as many separate IRAF/STSDAS tasks to produce the paper products. The following packages are necessary for the operation of the paper products software:
IRAF Version 2.10.4p2, or newer (preferably Version 2.11)
and STSDAS/TABLES Version 1.3.5, Version 2.0, or newer.
The simple, one task interface makes it easy to run, both interactively
by a user after extracting data from DADS and in batch mode from
a script, as done by the HST archive. This experience with processing
daily observations has proved that this task can function efficiently and effectively with minimal intervention.
© Copyright 1998 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: On the Need for Input Data Control in Pipeline Reductions
Up: Dataflow and Scheduling
Previous: Enhanced HST Pointing and Calibration Accuracy: Generating HST Jitter Files at ST-ECF
Table of Contents -- Index -- PS reprint -- PDF reprint
payne@stsci.edu