STPOA--The New Pipeline Package for the HST Post-Operational Archive

The ``Post-Operational Archive'' (POA) project at the Space Telescope European Coordinating Facility (ST-ECF) has recently released an upgraded version of the HST Faint Object Spectrograph (FOS) calibration pipeline. We present an overview of the capabilities of this STPOA release, showing examples of the resulting scientific gain. We discuss the lessons learned from the in-depth study of an HST Legacy instrument, the FOS, and the problems encountered therein, including erroneous header contents, difficulties in attaining and processing telemetry uplink/downlink information, and software environment restrictions.

1. Introduction

Archive research and data mining are now considered vital parts of scientific research, but scientific data integrity has to be checked, improved and validated before massive research projects can be started.

Typically, fewer resources are available for work on legacy instruments than are allocated for work on operational instruments. The HST Post Operational Archive (POA) project, based on an agreement between ESA and NASA, aims at improving the scientific value of HST legacy instrument data. We investigated the feasibility of improving the FOS data archive by resolving important calibration issues with the STSDAS off-line FOS calibration pipeline.

In July 2000, the POA project team released STPOA, a new IRAF external package. It contains `poa_calfos', a replacement for the standard FOS calibration pipeline, along with other tools for FOS re-calibration.

2. STPOA--The New Pipeline Package

The STPOA external IRAF package is available on-line¹and can be installed at any IRAF user location. The July 2000 (v1.0) release is already in place and being used at ST-ECF and STScI.

The POA project has officially taken over the user support of the FOS. Our website contains the most up-to-date documentation, technical information and software for the FOS. Also, our team operates a new helpdesk; the email address for questions and issues is ecf-poa@eso.org.

In its current version (v1.0), the new `poa_calfos' calibration pipeline software fixes the zero-point problems for most FOS/BLUE channel data. It takes into account the residual effects from the Geomagnetic Image Motion Problem (GIMP), the offsets introduced by the electronic setup of the detector (YBASE values), and the variations of the ambient temperature in the FOS optical bench.

The ST-ECF HST archive now supports FOS data retrieval with On-The-Fly (OTF) calibration using either the new pipeline `poa_calfos' or the old pipeline `calfos'. One can either retrieve re-calibrated FOS data or process the raw data using the new software.

3. Software and Science Improvements

Improvements reduce the uncertainty in the wavelength zero-point from as much as six pixels to less than one pixel. Figure 1 compares the results obtained using the old pipeline `calfos' (open symbols) and the new pipeline `poa_calfos' (closed symbols): various gratings are indicated by different grey scales; the time range is April 1993 to January 1997 (the end of the FOS lifetime); `poa_calfos' corrects the drift in the zero-point track to $\pm{1.25}$ pixels (or $\pm 0.75$ pixels if the data are oversampled, which is not shown here).

**Figure 1:** Comparison of wavelength zero-point offsets using `calfos' (open symbols) and `poa_calfos' (closed symbols).
$\begin{figure} \epsscale{.80} \plotone{P2-27a.eps} \end{figure}$

An example of the scientific improvement of data, using the new pipeline, is shown in Figure 2. The raw data were taken in Rapid-Readout mode, spanning approximately 40 minutes, observing the same object. A well known spectral line was used to compare the wavelength position to the expected value from literature:

**Figure 2:** Science Case: wavelength zero-point offsets using `calfos' (top) and `poa_calfos' (bottom curved) and `poa_calfos' oversampled (bottom flat).
$\begin{figure} \epsscale{.90} \plotone{P2-27b.eps} \end{figure}$

4. Verification of the On-board GIMP Procedure

In order to perform a verification of the GIMP using on-board information, the uplink HST telemetry data were checked to make sure HST did what it was commanded to do. The downlink engineering telemetry data were checked to match the uplink, as well as help solve the GIMP issue using on-board instrument readouts, info which is not in the science data or FOS data headers.

5. HST Telemetry Uplink Check

We needed the HST uplink telemetry data for our POA GIMP investigation. These data were never required to be archived in a database; they were written to 36,000 tapes and stored in a federal vault. The difficulties in attaining even a small fraction of the uplink HST telemetry information and being able to process it, made us realize that if we had to acquire the entire set and re-extract all the necessary information, our re-calibration task would have been close to impossible. A warning for the future! Fortunately we found some uplink telemetry (May '96 to Jan '97) from a colleague. We used the PASS software from STScI to extract the on-board GIMP related information. We thereby confirmed that the uplink GIMP coefficients were properly used by the on-board data processor.

6. HST Telemetry Downlink Check

Engineering telemetry downlink data are only routinely archived at STScI, so the ST-ECF HST archive had to be updated for the purpose of doing the on-board comparison. The existing downlink telemetry extraction software was never meant to be used outside STScI or distributed off-site. There is no general purpose engineering telemetry downlink reader; therefore these data are almost inaccessible. The ST-ECF archive team ported this software from VAX/VMS to DEC/Alpha; they modified it to access the parameters we needed. We used the on-board voltage, temperature and magnetometer readings to verify the geomagnetic field models used in `poa_calfos'. We confirmed that the instrument and on-board GIMP calculations were synchronized.

An important lesson learned from this experience is that one should view the Science and Engineering downlink telemetry as one unit! Uniting them and having the ability to easily access the information would greatly benefit all calibration work in the future.

7. Final POA/FOS Release--August 2001

The final version of the Post Operation Archive FOS pipeline is planned for release in August 2001. All 24,000 FOS datasets will be reprocessed with `poa_calfos' and re-ingested into the STScI archive. OTF re-calibration is (and will be) available at the ST-ECF HST archive for all FOS data. Substantial ``cleanup" of the FOS calibration reference file data will be performed. The FOS section of the HST Data Handbook will be updated accordingly. Technical reports on re-calibration issues will be available on our website.

8. Conclusions and a Look Ahead

When the POA work on the FOS is finished, we will have a homogeneous archive for this HST instrument. A researcher can view the FOS data from 1990 to 1997 together, in a coherent manner and confidently make comparisons with data from other sources.

POA work has the advantage of looking at the entire dataset range all at once; this is impossible to do while the instrument is in operation. Calibration issues evolve over time, the ``global view'' is not available over a short time scale during the ``live operational mode'' of an instrument. We will apply the expertise gained from our FOS work to the next HST legacy instrument project.

Long running satellite missions need to project ahead in terms of computer hardware/software and portability issues. Post Operation work will be done for most missions; such work will be made easier if all telemetry and pipeline processing software is portable to current technologies and is distributable.

Lessons learned from POA work should also influence the design of calibration procedures and archive architecture for future instruments.

Acknowledgments

We would like to thank the staffs of STScI and Goddard Space Flight Center for their help in unearthing documents and other ``dusty'' information on the FOS for our analysis.