65 kB PostScript reprint

---

Astronomical Data Analysis Software and Systems IV
ASP Conference Series, Vol. 77, 1995
Book Editors: R. A. Shaw, H. E. Payne, and J. J. E. Hayes
Electronic Editor: H. E. Payne

---

Evolution of EUVE Guest Observer Data Reduction

E. C. Olson
Center for EUV Astrophysics, University of California, 2150 N. Kittredge St., Berkeley, CA 94720
http://www.cea.berkeley.edu/ericco/ http://www.cea.berkeley.edu/ http://www.berkeley.edu/

Abstract:

Introduction

The Extreme Ultraviolet Explorer ( EUVE) Guest Observer (GO) Center is now in its second year of data processing. By using astronomical community standards like FITS and a common processing environment, IRAF, the Center was able to concentrate on software development and calibration analysis. The Center made full use of software developed at ST ScI and NOAO as well as CfA. The additional coordination required to enable this code reuse was well worth the time and effort. In an era of diminishing funding, these efforts must continue.

A large part of the success of the program is from selectively applying standard astronomical procedures and existing tools to the evolving EUVE data reduction problems. From its inception the GO Center expected to disseminate very complete raw data sets to GOs, calibration and reduction software, as well as the calibrated data products. Part of the motivation was that before launch, the operational characteristics of the telescope were unknown. Having maintained a stable reduction pipeline for over a year, the GO Center is considering streamlining (and reducing) the data products provided.

The aging core calibration tasks, written before launch in 1990, have had numerous improvements. These tasks are the result of various experiments in implementation: some were short lived, based on limited access to EUVE ground calibration data sets. Yet the initial design was general enough that they can support data sets and missions that were never considered in the design. Real EUVE data has been available since fall, 1992. By this time, all the major software components were completed and in final testing. By the spring of 1993 the software had been released a half dozen times. Under intense pressure and looming budget cuts, ``fast-track'' data sets were distributed to GOs for early EUVE data sets. By April, a functional reduction pipeline was in place. The fast-track data sets were immediately replaced with standard products and by summer, the data processing queue was eliminated. This effort was completed by the combined efforts of the entire GO Center staff. Since then additional functionality has been added to the software, the calibration data sets have been significantly improved, and the documentation has been written. Today, the GO Center is essentially providing the same raw (very complete) data products that it originally delivered, but with improved software and updated calibration data.

Background

The EUVE Guest Observer Center was created in the spring 1990 at the Center for EUV Astrophysics (CEA). The GO Center initially provided dedicated resources to the EUVE pointed spectroscopy program. The GO Center also provided EUVE spectrometer data and EUVE-specific analysis software. The GO Center is responsible for other activities including: science operations planning, GO proposal assessment and instrument calibration analysis.

The EUVE mission has two components after the initial in-orbit check out: a 6 month full-sky survey, and the pointed-mode Guest Observer phase. All EUVE data is archived at CEA on an optical disk jukebox. Data from the EUVE all-sky survey has been processed from raw telemetry into skymaps and catalogs at CEA.

Data processing for the pointed spectroscopy observations is handled by the GO Center. EUVE has seven photon counting detectors: three of the them produce spectral data. The band passes are 70--190 ÅÅ, 140--380 ÅÅ, and 280--760 ÅÅ for the short-, medium-, and long-wavelength detectors, respectively. EUVE photon event contain a detector position and time stamp.

EUVE generates approximately 100GB of telemetry per year. With a typical observation lasting s, about 150 GO observations are completed each year. Each observation corresponds to approximately 70MB of night time data. It is expected that handling each observation will require several hundred MB of disk space and a workstation-class computer.

Software Design

The main design goals of the EGO Center software were to provide a complete data set in a timely fashion with calibration data and software to scientists to complete analysis on their proprietary observations. Providing EUVE telemetry was never discussed as possible data product since it is optimized for satellite communication, and data extraction was a complex task. To date, there are still parts of the EUVE telemetry stream that are not extracted due to a lack of resources and other issues.

When we decided to represent the telemetry in STSDAS Tables, the TABLES package was not yet available and IRAF did not have a layered package mechanism. Representing EUVE telemetry as several STSDAS Tables proved to be a relatively compact format, but the most important feature was the mere availability of the TABLES package from ST ScI. By providing EUVE data in this standard format, EUVE inherited all of the staff-years of programming effort that ST ScI put into the TABLES package (which continues to be upgraded). A recent enhancement to TABLES was the addition was ASCII table support, and we are looking forward to direct FITS BINTABLE support. The TABLES package also provided FITS compatibility: at the time, ASCII tables were part of the evolving FITS standard but binary tables were not.

At the same time, CfA and NOAO were coordinating development a new IRAF image format for handling event data called QPOE (Quick Position Oriented Event) for ROSAT data. Although QPOE was not developed for EUVE, we readily made use of this facility as well. In both cases, EUV software was developed without the benefit of a preexisting application interface. But by adhering to interface definitions, most of the integration was smooth.

The GO Center made no plans for reprocessing GO data products. By providing complete raw data sets, the only reason to reprocess would be to fix significant errors in these level zero data sets. Instead of the GO Center reprocessing the data, the GOs could reprocess their observations at any time since all the GO reduction software is provided in a portable IRAF environment. In fact, using IRAF has not eliminated portability problems. Some problems were due to the flexibility of the SunOS F77 compilers; others were due to system errors. Today, there are still a number of GOs who use less common platforms that have difficulty using the EUV package.

In addition to coordinating software development with other institutes, the GO software also had to coordinate development with internal groups at CEA. These components turned out to be the past ones completed. A key component was the system to access EUVE telemetry which significantly delayed testing of the GO software. However, by working closely with the operations group at CEA, this subsystem's performance has been excellent since its implementation. The benefits of cooperation and conforming to standards are huge, especially for smaller missions that are short-lived and low-cost.

Having made several courageous but necessary decisions about the software development environment and tools, the principal design issues could be addressed. First and foremost was the need for the calibration pipeline to be flexible and robust. A problem was that calibration pipeline was designed before calibration data were available. Based on the ``end-to-end system'' pipeline developed at CEA for analyzing EUVE sky survey data, the GO calibration pipeline tried to retain flexibility while providing efficient access to the observation data set. In the end, this calibration pipeline consisted of less then a dozen processing modules. Some of the modules are quite specific to EUVE data sets, but most are general purpose modules. These processing modules apply the calibration data to the photon events and produce QPOE images of the observation.

QPOE image files are the key element in processing EUVE data sets. They provide an excellent format for efficiently manipulating event-oriented data sets by selectively applying data filters that select subsets of the observation event data. In addition to filter, QPOE images can be used as if they are normal binned images by IRAF tasks that have no a priori knowledge of event oriented data. This enables all IRAF programs to be used with QPOE images.

The other key element for handling EUVE data sets are time-stamped tables. Each row in a time-stamped table contains a valid time stamp for the row. All EUVE engineering information are supplied in this format. Additionally, analysis programs provided by CEA also produce and manipulate time-stamped tables. By sharing this simple common format, these programs can all work together. Some of these tools are now being used on other missions. GOs can use engineering data, post-reduction analysis software, and independently developed software to filter the event data in QPOE files. It turned out that EUVE QPOE images could not be used directly as binned images, due to a number of calibration issues.

Conclusions

Developing software prior to the availability of calibration and data was not too problematic. In fact, it drove the requirement for a flexible solution, although a more general solution than the current implementation might not be desirable either. However, it does appear that there is a great deal of commonality in event-oriented data processing pipelines used by various missions. A survey of such missions would be in order before proceeding with the development of a multi-mission, event-oriented pipeline. CEA will collaborate with CfA on an event-oriented IRAF package that may provide the framework for such a multi-mission event pipeline, although that is not the focus of the collaboration. In retrospect, it is tempting to suggest that the data products and the processing pipeline could have been streamlined earlier. But in an environment where calibration data sets under go significant structural changes, and the instrument can have unexpected characteristics, it is wiser to plan for a worst-case scenario.

By creating a flexible reduction pipeline, CEA has been able to incrementally add supplemental processing modules. For example, modules for handling flat fielding errors and for handling imaging detectors have been added. The final version of the software will include tools for pointed imaging data analysis as well as sky survey data analysis. These new tools will be integrated with the existing tools that handle instrument distortion corrections, data selection, and calibrations.

The possibility of applying these techniques to the AXAF processing and other missions has been discussed recently. Although the general concepts may be applicable, it is clear that this specific implementation does not provide all the necessary features needed for multi-mission support. For example, the use of time stamped tables seems generally applicable. However, the ``Comprehensive Event Pipeline'' that applies calibration data sets to events is anything but comprehensive.

Acknowledgments:

This research was supported by NASA contract NAS5--30180.

65 kB PostScript reprint