1768 kB PostScript reprint

---

Astronomical Data Analysis Software and Systems IV
ASP Conference Series, Vol. 77, 1995
Book Editors: R. A. Shaw, H. E. Payne, and J. J. E. Hayes
Electronic Editor: H. E. Payne

---

Organizing Observational Data at the Telescope

M. Peron, D. Baade, M. A. Albrecht, and P. Grosbøl
European Southern Observatory, Karl-Schwarzschild-Straße 2, D-85748 Garching, Germany

Abstract:

The Problem

There is a number of situations where users of astronomical observations would benefit from advanced software tools which can analyze the composition of the available data pool:

Data Acquisition Control:

Spectroscopists may wish to take after each scientific spectrum a new arc spectrum if, for instance, the spectrograph tilt angle has changed by more than a certain amount since the previous arc exposure. (Presumably, a new arc spectrum would not be required following acquisition images.) Given the numerous other on-line activities, a system knowledgeable enough to issue an automatic reminder to the observer, when and if necessary, would be very advantageous.

On-line Data Reduction:

The main difficulty of automatic on-line reduction procedures is to identify the optimal calibration files that are applicable to a given science exposure. These data may be either acquired during the observing run or supplied by an observatory-maintained database. If calibration data are temporarily unavailable, the on-line reduction tasks may also need to be suspended and resumed when all the necessary data become available.

Off-line Data Reduction:

This situation is very similar to the on-line case, except that the volume of data is larger and the user may no longer remember (or not know) the structure of the database. A system that can group science exposures and calibration data according to user-definable criteria would be a considerable asset, especially if it can also be interfaced to subsequent standard reduction procedures.

Archival Research:

The lack of a suitable overview of the structure and contents of the database is the standard problem of archival researchers who need to specify the calibration files they want to extract. Observatory staff are facing a similar situation when they wish to perform trend analyses for instruments.

Derived Requirements

The above scenarios have in common that the user wants to define a context-specific structure on the observational database. From this, the following requirements on the desired tool can be extracted: It must be able to classify each file into user-definable categories, and it must be able to establish relations between files of different categories (e.g., associate to a given science frame a set of suitable calibration files). The Data Organizer (DO) (Peron et al. 1993) allows users to perform these tasks in an easily customizable way. The main characteristics of the on-line implementation of the DO at the ESO New Technology Telescope (NTT) are described below.

Observation Summary Table

The database for all operations is a MIDAS (ESO-IPG 1993) table, the so-called Observation Summary Table (OST) which contains all relevant parameters extracted from the FITS keywords. Because the Data Organizer is built on existing capabilities of the MIDAS Table File System (Peron et al. 1992), standard relational database operations (e.g., select, merge, copy, and project) may be used.

Classification of Exposures

The information contained in the OST is used to classify the images according to different sets of attributes (e.g., optical elements, calibration sources, etc.). The classification is achieved by applying user-definable rules, and the result of the classification (e.g., an optical path) is saved in the OST. EMMI (Melnick et al. 1992), one of the instruments mounted at the NTT, allows a wide range of observing modes, from wide-field imaging to high-dispersion spectroscopy, including long-slit and multi-object spectroscopy. Because of its complexity, EMMI has been chosen for the first implementation of the DO in an on-line environment.

 
Figure: The Graphical User Interface to the Data Organizer as installed at the NTT. The main window in the upper left corner shows (part of) the OST which is the database on which all operations supported by the DO are performed. A table-like widget ("Classification rules") can be activated by a pushbutton to edit and modify classification rules. The rule ``BIAS'' for classifying BIAS exposures is being edited in the figure. Another table-like widget (``Classify'') can be used for entering classification rules to be applied as well as the character string which in the OST will identify the selected frames. Original PostScript figure (1732 kB)

Association of Exposures

The association of scientific frames with suitable calibration images is achieved by selecting and ranking all calibration frames which for a given scientific exposure match a set of user defined selection criteria. One may expand the search by submitting a ``second choice'' set of criteria when not enough frames match the original one. For instance, a search may be expressed in natural language as: ``Find for each scientific frame two dark exposures that have been observed within the same night, and for which the mean detector temperature did not vary by more than 1 degree. If unsuccessful, look for dark exposures observed within the same observing run, and for which the mean detector temperature did not vary by more than 2 degrees.'' Selected frames can be ranked by applying weights to the attributes invoked in the same process. For example, one may give more importance to the time difference than to the detector temperature difference.

On-line Implementation of the DO at the NTT

The DO has been conceived as a general purpose tool, whereas the implementation at the NTT provides very specific services. In particular, it knows about FITS keywords that characterize EMMI files. This customization was achieved by providing a set of NTT specific configuration tables which contain definitions of different exposure types (e.g., SCI, FFDOME, FFSKY, and BIAS) and instrument modes (e.g., blue imaging, red medium-dispersion spectroscopy). Each time a new file is delivered by the acquisition system, the exposure is classified according to a predefined set of rules, and the result is appended to the OST. Because it is essential that observers can interact efficiently with the tools offered to them in an on-line environment, a versatile Graphical User Interface has been fitted to the DO. It is shown and further explained in Figure 1.

Further Applications

After the adaptation of the association part of the DO to the particular requirements of EMMI, the basis for automatic on-line data reduction will be available. The first MIDAS package to be interfaced to it will be the CCD package. Observers and archival researchers will receive their data together with the corresponding OST. At the NTT, an observatory-supplied calibration database will be maintained. A dedicated OST will provide the overview of that database and enable observers to identify additional calibration observations which could be applicable to their own data. It is expected that at the Very Large Telescope (VLT) the Data Organizer will play a more central role in the on-line data flow.

References:

Peron, M., Ochsenbein, F., & Grosbøl, P. 1992, in Astronomy from Large Databases II, ESO Conference and Workshop Proceedings, No. 43, eds. A. Heck & F. Murtagh (Garching, ESO), p. 433

Peron, M., Albrecht, M. A., & Grosbøl, P. 1994, in Handling and Archiving Data from Ground-Based Telescopes, ESO Conference and Workshop Proceedings, No. 50, eds. M. A. Albrecht & F. Pasian (Garching, ESO), p. 57

Melnick, J., Dekker, H., D'Odorico, S., & Giraud, E. 1994, in EMMI & SUSI, ESO Operating Manual, No. 15, Version 2.0

1768 kB PostScript reprint