Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.adass.org/adass/proceedings/adass96/reprints/devriesc.pdf Дата изменения: Wed Jan 14 23:20:22 1998 Дата индексирования: Tue Oct 2 14:53:59 2012 Кодировка: Поисковые слова: vallis

Astronomical Data Analysis Software and Systems VI ASP Conference Series, Vol. 125, 1997 Gareth Hunt and H. E. Payne, eds.

The SRON-HeaD Data Analysis System
C. P. de Vries Space Research Organization Netherlands (SRON), Sorbonnelaan 2, 3584 CA Utrecht, Netherlands, E-mail: C.deVries@sron.ruu.nl Abstract. A data analysis system has b een develop ed at SRON, which has b een designed to allow rigorous control of the quality of its processed data products. In order to fulfill this requirement, all data processing steps are recorded in a central database. The system will initially b e used for analysis of SAX-WFC and XMM-RGS data at SRON.

1.

Introduction

A critical requirement for data analysis systems used for massive routine data processing is that the system is able to deliver processed data of controlled quality in an automatic fashion. In order to control quality of data, it is necessary to have the ability to trace the heritage of all data products. This means recording all parameters of all steps which lead to the establishment of the final data products (Figure 1). To check processing status and initiate subsequent processing steps, one should easily b e able to generate overviews of all available data and intermediary products based on data descriptions and processing heritage. The SRON-HeaD (SRON High Energy Astrophysics department Data analysis) system has b een develop ed to fulfill these requirements, and is based on earlier exp eriences with the CGRO-Comptel data analysis system (de Vries 1995). 2. Requirements

The following basic requirements were defined: · Full traceability of data processing. Storage of all parameters of all processing steps, including complete software configuration. · Complete catalogue of all available data products. Prop er user-interface for manual processing and access to data descriptions and data heritage. · Automatic processing based on data catalogue and processing status. · Automatic archiving/retrieval of bulk data from mass storage devices. · Use FITS format data files, where p ossible. · Allow external analysis packages (e.g., FTOOLS, IDL, etc.) in the system. · Separate "test environment" for testing of all system asp ects and data processing programs. Capable of running on a variety of UNIX systems. 62

© Copyright 1997 Astronomical Society of the Pacific. All rights reserved.


The SRON-HeaD Data Analysis System

63

File 1

File 2

File 3

Par 1 Par 2 Par 3

Task 1 File 6

File 4

File 5

Par Par Par Par Task 2

4 5 6 7

File 8 File 7

Task 3

Par 8 Par 9 Par 10

File 9

Figure 1. A processing pip eline is a sequence of tasks with several input/output files and parameters, which pass data from one task to the next. Heritage of any file (e.g., file 9) can b e established by recording all input/output files and parameters of all steps and by uniquely identifying each task, requiring thorough configuration control.

3.

Implementation

The core of the system is the recording of data descriptions and data heritage in the database, where this information may b e queried via user interfaces or the routine processing pip eline to start new jobs (Figure 2). The dataset heritage consists of actual processing parameter values and the software configuration used. Since data processing parameters are available either in the FITS headers or from input parameter files used by the generating programs, these parameters can b e recorded after actual data files have b een created. This means that no connection to a database is required during data processing and that external packages can easily b e incorp orated into the system. In addition, externally generated (FITS) data files can easily b e imp orted into the system, as well. The basic processing module is a "task" or program executable, called from a script or "job," which may also call other tasks. The script defines the control flow and communication b etween tasks within a job. The job script may prompt the user for task parameter input. Actual parameters are passed to tasks via IRAF format parameter files. Automatic recording of processing parameters is done for each output file at the end of job processing.


64

de Vries
User interface pipeline def.

Routine processing

Others .........

pipeline processor Forms select

parameter values par 1 par 2 ..... par n file 1 file 2 ...... file n

Data-base

software config

loader interface

Task Exec DAL

Parameter tracing DAL

Disk buffer + Archive System ( including database etc) New data file(s) + User parameters

data

subroutines

executable

script

sw package

Figure 2.

Top level functional breakdown of the HeaD system.

An Oracle client-server database architecture serves as the central database system (RDBMS), which holds the data catalogue and heritage, processing parameters description and software configuration. Software configuration at the system level is maintained through use of the RCS system, and use of this system is enforced by the appropriate user interfaces. The Tcl/Tk package plus extensions (TclX,OraTcl) is used to create the windows-based user interface (Figure 3). On-line help is available by means of a Tcl/Tk HTML viewer. Direct selection of data files for input to the appropriate user interfaces can b e made by selecting from lists resulting from the execution of SQL procedures. These can b e taken from a library of procedures which base selections on a variety of data descriptions, heritage, or processing status. In addition, Oracle-Forms can b e used for direct database queries. Data processing "jobs" can b e initiated manually by explicitly entering task parameters for sp ecific tasks, or automatically through manual or automatic


The SRON-HeaD Data Analysis System

65

Figure 3.

HeaD user interface.

marking of individual datasets or dataset typ es for further processing. In that case, sp ecial database tables define the processing flow. A data access layer (DAL) is available which separates the actual scientific code from basic data I/O, allowing for greater system p ortability. This layer is partially comp osed of the FITSIO library, modified to allow for communication with the archive system, and sp ecially develop ed routines. Currently the system contains sp ecially develop ed data processing programs as well as tasks taken from general packages like FTOOLS, IDL, SAOimage, etc. The system has b een initially implemented on Sun (SunOs, Solaris) and HP (HP-UX) systems. More information can b e found on HEAD Home Page.1 References de Vries, C. P. 1994, in ASP Conf. Ser., Vol. 61, Astronomical Data Analysis Software and Systems I I I, ed. D. R. Crabtree, R. J. Hanisch & J. Barnes (San Francisco: ASP), 399

1

http://ws13.sron.ruu.nl:8080/head/Welcome.html