Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.eso.org/projects/dfs/papers/mess84/
Дата изменения: Wed Jun 10 17:55:22 1998
Дата индексирования: Sun Dec 23 03:07:04 2007
Кодировка:
Поисковые слова: m 8

Data Flow System Overview

HOME

INDEX

HELP

NEWS

Introduction

Astronomers are currently at an important cross-road in their ability to gather information about the Universe. Firstly, computers and astronomical detectors have simultaneously increased in performance. Modern observatories using high-throughput, large-format detectors for imaging and spectroscopy are capable of producing tens of gigabytes of raw data every clear observing night. Although this allows rapid progress on observing programmes, the data rate and volume can easily saturate current network technologies and consume the disk space typically found on most workstations. Secondly, large telescopes in the 8 10-m class are going to be the workhorses of optical and IR astronomy well into the next century. These telescopes will be available to astronomers around the globe and competition for these expensive resources will be fierce. Observing programmes then have to be undertaken in the most efficient manner possible, minimising the effects of the weather in order to complete programmes in the shortest possible time.

Because of this "data deluge" and the need for maximum efficiency, astronomers are faced with the realisation that perhaps the classical observing cycle of trekking to the telescope and returning home with a tape of unprocessed data is no longer the most efficient or practical way to do astronomy. ESO astronomers stood at this crossroad and considered the options carefully when designing the Science Operation Plan for the VLT.

Operational Aspects

In the Science Operation Plan, ESO decided to enable the VLT to execute observing programmes in a service mode. Astronomers can specify programmes in a Phase I Phase II proposal process and the resulting observations would be completed by ESO astronomers and technicians at the VLT when the conditions are most favourable for that pro- gramme. In this way, ESO astronomers would be as efficient and competitive as possible in completing the types of frontier programmes that will be common place on the VLT. Furthermore, ESO decided to meet the challenge of the data volume potential of the VLT by insisting that the VLT should be able to deliver calibrated data to the astronomer with a well-defined accuracy. This has two immediate benefits for the astronomer. Firstly the data processing and storage necessary to calibrate the data is provided by ESO which helps relieve the deluge significantly since most data reduction systems expand raw data by factors of 2 to 5 before calibration is complete. Secondly, by guaranteeing the calibration process to a given accuracy, ESO has to continuously monitor the applicability of calibration data and the performance of telescopes and instruments over long periods of time. The quality control of the calibration process will ensure the longterm usefulness of the data.

The VLT Data Flow System

Once ESO adopted the principles of service observations and delivery of calibrated data for the VLT, it was obvious that the flow of data must be managed from start to end of the observing process. The design, development, operation, and maintenance of the VLT Data Flow System (DFS) are done by the Data Management Division (DMD). This software and hardware is responsible for the flow of scientific data through the VLT from Proposal Entry to Archival Research.

DFS Concepts and Design

In early 1995, the Data Flow Working Group was formed to formulate the requirements on the DFS following the guidelines of the Science Operation Plan. The group consisted of astronomers, instrumentation and software experts from several ESO divisions. Discussions within the group focused on the characterisation of the observing cycle from proposal entry to final archive of the data. This cycle was broken down into component processes and the outline of the DFS took shape.

A fundamental concept that holds the component processes together is that of the Observation Block. This quantum of data in the data flow system is an object which contains the essential target and instrument information to define an observation of a single object. An Observation Block is defined by either a classical observer at the telescope or a service mode observer. Whether the observer uses service-mode capabilities or not, the Observation Block is intrinsic to the flow of data from place to place in the VLT operation.

DFS Modules

Preliminary design documents describing the services offered by each of the modules in the DFS and their interrelationships were drawn up and distributed by the middle of 1995. The basic modules in the DFS are:

Programme Handling:
The electronic submission of Phase 1 proposals. Preparation of a 6-month schedule for service and classical mode observing sessions.

Observation Handling:
Phase II proposals are prepared using software tools. Observation Blocks are created for each target. Medium-Term Scheduler system prepares a list of possible Observing Blocks to be executed for the next few nights. Short-Term Scheduler systems select the next Observation Block to be executedby the VLT Control System (VCS) based on environment and configuration data.

Science Archive:
Provides on-line catalogue services to observers. Stores raw data passed by the VCS. Data distribution and archival research on VLT data.

Pipeline:
Organise raw data and calibration information for analysis. Schedule calibration processing of data. Apply approved calibration recipes.
Quality Control:
Maintain calibration plan for a given instrument and mode. Compare calibration data with simulators. Study long-term trends in calibration data. Assess quality of calibrated data with respect to standards.

[ESO VLT Machine] The DFS is the scientific operating system of the VLT. The VLT can be viewed as one machine with multiple components that interact. Unless the components of the VLT machine work together in an efficient manner, the scientific potential of the VLT will not be fully realised. With the DFS as a science operating system, the VLT machine can be scheduled and can make use of redundancy, pipeline and aggregation of resources to maximise throughput. In the design of the DFS, many opportunities exist to exploit the single machine vision of the VLT.
DFS and VCS
The concept of the DFS arrived late in the process of designing and implementing of the VCS. The VCS system is responsible for the control of the VLT Unit Telescopes and their associated instruments. This responsibility extends from low-level motor control to high-level assessments of the technical quality of data and its movement from instruments to storage. VCS was designed and partially implemented before concepts like Observation Blocks were introduced. It is therefore clear that the success of the DFS will depend critically on the close collaboration of software and system designers from DMD and the other ESO divisions. In September 1995, a Data Flow Project Team was formed by the Data Management Division to bring together astronomers and engineers from the VLT, Science and Instrumentation Division to address the many interfaces issues opened between the DFS and VCS. This Team has worked hard to crystallise the critical system interface issues and a number of collaborative software programmes are now under way between DMD and the VLT software group. A review of the status of VCS and DFS design in April 1996 pointed out the importance of the ongoing work of the Data Flow Project Team.

The development of a close collaboration and coupling between VCS and DFS has been aided and accelerated by the prototyping of DFS modules and VCS on the NTT. In July 1996, the NTT took off the air for six months in order to install VLT-specific hardware and software for testing. From the beginning of 1997, the NTT acted like a single unit telescope. Several modules of the DFS, including the Observation Block System, have been tested extensively using SUSI as a trial instrument. All modules of the DFS have been tested and and will modified within the NTT prototyping environment before going on a VLT unit telescope.

Status of the DFS

There are two critical interfaces between DFS and VCS which allow the basic operation of the DFS to proceed. The first allows Observation Blocks to flow from the Short Term Scheduler into VCS. Observation Block substructures call templated are interpreted by the Sequencer within the VCS, allowing instrument operation and target acquisition to proceed. A second interface allows completed science frames to be stored on the OnLine Archive System (OLAS). Within OLAS, science frames are recombined with associated technical data to supplement the associated Observation Block for the observation.

In order to meet the timeline of the VLT, it is essential to bring DFS components design to operational prototype as quickly as possible. To achive this, a rapid prototyping process was identified using the NTT. DFS prototypes have been commissioned and operated on the NTT in a fully VLT compliant environment. DMD identified SUSI as a tarher instrument on which to test the DFS. Prototypes for Phase II Proposal Preparation, Calibration Pipeline, On-Line Archive and Scheduling System at the NTT started in January 1997, after extensive testing in December 1996 at ESO Headquarters and the NTT. For 1997, prototypes for SOFI and EMMI are planned.