The Beowulf Project
	HOME INDEX SEARCH HELP NEWS

ESO's Data Management & Operations Division and California Institute of Technology, CACR are cooperating to investigate the appropriateness and means of applying Beowulf-class computing technology to the computing requirements of the ESO Very Large Telescope Data Flow System.

Beowulf

ESO initiated a contract with the Caltech Centre for Advanced Computing Research in January 1998. The one year program was to assess the usefullness of Beowulf technology to the data processing needs of the European Southern Observatory for the Very Large Telescope (VLT) project. The VLT is the world's largest optical observatory and is capable of data rates and data volumes in excess of any esisting ground or space based astronomical facility.

Objectives

1. Processing of astronomical data from the VLT archive: Once data from the telescopes arrive in Europe, they enter the VLT Archive. This facility will grow to 100 Tbytes within the first 5 years of VLT operations. Individual data frames can be as large as 0.1 - 0.5 Gbytes and hundreds of frames can be produced each night of observation. These frames will be processed before distribution to the astronomer in much the same way the Hubble Space Telescope delivers processed as well as raw data. Beowulf machines could be used to process these data files. The processing is "embarrassingly parallel" in that all frames are operated on by the same algorithm and each frame is essentially identical.... a good fit to the course-grain parallelism paradigm of a simple beowulf system. A prototype system has been assembled at Caltech and tests done on a realistic data set. The results are promising and if fully successful, a Beowulf system (initial configuration of approximately 16 PIII PC's) will be installed at ESO headquarters in Munich during 1999 for operational tests. Based on that system, a second Beowulf will be installed in 2000 with ~128 nodes.

2. Many data processing tasks for the VLT involve real-time feedback of information at the telescope. Detecting sources that vary in their brightness or move on the sky can be done in real time via applying Beowulf technology to the processing of raw data frames at the telescope. Based on the success of the machines at ESO headquarters, Beowulf systems may be operational at the observatory site some time in 2000

3. ESO is also interested in look at Beowulf technology for data storage and throughput. Using the PCs of the Beowulf as "smart disks" coupled with fiber-channel technology and 100baseT connections may allow us to stage large data sets (>100 GByte) for archival research.

A one year study of ESO data-flow computing requirements in 1998 has revealed an important opportunity for advancing those needs through Beowulf clustered computing technology. The proof-of-concept study performed by Caltech for ESO determined the suitability of Beowulf in this application domain and demonstrated the effectiveness of related application codes on a Beowulf, supported by software tools developed as part of the study. Parallel efficiency of order 70% (35x speedup using a 50 processor system) was demonstrated using ESO DFS analysis recipes and astronomical data running on the Caltech Beowulf through project-developed middleware.

The second phase of the project will include the delivery and installation of a prototype Beowulf system at ESO in Q3 1999 the development of a wide field digital camera data reduction pipeline inside a new DFS pipeline infrastructure that supports Data Reduction Systems (DRS) running on Beowulf-class resources a production environment running the prototype Beowulf facility at ESO on astronomical data from the Wide Field Imager on the ESO/MPI 2.2m telescope at La Silla Observatory in Q4 1999.

More information

• Who is Beowulf?
• CACR Beowulf homepage

Send comments to <Peter.Quinn@eso.org> Last update: July 7, 1999