Next: From a ``Launch Readiness'' System to an Astronomical Data Processing System - a Review of Four Years of CIA Development
Up: Software Development and Management
Previous: Software Development and Management
Table of Contents -
Subject Index -
Author Index -
Search -
PS reprint -
PDF reprint
Gaudet, S., Hill, N., Dunn, J., Jaeger, S., & Cockayne, S. 1999, in ASP Conf. Ser., Vol. 172, Astronomical Data Analysis Software and Systems VIII, eds. D. M. Mehringer, R. L. Plante, & D. A. Roberts (San Francisco: ASP), 3
The Gemini Data Handling System: A Case History
Séverin Gaudet, Norman R. Hill, Jennifer Dunn, Shannon Jaeger, Steve Cockayne
National Research Council Canada, Herzberg Institute of Astrophysics, 5071 West Saanich Rd., Victoria B.C., Canada V8X 4M6
Abstract:
The Gemini Data Handling System, developed by the Canadian Astronomy
Data Centre for the Gemini 8m Telescopes Project, provides the data
handling infrastructure for the Observatory Control System and the
instrument control systems. An overview of the preliminary design was
presented at ADASS '95. In August 1998, the DHS passed its acceptance
tests and was released operationally to Gemini North in Hawai'i. This
paper will present a case history of the project: how the
requirements changed, how the design evolved to its final
form, the approaches taken, the tools used, and the problems
encountered.
The Gemini Data Handling System (DHS), developed by the Canadian
Astronomy Data Centre (CADC) for the Gemini telescopes, provides the
data handling infrastructure for the Observatory Control System (OCS)
and the instrument control systems. An overview of the preliminary
design was presented at ADASS '95 (Gaudet 1996). This paper will not
discuss the functionality, design, user interfaces, nor
technologies of the DHS, as details on these topics are documented
elsewhere (Cockayne et al. 1998a, 1998b; Dunn et al. 1999a, 1999b; Hill et al. 1999a, 1999b; Jaeger et
al. 1999). Rather, we will present a case history of the project:
how the requirements changed, how the design evolved to its
final form, the approaches taken, the tools used, and the
problems encountered.
2. Evolution of the Project
The Gemini software system is being produced mostly by allocated work
packages to institutions in the partner countries to tap into existing
skills, expertise, and experience (McGonegal 1996). An international
committee worked on the high-level design of the whole control system
which defined the functionality and boundaries of all work packages,
including the DHS.
Early into the contract, it became obvious that the requirements
for the DHS were not complete. But as the various design steps
proceeded and the requirements were completed, the design evolved causing
changes in both the DHS functionality and the DHS boundaries with other
systems.
The design evolved from a sequenced single-threaded, single server
program for each instrument into a data-driven, multiple threaded servers,
each encapsulating some functionality. The DHS is also capable of
accepting data from many sources simultaneously. The system also
became highly configurable in terms of data structures, topology, and
data dictionary to allow Gemini to adapt it to the emerging summit
requirements. In addition, the system had to be distributed where
subsets of the DHS would run both at the summit and at the base
facility.
- Object-oriented. A team decision lead to the adoption of
object-oriented methodology for both the design (Booch) and
development (C++).
- Multi-threaded distributed servers. We used multi-threaded
servers to encapsulate functionality and to localize interfaces
to the rest of the Gemini software systems. For example, there
is only one server handling the data put and get requests.
- Persistent store. The SYBASE RDBMS was used to store file
status and location as well as history records.
- Code re-use. We have utilised many CADC data management
libraries as well as cfitsio (NASA), the IMP
messaging system and SDS data structures from AAO, ESO's
Skycat display tool, and the Gemini Project's tcl/tk shell
OCSWish.
To allow the team to work as efficiently as possible, the proper
working environment had to be created. This consisted of:
- A well supported development environment. The CADC already
had a code management process that allowed four programmers to
work simultaneously with a minimum of conflicts, allowed
for configuration management, and had well defined installation
and release procedures.
- Good tools. This included a good design tool (Rational
Rose), a good set of base classes implemented early in the
project (libdhs++, libGen, libdhsSta), a good debugging
environment with thread support (Visual Workshop), and a good
documentation tool (Framemaker).
- External contract vs. partnership. Since we were in a
contractual relationship with the Gemini Project, we could
have managed the evolving design with change orders, etc.
However, because of our partnership status, we had a stake in
the final result. We had to be driven by ``We have to produce
something that works, not something that meets spec!''. So it
became a balancing act between needs versus resources.
- External contract vs. internal work. The CADC has been
developing software for many years, mostly to support our core
archiving activity. As in most groups, internal development
does not follow a formal methodology but relies on
back-of-the-envelope design and ad-hoc testing. Although not
easy, the group managed the transition to a new culture of
a formal methodology.
- Not all interfaces were well defined. As the DHS design
evolved, some functions originally in the DHS were moved to
other work packages and some functions initially in other
packages were moved to the DHS. This necessitated the
development of new interfaces, some of which took a long time
to finalise.
- Separation of UNIX vs. real-time systems. The DHS is
mostly a UNIX system, but part of the core library is to be used
by instruments running under the VxWorks real-time system.
Porting the UNIX library was a Gemini Project responsibility
which was continually delayed by lack of resources.
- No instrument teams ready. The DHS team, with the help of
some Gemini scientists, had to make design decisions on data
structures and data processing without being able to consult
with instrument teams.
- Developers not involved in high-level design. The
high-level design of the DHS was done before the CADC became
involved with the project (see §2.). We
had to accept a work package with fixed costs and fixed
requirements, requirements which we could only interpret based
on our assumptions - some of which proved wrong!
- Good team. This is the essential ingredient for success.
The DHS team shared a vision of the final product, adapted to
the methodology and the environment and worked very hard and
very well together to attain the goal.
- Solid design. The design presented at the Preliminary
Design review proved to be a solid foundation upon which
detailed design of all the subsystems of the DHS could
proceed.
- Formal project methodology. Breaking the project down into
manageable, measurable tasks allowed for the trace-ability of
progress and costs. Microsoft Project was used to organize
this information.
- Interfaces. The fact that most major interfaces were fixed
early in the project helped minimize the assumptions on work
package boundaries.
- Meetings. Travel was part of the project budget and that
was a wise decision. There is nothing like face-to-face
meetings with other groups to resolve misunderstandings common
to distributed projects.
As mentioned above, the Phase 1 delivery of the DHS is up and running
on Mauna Kea in preparation for first light. Phase 2 has been
negotiated and the DHS final delivery is scheduled for January 1999.
Not all requirements will have been met because of both the evolving
requirements and the underestimation of the cost. The cost breakdown
for the DHS project to date is shown in Fig. 1.
Figure 1:
Total project and labour costs shown as percentages.
 |
Overall, the experience has been positive for the CADC. The formal
methodology and the new technologies have been good learning
experiences and have provided the team with intellectual challenges.
This knowledge can now be applied of internal CADC projects. In
addition, the CADC plans to re-use many parts of the DHS to augment the
CADC archiving systems.
Acknowledgments
As in all software projects, the success of this project is a
reflection of the team which included Norman Hill, Jennifer Dunn,
Shannon Jaeger, Steve Cockayne, Dayle Kotturi and Séverin Gaudet.
The support of the other members of the CADC, Daniel Durand, David
Schade, and David Bohlender was invaluable.
References
Cockayne, S., Dunn, J., Gaudet, S., Hill, N., & Jaeger, S. 1998a, The Gemini DHS Detailed Design Document, (Gemini 8m
Telescopes Project)
,
1998b, The Gemini DHS User Manual, (Gemini 8m Telescopes
Project)
Dunn, J., Cockayne, S., Gaudet, S., Jaeger, S., & Miguel, A. 1999b, this volume, 265
Dunn, J., Jaeger, S., Hill, N., Gaudet, S., & Cockayne, S. 1999a, this volume, 167
Gaudet, S. 1996, in ASP Conf. Ser., Vol. 101, Astronomical Data Analysis
Software and Systems V, ed. G. H. Jacoby & J. Barnes
(San Francisco: ASP), 388
Hill, N., Gaudet, S., Dunn, J., Jaeger, S., & Cockayne, S. 1999a, this volume, 163
, 1999b, this volume, 155
Jaeger, S., Dunn, J., Cockayne, S., Gaudet, S., & Hill, N. 1999, this volume, 159
McGonegal, R. 1996, in ASP Conf. Ser., Vol. 101, Astronomical Data Analysis
Software and Systems V, ed. G. H. Jacoby & J. Barnes
(San Francisco: ASP), 293
© Copyright 1999 Astronomical Society of the Pacific, 390 Ashton Avenue, San Francisco, California 94112, USA
Next: From a ``Launch Readiness'' System to an Astronomical Data Processing System - a Review of Four Years of CIA Development
Up: Software Development and Management
Previous: Software Development and Management
Table of Contents -
Subject Index -
Author Index -
Search -
PS reprint -
PDF reprint
adass@ncsa.uiuc.edu