Calibration Data Base and Operations

CDBS (Calibration Data Base System) is a pathway of software, protocol, and procedures which starts with several groups of Instrument Scientist teams collecting all of the various forms of calibration data, packaging it and delivering to CDBS. The data is put through CDBS Operations with the end result being delivery to HST's ground system reduction pipeline (OPUS), archiving to the Data Archiving and Distribution System (DADS), and population of the CDBS on-line archives.

Follow the links for more information about CDBS Operations, OPUS, DADS, and all calibration data.

Delivery Procedures for all instrument data.

Query the CDBS database....CDBS Query

• Instrument Information and Data:
  • WFPC2
  • GHRS
  • FOS
  • FOC
  • FGS
  • NICMOS
  • STIS
  • ACS
  • COS
  • WFC3

• Component and Reference Data:
  • Throughput Tables
  • Astronomical Catalogs
  • Modewave Data

• More CDBS Information
  • File and Table Structure
  • Ground System - OPUS
  • Component Throughput Data
  • CDBS Documentation
  • Miscellaneous Data
  • Data Delivery Procedures

Note: We no longer keep and WFPC (I),HSP, or STIS data and/or information on-line. Data is available in DADS using StarView.

Note: When accessing files, be aware of the different formats these files and tables are in . If you don't pay attention to these formats, you may end up with

CDBS OPERATIONS

CDBS (Calibration Database System) Operations maintains the HST Calibration Database. We receive and process calibration data delivered to us from instrument teams. The data will eventually be used in reduction routines both in OPUS and offline programs.

To get an idea of what happens when data comes into CDBS and is processed, we've captured the screen results from a WFPC2 delivery. Four sets of files came in along with an associated load file for each set. The particular reference file type in this example is DRK. Take a look at:
Deliver CDBS.

Many of the tasks, such as certify, expload, loopfits, mkcomptab, mkload, and uniqname, all part of deliver_cdbs, were written by Bernie Simon and has been documented at the following link: CDBS Tools

• design
• appendix
• directories
• comments_sql_gen
• expload
• install
• mkload
• recreate_load_file
• mkcomptab
• certify
• uniqname
• loopfits
• fitsverify
• cdbs_sql_gen
• opus_sql_gen
• update_opus
• cdbs_make_request

• We perform testing on data formats and structures as well as checking certain science value ranges, using certify. For history tracking purposes and clarity of use, comment lines and history records have been inserted into the header files of all of the data. We check for the existence of this information and look for concise reasons for data updates. All of these steps have been designed around certain data quality criteria and functionality.
• All of the data files are given unique new names following a date/time convention, using uniqname.
• If the files delivered are in GEIS format, loopfits and fitsverify are run to convert from GEIS to FITS and check the validity of the FITS files.
• check_load is run to check various keywords within the load files against the same keywords in the data files.
• cdbs_sql_gen is run to generate the sql statements for installation into CDBS, and installs a lock on the particular delivery number generated.
• run_delivery_sql installs the data and/or pointers into the database.
• check_cdbs is then run to make internal consistency checks on the database.
• opus_sql_gen generates a sql file for opus, representing the installation instruction software.
• update_ga_date is a last step in "locking-in" the delivery to CDBS, updates the general_availability_date within the database, and releases the lock on the delivery number.
• opus_catalog generates a catalog file for OPUS.
• The catalog file and the sql file, both for OPUS, and the reference data (FITS files) are transfered to a special directory in which an automatic process moves everthing over to OPUS.
• The reference data are then archived to DADS using a tool called cdbs_make_request.
• Finally, the reference data is moved to its final resting place, the CDBS online archives.

Once OPUS has installed the data, the ground system has the latest reference calibration data to be used for any observations coming through the system. A feedback mechanism brings information back to CDBS from OPUS on the verification of installation and on the exact installation time. Additional steps of operations involves entering paper forms which track the whole process into the CDBS library, and notifying Instrument Scientist of the installation into OPUS.

You may want to use our new Query system to directly query the Calibration Data Base [CDBS Query], or if you know what you are looking for (known filenames, for example), this may be an easy route for retrieving data. If you do not know what you are looking for and do not want to use the CDBS Query, it may be better to get the data through StarView, which will retrieve from DADS and not our on-line archives.

For history tracking reasons, we rename all of the master component and graph tables and keep them in a separate directory called MTAB . The MTAB directory contains HST master component and graph tables which have been delivered to OPUS, renamed using the uniqname schema. The extensions denote the type of file each represents:

• tmc -- component tables
• tmg -- graph tables
• tab -- individual component tables

More information about the naming of reference data, extensions, associated file types, and database relation names is in File Structures.

---

Back to the Calibration Data and Tools page

Maintained by Calvin Tullos
STScI Observatory Science Group
Last modified, Sept 16 1997