WFPC2 Instrument Handbook for Cycle 11

On-The-Fly Reprocessing Systems

---

The On-The-Fly Calibration (OTFC) system, publicly released in Dec. 1999, calibrated data at the time a user requested data from the archive. The advantages to using OTFC included the automatic application of improved calibration files and switches, use of most recent calibration software (allowing for rapid access to improved algorithms, new capabilities, and software fixes), and correction of header keywords if needed. An additional benefit is that only the uncalibrated data needs to be stored in the archive.

The On-The-Fly Reprocessing (OTFR) system replaced OTFC on May 16, 2001. The change will be transparent to most HST archive users. Requests for data will be submitted as usual via StarView or WWW; raw and freshly-calibrated data will be delivered. There will be no need to explicitly ask for OTFR: all requests for WFPC2 data will be handled by the OTFR system.

The primary difference between the two systems is that OTFR begins earlier in the data path. It uses the original telemetry files ("POD" files) received from Goddard Space Flight Center and performs all pipeline processing steps; OTFC performed only the last pipeline processing step (calibration), on raw files retrieved from the archive. An overview of the data flow for both systems is summarized in the table below. The benefits of the new OTFR system encompass the benefits in the OTFC system; in addition, OTFR data will need fewer header corrections (most problems will be fixed as part of the pre-calibration pipeline processing) and the system as a whole will require significantly less maintenance effort than OTFC.

UCHCOORD and OTFR. In the near-term, observers should not run the STSDAS task UCHCOORD (Feb. 1996 version or earlier) on data processed through the new OTFR system. This task updates the header keywords which define the mapping between the pixel and world coordinate systems. The keywords include the reference pixel locations (crpix*), the values of the world coordinate system at the reference location (crval*), the partial derivatives of the world coordinate system with respect to the pixel coordinates (cd*), and the orientation (orientat). Data processed through OTFR will already have all necessary corrections performed, and it is possible that UCHCOORD would make additional, un-needed (and inappropriate) corrections.

To address this issue, a new header keyword, PROCTIME, has been added to OTFR data to record when the system processed the data. If the keyword PROCTIME is present in the header, indicating that the data came from OTFR, UCHCCORD need not be run, and should not be run until the new software version is released. This new version of UCHCOORD will be part of the summer 2001 STSDAS release; it will correctly handle both old data and new OTFR data by detecting and using the PROCTIME keyword. UCHCOORD can, of course, still be run on any of the older OTFC or tape data observers already have (i.e., images without the PROCTIME keyword). WFPC2 observers with old data are still advised to execute the STSDAS task UCHCOORD to update the position-related group header keywords.

Table 8.1: Comparison of Dataflow in On-The-Fly Systems

OTFC	OTFR
Request for data is submitted to the archive via StarView or WWW interface; archive responds with acknowledgement email.	Same as OTFC.
Raw files are retrieved from the HST archive and passed to the OTFC system. For WFPC2, the raw files include d0, q0, q1, x0, and trl files.	POD file (original telemetry file) is retrieved from HST archive and passed to OTFR system. For WFPC2, there is typically 1 POD file for each image. Pre-calibration OPUS processing is performed: data partitioning, data editing, and generic conversion; these steps generate the raw files (d0, q0, q1, x0, and trl files).
Any problems in the header keywords are fixed by special lookup table.	Same as OTFC although the OPUS pre-calibration processing will have fixed the majority of keyword problems automatically (i.e., significantly fewer header corrections required in OTFR).
The best calibration files & switches are determined by separate standalone task, and header keywords updated accordingly.	Not needed. The best calibration files & switches are set by the pre-calibration OPUS code (generic conversion).
Images are calibrated by STSDAS calxxx module and sent back to the archive system.	Same as OTFC.
Archive delivers raw + calibrated data and emails completion notification to the requestor.	Same as OTFC.