Calibration memory and AB creation

This page deals with the operational issue of how to fill the calibration memory properly, and with the interplay of createAB and that memory.

The calibration memory is the list of all calibration product files currently available for association. It is used and maintained by the association tool createAB. There are two flavours of calibration products: virtual calibrations and real calibrations.

• Real calibration products are the processed and certified products, stored in $DFO_CAL_DIR. They are either dynamic (updated regularly) or static (updated never or rarely). The dynamic calibrations are created by the pipeline from raw files and are stored in $DFO_CAL_DIR/<date>. The static calibrations normally have no parent raw file and are stored in $GEN_CALDIR (e.g. $DFO_CAL_DIR/gen).
• Virtual calibrations are calibrations which are predicted and do not yet exist as fits files at creation time. With their predicted name, they can be used for association in a cascade. They either will become available later, or will only be used for association of raw files, in which case they will never become real.

The two flavours of calibration products are collected in two different pools:

• $DFO_CAL_DIR/MCAL is the pool of all real calibrations available for association by createAB. It is filled with symbolic links to the real files in the storage area ($DFO_CAL_DIR/<date>.
• $DFO_CAL_DIR/VCAL is the pool of virtual calibrations. It is filled with header files being extracted from the parent raw files.

In either case the calibration memory is only a subset of the available dates (for performance reasons). The date range is configured in config.createAB.

The calibration memory is managed by DFOS in the following way:

• MCAL is filled from scratch by createAB each time the tool is called. The memory depth (= the number of days contained in MCAL) is determined by its reference date (the latest date in the pool) and the number of dates. The number is configured as $N_MCAL_LIST in config.createAB. The reference date is either the latest available date in $DFO_CAL_DIR: this is the default choice if createAB is called in its standard way (createAB -m <mode> -d <process_date>); or a specified date (createAB -m <mode> -d <process_date> -D <reference_date>).
• VCAL is filled and depleted incrementally by createAB/updateAB in the course of day-to-day operations. It has a depth of $N_VCAL_LIST dates as configured in config.createAB.

Virtual calibrations. In routine operations the virtual calibrations, once they are certified and become real, are automatically removed from VCAL and enter MCAL, until they finally become outdated, are replaced by similar, new ones and disappear completely from the calibration memory. The QC scientist usually does not need to take care about these directories, except for fine tuning the memory depth parameters (typically 5-10 days).

There are two exceptional cases when entries in VCAL are removed, or kept, in a user-controlled way:

• in the course of certifyProduct, a product is rejected (certify='n'); then the user is asked whether or not its name should be deleted from VCAL.
• moveProducts has the configurable option VCAL_REMOVE to remove certain products, defined per RAW_TYPE.

Criteria for removing or not are:

• "case 1: bad raw, bad product, or no product" A raw file has bad quality (e.g. zero exposure level), hence cannot produce a reasonable product. It is then completely useless and should also be removed from VCAL (to prevent its association into other ABs). The QC scientist might consider to enter a comment and use hideFrames for hiding this file in the archive.
• "case 2: good raw, bad product or no product" A raw file is good, but the pipeline failed to produce a reasonable result: then you may want to keep associations of this raw file to science files or other calibrations and hence do not remove it from the VCAL directory.
• "case 3: good raw, no product" The pipeline is not able to correctly process certain setups or even RAW_TYPEs, independent of their quality; it never produces a product which could go through certification. Then, again, you would like to keep the VCAL product for the associations.

Cases 1 and 2 are a case-by-case decision which is done as part of certifyProducts. Case 3 is rule-based and hence is configured, as part of moveProducts.

Explicit deletion of virtual calibrations. Only when there is the need to create ABs outside of the daily workflow (e.g. for a historical set of June data while DFO date is already in December), some explicit actions are necessary for calibration memory maintenance:

• delete all entries under $DFO_CAL_DIR/VCAL/.
• run createAB -m CALIB -d <june-date> -D <june-date>

Then MCAL is correctly filled with June data, and no virtual calibs from December can be associated. If you would call createAB without explicitly specifying -D <reference_date>, it would scan and find the December data instead!

If you want to switch back to December, remove again the files in $DFO_CAL_DIR/VCAL/, and call createAB again in the standard way.

The current content of VCAL and MCAL is displayed in the dfoMonitor output.