make printable

Guidelines for hiding frames

compiled by Sabine Moehler

The dfos tool hideFrame can be used to hide frames without information or with misleading information. Here are a few rules about when a file should be hidden.

• Calibration frames without useful information or with misleading information should always be hidden, irrespective of whether they are part of the standard observatory calibration or part of a PI run.
• We do not commit to 100% completeness of all cases. We hide bad frames if we encounter them, but it is not required to 'hunt' for them.
• For any hidden data a comment should be entered in raw_comments (within certifyProducts, this is trivial; outside certifyProducts, use the button "enter comments: [file]" on the raw data report).

Examples for data to be hidden are

• files without useful information
  • significantly saturated frames (i.e. not just a few pixels)
    • flat fields (e.g. twilight flats, lamp flats)
    • standard star spectra (telluric/flux)
    • standard star images (only if at least all standard stars are saturated)
    • not necessarily saturated arc frames - these might be still useful
  • frames with corrupted headers that cannot be re-constructed (this also applies to science data)
• files with misleading information
  • frames with no flux at all which should have flux (e.g. flats or arcs without lamp)
  • a flat that had also the arclamp on, or other parasitic light sources

On the other hand, a valid flat that has a wrong header keyword (e.g. DPR.TYPE) should get this value corrected (also by using the hideFrames dialog but asking DBCM for the correction), and not be hidden.

Examples for grey areas are

• partly saturated frames, which may still be of some use (e.g. telluric standard saturated in wavelength ranges far from telluric absorption bands); add comment.
• data marked 'Not Valid', which are followed by a 'Valid' dataset. In that case the 'Not Valid' data may be hidden (or have DPR.CATG set to TEST) without further checks, but may also be checked and kept if good (also partially).

In the following cases files should not be hidden, but comments should be provided either for the raw data or the products:

• data that cannot be processed by the current pipeline but may otherwise be useful (i.e. could be processed by an ideal pipeline), e.g.
  • incomplete distortion data (SINFONI)
  • standard star images with large errors in their coordinates (FORS2)
• over-/underexposed data within a template that produces good results - the bad data are probably there by intention (e.g. saturated data in some linearity tests)
• calibration data providing a valid description of the current status even if they are out of specs, e.g.
  • 'bad' zeropoints/standard stars resulting from bad atmospheric conditions
  • bias data with persistent electronic noise pattern

If you decide to hide files *after you have certified them*, it is very important to check that they are not included in any harvested AB, and that they are not available for further associations. Otherwise a calSelector user will run into trouble since these download requests are doomed to fail. Therefore, these checks should be made:

• the files in the hide request must not be contained in the current VCAL directory ($DFO_CAL_DIR/VCAL);
• the files in the hide request must not be contained in any harvested SCIENCE AB (in $DFO_LOG_DIR/CALSELECTOR/<date>).

Note that hideFrame v1.3+ takes care of these requirements.

Master calibrations created from these files must be hidden as well.