make printable	new:	see also:
	qclib.py v1.5: new function arr_MAD (2016-03-08) qc_detlin.py v1.2.6 (2016-02-19) pyqc.tar v2.0 (May 2014) includes qclib.py v1.4 and qc_growth.py (new) Display multi-extension files: qc_imaovw.py (2014-01-28)	- description of library QClib - pyQC script structure - step-by-step installation tutorial - repository of example and tutorial scripts databases none dfos tools processQC, writeQC, scoreQC output used by certifyProducts upload/download none

pyQC: Python QC tools

Background

QC1 procedures are used to generate graphical reports of pipeline products, to calculate additional QC1 parameters that are not provided by pipeline recipes, and to control ingesting and scoring of QC1 parameters (by calling writeQC and scoreQC). They are invoked by processQC within the DFOS workflow. Due to the uniqueness of each instrument (and also due to historical development), each instrument has its own set of QC1 procedures and different scripting languages are in use. On the other hand, there are several tasks in each procedure that are common to other procedures from the same instrument or other instruments. There are also some common datatypes (mainly bias, dark, and flat frames) which do not differ much from instrument to instrument.

The pyQC tools are an offer for Python programmers. They aim at:

• providing a tool set of instrument-independent Python classes and functions that encapsulate common tasks used in QC1 procedures,
• providing some basic Python scripts (i.e. QC1 procedures) that can be configured for different instruments and can be used as starting points for own developments,
• exchanging useful procedures between instruments and defining ways for code reuse.

Overview of concept

PyQC has three main components. On the lowest level, there is a collection of Python functions and classes (QClib) that offer standardized procedures for typical tasks (reading an AB, opening raw and product frames, plotting a cut through a 2D image, etc.). This library is imported into plot modules which typically produce a figure with several sub plots. The plot modules are coded so that they can be used for different raw types and also for different instruments; raw-type- and instrument-dependent definitions are contained in dedicated configuration files. Finally, there is typically one wrapper script per raw type that imports plot modules and executes them. The interaction between the components is illustrated in the following figure:

The above example has wrapper scripts for two raw types. There is one configuration file associated with each raw type. The first wrapper script uses plot modules A and B; the second wrapper also uses module A but adds functionality from module C instead of B. All modules use code from QClib.

Content

The installation pyqc.tar offers the following Python files:

1. QClib

qclib.py: a library of Python functions and classes for QC
vimos_lib.py: an example library of additional instrument-specific Python functions and classes

2. Plot modules:

qc_mstplt.py:
creates a figure per detector with cuts through raw and product frames, histograms, etc. Useful for master frames (dark, bias, flat). Instrument-independent. Example plot VIMOS.
NEW with v2.0: plot 6 can be configured so that it contains either col/row averages (as before) or a 2D image. Example plot.
qc_genplt.py:
similar to qc_mstplt.py but with more flexibility regarding plot configuration. Instrument-independent. Example plot.
qc_detlin.py:
visualization of detector monitoring raw frames and product tables (gain and linearity); instrument-independent. Example plot.
qc_mstdisp.py:
creates 2D images from a single detector (first raw frame, master, comparison to master). Instrument-independent. Example plot.
qc_imadisp.py:
creates a 2D image from a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plot.
NEW: qc_imaovw.py:
creates an overview from all (or a selected number of) extensions of a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plots: KMOS master dark, KMOS master flat, KMOS illumination correction.
NEW: qc_growth.py:
creates flux growth curves for a sequence of skylight flats. Requires some additional configuration which can be put in the same configuration file as for qc_mstplt. Instrument-independent. Example plot: KMOS sky flats.
vimos_4Q.py:
creates one figure with 2D images of all 4 VIMOS detectors. VIMOS-specific example script. Example plot.

All plot modules can be used standalone or can be called from wrapper scripts.

3. Wrapper scripts:

vimos_bias.py:
example for putting all pieces together. Imports vimos_lib.py and calls qc_mstplt.py, qc_mstdisp.py, and vimos_4Q.py.

4. Configuration files:

config_cf.py: contains the connection between raw types and configuration files.
config_bias_lg.py: example configuration file (from VIMOS) for using the above scripts.
config_detlin.py: configuration file for qc_detlin.py.

How to use

See also the step-by-step tutorial.

1. Download the tar file pyqc.tar. Unpack it into a dedicated directory of your dfo machine (e.g. into <HOME_DIR>/python). All files stay in the same directory.

2. Copy and edit configuration files: there is typically one configuration file per raw type. It is also possible to have several configuration files per raw type that are split by instrument setting (see config_bias_lg.py). The connection between raw types and configuration files is given in config_cf.py. The latter file must keep its name.

3. The scripts expect the usual DFOS environment. ABs are expected to be in <DFO_AB_DIR>, raw and product files must be in the places specified in the AB. The AB name is given as command line option when calling a script, e.g.:

<HOME_DIR>/python/qc_mstplt.py -a <AB> .

Output is written as png files into the current working directory. Typical command line options accepted by the scripts are:

--version		print version number and exit
-h, --help		print help on command line parameters and exit
-a <AB>		name of AB [required]
-i, --ingest		ingest QC1 parameters using writeQC, may require additional coding [optional, only for wrapper scripts]

4. For making the scripts operational, add their calls to your config.processQC.

More details and support material

More details on pyQC: description of classes and functions within the library QClib and an outline of the structure of wrapper scripts and plot modules.

The official Python documentation. External packages that are frequently used: matplotlib, NumPy/SciPy, PyFITS.

Editing of scripts and own additions

Feel free to use and edit pyQC scripts and modules as you like. If you make changes to instrument-independent plot modules like qc_mstplt.py it is recommend to make a copy (e.g. kmos_mstplt.py) and edit that copy. If you would like to make changes to qclib.py it is recommended to copy the affected functions or classes to an instrument-specific library file (e.g. vimos_lib.py) and to import this module in your scripts after qclib.

Suggestions for additions to qclib.py are highly welcome if they are of general interest and can be coded intrument-independently. The same is true for changes to qc_mstplt.py and other modules and for additions in the form of new modules.

Acknowledgements

Thanks to Mark and Wolfgang for starting to code QC procedures using Python and to John for testing pyQC with HAWK-I.