Документ взят из кэша поисковой машины. Адрес
оригинального документа
: http://www.eso.org/~qc/tqs/pyqc/pyqc.html
Дата изменения: Tue Mar 8 12:54:47 2016 Дата индексирования: Sun Apr 10 19:51:43 2016 Кодировка: Поисковые слова: р р р с р с р р р с р с р р р с с р р с р с р р п п р п п п п |
Common Trending and QC tools:
|
tqs = Trending and Quality Control System |
![]() |
new: | see also: | ||||||||
qclib.py v1.5: new function arr_MAD (2016-03-08) qc_detlin.py v1.2.6 (2016-02-19) pyqc.tar v2.0 (May 2014) Display multi-extension files:
|
- description of library QClib
|
files | description | version | date | download |
pyqc.tar | complete installation | 2.0 | 2014-05-21 | download |
QClib | ||||
qclib.py | library with generic classes and functions | 1.5 | 2016-02-18 | download |
plot modules | ||||
qc_mstplt.py | instrument-independent module plotting cuts through 2D frames, histogrames, etc. | 2.0 | 2014-05-21 | download |
qc_genplt.py | instrument-independent plotting module similar to qc_mstplt.py but with more flexibility | 1.0.2 | 2014-04-04 | download |
qc_mstdisp.py | instrument-independent module creating 2D images from a single detector (raw frame, master, comparison to reference) | 1.3.3 | 2014-04-08 | download |
qc_detlin.py | instrument-independent module for detector monitoring (needs also configuration config_detlin.py) |
1.2.6 | 2016-02-19 | download |
qc_imadisp.py | instrument-independent module creating a simple 2D image from a raw or product file; does not need additional configuration | 1.5.3 | 2014-04-07 | download |
qc_imaovw.py | instrument-independent module creating an overview of all (or of a selected number of) extensions in a raw or product file; does not need additional configuration | 1.0.2 | 2014-04-08 | download |
qc_growth.py | instrument-independent module for plotting flux growth curves of skylight flats | 1.0 | 2014-05-20 | download |
examples | ||||
repository of general example and tutorial scripts |
QC1 procedures are used to generate graphical reports of pipeline products, to calculate additional QC1 parameters that are not provided by pipeline recipes, and to control ingesting and scoring of QC1 parameters (by calling writeQC and scoreQC). They are invoked by processQC within the DFOS workflow. Due to the uniqueness of each instrument (and also due to historical development), each instrument has its own set of QC1 procedures and different scripting languages are in use. On the other hand, there are several tasks in each procedure that are common to other procedures from the same instrument or other instruments. There are also some common datatypes (mainly bias, dark, and flat frames) which do not differ much from instrument to instrument.
The pyQC tools are an offer for Python programmers. They aim at:
PyQC has three main components. On the lowest level, there is a collection of Python functions and classes (QClib) that offer standardized procedures for typical tasks (reading an AB, opening raw and product frames, plotting a cut through a 2D image, etc.). This library is imported into plot modules which typically produce a figure with several sub plots. The plot modules are coded so that they can be used for different raw types and also for different instruments; raw-type- and instrument-dependent definitions are contained in dedicated configuration files. Finally, there is typically one wrapper script per raw type that imports plot modules and executes them. The interaction between the components is illustrated in the following figure:
Wrapper script 1: RAW_TYPE_1 | ||||
Configuration
file for RAW_TYPE_1 |
Plot module A | Plot module B | ||
Conf file 1 | QClib | Conf file 1 | QClib |
Wrapper script 2: RAW_TYPE_2 | ||||
Configuration
file for RAW_TYPE_2 |
Plot module A | Plot module C | ||
Conf file 2 | QClib | Conf file 2 | QClib |
The above example has wrapper scripts for two raw types. There is one configuration file associated with each raw type. The first wrapper script uses plot modules A and B; the second wrapper also uses module A but adds functionality from module C instead of B. All modules use code from QClib.
The installation pyqc.tar offers the following Python files:
1. QClib
qclib.py: a library of Python functions and classes for QC
vimos_lib.py: an example library of additional instrument-specific Python functions and classes
2. Plot modules:
qc_mstplt.py:
creates a figure per detector with cuts through raw and product frames, histograms, etc. Useful for master frames (dark, bias, flat). Instrument-independent. Example plot VIMOS.
NEW with v2.0: plot 6 can be configured so that it contains either col/row averages (as before) or a 2D image. Example plot.
qc_genplt.py:
similar to qc_mstplt.py but with more flexibility regarding plot configuration. Instrument-independent. Example plot.
qc_detlin.py:
visualization of detector monitoring raw frames and product tables (gain and linearity); instrument-independent. Example plot.
qc_mstdisp.py:
creates 2D images from a single detector (first raw frame, master, comparison to master). Instrument-independent. Example plot.
qc_imadisp.py:
creates a 2D image from a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plot.
NEW: qc_imaovw.py:
creates an overview from all (or a selected number of) extensions of a single raw or product frame. Instrument-independent; can be called from the command line with some options; does not require additional configuration. Example plots: KMOS master dark, KMOS master flat, KMOS illumination correction.
NEW: qc_growth.py:
creates flux growth curves for a sequence of skylight flats. Requires some additional configuration which can be put in the same configuration file as for qc_mstplt. Instrument-independent. Example plot: KMOS sky flats.
vimos_4Q.py:
creates one figure with 2D images of all 4 VIMOS detectors. VIMOS-specific example script. Example plot.All plot modules can be used standalone or can be called from wrapper scripts.
3. Wrapper scripts:
vimos_bias.py:
example for putting all pieces together. Imports vimos_lib.py and calls qc_mstplt.py, qc_mstdisp.py, and vimos_4Q.py.
4. Configuration files:
config_cf.py: contains the connection between raw types and configuration files.
config_bias_lg.py: example configuration file (from VIMOS) for using the above scripts.
config_detlin.py: configuration file for qc_detlin.py.
See also the step-by-step tutorial.
1. Download the tar file pyqc.tar. Unpack it into a dedicated directory of your dfo machine (e.g. into <HOME_DIR>/python). All files stay in the same directory.
2. Copy and edit configuration files: there is typically one configuration file per raw type. It is also possible to have several configuration files per raw type that are split by instrument setting (see config_bias_lg.py). The connection between raw types and configuration files is given in config_cf.py. The latter file must keep its name.
3. The scripts expect the usual DFOS environment. ABs are expected to be in <DFO_AB_DIR>, raw and product files must be in the places specified in the AB. The AB name is given as command line option when calling a script, e.g.:
<HOME_DIR>/python/qc_mstplt.py -a <AB> .
Output is written as png files into the current working directory. Typical command line options accepted by the scripts are:
--version print version number and exit -h, --help print help on command line parameters and exit -a <AB> name of AB [required] -i, --ingest ingest QC1 parameters using writeQC, may require additional coding [optional, only for wrapper scripts]
4. For making the scripts operational, add their calls to your config.processQC.
More details on pyQC: description of classes and functions within the library QClib and an outline of the structure of wrapper scripts and plot modules.
The official Python documentation. External packages that are frequently used: matplotlib, NumPy/SciPy, PyFITS.
Feel free to use and edit pyQC scripts and modules as you like. If you make changes to instrument-independent plot modules like qc_mstplt.py it is recommend to make a copy (e.g. kmos_mstplt.py) and edit that copy. If you would like to make changes to qclib.py it is recommended to copy the affected functions or classes to an instrument-specific library file (e.g. vimos_lib.py) and to import this module in your scripts after qclib.
Suggestions for additions to qclib.py are highly welcome if they are of general interest and can be coded intrument-independently. The same is true for changes to qc_mstplt.py and other modules and for additions in the form of new modules.
Thanks to Mark and Wolfgang for starting to code QC procedures using Python and to John for testing pyQC with HAWK-I.
Send
comments to <bwolff@eso.org>
Last update: March 8, 2016 |