make printable	new:	see also:
	updates for v2.0 (May 2014)	- overview of pyQC - description of qclib databases none dfos tools processQC, writeQC, scoreQC output used by certifyProducts upload/download none

pyQC: structure of scripts and modules

All pyQC scripts and modules share a similar structure.

1. Initialization

1.1 Module import

Python offers to group functions, classes, etc. into modules which can be imported by different scripts. There are several options for module import. In order to avoid confusion if names are used by different (especially external) modules, the QC scripts use the 'from' statement only for pyQC modules, i.e.:

# QC configuration and library
from config_cf import *
from qclib import *
# instrument-specific library, imported after general library
from vimos_lib import *

Names (e.g. classes, functions) from these modules can be referenced without giving the module name. For other modules, the simple 'import' statement is used:

import string
import time

Their names must be referenced using the <module>.<name> mechanism, e.g.:

print time.localtime()

The wrapper scripts have also a section where they import the actual plotting scripts:

import qc_mstplt
import crires_mstdisp

1.2. Parse options, set up logging, and read AB

With pyQC v2.0, parsing of command line options uses argparse module:

parser = argparse.ArgumentParser(...)
# you may add your own options:
parser.add_argument('--version', action='version', version='%(prog)s ' + _version_)
# parse arguments/options:
args = parser.parse_args()

For writing logging messages to stdout, the module logging is used. This module allows control over which types of messages are written (e.g. 'info', 'warning', 'error' messages). This logging level is set:

set_logging(level='info')

The AB content is read like this:

AB = AssociationBlock(args.ab)

See also qclib.

1.3. Importing script-specific configuration

Python modules are also used for keeping script configuration. Which module is loaded depends on RAW_TYPEs and RAW_MATCH_KEYs as read from the AB. Since the module name cannot be hard coded in the script, a trick has to be used for loading:

module_name = get_confname(config_files, AB)
if module_name == '':
     logging.error('configuration file could not be found. Exit.')
     sys.exit(1)
else: 
     exec 'from ' + module_name + ' import * '
     logging.info('configuration imported from ' + module_name)

This allows to use the same script for different raw types and instrument settings by changing only the configuration file.

Some Python modules also import an optional general configuration file which can be used e.g. to globally set plot colours:

try:
     exec 'from config_gen import *'
except:
	  pass
else: 
     logging.info('configuration imported from config_gen')

1.4. Opening fits files

rawHDUs = AB.get_raw()
proHDUs = AB.get_pro()

The class methods return a Python list or a dictionary with the header units of the fits files. This makes use of PyFITS. See also qclib.

2. Main part

The wrapper scripts execute functions from the plot modules:

plot_index = qc_mstplt.draw_plots(AB, rawHDUs, proHDUs, plot_index=1)
plot_index = crires_mstdisp.display2D(AB, options, proHDUs, plot_index)

The name 'plot_index' is used internally for determining a tag for the output plot file name (e.g. '*X00.png').

The plot modules have the function definitions here:

def draw_plots(AB, rawHDUs, proHDUs, plot_index=0):
     ...
return plot_index 

3. Post-processing

3.1 Calculate QC1 parameters

QC1 parameters that are not (yet) determined by pipeline recipes can be calculated here, e.g.:

if not args.calc_qc:
    snr = [0, 0, 0, 0]
else:
    logging.info('calculating additional QC1 parameters')
    snr = crires_calcqc.get_snr(proHDUs, catg_master)

3.2 Storing into QC1 database

if not args.ingest:
    logging.info('QC1 parameters not ingested')
else:
    logging.info('ingesting QC1 parameters')
   
    civil_date = AB.content['DATE']
    tmp_dir = os.path.expandvars('$TMP_DIR')
    for ext in allow_ext:
         os.system('writeQC -m PREPARE -a ' + AB.content['AB_NAME'] + ' -e ' + str(ext))
         execfile(tmp_dir + '/' + AB.content['AB_NAME'] + '_' + str(ext) + '/writeQC.py')
         os.system('writeQC -m INGEST -a ' + AB.content['AB_NAME'] + ' -e ' + str(ext))
    # if applicable: average QC1 parameters across detectors
    os.system('writeQC -m AVERAGE -a ' + AB.content['AB_NAME'])

4. Standalone usage of plot modules

Python does not distinguish between programs and modules: all programs are modules. It is possible to code Python modules in a way so that they can be imported into other modules and can also be used standalone. This requires that an additional function is defined and only executed in the standalone case:

def draw_plots(AB, rawHDUs, proHDUs):
    # define plotting function
    ...
    return

def main():
    # main() is only called in standalone mode
    # do some preparations first
    ...
    # execute function(s)
    draw_plots(AB, rawHDUs, proHDUs)

# distinguish between module import and standalone usage
if __name__ == '__main__':
    main()

This is especially useful for testing individual procedures.