make printable	new:	see also:
	v1.6: - creating raw file screenshots calling rawdisp.py	- createJob writes the calls of processQC into the configured JOB_FILE. - While processQC calls QC reporting, processAB calls the pipeline. - rawdisp.py is the tool to create RAW screenshots.
	v1.6.1: - added optional config key RAWDISP_SUPPORT v1.6.2: - added optional config key FINAL_PLUGIN	databases none (QC procedures and scoreQC write into QC1 database) dfos tools called: QC procedures; scoreQC; funpack to uncompress fits.fz files (tile compression, VCAM) output QC reports and QC1 parameters; used by certifyProducts upload/download none

processQC

Description

	enabled for parallel execution
	enabled for condor execution

The tool processQC is called for the specified AB. It does the following:

• it calls the rawdisp.py tool for the list of RAWFILEs extracted from the AB (if MODE=CALIB and DATE not older than 7 days);
• it reads RAW_TYPE and DATE from the AB, goes to the directory $DFS_PRODUCT/<RAW_TYPE>/<DATE> with the pipeline products, reads $PRIM_FILE from the configuration file, and launches the configured QC procedure;
• it calls scoreQC for the specified AB.

The QC procedures are instrument-specific and user-provided. They are documented here. In general they do the following:

• they extract QC1 parameters from the products of the AB;
• in some cases, they compute QC1 parameters if they are not provided by the pipeline;
• they collect the QC1 parameters and ingest them into the corresponding QC1 database (using writeQC and qc1Ingest);
• they provide customized QC plots, using the QC python library pyQC.

processQC is enabled to execute parallel jobs (N processQC calls executed simultaneously). Of course, that option requires the corresponding QC procedure to support parallel processing:

• each AB needs its own workspace;
• no temporary files under $TMP_DIR or $DFS_PRODUCT directories.

The processQC tool is usually called as part of the process job file created by createJob (execQC), but can also be called from the command line.

Raw file screenshots. As the first step, processQC calls the tool rawdisp.py which is a python tool to create screenshots. The tool creates screenshots of all RAWFILEs in the AB (or, as configured, of a subsample of those). It does so only for the 7 latest dates (counted from TODAY), since the screenshots are intended to support the daytime QC process on Paranal only.

They are organized in a web page per RAW_TYPE, collecting thumbnail plots that can be inspected at larger detail which then also includes histograms, cuts etc. Existing plots are not created again. Find more here. These web pages are created by the tool getStatusAB.

In exceptional cases (instruments with raw files as bintables), you can configure RAWDISP_SUPPORT=NO which then prevents calling qc_rawdisp.py.

Coversheet. As the last step, processQC can create a coversheet page as entry point to the QC reports. In complex situations there is more than one QC report per AB. The coversheet mechanism has been developed to support those cases in a convenient way. The coversheet is an html file which collects all QC reports for a given AB in a convenient way. It provides easy access to the QC reports in case of complicated situations, in particular MEF instruments. There are two options for its creation:

• It is created in a standard way by the function 'createCS' within processQC . If you are happy with the automatic output, you do not need to do anything.
• If not, you can create and use your own customized pgi script. Configure its name in PGI_COVER, and provide it under $DFO_BIN_DIR. Check for the procedure createCS function already implemented in processQC to have a simple-minded starting point.

A coversheet is always automatically created by processQC if more than one QC report with the $PROD_ROOT_NAME and extension gif|png|jpg is found. The coversheet name is constructed from the AB name, with extension .ab replaced by _cs.html. It is created under the same $DFS_PRODUCT directory as all other products, and later moved to $DFO_PLT_DIR (not to $DFO_LOG_DIR!).

The following rules apply (this is relevant only if you plan to use your own PGI_COVER procedure!):

• The list of all QC report files per AB (extensions gif|png|jpg) is available in $JTMP_DIR/list_graphics. The directory name and the list name is known to PGI_COVER.
• Also available to PGI_COVER is $PROD_PATH.

[Note that the pgi tool pgi_coversheet is delivered by dfosInstall into $DFO_GUI_DIR but not automatically moved to $DFO_BIN_DIR (to avoid overwriting your customized version the next time).]

Note that there is an "advanced" feature in scoreQC which you could use in your custom-built pgi_coversheet: those of your links which you mark with <!--SCORE_REPORT--> are extracted into the "QC REPORT" section of your score report. See more in the scoreQC documentation.

Output

Depending on the QC procedures; typically a set of graphical information will be created (png, ima, plt files etc.), as well as QC1 parameters which are ingested by the QC procedures, and scores created by scoreQC.

How to use

Type processQC -h for on-line help, processQC -v for the version number,

processQC -a <ab_name>

to launch the QC report for a specific AB.

For execution by condor you have to specify the option -u <user>, e.g. -u vircam, in order to make condor find the home directory and source the environment properly.

Installation

All configured QC procedures need to exist and need to be called in the proper syntax. That syntax is defined in the config file. The example MIDAS procedure bias.prg remains under $DFO_INSTALL_DIR/processQC/.

Configuration file

config.processQC defines:

Status information

None. The DFO status cal_QC and sci_QC are written by the execution file, $DFO_JOB_DIR/execQC.

Operational aspects

• Typical call for processQC: within the job file execQC
• you can call it anytime from the command line
• write your QC procedures such that they write all graphical files (png, gif, ps) to the same $DFS_PRODUCT/<RAW_TYPE>/<DATE> directory as the associated product files, otherwise the subsequent moveProducts tool will not find them, and they get lost.

Workflow description

1. prepare

1.1 Check if AB exists
1.2 read from AB: RAW_TYPE, DATE
1.3 find name of product file from config file
1.4 check if $DFS_PRODUCT/<RAW_TYPE>/<DATE> exists; go there

2. if MODE=CALIB and DATE not older than TODAY-7d and RAWDISP_SUPPORT=YES: call qc_rawdisp.py for the AB; graphics go to $DFS_PRODUCT/RAWDISP/<DATE>

3. call configured QC report

[2.1 unpack fits.fz files from RAW section in AB (relevant for VIRCAM only) ]
2.2 Launch QC report with configured call

4. call procedure createCS (or the pgi $PGI_COVER if configured) if more than one graphics file is found

5. call scoreQC (unless called per detector)

6. Measure execution time in sec and write into AB (as TQCEXEC)

7. Update .tab files (QC report, coversheet, scores)

8. Call FINAL_PLUGIN if configured