make printable	new:	see also:
	v2.1: - supports DFOS and PHOENIX, no functional change	off-line calls of ingestProducts are collected in JOBS_INGEST and managed by dfoMonitor
	v2.2: - VCAL mechanism simpler and safer; VCAL_REMOVE obsolete	databases none dfos tools ingestProducts, renameProducts, updateAB output distributed/updated files; VCAL directory managed upload/download export to qcweb: all files (ex hdr/fits) in $DFO_LOG_DIR/<date>; all files (ex hdr/fits, ex ps) in $DFO_PLT_DIR/<date>

moveProducts

Description

This tool is called to distribute, after certification, all or selected pipeline products and graphical information to the final directories. The final directories on the DFOS systems are: $DFO_CAL_DIR/$DATE for calib data, $DFO_SCI_DIR/$DATE for science data, $DFO_PLT_DIR/$DATE for graphical information, and $DFO_LOG_DIR/$DATA for pipeline logs, association logs, trending logs, and ABs. Some data are also exported to the QC server and to the ESO web server.

The tool also calls the DFOS tools renameProducts and updateAB. Calls for product ingestion are written into the JOBS_INGEST file under $DFO_JOB_DIR. The tool also manages the content of $DFO_CAL_DIR/VCAL (the virtual calibrations container), i.e. removes those virtual calibs which became certified (REAL) in the end. All logs and plots (except postscript files) are exported to the QC web server qcweb for on-line reference.

An optional, user-provided plugin can be used e.g. to store certain products in a temporary workspace for detailed analysis. The package has an example plugin.

Find its place in the daily workflow here.

Output

How to use

Type moveProducts -h for on-line help, moveProducts -v for the version number, and

moveProducts -m CALIB -d 2014-07-12

to move all pipeline products for mode CALIB on that date to their final place (if they are configured to be moved).

Installation

Use dfosExplorer or dfosInstall . There is an optional plugin, pgi_moveProducts, which remains in $DFO_INSTALL_DIR/moveProducts. Check its function and content and decide if you want to use it.

Configuration file

config.moveProducts defines:

Status information

The tool writes the DFO status cal_Finished and sci_Finished.

Operational aspects

• The optional SPECIAL_PLUGIN can be used e.g. to store certain products in a temporary workspace for detailed analysis.
• If run in the PHOENIX environment, the tool checks if it is called from within phoenix (via 'phoenix -M' or just 'phoenix'; then it is ok), or not (i.e. from the command line). In the latter case, it refuses to start since the command-line call would transfer information to qcweb which gets mixed up with the operational (DFOS) account.

Workflow description

1. read list of RAW_TYPEs for selected mode in config file

2. loop on RAW_TYPEs:

2.1 go to $DFS_PRODUCT/<RAW_TYPE>/<DATE>

2.2 Check for existence of $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>. If it already exists, ask the user (in mode INTER) if it should be recreated (hence all existing files there are deleted) or not. If no, all existing files are named back to their PIPEFILE name. Thus, it is guaranteed that later (in step 2.12) all new files are either added, or overwrite pre-existing solutions, and in any case the versioning scheme remains valid (remember that versioning can only work on the whole data pool and not file by file).

2.3 read, under DELETE, list of PRO.CATGs to move for $RAW_TYPE in config file (or read entry ALL under RAW_TYPE);
if no PRO.CATG is listed for $RAW_TYPE (and no entry ALL exists), continue with next RAW_TYPE

2.4 create list of fits file names having movable PRO.CATG ('file_list'). If none found, continue with next RAW_TYPE
2.5 create list of associated ABs, association/trending/processing/scoring logs (evaluate ROOT_NAMEs in file_list; 'ab_list')
2.6 create list of associated further information (evaluate ROOT_NAMEs in file_list; extensions configured per RAW_TYPE in config file, e.g. gif files; 'graph_list'); if none found, or no extensions specified, skip this step
2.7 Call SPECIAL_PLUGIN, if configured; refresh file_list afterwards (since the plugin may have removed files)
2.8 move the fits files in file_list to $DFO_CAL_DIR/<DATE> or $DFO_SCI_DIR/<DATE>
2.9 move the graphics files in graph_list and the coversheet to $DFO_PLT_DIR/<DATE> and other FURTHER information to $DFO_LOG_DIR/<DATE>
2.10 delete $DFS_PRODUCT/<RAW_TYPE>/<DATE>
2.11 remove all $MODE files for $DATE from $DFO_CAL_DIR/VCAL (the container for virtual calibrations created by createAB) unless they have no MCAL counterpart
2.12 call renameProducts for MODE and DATE (if configured)
2.13 call updateAB to update the ABs with the final file names and locations
2.14 move the ABs, AB logs and processing logs to $DFO_LOG_DIR/<DATE>
2.15 update links in scoring logs (HTML), move to $DFO_LOG_DIR/<DATE>

2.16 write call of ingestProducts into $ING_FILE_NAME

2.18 CALIB: export content of $DFO_LOG_DIR/<DATE> and $DFO_PLT_DIR/<DATE> to qcweb; all links in HTML files are edited properly
2.18 SCIENCE: export content of $DFO_LOG_DIR/<DATE> to qcweb; all links in HTML files are edited properly

3. Set process status to cal_Finished/sci_Finished.