Common DFOS tools:
Documentation
make
printable
Common DFO steps and DFOS tools: architecture
1. Daily workflow
| DFOS | pre_pipeline part | part interacting with pipelines | part interacting with QC procedures | post_pipeline part |
| non-DFOS | pipeline infrastructure (esorex,
CONDOR) provided by DFS |
QC procedures (instrument specific) provided by the QC scientist |
||
| pipeline recipes (instrument specific) provided by DFS |
Details:
These steps are called per DATE, the middle part also per MODE (CALIB or SCIENCE).
| I.
Pre processing Ia. Automatic (within autoDaily), once per hour |
Ib. When new data available | II. AB processing and QC checks | ||||
| workflow step | create data report | CALIB: create ABs |
create jobs | process ABs | create QC reports | certification |
| SCIENCE: create ABs |
||||||
| DFOS tool | createReport | createAB | createJob | [JOBS]* | [JOBS]* | certifyProducts |
| calling ... | fitsreport | ABbuilder, filterRaw, fitsreport | CALIB: processAB, ngasClient |
processQC; qc1Ingest; scoreQC | ||
| status flag | report_Exist | cal_AB/ sci_AB | cal_Queued/ sci_Queued | cal_QC/ sci_QC | cal_Certif/ sci_Certif |
| III. Post processing | |||||
| workflow step |
CALIB: move to final directories |
rename | update ABs | ingest to NGAS | clean up |
| SCIENCE: | update ABs | extract associations | |||
| DFOS tool | moveProducts | [JOBS]* | finishNight | ||
| calling ... | renameProducts | updateAB | ingestProducts; dpIngest SCIENCE: verifyAB |
extractStat; histoMonitor | |
| status flag | cal_Finished/ sci_Finished |
cal_Renamed/ sci_Renamed |
cal_Updated/ sci_Updated |
cal_Ingested/ sci_Verified |
finished/ removed |
| Explanation | |
| DFOS tool | level1 (main) tool, called from the command line |
| calling ... | level 2 (helper) tool, called from a level1 tool |
| status flag | status flag to be set in log file |
| ALL | process for all files |
| CALIB | process for CALIB files |
| SCIENCE | process for SCIENCE files |
2. Packing workflow
This workflow was supported until October 2011. If you have historical interest, find its description here.
3. Workflow tools
| process step | calChecker (and telltracker*) | auto-download, CALIB processing and QC | pre-Imaging workflow | PHOENIX workflow for science processing |
| DFOS tool | calChecker | autoDaily | processPreImg* | phoenix* |
| tools called | HdrDownloader, createReport | all tools from daily workflow part I up to II (except certification) | createAB ... moveProducts, incl. processAB | all tools from daily workflow, except for createAB, certifyProducts |
| status flag | hdr_Exist, report_Exist |
4. Monitors and GUIs
| process step | installation monitor | calibration completeness | daily workflow monitor | scores overview | AB product monitor (processing and QC reporting) | productExplorer | muc Monitor (condor queue on muc blades) | cascadeMonitor (overview of executed cascades) | historical record ("nautilus") |
| DFOS tool | dfosExplorer | calChecker | dfoMonitor | scoreHC -> scoreManager.html | getStatusAB | productExplorer | mucMonitor | cascadeMonitor | histoMonitor/ phoenixMonitor* |
| calling ... | dfosInstall | HdrDownloader, createReport |
5. Trending and QC system
| process step | ingest QC1 parameters | score product quality | trending | documentation of QC procedures | maintenance of navigation bars | browse | plot | hide | python QC library |
| DFOS tool | scoreQC | trendPlotter | qcDocu | webNavbar | qc1Browser* | qc1Plotter* | qc1Hide* | pyQC** | |
| calling ... | qc1Ingest | pet.py**; scoreHC |
** set of python procedures
6. Other tools
| process step | download headers | hide files or change file headers | query products in NGAS | replace product fits files with their headers | tools for calSelector rules management |
| DFOS tool | dataLoader | hideFrame | productExplorer | cleanupProducts | createCalibMap; calselManager |
| calling ... | HdrDownloader |
|
DFOS tools: overview
|
|
Daily workflow:
pre-processing |
||||
| DFOS tool | source | DFOS tool | source | |
| HdrDownloader | DFS | trendPlotter | QC | |
| dpQuery | DFS | pet.py | QC | |
| createReport | QC | scoreHC | QC | |
| fitsreport | DFS | scoreQC | QC | |
| filterRaw | QC | webNavbar | QC | |
|
|
qcDocu | QC | ||
| createAB | QC | qc1Ingest |
DFS | |
| ABbuilder | DFS | qc1Browser |
DFS | |
| createJob | QC | qc1Plotter |
DFS | |
|
processAB
|
QC | qc1Hide |
DFS | |
|
ngasClient
|
QC | pyQC | QC | |
|
ngamsCClient
|
DFS | |||
|
hotfly
|
DFS | |||
|
processQC
|
QC | calChecker | QC | |
|
scoreQC
|
QC | tellTracker | QC | |
| cascadeMonitor, mucMonitor | QC | |||
| certifyProducts | QC | |||
Daily workflow: post-processing |
||||
| moveProducts | QC | |||
renameProducts |
QC | |||
ingestProducts |
QC | |||
dpIngest |
DFS | |||
| updateAB | QC | |||
|
verifyAB
|
QC | |||
|
calSelector
|
DFS | |||
| calselManager | QC | |||
| createCalibMap | QC | |||
| finishNight | QC | |||
extractStat |
QC | |||
| cleanupProducts | QC | |||
QC: tool
developed and maintained by QC
DFS: tool developed and maintained by DFS