Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.eso.org/~qc/dfos/automatic1.html
Дата изменения: Mon May 12 15:57:17 2014
Дата индексирования: Sun Apr 10 00:40:04 2016
Кодировка:
Поисковые слова: р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п р п
Operations with DFOS: daily cronjobs

Common DFOS tools:
Documentation
dfos = Data Flow Operations System, the common tool set for DFO
*make printable

Operations with DFOS: daily cronjobs

The automatic dfos system ensures that

Three core parts of dfos tools establish the automatic dfos system:

The following table gives an overview how these components work together (all tools work on latest date).

component    
calChecker
autoDaily
dfosCron -t JOBS_TREND
main step   tool      
reporting   hdr download HdrDownloader
yes
data report createReport+
yes
yes
calibration completeness   calChecker
yes
 
opslog extraction  qc1Parser
yes
yes
AB creation and processing    AB creation createAB  
yes
 
processing processAB/QC  
yes
 
scoring scoreQC  
yes
 
HC monitor affected plots trendPlotter/ JOBS_HEALTH  
yes
 
all plots trendPlotter/ JOBS_TREND    
yes
update frequency:
0.5 hr*
1 hr**
24 hr
multiple instances:
killer! younger one killed*** not an issue****

* daytime/nighttime
** while daytime calibs executing
*** based on the assumption that the older process is not hanging but still executing due to large data volume
**** trendPlotter enabled for parallel execution
+ final refresh upon histoMonitor

The following table details the autoDaily tasks.

1. PREpare
parse QC1 opslogs qc1Parser part of autoDaily
    download headers, creation of night logs: automatically done by calChecker
ngasMonitor ngasMonitor detect new data
no new data found: exit; new data found:
create report createReport -d 2010-11-15 refresh data report
create ABs: CALIB createAB -m CALIB -d 2010-11-15  
filter data
filterRaw -d 2010-11-15 -m SCAN [done within createAB ]

Next, the PROCessing part of the workflow starts, for mode CALIB:

2. PROCessing (CALIB)
create processing jobs createJob -m CALIB -d 2008-11-15 pipeline jobs (ABs) are written into a job file, for off-line processing
execute the job file [execAB call in JOBS_AUTO]

calls:
processAB --> ngasClient (data download)
execute QC jobs [execQC call in JOBS_AUTO]
included in job file
update HC monitor [execHC call in JOBS_AUTO] included in job file
Send comments to <rhanusch@eso.org>
Last update: May 12, 2014