Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ îðèãèíàëüíîãî äîêóìåíòà : http://www.eso.org/~qc/dfos/automatic1.html
Äàòà èçìåíåíèÿ: Mon May 12 15:57:17 2014
Äàòà èíäåêñèðîâàíèÿ: Sun Apr 10 00:40:04 2016
Êîäèðîâêà:
Ïîèñêîâûå ñëîâà: asteroid

Operations with DFOS: daily cronjobs

Common DFOS tools:
Documentation

dfos = Data Flow Operations System, the common tool set for DFO

make printable

Operations with DFOS: daily cronjobs

The automatic dfos system ensures that

all new files (headers and fits files) are detected
reports are updated
calibration completeness is monitored
CALIB data are processed
the QC1 database is populated with new QC1 parameters
new opslog data are found
affected Health Check plots are updated

Three core parts of dfos tools establish the automatic dfos system:

calChecker
autoDaily
JOBS_TREND and JOBS_HEALTH.

The following table gives an overview how these components work together (all tools work on latest date).

component			calChecker	autoDaily	dfosCron -t JOBS_TREND
main step		tool
reporting	hdr download	HdrDownloader	yes
reporting	data report	createReport+	yes	yes
calibration completeness		calChecker	yes
opslog extraction		qc1Parser		yes	yes
AB creation and processing	AB creation	createAB		yes
	processing	processAB/QC		yes
	scoring	scoreQC		yes
HC monitor	affected plots	trendPlotter/ JOBS_HEALTH		yes
HC monitor	all plots	trendPlotter/ JOBS_TREND			yes
update frequency:			0.5 hr*	1 hr**	24 hr
multiple instances:			killer!	younger one killed***	not an issue****
* daytime/nighttime while daytime calibs executing * based on the assumption that the older process is not hanging but still executing due to large data volume **** trendPlotter enabled for parallel execution + final refresh upon histoMonitor

The following table details the autoDaily tasks.

1. PREpare
parse QC1 opslogs	qc1Parser	part of autoDaily
		download headers, creation of night logs: automatically done by calChecker
ngasMonitor	ngasMonitor	detect new data
no new data found: exit; new data found:
create report	createReport -d 2010-11-15	refresh data report
create ABs: CALIB	createAB -m CALIB -d 2010-11-15
filter data	filterRaw -d 2010-11-15 -m SCAN	[done within createAB ]

Next, the PROCessing part of the workflow starts, for mode CALIB:

2. PROCessing (CALIB)
create processing jobs	createJob -m CALIB -d 2008-11-15	pipeline jobs (ABs) are written into a job file, for off-line processing
execute the job file	[execAB call in JOBS_AUTO]	calls: processAB --> ngasClient (data download)
execute QC jobs	[execQC call in JOBS_AUTO]	included in job file
update HC monitor	[execHC call in JOBS_AUTO]	included in job file

Common DFOS tools: Documentation

Operations with DFOS: daily cronjobs

Common DFOS tools:
Documentation