make printable	new:	see also:
	v2.1.5: - enabled for NLT	dfoMonitor createReport
	v2.2: - if PHOENIX-enabled, email signal is sent if new month is started; no changes otherwise	databases DFO_db (daily_comments); obs_metadata..data_products dfos tools getStatusAB, ngasClient , qc1Ingest output $DFO_MON_DIR/FINISHED upload/download upload to qcweb (monitor/FINISHED)

histoMonitor

Description

This tool creates and maintains a historical record of the DFO operations for a particular instrument. It is also known as the "nautilus". Why?

The histoMonitor creates a local web site that is also exported to the QC web server qcweb. It has two main parts:

1. DFO history

2. Package history

DFO history. The DFO history is the history of the daily processing workflow. It comes in three levels:

• level 1: cover page with an overview of all years currently recorded, and direct links to all months
• level 2: one page per month (the "month" page, e.g. 2008-01.html) with a record for each processed date of that month
• level 3: the AB monitor page with details for each processed AB

Level 1: This page is dynamically created each time the tool is called.

Screenshot of the histoMonitor level1 page [click to enlarge]

Level 2: This page is incrementally updated. Usually, with finishNight a new entry is created. Once a date is removed from the dfoMonitor, it shows up permanently on the histoMonitor. There is one line per date, e.g.:

DATE	NR	cdb?	sci?	SM_VM	report	statistics		ABs	comments
2007-10-06	NR			SM	report	20/3		done (14/14)	Begin of science ingestion!

• NR: link to nightlog
• cdb?: indicates that all master calibrations have been ingested
• sci?: science ABs have been verified and stored on qcweb
• SM_VM: flags if SM and/or VM science data have been taken on that date
• report: links to the data report (orange if HTML version exists)
• statistics: number of ALL data/science data; the graphical bar is proportional in length to the number of science data (truncated at a configurable value)
• ABs: link to Ab monitor
• comments: the tool page has an option to enter a comment applicable to this date

Level 3: This page is the AB monitor. Contrary to the version visible in the course of the daily workflow, this page contains all ABs, both science and calib. It is a coversheet to all log, graphical and scoring information that is stored permanently in the local file system:

• the AB itself (stored in $DFO_LOG_DIR/<date>)
• the association log (same)
• the processing log (same)
• the (first) QC plot (under $DFO_PLT_DIR/<date>)
• the scoring log ($DFO_LOG_DIR/<date>).

This page has exactly the format as the usual AB monitor, it is created by getStatusAB scanning all ABs in $DFO_LOG_DIR/<date>.

Package history (supported until 2011-10)

The package history is sorted by periods. Two periods constitute a year.

The package history comes in two levels.

Level 1: the main page, same as above. This page is dynamically created each time the tool is called.

Level 2: the period page. It lists all released packages for that period. The relation package - period is established on the basis of data acquisition: all data acquired in a given period constitute a package for that period (depending on size, this package could also come in partial packages for that period). Data for the same run_id but taken in a different period build another package, sorted under that period.

For each package, a set of parameters, files or directories is linked, like e.g. the archive list, the TOC, the observing reports etc.

DFO database table. The daily comments are written in, and read from, the database table daily_comments that is part of the DFO database (formally this is just another QC1 database table). The database is used as central storage and backup facility here, but you may also browse entries.

The histoMonitor creates a set of HTML pages under $DFO_MON_DIR/FINISHED. The pages are collected under <year>/<ym> as <ym>.html where ym stands for e.g. 2008-01 (for the data processing history), or under <period> as <period>.html (for the package history tree). The comments are stored in text files, per day, under the same directory tree.

For each date, there is also the standardAB monitor.

How to use: initial setup

The tool creates and maintains a local web site. To obtain the first set of files will require some time, afterwards it goes much faster.

Step 1: call the tool for one complete, recent month, e.g. 'histoMonitor -m 2007-12'.
It will scan all days belonging to this month.

Per date:
information is extracted from $DFO_LOG_DIR/<date>
the tool creates the AB monitor
nightlog: check in getObInfo if NLT applicable: if yes, link to NLT URL; if not, check in $DFO_LST_DIR if found locally; if not, it is created and linked
data report:
- if found, it is linked
- if not found, but headers exist: data report is created from scratch and linked
- if not found, and no headers are found: headers are downloaded with ngasClient
- if not found, no headers found, no download possible: fits files are downloaded (!), their header extracted, the fits files deleted
ingestion: either DFO_STATUS is scanned, or the list files from the ingestion are searched
statistics and SM_VM flags: taken from the report
AB monitor: created from the ABs

Note: for recent months, you will most likely have all information in your system, and the only performance issue is getting the AB monitor. For older months, with no hdr information in your system, things become more time consuming, with the header extraction becoming the dominant factor.

Step 2: Once you have a feeling how this works and how long it takes, do the same for the other months of the same year, by hand. You can also call 'histoMonitor -y 2007' to do it in one run for the whole year.

Step 3: The final step is to do this for more and more years, backwards in time, or to just call 'histoMonitor' that does the complete scan in one go (but this may take a couple of hours!).

Step 4: Fine tuning. Reviewing the full set, you may want to decide to optimize e.g. the length of the statistics bar. Re-running the tool, once the complete data reports and AB monitor pages exist, is not critical and can be done anytime.

As a rule of thumb, a full month w/o AB monitor refresh takes a few seconds, a full year about a minute.

How to use: daily increments

To add a new date, typically after finishNight:

call 'histoMonitor -m <yyyy-mm>' where the month corresponds to the date to be added. This will effectively add the new date (all steps done listed under 'step 1' above) and embed it properly in the pre-existing dates.

How to use: other calls for the DFO workflow

• histoMonitor -h for on-line help, histoMonitor -v for the version number
• histoMonitor -d 2008-01-10: will enforce re-creation of the entry for the specified date; only needed in non-standard cases (e.g. you want to test what happens with date 2004-01-10 where you have ABs but certainly no hdr information, nor a data report).
• histoMonitor -m 2006-07: create monthly overview, update navigation (but not the AB monitors)
• histoMonitor -m 2006-07 -A: same, plus enforce creation of the AB monitors, even if pre-existing (will take minutes!)
• histoMonitor -y 2006: create all monthly overviews for the specified year, update navigation (but not the AB monitors)
• histoMonitor -y 2006 -A: same, plus enforce creation of the AB monitors, even if pre-existing (will take hours!)
• histoMonitor (without argument): create all monthly overviews for all years, update navigation (but not the AB monitors)

Exported version

The qcweb version in general is visible only ESO internally. Since packages contain sensitive information (at least within the proprietary period) the package part of histoMonitor is also password protected, like e.g. the nightlog information on the calChecker output.

Configuration file

config.histoMonitor defines:

Notes:

- the tool provides a marking of the current last date visible in the dfoMonitor

- the dfoMonitor has a link to the current last page of the histoMonitor

- the histoMonitor web site is exported to qcweb.hq.eso.org/~qc/<instr>/monitor/FINISHED/histoMonitor.html that is visible ESO internally only

- the tool offers a comfortable navigation bar for the DFO workflow that however needs to be updated, under certain circumstances, on all or many other pages:

• new month added: all pages of the same year need to be refreshed (histoMonitor -y <yyyy>)
• new year added: all pages (!) need to be refreshed (histoMonitor)
• the 'home' page of histoMonitor is always updated automatically and does not need specific care

- if PHOENIX-enabled (configuration key PHOENIX_ENABLED in config.dfoMonitor is YES):

• an email is sent if a new month is started for DFOS operations
• this is interpreted as signal that for the previous month PHOENIX processing can be started on PHOENIX_ACCOUNT (also configured in config.dfoMonitor) because now all master calibrations are available, and all science ABs have been created, verified and stored.
• Find more about PHOENIX configuration on the dfoMonitor documentation site, and more about PHOENIX here.

- ignore this if your account is not PHOENIX-aware.