Common DFOS tools:
Documentation
|
| dfos
= Data Flow Operations System, the common tool set for DFO |
scoreHC
 enabled
for parallel execution |
|
embedded in trendPlotter:
|
![[ top ]](/images/arr1red-up.gif "[top]")
Description
This tool connects scores, as created by scoreQC, and the Health
Check (HC) plots, as created by trendPlotter. It re-arranges the scores,
originally derived per QC1 parameter and AB, into a scheme which focuses on instrument
properties. The tool is called by trendPlotter, but can also be called
from the command line.
The goal of trendPlotter+scoreHC is to come from QC1 parameters of
individual frames to a statement about the current instrument status, and raise alerts
in due time. Technically, this is achieved by representing each HC plot by its score report
counterpart, also known as QUICK-LOOK version. The creation and delivery of that
version is provided by scoreHC.
The tool is called for a specified report: scoreHC
-r <report_name>
(just like trendPlotter).
It collects all QC1 parameters used in that report over the last 7 days,
and reads their scores from the qc1_score database
table. This is a two-step reduction of complexity: an instrument property is represented
by a set of QC1 parameters (step 1), and these values are then scored and replaced by numbers
0 or 1 (step 2).
Resulting scores from the last 7 days before the last date indicated in the
report are finally aggregated (older scores are ignored since they are considered
to be irrelevant for the current status):
- all scores per plot are evaluated: the last one is used for higher-level scores, the
others are just displayed
- higher-level scores: each last score gives the plot score
- all plot scores are aggregated to the report score
- all report scores are aggregated to the group score (representing one aspect/component
of the instrument)
- all group scores result finally in the instrument score
Aggregation means a value of 0 or 1 where 0 can only be achieved if all component scores
are 0. Thereby a single last plot score will turn all affected higher-level scores to 1,
including the instrument score. This is an efficient hierarchical alert system.
All aggregated scores (derived from scores of QC1 parameters) are also called
"Health Check scores" or HC scores (thus the name of the tool). HC scores come
either as 0 (displayed green) or 1 (red).
Contrary to QC1 scores, there are no yellow scores.
SCORE_DEPTH is hard-coded to 7 days (to align with calChecker). SCORE_DEPTH
is counted backwards from the date with the last available score (as indicated in the score
report). Thereby it is usually guaranteed that at least one score is visible and evaluated,
even if it is already weeks old. This is because that score is considered as the latest
available information, which might be outdated but will be replaced only by a new measurement.
Score calculation and output. The tool reads the trendPlotter configuration
of the HC report, to derive the names of the QC1 parameters. Then, it reads the corresponding
scores from qc1_score, and arranges the results in a graphical way using HTML format:
| FF level (scores, last 7 days up
to 2010-01-31*) |
 |
1: Med1
3| |
2: Med2
2| |
|
4: IFU1
4|  |
5: IFU2
2| |
6: Argus
3| |
|
QUICK-LOOK version |
The format of this table is arranged exactly the same way as the original
plot:
 [click
to enlarge] |
HEALTH CHECK (HC) plot |
Thereby, the HC report can be represented by its QUICK-LOOK version for
quick reference. The resulting HC report then comes in two flavours, one of which is provided
by scoreHC
(just scores and HTML) ,
the other by trendPlotter (graphical information).
Information on demand. Whenever useful, you can get more score information
on demand:
- The little score symbols are hyper-linked to the corresponding
score report (try in the example score report).
- On moving the mouse over the score symbols you can see the corresponding
date to which the score applies.
Comment
section. Just below the score report,
there is the News section, replicated from the HC report. It displays three kinds of news:
- 'General news' (of general interest for the HC monitor)
- 'Instrument news' (applicable to the instrument as a whole) and
- 'Report news' (news for this report).
The 'Instrument news' and the 'Report news' are
managed locally with the scoreManager. With v2.0, the 'Report
news' are managed with the web-based php script hcComments.php,
see below.
There is also an extraction of the product comments.
This is intended to align with the calChecker ANALYSIS section. For each
score with a product comment (entered upon certification), there is a row with the corresponding
DATE (linked to the AB product monitor), the name of the AB (linked to the score report),
the score symbol, and the product quality comment. Note that the comment does not necessarily
fit to the particular report displayed here, it is usually entered for the whole set
of products and QC1 parameters.
| General news: |
2010-02-01:
DTS works again. |
| GIRAFFE news: |
2010-02-02: FFLAMP flux is now ok.
|
Report
news:  |
2010-01-23:
IFU1 flats saturated
|
|
News section |
| FF level (scores, last 7 days up
to 2010-01-31*) |
| |
1: Med1
3| |
2: Med2
2| |
|
4: IFU1
4| |
5: IFU2
2| |
6: Argus
3| |
|
score report |
|
|
Product quality comments |
SCORE OVERVIEW AND MANAGER. In addition to
the QUICK-LOOK score output, scoreHC also creates two
overview pages:
Overview. The file score_overview.html provides an overview of all
current scores per instrument. It is exported to the HC site and can be found under the
URL http://www.eso.org/qc/<INSTRUMENT>/common/score_overview.html.
Find the SINFONI example here.
It comes in the following versions:
The red score comments are helpful, both for Paranal and QC, to better and
faster understand issues related to red scores. The all score comments are
primarily useful for the QC scientist to monitor outdated report comments. The query form
can be used by Paranal and by QC to create, update or delete comments.
The comment interface has a mandatory DATE field to be filled. This field
is used by 'trendPlotter' to check if the comment is still
valid. After 14 days, the comment is considered outdated, and the tool starts to send an
email to the QC scientist with this information. It is up to the QC scientist to decide
what to do: most likely delete, or update. The tool does not delete outdated
comments by itself. Report comments with historical interest should
go to the
"HISTORY" part of the tutorial pages. General comments should go to documentation
section ('This plot').
Score manager. There is also a local scoreManager file ($DFO_TREND_DIR/reports/scoreManager/scoreManager.html).
This is an interactive GUI to manage scores. Here you can obtain an overview of
all AB which contributed to the current instrument score. It helps you understanding
how the HC scores are linked to the QC1 scores.
The local score manager has, apart from the main page, one detailed page
per report, which is described below.
stat_grp file. The main scoreManager.html file has a useful feature
that is not available on the exported score_overview.html file: it has a column named stat_grp that
links to the technical trendPlotter result tables under $DFO_TREND_DIR/reports/stat_files.
These files show you the names and group scores for each HC group and thereby help to
better understand, and possibly debug, the configuration of a report group.
The 'DETECTOR' link displays the stat_grp_DETECTOR file (the report group
file for the DETECTOR group):
|
DETNOISE -99
DETCONTA 0
BIASFULL -99
DETCONTAFULL -99
DETLIN 0
DARK 0
BIAS 0
|
It has a listing all report names for that group and their report scores (0:
OK; >0:
NOK; -99: not implemented).
Find a complete example here.
Consistency
checks I. The scoreManager assists
you in checking for hidden 'zombie' group scores. This is a notorious problem if at some
point your HC report system has been modified (groups have been re-arranged; group names
or report names have been changed; tests have been executed). Then it might happen easily
that a score file of an inactive report, or group, survives. If that inactive file would
store a red score, this alert would propagate and could not be fixed by any action on active
reports.
Therefore the scoreManager scans your system for inactive group
scores. It checks for two aspects: inactive group files, and invalid content of (active
or inactive) group files. In either case, the found candidates are displayed and marked
in red (example).
Their analysis and removal is left to you. If the system is consistent, all is marked green,
like in the example here.
A brute-force approach, as devised by Mark Neeser:
1. delete all stat_* files in your $DFO_TREND_DIR/reports/statFiles/ directory
2. re-create the contents of your statFiles directory by running scoreHC
-r <report_name> for all the reports listed in your JOBS_TREND file.
This should eliminate all of the zombies created in past scoring experiments, but make
sure to reconstruct the contents (step 2) as soon as possible after the deletion, otherwise
your HC scoring system is out of order!
Consistency checks II. The tool does a second
kind of consistency check. It checks, for a given QC1 parameter and data type, if there
is an inconsistency between thresholds configured in config.scoreQC, and in a related
HC report (config.tp_<report>).
An example: the struct_row parameter for
data type BIAS is
scored with fixed thresholds 0.07 and 0.10, while there is a BIAS HC
report for the same parameter with thresholds 0.05 and 0.12. Then, a value of struct_row=0.11
would produce a red score without graphically indicating an outlier in the HC report.
This would cause confusion. Instead the user should link
the scoreQC configuration to the trendPlotter configuration (dynamic configuration).
This configuration problem is hard to discover. Therefore
scoreHC checks for those cases. It sends an email reminder to the QC scientist when
a potential inconsistency is found, along with some proposal how to improve.
There might be cases where this non-standard situation nevertheless makes
sense, then the user can keep the configuration and avoid future alerts by marking the corresponding
line in config.scoreQC by #STATIC (at the end of the line). Known such exceptions are:
- The HC plot displays multiple data sets (distinguished by setup), but
scoreQC has configured "ANY". Then you should take care to configure the "most appropriate"
value in scoreQC and flag it (#STATIC).
- The HC plot displays a subsample or a supersample of the data set scored in scoreQC.
Consistency checks III and IV. These checks have been introduced with v2.3.
Because the HC_PLOT section of config.scoreQC is important for the incremental update of HC reports within autoDaily, the configuration is checked in two directions:
- All HC reports configured in the HC_PLOT section of config.scoreQC should have their config file in $DFO_CONFIG_DIR/trendPlotter/config.tp_<...>. This check protects against obsolescence of entries in the HC_PLOT section.
- All config.tp_<...> files should have a report configured in the HC_PLOT section. This check protects against forgotten HC plots in the scoreQC configuration, which can happen easily with new reports. The consequence would be that the corresponding HC report does not get updated by autoDaily (only by the bulk refresh once a day but even then only if included in the jobs file.).
The results of both checks appear as section 3 and 4 on the scoreManager. Inconsistencies in part 4 are also mailed once a day to $OP_ADDRESS, because of their fundamental importance.
Detail pages per report.
Quick
updates. The detail
report pages have on top a function to quickly update the NEWS file. The 'edit' button
calls the web-based form hcComments.php (the
same which is linked to the score overview page). All edits are done on the web. This
local page cannot be refreshed by the php form (since it cannot start local processes),
and it cannot include the new content of the news (since this works only on the web server).
Hence only you can refresh the local content, and this is the purpose of the 'refresh'
button. If you forget to refresh that page, the next automatic trendPlotter process will
do.
| GIRAFFE news: |
|
 |
| Report news: |
|
If you have edited the report news, they are visible on the Web
but not locally. Don't
forget to refresh the page.
|
| HC
report |
|
|
|
Each detail report page is structured like the following example (find the full
page here).
The content is extracted from the AB status pages:
1. Score results
[this repeats the score report] |
| BIAS parameters
(scores, last 7 days up to 2008-12-12) |
 |
1: BIAS_level
6| |
2: RON_raw
6|  |
3: RON_master
6| |
4: struct_X
6| |
5: struct_Y
6| |
|
|
| |
2. Extraction from AB monitors [don't
expect links to work in this environment!]
(list of all ABs which have entered the current score result table, sorted by DATE, extracted
from AB monitors) |
|
|
3. Details per plot and QC1 parameter [don't
expect links to work in this environment!]
This is a list of all QC1 parameters and scores for
the BIAS report on the HC monitor. It is not complete in terms of ABs.
This list can be used to correct, remove or override scores interactively. |
 Workflow
for correction: |
1a. modify configuration |
1b. call trendPlotter for report --> update stat_text file
and plots |
1c. call scoreQC for AB --> write new scores into db |
2. call scoreHC --> update HC monitor |
 Workflow
for deletion: |
1. delete all scores for selected AB in db |
2. call scoreHC --> update HC monitor |
 Workflow
for NOK override: |
1. force all scores for selected AB to green/OK |
2. call scoreHC --> update HC monitor |
|
| Action |
Plot |
QC1 table |
QC1 param |
DATE |
MJD_OBS |
score |
AB |
| Plot 1: | giraffe_bias | median_master | 2008-10-09 | 54749.646892 | 0 | GIRAF.2008-10-10T15:31:31.463_tpl.ab |
| Plot 1: | giraffe_bias | median_master | 2008-10-11 | 54751.426903 | 0 | GIRAF.2008-10-12T10:14:44.424_tpl.ab |
| Plot 1: | giraffe_bias | median_master | 2008-10-12 | 54752.504637 | 0 | GIRAF.2008-10-13T12:06:40.624_tpl.ab |
| Plot 1: | giraffe_bias | median_master | 2008-10-12 | 54752.777497 | 0 | GIRAF.2008-10-13T18:39:35.736_tpl.ab |
| Plot 1: | giraffe_bias | median_master | 2008-10-14 | 54754.531288 | 0 | GIRAF.2008-10-15T12:45:03.281_tpl.ab |
| Plot 1: | giraffe_bias | median_master | 2008-10-15 | 54755.537979 | 0 | GIRAF.2008-10-16T12:54:41.381_tpl.ab |
|
| Plot 2: | giraffe_bias | sigma_raw | 2008-10-09 | 54749.646892 | 0 | GIRAF.2008-10-10T15:31:31.463_tpl.ab |
| | Plot 2: | giraffe_bias | sigma_raw | 2008-10-11 | 54751.426903 | 1 | GIRAF.2008-10-12T10:14:44.424_tpl.ab |
| | Plot 2: | giraffe_bias | sigma_raw | 2008-10-12 | 54752.504637 | 1 | GIRAF.2008-10-13T12:06:40.624_tpl.ab |
| | Plot 2: | giraffe_bias | sigma_raw | 2008-10-12 | 54752.777497 | 1 | GIRAF.2008-10-13T18:39:35.736_tpl.ab |
| etc. ... |
|
The
red button will call 'scoreQC -a <ab>'. This might be reasonable
when you have modified thresholds definitions. Because of the complex link between config.scoreQC
and HC configuration (dynamic thresholds), the HC report needs then to be refreshed, then
scoreQC is called, and finally scoreHC is called (for a second time) to transfer the updated
scores to the HC monitor.
The blue button (offered only for NOK scores)
offers deletion in qc1_score. This might be reasonable when you discover that a certain
file was invalid, and its scores would raise a false alert. The associated workflow also
updates the HC monitor.
The green button (offered only for NOK scores) offers overriding a red score.
This might become useful when you have analyzed an issue, find that in this particular
case there is no issue, or the issue has been solved, but you have good reasons to not
modify the configuration. The associated workflow also
updates the HC monitor. Make sure to not call 'scoreQC' again for that AB: since you have
not modified the configuration, you will see agin red scores.
Note that for instruments producing MEF files (like CRIRES or HAWKI), results
from a particular AB will display in several lines. For instance, for CRIRES, there will
be six lines (4 detector entries, plus AVG and RMS). For consistency, the blue buttons
will delete all scores for all extensions from the AB. For our CRIRES example this means
a single delete action will delete all six lines at once.
score_list_<instr>.txt file. This is a temporary output file used for the evaluation
of the Health Check site. It is created under $TMP_DIR.
MULTIPLE.
As a general rule, scoreHC requires single data sets per plot (where OPLOG data sets are
not counted since they never get scored). This has two reasons: one is that the tool would
then become overly complex. The other reason is that displaying multiple data sets in a
single plot with different threshold values and rules makes the graph quickly unreadable.
Therefore, scoreHC checks for these cases and returns "multiple
dataset" instead of
scores.
There is one exception to this general
rule: if you define two or more data sets for a given plot in
your trendPlotter configuration file, but make sure that you have only one dataset
with a scoring condition (section SCORE_COND in the trendPlotter configuration), then
that dataset gets scored and displayed as QUICK-LOOK (but all others not). A typical application
would be that you want to display all flatfield fluxes in the J filter (dataset #1), and
also overplot (in different colors) the ones for different plate scales (data sets 2...n).
Then you can score dataset #1 and leave the others
unscored, by defining SCORE_COND for that dataset only. As soon as the tool detects more
than one SCORE_COND for a plot, this is handled as "multiple
dataset" case. An example is found here.
Output
- score HTML files: all HC score results are wrapped in a piece of HTML code,
exported to the Health Check web site and linked dynamically (through the virtual include
mechanism) to the HC pages.
- score_overview.html and scoreManager.html
How to use
Type scoreHC -h for
on-line help, scoreHC -v for
the version number,
scoreHC -r <report_name>
to create the QUICK-LOOK version of report <report_name>. This syntax is
exactly the same as for trendPlotter.
Type
scoreHC -P
to create the php scripts and directory structure for the comment interface,
or to update its code.
Type
scoreHC -I
to create only the local scoreManager interface (without creating a score
report, and hence faster).
You can call the tool on the command-line. The standard use is from trendPlotter.
There is also a special option
scoreHC -O
that creates the common score
overview page including the 'last 7' and 'last 30' pages. It is *not* required to be run
in routine operations.
Configuration file
For normal use, there is no tool-specific configuration. Most required configuration is read from config.trendPlotter or from the report configuration file, config.tp_<report_name>.
Note that some information is also read from config.scoreQC, and that the usage of multiple configuration there (MULTI_CONFIG=YES) has certain implications:
If MULTI_CONFIG=YES in config.scoreQC, and if
a REPORT contains plot(s) that have mixed RAW_TYPE, then you should specify the proper config file for scoreQC for each plot in the report. This has to be done in config.scoreQC, in the (new) section 4.1A. Please check details there.
For the special mode 'scoreHC -O' that creates the all_score_overview pages for the last 7 and 30 days (accessible from the shiftleader's page), there is the configuration file config.scoreHC. It is called by the hard-coded account $CREATE_OVERVIEW_USER. This configuration file has the currently active TELescopes (UT1-4, VLTI, survey telescopes) and INStruments.
Operational
aspects
- Manage the report news such as to have them always up-to-date and relevant. "News" will
typically become outdated after two weeks or so. They should then be removed.
- See also sections Hints and Pitfalls in the scoreQC and trendPlotter documentation.
- The consistency checker of the scoreManager lists in section #4 all config.tp_<...> files that are not configured in the HC_PLOT section of config.scoreQC. Because of the operational importance of that section, any inconsistencies are also mailed, once a day. There are several options to resolve these inconsistencies, and get rid of the daily reminder mails:
- A config.tp file is really obsolete: then you may want to delete it, or move it to some OBSOLETE folder; scoreHC will only scan $DFO_CONFIG_DIR/trendPlotter.
- A config.tp file is obsolete for operations but you are not yet sure: rename to e.g. AA_config.tp; scoreHC will only scan for files named like config.tp_<...>.
- A config.tp file is operationally used for a FULL plot but not configured in the HC_PLOT section: include it in the HC_PLOT section.
- How to include the scoreManager in the local monitor navigation bar:
- edit config.dfoMonitor to include 'scoreManager' (col. #3) and '<expanded path to
scoreManager>scoreManager.html' (col. #4)
- push "navigation bar: [refresh]" on the dfoMonitor
- now the scoreManager is embedded in the navigation bar of dfoMonitor; the other tools
follow as soon as they are refreshed.
make printable![[ used databases ]](/%7Eqc/dfos/database.png "[ used databases ]"){kind=small}
databases ![[ used dfos tools ]](/%7Eqc/dfos/tool_icon.gif "[ used dfos tools ]"){kind=icon}
dfos
tools ![[ output used by ]](/images/text.gif "[ output used by ]"){kind=small}
output
used by![[ upload/download ]](/%7Eqc/dfos/download-page-blue.gif "[ upload/download ]"){kind=small}
upload/download
