make printable

Note: The tool is distributed through the utilities package (utilPack).

checkConsist

Description

This tool checks for inconsistencies between the dp_headers and the data_products tables in the observations database. Such inconsistencies may rarely occur if e.g. the replication process is broken. Since the result of a scan of dp_headers ("header download") is used as reference list for fits data downloads, a file missing in dp_headers will also not be downloaded as fits file.

The tool can also check (if called with option -C) the existance of all headers in the DFO_HDR_DIR directory. It is called with that option at the begin of the certifyProducts workflow. It can then not only catch missing headers, but also call 'autoDaily -D' and thereby createAB, processAB and processQC the missing ABs, without creating unnecessary overheads and inconsistencies with the already processed ABs for that date.

It takes into account the dataclient behaviour for the last "broken" template (which is moved to the next date) and does not fire false alerts if it finds those headers.

Output

How to use

Type checkConsist -h for on-line help, checkConsist -v for the version number, and

checkConsist -d 2024-06-10

to start checking date 2024-06-10, and

checkConsist -d 2024-06-10 -C

to extend the check to the downloaded headers (operationally this is done within certifyProducts).

Installation

The tool comes as part of utilPack. Use dfosExplorer, or dfosInstall -t utilPack to install.

Configuration file

None.

Details

1. observations..dp_headers is scanned. All files within the specified date are searched, $DFO_OFFSET is taken into account.

2. observations..data_products is scanned. All files within the specified date are searched, $DFO_OFFSET is taken into account.

3. Both file lists are compared, inconsistencies are written into $TMP_DIR/con_mail and emailed to $OP_ACCOUNT as an alert. The email is only sent if the problem is detected for the first time (to avoid hourly repetitions since this tool is usually embedded in autoDaily).

The problem fix has to be initiated at a case-by-case basis.

With option -C, there is an additional test of the results of the dp_headers scan versus the content of the DFO_HDR_DIR directory.