make printable	new:	Note:
	v1.4: - display virtual products (for calSelector) v1.5: - new column 'pipefile'; mark multiple entries	databases qc_metadata..qc_products; qc1_daily_comments; ngas..ngas_files dfos tools ngasClient cpDelete histoMonitor output $DFO_MON_DIR/productExplorer.html and $DFO_MON_DIR/productExplorer1.html upload/download none

productExplorer

Description

This is a helper tool for exploring and managing the content of the qc_metada..qc_products database. This database stores all QC-ingested product fits files (calib, science: until 2011-10, ancillary: until 2014-05). It offers a user-friendly way to query ("explore") its content, and fine-tune the query. There are options to download the results or to delete entries.

It launches a direct isql query to qc_metada..qc_products.

It displays its result in a html table which can be sorted and filtered in the standard way, plus an executable download file, plus a deletion file for ingested products (to prepare for ingestion of an improved version).

How to use

Type productExplorer -h for on-line help, productExplorer -v for the version number,

productExplorer -d 2017-10-15

to explore calib for that date and the default $DFO_INSTRUMENT,

productExplorer -d 2017-10-15 -D 2007-10-30 -i CRIRES

for the specified time range and instrument,

productExplorer -d 2017-01-01 -p LINE_TABLE

for the specified pro_catg value only, and

productExplorer -d 2017-01-01 -p LINE_TABLE -s grism_256

to filter from the result set those files with setup value "grism_256". This filter option is done after the isql download, at the unix level, for performance reasons.

Result HTML file. The result file comes as an HTML file, similar in format to the (raw) data reports created by createReport. It uses color-coding to distinguish calib (green) from science (blue) files. It also supports ancillary (log) fits files (displayed yellow). In standard (date) mode, it displays daily comments just like the raw data reports.

Product report for 2008-03-13 (products ingested into NGAS)
CAL SCI ANC ALL number: 77 size (MB): 115.1 3.2 8.8 127.1
Date comment:	Last night of old CCD Bruce; IFLATs taken; STD efficiency
[read query] | [read download file] | [read deletion file]

type*	origfile	setup**	pipefile	mjd_obs	pro_catg	ins_mode	dp_id
VIRTUAL	V.GIRAF.2008-03-14T02:29:04.977_005.fits		NULL	54539.163178	SCIENCE_RBN_SPECTRA	MED	V.GIRAF.2008-03-14T02:29:04.977_005.fits
SCIENCE	GI_SRBS_287119_2008-03-14T02:29:04.977_IFU2_L543.1nm_o3.fits	IFU2_L543.1nm_o3	r.GIRAF.2008-03-14T02:29:04.977_tpl_0001.fits	54539.103530	SCIENCE_RBNSPECTRA	IFU	S.GIRAFFE.2008-04-01T09:28:10.770
CALIB	GI_PIME_080313A_Argus_L427.2nm_o4_LTT3218.fits	Argus_L427.2nm_o4_LTT3218	r.GIRAF.2008-03-14T10:39:42.5009_tpl_0006.fits	54539.163178	STD_RCERRORS	ARG	M.GIRAFFE.2008-04-01T09:00:16.159

Filter and sort. The HTML table can be filtered (any string from the content can be used). You can filter e.g. to display only MBIA files, or files containing the string U_BESS etc. Sorting can be done by any column.

Virtual products. With v1.4, virtual products (defining the calSelector datasets) are displayed. You can toggle between displaying and hiding them, nearby the 'Filter' option.

Multiple entries. In particular static calibration files could have been entered multiple times with different origfile names. These files, as being identified by the combination of pipefile, pro_catg and mjd_obs, are marked in red font color.

Download. The files can be downloaded, by using the executable file $TMP_DIR/mcalibDownload, either as it stands or after further editing. It uses ngasClient for the download. All downloads go into $DFO_CAL_DIR/DOWNLOAD and can then be further distributed by the user.

Deletion. The tool supports deletion of products. It provides the results of the query in a text file, with the proper syntax prepared for deletion of these files. However, without further editing, deletion will be performed only in simulation mode. Only if the user edits the deletion file to the full syntax, e.g.

dpDelete GI_MBIA_...fits -force

these files will get deleted. Deletion means physical removal and deletion in all database tables. Deletion is not hiding: after deletion, you cannot reconstruct the file content from information stored anywhere in NGAS. You can, however, ingest a new version of this file from your local file system, and this is the main purpose of the tool.

The deletion mode should be used with care.

Installation

Use dfosInstall.

Configuration file

none.

Operational aspects

• Note that this tool applies $DFO_OFFSET in the definition of the time ranges. This is your local, current value, so it might deviate from the historical value on your colleague's machine when the data were named and ingested.