make printable	new:	see also:
	v1.3: - dataclient replaced by HdrDownloader	HdrDownloader (used for header downloads) ngasClient (used by dataLoader as download engine) dpQuery (used for mcalib downloads) hotfly (used by ngasClient to update the headers)
	v1.3.2: - calls new version of HdrDownloader (v1.1)	databases ngas and obs_metadata..data_products (for file availability) dfos tools ngasClient for downloading, hotfly for updating output downloaded hdr/fits files, raw and products upload/download download of hdr/fits files, raw and products

dataLoader

Description

This tool is designed to download headers and fits files from the archive (raw and products). It works on the complete set of files for the specified date. It has the following properties:

	dataLoader
download mechanism	on-line, through HdrDownloader (headers) or ngasClient (fits files)
headers: source	from SAF_IQ (keyword_repository): fully up-to-date (raw hdrs as well as product hdrs)
headers: availibility	very close to real-time (DTS transfer, within 5 minutes or less)
fits files: source	NGAS (through DTS)
fits files: header updates	fully available (raw and products)

The typical use cases for dataLoader are:

• headers: embedded in workflow tool (autoDaily)
• fits files: not needed for daily workflow (since fits files are usually downloaded by ngasClient within processAB ), but useful for historical (re) processing
• if you have lost a $DFO_CAL_DIR or $DFO_SCI_DIR directory: reconstruct it with dataLoader -d <date> -t [c | P] (product headers or fits files)

dataLoader has two main parameters: -t[ype] and -d[ate].

• -t[ype] can be h[eader], r[aw_fits], P[roduct header], c[product fits file]. The values are the same as for ngasClient (which provides internally the file-by-file download).
• -d[ate] is the operational date in DFO format (e.g. 2024-07-30).

Output

For mode h[eader], the tool downloads the header files into $DFO_HDR_DIR/<date>, by calling HdrDownloader. Headers are fully updated. Last broken template is recognized and shifted to next date, in order to avoid processing failures. Find more here.

In mode r[aw_fits], it checks, file by file, their availability in NGAS. Also, it checks if the file is already in the system. If so, the file is skipped. All available and not yet existing files are then downloaded (again with updated headers) into $DFO_RAW_DIR/<date>.

Product headers and product fits files are always downloaded in two channels: CALIB data first (into $DFO_CAL_DIR/<date>), then SCIENCE data (into $DFO_SCI_DIR/<date>). You cannot download CALIB or SCIENCE products only. You cannot download ANCILLARY files with dataLoader. All product headers or fits files go through hotfly and have updated headers. The tool calls productExplorer to get the list of all files to be downloaded.

How to use

Type dataLoader -h for on-line help.

The typical call for headers is:

dataLoader -t h -d 2025-07-30 (raw headers, updated, broken-template enabled)

dataLoader -t P -d 2025-07-30 (product CALIB and SCI headers, updated)

Fits files are downloaded by:

dataLoader -t r -d 2025-07-30 (raw fits files, headers updated)

dataLoader -t c -d 2025-07-30 (product CALIB and SCI fits files, headers updated)

Configuration file

config.dataLoader contains:

Installation

Use dfosExplorer, or type dfosInstall -t dataLoader .

Logging and status

dataLoader in mode 'header' writes the status flag 'hdr_Exist' (or 'hdr_Notexist') into $DFO_MON_DIR/DFO_STATUS. In the other modes no status flags are written.

Operational aspects

• There is no need to use dataLoader -t r routinely within the daily workflow. All files are auto-delivered by processAB and ngasClient.

• The tool is typically used to reconstruct data directories ($DFO_HDR_DIR, $DFO_RAW_DIR, $DFO_CAL_DIR, $DFO_SCI_DIR) for reprocessing.