Документ взят из кэша поисковой машины. Адрес
оригинального документа
: http://www.eso.org/~qc/dfos/ingestProducts.html
Дата изменения: Mon Apr 27 14:39:45 2015 Дата индексирования: Sun Apr 10 00:53:36 2016 Кодировка: Поисковые слова: п п п п п п п п п п п п п п п п п п п |
Common DFOS tools:
|
dfos = Data Flow Operations System, the common tool set for DFO |
![]() |
new: | see also: | ||||||||
v2.3: - new option -f to call fitsverify only |
- called by moveProducts - dpIngest information - how to ingest general calibration products |
|||||||||
v3.0:
|
|
This tool ingests pipeline products into the NGAS archive, after they have been certified, distributed, and renamed. Calibration products (master calibrations) as well as science products (IDPs) are supported. The tool ingests all, or selected, products from the product directory. The product directory is $DFO_CAL_DIR/<DATE> for master calibrations, and $DFO_SCI_DIR/<DATE> for IDPs.
The tool is enabled to ingest IDPs (internal data products, being science products generated by QC through the phoenix process). This feature is documented in more detail on the PHOENIX branch of the DFOS documentation here.
With version 3, the tool is enabled to ingest master calibrations created by the phoenix process. The only difference to the normal (operational) ingestion of master calibrations is the configurable behaviour how to handle previously existing instances. This is documented in more detail on the PHOENIX branch of the DFOS documentation here. If you are not using the tool for a PHOENIX project, you can safely ignore this part.
The tool is normally called from the ingestion file under $DFO_JOB_FILE that is maintained by moveProducts (this is the usual operational setup for larger data sets). It can also be called from the command line.
The tool writes a log file list_ingest_$MODE_$DATE.txt into $DFO_LST_DIR. The monitor tools dfoMonitor and histoMonitor check this file and display whether or not a certain night with calibration products has already been ingested. (Note: versions earlier than v1.2 had the log file called list_ingest_$DATE.txt.)
It uses the DFS tool dpIngest that comes with the standard DFS installation. If the configuration key QC1_UPDATE is set to YES in the configuration file, the key cdbfile in the tables of the QC1 database is updated with the new name of the calibration product (since it has been renamed from the PIPEFILE format to the CALIBFILE format in the meantime). Find more information about dpIngest here.
If configured, the tool will send an email if errors or warnings are found in the log file.
The product data are ingested into NGAS. Their metadata description is extracted from the header keys and is stored in the qc_products database (qc_files).
In order for dpIngest to work properly, two entries in $HOME/.dbrc are required:
QC_PRODUCT_HANDLER | acdb.hq.eso.org:2025 | <user> | <pass> |
KEYWORDS_REPOSITORY | asestage.hq.eso.org:7500 | <user> | <pass> |
Also, you need to make sure that the fitsio tools, in particular fitsverify, are in your $PATH. Type 'which fitsverify', and in case of 'not found', enter something like
export PATH=${PATH}:/opt/cfitsio/bin |
in your $HOME/.qcrc.
cleanupProducts. The tool writes, in mode CALIB, a call 'cleanupProducts -d $DATE' into the job file $DFO_JOB_DIR/JOBS_CLEANUP that is monitored by dfoMonitor and ready for execution after finishNight has been executed. Find more information about cleanupProducts here.
JOBS_CLEANUP: |
cleanupProducts -d 2010-04-16 |
cleanupProducts -d 2010-04-17 |
The tool scans the log file $DFO_LST_DIR/list_ingest_$MODE_$DATE.txt for error messages from dpIngest and for unusual content. It reports about occurrences such as WARNING or ERROR before finishing. It also sends an email if SEND_MAIL=YES in the configuration file (highly recommended!). Error and warning messages must be carefully observed and understood in order to ensure proper ingestion of files.
The following error message from dpIngest is especially worth mentioning:
All other messages are likely caused by problems with connecting to the archive. It must be checked that files have been successfully ingested.
ingestProducts -m CALIB -d 2004-10-12 ingest master calibrations for that date
[ingestProducts -m SCIENCE -d 2013-10-12 ingest IDPs, if PHOENIX enabled ]
[ingestProducts -m CALIB -d 2003-10-12 ingest MCALIBs, if PHOENIX and MCALIB enabled ]
Type ingestProducts -h for on-line help, ingestProducts -v for the version number, and
ingestProducts -m CALIB -d 2004-10-12
to ingest all (or specified) pipeline products for mode CALIB on that date into the product archive.
Type
ingestProducts -m CALIB -d 2004-10-12 -f
to call fitsverify only on the fits files in $DFO_CAL_DIR/<date>, and verify their fitsverify compliance. No other effect. If the tool returns an error (exit code != 0), you may want to investigate the files with an error, by following the instructions. In case the fitsverify error is a real one, and not just a warning, you should create a pipeline ticket, with the exact description of the error, to get the file structure fixed.
For MCALIB projects under PHOENIX, type
ingestProducts -m CALIB -d 2003-08-02 -D
to call the tool in DEBUG mode, with interactivity and confirmation of the important steps of deletion of old instances and ingestion of new instances. See here for more.
config.ingestProducts defines:
Section 1: general | ||
INGEST_SWITCH | ALL | SELECT | DESELECT | ALL: ingest all products found SELECT: ingest the products with pro_catg matching the values in section 2 DESELECT: ingest the products with pro_catg not matching the section 2 values |
QC1_UPDATE | YES | NO | update the cdbfile key in QC1 database tables (default: YES) |
SEND_MAIL | YES | NO | YES: an email is sent if ERRORS or WARNINGS are found in the log file. |
# special config keys for PHOENIX: | ||
PATH_TO_IT | find details here | |
CONVERTER | ||
# parameters defining the stream | ||
PROGRAM_NAME | find details here | |
COLLECTION_NAME | ||
RELEASE_TAG | ||
Section 2: List of PRO_CATGs |
||
PRO_CATG | MASTER_BIAS | |
PRO_CATG | MASTER_FLAT | |
Section 3 for phoenix-generated master calibrations only: find details here |
||
PHX_DELETE | PRO_CATG | INS_MODE |
The tool writes the DFO status cal_Ingested or sci_Ingested.
- all calib products must be ingested that are associated to science data later.