Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.eso.org/~qc/dfos/ABbuilder.txt Дата изменения: Fri Feb 17 16:02:48 2006 Дата индексирования: Tue Oct 2 01:19:48 2012 Кодировка: Поисковые слова: п п п п п п п п п п п

NAME
ABbuilder - DFO data organization tool

SYNOPSIS
ABbuilder
where are:
--raw-dir directory containing raw data
--cal-dir directory containing calib data
--cal-ldir directory containing symbolic links to calib files
--cal-list mcalib list
--vcal-dir directory containing virtual products
--pro-dir where pipeline products are created
--ab-dir where association blocks are created
--oca-rul file containing OCA rules
--instrume instrument
--date YYYY-MM-DD
--mode { CALIB | SCIENCE | ALL(default} }
--help print this help
--version print version


DESCRIPTION
The ABbuilder tool organizes the input data set into Association Blocks
according to the given Classification-Organization-Association rules.
The ABbuilder application performs the following steps:

1- load the OCA rules. If a syntax error is detected during this
phase, ABbuilder reports a message indicating where the syntax
error occurred, and the application terminates.

2- read the required keyword values from all of the *.fits *.tfits *.hdr files
under the raw data directory (including subdirectories).

3- read the required keyword values from all of the *.fits *.tfits *.hdr files
under the calib data directory (including subdirectories).
If the virtual product directory is specified (see option --vcal-dir),
read the relevant keywords from all virtual products under this directory.

4- classify all of the raw files, using the OCA classification rules
("if..then" statements)

5- organize the raw files into groups, according to the OCA organization rules
("select execute" statements)

6- for each group defined in the previous step, find the associated files
and recipe using the association rules specified in the relevant
action

7- for each group, create an association block in the AB directory

8- If the virtual product directory is specified (see option --vcal-dir),
create virtual products in HDR format under this directory.


SPECIFIC PARAMETERS
ABbuilder expects the following parameters to be defined in the OCA rules.

- PACK.DIR : using the classification rules, the user can define data type
specific pack dirs. For example:
if RAW.TYPE=="BIAS" then PACK.DIR="/data/pack/bias"
The value of PACK.DIR is written in the Association Block, under PACK_DIR.

- PRO.EXT : this parameter should be used in the (virtual) product definition
to specify the predicted extension of the product filename. For example:
product mbias { PRO.CATG="MASTER_BIAS"; PRO.EXT="0001.fits"; }

If PACK.DIR or PRO.EXT is not specified, ABbuilder will generate an error message
but the association blocks will still be created.

OPTIONS
The following options are available:

--raw-dir
Specify a directory containing raw (input) data.
ABbuilder will organize all FITS and HDR files under this directory
(including sub-directories).
This option is mandatory.

--cal-dir
Specify a directory containing calibration data.
ABbuilder will build a catalog of all FITS, TFITS and HDR files under this directory
(including sub-directories). This catalog is the local calibration database which is
used during the association process.
Either this option or the options --cal-ldir and --cal-list must be used.

--cal-ldir
Specify a directory where ABbuilder creates symbolic links to calibration files
listed in the mcalib list (See option --cal-list).
Either this option or the option --cal-dir must be given. This option
must be used together with --cal-list option.

--cal-list
This option must be used in connection with option --cal-ldir, to specify
a file containing a catalog of calibration data (mcalib list).
If this option is used, ABbuilder will create symbolic links to the calibration
files listed in the mcalib list under the directory given by --cal-ldir.

--vcal-dir
Specify a repository of virtual products, stored as HDR files.
Virtual products are the expected products that will be generated
as result of the execution of an Association Block.
Virtual products can be defined in the OCA association rules (See OCA
rules documentation).
If this option is used, ABbuilder will load into the local calibration database
all virtual products already stored in the given directory, and
create new virtual products in the given directory.

--pro-dir
Specify the base directory under which pipeline products are created.
See also PROD_PATH in Association Block.
This option is mandatory.

--ab-dir
Specify a directory where Association Blocks are created.
This option is mandatory.

--oca-rul
Specify a file containing OCA rules, to be used for the data organization.
This option is mandatory.

--instrume
Specify the instrument name to be stored in the Association Block, as value
of INSTRUMENT parameter.
If this option is not specified, the instrument name will be read from the
FITS header (keyword INSTRUME).

--date
Specify the DFO date. In the current version this information is not used.

--mode { CALIB | SCIENCE | ALL(default} }
Specify the category (DPR.CATG) of the raw files to process. The possible
values are:
CALIB - only calibration ABs are created
SCIENCE - only science ABs are created
ALL - default value, all raw data under directory given by --raw-dir is processed
(organized).

--help print help and exit

--version print version and exit


CONFIGURATION FILES
ABbuilder configuration files are stored under directory "conf"
of ABbuilder installation tree. At present the following configuration files
are available:

- log4j.cf : configuration file for the logging system (log4j)

- ABbuilder.cfg : contains some specific configuration parameters, which
are not supported by the OCA rules, e.g.PACK_ADD

EXAMPLES
1) calibration data is available under /data/calib.

ABbuilder --raw-dir /data/raw
--cal-dir /data/calib
--pro-dir /data/product
--ab-dir /data/ab
--oca-rul /data/rul/test.rul
--instrume TEST
--vcal-dir /data/virtual

2) a catalog of calibration data is provided (mcalib_list). Symbolic
links are created under /data/calib/links. Instrument is
not specified, will be read from the fits header.
Virtual products (headers) will not be stored, since vcal-dir is not specified.

ABbuilder --raw-dir /data/raw
--cal-ldir /data/calib/links
--cal-list /data/calib/mcalib_list
--pro-dir /data/product
--ab-dir /data/ab
--oca-rul /data/rul/test.rul

KNOWN ISSUES
- Possible performance issues in connection with very large calibration databases
(several thousends of files)


FUTURE DEVELOPMENTS
- for large calibration data sets, the performance of association could be improved by
storing the relevant keywords in a relational database, to avoid
re-reading the fits files each time the ABbuilder is executed.

SEE ALSO
- OCA rules
- Association Block