Common DFOS tools:
Documentation

make printable	new:	see also:
	"between" and "tryBetween" operators	createAB documentation, ABbuilder documentation filterRaw
	RAW_MINNUM: supported by filterRaw	AB concepts

OCA configuration and syntax

OCA stands for Organisation, Classification, and Association. It is the classification and association framework used by various dfos tools (and also non-dfos tools like calSelector or the Data Transfer System).

Configuration. The association configuration (used by createAB, calChecker, and filterRaw) is somewhat more complex than for most other DFOS tools. All relevant configuration is kept under $DFO_CONFIG_DIR/OCA (where $DFO_CONFIG_DIR could be the classical one as defined in .dfosrc or a separate workspace like for calChecker or calSelector):

• OCA_macro.h: general macros for ABbuilder
• <instr>_macro.h: instrument specific macros ( <instr> is short for $DFO_INSTRUMENT)
• log4j.cf: a java logging configuration file

AND

• a set of config files in OCA syntax for createAB:
  • <instr>_classification.h : all classification configuration
  • <instr>_organisation.h: all organization configuration
  • <instr>_association.h: all association configuration
  • <instr>.h: all 'include' information
  • <instr>.RLS: the pre-compiled configuration file, ready to be used by ABbuilder; this is dynamically built by createAB on runtime

OCA syntax. The OCA syntax is SQL-like and consists of a set of

• classification rules ( if <conditions> then <classes>)
• organization rules (select ... from ... where <conditions> group by <grp rules> )
• association rules (defines actions with sets of associations to find)
  

OCA syntax is very flexible. For instance:

• fuzzy matches ("find mcalib which has same central wavelength within 10%") are possible
• full support for MASSOC and RASSOC is provided
• multiple actions per <raw_type> are supported
• creation of step2 (child) ABs is supported
• multi-valued templates are supported.

The same syntax is used for other applications like filterRaw or calChecker.

The main configuration files are <instr>_<oca>.h (where <oca> is short for organization, classification, association). The usage of macros is supported. These are defined in two macro files:

• OCA_macro.h is the general macro file and should not be modified unless in a coordinated way
• <instr>_macro.h contains the specific macros (like match keys).

The macro files need to be defined (# include) in the <instr>_<oca>.h files. All <instr>_<oca>.h files need to be defined (# include) in a <instr>.h file. All these files are then pre-compiled using gcc into a single <instr>.RLS file which is dynamic and is read by ABbuilder. All these files have a special syntax:

• '# include' marks macro files to be included by the precompiler (not a comment!)
• // marks a single line comment
• /*
  *
  */ marks multi-line comments

<instr>_classification.h:

consists of multiple entries (one per DO.CLASS) like the following:

if DPR.CATG=="CALIB" and DPR.TYPE=="FLAT,LAMP" then { RAW.TYPE = "FLAT"; DO.CLASS = "FIBER_FLAT"; PACK.DIR = "FLAT"; CATG = "CALIB"; }

<instr>_organisation.h:

consists of multiple entries (one per ACTION) like the following:

The match and grouping rules RAW_MATCH_RULE2,GRP_RULE_TPL_A can be macros which then need to be defined in <instr>_macro.h.

You can define multiple ACTIONs per RAW.TYPE in <instr>_organisation.h.

The where clause can contain complex expressions, like

where RAW.TYPE=="SCI_IMG" and TPL.NEXP>3

This may become useful when you want to trigger more than one action on a given RAW_TYPE, like in the following example:

select execute(ACTION_SCI_IMG) from inputFiles where RAW.TYPE=="SCI_IMG"
  group by IMA_MATCH_RULE,GRP_RULE_SINGLE;

select execute(ACTION_FLAT_NIGHT) from inputFiles where RAW.TYPE=="SCI_IMG" and TPL.NEXP>3
  group by FLAT_NIGHT_MATCH_RULE,GRP_RULE_TPL_A;

First, ACTION_SCI_IMG is triggered, applying the match rule IMA_MATCH_RULE and the grouping rule GRP_RULE_SINGLE on all input files of RAW.TYPE "SCI_IMG". Then, ACTION_FLAT_NIGHT is called on those SCI_IMG raw files which also satisfy the condition TPL.NEXP>3. These are then grouped by GRP_RULE_TPL_A.

RAW_MINNUM: there is an optional, non-OCA configuration possible in <instr>_organisation.h: a minimum number of input raw files for an AB to be built. This information is read by the filter tool filterRaw. There is one line per condition. The exact location within the OCA classification does not matter. Traditionally, it is found in the same place as the other configuration for <raw_type>:

This entry is interpreted by filterRaw to create ABs for <raw_type> only if at least <value> input raw files have been found, otherwise no AB is created.

<instr>_association.h:

consists of multiple entries (one per ACTION) like the following (the string ACTION_FLAT MASTER_BIAS DELTAT_RULE=2.5 is not part of OCA but is evaluated by createAB to establish the rule that for FLAT raw types a MASTER_BIAS is needed within a validity range of 2.5 days; the string FLAT needs to be identical to the RAW.TYPE in <instr>_classification.h and <instr>_organisation.h):

*: note to use rawFiles here instead of the usual inputFiles (see below for an explanation).
**: any content between "..." is interpreted as valid; you can also enter blanks etc.

MATCH_RULE1 and NEXT can be macros to be defined in <instr>_macro.h.

Note that lines starting with '// ACTION_' will be interpreted by tools like createAB. Any other comment is permitted.

Syntax rules

Possible operators in the condition section (if...then or where...) are:

Key arithmetics. With mathematical operators, you can do e.g. things like

if ... then {
  KEY = KEY + 1;
 }
and then use the modified KEY in a grouping rule.

Meta keywords. Another powerful mechanism is meta keywords. These are "artificial" keywords created and handled by OCA in the same way as fits header keywords. One example is MJD-OBS_MOD being created by ABbuilder and written into ABs and headers of virtual calibration files. Another way of using this mechanism is to define a meta key, e.g. by saying

if KEY >= 1 and KEY < 2 then M.KEY = 1;

and then use M.KEY within a grouping rule.

Data sources. Currently ABbuilder knows three data sources:

• inputFiles (defines the set of input files for grouping, that is all input files classified as CALIB or SCIENCE)
• rawFiles (defines the set of all input files, being a superset of inputFiles; these are required for finding RASSOC files)
• calibFiles (set of all mcalib and vcalib calibration products)

Step2 ABs. There are sometimes cases when a certain raw_type triggers a recipe, and the products of that recipe need to be processed in a second step, without having raw input files. The concept of step1-step2, or parent-child, ABs is supporting this scenario. In OCA, there is no specific syntax required to support this scenario.

Macros. A very useful feature is the availability of macros. Rather than having many matchkey definition lines, which repeat in many rule files, one can use one unique string and define it in the macro files once. There is the OCA_macro.h file which has the OCA system-wide definitions (e.g. grouping rules, time match rules), and the <instr>_macro.h collecting the match keys.