EsoRex
Introduction
One of the features provided by the CPL is the ability to create data-reduction algorithms that run as plugins (dynamic libraries). These are called recipes and are one of the main aspects of the CPL data-reduction development environment.As these recipes are dynamic libraries, it is not possible to run them directly from the command line. However, ESO provides several tools to do this, thus saving recipe developers the need to write such an application themselves. One of these is GASGANO (a GUI-based tool) and the other is EsoRex (which runs from the command line) and is described here.
List of features
In addition to running recipes, EsoRex provides the following features:
- List recipes available in the specified directories
- Multiple recipe-path specification
- Recursive recipe-path searching
- Log file level-control and management
- Ability to create links to recipe products
- Recipe product naming control
- Generation of recipe man pages
- Permission control for recipe products
- Command line and configuration file parameters
- Can automatically generate configuration files
- Able to measure and report recipe execution times
- Optionally, print a report of memory-leaks in the recipe
Obtaining and Installing EsoRex
EsoRex may be obtained from the download page. Installation instructions are provided in the README file that is included in the EsoRex distrubution.Using EsoRex
EsoRex takes a number of command parameters in order to determine the required operation. The "usage" for the tool is as follows.
esorex [esorex-options] [recipe [recipe-options] [sof]]
Notice that after the command itself, all the command-line arguments are grouped according to their function. First come the EsoRex options (a full list of these can be determined by typing esorex --help. Then, optionally, the recipe may be specified. Following this are any command line options for the recipe itself, and a sof (set of frames) file. (Note that iit is possible to list several sof files, in which case EsoRex will treat them as if they were appended in a single file.)
The use of these options is described in the following sections.
Getting help on EsoRex
% esorex --help
This lists all the command-line options for the EsoRex application itself.
Using an EsoRex configuration file
EsoRex has quite a lot of input parameters, so it is often much more convenient to use a configuration file. It is even possible for EsoRex to create a configuration file for you (see below).The configuration file contains the EsoRex options, less the -- switch, but prefixed with esorex.caller.. Blank lines are ignored and lines beginning with # are treated as comments.
Here is an example configuration file.
# Example EsoRex configuration file # esorex.caller.recipe-dir=/home/username/EsoRex/Plugins esorex.caller.log-dir=. esorex.caller.log-file=esorex.log esorex.caller.output-dir=. esorex.caller.output-prefix=out_
Then, to use this configuration file, use the --config command switch. For example:
% esorex --config=myesorex.rc
Note that the configuration file created here, contains the essential configuration options for normal EsoRex use. This configuration file will be assumed in subsequent examples.
Note also, that EsoRex will also search for a configuration file called esorex.rc in the $HOME/.esorex directory. Any values in this file will be used by default, however any value specified in an explicit configuration file (using the --config option), or directly on the command line will overwrite these values.
Using EsoRex to generate a configuration file
Rather than coding a configuration file by hand, it is possible for EsoRex to create its own configuration file, using sensible defaults, and over-riding them with any additional ones that you provide on the command line. To do this, use:% esorex --create-config
This will create a file .esorex/esorex.rc in your home directory. If the directory .esorex doesn't exist, it will be created for you. EsoRex will fill in most of the parameters in the configuration file for itself, but if there are any for which sensible defaults can not be determined, then EsoRex will put them in blank but commented out, and write a warning message to the screen.
If an configuration file already exists, then the old one will be copied to .esorex/esorex.rc.bak. This feature can be disabled by using the --cleanup option.
Displaying EsoRex settings
% esorex --config=myesorex.rc --params
This will show the caller (i.e. EsoRex) parameters. However, unlike the --help which shows the parameters as command line options, with their description, --params will display the variable and its current value, whether that is from the command line or the configuration file.
Listing all available recipes
% esorex --config=myesorex.rc --recipes
This will search for recipes in the recipe directory (which is specified in either the configuration file, or using the --recipe-dir command line switch). The directory search is _recursive_. That is, EsoRex will look not only in the recipe directory itself, but in all sub-directories.
*Warning:* If non-recipe dynamic libraries exist in the recipe directory tree, they may (depending on contents) cause EsoRex to crash. This is due to a limitation in the ltdl library, and is a known problem. There is no immediate solution, other than to ensure that the recipe directory tree does not include any non-identifiable dynamic libraries, other than genuine recipes.
Getting help on a specific recipe
% esorex --config=myesorex.rc --help myrecipe
This will display information about the recipe myrecipe, including a list of the recipes options.
Displaying recipe parameters
% esorex --config=myesorex.rc --params myrecipe
This will show the caller (i.e. EsoRex) parameters, followed by the recipe parameters. It will show the value that would be used were the recipe actually executed. These values will be the latest specified from either the configuration file or the command line. If the parameter was not specified in either of these locations, the default will be displayed.
Setting recipe parameters
To specify parameters for a recipe, put the options after the recipe name. For example:% esorex --config=myesorex.rc myrecipe --myoption
Of course, one can use the --params option to ensure that the values are correctly read. For example:
% esorex --config=myesorex.rc --params recipeone --op=subtract
Using a recipe configuration file
Like EsoRex itself, it is possible to also specify a configuration file for the recipe options. Note that this is still an EsoRex command, and so should go before the recipe name (only actual recipe options should go after the recipe name). Thus:% esorex --config=myesorex.rc --params --recipe-config=myrecipe.rc myrecipe
would display the parameters that would be used, given the recipe configuration file myrecipe.rc.
The recipe parameters must be specified in the configuration file as their full names. As an example, here is what a configuration file for the mythological "ZIMOS" instrument might look like:
# # EsoRex configuration file for ZIMOS 'zmspcaldisp' CPL recipe. # zimos.Parameters.bias.removing.method=Zmaster zimos.Parameters.extraction.fuzz=5 zimos.Parameters.lamp.frames.validate=true zimos.Parameters.ids.refine=false zimos.Parameters.extraction.window=-1 zimos.Parameters.flat.apply=false zimos.Parameters.badpixel.clean=false zimos.Parameters.cosmics.clean=false zimos.Parameters.BLA=Input zimos.Parameters.slit.model=false zimos.Parameters.slit.order=0
Note that like the esorex.rc configuration file, EsoRex will also search $HOME/.esorex for any recipe configuration files and use them as a default. It is also possible to use the --create-config option when a recipe name is also specified, and EsoRex will create a configuration file for that recipe. This can then be altered with a text editor.
Actually running a recipe
Without any --params or --help options, EsoRex will attempt to execute any specified recipe.% esorex --config=myesorex.rc --recipe-config=myrecipe.rc myrecipe data.sof
See the section on Sets of frames (below) for more details on how to actually specify the data.
Note, that EsoRex will create all temporary and intermediate files
in the current working directory. Thus, you cannot run mutliple instances
of EsoRex to execute several recipes in parallel in the same directory -
internal data would be overwritten in a non-deterministic way.
As a solution move to different directories before starting EsoRex in
parallel to an already running EsoRex module.
The command-line-option --output-dir only stores the final product in that directory, all temporary files are still put in the current working directory.
Sets of frames
A sof file contains a list of the input data. This data is specified in an sof file (which is just a text file), where each input file is specified with its associated classification and category. The format of each line in the sof file is as follows:full-path-to-file classification
An example file, for the mythological "ZIMOS" instrument, might look like this:
/home/user/data/mos/ZIMOS.2003-12-26T01:05:06.233.fits MOS_SCIENCE /home/user/data/mos/ZIMOS.2003-12-26T01:26:00.251.fits MOS_SCIENCE /home/user/data/mos/ZIMOS.2003-12-26T01:47:04.050.fits MOS_SCIENCE /home/user/data/cal/master_bias4.fits MASTER_BIAS /home/user/data/cal/grs_LR_red.3.tfits GRISM_TABLE /home/user/gasgano/extract_table2.fits EXTRACT_TABLE /home/user/data/cal/badpixel.3.tfits CCD_TABLE
For an concrete example for a specific instrument, check the documentation for that instrument.
Optionally, a third column may be provided. Permitted values are either RAW or CALIB. This is for when a recipe does not identify the type of input file, but as all ESO recipes are required to do so, this column is typically not needed.
Hints for using EsoRex
File permissions
(EsoRex v1.x) When a recipe is used with the --suppress-prefix option, and the --output-dir is set to the current working directory, then the first execution of a recipe will work correctly, but subsequent executions may fail. This is due to output products being given "read-only" permission (to avoid the potential inadvertant loss of products). The recipe itself is unable to modify the permissions, and thus it fails when attempting to create the file. The solution (other than using a different output directory or prefixes) is to change the permission of these output files or delete them prior to any subsequent execution of that recipe.(EsoRex v2+) This problem is less likely to occur in EsoRex v2+, due to the replacement of the --output-overwrite option with the --output-readonly (which is disabled by default). However, a determined user can still reach this situation, in which case the non-readable products must have their permissions changed, as described above.