 |
The FITS archive: How to archive a new dataset |
|
| Fist Archive | Structure of
HFA | Keywords | FAQ
| Introduction | Archived
data | Software | Pipeline |
This page explains the different steps for archiving a new dataset.
Introduction
Archiving a dataset should be
implemented in the "data-flow" between
the acquisition at the telescope and the reduction, in other words
the data are archived as the are produced by the telescope and the
reduction is performed on the archived data using the pipeline
procedure.
Archiving data
consists in adding in the FITS header all the keywords necessary
for a full description of the data.
This principle is often going against the usual practice, we will
explain
now why you should nevertheless adopt this line and maybe change your
habits.
The goal of an astronomer when reducing data is to get his results as
fast as
possible. In practice, very often the procedures fail and the work has
to be re-done in an improved manner. There are two basic alternative
ways to make the "re-do" easy:
- Write a procedure for each individual frame to reduce.
- Write the same procedure for all the frames and use the keywords
to get the parameters specific to a given file.
The second solution is of course
better because only one file will need
to be edited to change the reduction method. The second method needs
the
relevant keywords to have been created: this is exactly what
"archiving"
will do. Anyway, whatever method is adopted the information will have
to be given once...
If necessary the two methods can be combined by using the header
information when available and otherwise adding specific information
in the command files. This will make the command files lighter, but I
still think it is not the best solution.
The overwhelming advantages of archiving the data before the
data-reduction appear when the data are to be re-used, sometimes
years after, in order to measure other parameters (for example
do spectrophotometry of data which were first used for kinematics)
or to apply new reduction methods (eg. improved description of the
LOSVD).
Even if the command files were carefully kept, it will not be
straightforward
to re-use them: information like object identification or position will
have to be re-determined, and hence possibly incompatible with the
initial reduction.
My advice: Load as much information as possible early in the
data-reduction
Pre-requisites
- PLEINPOT is installed on the current host. (PLEINPOT commands
are used to set some fits keywords).
A basic knowledge of PLEINPOT is a clear advantage: The HTML documentation
is available at http://leda.univ-lyon1.fr/pleinpot/pleinpot.html.
- The files are available in the FITS format. These files may be
raw or processed observations, in this document they will anyway be
referenced as "raw observations" opposed to "mean calibrations". Mean
calibrations are the processed files necessary for making the
reduction, eg. normalized flat-fields...
Steps
- If you have enough space on disk, copy the whole dataset to a
new
directory (for security).
The name of the directory containing the fits file should be the
identifier
of the dataset. This is a 9 digit character string, obeying a (somewhat
flexible) convention.
For example,
L99021HP2 is decoded as:
L -> archived in Lyon
99 -> millesime of the year of observations
02 -> month of observations
HP -> Haute-Provence Observatory
2 -> 1.2 m telescope (1 is for the 1.9 m)
- Create the files: AAAReadMe and AAAHistory
These files must be filled with (free text format)
AAAReadMe: Detailled description of the dataset.
AAAHistory: Log of the different steps for archiving in HFA.
- Uncompress the FITS files (if necessary)
- Allow write access to the FITS files (if necessary;
chmod +w *.fits)
- Rename all the FITS files into filenames formatted as:
nnnnn.fits
Where nnnnn is a running number.
The raw-observation files must be in the range 00001 to 98999.
The order is not really important, but it is recommended to use
consecutive numbers.
(to create the numbers you can use something like:
ls p*.fits | awk '{printf "mv %+13s %05d.fits\n", $0, FNR}'
)
The numbers between 99001 and 99999 are reserved for naming the
"computed" files, as eg. average flat-field or mean calibration file.
The program archilist can be used to display a listing of
the dataset
- Create a file SETUP.fits which will be used to set the default
value of some keywords common to the whole dataset execute the PLEINPOT
command:
archiset
The program archiset asks the default value for some keywords, try to
give as much as possible. If you do not set some values, the other
archiving programs will ask them...
SETUP.fits is used by the archiving programs: archispe
and archiver
You may update SETUP.fits at any time using archiset or possibly using
any keyword editing program, for example futil. You can
initialize any keyword (associated with a PLEINPOT descriptor), even
those not initialized by archiset.
Note: When you run archiset for the first time, the program will
probably not be able to suggest default value for the bias and
flatfield files. You may omit to initialize this value directly, and go
directly to the program archispe. After archispe
you may re-run archiver.
If the dataset consists in spectra not calibrated in wavelength, archiset
asks the "identifier" of the relation of calibration (keyword H_WCALIB).
It can be a separate FITS frame, hence identified as eg. L1999101HP1/990005,
or an extension attached to any FITS frame, for example: L1999101HP1/990001[WCA]
(extension named WCA in L1999101HP1/990001.
- Execute
archispe to load the keywords in the special
files.
- Execute
archiver to load the keywords in the
"regular" files. You may also use any keyword editing program, for
example futil, or archikey to edit a
keyword in a series of files in the dataset. For example, to set the
value of H_AXIS1 for all the regular files of the dataset
L1996KINN, you may use: ls L1996KINN/0*.fits |
xargs -i futil {} ke H_AXIS1 2
archiver initialize some kewords mandatory for the
archiving. User is prompted to provide (or confirm) the value for these
keywords. The default values for the un-initialized keywords are taken
from SETUP.fits (see above).
- To define the position of the object in the frame, you can use:
archiview.
This program visualize each frame in turn, and you can use the usual
keyboard commands (as in grlise) to set the center
(actually ctrl-C).
- Spectroscopy: To define the pointers to the arc files
taken
before and after each science observation, use
archiarc
- To move the dataset into the HFA, contact the HFA administrator,
at present Ph. Prugniel.
drawbacks
For the moment the calibration files
(eg. flatfield...) are referred with
their identification in the HyperLeda FITS archive: as eg. fa:L95011HP2/99004.
For this reason the PLEINPOT programs
for processing these files will not work until the dataset is actually
moved in HFA.
There is not program (yet) for writing
the pointers to wavelength calibration
and flux calibration relations (this must e done by hand).
Documentation of PLEINPOT
programs used for archiving
- archilist
- archiset
- archiver
- archispe
- archikey
In some cases, you will need to know the list of keywords used by
HyperLeda. The
list is not always up-to-date, since we
are still
developing the system of keyword. You can also see the list of keywords
already defined in your file using the program futil
(with command
KL). Giving the value "undefined", ie. entering "--"
when you are
prompted for the keyword's value, is equivalent of killing the keyword.
- archiview
-