Format conversion

If you have a program to convert data to MIDAS input format, fine. Use it and go on to the next section.

If you do not, first see if there is any existing program that reads your data format. (Very likely there is, unless the instrument is new.) All you need to do is cannibalize the part of the existing program that reads the data, and add a short piece of code to re-format the data first as a flat ASCII file, one measurement per line; and then convert this to *.tbl format. Sound familiar? It's just like the process of creating a program-star table. Once again, a dummy *.fmt file is provided to help you.

That's usually the easiest way to go. You may decide to write a short MIDAS script to apply your format-conversion program and the edited *.fmt file to each night's data. That can be put into a procedure, given a name, and invoked as needed. This has already been done for data in the formats produced by the photometers at the 1-meter ESO and the Danish telescopes on La Silla; the command CONVERT/PHOT will convert their data files to the proper MIDAS tables.

If you are very lucky, your data may already be in the form of a flat ASCII file of consistent format; then all you need to do is edit the *.fmt file and apply it to generate the *.tbl files MIDAS needs. More likely, you have most of your data in this form, but interspersed with comments and/or records of other types. In this case, it may be possible to write a simple pre-processor (or use some UNIX utility, like awk) to strip out the data as a fixed-format ASCII file.

If there are just a few distinct record formats in your data files, it may be most convenient to strip out each type separately; convert each one first to a homogeneous ASCII file, and then to a MIDAS *.tbl file; and then use MERGE/TABLE to combine the separate tables into the final file for input to the reduction program. If each line of each file is correctly time-tagged, a simple sort on the MJD_OBS column will then put everything back together in the right order.

In any case, you should automate this procedure as much as possible. That means writing a short MIDAS or shell script. Whatever you do, don't resort to copying things out by hand! At worst, you might need to insert a little timing information by hand to label comments in the original data.

You should be careful to make sure enough information is in the final *.tbl file to tell the reduction program everything it needs to know. For example, if you measure dark current, and have more than one detector channel, the table must show which detector's dark current is being measured. That is usually indicated by the filter position; so you will probably need to have a non-blank filter position for ``dark'' measurements. Note, however, that you do not need to distinguish among standard, extinction, program, and variable stars in data files, as this information is either supplied when you read in the star files, or determined dynamically during the reductions.

The minimum of essential columns in a data file are object identification, band identification, signal, time, and integration time. The section ``Required observational data'' in the Appendix describes these columns. For standard band names in commonly-used systems, see the Appendix (particularly Table 2 in the section ``Standard values'' under ``Star tables'', and the description of the BAND column in the ``Passbands'' subsection under Section 5). Usually, you will also need the STARSKY column, to distinguish between star and sky measurements. But many other data can be useful: temperatures, relative humidity, various instrumental settings, and error estimates. The section ``Additional information'' in the Appendix describes these in detail.

Note that temperature and humidity data can be interpolated, if they are not routinely recorded with each observation. If your instrument table shows that neutral-density filters may be used, you need to include this information as part of the band designation in the data files. Conversely, if your data indicate that ND filters have been used, their presence and nominal attenuation factors should be included in the instrument table.

It would be nice to have a universal data-conversion utility. Unfortunately, the amazing variety of formats produced by dozens of home-grown data-logging systems seems to prevent it. You know your data better than anyone else; so you can put them in standard form better than anyone else.