Converting Arecibo Interim Correlator Data Into SDFITS (For Aips++) or CLASS Internal Format

Two nearly-identical programs convert from raw Arecibo correlator data to either CLASS and Single-Dish FITS formats on-the fly, with each record getting dumped as it is written (or within a second, at any rate). The programs that do this conversion are /usr/S2local/bin/cltest and /usr/S2local/bin/sdtest, respectively. They behave very similarly, except where necessary because of the differences between SDFITS and CLASS format data.

Before running them you need to know the following:

• Where is your data located (i.e. at /share/olcor/... or /proj/a9999/...)
• What was the system temperature for both channel A & B ? (This is only used for the data headers).
• Whether the receiver was set to linear or circular polarization
• The total number of scans/cycle (i.e. its four if you took: ON, OFF, CAL ON, CAL OFF)

Once you feel confident you know these things, login to an Arecibo computer and then go to whatever directory you wish your CLASS files to reside in. Type cltest to run the CLASS or sdtest to run the SDFITS conversion program. The program will will ask a few questions and then start converting your data.

Note that you can also run these programs "on-the-fly" to continuously update a datafile with the online data via: cltest -c or sdtest -c

In that case, when you are finished you must quit out of the conversion program by typing ^C (possibly a few times).

You can specify the file names on the command line too:

cltest -c -A /share/olcor/corfile -F <output_file> -L 30 -R 28 -P c

where:

• "/share/olcor/corfile" is the name of the online datafile
• <output_file> is the name of the class file to which the data is being output. This file must have an extension (.something) for class to be able to see it (CLASS assumes any file with no . is .bur).
• "30" is the system temperature of the "A" channel (used in the data header)
• "28" is the system temperature of the "B" channel (used in the data header)
• "c" indicates circular polarization (the choices are "c" or "l")

(The other available options can be found via typing "cltest -h")

Notes

• I suggest that you put a round number in for the system temperatures. It varies somewhat with pointing, and a round number will help with final calibration later. This information should come from our internal calibration information, but so far does not.
• Make sure that your output data file is in a place with sufficient disk space, and not on the /share/olcor disk. The output data file will be somewhat larger than the raw data file. If you fill the /share/olcor disk while taking data, your data will be lost.
• CLASS file names must have a period in them, or CLASS won't see them, because CLASS adds .bur to any filename with no extension.
• If data is being taken faster than the conversion can run, it will get behind, but no particular ill effects are expected in the converted data. Datataking at extremely fast rates could present a problem if the original raw data can't be written to disk as fast as they are taken.

CLASS

Class Data

CLASS format has a very limited space for describing observations. Cltest encodes the observation in the telescope name. So the telescope looks like: AO-BPT: B is Board, a number from 1 to 4. P is polarization, and can be X, Y (linear), L, R (circular), I, Q, U, V (stokes). Note that each board can be set up to take 1, 2, or 4 polarizations. T is type, and can be O, F, C, D, meaning On-source, Off-source, Cal-on, Cal-off (respectively). Thus, AO-1LF means Arecibo, correlator board 1, left circular polarization, Off source, and if you use 4 boards with two polarizations per board, you will get

AO-1XO
AO-1YO
AO-2XO
AO-2YO
AO-3XO
AO-3YO
AO-4XO
AO-4YO

for each dump.

The "scan numbers" for class data are the last six digits of the AO scan number, that is, the least significant digit of the day-of-year, and the 5-digit number for the scan id.

The object names are taken from the header, but the AO format allows longer names than does class, so they can be truncated.

CLASS Example

A data file with only two observations, each of a different galaxy (UGC 3489 and UGC 4131).
Each cycle consisted of ON, OFF, CAL ON, CAL OFF

/home/nolan/src2/fi/cltest -c
Enter Input (AO) filename: /proj/a9999/corfile.01jan00.a9999.1
Enter Output File: 01Jan00a.cls

Channel A system temp? :30

Channel B system temp? :28

Linear or Circular inputs? :C

Total number of scans in a cycle:4

Scan 103700038, UTC=2001-02-06T08:45:01, AST=2001-02-06T04:45:01
Object: U5440
Scan 103700038: U5440-1.  RA 10:05:36.20 Dec +04:16:23.0
Scan 103700039: U5440-2.  Offset 5400.000000 (5400.215304)
Scan 103700040: U5440-3.  Offset 5400.000000 (5400.215304)
Scan 103700041: U5440-4.  Offset 5400.000000 (5400.215304)
Scan 103700090, UTC=2001-02-06T10:01:49, AST=2001-02-06T06:01:49
Object: U5790
Scan 103700090: U5790-1.  RA 10:39:04.70 Dec +04:38:51.0
Scan 103700091: U5790-2.  Offset 5400.000000 (5400.264480)
Scan 103700092: U5790-3.  Offset 5400.000000 (5400.264480)
Scan 103700093: U5790-4.  Offset 5400.000000 (5400.264480)
Scan 103700094, UTC=2001-02-06T10:13:53, AST=2001-02-06T06:13:53
Object: U5790
Scan 103700094: U5790-1.  RA 10:39:04.70 Dec +04:38:51.0
.
.
.

CLASS_WRITER: scan 103700038 written.
CLASS_WRITER: scan 103700039 written.

Commands are (all caps required):
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <scan>  to continue for num or until reaching scan,
  SKIP  <num> to skip to scan <num>,
  ASKFREQ to prompt for frequencies each time,
  ASKMAP to prompt for map pos.
  ASK to ask basic questions again.(cr will keep U3489): U3489
Scan 930900009: U3489-1.  RA 11:12:24.40 Dec +31:24:41.0
Scan 930900010: U3489-2.  Offset 5400.000000 (5400.165333)
Scan 930900011: U3489-3.  Offset 5400.000000 (5399.659220)
Scan 930900012: U3489-4.  Offset 5400.000000 (5399.719945)
Object:U4131
Number of scans in an image:4
Number of dumps in a scan:300

Commands are (all caps required):
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <scan>  to continue for num or until reaching scan,
  SKIP  <num> to skip to scan <num>,
  ASKFREQ to prompt for frequencies each time,
  ASKMAP to prompt for map pos.
  ASK to ask basic questions again.(cr will keep U4131): U4131
Scan 930900009: U4131-1.  RA 07:59:11.43 Dec +31:48:23.6
Scan 930900010: U4131-2.  Offset 5400.000000 (5400.165333)
Scan 930900011: U4131-3.  Offset 5400.000000 (5399.659220)
Scan 930900012: U4131-4.  Offset 5400.000000 (5399.719945)
9632 records translated

SDFITS

SDFITS and Aips++

Aips++ can read single-dish FITS data. These data are stored as FITS binary tables with a single row for each spectrum: A single dump usually generates 4 or 8 spectra (four subcorrelators with either one or two polarizations per board). If data are taken with different numbers of channels per spectrum, the file size will be as if they all had the largest number of channels. If the number of channels increases during the conversion, then sdtest will resize the entire file, which may take a while.

SDFITS Example #1

A data file with only two observations, each of a different galaxy (UGC 3489 and UGC 4431). Each cycle consisted of ON, OFF, CAL ON, CAL OFF

sdtest
Enter Input (AO) filename: /proj/a9999/corfile.01jan00.a9999.1
Enter Output File: 01Jan00a.fits

Channel A system temp? :35

Channel B system temp? :35

Linear or Circular inputs? :C
Total number of scans in a cycle:4
Scan 930900005, UTC=2000-01-05T12:06:42, AST=2000-01-05T08:06:42
Object: U3489
nter Command (UPPER CASE) or Object Name: (cr will keep U3489
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <num|scan>  to continue for num or until reaching scan,
  NEWCYCLE <num> to change the number of scans in a cycle to num,
  SKIP  [num | scan] to skip this scan, or num scans, or until scan
  ASKFREQ to prompt for frequencies each time,
  ASK to ask basic questions again.): U3489
Scan 930900009: U3489-1.  RA 11:12:24.40 Dec +31:24:41.0
Scan 930900010: U3489-2.  Offset 5400.000000 (5400.165333)
Scan 930900011: U3489-3.  Offset 5400.000000 (5399.659220)
Scan 930900012: U3489-4.  Offset 5400.000000 (5399.719945)
Object:U3489
Enter Command (UPPER CASE) or Object Name: (cr will keep U3489
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <num|scan>  to continue for num or until reaching scan,
  NEWCYCLE <num> to change the number of scans in a cycle to num,
  SKIP  [num | scan] to skip this scan, or num scans, or until scan 
  ASKFREQ to prompt for frequencies each time,
  ASK to ask basic questions again.): U4431
Scan 930900009: U4131-1.  RA 07:59:11.43 Dec +31:48:23.6
Scan 930900010: U4131-2.  Offset 5400.000000 (5400.165333)
Scan 930900011: U4131-3.  Offset 5400.000000 (5399.659220)
Scan 930900012: U4131-4.  Offset 5400.000000 (5399.719945)
9632 records translated

SDFITS EXAMPLE #2

A data file with three observations, two of one galaxy (U5361) and one of another (U5592). Each cycle consisted of ON, OFF, CAL ON, CAL OFF

sdtest
Enter Input (AO) filename: /proj/a9999/corfile.01jan00.a9999.4 
Enter Output File: 01Jan00d.fits   

Channel A system temp? :9.2

Channel B system temp? :9.2

Linear or Circular inputs? :L
Total number of scans in a cycle:4
Scan 930900061, UTC=2000-01-05T15:30:13, AST=1999-11-05T11:30:13
Object: U5361
Enter Command (UPPER CASE) or Object Name: U5361
Enter Command (UPPER CASE) or Object Name: (cr will keep U5361
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <num|scan>  to continue for num or until reaching scan,
  NEWCYCLE <num> to change the number of scans in a cycle to num,
  SKIP  [num | scan] to skip this scan, or num scans, or until scan 
  ASKFREQ to prompt for frequencies each time,
  ASK to ask basic questions again.): U5361
Scan 930900061: U5361-1.  RA 09:55:03.25 Dec +25:12:10.6
Scan 930900062: U5361-2.  Offset 8955.000000 (8948.313686)
Scan 930900063: U5361-3.  Offset 8955.000000 (8948.299047)
Scan 930900064: U5361-4.  Offset 8955.000000 (8948.302798)
Scan 930900065, UTC=2000-01-05T15:42:43, AST=1999-11-05T11:42:43
Object: U5361
Enter Command (UPPER CASE) or Object Name: (cr will keep U5361
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <num|scan>  to continue for num or until reaching scan,
  NEWCYCLE <num> to change the number of scans in a cycle to num,
  SKIP  [num | scan] to skip this scan, or num scans, or until scan 
  ASKFREQ to prompt for frequencies each time,
  ASK to ask basic questions again.): 
Scan 930900065: U5361-1.  RA 09:55:03.25 Dec +25:12:10.2
Scan 930900066: U5361-2.  Offset 8955.000000 (8948.189632)
Scan 930900067: U5361-3.  Offset 5400.000000 (5399.836752)
Scan 930900068: U5361-4.  Offset 8955.000000 (8948.473693)
Scan 930900069, UTC=2000-01-05T15:57:01, AST=1999-11-05T11:57:01
Object: U5361
Enter Command (UPPER CASE) or Object Name: (cr will keep U5361
  QUIT to stop converting,
  DRIVE to finish entire file with same object name,
  UNTIL <num|scan>  to continue for num or until reaching scan,
  NEWCYCLE <num> to change the number of scans in a cycle to num,
  SKIP  [num | scan] to skip this scan, or num scans, or until scan 
  ASKFREQ to prompt for frequencies each time,
  ASK to ask basic questions again.): U5592
Scan 930900069: U5592-1.  RA 10:21:31.63 Dec +22:32:51.0
Scan 930900070: U5592-2.  Offset 5400.000000 (5401.432019)
Scan 930900071: U5592-3.  Offset 5400.000000 (5400.222629)
Scan 930900072: U5592-4.  Offset 5400.000000 (5400.127029)
14448 records translated

Last modified: Tue May 18 17:49:17 AST 2004

Contact: Mike Nolan