ATNF VLBI Wiki

DiFX can support a wide range of spectral resolutions. Achieving the highest resolutions can be tricky for a number of reasons, many with adverse performance side effects:

• Typically transform sizes get large
• Lots of memory is needed / cache memory becomes less effective
• Bandwidth.time product becomes small (as low as 1), meaning less reduction of data
  • Output data could be larger than input data!
• Individual FFTs become long (in time), butting up against
• Due to timekeeping mechanisms in mpifxcorr, an FFT cannot span more than 2^31-1 nanoseconds
  • A hair over 2 seconds is the longest allowed transform
  • 0.5 Hz is thus the finest frequency resolution allowed

Some general tips to consider:

• Start with small input bandwidths (e.g., plan the experiment properly!)
• Use factors of 10 in FFT spectral resolution, rather than only factors of 2
  • E.g., set spectral resolution (specRes and fftSpecRes) to 0.001 (MHz) rather than 0.0009765625 (MHz)
• Make sure integration time (tInt) multiplied by specRes is not less than 0.0000001
• Make use of zoom band to only save part of the spectrum, if consistent with the science
• To minimize memory usage, numBufferedFFTs should probably be set to 1 (may greatly hurt performance)

Some example .v2d file content leading to 0.5 Hz resolution:

# Might need to increase the largest read size to accomodate long FFTs
maxReadSize = 40000000
 
# Correlate a 1 kHz portion of the input band centered on 4101.3 MHz
ANTENNA FD { toneSelection=none filelist=fd.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA KP { toneSelection=none filelist=kp.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA NL { toneSelection=none filelist=nl.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA OV { toneSelection=none filelist=ov.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
ANTENNA PT { toneSelection=none filelist=pt.filelist addZoomFreq=freq@4101.2995/bw@0.001 }
 
SETUP default
{
  tInt = 2 # this is the shortest integration allowed for half Hz
  fftSpecRes = 0.0000005
  specRes    = 0.0000005
  doPolar = False
  numBufferedFFTs = 1 # When 1 FFT per integration, this is the only legal value
  maxNSBetweenACAvg = 2000000000 # change to be integration time; avoid warnings
}

Note that the above 5 station correlation of a pair of 4 MHz channels requires 12 GB of RAM for each core process.