Problem:

---

Background:

The steps in going from acf to spectrum in pulsar mode using the wapps  are:

1. accumulate the acf in the correlator chip then read it out. For pulsars this is typically every 64 to 1024 usecs.
2. Combine multiple chips if 9 level  or interleaved sampling is used.
• this is done in one of the wapp demux routines.
3. correct for any bias/offsets of the 3/9 level correlator chips (e.g. -1,0,1 is recorded as 0,1,2 in the 3 level chips).
4. If pulsar folding mode then accumulate the acf in the correct phase bin.
5. Compute the digitizerThreshold/rmsVoltage using the zero lag.
6. 3 or 9 level correct the acf
7. fourier transform the acf to get the spectrum (after symmeterizing or zero extending the acf).

Some notes:

• If you are storing acf's, then the data is written to disc after step 4. If spectra are stored then they are written to disc after step 7.
• For 9 level sampling, four 3 level correlator chips are combined to measure 9 levels. The input signal is split into two ternary digits HL (base 3). The 4 chips do the La*Lb, Ha*Hb, Ha*Lb, Hb*Lb where a is the immediate data and b is the delayed data that goes into each lag multiply. The multiplies are scaled by : 1*1, 3*3, 3*1, 1*3 corresponding to the ternary digit they belong to. This ends up scaling the acf by : 1+9+3+3=16.

level correction history:

• The interim correlator 3 level  correction comes from kulkarni and heiles (astronomical journal)
• The interim correlator 9 level correction came from murray lewis. Only the 0 lag (total power) correction is done. The other lags are not modified.
• The wapp original  3 level correction came from andy dowd and the 9 level correction was that of murray lewis.
• Around 2002 fred schwab (nrao) published an algorithm for multi level correction. In the paper he included routines for 3 and 9 level correction. The wapps eventually switched to this code.
• Idl routines (written by me) are able to read pulsar wapp acf data and do the kulkarni/heiles 3 level correction and  murray lewis's 9 level correction (actually the 9 level idl code was written to find this problem and is probably not in the production library yet..).

---

Plots of the problem (and solution).

• No 9 level spectral correction (i did do the 9 level total power correction)
• Use foldfft with the 9 level bug in it
• Use idl and murray lewis's 9 level correction to process the data.
• Use foldfft with the fixed code for 9 level.
• Use 3 level spectra with no correction (i did do the 3 level total power correction).
• Use 3 level spectra with foldfft
• Use idl and kulkarni/heiles 3 level correction.

• Fig 1 Top: This shows the absorption spectrum for the 7 methods. The horizontal scale is in units of pulsar continuum. An offset has been added for plotting.  The colored lines are:
• black : no 9 level spectral correction.
• red : foldfft with the 9 level bug.
• green: foldfft with the 9 level fix.
• darkBlue:  9 level correction of idl and murray lewis.
• light blue:  3 level no spectral correction.
• purple:  3 level correction of kulkarni/heiles.
You can see the large error that the foldfft 9 level bug is making. This pulsar is not supposed to have any absorption.
• Fig 1 bottom: Spectra from the first phase bin. The blue (9lvl_idl_bml) and black (9lvl_nocorrection) lie on top of each other. The 3 level and 9 level spectra can have different power levels since they went through different filters/amplifiers. The edges of each spectra should hit .3 (relative to the median value). This is the expected alias from the digital filters. The red plot (foldfft with bug) has spectral edges that fall below 0.
• Fig 2 top: This shows the difference between 3 level  spectra: foldfft (fred schwab)  -  (kulkarni/heiles). The difference is around 1e-7 which is close to the resolution of the floating point numbers. So these two routines are giving the same results.
• Fig 2 middle: This shows 9 level spectra: foldfftCorrected - murrayLewis. There is an offset of about 3 % Tsys and a spectral variation of .0002 Tsys. If things were perfect, this difference should be the same range as the 3 level differences.
• Fig 2 bottom:  This shows the 9 level absorption spectrum difference (it's units are pulsar continuum rather than correlator counts). The bump that was seen in the difference is not evident.  Since we are taking (onPulse-offPulse)/median(onpulse-offpulse) we are removing any offsets and linear scaling terms. So to within and additive and multiplicative constant the values are pretty close (although it would be nice to know why they aren't exactly the same).

---

What is causing the problem:

    One of these routines in the level correction: thresh() (in /share/wappsrc/lib/vanvleck.c) is used to compute the digitizer threshold/rmsVoltage. It expects the 0 lag to be input without the factor of 16 removed. The online folding routine had removed this factor (see /share/wappsrc/wapp/fold.c line 109:
 

The thresh() thought that there was 16 times less power than actually present. This caused an incorrect threshold to be used in the later 9 level correction which made a big difference in the corrected spectrum.

The routine thresh() is called from: vanvleck_z(), pow9lev(), attndb9lev() all in /share/wappsrc/lib/vanvleck.c . These routines are then called from a bunch of other places.
 

---

How to fix the problem(s):

• For the offline program /share/wappsrc/utility/foldfft():
I've created a version of foldfft() that works for 9 level sampling. All you need to do is multiply the zero lag passed into vanvleck_z() and pow9lev() by 16 (but only do this for 9 level sampling).
• If you did 9 level online folding and you stored spectra:
Your spectra have this problem. To fix it you would have to go back to the acf domain, undo the "wrong" 9 level correction, and then redo the correct one (i'm not sure how easy this is..).
• To fix online folding and recording of spectra code you need to :
In /share/wappsrc/wapp/fold.c routine fold_output.c line 214 :
div=1./(double)o->count[b]  change to:

 scl=(((c->head->lagformat & ~FOLD32) == FLOATSPEC ) &&
          (c->mode->level == NINELEV)) ? 16. : 1.;
  div= scl/(double)o->count[b];

I think this should fix the problem (although i'd spend some time checking it..).

---

What is not broken:

 home_~phil