Intro:

• IF1: Is the first IF which is upstairs on the platform.
•  The mixers use the synthesizer in the dome to mix from the 1st to the 2nd if
• IF2: this is the 2nd IF. It is downstairs in the receiver room.
• . The mixers to go from the 1st IF to up to 8 2nd separate IF'S
  

The problem

a2705 : cband galactic mapping:

• 10sep12 - when  some of the data was bad. There were 46 spectra of 180 seconds each
  
• 11sep12 - the data looked ok. There were 44 strips of 180 seconds each.
  

• Page 1: average spectra
• top: 10sep12
• bottom: 11sep12
• The colors are different mock boxes.
• Page 2: blowup spectra which used the synths that had the problem
• black: 10sep12
• red: 11sep12
• You can see that the rfi for B4, and B6 are missing on the10sep12 spectra.
• Looking at the x111 data, i could tell that the B0, and B2 data were also bad.
• Page3:blowup spectra which used the synths that were not affected
• black: 10sep12
• red: 11sep12
• you can see the rfi shows up in the same place on both days (the spike i the middle is dc).

x111 rfi monitoring data on 09sep12,10sep11.

• This used the 4 synths affected to generate 4 separate frequency bands.
• Whenever there was rfi in 1 of the 4  bands, the rfi was in the same relative position in the other 4 bands It used the 4

• The frequency range for each of the 4 spectra should be separated by 25 MHz.
• The large rfi spike appears at the same relative position in the 4 spectra. This would happen if all 4 2nd lo synths were all at the same frequency.
  

Timeline

• 09sep12
  
• 05:35 vme crates pnt, cor, and da were rebooted.
  
• the radar experiment that was starting had some troubles so they tried rebooting all of the crates
• 10:07 trouble talking to interim correlator
  
• The operator rebooted the da crate and then they rebooted the cor vme crate
• When the da crate was rebooted, the message log had:
• Sep  9 10:29:23 da :if2Prog:E:3f7000b:Error reading in amplitude for hp synth:1
• Sep  9 10:29:28 da :if2Prog:E:3f7000b:Error reading in freq for hp synth:2
• Sep  9 10:29:33 da :monTemp:E:3f7000b:reading hp34401. acpwr off?
• Sep  9 10:29:38 da :if2Prog:E:3f7000b:Error reading in freq for hp synth:3
• Sep  9 10:29:48 da :if2Prog:E:3f7000b:Error reading in freq for hp synth:4
• After this, the 4 if2 synthesizers (1,2,3,4) did not get updated until 11:00 am on 11sep12.
• During this time, the gpib writes to these synthesizers did not generate any errors messages
• 10sep12
• all day the 4 gpib synths did not update as requested. there were no error messages reported.
  
• 11sep12
• 10:30 am dana notices that the synthesizers were not working when the tsys for the carriage house came out bad.
• The power on the synths were cycled. The if2Prog was also restarted. The synthesizers started communicating again.
• the tsys for 9-11 aug for the other receivers did not look very bad. The frequency in the synthesizers must have be close enough to give some output power.

Which projects were affected:

• alfa does not use the 2nd IF's so alfa experiments were  not affected.
• Mock experiments:
• the mocks use up to 7 synths to generate separate bands.
• boxes/beams: b0s1gN,  b2s1gN, b4s1gN, b6s1gN
  
• use the affected synthesizers. These bands will have the incorrect frequencies
• Boxes/beams: b1s1gN, b3s1gN, b5s1gN
•  use synthesizers on ethernet. They will not be affected. These spectra will be ok.

To do:

• Writing to the synthesizers does not report a problem when this happens.
• Reading back the frequency on each write would probably generate an error. This would slow things down a little.
• Measuring the system temperature did not show the problem since the frequency was close enough to what it was supposed to be.
  
• The lro power meter was added to the gpib.After this addition, the problem seemed to occur more often.
• Maybe we should move this power meter to a separate gpib ethernet.
• I tried  looking at the rfi  in the x111 to determine what frequency was actually being used. I think it might have been close to 1080 MHz.
  
• It's probably better to just reschedule:
• p2679.. sorry Joanna.. this was supposed to be a makeup from a previous problem!!
• a2705.. only 3 out of the 7 bands have valid data,