• movies 2013
• movies 2014
• movies 2015
• nov15 heating campaign
• movies 2016
• 160105: world day
  
• 160205: world day
  

• nov15
  

• npl: nighttime plasma line
  

Location of raw Coded long pulse (clp) data taken with echotek

• filename (without the directory path)
  
• experiment name. This is the directory where the data was found.
• The naming convention is ProjectId_date (where date is the star to run).
  
• index into the path arrays that we search through (see below). If the index is < 0 then this file is no longer on disc.
• tmSesFrom1970 (unix time). This is the time of the last write to the file (this assumes that the files are copied preserving the last write time..)

• 1st line: last date that idl script searched for these files.
  
• col1: index number (sorted by project name)
• col2: the directory name holding the files
• col3: total number of files we have seen
• col4: total number of files that are currently on disc (as of the last run of the idl save script).
• col5: the year,time for the first file in the directory (from col2)

Files,locations:

• /share/megs/phil/x101/atm/clpshs/filelist .. directory holding info
• flist.pro - idl program to search for files
• shsfiles.sav - archive holding the files found
• flbyexpnum.pro  - print out files by experiment name

may14:  plasma line movies (top)

    I've made movies of the 10 second decoded plasma line images. The process is:

• do the regular decoding /spectral computation of each 10 second average
• create a jpeg file for each 10 sec spectra
• use mencode (on linux) to make a .avi file from all of the jpegs of a run.
• the command i'm using (10jan15..)
  
• mencoder "mf://$inpdir/*_${chn}.jpg" -o $outf -mf type=jpg:fps=30 \
              -ovc lavc -lavcopts vcodec=mjpeg
• Prior to 10jan15 is was using -ovc copy.. this didn't work on quicktime, and positioning didn't work
• I redid most of the movies to use the -ovc lavc encoder.
• I've tested this on windows with quicktime and media player. they can both display it. Positioning also works.
  
• I use mplayer (on linux) to then look at the movie.
• the movie has the data, time, az,zaDome,zaCh and filename as a header  for each frame of the movie
  

Making the movies:

programs/Files used:

Current hit movies:

• movies located in :/share/megs2_u1/phil/data/clpshs
• sometimes the data is not available  at night.
• sometimes the individual frames were inspected (prior to making a movie). If no signal was seen at night, then the raw data was deleted (to save on disc space).
• We could have missed some weak signals that could have been seen with a movie
• Other times, there is a break in the data because of transmitter problems.
• Notes on images:
• the sensitivity is not the same for all movies
• We can run the xmter:
• single beam: We use only the carriage house (ch). It gets all of the power from the xmiter
• dual beam: we use the ch and the gregorian dome
• we have the option of splitting the power any way we want between the ch and the dome. Typically we set it so the both return similar s/n power profiles.
• It is possible to put all of the power in the dome (if we just want the plasma line data..)
• If the xmter is having problems, we may run at a reduced power output.
  
• The vertical (range) oscillation of the up shifted line is probably from the beam swinging (parallel and perp to the magnetic field  lines).
• sometimes we see plasma lines during the night time.
• On some days the line appears before  sunrise.. even before sunrise of the conjugate point (at the altitude of interest).
  
• Occasionally there is a lot of jumping near the peak at sunrise.
• horizonal strips equal to the baud length  are usually lightning. You will see them in the afternoon
• there are still a few datasets i need to process in 2014.. (data no longer on disk). and all of 2013 needs to be done with the bands combined
• chn12 covers 433 to 442 Mhz (+3 to +12)
• chn1 covers 433 to 438, chn2 covers 437 to 442.
  

• linux: use .avi
• mac: use .mov (quicktime movie)
• windows .. whatever works...
  

• 02may15: t2686 frank and herb
• ran coded (20 sec) then uncoded (20sec) clp
• bands at 5.5 and 10 Mhz6
  
• processed both bands
• fof2 8-10 mhz all night
• 20140502c_chn1.avi, 20140502c_chn2.avi . 447 Mbytes each
  
• lower band. lost signal around 7:30.
• was going out of image (700 km around 4 mhz).
• upper band.
• could follow fof2 until around 10:30 (very weak)
• looks like it might have been coming back around 3am but not sure..
  

04apr14 - t2876 measuring the electron line (top)

    t2876 was trying to measure the electron line. It used d-region spectra and coded long pulse.
The coded long pulse setup was:

• 10 usecond baud, 250 usec rf pulse, and 5 millisec ipp
• data was acquired in 30 second blocks.
• heights were taken from from 48 to 190 km (spaced by 5usecs or .75 km)
• The data was processed by decoding, computing spectra and then averaging
• When processing, i had to turn off the rfi excision since it was cutting out the ion line..
  

• the data came from file t2876_20140404_007.shs.. the first 30 second block of clp data in the file.
  
• 4 ranges are over plotted in color (85.5, 123, 160.5 and 183 km).
• Top: +/- 2.5 Mhz displayed
• middle: +/- .5 Mhz displayed
• bottom: +/- .1 Mhz displayed.

• The steps at .1 Mhz are from the 10 usec baud transistions
• the peak in the middle is the ion line. It gets broader as you go up in height
• +/- 100 khz spectra. this is probably clutter from the ion line caused by the clp decoding
• > 100 Khz.. hopefully there's an electron line in there..

processing: x101/140403/clpeline.pro

01feb14 - removing rfi from clp data (top)

    Rfi removal was tested on clp data taken 03feb14 (file 15, block 1). The removal process was dpne on an ipp  basis:

• transform 22000 height points -> Vspc[22000]
• compute power -> Pspc[22000]
• compute a robust rms of the 22000 power points:
• sigma1=rms of the initial points
• find all points within Nsigma * sigma1of the mean value ( i used Nsigma = 5.)
  
• rms2=rms off all points within Nsigma*sigma1 of the mean
• set sigma = ( rms2/rms2 > 2)?sigma2 : sigma1
• This tries to throw away outliers.
• find all points > 5*sigma from the mean power
• Set Vspc[ii] = 0 for these points
• inverse transform back to the time domain.

Looking at the results

• The first image shows no clipping (.gif)
• Clipping the rfi with using 5 sigma
  
• 5 sigma clipping replacing the bad data with 0's (.gif)

05sep12: power profile cal dropouts. (top)

    On 05sep12 a world day run was in progress. Nestor noticed that the cal (at the end of the ipp) was not always present in the power profiles. The experiment was:

• dual beam,
• za ch =0, za gr=15, az swinging at .04 deg/sec
• running: clp,mracf,tpsd, and power.

• data processing:
• files t1193_04sep2012.073-121 were processed. (about 8:00 to24:00).
  
• 10 sec blocks of power profile data were processed as a unit.
• For each ipp in a block, the average calOn, calOff power was computed
  
• there were 200 calOn samples followed by 200 cal off samples (sample rate=2usecs)
• 25 samples at the start of calon and caloff were discarded to eliminate the cal transition
• Page 1: calOn,calOff  power  in a/d units
• top Frame: ch, bottom frame: Gregorian
• Black is calOn, red is calOff
• The median calon,caloff for each 10sec block was plotted
• When calOn (black) was close to calOff (red) , then the cal had not  fired
• You can see this at: 8-9am, 12.5 to 13
• 13:05:  i changed the cable going from the sps to the cal multiplexer calport 7 input.(flagged with a green line)
• After this cable change, there were no more cal dropouts.
• 18:00 must have been a crowbar, the system was put on load for awhile
• 21:00: the galaxy transited overhead.
•  the line feed shows it drifting through the beam
• The dome was swinging +/- 15 degrees so it cut through the galaxy many times during the swinging.
• Page 2: cal deflections in degK.
• Same as page 1 except the units are degK (computed from the cal).
• For the times where the cal did not fire, i used the median cal deflection for the entire dataset.
• This shows:
•  the strength of the galaxy (about 100K)
• The load temperature (about 350K for ch)
  
• Page 3: a blowup where the cal failure started:
• three 10 second blocks of cal data are plotted (.01 sec resolution).
  
• The gaps are about 1 minute of time (when the other programs ran).
• The bottom plot is a blowup of the final 10 second block.
  
• colors:
• ch black calOn, red Caloff
• gr Green calOn, blue calOff
• For the 1st 10 seconds the cal deflection was correct.
• For the middle 10 seconds the Gregorian cal deflection increased by about 10db. jumping back and forth
• The cal off measurement remained stable. This sample is 400 usecs after the cal on sample so the problem was definitely the cal turning on
• The final 10 second  block shows:
•  the ch cal stopped firing around 2.6 seconds
• the dome cal started to decrease at this time, but it took it about 2 seconds to turn off.

Summary:

• The cable from the sps to the cal multiplexer was the problem. It was a shinny long cable (i've given it to electronics to look at).
  
• I've been told that the calon signal goes first to the ch and then over to the dome.
• The dome cal jumping way up may have been  a problem with the cal ttl signal at the cal diodes. It may have been close to the ttl transition level causing funny things to happen.
  
• The bad cable was from the sps to cal multiplexer (calport 7). I don't understand why both ch and dome cals didn't stop at the same time
• suppose the bad cable caused the voltage at the cal multiplexor input to full below a valid ttl on. It the cal multiplexor receiving circuit was built correctly it should have sent a solid 0 level upstairs to both the ch and then the dome. It looks like this didn't happen.
• Maybe we should take a look at the cal multiplexer calport 7 receiver chip to see what it is doing

28nov11:Rediscovering the Galaxy with atm az swings. (top)

• Dual beam mode with zach=0 and zaDome=15 deg.
• Az rotation was  .39 deg/sec
• The dbraw routine ran clp,mrac,power, and topsd.
  
• Each ran for 10 to 20 seconds and then stepped to the next program.

• Dynamic spectra were made of the power profile data.
• Data from 14:30 to 16:42 AST on 28nov11 were used.
  
• The power was computed for each of the 2400 power profile heights of each 10 millisec ipp.
  
• The barker code was not decoded.
• The heights were averaged and then decimated by 3 to give 800 heights (2*3=6usecs spacing)
• 100 ipps were averaged to give 1 second time resolution.
• The images were scaled to N sigma using a robust algorithm for computing sigma (N=3.5 for ch, 4.5 for dome)

• The horizontal scale is hours (AST) the vertical is range (km)
• The large strip at 16.5 hours is when the system was switched from sky to load.
  
• Some atmospheric signals:
• 110 Km = this is probably the e-region. Since the barker code is not decoded, any lines get smeared out to 13*2 26 usec + the average by 3 heights.
• 250 Km.. the F-region
• The vertical stripes duration is the 10 seconds of each profile.
• I've plotted each 10 second power profile dataset adjacent to each other rather than leaving blank space where mracf, clp, and tpsd ran.
• Carriage house data:
• There is an increase in power along all heights around 15.48 hours
• Dome data.
• There are multiple vertical stripes of increased power regularly spaced.
• The bright strips are spaced by about 16 minutes.
• The time for 1 rotation istime spacing is 15.1 minutes + the turn around time overhead (probably 20 secs). This is pretty close to the spacing of the peaks. The galaxy also moves a bit during this time.
  

• The data from the dynamic spectra were averaged over the 2400 heights and 10 seconds of each power profile giving  2 power samples every 10 secs of a power profile run (1 for the dome and one for the ch).
• Page 1: positions
  
• TOP: Dome azimuth vs hour of day for the power profiles.
• Bottom: right ascension and declination for each of the power samples.
• Black is the ch.
• At 0 degs za it looks at 18.3 degrees declination
• Red is the dome
• At 15 deg za, the dome swings between 3 and 33 deg declination (18-15, 18+15 where 18 is the  declination of AO).
• Page 2: power vs positions
• Top: Total power vs hour of day (AST).
• black is ch, red is dome.
• The dome has many peaks (which showed as the vertical stripes i the dynamic spectra).
• The ch has one main peak around 15.5 hours AST.
• Middle: power vs right ascension.
• The ch peaks at 19.5 hours RA. This is probably the galaxy transiting at declination = 18.3 deg.
• The peak is about 1.5 * Tsys.
  
• The dome peaks earlier at 19.0 hours. It's peak more than doubles Tsys.
  
• Since the dome is swinging +/-15 degs declination it probably crosses a portion of the galaxy that is stronger than declination = 18.3.
• When the dome crosses 19.5 hours RA, it's level is comparable with the ch value (although the baselines vary by about 10%).
  
• Bottom: Total power vs azimuth.
• Page 3: power vs Azimuth for ch (last 3 swings)
• Previously it was noted that some noise was coming from the transmitter building (az = -20deg).
• With the ch at za=0 and the domeza=15, the dome should shield the ch from this noise every time the dome passed between the ch and transmitter building (az=-20).
• Looking at the last last 3 azswings for the ch, there is not obvious dip in the total power near -40 to 0 deg azimuth. This may just be that the sampling rate is to low, or that 10 second averaging has washed out the narrow noise spikes.

Summary:

• The galaxy was seen in the dome and ch during the azimuth swings.
• The ch has a single peak since it stays at za =0 (dec=18.3)
• The dome sees many peaks (in time) since it is continually swinging through the galaxy while the galaxy transits.
• The galaxy increases the ch Tsys by 50%. The galaxy double the tsys on the dome.
  

30aug11: removing clutter from clp echotek images.  (top)

• the clp clutter.
• various baseline removal techniques.
• the difference between removing the baseline at the peak (narrow plasma line) versus removing the baseline where the vertical gradient is smallest (giving the most smearing within 1 range gate).

Processing:

• The datafile used was: daeron/t2634_30aug11/t2634_30aug2011_000_004.shs.
• The 10 seconds of data started at 8:10:49 ast.
  
• The fregion was till low, so the vertical gradient was at minimum (causing more altitude smearing).
• The clp data was taken with:
• 2 usec baud, 1 usec spacing between heights.
  
• 5 MHz bandwidth (from the dome). Cfr about 436.5 MHz.
  
• 2400 heights (with 1 usec spacing) were decoded (90 to 450 km range).
  
• decoding used 4096 length fft giving 1.2Khz frequency channel width.
• 1000 ipps or 10 seconds of data were processed for the images.
• 8 threads were used to process the data on aserv21.
• The data was processed on aserv21 from the directory /share/megs/phil/svn/pdev/pdev/clp using the command:
• clp1shs.sc --endtime   --nippaccum=1000 --fftlen=4096 --nthreads=8  --nfiles=1 --odir=/tmp/clp1shs/
   --inpdir=/mnt/daeron/t2634_30aug2011 --inpfilepre=t2634_30aug2011_
  
  

• Page 1: Raw spectra and band pass correction:
  
• Top Raw spectra:
• black: spectra a Fregion peak (210 km)
• green: spectra at minimum height gradient (184 km)
• red: avg spectra (200 heights) 420-449 km. This should not have any atmosphere in it.
• The power goes up at the edges of each spectra. This is aliased power from outside the 5 MHz band
• Bottom: spectra with red bandpass correction divided out:
• the red spectra from the top plot was divided into the black and green spectra to remove the system filter shape.
• This is incorrect for the edges of the spectra because of the aliased power.
• Page 2: data  and various smoothing filters to try  and remove clutter:
  
• The increase in the spectra around freq channel number 2400 is caused by clutter.
  
• Various bandpass filters were computed to be laster used to subtract off the clutter:
• boxcar smoothing of 32 (red),  64 (green) channels, and median filtering with 71 channel length (blue) were computed.
  
• top,2nd: 210 Km fregion peak.
• top: spectra at peak with 3 filtered spectra over plotted. The plasmaline is to the right of the clutter peak.
  
• 2nd. blowup around plasmaline of spectra at peak and  3 filters
• The  plasmaline is about 10 frequency channels wide.
• The colored lines show how much each filter would lower the amplitued of the plasmaline.
• 3rd,4th spectra at 184 km (gradient minimum).
• 3rd spectra with the 3 filtered spectra overplotted. The plasmaline is around channel 1600.
• 4th blowup around the plasmaline with the 3 filters overplotted.
• The plasmaline is about 30 channels wide.
• the filters (especially the 32 channel boxcar) will remove much of the plasmaline.

• at the peak where the plasmaline is narrow, boxcar filters can be used to remove the clutter
• Where the vertical gradient is small, boxcar filters will remove much of the plasmaline.
• using wider boxcar filters will fail at the peak where the clutter has the most curvature.

Dynamic spectra showing various filters:

• You should not see a remnant  of the plasmaline in the filter.
• You should see little clutter in the data - filter
• The plasmaline should be stronger below the peak than above the peak.

Summary:

• Filtering the spectra to remove the clutter does not do a good job at all altitudes:
• At  the minimum gradient in altitude the plasmaline is smeared (because of our height resolution).
  
• This requires a wide filter
• At the peak, a wide filter does not do a good job of removing the clutter.
• The shadow plasmalines are caused by the buad transitions (2 usecs = .5 MHz) beating with the actual plasmaline.

 home_~phil