─юъєьхэЄ тч Є шч ъ¤°р яюшёъютющ ьр°шэ√. └фЁхё юЁшушэры№эюую фюъєьхэЄр : http://www.atnf.csiro.au/vlbi/dokuwiki/lib/exe/fetch.php/difx/meetings/korea2013/korea_difx_vdif_oct2013.pdf
─рЄр шчьхэхэш : Tue Oct 15 12:01:03 2013
─рЄр шэфхъёшЁютрэш : Tue Apr 12 12:45:47 2016
╩юфшЁютър:
╧юшёъют√х ёыютр: тэх°эшх яырэхЄ√

A New VDIF (and Mark5B) Decoder
7th DiFX Users/Developers Meeting, Tamna Observatory, South Korea

Walter Brisken
NRAO - Socorro

DiFX at NRAO
╖ ╖ VLBA Sensitivity Upgrade Project has been completed н This required some changes to DiFX Many new Mark5C specific updates
н н

Support for new Directory format Diagnostic programs for VDIF data

vdifspec, vdifd, vdifbstate, vdiffold Perform "fix" operation on copy (more on this later)

н

Special option in mk5cp for Mark5B data

╖ ╖

Support for VDIF / Phased VLA
н

Starting with considerable effort by Adam LMT (w/ Haystack) International fringes to SHAO 65m
DiFX7@Tamna 2013 2

Some "first fringes":
н н

VLBA DiFX Correlator
╖ ╖ ╖ Upgraded from 20 to 24 nodes (200 to 232 cores) Occasionally encounter insufficient memory conditions н For 200+ phase centers 14 Mark5 playback devices
н н

Mark5A systems at VLBA antennas are being retired Will increase correlator to >=20 units this year

╖

Thanks to Steven Tingay we will be getting DiFX hooked up to a Lustre filesystem
н н

Should speed up some operations Will provide more storage

DiFX7@Tamna 2013

3

Introduction and Motivation
╖ ╖ ╖ ╖ Mark5B support in DiFX is fairly mature already VDIF support has been functional for a while Both cope with minor interruptions to perfect data flow Some important cases have been difficult to deal with: н VDIF from VLA: sometimes threads start at vastly different times н VDIF from VLA: sometimes there are large gaps in a recording н Mark5B and VDIF at VLBA: new X-cube soft-switch drops ~10-5 packets н VLBA: Either Mark5C or RDBE occasionally inserts extra data into stream н Basically, hardware is not perfect, so fix in software VDIF performance in Mark5 playback suffers due to overtaxing the Mark5 CPU Want robust, hiFX7@Tamrfo013 nce VDIF and Mark5B Di gh pe na 2 rma 4

╖

Design Principles
╖ ╖ ╖ ╖ Keep complicated algorithms in external libraries н Simplify testing and reuse Don't compromise on speed Full reporting of statistics Simple interface for dealing with abnormal data н Avoid having to deal with many different exit conditions The above principles can apply to most developments ╖ Separate record scan and schedule scan concepts н It is possible one record scan contains more than one schedule scan н It is possible one scheduled scan is split into multiple record scans
DiFX7@Tamna 2013 5

General approach
╖ ╖ ╖ ╖ First: jump to starting point in media Next: read a chunk of data (from file, Mark5 or network) Then: "condition" the data with code to reorganize/filter Finally: put data into datastream buffer section and assign to that section the timestamp of the first valid packet

DiFX7@Tamna 2013

6

VDIF pre-processing:
╖ vdifio : vdifmux() does the following н Reorders incoming packets н Pads missing data with masked data н Excises unwanted "interloper " data н Corner-turns multi-thread VDIF into single-thread VDIF н Output is uniform time grid of data; incomplete records are masked Statistics generated н # valid and invalid input frames н # duplicate frames (should never happen, but has been seen to) н # wrong thread frames (indicates correlator configuration problem) н # skipped bytes н # bytes processed in total н # of total good single-thread VDIF frames produced
DiFX7@Tamna 2013 7

╖

Mark5B pre-processing
╖ mark5access : mark5bfix н Excises unwanted "interloper " data н Ensure first output packet has timestamp that has integer ns start time н Pads data with fill packet to replace missing packets н Mark5C records 1 Mark5B frame as 2 packets: Look for both halves of a frame (test is not perfect though) н Outputs uniform time series of Mark5B packets Statistics generated н # valid and invalid input frames н # of interloper bytes н # of missing packets н # of bytes processed
DiFX7@Tamna 2013

╖

8

Performance enhancing tricks (VDIF)
╖ Minimal data copying is done н First pass through data identifies data movements to be performed н Second pass through data corner-turns from initial location to final location Optimized corner turning algorithms н Separate algorithm for each #channels / #bits case In some cases separate algorithms for 32- and 64bit CPUs н Makes use of bit mask-and-shift operations Usually less than 1 integer operation per sample н I will investigate Harro's SSE2 corner turners Use OpenMP for instruction-level parallelization
DiFX7@Tamna 2013 9

╖

╖

Native Mark5 implementation
╖ Data is gotten from a Mark5 module with XLRRead(...) н This function blocks Any extra processing will reduce datastream throughput н It performs a DMA transfer and takes effectively zero C PU н It has known problems for reads > 25 MB A new posix thread is now spawned for the reading of data н Reads > 20 MB are split into a series of smaller reads A posix barrier is used to synchronize reading with preprocessing н A series of posix mutex locks are used to control a circular buffer for incoming data
DiFX7@Tamna 2013 10

╖ ╖

File implementation
╖ ╖ Not that different from existing Mk5 datastream Uses new vdifsum() and mark5bsum() functions to generate summaries of input files to allow for a quick jump to the expected start of job data. Could possibly benfit from a similar threading model used by Mark5 datastream, but thought to be less important as OS-level file read-ahead effectively does this for free

╖

DiFX7@Tamna 2013

11

Network (eVLBI) implementation
╖ ╖ ╖ Not yet started н Much thought has gone into it though A double buffer system similar to that used by the Mark5 implementation will be used End result code may be simpler than existing eVLBI code in DiFX as the packet reordering / gap filling code is already compartmentalized Want to support UDP, TCP and raw socket operations Support for simultaneous reads from multiple network cards is a natural extension н This may be needed by LOBO (see slides at end)...

╖ ╖

DiFX7@Tamna 2013

12

Development status
╖ ╖ VDIF work began in early 2013 Work accellerated in July н X-cube packet loss in Mark5B data prompted immediate effort on the Mark5B front Currently the code is working, is fast, and is in use by the VLBA for all wideband data Code is being developed in the vdifnew branch of mpifxcorr Currently there is a separate class derived from DataStream for each of: н VDIF on Mark5 (vdifmark5.cpp/h) н VDIF on files (vdiffile.cpp/h) н Mark5B on Mark5 (mark5bmark5.cpp/h) н Mark5B on files (mark5bfile.cpp/h) н Ultimately some consolidation of code should occur
DiFX7@Tamna 2013 13

╖ ╖ ╖

Development status (2)
╖ Some functionality is still missing: н "Fake" mode for benchmarking н Network datastream See section on VLITE below н Jump The new datastream classes ignore the requested data time in the manager's data request н Cases where a correlator job skips scans require reading all intermediate data A jump at the beginning of the job is performed properly Workshop goal: pave way for merging this into trunk and getting a new DiFX release planned before 2013 comes to an eDiFX7@Tamna 2013 nd!

14

VLITE
╖ ╖ VLA Low-band Ionosphere and Transient Experiment A commensal P-band (~330 MHz) system н Observes all the time at 330 MHz, even when used for other science Funded by Naval Research Laboratory A 10-antenna ramp up to LOBO: the LOw-Band Observatory Consists of н Samplers/channelizer/formatter for 10 VLA antennas н Real-time DiFX correlator н Real-time integration with VLA M&C system Being developed by NRAO Oct 2013 to Sept 2014 н I will do the DiFX development
DiFX7@Tamna 2013 15

╖ ╖ ╖

╖

New calibration data path
╖ ╖ difx2fits attaches Tsys, Pcal, weather and flag data to FITS files Previously such data could only be provided as one-file-pertable (tsys, pcal, weather, flag) except when DiFX extracted Pcal or Tsys Now difx2fits looks for .. н is experiment name: BD152, ...
н н

╖

is antenna code: HH, ... is one of: tsys, pcal, weather, flag 1. DiFX extracted (pcal / tsys only) 2. .. 3.
DiFX7@Tamna 2013 16

╖

Priority now is:
н н н

WFB in 2014 and beyond...
╖ ╖ ╖ I am moving! Off to Minneapolis Minnesota Will retain a 50% position with NRAO н Less focus on operations/VLBA development н Heavy focus on VLITE project No end of my DiFX involvement in sight! н Beyond Sep 2014 my NRAO role becomes less clear... Will acquire a 50% position at UMN н Working with the Cerenkov T lescope Array project e Beyond Sept. 2014, some uncertainty of NRAO position

╖ ╖

DiFX7@Tamna 2013

17