ATNF VLBI Wiki

This page briefly explains the purpose of the myriad utilities that have been developed in aid of debugging and operations support. These are organized by topic. For an alphabetical list of these with considerable more detail, visit the Reference Manual.

• mpifxcorr : The software correlator itself
• vex2difx : Makes .input and other files used by mpifxcorr from supplied .vex file
• difx2fits : Takes the output of mpifxcorr and generate .FITS files
• difx2mark4 : Takes the output of mpifxcorr and generates Mark4 format files
• calcif2 : Takes .calc file made by vex2difx and produces the delay model (.im) file
• CalcServer : A service that makes calls to the CALC program upon request by an RPC call

• difxcopy : Copies a .input file, and other files referenced within, to a new path, updating full paths internally
• difxlog : Captures multicast status and messages from a running DiFX job and writes to a text file
• difxsniff : Concattenates as appropriate output sniffer files from mpifxcorr and plots using plotwt, plotapd and plotbp
• genmachines : Determines location of Mark5 modules via request to mk5daemon processes and makes .threads and .machines files
• makefits : Simplified wrapper interface for difx2fits, tailored to NRAO's usage
• oms2v2d : Makes skeleton .v2d file based on the sched .oms output file
• startdifx : Command line utility to start mpifxcorr
• stopdifx : Command line utility to stop a single running instance of mpifxcorr
• vexpeek : Program that reads a .vex file and prints the project name and list of stations and the time period each observed; used by db2vex
• vlog : Splits up a TSM style text log file into pcal, weather, tsys and flags files; mainly for VLBA operations
• startcorr.pl : Simple DiFX launcher, with eVLBI support
• difxwatch : Monitors progess of running mpifxcorr. Kills job if no progess for longer than 300s.
• espresso: a lightweight suite of programs designed for management of file-based correlation.

• difxarch : Calls e2ecopy to transfer correlator output to a final archive destination, consulting a list of expected files along the way
• difxclean : After archiving of data is complete, removes all DiFX input and output files from the correlation staging area
• difxqueue : Copies .input and associated files (with difxcopy) to a staging area and updates a database of jobs to run
• difxusage : Queries database for total correlator usage over a requested time period
• db2vex : Database utility that reads a .vex file, queries an operations database, and makes a new .vex file containing additional information such as clock offsets, earth orientation parameters and Mark5 module VSNs
• getshelf : Requests the current shelf location of a module from the operations database
• jobdisks : Looks at a .input file for the list of Mark5 module VSNs
• joblist : Lists all the jobs in the current directory and summarizes antennas and bands to be correlated
• jobstatus : Broken for DiFX 2.0
• listcpus : Uses ssh to determine CPU type of all machines in a DiFX cluster
• makefits : Simplified wrapper interface for difx2fits, tailored to NRAO's usage

• checkmpifxcorr : Checks the contents of .input file, and files referenced within, for syntax compliance
• difxcalculator : Uses difxio to load the .input file and prints a summary of memory, cpu, and network utilization for a job

• cpumon : Displays multicast information, as usually sent by mk5daemon, such as CPU, memory and network utilization, from DiFX cluster members
• errormon : Prints multicast messages to screen
• errormon2 : Similar to errormon, perhaps with more useful defaults
• mk5mon : Displays multicast information about Mark5 units
• statemon : Prints condensed summary of the DiFX state messages

• diffDiFX.py : Generates a context-sensitive difference of two DiFX output files for detailed version testing. Corresponding visibility records are differenced and statistics on the differences are accumulated and printed at the end of the processing.
• neuteredmpifxcorr : : document in user guide
• plotDiFX.py : Given one or more .difx output files, plots amplitude, phase and lag information to the screen (overplotting in different colours when >1 file given) one subband at a time. Loops through the records sequentially. : document in user guide
• plotDynamicSpectrum.py : Given a .difx file, makes a dynamic spectrum of the visibilities (either a single baseline or scalar averaged over all baselines). : document in user guide
• printDiFX : Prints summary visibility information of DiFX output files
• snipDiFX.py : Given an input .difx file and a timerange, extract only those visibilities falling in the specified timerange and copy them to a new .difx file. : document in user guide
• testdifxinput : Prints a summary of a .input file based on the perspective of the difxio library
• testdifxmessagereceive : Prints details of multicast DiFX messages
• vlbi_fake : A program to benchmark the performance of sending data around the network : document in user guide

• condition : Queries database for results of module conditioning; also allows injection of new conditioning results
• condition_watch : A service that listens for multicast conditioning information and adds new records to database
• mk5cat : Open for read a scan or byte range on a Mark5 module and send to stdout; there seems to be some minor issues left with this (work in progress)òÀæ
• mk5control : Utility to send a command to one or more mk5daemon processes
• mk5cp : Copies one or more scans or a byte range from a Mark5 module to one or more files
• mk5daemon : A process that runs on all DiFX cluster members, not just Mark5 units, that logs and multicasts status and is responsible for starting DiFX jobs and fielding commands sent by mk5control
• mk5dir : Reads the user directory from a Mark5 module and stores it in a file; by default this reads a bit of data from the beginning and end of each scan and is thus useful for identifying malformed data
• mk5erase : Erases a Mark5 module; like the Haystack SSErase program but with more options; can set the directory to be either legacy or new format and transmits progress via multicast messages
• recover : Like the Mark5A recover command
• testmod : Does read and/or write tests of Mark5 modules
• vsn : Retrieves or sets the Volume Serial Number of a Mark5 module; also prints the serial numbers, model numbers and capacities of the drives in the module.

• cleanVDIF : Walks packet-by-packet through a VDIF file looking for packets of invalid length (often ARP packets from a switch inadvertently captured by a Mark5C) or corrupted header info, stripping out bad packets and writing a 'clean' VDIF file. : document in user guide
• m5d : Decodes the header and samples of baseband data from a mark5access supported baseband data file
• m5fold : Folds in time the voltage squared (akin to pulsar processing); can be used to identify switched power in a datastream
• m5pcal : Extracts pulse cals from a baseband data file
• m5spec : Forms spectra of each sampled channel in a baseband data file
• m5test : Reads through a baseband data file looking for missing or damaged frame headers
• m5tsys : Extracts switched power from a baseband data file (rather, it extracts Pon and Poff for each sampled channel)
• multi2singlethreadVDIF : Takes a VDIF file with 2^N single-channel threads and multiplexes these channels into a single VDIF thread with multiple channels. The resultant single thread output file can be processed by e.g. m5spec and other utilities, and by mpifxcorr. : document in user guide
• padVDIF : Walks packet-by-packet through a VDIF file looking for missing packets and inserting packets as necessary to make a continuous packet stream. The inserted packets have the invalid bit set. : document in user guide
• printVDIF : Loops through a VDIF file inspecting each packet header and printing some basic summary info (time etc)
• stripVDIF : Strips UDP overhead from VDIF packets captured using wireshark, leaving a correct VDIF file. : document in user guide
• zerocorr : A program that takes two baseband data signals and performs a zero-fringe-rate cross-correlation; can handle mismatched bands/sidebands and different data formats and sample rates

• e2ecopy : NRAO specific program to move, rename, and set permissions of files bound for the VLBA archive
• getvex : Copies .vex file and other bits from a standard filesystem location within NRAO operations

• dedisperse_difx : : document in user guide