Документ взят из кэша поисковой машины. Адрес
оригинального документа
: http://www.atnf.csiro.au/computing/software/askapsoft/sdp/docs/current/tutorials/intro.html
Дата изменения: Unknown Дата индексирования: Tue Apr 12 13:02:55 2016 Кодировка: ISO8859-5 Поисковые слова: п п п п п п п п п п п п п п п |
This tutorial demonstrates basic operation of the ASKAP Central Processor platform, and does so by walking you through the process of imaging a simulated BETA dataset. The primary goal of this tutorial is to introduce you to the platform and the program execution environment.
The simulated measurement set has the following specifications:
You should read the Central Processor Platform documentation and in particular have setup your environment per the section entitled “Setting up your account”.
Your working directory will not be within your home directory, instead it will reside on the fast Lustre filesystem:
cd /scratch2/askap/$USER
mkdir introtutorial
cd introtutorial
The measurement set used in this tutorial resides on the “Commissioning Archive” and can be retrieved and extracted using the following commands:
ashell.py
<ivec.offline>login
<ivec.online>get /projects/ASKAP Commissioning Data/tutorials/basic1.tar
<ivec.online>quit
tar xvf basic1.tar
You may notice the “get” may stall. This is likely due to the fact the data has not been fetched (staged) from tape to disk. This is quite normal, and the length of the stall depends upon the load on the system (e.g. other users).
Since the measurement set has no calibration errors we can move straight to imaging. First we will produce a dirty image with the cimager program. Create the following configuration file for cimager, lets call it dirty.in:
Cimager.dataset = coarse_chan.ms
# Each worker will read a single channel selection
Cimager.Channels = [1, %w]
#
Cimager.Images.Names = [image.i.dirty]
Cimager.Images.shape = [3328,3328]
Cimager.Images.cellsize = [10arcsec,10arcsec]
Cimager.Images.image.i.dirty.frequency = [1.270e9,1.270e9]
Cimager.Images.image.i.dirty.nchan = 1
Cimager.Images.image.i.dirty.direction = [12h30m00.00, -45.00.00.00, J2000]
#
Cimager.gridder.snapshotimaging = true
Cimager.gridder.snapshotimaging.wtolerance = 1000
Cimager.gridder = AWProject
Cimager.gridder.AWProject.wmax = 1000
Cimager.gridder.AWProject.nwplanes = 33
Cimager.gridder.AWProject.oversample = 4
Cimager.gridder.AWProject.diameter = 12m
Cimager.gridder.AWProject.blockage = 2m
Cimager.gridder.AWProject.maxfeeds = 36
Cimager.gridder.AWProject.maxsupport = 512
Cimager.gridder.AWProject.variablesupport = true
Cimager.gridder.AWProject.offsetsupport = true
Cimager.gridder.AWProject.frequencydependent = true
#
Cimager.solver = Dirty
Cimager.solver.Dirty.tolerance = 0.1
Cimager.solver.Dirty.verbose = True
Cimager.ncycles = 0
Cimager.preconditioner.Names = [Wiener, GaussianTaper]
Cimager.preconditioner.GaussianTaper = [30arcsec, 30arcsec, 0deg]
Cimager.preconditioner.Wiener.robustness = 0.0
Cimager.preconditioner.Wiener.taper = 64
#
Cimager.restore = true
Cimager.restore.beam = fit
Next create a file called dirty.sbatch, this is a description of the batch job that the system will execute:
#!/usr/bin/env bash
#SBATCH --ntasks=305
#SBATCH --ntasks-per-node=20
#SBATCH --time=02:00:00
#SBATCH --job-name=cimager
#SBATCH --export=NONE
aprun -n 305 -N 20 -ss cimager -c dirty.in > dirty_${SLURM_JOB_ID}.log
Before submitting the job for execution, lets dissect this file line by line. This line requests 305 MPI processes, which for the cimager results in one master process and 304 worker processes. Each worker process will process a single spectral channel from the measurement set:
#SBATCH --ntasks=305
Next –ntasks-per-node=20 indicates 20 processes should be launched on each node. The nodes in the system each have 20 CPU cores, so this results in an optimal assignment of one process per core. This job will then require 16 compute nodes.
#SBATCH --ntasks-per-node=20
The time limit is set to 2 hours, meaning if the job has not finished in 2 hours it will be killed. It is useful to set such a reasonable limit to ensure your job doesn’t run indefinitely (which can happen in the case of a bug or misconfiguration):
#SBATCH --time=02:00:00
The “–job-name” option sets the job name to cimager. This can be anything (there are some restrictions) and is used to identify your job in the list of all jobs running on the system:
#SBATCH --job-name=cimager
This final line actually executes the program. The aprun wrapper is used to execute all jobs on the Cray compute nodes. Here “-n 305” and “-N 20” repeat those numbers already described earlier. The “-ss” option specifies strict memory containment per NUMA node. This is just a performance optimisation and can be ignored for now. The cimager program is then executed by aprun with command line arguments “-c dirty.in” which specifies the configuration parameter set created above. Finally, the output is directed to a file named dirty_1234.log (where 1234 is the unique job ID). Without this redirection, the output will go to stdout and will only be written to disk once the job is complete. By redirecting the output to a file, the file can be inspected at run time to track progress:
aprun -n 305 -N 20 -ss cimager.sh -c dirty.in > dirty_${SLURM_JOB_ID}.log
Now this job can be submitted to the SLURM scheduling system for execution with the sbatch command like so:
sbatch dirty.sbatch
The sbatch program returns the Job ID of the created job (e.g. 1234) which you can then use to monitor the status of your job:
squeue -j 1234
JOBID USER ACCOUNT NAME ST REASON START_TIME TIME TIME_LEFT NODES CPUS
1024 user123 astrono cimager R None 2014-05-21T10:00:27 2:00:00 2:00:00 16 320
Alternatively, you can use the command “squeue -u $USER” to list all of your incomplete jobs. You can also (once the job begins running) tail the file dirty_1234.log (where 1234 is your job id) to track its progress.
When the job completes it produces the following output files:
Filename | Description |
---|---|
image.i.dirty | Ignore this image, it will be empty |
image.i.dirty.restored | The dirty image |
psf.i.dirty | PSF image |
sensitivity.i.dirty | Sensitivity pattern image |
weights.i.dirty | Weights image |
If you have CASA installed on your desktop computer you may download and visualise these images with casaviewer. First copy the file from the /scratch2 filesystem to your desktop computer:
scp -r galaxydata.ivec.org:/scratch2/askap/<USERID>/introtutorial/image.i.dirty.restored .
scp -r galaxydata.ivec.org:/scratch2/askap/<USERID>/introtutorial/psf.i.dirty .
scp -r galaxydata.ivec.org:/scratch2/askap/<USERID>/introtutorial/sensitivity.i.dirty .
scp -r galaxydata.ivec.org:/scratch2/askap/<USERID>/introtutorial/weights.i.dirty .
The string <USERID> needs to be replaced with your iVEC userid. Alternatively, you may wish to use the BBCP program for faster data transfer if you have it installed:
bbcp -z -P 10 -s 16 -w 2M -r galaxy-data1.pawsey.ivec.org:/scratch2/askap/<USERID>/introtutorial/image.i.dirty.restored .
bbcp -z -P 10 -s 16 -w 2M -r galaxy-data1.pawsey.ivec.org:/scratch2/askap/<USERID>/introtutorial/psf.i.dirty .
bbcp -z -P 10 -s 16 -w 2M -r galaxy-data1.pawsey.ivec.org:/scratch2/askap/<USERID>/introtutorial/sensitivity.i.dirty .
bbcp -z -P 10 -s 16 -w 2M -r galaxy-data1.pawsey.ivec.org:/scratch2/askap/<USERID>/introtutorial/weights.i.dirty .
Once the files have finished downloading, view the restored image with casaviewer:
casaviewer image.i.dirty.restored
You may need to adjust the data range so as to see the fainter sources:
All four images are tiled and shown below - image.i.dirty.restored (top left), psf.i.dirty (top right), sensitivity.i.dirty (bottom left), weights.i.dirty (bottom right)