Документ взят из кэша поисковой машины. Адрес оригинального документа : http://www.atnf.csiro.au/computing/software/askapsoft/sdp/docs/current/_sources/platform/externaltransfer.txt Дата изменения: Tue Mar 8 06:55:19 2016 Дата индексирования: Tue Apr 12 13:10:30 2016 Кодировка:

External Data Transfer
======================

You will often want to transfer data between the ASKAP Central Processor and computers
at your home institution. For example, upon completion of imaging you may want to visualise
those images on your desktop computer.

You may use the secure copy program (scp) to copy data to/from either your home directory or
the scratch filesysyem. A special node has been setup, with hostname *galaxydata.ivec.org*,
for this purpose. It is possible to copy data through the login nodes, however this should
be avoided if possible so as to reduce the load on the login nodes.

Here is an example of copying a file from both the home directory and the scratch filesystem.
Note these commands are executed on your local host (e.g. your desktop or laptop)::

scp -r galaxydata.ivec.org:/scratch2/askap/user123/image.fits .
scp -r galaxydata.ivec.org:~/1934-638.ms .

or to copy from your laptop/desktop to the Central Processor::

scp -r image.fits galaxydata.ivec.org:/scratch2/askap/user123
scp -r 1934-638.ms galaxydata.ivec.org:~

Note the "-r" performs a recursive copy so can be used to copy files or directories.

Using BBCP
----------

Using *scp* can be quite slow and a program called *bbcp* is suggested for large files.

* Binary packages: http://www.slac.stanford.edu/~abh/bbcp/bin/
* Obtaining the source with git: "git clone http://www.slac.stanford.edu/~abh/bbcp/bbcp.git"
* Command line parameters: http://www.slac.stanford.edu/~abh/bbcp/

The ASKAP software team can supply Debian packages. Usage is similar to scp, but with
a few extra parameters. To copy a file from the /scratch2 filesystem::

bbcp -z -P 10 -s 32 -w 2M -r galaxy-data1.pawsey.ivec.org:/scratch2/askap/user123/image.fits .

and to copy a file to the /scratch2 filesystem::

bbcp -P 10 -s 32 -w 2M -r image.fits galaxy-data1.pawsey.ivec.org:/scratch2/askap/user123

.. note:: The hostname necessary to use bbcp is *galaxy-data1.pawsey.ivec.org*. This is one of the
two hosts to which the *galaxydata* DNS alias points to (the other is *galaxy-data2*).
This is necessary as bbcp doesn't reliably establish connections via the galaxydata
alias due to the fact connections are round-robined between the two nodes.

The three additional options result in production of progress messages every 10 seconds,
sets the number of parallel network streams to be used for the transfer to 32, and sets the
preferred size of the TCP window to 2MB. On some networks increasing the number of streams
from 32 (up to a maximum of 64) may result in even higher throughput.