Äîêóìåíò âçÿò èç êýøà ïîèñêîâîé ìàøèíû. Àäðåñ
îðèãèíàëüíîãî äîêóìåíòà
: http://www.atnf.csiro.au/vlbi/dokuwiki/doku.php/lbaops/datatransfers
Äàòà èçìåíåíèÿ: Unknown Äàòà èíäåêñèðîâàíèÿ: Sun Apr 10 23:20:00 2016 Êîäèðîâêà: IBM-866 Ïîèñêîâûå ñëîâà: manicouagan crater |
These instructions are for data transfer from Parkes (pam-store-ext, pkvsi1-ext, pkvsi2-ext), ATCA (cavsi1-ext, cavsi2-ext, caxcube-ext), Mopra (caxcube-ext, mpvsi1-ext, mpvsi2-ext) and ASKAP (cira10).
Requirements:
ssh authentication key setup:
In order to use globus GridFTP to push data to Pawsey via your own user account, you will need to have ssh keys set up.
On each recorder host that you need to transfer data from (iff each has its own independent home directory), use ssh-keygen
to generate your own ssh keys with a passphrase that you can remember. ssh-keygen
will prompt for a filename to save to; use a new unique file in the ~/.ssh area, e.g. òÀÜmyname_rsaòÀÝ. Append the public key to your authorized_keys on magnus with ssh-copy-id -i ~/.ssh/myname_rsa.pub user@magnus-data.pawsey.org.au
(this will prompt you for your Pawsey account password).
After this you should be able to ssh from the recorder host to magnus without needing to enter your password.
Optionally, to manage your keys on the recorder host type ssh-agent bash
and ssh-add ~/.ssh/myname_rsa
. ssh-add
will ask for your passphrase. After this you should be able to log into magnus without being prompted for your passphrase.
To avoid having to repeat this every time you log in again, you can use ssh-agent within screen
(man screen
for info on screen), and re-attach to the screen to continue transfers when you next log in.
After each run (including mini-runs):
How to transfer data:
screen
- see above./scratch/pawsey0169/transfers/{year}/{experiment}/{antenna}
{experiment}/{antenna}/{DASn}
directories./scratch/pawsey0169/transfers/2016/{antenna}
on magnus-data. {antenna}/{experiment}
to the correct {experiment}/{antenna}
directories on magnus-data. To help with this rearrange, Chris has a script ~cphillips/bin/movedir.sh
which works with alias 'movedir' (to be run from within the experiment directory where the data are currently located). To use it, you can add alias movedir='. ~cphillips/bin/run_movedir.sh
' to your own ~/.profile
or similar, and add Chris' ~/bin directory to your path (add the line export PATH=/home/cphillips/bin:$PATH
to your .bashrc).~cphillips/bin/filesum.sh
at Pawsey) on both the copy and the original data at the observatories to compare the two sets of data (check that total data volume, #files, smallest and largest files and first and last are the same). Once this comparison is made, use alias òÀÜmovediròÀÝ on the original data (at the observatories; the alias is already set in the vlbi account) to move it to a directory called òÀÜcheckedòÀÝ. Note: Data can also be transferred directly to cass-01-per.it.csiro.au
Data recorded by ATNF telescopes for RadioAstron experiments go to the subdirectory transfers/2016/radioastron/{experiment}
at Pawsey. The script for moving RadioAstron data to the correct subdirectory structure can be accessed using alias movera='. ~cphillips/bin/run_movera.sh
'
The data are transferred to ASC for correlation using the tsunami transfer protocol. Set up the tsunami server and let Dmitri know [TO BE UPDATED].
Q: Are the notes below still relevant?
As root:
> apt-get install build-essential bzip2 autoconf libxml-parser-perl xinetd openssl telnet
> adduser arcs
> mkdir /opt/globus-4.2.1
> chown arcs:arcs /opt/globus-4.2.1
Add the following to the appropriate interface in /etc/network/interfaces
up route add -host pbstore.ivec.org gw 202.158.222.113 down route del -host pbstore.ivec.org gw 202.158.222.113 up route add -host arcs-df.ivec.org gw 202.158.222.113 down route del -host arcs-df.ivec.org gw 202.158.222.113
Copy .ssh/authorized_keys to ~arcs/.ssh/ from arcs@cavsi2