Deleting data from Parkes DFB data disks

This procedure checks that data files are present on all of a) the Epping data archive, b), the Parkes DLT backup and c) the Parkes data disk. Files which satisfy this may be deleted from the Parkes data disk. Currently this procedure is complete for PDFB2, PDFB3 and PDFB4.

First log on to an Epping linux machine and get a listing of files on the Epping archive, e.g.:

%ssh pulsar@tycho

%TAsummary.csh PDFB3

This produces a file SF_epp_dfb3.dat. The procedure is similar for PDFB2 and PDFB4.

Log into a Parkes linux machine, e.g. lagavulin, and copy the file to ~pulsar:

%ssh pulsar@lagavulin

%scp tycho:./SF_epp_dfb3.dat .

In a separate window, log into the correlator computer and cd to the data disk:

%ssh corr@pkccc3

>cd /data1/PDFB3_1

First get rid of short files which have no data:

>ll s1* | awk '{if($5<105000)print "rm -f",$9}' > del.lis

>cat del.lis

and check that it really is a listing of short files (and nothing else!).

Then when satisfied:

>source del.lis

Then get a listing of files on the PDFB3 data disk and which have been backed up to DLT. (The DLT backup is normally done by John Sarkissian using psrmarc on pegasus. Simon Hoyle is also familiar with this procedure - having written the program.) On the pulsar@lagavulin window:

%TAsummary.csh PDFB3

This produces files SF_pks_dfb3.dat and SF_DLT_dfb3.dat.

Find those files on the data disk which have been backed up:

%tranScan -L SF_pks_dfb3.dat -R SF_DLT_dfb3.dat -c

%mv matches.dat SF_pksbk_dfb3.dat

Find files which are at both Parkes and Epping and which have been backed up (i.e., which are ripe for deletion!)

%tranScan -L SF_pksbk_dfb3.dat -R SF_epp_dfb3.dat -c

%mv matches.dat SF_del_dfb3.dat

Before proceeding CHECK THAT SF_del_dfb3.dat IS WHAT YOU THINK IT IS!! For example, head it and tail it and pick a few random file names, and grep for these in SF_epp_dfb3.dat and SF_DLT_dfb3.dat. ls them on tycho to make sure they're actually there. wc -l SF_pks_dfb3.dat SF_del_dfb3.dat to see that the number of files to be deleted is less than the total number of files on the data disk.

When satisfied all is well, proceed to the deletion step. In the corr@pkccc3 window:

%scp pulsar@lagavulin:SF_del_dfb3.dat .

%df -h .

and take note of how much space is free.

%awk '{print $1}' < SF_del_dfb3.dat | xargs -n 100 echo rm

as a final check; then take a deep breath and ...

%awk '{print $1}' < SF_del_dfb3.dat | xargs -n 100 rm

and wait for the deletion to complete.

%df -h .

to see how much space is now available.