HTAR : save/restore private directory (large set of small files) in HPSS
For saving all files in directory 'aaa' on disk to your HPSS target bbb/ccc.tar and keep log file in ~/0x/ddd.log1 (needed if you want to kill/quit the window) do:
cd aaa
du -hs .
htar -cf bbb/ccc.tar file1 file2 fileX* ... >& ~/0x/ddd.log1 &
(wait until job is done, for 50GB make take severa hours)
hsi (goes directly to HPSS, be careful!)
?ls -l bbb/
?quit (exit hsi)
You should see 2 files : ccc.tar & ccc.tar.idx, the size of file ccc.tar should be close to the size of your oryginal directory.
The .tar file shows up immediately but sinking is NOT finished until you see .idx file.
The log file should contain the string: HTAR: HTAR SUCCESSFUL , no warnings.
However the most reliable way to verify the storage did not failed is to:
retrieving files from HPSS to disk directory eee do:
mkdir eee
cd eee
htar -xf bbb/ccc.tar >& ~/0x/ddd.log2 &
http://drupal.star.bnl.gov/STAR/comp/sofi/hpss/htar
Known problems:
WARNING: htar_PreallocateSpace: HPSS OUT-OF-SPACE error preallocating
**** bytes for file=[/home/salur/myfile.tar]
This is a directory with 350G. Do I have to divide my files into smaller.
there is a 60GB max file size limit [when creating tarballs], Another
limit to be aware of, however, is an 8GB limit on member files
Mass restoring of ~100GB files from HPSS using Data Carusel
Prepare input file xxx.hpss with list of source +destination paths for all files you need, one line per file
Execute the command (at any location at rcas6nnn)
hpss_user.pl -f xxx.hpss
Wait 10-1000 minutes fro the files to show up at your destination (if all paths are correct)