[Lustre-devel] SAM-QFS, ADM, and Lustre HSM

[Nathaniel Rutman <Nathan.Rutman@Sun.COM>]

Andreas Dilger wrote:
> On Jan 23, 2009  10:46 -0600, Harriet G. Coverston wrote:
>   
>> SAM supports classification policy rules for files  -- (1) number of  
>> copies, up to 4 (2) where to put the copies  on which vsn pools  -  
>> disk and/or tape, local and/or remote) (3) when to make the copies  
>> (time based archiving). You specify the policy in the archiver.cmd  
>> file. You can group files for a policy rule by pathname, owner, group,  
>> size, wildcard, and access time.
>>     
My point about this is that files will be stored using the FID as the 
file name, so name-based policies at the copytool level are worthless.   
Unless we a.) add the path/filename back to the file (EA?), and b.) 
modify the SAM policy engine to use the "real" path/filename instead of 
the FID.
>> This brings up the question of restore. In case of a Lustre disk  
>> failure, how are you going to restore your Lustre file system?
>>     
>
> ...
>   
> - since the archive does not contain a copy of the namespace (it only
>   has 128-bit FIDs as identifiers for the file) we would need to make
>   a separate backup of the MDS filesystem (which is all namespace).
>   There are already several mechanisms to do this, either using the
>   ext2 "dump" program to read from the raw device, or to make an LVM
>   snapshot and use e.g. tar to make a filesystem-level backup.  Both
>   of these need to include a backup of the extended attributes.
>   
Or include the path/filename in each file, and the restore process uses 
this to repopulate the filesystem.
>   
>> Agree. I don't see any SAM-QFS code changes required. The Lustre  
>> copytool will write to HPSS using the HPSS APIs and write to SAM-QFS  
>> with a ftp or pftp interface. This is minimum changes.
>>     
>
> We weren't thinking of using an FTP interface to SAM, though I guess
> this is possible.  Rather we were thinking of just mounting both QFS
> and Lustre on a Linux client and using "cp" or equivalent tool.
>   
Harriet already knew this, she just forgot :)