[Lustre-devel] Lustre HSM - some talking points.

[Colin Ngam <Colin.Ngam@Sun.COM>]

Vicky White wrote:
>
>> 6.  No namepace.  No namespace.  Lustre pathnames can be stored as 
>> Extended
>> Attributes.
>
>
> I realize you are talking about SAMQFS, but do we want to keep the 
> design consistent with what we do for hpss?
>
> HPSS will support Extended Attributes in release 7.2, to be available 
> September 2009, but it will be expensive to use these EAs for 
> pathnames, though it can be done.   The current design is to specify 
> EAs in XML, and for the most efficient storage of the EA in the 
> database, the containing XML document needs to be 512 bytes or fewer 
> so that it can be stored inline.  Larger XML objects will have to be 
> stored externally as LOBs (large objects), which will make queries 
> cost more.
>
> So we need to think about what that cost will be when we are 
> considering repositories with millions or billions of files.
>
> Vicky

Hi Vicky,

I do not see why we need to query these EAs in normal operation.  These 
EAs will only be accessed when we need to perform Ultimate Disaster 
Recovery - when you have lost all data on disks and all you have are tapes.

I was thinking about XML - but it is "opaque" to Object SAMQFS so, it is 
up to the Lustre side.  Whatever it is, the Applications - 
Lustre-Restore for example, is the one that has to understand the 
format.  I am not a Tar Header expert - but I assume that these EAs can 
go with the file in the tar ball.  I do not expect to keep any of it on 
line on disk cache, on the SAMFS side.  I see no reason.

With respect to whether it should be consistent with HPSS - I would say 
if that is all we need and it is sufficient - why not.  Otherwise, let's 
make it better than HPSS.  It must be in SUN's best interest to sell SAMQFS?

I do apologize Vicky, are you a SUN employee?

Thanks.

colin