[Lustre-devel] Feed API and Changelog HLD

[Lustre-devel] Feed API and Changelog HLD

[Nathaniel Rutman]

The Feed API is a subset of the Changelog HLD.
Changelog HLD is not complete, but hopefully decently fleshed out at 
this point.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: feed_api.pdf
Type: application/pdf
Size: 93001 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20080201/1b3cf07f/attachment.pdf>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: changelog-hld.pdf
Type: application/pdf
Size: 152806 bytes
Desc: not available
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20080201/1b3cf07f/attachment-0001.pdf>

[Lustre-devel] Feed API and Changelog HLD

[Peter J Braam]

On page 10 of the changelog API: recursive behavior for directories in 
filesets must exclude directories that are mount point of other filesets 
(as implemented for the Path Forward a few years back).

I didn't see where ABORT sizes are discussed, but as I mentioned before 
there are few cases where ditching records is possible.

The storage of the records should be discussed.  The llog headers may 
need changes and probably the API's for canceling records will change to 
indicate what consumer is canceling.

It would be wise to discuss how disk full failures etc are handled in 
some detail.

- peter -

Nathaniel Rutman wrote:
> The Feed API is a subset of the Changelog HLD.
> Changelog HLD is not complete, but hopefully decently fleshed out at 
> this point.
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Lustre-devel mailing list
> Lustre-devel at lists.lustre.org
> http://lists.lustre.org/mailman/listinfo/lustre-devel
>   

[Lustre-devel] Feed API and Changelog HLD

[Peter J Braam]

Nathaniel Rutman wrote:
> Peter J Braam wrote:
>> On page 10 of the changelog API: recursive behavior for directories 
>> in filesets must exclude directories that are mount point of other 
>> filesets (as implemented for the Path Forward a few years back).
>>
>> I didn't see where ABORT sizes are discussed, but as I mentioned 
>> before there are few cases where ditching records is possible.
>>   
> I don't know what you're talking about with ABORT - you mean e.g. HSM 
> should
> abort copying an old file if the file is modified after the copy 
> starts?  Or something else?
>
Hmm, this was my jetlagged head, sorry.  I meant when you want to force 
a truncation of the log (and throw records away), how you do handle 
this?  You need this mechanism, even though it will be used only rarely.

While I am writing this, I also realize that there are some logs that 
may require re-compacting.  This happens for the LRU logs (tracking most 
recent access to files for HSM migration management) I think.  The idea 
for these logs is to (1) append a new record when an object is accessed, 
and (2) cancel the previous record for that object using a pointer to 
that record in the object.  These records can be tiny (obj id / parent 
obj id), but the mechanisms causes llog files to be non-empty with holes.


>> The storage of the records should be discussed.  The llog headers may 
>> need changes and probably the API's for canceling records will change 
>> to indicate what consumer is canceling.
>>
>> It would be wise to discuss how disk full failures etc are handled in 
>> some detail.
>>   
> ok.  Does it seem to be on the right track in general now?
At a 10,000 feet level yes.  My concerns at this point are error 
handling and performance and multiple consumer management. That's not so 
clear yet.

- Peter -