[Lustre-devel] Role of the Metadata Server during File I/O

[Lustre-devel] Role of the Metadata Server during File I/O

[office at hailoo.com]

One of Lustre's key innovations seems to be the separation of metadata from 
file data.  According to Sun, "Lustre file operations bypass the MetaData 
server completely and fully utilize the parallel data paths to all OSSs in 
the cluster."  (See 
http://www.sun.com/software/products/lustre/features.xml) 

However, can this really be the case? 

In POSIX-compliant file I/O, a call to write() that starts at an offset 
which is greater than the file size must write zeroes to disk to make up for 
the missing space.  So if you have a file size of 0 bytes, and then you 
write a single byte at offset 10, bytes 0 through 9 of the file will contain 
zeros. 

But if a file on a Lustre system is striped across multiple OSTs, how does 
Lustre avoid communicating with the Metadata Server at every write 
operation?  Consider the following scenario: 

You have 4 OSTs and you create a new file and stripe it across all 4 OSTs, 
and you set the stripe size to 4 bytes.  (I know that is too small but I'm 
just keeping this simple.) 

Now, suppose you call write() and write 1 byte to the file at offset 5.  
Lustre must now write 4 zero bytes on the first OST, and 1 non-zero byte on 
the second OST.  But in order to know that it is necessary to write zeroes 
to the first OST, the client would need access to global information about 
the total size of the file.  Therefore, wouldn't it need to check with the 
Metadata Server to determine the total file size before every call to 
write()? 

Any information anyone can provide me on the implementation 
details/strategies used here would be greatly appreciated. 

 -Charles Salvia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20090810/dd95b56f/attachment.htm>

[Lustre-devel] Role of the Metadata Server during File I/O

[Eric Barton]

The file size is computed from the sizes of all its stripes to avoid

unnecessary communication with the MDS while a file is being

extended.

 

Obviously the next question is ?does that mean you have to consult all

the OSTs whenever you stat a file?? and currently the answer is yes.

However we are developing a new feature to allow Lustre to cache the

sizes of inactive files on the MDS to give us the best of both worlds.

Cheers, 
                   Eric 

 

From: lustre-devel-bounces@lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of office at hailoo.com
Sent: 11 August 2009 12:32 AM
To: lustre-devel at lists.lustre.org
Subject: [Lustre-devel] Role of the Metadata Server during File I/O

 

One of Lustre's key innovations seems to be the separation of metadata from file data.  According to Sun, "Lustre file operations bypass the MetaData server completely and fully utilize the parallel data paths to all OSSs in the cluster."  (See http://www.sun.com/software/products/lustre/features.xml) 

However, can this really be the case? 

In POSIX-compliant file I/O, a call to write() that starts at an offset which is greater than the file size must write zeroes to disk to make up for the missing space.  So if you have a file size of 0 bytes, and then you write a single byte at offset 10, bytes 0 through 9 of the file will contain zeros.  

But if a file on a Lustre system is striped across multiple OSTs, how does Lustre avoid communicating with the Metadata Server at every write operation?  Consider the following scenario: 

You have 4 OSTs and you create a new file and stripe it across all 4 OSTs, and you set the stripe size to 4 bytes.  (I know that is too small but I'm just keeping this simple.) 

Now, suppose you call write() and write 1 byte to the file at offset 5.  Lustre must now write 4 zero bytes on the first OST, and 1 non-zero byte on the second OST.  But in order to know that it is necessary to write zeroes to the first OST, the client would need access to global information about the total size of the file.  Therefore, wouldn't it need to check with the Metadata Server to determine the total file size before every call to write()? 

Any information anyone can provide me on the implementation details/strategies used here would be greatly appreciated. 

-Charles Salvia 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20090811/38e6b897/attachment.htm>

[Lustre-devel] Role of the Metadata Server during File I/O

[office at hailoo.com]

Thanks for the information.  Doesn't this entail that for every call to 
write() involving a striped file, Lustre must necessarily consult all OSSs, 
in order to determine 1) the file size and 2) if the current write operation 
will extend the file? 

If so, then I think it would be advisable for Lustre clients working with 
striped files to avoid small unbuffered file writes, since every call to 
write() requires a lot of network overhead. 

Eric Barton writes: 

> The file size is computed from the sizes of all its stripes to avoid 
> 
> unnecessary communication with the MDS while a file is being 
> 
> extended. 
> 
>   
> 
> Obviously the next question is ?does that mean you have to consult all 
> 
> the OSTs whenever you stat a file?? and currently the answer is yes. 
> 
> However we are developing a new feature to allow Lustre to cache the 
> 
> sizes of inactive files on the MDS to give us the best of both worlds. 
> 
> Cheers, 
>                    Eric  
> 
>   
> 
> From: lustre-devel-bounces at lists.lustre.org [mailto:lustre-devel-bounces at lists.lustre.org] On Behalf Of office at hailoo.com
> Sent: 11 August 2009 12:32 AM
> To: lustre-devel at lists.lustre.org
> Subject: [Lustre-devel] Role of the Metadata Server during File I/O 
> 
>   
> 
> One of Lustre's key innovations seems to be the separation of metadata from file data.  According to Sun, "Lustre file operations bypass the MetaData server completely and fully utilize the parallel data paths to all OSSs in the cluster."  (See http://www.sun.com/software/products/lustre/features.xml)  
> 
> However, can this really be the case?  
> 
> In POSIX-compliant file I/O, a call to write() that starts at an offset which is greater than the file size must write zeroes to disk to make up for the missing space.  So if you have a file size of 0 bytes, and then you write a single byte at offset 10, bytes 0 through 9 of the file will contain zeros.   
> 
> But if a file on a Lustre system is striped across multiple OSTs, how does Lustre avoid communicating with the Metadata Server at every write operation?  Consider the following scenario:  
> 
> You have 4 OSTs and you create a new file and stripe it across all 4 OSTs, and you set the stripe size to 4 bytes.  (I know that is too small but I'm just keeping this simple.)  
> 
> Now, suppose you call write() and write 1 byte to the file at offset 5.  Lustre must now write 4 zero bytes on the first OST, and 1 non-zero byte on the second OST.  But in order to know that it is necessary to write zeroes to the first OST, the client would need access to global information about the total size of the file.  Therefore, wouldn't it need to check with the Metadata Server to determine the total file size before every call to write()?  
> 
> Any information anyone can provide me on the implementation details/strategies used here would be greatly appreciated.  
> 
> -Charles Salvia  
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20090811/9c3100c1/attachment.htm>

[Lustre-devel] Role of the Metadata Server during File I/O

[Oleg Drokin]

Hello!

On Aug 11, 2009, at 1:24 PM, office at hailoo.com wrote:

> Thanks for the information.  Doesn't this entail that for every call  
> to write() involving a striped file, Lustre must necessarily consult  
> all OSSs, in order to determine 1) the file size and 2) if the  
> current write operation will extend the file?

Why does write need to know the entire file size? We only care if we  
are extending currently accessed stripe. We know this the moment we  
obtained the lock on
the stripe region we are interested in.
> If so, then I think it would be advisable for Lustre clients working  
> with striped files to avoid small unbuffered file writes, since  
> every call to write() requires a lot of network overhead.
>

It is indeed not advisable to use small unbuffered writes (though we  
relieved the situation somewhat), but not because of the network  
overhead, rather the lustre
code is not exactly light-weight doing thelock processing and lookup  
and stuff.
I remember when LLNL performed experiments, and 4 byte unbuffered  
writes were 100 times slower than 1M writes.
But since then we improved the situation and I am no longer sure how  
these compare.

Bye,
     Oleg

[Lustre-devel] Role of the Metadata Server during File I/O

[office at hailoo.com]

Oleg Drokin writes: 

> Hello! 
> 
> On Aug 11, 2009, at 1:24 PM, office at hailoo.com wrote: 
> 
>> Thanks for the information.  Doesn't this entail that for every call  to 
>> write() involving a striped file, Lustre must necessarily consult  all 
>> OSSs, in order to determine 1) the file size and 2) if the  current write 
>> operation will extend the file?
> 
> Why does write need to know the entire file size? We only care if we  are 
> extending currently accessed stripe. We know this the moment we  obtained 
> the lock on
> the stripe region we are interested in.

Consider the following situation: 

You have 4 OSTs and you create a file striped across all 4 OSTs, and you set 
the stripe size to 4 bytes.  (Obviously that is too small, but I just want 
to keep this simple.) 

The file is created and it starts out as a 0 byte file.  Now, suppose you 
write one byte to offset 5.  So now Lustre has to write one byte to the 
second OST.  But, in POSIX compliant file I/O, if you write to an offset 
that is greater than the file size, the file system must write zeros to the 
disk to fill the gap between the old end of the file and the offset.  So, in 
the case of Lustre, the system must not only write a single byte to the 
second OST, it also must write 4 zero-bytes to the first OST.  But in order 
to even know that is has to do this, wouldn't Lustre need to know the entire 
file size?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lustre.org/pipermail/lustre-devel-lustre.org/attachments/20090813/d2a7cfd9/attachment.htm>

[Lustre-devel] Role of the Metadata Server during File I/O

[Oleg Drokin]

Hello!

On Aug 13, 2009, at 2:43 PM, office at hailoo.com wrote:
>>> Thanks for the information.  Doesn't this entail that for every  
>>> call  to write() involving a striped file, Lustre must necessarily  
>>> consult  all OSSs, in order to determine 1) the file size and 2)  
>>> if the  current write operation will extend the file?
>> Why does write need to know the entire file size? We only care if  
>> we  are extending currently accessed stripe. We know this the  
>> moment we  obtained the lock on
>> the stripe region we are interested in.
>>
> Consider the following situation:
> You have 4 OSTs and you create a file striped across all 4 OSTs, and  
> you set the stripe size to 4 bytes.  (Obviously that is too small,  
> but I just want to keep this simple.)
> The file is created and it starts out as a 0 byte file.  Now,  
> suppose you write one byte to offset 5.  So now Lustre has to write  
> one byte to the second OST.  But, in POSIX compliant file I/O, if  
> you write to an offset that is greater than the file size, the file  
> system must write zeros to the disk to fill the gap between the old  
> end of the file
>

This is a misconception. Nowhere does it says we must write zeroes to  
disk. We are fine as long as subsequent reads would get zeroes.
> and the offset.  So, in the case of Lustre, the system must not only  
> write a single byte to the second OST, it also must write 4 zero- 
> bytes to the first OST.  But in order to even know that is has to do  
> this, wouldn't Lustre need to know the entire file size?
>

No.
What happens is we write the data and necessary zeroes to 2nd OST (to  
fill the page where this one byte fits) and size on 1st ost remains 0,
lustre is smart enough to fill the next read with zeroes when it  
encounters access to that part of the file.
The file size on the other hand would be properly composed because we  
look at file size at every OST.

Bye,
     Oleg