[NFS] Does "sync" cause the FUA bit to be set?

[NFS] Does "sync" cause the FUA bit to be set?

[Frank Steiner]

Hi,

we have a raid controller serving a file system that we export and
mount with NFS. Depending on the performance profile we chose for the 
raid controller, the are huge performance differences on the NFS client 
side between two profiles. Those two profiles both use write cache and
no journaling, but one honors the FUA bit if it is set, and the other 
profile ignores it (there might be more differences, but the manual
doesn't state any others but the FUA bit).

With the profile ignoring the FUA bit, copying or deleting directories
of e.g. 10M with a about 1000 files is factor 5 faster than with the
profile honoring the FUA bit.

We export with the "sync" option. Does that option maybe set the FUA bit
for all write operations on the NFS server?

cu,
Frank


-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[Benny Halevy]

On Jun. 10, 2008, 13:43 +0300, Frank Steiner <fsteiner-mail1-G0GEQqhI7DhYiKXMg8wJIg@public.gmane.org> wrote:
> Hi,
> 
> we have a raid controller serving a file system that we export and
> mount with NFS. Depending on the performance profile we chose for the 
> raid controller, the are huge performance differences on the NFS client 
> side between two profiles. Those two profiles both use write cache and
> no journaling, but one honors the FUA bit if it is set, and the other 
> profile ignores it (there might be more differences, but the manual
> doesn't state any others but the FUA bit).
> 
> With the profile ignoring the FUA bit, copying or deleting directories
> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> profile honoring the FUA bit.
> 
> We export with the "sync" option. Does that option maybe set the FUA bit
> for all write operations on the NFS server?

No, it's not supposed to that for *all* writes. It's just supposed
to honor the client's synchronous writes and commit calls.
However, the directory is synced after each change (you say above that
files are copied or deleted) with the sync flag and this is more
likely to cause the slowdown you see.

Benny

> 
> cu,
> Frank
> 
> 


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[Wendy Cheng]

Frank Steiner wrote:
> With the profile ignoring the FUA bit, copying or deleting directories
> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> profile honoring the FUA bit.
>   
FUA bit is normally combined with write-thru scsi command that bypasses 
storage write cache. I would imagine it needs to well synchronize 
various pieces before issuing this command. It could hurt the 
performance if not done well, particularly for meta data. So your result 
is not surprising.
> We export with the "sync" option. Does that option maybe set the FUA bit
> for all write operations on the NFS server?
>   
It depends on how the filesystem (and its associated disk subsystem) is 
implemented. The "sync" export option itself has a heavy performance 
impact, regardless how FUA bit is handled. Some vendors uses specialized 
HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem 
doesn't have this type of support, you should expect "sync" option runs 
much much slower than "async". It is a choice (or balance) between cost, 
performance, and data reliability.

Don't you have the vendor's support group to go ? Better get this answer 
from the storage vendor directly.

-- Wendy


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[Wendy Cheng]

Benny Halevy wrote:
> On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
>   
>> Frank Steiner wrote:
>>     
>>> With the profile ignoring the FUA bit, copying or deleting directories
>>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
>>> profile honoring the FUA bit.
>>>   
>>>       
>> FUA bit is normally combined with write-thru scsi command that bypasses 
>> storage write cache. I would imagine it needs to well synchronize 
>> various pieces before issuing this command. It could hurt the 
>> performance if not done well, particularly for meta data. So your result 
>> is not surprising.
>>     
>>> We export with the "sync" option. Does that option maybe set the FUA bit
>>> for all write operations on the NFS server?
>>>   
>>>       
>> It depends on how the filesystem (and its associated disk subsystem) is 
>> implemented. The "sync" export option itself has a heavy performance 
>> impact, regardless how FUA bit is handled. Some vendors uses specialized 
>> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem 
>> doesn't have this type of support, you should expect "sync" option runs 
>> much much slower than "async". It is a choice (or balance) between cost, 
>> performance, and data reliability.
>>     
>
> Wendy, I *think* what you have in mind is the sync mount option
> rather than the sync export flag.  The latter just tells the server
> not to cheat and do everything asynchronously.  It should *not*
> have a heavy performance penalty for I/O intensive writes if the
> client is using async writes and commits.
>
>   
No, I didn't get confused ... We can use Linux as an example :) .. check 
out:
http://marc.info/?l=linux-nfs&m=119618886105337&w=2

-- quote

The default export might have been "async", but unless the option "sync"
in /etc/exports was being ignored I was already using "sync". Nevertheless
I will try to change to async and test if it makes a difference.

(one day later: )

I have now tried it and the load on the NFS server is much lower and KDE
logins seem to be reasonably fast now.

-- un-quote


-- Wendy

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[J. Bruce Fields]

On Tue, Jun 10, 2008 at 01:52:08PM -0500, Wendy Cheng wrote:
> Benny Halevy wrote:
> > On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
> >   
> >> Frank Steiner wrote:
> >>     
> >>> With the profile ignoring the FUA bit, copying or deleting directories
> >>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
> >>> profile honoring the FUA bit.
> >>>   
> >>>       
> >> FUA bit is normally combined with write-thru scsi command that bypasses 
> >> storage write cache. I would imagine it needs to well synchronize 
> >> various pieces before issuing this command. It could hurt the 
> >> performance if not done well, particularly for meta data. So your result 
> >> is not surprising.
> >>     
> >>> We export with the "sync" option. Does that option maybe set the FUA bit
> >>> for all write operations on the NFS server?
> >>>   
> >>>       
> >> It depends on how the filesystem (and its associated disk subsystem) is 
> >> implemented. The "sync" export option itself has a heavy performance 
> >> impact, regardless how FUA bit is handled. Some vendors uses specialized 
> >> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem 
> >> doesn't have this type of support, you should expect "sync" option runs 
> >> much much slower than "async". It is a choice (or balance) between cost, 
> >> performance, and data reliability.
> >>     
> >
> > Wendy, I *think* what you have in mind is the sync mount option
> > rather than the sync export flag.  The latter just tells the server
> > not to cheat and do everything asynchronously.  It should *not*
> > have a heavy performance penalty for I/O intensive writes if the
> > client is using async writes and commits.
> >
> >   
> No, I didn't get confused ... We can use Linux as an example :) .. check 
> out:
> http://marc.info/?l=linux-nfs&m=119618886105337&w=2
> 
> -- quote
> 
> The default export might have been "async", but unless the option "sync"
> in /etc/exports was being ignored I was already using "sync". Nevertheless
> I will try to change to async and test if it makes a difference.
> 
> (one day later: )
> 
> I have now tried it and the load on the NFS server is much lower and KDE
> logins seem to be reasonably fast now.

Yeah, the "async" export options is expected to improve performance on a
workload like that (with lots of directory creates/deletes).  But it
does so by violating the promises that the protocol makes to the client.

--b.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[Wendy Cheng]

Benny Halevy wrote:
> On Jun. 10, 2008, 21:52 +0300, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
>   
>> Benny Halevy wrote:
>>     
>>> On Jun. 10, 2008, 18:44 +0300, Wendy Cheng <s.wendy.cheng@gmail.com> wrote:
>>>   
>>>       
>>>> Frank Steiner wrote:
>>>>     
>>>>         
>>>>> With the profile ignoring the FUA bit, copying or deleting directories
>>>>> of e.g. 10M with a about 1000 files is factor 5 faster than with the
>>>>> profile honoring the FUA bit.
>>>>>   
>>>>>       
>>>>>           
>>>> FUA bit is normally combined with write-thru scsi command that bypasses 
>>>> storage write cache. I would imagine it needs to well synchronize 
>>>> various pieces before issuing this command. It could hurt the 
>>>> performance if not done well, particularly for meta data. So your result 
>>>> is not surprising.
>>>>     
>>>>         
>>>>> We export with the "sync" option. Does that option maybe set the FUA bit
>>>>> for all write operations on the NFS server?
>>>>>   
>>>>>       
>>>>>           
>>>> It depends on how the filesystem (and its associated disk subsystem) is 
>>>> implemented. The "sync" export option itself has a heavy performance 
>>>> impact, regardless how FUA bit is handled. Some vendors uses specialized 
>>>> HW (e.g. NVRAM) to alleviate this performance hit. If your filesystem 
>>>> doesn't have this type of support, you should expect "sync" option runs 
>>>> much much slower than "async". It is a choice (or balance) between cost, 
>>>> performance, and data reliability.
>>>>     
>>>>         
>>> Wendy, I *think* what you have in mind is the sync mount option
>>> rather than the sync export flag.  The latter just tells the server
>>> not to cheat and do everything asynchronously.  It should *not*
>>> have a heavy performance penalty for I/O intensive writes if the
>>> client is using async writes and commits.
>>>
>>>   
>>>       
>> No, I didn't get confused ... We can use Linux as an example :) .. check 
>> out:
>> http://marc.info/?l=linux-nfs&m=119618886105337&w=2
>>
>> -- quote
>>
>> The default export might have been "async", but unless the option "sync"
>> in /etc/exports was being ignored I was already using "sync". Nevertheless
>> I will try to change to async and test if it makes a difference.
>>
>> (one day later: )
>>
>> I have now tried it and the load on the NFS server is much lower and KDE
>> logins seem to be reasonably fast now.
>>
>> -- un-quote
>>
>>
>> -- Wendy
>>     
>
> I'm not sure how the problem you referred to is related to the
> one at hand.  what I'm trying to say is that the sync _exports_
> flag has stronger affect on namespace modifying workloads, like
> what I believe Frank's workload might be ... [snip]
The referred thread showed "sync" export has performance impacts on 
Linux server *too*.

-- Wendy


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs

Re: [NFS] Does "sync" cause the FUA bit to be set?

[Frank Steiner]

Thanks to the two of you for the detailed explanations! It did help me
understanding the impacts to my raid controllers profile!

cu,
Frank


-- 
Dipl.-Inform. Frank Steiner   Web:  http://www.bio.ifi.lmu.de/~steiner/
Lehrstuhl f. Bioinformatik    Mail: http://www.bio.ifi.lmu.de/~steiner/m/
LMU, Amalienstr. 17           Phone: +49 89 2180-4049
80333 Muenchen, Germany       Fax:   +49 89 2180-99-4049
* Rekursion kann man erst verstehen, wenn man Rekursion verstanden hat. *

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
_______________________________________________
Please note that nfs@lists.sourceforge.net is being discontinued.
Please subscribe to linux-nfs@vger.kernel.org instead.
    http://vger.kernel.org/vger-lists.html#linux-nfs