Re: Re: [PATCH] Smooth out NFS client writeback

Re: Re: [PATCH] Smooth out NFS client writeback

[Peter Staubach]

Shantanu Goel wrote:

>Hi Trond,
>
>Attached is the long delayed revised version of the
>writeback smoothing patch this time against
>2.6.12-mm2.  I have omitted the commit w/range and
>mmap writeback from this one.  If this one is deemed
>acceptable for inclusion I'll post the other 2 later. 
>The commit w/range really should be restored as it
>makes quite a difference against Solaris NFS servers
>with regular disks.  I observed a difference of 2-3
>MB/s under sustained writes.  It makes no difference
>with the Linux NFS server since it ignores the range.
>

On Solaris, at least with UFS as the underlying file system, the COMMIT
operations are processed by looking through the entire cached page list
or by doing page lookup operations on each individual page.  If the entire
file is specified, ie. len = 0, then the page list is walked.  If a range
is specified, then just the pages within the range are looked up.

Specifying the range can result in significantly less CPU overhead on the
server.  This is why the NFSv3 COMMIT operation has a range which can be
specified...  :-)

       ps


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: [PATCH] Smooth out NFS client writeback

[Lever, Charles]

hi peter-

> On Solaris, at least with UFS as the underlying file system,=20
> the COMMIT
> operations are processed by looking through the entire cached=20
> page list
> or by doing page lookup operations on each individual page. =20
> If the entire
> file is specified, ie. len =3D 0, then the page list is walked.=20
>  If a range
> is specified, then just the pages within the range are looked up.
>=20
> Specifying the range can result in significantly less CPU=20
> overhead on the
> server.  This is why the NFSv3 COMMIT operation has a range=20
> which can be specified...  :-)

a server CPU inefficiency hardly qualifies as a client bug.  in the most
common cases where the client is creating and writing to a file, then
closing with a COMMIT(0,0) request, the server should be changed to
behave in a more efficient manner.

in other words, i think the client should optimize the number of
requests on the wire, and the server should optimize for using its CPU
and disks most efficiently.  i haven't looked closely at shantanu's
patch, but i'm a little leary of the change if it means more wire
operations are generated than before.


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: [PATCH] Smooth out NFS client writeback

[Peter Staubach]

Lever, Charles wrote:

>hi peter-
>
>  
>
>>On Solaris, at least with UFS as the underlying file system, 
>>the COMMIT
>>operations are processed by looking through the entire cached 
>>page list
>>or by doing page lookup operations on each individual page.  
>>If the entire
>>file is specified, ie. len = 0, then the page list is walked. 
>> If a range
>>is specified, then just the pages within the range are looked up.
>>
>>Specifying the range can result in significantly less CPU 
>>overhead on the
>>server.  This is why the NFSv3 COMMIT operation has a range 
>>which can be specified...  :-)
>>    
>>
>
>a server CPU inefficiency hardly qualifies as a client bug.  in the most
>common cases where the client is creating and writing to a file, then
>closing with a COMMIT(0,0) request, the server should be changed to
>behave in a more efficient manner.
>
>in other words, i think the client should optimize the number of
>requests on the wire, and the server should optimize for using its CPU
>and disks most efficiently.  i haven't looked closely at shantanu's
>patch, but i'm a little leary of the change if it means more wire
>operations are generated than before.
>  
>

I wouldn't claim that this is a client side bug either.  I would claim that
it is an opportunity, for very little cost, to help a server to perform
better and this makes the client, and NFS in general, look better.

I don't think that there will be more over the wire operations generated
when using the range versus always specifying the entire file.  Typically,
COMMITs are done for a range of the file or for the entire file at close.
The client typically needs to know what data is marked as needing to be
committed anyway, so it isn't very hard to figure out what the range that
needs to be committed is at the same time.  I would claim that a client,
which simply issues a blanket COMMIT(0,0), without already having gathered
up the buffers/pages/whatever that need committing, is broken.  It will be
unsafe because it will be subject to races like more data getting written
with UNSTABLE while the COMMIT is happening.  This new data may or may not
have been committed by the COMMIT.

Bottom line for me -- if the client can do something to help the server to
help the client and it is an overall win, then I think that it should do 
so...

    Thanx...

       ps


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: Re: [PATCH] Smooth out NFS client writeback

[Lever, Charles]

> I don't think that there will be more over the wire=20
> operations generated
> when using the range versus always specifying the entire=20
> file.

committing the whole file at once is simply more efficient from a
network perspective when an application is performing random writes
(unless it is using O_SYNC).  there are some cases where it won't make a
difference whether a range or a whole file commit is used, but i think
it would be really hard to figure out a client-side heuristic to decide
which is better.

> I would claim that a client,
> which simply issues a blanket COMMIT(0,0), without already=20
> having gathered
> up the buffers/pages/whatever that need committing, is=20
> broken.  It will be
> unsafe because it will be subject to races like more data=20
> getting written
> with UNSTABLE while the COMMIT is happening.  This new data=20
> may or may not
> have been committed by the COMMIT.

the linux client already keeps track of the order of writes and commits
well enough that this isn't an issue.

> Bottom line for me -- if the client can do something to help=20
> the server to
> help the client and it is an overall win, then I think that=20
> it should do so...

we're talking about potentially adding complexity to the client and
increasing the number of write and commit operations on the wire, which
could have a negative performance impact in other environments (think
WAN).  all this to optimize a particular workload against a particular
server implementation.

i'm not saying this type of work shouldn't be explored, but as we
consider the change, we should be very careful especially since we don't
have adequate performance regression test coverage yet.


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: [PATCH] Smooth out NFS client writeback

[Peter Staubach]

Lever, Charles wrote:

>>I don't think that there will be more over the wire 
>>operations generated
>>when using the range versus always specifying the entire 
>>file.
>>    
>>
>
>committing the whole file at once is simply more efficient from a
>network perspective when an application is performing random writes
>(unless it is using O_SYNC).  there are some cases where it won't make a
>difference whether a range or a whole file commit is used, but i think
>it would be really hard to figure out a client-side heuristic to decide
>which is better.
>
>  
>

The network perspective is important, to be sure.  However, so is disk
i/o.

Having done an implementation, it isn't so hard to develop the heuristics,
actually.

Please keep in mind that COMMIT operations are very expensive.  They are
to be avoided whenever possible.  Clients should delay as long as possible
to commit data because they may be able to avoid committing it at all.
These benefits can include data which is repeatedly overwritten, small
writes to the same file system block on the server, temporary files, etc.
The list goes on and on.  Not going to stable storage on the server is a
huge win in performance all that way around.

>>I would claim that a client,
>>which simply issues a blanket COMMIT(0,0), without already 
>>having gathered
>>up the buffers/pages/whatever that need committing, is 
>>broken.  It will be
>>unsafe because it will be subject to races like more data 
>>getting written
>>with UNSTABLE while the COMMIT is happening.  This new data 
>>may or may not
>>have been committed by the COMMIT.
>>    
>>
>
>the linux client already keeps track of the order of writes and commits
>well enough that this isn't an issue.
>
>  
>

Good to hear!

>>Bottom line for me -- if the client can do something to help 
>>the server to
>>help the client and it is an overall win, then I think that 
>>it should do so...
>>    
>>
>
>we're talking about potentially adding complexity to the client and
>increasing the number of write and commit operations on the wire, which
>could have a negative performance impact in other environments (think
>WAN).  all this to optimize a particular workload against a particular
>server implementation.
>
>i'm not saying this type of work shouldn't be explored, but as we
>consider the change, we should be very careful especially since we don't
>have adequate performance regression test coverage yet.
>  
>

I think that there may be more server implementations out there that
could benefit from the client strategy than just the Solaris with UFS
NFS server.  I would guess that most, if not all, would benefit.  I
even suspect that we could make the Linux server go faster if it paid
attention to the byte range specified, instead of always sync'ing the
entire file.  The cost of looking at clean pages/buffers is small, but
adds up quickly...  :-)

       ps


-------------------------------------------------------
SF.Net email is sponsored by: Discover Easy Linux Migration Strategies
from IBM. Find simple to follow Roadmaps, straightforward articles,
informative Webcasts and more! Get everything you need to get up to
speed, fast. http://ads.osdn.com/?ad_id=7477&alloc_id=16492&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Re: [PATCH] Smooth out NFS client writeback

[Shantanu Goel]

> Lever, Charles wrote:
> >committing the whole file at once is simply more
> efficient from a
> >network perspective when an application is
> performing random writes
> >(unless it is using O_SYNC).  there are some cases
> where it won't make a
> >difference whether a range or a whole file commit
> is used, but i think
> >it would be really hard to figure out a client-side
> heuristic to decide
> >which is better.

The patch I posted is actually much better in terms of
the # commit requests it issues compared to the stock
client.  There are some #'s for iozone -r4k -s256m -c:

Proc            Stock         Patch
write           16486         16384
commit            518            80

Here are the numbers for write throughput as seen by
the children in KB/s in the iozone test above to see
the difference that commit range makes.

Stock    Patch w/o commit   Patch w/commit
7064     6038               10226    =20

When I get a chance I'll modify the Linux NFS server
to honour the range as well and see what if any
difference it makes.

Thanks,
Shantanu



	=09
____________________________________________________=20
Yahoo! Sports=20
Rekindle the Rivalries. Sign up for Fantasy Football=20
http://football.fantasysports.yahoo.com


-------------------------------------------------------
This SF.Net email is sponsored by the 'Do More With Dual!' webinar happen=
ing
July 14 at 8am PDT/11am EDT. We invite you to explore the latest in dual
core and dual graphics technology at this free one hour event hosted by H=
P,=20
AMD, and NVIDIA.  To register visit http://www.hp.com/go/dualwebinar
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs