NFS using too much memory on diskless system

NFS using too much memory on diskless system

[Gareth Glaccum]

Hi,
I have a diskless system built and am having weird issues with the way NFS 
client is working. I am using SLES 9.0 with the stock kernel (2.6.5-7.111-30 
I think from memory).
I have mounted an NFS partition from another machine without any issues. 
However, when I use anything to write to the partition (for example, see 
later) the system uses memory to cache the files as I am mounted over GbE.
However, the machine is not limiting the memory used at all. In fact, it is 
using so much it is crashing the machine or causing other programs to be 
OOM'd.

To recreate the problem, the following command took the machine out in less 
than 3 seconds dd if=/dev/zero of=/mnt/NFSMOUNT bs=1024k count=4000 The 
machine has 4GB of memory and is using around 300Mb for its OS all told.
Is there some way of limiting the NFS cache size like there used to be in 
the 2.4 kernel? (see section B7 of the nfs FAQ I believe)

Any ideas gratefully received,
Gareth

_________________________________________________________________
It's fast, it's easy and it's free. Get MSN Messenger today! 
http://www.msn.co.uk/messenger



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS using too much memory on diskless system

[Bill Rugolsky Jr.]

On Thu, Feb 24, 2005 at 12:19:15PM +0000, Gareth Glaccum wrote:
> To recreate the problem, the following command took the machine out in less 
> than 3 seconds dd if=/dev/zero of=/mnt/NFSMOUNT bs=1024k count=4000 The 
> machine has 4GB of memory and is using around 300Mb for its OS all told.
> Is there some way of limiting the NFS cache size like there used to be in 
> the 2.4 kernel? (see section B7 of the nfs FAQ I believe)
 
There was definitely a known issue with NFS write-throttling back in early 2.6.
Andrew Morton had an "nfs-write-throttling.patch" in the -mm series for a while;
I'm not sure when it was merged.  Perhaps try a recent mainline kernel, or
dig out the patch(es) and see whether they apply to your SuSE kernel.

Regards,

	Bill Rugolsky


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS using too much memory on diskless system

[Gareth Glaccum]

Thanks Bill, I shall investigate these patches as a possible permanent fix.
Gareth

>From: "Bill Rugolsky Jr." <brugolsky@telemetry-investments.com>
> > Is there some way of limiting the NFS cache size like there used to be 
>in
> > the 2.4 kernel? (see section B7 of the nfs FAQ I believe)
>
>There was definitely a known issue with NFS write-throttling back in early 
>2.6.
>Andrew Morton had an "nfs-write-throttling.patch" in the -mm series for a 
>while;
>I'm not sure when it was merged.  Perhaps try a recent mainline kernel, or
>dig out the patch(es) and see whether they apply to your SuSE kernel.
>
>Regards,
>
>	Bill Rugolsky

_________________________________________________________________
Express yourself with cool new emoticons http://www.msn.co.uk/specials/myemo



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS using too much memory on diskless system

[Neil Horman]

Gareth Glaccum wrote:
> Hi,
> I have a diskless system built and am having weird issues with the way 
> NFS client is working. I am using SLES 9.0 with the stock kernel 
> (2.6.5-7.111-30 I think from memory).
> I have mounted an NFS partition from another machine without any issues. 
> However, when I use anything to write to the partition (for example, see 
> later) the system uses memory to cache the files as I am mounted over GbE.
> However, the machine is not limiting the memory used at all. In fact, it 
> is using so much it is crashing the machine or causing other programs to 
> be OOM'd.
> 
> To recreate the problem, the following command took the machine out in 
> less than 3 seconds dd if=/dev/zero of=/mnt/NFSMOUNT bs=1024k count=4000 
> The machine has 4GB of memory and is using around 300Mb for its OS all 
> told.
> Is there some way of limiting the NFS cache size like there used to be 
> in the 2.4 kernel? (see section B7 of the nfs FAQ I believe)
> 
> Any ideas gratefully received,
> Gareth
> 
It sounds to me your kernel has some sort of vm bug, but if you want to 
shrink page cache usage, you may want to play with the dirty_vm_* 
settings in /proc/sys/vm.  I'm still getting used to the 2.6 sysctls, 
but I believe that those settings allow you to specify when your kernel 
decides its under sufficient memory pressure to start moving data back 
to disk.

HTH
Neil
> _________________________________________________________________
> It's fast, it's easy and it's free. Get MSN Messenger today! 
> http://www.msn.co.uk/messenger
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs


-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS using too much memory on diskless system

[Gareth Glaccum]

Hi Neil,
Thanks for the pointer,
a
sysctl -w vm.dirty_ratio=0
appears to have removed the problems, I shall continue to investigate this.
Thanks,
Gareth

>From: Neil Horman <nhorman@redhat.com>
>>Is there some way of limiting the NFS cache size like there used to be in 
>>the 2.4 kernel? (see section B7 of the nfs FAQ I believe)
>>
>>Any ideas gratefully received,
>>Gareth
>>
>It sounds to me your kernel has some sort of vm bug, but if you want to 
>shrink page cache usage, you may want to play with the dirty_vm_* settings 
>in /proc/sys/vm.  I'm still getting used to the 2.6 sysctls, but I believe 
>that those settings allow you to specify when your kernel decides its under 
>sufficient memory pressure to start moving data back to disk.
>
>HTH
>Neil
>>_________________________________________________________________

_________________________________________________________________
Express yourself with cool new emoticons http://www.msn.co.uk/specials/myemo



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: NFS using too much memory on diskless system

[Neil Horman]

Gareth Glaccum wrote:
> Hi Neil,
> Thanks for the pointer,
> a
> sysctl -w vm.dirty_ratio=0
> appears to have removed the problems, I shall continue to investigate this.
> Thanks,
> Gareth
> 
Glad to hear it worked :)
Neil
>> From: Neil Horman <nhorman@redhat.com>
>>
>>> Is there some way of limiting the NFS cache size like there used to 
>>> be in the 2.4 kernel? (see section B7 of the nfs FAQ I believe)
>>>
>>> Any ideas gratefully received,
>>> Gareth
>>>
>> It sounds to me your kernel has some sort of vm bug, but if you want 
>> to shrink page cache usage, you may want to play with the dirty_vm_* 
>> settings in /proc/sys/vm.  I'm still getting used to the 2.6 sysctls, 
>> but I believe that those settings allow you to specify when your 
>> kernel decides its under sufficient memory pressure to start moving 
>> data back to disk.
>>
>> HTH
>> Neil
>>
>>> _________________________________________________________________
> 
> 
> _________________________________________________________________
> Express yourself with cool new emoticons 
> http://www.msn.co.uk/specials/myemo
> 
> 
> 
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> NFS maillist  -  NFS@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs


-- 
/***************************************************
  *Neil Horman
  *Software Engineer
  *Red Hat, Inc.
  *nhorman@redhat.com
  *gpg keyid: 1024D / 0x92A74FA1
  *http://pgp.mit.edu
  ***************************************************/


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs