Odd swapping issue

Odd swapping issue

[Rafal Wijata]

It happened twice recently(within one week), therefore I'm posting here.

32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon

My machine went sluggish. Tried to debug it, and observed, that free 
ram(the unused one) was increasing while swap usage was increasing as 
well. It stopped after whole swap partition was used. Then it came back 
normal. Required processes were brought back to ram and server become 
responsible again.

Now the most strange thing is that there was 5G free(unused) ram, during 
those few minutes it growed to 12G! In the very same time swap usage 
increased from ~4G to 12G.
This is irrational to me. Can anybody explain? How to prevent?

vm.swappiness was decreased to 30, to avoid excessive swap usage, but no 
avail.

It's application and nfs server/client if it matters. During that 
"memory sweep" I observed many nfsd processes(kernel threads?) in D state.

Re: Odd swapping issue

[Peter Zijlstra]

On Wed, 2008-07-23 at 18:53 +0200, Rafal Wijata wrote:
> It happened twice recently(within one week), therefore I'm posting here.
> 
> 32bit Fedora7, 2.6.23.17-88.fc7PAE, 16G ram, 12G swap, 2*dualXeon

16g ram on a 32bit kernel is stretching it... may I suggest running a
64bit kernel?

Re: Odd swapping issue

[David Newall]

Peter Zijlstra wrote:
> 16g ram on a 32bit kernel is stretching it... may I suggest running a
> 64bit kernel?
>   

Is it?  Do we expect that behaviour with so much RAM?

Re: Odd swapping issue

[Dan Noé]

David Newall wrote:
> Peter Zijlstra wrote:
>> 16g ram on a 32bit kernel is stretching it... may I suggest running a
>> 64bit kernel?
>>   
> 
> Is it?  Do we expect that behaviour with so much RAM?

The memory must be mapped into the 4GB (32-bit) virtual address space to
be used.  Having this much memory on a system with 32 bit pointers leads
to a lot of VM overhead and strange behavior.

Cheers,
Dan

-- 
                    /--------------- - -  -  -   -   -
                    |  Dan Noé
                    |  http://isomerica.net/~dpn/

Re: Odd swapping issue

[Peter Zijlstra]

On Thu, 2008-07-24 at 06:47 +0930, David Newall wrote:
> Peter Zijlstra wrote:
> > 16g ram on a 32bit kernel is stretching it... may I suggest running a
> > 64bit kernel?
> >   
> 
> Is it?  Do we expect that behaviour with so much RAM?

I expect your 32bit machine to go wonky after say 8gb,.. maybe even
earlier for some workloads.

The thing is, 32bit machines have less than 1gb of kernel memory - the
more memory you have the less of that remains for system duties since
such things as the page frame array grow with the number of physical
pages present.

Also, all such things as buffer cache reside in that tiny space. At some
point the kernel itself is just going to run out of memory trying to
manage your machine.

Re: Odd swapping issue

[KOSAKI Motohiro]

Hi Rafal

> vm.swappiness was decreased to 30, to avoid excessive swap usage, but no 
> avail.
> It's application and nfs server/client if it matters. During that 
> "memory sweep" I observed many nfsd processes(kernel threads?) in D state.

I think no related.

nfsd always remain D state.
because if nfsd waiting to new packet, ps (or other tool) display "D".

Re: Odd swapping issue

[Rafal Wijata (NULL)]

OK, things are little more clear now I suppose.

I tried (just for fun) echo 8 > /proc/sys/vm/nr_hugepages
and it happened again, and no hugepages were allocated of course.

But this brought the following to my mind.

Since it's nfs server/client, and the nfs is rather heavily utilized(at 
least sometimes), maybe nfsd tasks were unable to find long enough lowmem 
chunk. How can we get such ram? By sweeping pages out to swap, isn't it?

Now the question is, if Linux actually has such feature like blind swapping 
out if lowmem is too fragmented to find requested size?
Can anybody confirm or negate?

Next, I was thinking then I could run 64bit system, but I don't want to 
reinstall it. Anybody knows if it's possible to have installed 32bit OS, but 
run 64bit kernel alone(no 64bit glibc or anything else)?

Thanx in advance.

Re: Odd swapping issue

[Peter Zijlstra]

On Thu, 2008-07-24 at 15:45 +0200, Rafal Wijata (NULL) wrote:
> OK, things are little more clear now I suppose.
> 
> I tried (just for fun) echo 8 > /proc/sys/vm/nr_hugepages
> and it happened again, and no hugepages were allocated of course.
> 
> But this brought the following to my mind.
> 
> Since it's nfs server/client, and the nfs is rather heavily utilized(at 
> least sometimes), maybe nfsd tasks were unable to find long enough lowmem 
> chunk. How can we get such ram? By sweeping pages out to swap, isn't it?
> 
> Now the question is, if Linux actually has such feature like blind swapping 
> out if lowmem is too fragmented to find requested size?
> Can anybody confirm or negate?

Well, not blind - we have lumpy reclaim that tries to focus on the right
spot for large order allocations. However there are a myrad of
conditions that can make it impossible to reclaim certain pages,
therefore defeating this strategy.

> Next, I was thinking then I could run 64bit system, but I don't want to 
> reinstall it. Anybody knows if it's possible to have installed 32bit OS, but 
> run 64bit kernel alone(no 64bit glibc or anything else)?

Yes running as I type ;-)