kswapd/tg3 issue

kswapd/tg3 issue

[Yaroslav Halchenko]

Dear Kernel People,

Just got a logwatch daily mail which revealed a problem:
[2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20
and a lengthy backtrace with head

,------------------------------------------------------------------------
| [2024412.795212] Call Trace:
| [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
| [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
| [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
| [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
| [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
| [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
| [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
| [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
`---
full dmesg is at
http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg

is that critical? seems to behave ok but...

More details on the system and error in particular can be
http://www.onerussian.com/Linux/bugs/bug.kswapd/

-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        

Re: kswapd/tg3 issue

[Alan]

On Thu, 30 Nov 2006 09:43:55 -0500
Yaroslav Halchenko <yoh@psychology.rutgers.edu> wrote:

> Dear Kernel People,
> 
> Just got a logwatch daily mail which revealed a problem:
> [2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20
> and a lengthy backtrace with head
> 
> ,------------------------------------------------------------------------
> | [2024412.795212] Call Trace:
> | [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
> | [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
> | [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
> | [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
> | [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
> | [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
> | [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
> | [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
> `---
> full dmesg is at
> http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg
> 
> is that critical? seems to behave ok but...

Its tell us that the machine got very very tight on memory, far tighter
than it probably ever should in normal situations. It is harmless of
itself and if you only get the odd one is not a worry.

Re: kswapd/tg3 issue

[Yaroslav Halchenko]

Thank you Alan

Ok - I am adding more memory in my purchasing plan ;-) For now I guess
adding swap space should help, right?

> > >...<
> Its tell us that the machine got very very tight on memory, far tighter
> than it probably ever should in normal situations. It is harmless of
> itself and if you only get the odd one is not a worry.



-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        

Re: kswapd/tg3 issue

[Arjan van de Ven]

On Thu, 2006-11-30 at 10:10 -0500, Yaroslav Halchenko wrote:
> Thank you Alan
> 
> Ok - I am adding more memory in my purchasing plan ;-) For now I guess
> adding swap space should help, right?


actually since this was networking...
you probably should bump the value in

/proc/sys/vm/min_free_kbytes

a bit (like by 50%); that makes the kernel keep a bigger pool free for
emergencies/spikes...

That might be enough already if your system isn't swapping a whole lot.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: kswapd/tg3 issue

[Yaroslav Halchenko]

Thank you Arjan for advice

I had 5746, made it 8619.

Is that a good practice in general to have that value higher for a
server with lots of I/O including networking? (there is a RAID on that
system and 2 bonded gigabit interfaces) Is there any heuristic to decide
on that value ?

On Thu, 30 Nov 2006, Arjan van de Ven wrote:

> >...<

> actually since this was networking...
> you probably should bump the value in
> /proc/sys/vm/min_free_kbytes
> a bit (like by 50%); that makes the kernel keep a bigger pool free for
> emergencies/spikes...
> That might be enough already if your system isn't swapping a whole lot.
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        

Re: kswapd/tg3 issue

[Alan]

On Thu, 30 Nov 2006 10:10:03 -0500
Yaroslav Halchenko <yoh@psychology.rutgers.edu> wrote:

> Thank you Alan
> 
> Ok - I am adding more memory in my purchasing plan ;-) For now I guess
> adding swap space should help, right?

Under heavy network or I/O pressure it may not have time to swap to get
the memory. Thus adding swap won't usually help. Adding RAM may do but
its often not the best answer. Arjan's suggestion should sort it, and -
yes typically boxes with very high I/O and network load need more of a
pool of memory free for immediate use than other systems.

Re: kswapd/tg3 issue

[Avi Kivity]

Alan wrote:
> Under heavy network or I/O pressure it may not have time to swap to get
> the memory. Thus adding swap won't usually help. Adding RAM may do but
> its often not the best answer. Arjan's suggestion should sort it, and -
> yes typically boxes with very high I/O and network load need more of a
> pool of memory free for immediate use than other systems.
>   

It could be nice if the kernel could autotune this, for example by 
raising the free memory goal when memory shortage is detected, and 
lowering it gradually when not.

The sysctl could be a minimum from which this is calculated.

-- 
error compiling committee.c: too many arguments to function

Re: kswapd/tg3 issue

[Yaroslav Halchenko]

On another box which has 4 times more RAM or a bit more than twice total
memory, it had twice as high vm.min_free_kbytes
on another node with even more RAM it is 13821..

hm - so what is the algorithm which sets it? percent of available RAM?

For now I am adjusting it on that server to be twice from default.

Thanks everyone for your help

On Thu, 30 Nov 2006, Avi Kivity wrote:

> Alan wrote:
> >Under heavy network or I/O pressure it may not have time to swap to get
> >the memory. Thus adding swap won't usually help. Adding RAM may do but
> >its often not the best answer. Arjan's suggestion should sort it, and -
> >yes typically boxes with very high I/O and network load need more of a
> >pool of memory free for immediate use than other systems.


> It could be nice if the kernel could autotune this, for example by raising the free memory goal when memory shortage is detected, and lowering it 
> gradually when not.

> The sysctl could be a minimum from which this is calculated.
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik        

Re: kswapd/tg3 issue

[Herbert Xu]

Alan <alan@lxorguk.ukuu.org.uk> wrote:
>> 
>> Just got a logwatch daily mail which revealed a problem:
>> [2024412.788680] kswapd1: page allocation failure. order:2, mode:0x20
>> and a lengthy backtrace with head
>> 
>> ,------------------------------------------------------------------------
>> | [2024412.795212] Call Trace:
>> | [2024412.799768]  <IRQ> [<ffffffff8020c852>] __alloc_pages+0x27a/0x291
>> | [2024412.806452]  [<ffffffff802a08e3>] kmem_getpages+0x5e/0xd8
>> | [2024412.812370]  [<ffffffff80212c68>] cache_grow+0xd0/0x185
>> | [2024412.818064]  [<ffffffff80245c4f>] cache_alloc_refill+0x18c/0x1da
>> | [2024412.824625]  [<ffffffff802a1979>] __kmalloc+0x93/0xa3
>> | [2024412.830145]  [<ffffffff80222e9e>] __alloc_skb+0x54/0x117
>> | [2024412.835958]  [<ffffffff803b8a55>] __netdev_alloc_skb+0x12/0x2d
>> | [2024412.842347]  [<ffffffff80370292>] tg3_alloc_rx_skb+0xbb/0x146
>> `---
>> full dmesg is at
>> http://www.onerussian.com/Linux/bugs/bug.kswapd/dmesg
>> 
>> is that critical? seems to behave ok but...
> 
> Its tell us that the machine got very very tight on memory, far tighter
> than it probably ever should in normal situations. It is harmless of
> itself and if you only get the odd one is not a worry.

Since this is a 2nd order allocation, it could also be that you have
memory but it's fragmented.  If you aren't using jumbograms you can
try disabling that.

Cheers,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: kswapd/tg3 issue

[Yaroslav Halchenko]

> Since this is a 2nd order allocation, it could also be that you have
> memory but it's fragmented. 
Thanks for the info!

> If you aren't using jumbograms you can
> try disabling that.
disabling 2nd order allocation?
and I do use jumbos on that box (it is an NFS server so jumbo frames --
MTU 9000 -- presumable help a bit on CPU utilization)

> Cheers,
-- 
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: (973) 353-5440x263 | FWD: 82823 | Fax: (973) 353-1171
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik