__GFP flags and kmalloc failures

__GFP flags and kmalloc failures

[Pierre Ossman]

I'm trying to allocate a buffer to be used for ISA DMA and I'm 
experiencing some difficulties.

I'm allocating a 64kB buffer (max size for low ISA DMA) using:

kmalloc(65536, GFP_KERNEL | GFP_DMA);

The choice of flags are from another driver that does ISA DMA so I 
didn't put too much thought into them at first.

The problem is now that this allocation doesn't always succeed. When it 
fails I get:

insmod: page allocation failure. order:4, mode:0x11

and a nice little stack dump.

Digging around in gfp.h to see if I have the proper flags I find that I 
currently have the following:

* __GFP_WAIT : This seems to indicate that the process should be put to 
sleep until the allocation can succeed. Doesn't seem to work that way 
though.

* __GFP_IO : What is meant with physical IO? PCI DMA? This buffer needs 
only be read by the ISA DMA controller and the driver in kernel space. 
Any useful data is copied to other buffers.

* __GFP_FS : Since the data is copied before use this probably isn't needed.

* __GFP_DMA : From what I've been told, this flags causes the allocator 
to do the magic required for the buffer to end up i memory accessible 
from the ISA DMA controller. So this seems to be the only flag that 
actually does anything useful.

My question is now, why does the allocation fail (sometimes) and what 
should I do about it?

Memory fragmentation and overusage seems like reasons to why but why 
doesn't the kernel throw out cache pages and reorganise user pages so 
that the allocation can succeed?

As for solutions I've tried using __GFP_REPEAT which seems to do the 
trick. But the double underscore indicates (at least to me) that these 
are internal defines that shouldn't be used except for very special 
cases. What is the policy about these?

Rgds
Pierre

Re: __GFP flags and kmalloc failures

[Marcelo Tosatti]

On Tue, Nov 02, 2004 at 04:49:20PM +0100, Pierre Ossman wrote:
> I'm trying to allocate a buffer to be used for ISA DMA and I'm 
> experiencing some difficulties.
> 
> I'm allocating a 64kB buffer (max size for low ISA DMA) using:
> 
> kmalloc(65536, GFP_KERNEL | GFP_DMA);
> 
> The choice of flags are from another driver that does ISA DMA so I 
> didn't put too much thought into them at first.
> 
> The problem is now that this allocation doesn't always succeed. When it 
> fails I get:
> 
> insmod: page allocation failure. order:4, mode:0x11

This is a big allocation and the kernel is having problem finding such a 
big page, due to memory fragmentation (as you mention below).

What kernel version are you using?

-mm contains a series of patches from Nick which should make the situation 
better, have you tried it? Currently kswapd doenst honour high order 
page shortage.

> and a nice little stack dump.
> 
> Digging around in gfp.h to see if I have the proper flags I find that I 
> currently have the following:
> 
> * __GFP_WAIT : This seems to indicate that the process should be put to 
> sleep until the allocation can succeed. Doesn't seem to work that way 
> though.
> 
> * __GFP_IO : What is meant with physical IO? PCI DMA? This buffer needs 
> only be read by the ISA DMA controller and the driver in kernel space. 
> Any useful data is copied to other buffers.
> 
> * __GFP_FS : Since the data is copied before use this probably isn't needed.
> 
> * __GFP_DMA : From what I've been told, this flags causes the allocator 
> to do the magic required for the buffer to end up i memory accessible 
> from the ISA DMA controller. So this seems to be the only flag that 
> actually does anything useful.
> 
> My question is now, why does the allocation fail (sometimes) and what 
> should I do about it?
> 
> Memory fragmentation and overusage seems like reasons to why but why 
> doesn't the kernel throw out cache pages and reorganise user pages so 
> that the allocation can succeed?

We're working on that.

> As for solutions I've tried using __GFP_REPEAT which seems to do the 
> trick. But the double underscore indicates (at least to me) that these 
> are internal defines that shouldn't be used except for very special 
> cases. What is the policy about these?

Its OK to use these flags externally. They might change in future major kernel
versions though, or even future v2.6 release.  ie its not a stable API.

Re: __GFP flags and kmalloc failures

[Pierre Ossman]

Marcelo Tosatti wrote:

>On Tue, Nov 02, 2004 at 04:49:20PM +0100, Pierre Ossman wrote:
>  
>
>>
>>The problem is now that this allocation doesn't always succeed. When it 
>>fails I get:
>>
>>insmod: page allocation failure. order:4, mode:0x11
>>    
>>
>
>This is a big allocation and the kernel is having problem finding such a 
>big page, due to memory fragmentation (as you mention below).
>
>What kernel version are you using?
>  
>
I'm currently running 2.6.9. No external patches (except for my own 
stuff related to this driver).

>-mm contains a series of patches from Nick which should make the situation 
>better, have you tried it? Currently kswapd doenst honour high order 
>page shortage.
>
>  
>
No I haven't. Only saw it today and I usually don't use the -mm tree. 
I've gotten the impression it's a bit too bleeding edge for me ;)
What do these patches add to the mix?
I'm also not familiar what the order means. I guess it's some kind of 
priority system? Is there a way I can raise my priority to get access to 
the memory that kswapd actually keeps available?

>>As for solutions I've tried using __GFP_REPEAT which seems to do the 
>>trick. But the double underscore indicates (at least to me) that these 
>>are internal defines that shouldn't be used except for very special 
>>cases. What is the policy about these?
>>    
>>
>
>Its OK to use these flags externally. They might change in future major kernel
>versions though, or even future v2.6 release.  ie its not a stable API.
>  
>
Is there any other way of increasing the chances of actually getting the 
pages I need? Since it is DMA it needs to be one big block.

Rgds
Pierre

Re: __GFP flags and kmalloc failures

[Marcelo Tosatti]

On Tue, Nov 02, 2004 at 07:01:55PM +0100, Pierre Ossman wrote:
> Marcelo Tosatti wrote:
> 
> >On Tue, Nov 02, 2004 at 04:49:20PM +0100, Pierre Ossman wrote:
> > 
> >
> >>
> >>The problem is now that this allocation doesn't always succeed. When it 
> >>fails I get:
> >>
> >>insmod: page allocation failure. order:4, mode:0x11
> >>   
> >>
> >
> >This is a big allocation and the kernel is having problem finding such a 
> >big page, due to memory fragmentation (as you mention below).
> >
> >What kernel version are you using?
> > 
> >
> I'm currently running 2.6.9. No external patches (except for my own 
> stuff related to this driver).
> 
> >-mm contains a series of patches from Nick which should make the situation 
> >better, have you tried it? Currently kswapd doenst honour high order 
> >page shortage.
> >
> > 
> >
> No I haven't. Only saw it today and I usually don't use the -mm tree. 
> I've gotten the impression it's a bit too bleeding edge for me ;)
> What do these patches add to the mix?

They make kswapd free pages when high order allocations fail. 

> I'm also not familiar what the order means. I guess it's some kind of 
> priority system? Is there a way I can raise my priority to get access to 
> the memory that kswapd actually keeps available?

order indicates the order which 2 will be elevated, to indicate how many
physical pages this "high order page" is, in size.

"2 ^ 0" = 1 page, order 0 , "2 ^ 1" = 2 pages, order 1,
 "2 ^ 2" = 4 pages, order 2 and so on.
> 
> >>As for solutions I've tried using __GFP_REPEAT which seems to do the 
> >>trick. But the double underscore indicates (at least to me) that these 
> >>are internal defines that shouldn't be used except for very special 
> >>cases. What is the policy about these?
> >>   
> >>
> >
> >Its OK to use these flags externally. They might change in future major 
> >kernel
> >versions though, or even future v2.6 release.  ie its not a stable API.
> > 
> >
> Is there any other way of increasing the chances of actually getting the 
> pages I need? Since it is DMA it needs to be one big block.

__GFP_NOFAIL, from gfp.h:

 * Action modifiers - doesn't change the zoning
 *
 * __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
 * _might_ fail.  This depends upon the particular VM implementation.
 *
 * __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
 * cannot handle allocation failures.
 *
 * __GFP_NORETRY: The VM implementation must not retry indefinitely.
 */

Re: __GFP flags and kmalloc failures

[Pierre Ossman]

Marcelo Tosatti wrote:

>On Tue, Nov 02, 2004 at 07:01:55PM +0100, Pierre Ossman wrote:
>  
>
>>Is there any other way of increasing the chances of actually getting the 
>>pages I need? Since it is DMA it needs to be one big block.
>>    
>>
>
>__GFP_NOFAIL, from gfp.h:
>
> * Action modifiers - doesn't change the zoning
> *
> * __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
> * _might_ fail.  This depends upon the particular VM implementation.
> *
> * __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
> * cannot handle allocation failures.
> *
> * __GFP_NORETRY: The VM implementation must not retry indefinitely.
> */
>
>  
>
Yes, I've browsed through these. __GFP_NOFAIL seems like it can hang for 
a very long time (I don't know if there is an upper bound on how long it 
will have to wait for a free page). __GFP_REPEAT seems to work good 
enough in this case.
My question was meant to be more along the lines of "Is there anything I 
can do without resorting to unstable/interal API:s?".

Rgds
Pierre

Re: __GFP flags and kmalloc failures

[Marcelo Tosatti]

On Tue, Nov 02, 2004 at 07:31:39PM +0100, Pierre Ossman wrote:
> Marcelo Tosatti wrote:
> 
> >On Tue, Nov 02, 2004 at 07:01:55PM +0100, Pierre Ossman wrote:
> > 
> >
> >>Is there any other way of increasing the chances of actually getting the 
> >>pages I need? Since it is DMA it needs to be one big block.
> >>   
> >>
> >
> >__GFP_NOFAIL, from gfp.h:
> >
> >* Action modifiers - doesn't change the zoning
> >*
> >* __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
> >* _might_ fail.  This depends upon the particular VM implementation.
> >*
> >* __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
> >* cannot handle allocation failures.
> >*
> >* __GFP_NORETRY: The VM implementation must not retry indefinitely.
> >*/
> >
> > 
> >
> Yes, I've browsed through these. __GFP_NOFAIL seems like it can hang for 
> a very long time (I don't know if there is an upper bound on how long it 
> will have to wait for a free page). __GFP_REPEAT seems to work good 
> enough in this case.
> My question was meant to be more along the lines of "Is there anything I 
> can do without resorting to unstable/interal API:s?".

Not really. 

They are not that unstable, I shouldnt mean that.

These defines are not as stable as system calls - VM internals might change 
in v2.7 and the flags also - but for v2.6 they are very likely to remain 
untouched.

Its just like any driver API in Linux - they change.

Just keep an eye.

Re: __GFP flags and kmalloc failures

[Pierre Ossman]

Marcelo Tosatti wrote:

>On Tue, Nov 02, 2004 at 07:31:39PM +0100, Pierre Ossman wrote:
>  
>
>>
>>Yes, I've browsed through these. __GFP_NOFAIL seems like it can hang for 
>>a very long time (I don't know if there is an upper bound on how long it 
>>will have to wait for a free page). __GFP_REPEAT seems to work good 
>>enough in this case.
>>My question was meant to be more along the lines of "Is there anything I 
>>can do without resorting to unstable/interal API:s?".
>>    
>>
>
>Not really. 
>
>They are not that unstable, I shouldnt mean that.
>
>These defines are not as stable as system calls - VM internals might change 
>in v2.7 and the flags also - but for v2.6 they are very likely to remain 
>untouched.
>
>Its just like any driver API in Linux - they change.
>
>Just keep an eye.
>  
>
Ok. I'll stick with __GFP_REPEAT for now then. If things break in the 
future it should be fairly obvious that this is the problem. Any users 
of the driver will probably point out that it's broken rather quickly :)

Thanks for the help!

Rgds
Pierre