Can Linux live without DMA zone?

Can Linux live without DMA zone?

[Jun Sun]

I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
from kernel.  A consequence is that DMA zone now has size 0.  That causes
many drivers to grief (OOMs).

I see two ways out:

1. Modify individual drivers and convince them not to alloc with GFP_DMA.
   I have been trying to do this but do not seem to see an end of it.  :)

2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
   the whole memory region belongs to DMA zone.

#2 sounds pretty hackish.  I am sure something bad will happen
sooner or later (like what?). But so far it appears to be working fine.

The fundamental question is: Has anybody tried to run Linux without 0 sized
DMA zone before?  Am I doing something that nobody has done before (which is
something really hard to believe these days with Linux :P)?

Cheers.

Jun

RE: Can Linux live without DMA zone?

[Conke Hu]

It seems a good idea.
Is dma zone is still necessay on most modern computers?

Best regards,
Conke @ AMD, Inc.

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jun Sun
Sent: 2006年11月2日 10:16
To: linux-kernel@vger.kernel.org
Subject: Can Linux live without DMA zone?


I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
from kernel.  A consequence is that DMA zone now has size 0.  That causes
many drivers to grief (OOMs).

I see two ways out:

1. Modify individual drivers and convince them not to alloc with GFP_DMA.
   I have been trying to do this but do not seem to see an end of it.  :)

2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
   the whole memory region belongs to DMA zone.

#2 sounds pretty hackish.  I am sure something bad will happen
sooner or later (like what?). But so far it appears to be working fine.

The fundamental question is: Has anybody tried to run Linux without 0 sized
DMA zone before?  Am I doing something that nobody has done before (which is
something really hard to believe these days with Linux :P)?

Cheers.

Jun
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: Can Linux live without DMA zone?

[Peter Zijlstra]

On Thu, 2006-11-02 at 11:43 +0800, Conke Hu wrote:
> It seems a good idea.
> Is dma zone is still necessay on most modern computers?

(don't top post!)

if you would have used google, you'd have found this:
  http://www.mail-archive.com/linux-arch@vger.kernel.org/msg01623.html


cheers

> -----Original Message-----
> From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Jun Sun
> Sent: 2006年11月2日 10:16
> To: linux-kernel@vger.kernel.org
> Subject: Can Linux live without DMA zone?
> 
> 
> I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
> from kernel.  A consequence is that DMA zone now has size 0.  That causes
> many drivers to grief (OOMs).
> 
> I see two ways out:
> 
> 1. Modify individual drivers and convince them not to alloc with GFP_DMA.
>    I have been trying to do this but do not seem to see an end of it.  :)
> 
> 2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
>    the whole memory region belongs to DMA zone.
> 
> #2 sounds pretty hackish.  I am sure something bad will happen
> sooner or later (like what?). But so far it appears to be working fine.
> 
> The fundamental question is: Has anybody tried to run Linux without 0 sized
> DMA zone before?  Am I doing something that nobody has done before (which is
> something really hard to believe these days with Linux :P)?
> 
> Cheers.
> 
> Jun
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Can Linux live without DMA zone?

[Arjan van de Ven]

On Wed, 2006-11-01 at 18:15 -0800, Jun Sun wrote:
> I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
> from kernel.  A consequence is that DMA zone now has size 0.  That causes
> many drivers to grief (OOMs).
> 
> I see two ways out:
> 
> 1. Modify individual drivers and convince them not to alloc with GFP_DMA.
>    I have been trying to do this but do not seem to see an end of it.  :)
> 
> 2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
>    the whole memory region belongs to DMA zone.
> 
> #2 sounds pretty hackish.  I am sure something bad will happen
> sooner or later (like what?). But so far it appears to be working fine.
> 
> The fundamental question is: Has anybody tried to run Linux without 0 sized
> DMA zone before?  Am I doing something that nobody has done before (which is
> something really hard to believe these days with Linux :P)?

on a PC there are still devices that need memory in the lower 16Mb.....
(like floppy)

Maybe you should reserve another area of memory instead!


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: Can Linux live without DMA zone?

[Paul Mundt]

On Wed, Nov 01, 2006 at 06:15:47PM -0800, Jun Sun wrote:
> The fundamental question is: Has anybody tried to run Linux without 0 sized
> DMA zone before?  Am I doing something that nobody has done before (which is
> something really hard to believe these days with Linux :P)?
> 
There's patches that rip out ZONE_DMA for platforms that don't have a DMA
limitation (and were pretty much putting all of ZONE_NORMAL in ZONE_DMA),
all of these are already in -mm.

RE: Can Linux live without DMA zone?

[Conke Hu]

Most PCs do not have ISA or floppy, so maybe we could add an option to enable DMA zone or not.

Best regards,
Conke @ AMD, Inc.

-----Original Message-----
From: linux-kernel-owner@vger.kernel.org [mailto:linux-kernel-owner@vger.kernel.org] On Behalf Of Arjan van de Ven
Sent: 2006年11月2日 17:16
To: Jun Sun
Cc: linux-kernel@vger.kernel.org
Subject: Re: Can Linux live without DMA zone?

On Wed, 2006-11-01 at 18:15 -0800, Jun Sun wrote:
> I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
> from kernel.  A consequence is that DMA zone now has size 0.  That causes
> many drivers to grief (OOMs).
> 
> I see two ways out:
> 
> 1. Modify individual drivers and convince them not to alloc with GFP_DMA.
>    I have been trying to do this but do not seem to see an end of it.  :)
> 
> 2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
>    the whole memory region belongs to DMA zone.
> 
> #2 sounds pretty hackish.  I am sure something bad will happen
> sooner or later (like what?). But so far it appears to be working fine.
> 
> The fundamental question is: Has anybody tried to run Linux without 0 sized
> DMA zone before?  Am I doing something that nobody has done before (which is
> something really hard to believe these days with Linux :P)?

on a PC there are still devices that need memory in the lower 16Mb.....
(like floppy)

Maybe you should reserve another area of memory instead!


-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

RE: Can Linux live without DMA zone?

[Arjan van de Ven]

On Thu, 2006-11-02 at 18:33 +0800, Conke Hu wrote:
> Most PCs do not have ISA or floppy, so maybe we could add an option to enable DMA zone or not.
> 

please don't top-post. 


also floppy is still there a lot unfortunately ... as are some of the
more crappy soundcards. ZONE_DMA is a 32 bit linux thing; most modern
systems are 64 bit capable now....

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

RE: Can Linux live without DMA zone?

[Alan Cox]

Ar Iau, 2006-11-02 am 18:33 +0800, ysgrifennodd Conke Hu:
> Most PCs do not have ISA or floppy, so maybe we could add an option to enable DMA zone or not.

Lots of hardware still needs memory below the 2GB boundary, or for
floppy etc (which is common) in the low 16MB. People are still shipping
wireless hardware with these kind of limits even today.

RE: Can Linux live without DMA zone?

[Christoph Lameter]

That code is in 2.6.19-rc4-mm1. If you clear CONFIG_ZONE_DMA then you 
wont have a DMA zone anymore.

Re: Can Linux live without DMA zone?

[Phillip Susi]

Shouldn't only ancient ISA drivers be using GFP_DMA?  You know, ones 
that actually require it?  PCI drivers should not have this limit.

Jun Sun wrote:
> I am trying to reserve a block of memory (>16MB) starting from 0 and hide it 
> from kernel.  A consequence is that DMA zone now has size 0.  That causes
> many drivers to grief (OOMs).
> 
> I see two ways out:
> 
> 1. Modify individual drivers and convince them not to alloc with GFP_DMA.
>    I have been trying to do this but do not seem to see an end of it.  :)
> 
> 2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that
>    the whole memory region belongs to DMA zone.
> 
> #2 sounds pretty hackish.  I am sure something bad will happen
> sooner or later (like what?). But so far it appears to be working fine.
> 
> The fundamental question is: Has anybody tried to run Linux without 0 sized
> DMA zone before?  Am I doing something that nobody has done before (which is
> something really hard to believe these days with Linux :P)?
> 
> Cheers.
> 
> Jun

Re: Can Linux live without DMA zone?

[Arjan van de Ven]

On Thu, 2006-11-02 at 11:32 -0500, Phillip Susi wrote:
> Shouldn't only ancient ISA drivers be using GFP_DMA?  You know, ones 
> that actually require it?  PCI drivers should not have this limit.


that is a nice theory, but unfortunately there is just a lot of "PCI"
hardware out there for which the designers decided to save a bit of
copper and only wire up the lower X address lines (for various values of
X)
-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: Can Linux live without DMA zone?

[Alan Cox]

Ar Iau, 2006-11-02 am 11:32 -0500, ysgrifennodd Phillip Susi:
> Shouldn't only ancient ISA drivers be using GFP_DMA?  You know, ones 
> that actually require it?  PCI drivers should not have this limit.

"Should" and "Do not" are different things. Many PCI drivers have
interesting little limits.

Re: Can Linux live without DMA zone?

[Phillip Susi]

Arjan van de Ven wrote:
> that is a nice theory, but unfortunately there is just a lot of "PCI"
> hardware out there for which the designers decided to save a bit of
> copper and only wire up the lower X address lines (for various values of
> X)

Yea, but shouldn't PCI drivers be using another means than allocating 
from GFP_DMA?  Wasn't there some sort of bounce buffers call I can't 
quite remember the details of?  That performs any required translations 
to bus hardware addresses, and copies the buffer to a more appropriate 
location if required, based on the specific requirements of that device?

I know that most 32 bit PCI devices can't handle addresses above the 4 
GB mark on 64 bit machines, but those drivers should NOT be limiting DMA 
to the first 16 MB.  Especially since most machines don't have over 4 GB 
of ram anyhow, but quite often original buffers will be above 16 MB.

Re: Can Linux live without DMA zone?

[Arjan van de Ven]

On Thu, 2006-11-02 at 14:08 -0500, Phillip Susi wrote:
> Arjan van de Ven wrote:
> > that is a nice theory, but unfortunately there is just a lot of "PCI"
> > hardware out there for which the designers decided to save a bit of
> > copper and only wire up the lower X address lines (for various values of
> > X)
> 
> Yea, but shouldn't PCI drivers be using another means than allocating 
> from GFP_DMA?  Wasn't there some sort of bounce buffers call I can't 
> quite remember the details of?  That performs any required translations 
> to bus hardware addresses, and copies the buffer to a more appropriate 
> location if required, based on the specific requirements of that device?

that's for the 32 bit boundary. THe problem is that there are 31, 30, 28
and 26 bit devices as well, and those are in more trouble, and will
eventually fall back to GFP_DMA (inside the x86 PCI code; the driver
just uses the pci dma allocation routines) if they can't get suitable
memory otherwise....

It's all nice in theory. But then there is the reality that not all
devices are nice pci device that implement the entire spec;)

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org

Re: Can Linux live without DMA zone?

[Phillip Susi]

Arjan van de Ven wrote:
> that's for the 32 bit boundary. THe problem is that there are 31, 30, 28
> and 26 bit devices as well, and those are in more trouble, and will
> eventually fall back to GFP_DMA (inside the x86 PCI code; the driver
> just uses the pci dma allocation routines) if they can't get suitable
> memory otherwise....
> 
> It's all nice in theory. But then there is the reality that not all
> devices are nice pci device that implement the entire spec;)
> 

Right, but doesn't the bounce/allocation routine take as a parameter the 
limit that the device can handle?  If the device can handle 28 bit 
addresses, then the kernel should not limit it to only 24 bits.

Re: Can Linux live without DMA zone?

[Arjan van de Ven]

On Thu, 2006-11-02 at 16:26 -0500, Phillip Susi wrote:
> Arjan van de Ven wrote:
> > that's for the 32 bit boundary. THe problem is that there are 31, 30, 28
> > and 26 bit devices as well, and those are in more trouble, and will
> > eventually fall back to GFP_DMA (inside the x86 PCI code; the driver
> > just uses the pci dma allocation routines) if they can't get suitable
> > memory otherwise....
> > 
> > It's all nice in theory. But then there is the reality that not all
> > devices are nice pci device that implement the entire spec;)
> > 
> 
> Right, but doesn't the bounce/allocation routine take as a parameter the 
> limit that the device can handle?  If the device can handle 28 bit 
> addresses, then the kernel should not limit it to only 24 bits.

you're right in theory, but the kernel only has a few pools of memory
available, but not at every bit boundary. there is a 32 bit pool
(GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24
bit pool (GFP_DMA) with basically nothing inbetween.

-- 
if you want to mail me at work (you don't), use arjan (at) linux.intel.com
Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
 

Re: Can Linux live without DMA zone?

[Jun Sun]

On Thu, Nov 02, 2006 at 11:19:05PM +0100, Arjan van de Ven wrote:
> On Thu, 2006-11-02 at 16:26 -0500, Phillip Susi wrote:
> > Arjan van de Ven wrote:
> > > that's for the 32 bit boundary. THe problem is that there are 31, 30, 28
> > > and 26 bit devices as well, and those are in more trouble, and will
> > > eventually fall back to GFP_DMA (inside the x86 PCI code; the driver
> > > just uses the pci dma allocation routines) if they can't get suitable
> > > memory otherwise....
> > > 
> > > It's all nice in theory. But then there is the reality that not all
> > > devices are nice pci device that implement the entire spec;)
> > > 
> > 
> > Right, but doesn't the bounce/allocation routine take as a parameter the 
> > limit that the device can handle?  If the device can handle 28 bit 
> > addresses, then the kernel should not limit it to only 24 bits.
> 
> you're right in theory, but the kernel only has a few pools of memory
> available, but not at every bit boundary. there is a 32 bit pool
> (GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24
> bit pool (GFP_DMA) with basically nothing inbetween.
>

Perhaps a better solution is to 

1. get rid of DMA zone

2. have another alloc funciton (e.g., kmalloc_range()) which takes an
   extra pair of parameters to indicate the desired range for the
   allocated memory.  Most DMA buffers are allocated during start-up.
   So the alloc operations should generally be successful.

3. convert drivers over to use the new function.

Cheers.

Jun
   are allocated at start-up time.

Re: Can Linux live without DMA zone?

[Stephen Hemminger]

On Thu, 2 Nov 2006 15:17:15 -0800
Jun Sun <jsun@junsun.net> wrote:

> On Thu, Nov 02, 2006 at 11:19:05PM +0100, Arjan van de Ven wrote:
> > On Thu, 2006-11-02 at 16:26 -0500, Phillip Susi wrote:
> > > Arjan van de Ven wrote:
> > > > that's for the 32 bit boundary. THe problem is that there are 31, 30, 28
> > > > and 26 bit devices as well, and those are in more trouble, and will
> > > > eventually fall back to GFP_DMA (inside the x86 PCI code; the driver
> > > > just uses the pci dma allocation routines) if they can't get suitable
> > > > memory otherwise....
> > > > 
> > > > It's all nice in theory. But then there is the reality that not all
> > > > devices are nice pci device that implement the entire spec;)
> > > > 
> > > 
> > > Right, but doesn't the bounce/allocation routine take as a parameter the 
> > > limit that the device can handle?  If the device can handle 28 bit 
> > > addresses, then the kernel should not limit it to only 24 bits.
> > 
> > you're right in theory, but the kernel only has a few pools of memory
> > available, but not at every bit boundary. there is a 32 bit pool
> > (GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24
> > bit pool (GFP_DMA) with basically nothing inbetween.
> >
> 
> Perhaps a better solution is to 
> 
> 1. get rid of DMA zone
> 
> 2. have another alloc funciton (e.g., kmalloc_range()) which takes an
>    extra pair of parameters to indicate the desired range for the
>    allocated memory.  Most DMA buffers are allocated during start-up.
>    So the alloc operations should generally be successful.
> 

Network devices don't allocate buffer until they are brought up.
By then a lot of memory allocation has happened.  You could add an
interface that allows a device to say:
	kmalloc_range_intent(unsigned long mask, unsigned count, unsigned size)
to cause reservation before use.

-- 
Stephen Hemminger <shemminger@osdl.org>

Re: Can Linux live without DMA zone?

[Krzysztof Halasa]

Arjan van de Ven <arjan@infradead.org> writes:

> you're right in theory, but the kernel only has a few pools of memory
> available, but not at every bit boundary. there is a 32 bit pool
> (GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24
> bit pool (GFP_DMA) with basically nothing inbetween.

Perhaps naive question, but... what's wrong with allocating memory
from the top (within given address mask[1], size-wise)? I think we
don't allocate more than 1 page with kmalloc anymore, do we?

[1] - some devices (a specific StrongARM only?) need non-continuous
masks due to a hardware bug(s).
-- 
Krzysztof Halasa

Re: Can Linux live without DMA zone?

[Phillip Susi]

Jun Sun wrote:
> Perhaps a better solution is to 
> 
> 1. get rid of DMA zone
> 
> 2. have another alloc funciton (e.g., kmalloc_range()) which takes an
>    extra pair of parameters to indicate the desired range for the
>    allocated memory.  Most DMA buffers are allocated during start-up.
>    So the alloc operations should generally be successful.
> 
> 3. convert drivers over to use the new function.
> 
> Cheers.
> 
> Jun
>    are allocated at start-up time.

That is what I was thinking.  You don't need lots of separate pools, you 
just need the standard allocator to prefer higher addresses, and then 
the bounce routines need to simply check if the existing user buffer 
happens to already be within the area the hardware can address ( which 
it often will be ), and if not, copy the data to pages allocated in 
lower memory.