[PATCH 0/8] Optional ZONE_DMA V1

[PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

Optional ZONE_DMA

This patch follows up on the earlier work in Andrew's tree to reduce
the number of zones. The patches allow to go to a minimum of 2 zones.
This one allows also to make ZONE_DMA optional and therefore the
number of zones can be reduced to one.

ZONE_DMA is usually used for ISA DMA devices. Typically modern hardware
does not have any of these anymore. So we frequently do not need
the zone anymore. The presence of an additional zone unnecessarily
complicates VM operations. It must be scanned and balancing logic
must operate in it etc etc. If one has a 1-1 correspondence between
zones and nodes in a NUMA system then various other optimizations
become possible.

Many systems today (especially 64 bit but also 32 bit machines with less
than 4G of memory) can therefore operate just fine with a single zone.
With a single zone various loops can be optimized away by the
compiler. Many system currently do not place anything in ZONE_DMA. On
most of my systems ZONE_DMA is completely empty. Why constantly look
at an empty zone in /proc/zoneinfo and empty slab in /proc/slabinfo?
Non i386 also frequently have no need for ZONE_DMA and zones stay
empty.

The patchset was tested on i386 (UP / SMP), x86_64 (UP, NUMA) and
ia64 (NUMA).

The RFC posted earlier (see
http://marc.theaimsgroup.com/?l=linux-kernel&m\x115231723513008&w=2)
had lots of #ifdefs in them. An effort has been made to minize the number
of #ifdefs and make this as compact as possible. The job was made much easier
by the ongoing efforts of others to extract common arch specific functionality.

I have been running this for awhile now on my desktop and finally Linux is
using all my available RAM instead of leaving the 16MB in ZONE_DMA untouched:

christoph@pentium940:~$ cat /proc/zoneinfo
Node 0, zone   Normal
  pages free     4435
        min      1448
        low      1810
        high     2172
        active   241786
        inactive 210170
        scanned  0 (a: 0 i: 0)
        spanned  524224
        present  524224
    nr_anon_pages 61680
    nr_mapped    14271
    nr_file_pages 390264
    nr_slab_reclaimable 27564
    nr_slab_unreclaimable 1793
    nr_page_table_pages 449
    nr_dirty     39
    nr_writeback 0
    nr_unstable  0
    nr_bounce    0
    cpu: 0 pcp: 0
              count: 156
              high:  186
              batch: 31
    cpu: 0 pcp: 1
              count: 9
              high:  62
              batch: 15
  vm stats threshold: 20
    cpu: 1 pcp: 0
              count: 177
              high:  186
              batch: 31
    cpu: 1 pcp: 1
              count: 12
              high:  62
              batch: 15
  vm stats threshold: 20
  all_unreclaimable: 0
  prev_priority:     12
  temp_priority:     12
  start_pfn:         0

[PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Lameter]

Introduce CONFIG_ZONE_DMA

This patch simply defines CONFIG_ZONE_DMA for all arches. We later do
special things with CONFIG_ZONE_DMA after the VM and an arch are
prepared to work without ZONE_DMA.

CONFIG_ZONE_DMA can be defined in two ways depending on how
an architecture handles ISA DMA.

First if CONFIG_GENERIC_ISA_DMA is set by the arch then we know that
the arch needs ZONE_DMA because ISA DMA devices are supported. We
can catch this in mm/Kconfig and do not need to modify arch code.

Second, arches may use ZONE_DMA in an unknown way. We set CONFIG_ZONE_DMA
for all arches that do not set CONFIG_GENERIC_ISA_DMA in order to insure
backwards compatibility. The arches may later undefine ZONE_DMA
if their arch code has been verified to not depend on ZONE_DMA.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.18-rc6-mm1/mm/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/mm/Kconfig	2006-09-11 15:42:32.736665338 -0500
+++ linux-2.6.18-rc6-mm1/mm/Kconfig	2006-09-11 15:53:12.072397960 -0500
@@ -139,6 +139,10 @@
 	default "4096" if PARISC && !PA20
 	default "4"
 
+config ZONE_DMA
+	def_bool y
+	depends on GENERIC_ISA_DMA
+
 #
 # support for page migration
 #
Index: linux-2.6.18-rc6-mm1/arch/ia64/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/ia64/Kconfig	2006-09-11 15:42:32.746431796 -0500
+++ linux-2.6.18-rc6-mm1/arch/ia64/Kconfig	2006-09-11 15:53:12.089977582 -0500
@@ -22,6 +22,10 @@
 	bool
 	default y
 
+config ZONE_DMA
+	bool
+	default y
+
 config MMU
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/cris/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/cris/Kconfig	2006-09-11 15:42:32.754244962 -0500
+++ linux-2.6.18-rc6-mm1/arch/cris/Kconfig	2006-09-11 15:53:12.101697331 -0500
@@ -9,6 +9,10 @@
 	bool
 	default y
 
+config ZONE_DMA
+	bool
+	default y
+
 config RWSEM_GENERIC_SPINLOCK
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/s390/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/s390/Kconfig	2006-09-11 15:42:32.764988066 -0500
+++ linux-2.6.18-rc6-mm1/arch/s390/Kconfig	2006-09-11 15:53:12.112440433 -0500
@@ -7,6 +7,10 @@
 	bool
 	default y
 
+config ZONE_DMA
+	bool
+	default y
+
 config LOCKDEP_SUPPORT
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/xtensa/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/xtensa/Kconfig	2006-09-11 15:42:32.773777878 -0500
+++ linux-2.6.18-rc6-mm1/arch/xtensa/Kconfig	2006-09-11 15:53:12.124160181 -0500
@@ -7,6 +7,10 @@
 	bool
 	default n
 
+config ZONE_DMA
+	bool
+	default y
+
 config XTENSA
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/h8300/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/h8300/Kconfig	2006-09-11 15:42:32.782567690 -0500
+++ linux-2.6.18-rc6-mm1/arch/h8300/Kconfig	2006-09-11 15:53:12.134903284 -0500
@@ -17,6 +17,10 @@
 	bool
 	default n
 
+config ZONE_DMA
+	bool
+	default y
+
 config FPU
 	bool
 	default n
Index: linux-2.6.18-rc6-mm1/arch/v850/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/v850/Kconfig	2006-09-11 15:42:32.791357502 -0500
+++ linux-2.6.18-rc6-mm1/arch/v850/Kconfig	2006-09-11 15:53:12.145646386 -0500
@@ -10,6 +10,9 @@
 config MMU
        	bool
 	default n
+config ZONE_DMA
+	bool
+	default y
 config RWSEM_GENERIC_SPINLOCK
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/sh/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/sh/Kconfig	2006-09-11 15:42:32.800147314 -0500
+++ linux-2.6.18-rc6-mm1/arch/sh/Kconfig	2006-09-11 15:53:12.158342780 -0500
@@ -14,6 +14,10 @@
 	  gaming console.  The SuperH port has a home page at
 	  <http://www.linux-sh.org/>.
 
+config ZONE_DMA
+	bool
+	default y
+
 config RWSEM_GENERIC_SPINLOCK
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/frv/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/frv/Kconfig	2006-09-11 15:42:32.808937126 -0500
+++ linux-2.6.18-rc6-mm1/arch/frv/Kconfig	2006-09-11 15:53:12.171039174 -0500
@@ -6,6 +6,10 @@
 	bool
 	default y
 
+config ZONE_DMA
+	bool
+	default y
+
 config RWSEM_GENERIC_SPINLOCK
 	bool
 	default y
Index: linux-2.6.18-rc6-mm1/arch/m68knommu/Kconfig
=================================--- linux-2.6.18-rc6-mm1.orig/arch/m68knommu/Kconfig	2006-09-11 15:42:32.820656875 -0500
+++ linux-2.6.18-rc6-mm1/arch/m68knommu/Kconfig	2006-09-11 15:53:12.183735568 -0500
@@ -17,6 +17,10 @@
 	bool
 	default n
 
+config ZONE_DMA
+	bool
+	default y
+
 config RWSEM_GENERIC_SPINLOCK
 	bool
 	default y

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Hellwig]

On Mon, Sep 11, 2006 at 03:27:39PM -0700, Christoph Lameter wrote:
> Introduce CONFIG_ZONE_DMA
> 
> This patch simply defines CONFIG_ZONE_DMA for all arches. We later do
> special things with CONFIG_ZONE_DMA after the VM and an arch are
> prepared to work without ZONE_DMA.
> 
> CONFIG_ZONE_DMA can be defined in two ways depending on how
> an architecture handles ISA DMA.
> 
> First if CONFIG_GENERIC_ISA_DMA is set by the arch then we know that
> the arch needs ZONE_DMA because ISA DMA devices are supported. We
> can catch this in mm/Kconfig and do not need to modify arch code.
> 
> Second, arches may use ZONE_DMA in an unknown way. We set CONFIG_ZONE_DMA
> for all arches that do not set CONFIG_GENERIC_ISA_DMA in order to insure
> backwards compatibility. The arches may later undefine ZONE_DMA
> if their arch code has been verified to not depend on ZONE_DMA.

Let's pass this on to linux-arch so that the arch maintainers can give input
whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
various architectures will just keept it around despite not needing it.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Martin Schwidefsky]

On Mon, 2006-09-18 at 14:55 +0100, Christoph Hellwig wrote:
> > Second, arches may use ZONE_DMA in an unknown way. We set CONFIG_ZONE_DMA
> > for all arches that do not set CONFIG_GENERIC_ISA_DMA in order to insure
> > backwards compatibility. The arches may later undefine ZONE_DMA
> > if their arch code has been verified to not depend on ZONE_DMA.
> 
> Let's pass this on to linux-arch so that the arch maintainers can give input
> whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
> various architectures will just keept it around despite not needing it.

s390 needs ZONE_DMA for the memory area below 2GB. That is required for
the control structure of channel based I/O. It is not ISA but
conceptually related. Can we rename GENERIC_ISA_DMA into GENERIC_IO_DMA
or something similar?

-- 
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Lameter]

On Mon, 18 Sep 2006, Martin Schwidefsky wrote:

> s390 needs ZONE_DMA for the memory area below 2GB. That is required for
> the control structure of channel based I/O. It is not ISA but
> conceptually related. Can we rename GENERIC_ISA_DMA into GENERIC_IO_DMA
> or something similar?

The patchset defines CONFIG_ZONE_DMA for s390 so you are fine.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Martin Schwidefsky]

On Mon, 2006-09-18 at 10:28 -0700, Christoph Lameter wrote:
> > s390 needs ZONE_DMA for the memory area below 2GB. That is required for
> > the control structure of channel based I/O. It is not ISA but
> > conceptually related. Can we rename GENERIC_ISA_DMA into GENERIC_IO_DMA
> > or something similar?
> 
> The patchset defines CONFIG_ZONE_DMA for s390 so you are fine.

Ok, I've looked at the patch and defining ZONE_DMA is the
straightforward solution. No need to make detours over GENERIC_ISA_DMA.

-- 
blue skies,
  Martin.

Martin Schwidefsky
Linux for zSeries Development & Services
IBM Deutschland Entwicklung GmbH

"Reality continues to ruin my life." - Calvin.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Russell King]

On Mon, Sep 18, 2006 at 02:55:59PM +0100, Christoph Hellwig wrote:
> Let's pass this on to linux-arch so that the arch maintainers can give input
> whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
> various architectures will just keept it around despite not needing it.

Thanks Chris - I'd have missed this otherwise.

FWIW, several ARM platforms do make use of ZONE_DMA, though not all, so
it's not safe to assume that it can be turned off just because it's ARM.

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[James Bottomley]

On Mon, 2006-09-18 at 14:55 +0100, Christoph Hellwig wrote:
> Let's pass this on to linux-arch so that the arch maintainers can give input
> whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
> various architectures will just keept it around despite not needing it.

Right at the moment, parisc has no use for ZONE_DMA.  We basically set
ZONE_DMA = ZONE_NORMAL.  I don't believe our 32 bit systems will ever
need it.  On the 64 bit systems currently we insist on having an IOMMU
for every transaction, so ZONE_DMA = ZONE_NORMAL still.  However, on
the zx1 systems, I can see us eventually implementing IOMMU bypass and
for that case, we might need ZONE_DMA (primarily for GFP_DMA32, I
suppose, which we could do internally).

James

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Lameter]

On Mon, 18 Sep 2006, James Bottomley wrote:

> On Mon, 2006-09-18 at 14:55 +0100, Christoph Hellwig wrote:
> > Let's pass this on to linux-arch so that the arch maintainers can give input
> > whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
> > various architectures will just keept it around despite not needing it.
> 
> Right at the moment, parisc has no use for ZONE_DMA.  We basically set
> ZONE_DMA = ZONE_NORMAL.  I don't believe our 32 bit systems will ever
> need it.  On the 64 bit systems currently we insist on having an IOMMU
> for every transaction, so ZONE_DMA = ZONE_NORMAL still.  However, on
> the zx1 systems, I can see us eventually implementing IOMMU bypass and
> for that case, we might need ZONE_DMA (primarily for GFP_DMA32, I
> suppose, which we could do internally).

What does parisc do with GENERIC_ISA_DMA? I see

config GENERIC_ISA_DMA
        bool

No default and no explanation?

Can I drop this from parisc's Kconfig? Is GENERIC_ISA_DMA set or not?

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Paul Mundt]

On Mon, Sep 18, 2006 at 02:55:59PM +0100, Christoph Hellwig wrote:
> On Mon, Sep 11, 2006 at 03:27:39PM -0700, Christoph Lameter wrote:
> > Second, arches may use ZONE_DMA in an unknown way. We set CONFIG_ZONE_DMA
> > for all arches that do not set CONFIG_GENERIC_ISA_DMA in order to insure
> > backwards compatibility. The arches may later undefine ZONE_DMA
> > if their arch code has been verified to not depend on ZONE_DMA.
> 
> Let's pass this on to linux-arch so that the arch maintainers can give input
> whether they need ZONE_DMA or not.  If this just sneaks in unseen I'd bet
> various architectures will just keept it around despite not needing it.
> 
sh and sh64 have no use for ZONE_DMA, it's effectively ZONE_DMA =
ZONE_NORMAL anyways, so it can be safely switched off (though
arch/sh/mm/init.c should then be changed to use ZONE_NORMAL if ZONE_DMA
goes away, as we currently place all lowmem in ZONE_DMA, likewise for
sh64). None of our DMA controllers have any particular limitations where
ZONE_DMA would be useful at least.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Lameter]

On Tue, 19 Sep 2006, Paul Mundt wrote:

> sh and sh64 have no use for ZONE_DMA, it's effectively ZONE_DMA =
> ZONE_NORMAL anyways, so it can be safely switched off (though
> arch/sh/mm/init.c should then be changed to use ZONE_NORMAL if ZONE_DMA
> goes away, as we currently place all lowmem in ZONE_DMA, likewise for
> sh64). None of our DMA controllers have any particular limitations where
> ZONE_DMA would be useful at least.

Ok. I dropped the CONFIG_ZONE_DMA for sh so you will no longer have a DMA 
zone when this patch goes in.
 
sh64 has the same strange code as parisc:

config GENERIC_ISA_DMA
        bool

You do not have ISA_DMA so I should drop these lines?

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Paul Mundt]

On Mon, Sep 18, 2006 at 10:33:00AM -0700, Christoph Lameter wrote:
> sh64 has the same strange code as parisc:
> 
> config GENERIC_ISA_DMA
>         bool
> 
> You do not have ISA_DMA so I should drop these lines?
> 
It doesn't really matter, this notation basically keeps it disabled
anyways (you'll note the absence of it from the defconfigs).

On Mon, Sep 18, 2006 at 11:36:55AM -0700, Christoph Lameter wrote:
> Index: linux-2.6.18-rc6-mm2/arch/sh/mm/init.c
> =================================> --- linux-2.6.18-rc6-mm2.orig/arch/sh/mm/init.c	2006-09-18 12:54:04.733274009 -0500
> +++ linux-2.6.18-rc6-mm2/arch/sh/mm/init.c	2006-09-18 12:58:58.563038661 -0500
> @@ -156,7 +156,6 @@ void __init paging_init(void)
>  	 * Setup some defaults for the zone sizes.. these should be safe
>  	 * regardless of distcontiguous memory or MMU settings.
>  	 */
> -	zones_size[ZONE_DMA] = 0 >> PAGE_SHIFT;
>  	zones_size[ZONE_NORMAL] = __MEMORY_SIZE >> PAGE_SHIFT;
>  #ifdef CONFIG_HIGHMEM
>  	zones_size[ZONE_HIGHMEM] = 0 >> PAGE_SHIFT;

You've missed the other ZONE_DMA references, if you scroll a bit further
down that's where we fill in ZONE_DMA, this is simply the default zone
layout that we rely on for nommu.

sh64 part looks fine.

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Christoph Lameter]

On Tue, 19 Sep 2006, Paul Mundt wrote:

> You've missed the other ZONE_DMA references, if you scroll a bit further
> down that's where we fill in ZONE_DMA, this is simply the default zone
> layout that we rely on for nommu.

Are you sure that sh does not need ZONE_DMA? There is code in there
to check for the DMA boundary. The following patch disables that
code if CONFIG_ZONE_DMA is not set.



sh / sh64: Remove ZONE_DMA remains.

Both arches do not need ZONE_DMA

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.18-rc6-mm2/arch/sh/mm/init.c
=================================--- linux-2.6.18-rc6-mm2.orig/arch/sh/mm/init.c	2006-09-18 13:27:48.125691412 -0500
+++ linux-2.6.18-rc6-mm2/arch/sh/mm/init.c	2006-09-18 17:56:23.632421140 -0500
@@ -156,7 +156,6 @@ void __init paging_init(void)
 	 * Setup some defaults for the zone sizes.. these should be safe
 	 * regardless of distcontiguous memory or MMU settings.
 	 */
-	zones_size[ZONE_DMA] = 0 >> PAGE_SHIFT;
 	zones_size[ZONE_NORMAL] = __MEMORY_SIZE >> PAGE_SHIFT;
 #ifdef CONFIG_HIGHMEM
 	zones_size[ZONE_HIGHMEM] = 0 >> PAGE_SHIFT;
@@ -186,13 +185,16 @@ void __init paging_init(void)
 		max_dma = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;
 		low = MAX_LOW_PFN;
 
-		if (low < max_dma) {
-			zones_size[ZONE_DMA] = low - start_pfn;
-			zones_size[ZONE_NORMAL] = 0;
-		} else {
+#ifdef CONFIG_ZONE_DMA
+		if (low < max_dma)
+#endif
+			zones_size[ZONE_NORMAL] = low - start_pfn;
+#ifdef CONFIG_ZONE_DMA
+		else {
 			zones_size[ZONE_DMA] = max_dma - start_pfn;
 			zones_size[ZONE_NORMAL] = low - max_dma;
 		}
+#endif
 	}
 
 #elif defined(CONFIG_CPU_SH3) || defined(CONFIG_CPU_SH4)
Index: linux-2.6.18-rc6-mm2/arch/sh64/mm/init.c
=================================--- linux-2.6.18-rc6-mm2.orig/arch/sh64/mm/init.c	2006-09-18 13:27:48.134481203 -0500
+++ linux-2.6.18-rc6-mm2/arch/sh64/mm/init.c	2006-09-18 17:54:14.263532486 -0500
@@ -118,10 +118,7 @@ void __init paging_init(void)
 
 	mmu_context_cache = MMU_CONTEXT_FIRST_VERSION;
 
-        /*
-	 * All memory is good as ZONE_NORMAL (fall-through) and ZONE_DMA.
-         */
-	zones_size[ZONE_DMA] = MAX_LOW_PFN - START_PFN;
+	zones_size[ZONE_NORMAL] = MAX_LOW_PFN - START_PFN;
 	NODE_DATA(0)->node_mem_map = NULL;
 	free_area_init_node(0, NODE_DATA(0), zones_size, __MEMORY_START >> PAGE_SHIFT, 0);
 }
Index: linux-2.6.18-rc6-mm2/arch/sh64/Kconfig
=================================--- linux-2.6.18-rc6-mm2.orig/arch/sh64/Kconfig	2006-09-18 13:27:48.145224281 -0500
+++ linux-2.6.18-rc6-mm2/arch/sh64/Kconfig	2006-09-18 17:54:14.285995416 -0500
@@ -36,9 +36,6 @@ config GENERIC_CALIBRATE_DELAY
 config RWSEM_XCHGADD_ALGORITHM
 	bool
 
-config GENERIC_ISA_DMA
-	bool
-
 source init/Kconfig
 
 menu "System type"

Re: [PATCH 2/6] Introduce CONFIG_ZONE_DMA

[Paul Mundt]

On Mon, Sep 18, 2006 at 03:58:52PM -0700, Christoph Lameter wrote:
> On Tue, 19 Sep 2006, Paul Mundt wrote:
> > You've missed the other ZONE_DMA references, if you scroll a bit further
> > down that's where we fill in ZONE_DMA, this is simply the default zone
> > layout that we rely on for nommu.
> 
> Are you sure that sh does not need ZONE_DMA? There is code in there
> to check for the DMA boundary. The following patch disables that
> code if CONFIG_ZONE_DMA is not set.
> 
Yes, MAX_DMA_ADDRESS (in include/asm-sh/dma.h) is from when we needed it for
alloc_bootmem(), we have no interest in it, but we can't kill off the
definition either since some drivers seem to rely on it.. It was also left
around in case some CPU variants with an arbitrary limitation in their
respective DMACs popped up.

All lowmem fits < MAX_DMA_ADDRESS and so gets stuffed in ZONE_DMA, as
per:

                if (low < max_dma) {
                        zones_size[ZONE_DMA] = low - start_pfn;
                        zones_size[ZONE_NORMAL] = 0;

So we may as well just do away with it entirely, via something like this:

Signed-off-by: Paul Mundt <lethal@linux-sh.org>

diff --git a/arch/sh/mm/init.c b/arch/sh/mm/init.c
index 8ea27ca..40494f9 100644
--- a/arch/sh/mm/init.c
+++ b/arch/sh/mm/init.c
@@ -156,7 +156,6 @@ void __init paging_init(void)
 	 * Setup some defaults for the zone sizes.. these should be safe
 	 * regardless of distcontiguous memory or MMU settings.
 	 */
-	zones_size[ZONE_DMA] = 0 >> PAGE_SHIFT;
 	zones_size[ZONE_NORMAL] = __MEMORY_SIZE >> PAGE_SHIFT;
 #ifdef CONFIG_HIGHMEM
 	zones_size[ZONE_HIGHMEM] = 0 >> PAGE_SHIFT;
@@ -168,7 +167,7 @@ #ifdef CONFIG_MMU
 	 * the zone sizes accordingly, in addition to turning it on.
 	 */
 	{
-		unsigned long max_dma, low, start_pfn;
+		unsigned long low, start_pfn;
 		pgd_t *pg_dir;
 		int i;
 
@@ -183,16 +182,10 @@ #ifdef CONFIG_MMU
 
 		/* Fixup the zone sizes */
 		start_pfn = START_PFN;
-		max_dma = virt_to_phys((char *)MAX_DMA_ADDRESS) >> PAGE_SHIFT;
 		low = MAX_LOW_PFN;
 
-		if (low < max_dma) {
-			zones_size[ZONE_DMA] = low - start_pfn;
-			zones_size[ZONE_NORMAL] = 0;
-		} else {
-			zones_size[ZONE_DMA] = max_dma - start_pfn;
-			zones_size[ZONE_NORMAL] = low - max_dma;
-		}
+		/* No DMA limitation, shove all of lowmem in ZONE_NORMAL. */
+		zones_size[ZONE_NORMAL] = low - start_pfn;
 	}
 
 #elif defined(CONFIG_CPU_SH3) || defined(CONFIG_CPU_SH4)

[PATCH 3/6] Optional ZONE_DMA in the VM

[Christoph Lameter]

Make ZONE_DMA optional in core code.

- ifdef definitions for ZONE_DMA and related code following
  the example for ZONE_DMA32 and ZONE_HIGHMEM.

- Without ZONE_DMA, ZONE_HIGHMEM and ZONE_DMA32 we get to a
  ZONES_SHIFT of 0.

- Modify the VM statistics to work correctly without a DMA
  zone.

- Modify slab to not create DMA slabs if there is no
  ZONE_DMA.

Signed-off-by: Christoph Lameter <clameter@sgi.com>

Index: linux-2.6.18-rc6-mm1/include/linux/mmzone.h
=================================--- linux-2.6.18-rc6-mm1.orig/include/linux/mmzone.h	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/include/linux/mmzone.h	2006-09-11 16:45:01.863158377 -0500
@@ -90,6 +90,7 @@
 #endif
 
 enum zone_type {
+#ifdef CONFIG_ZONE_DMA
 	/*
 	 * ZONE_DMA is used when there are devices that are not able
 	 * to do DMA to all of addressable memory (ZONE_NORMAL). Then we
@@ -110,6 +111,7 @@
 	 * 			<16M.
 	 */
 	ZONE_DMA,
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	/*
 	 * x86_64 needs two ZONE_DMAs because it supports devices that are
@@ -147,7 +149,11 @@
  */
 
 #if !defined(CONFIG_ZONE_DMA32) && !defined(CONFIG_HIGHMEM)
+#if !defined(CONFIG_ZONE_DMA)
+#define ZONES_SHIFT 0
+#else
 #define ZONES_SHIFT 1
+#endif
 #else
 #define ZONES_SHIFT 2
 #endif
@@ -447,7 +453,11 @@
 
 static inline int is_dma(struct zone *zone)
 {
+#ifdef CONFIG_ZONE_DMA
 	return zone = zone->zone_pgdat->node_zones + ZONE_DMA;
+#else
+	return 0;
+#endif
 }
 
 /* These two functions are used to setup the per zone pages min values */
Index: linux-2.6.18-rc6-mm1/mm/page_alloc.c
=================================--- linux-2.6.18-rc6-mm1.orig/mm/page_alloc.c	2006-09-11 16:44:51.000000000 -0500
+++ linux-2.6.18-rc6-mm1/mm/page_alloc.c	2006-09-11 16:45:01.883667932 -0500
@@ -71,7 +71,9 @@
  * don't need any ZONE_NORMAL reservation
  */
 int sysctl_lowmem_reserve_ratio[MAX_NR_ZONES-1] = {
+#ifdef CONFIG_ZONE_DMA
 	 256,
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	 256,
 #endif
@@ -90,7 +92,9 @@
 EXPORT_SYMBOL(zone_table);
 
 static char *zone_names[MAX_NR_ZONES] = {
+#ifdef CONFIG_ZONE_DMA
 	 "DMA",
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	 "DMA32",
 #endif
Index: linux-2.6.18-rc6-mm1/include/linux/gfp.h
=================================--- linux-2.6.18-rc6-mm1.orig/include/linux/gfp.h	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/include/linux/gfp.h	2006-09-11 16:45:01.902224197 -0500
@@ -80,8 +80,10 @@
 
 static inline enum zone_type gfp_zone(gfp_t flags)
 {
+#ifdef CONFIG_ZONE_DMA
 	if (flags & __GFP_DMA)
 		return ZONE_DMA;
+#endif
 #ifdef CONFIG_ZONE_DMA32
 	if (flags & __GFP_DMA32)
 		return ZONE_DMA32;
Index: linux-2.6.18-rc6-mm1/mm/slab.c
=================================--- linux-2.6.18-rc6-mm1.orig/mm/slab.c	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/mm/slab.c	2006-09-11 16:45:01.919803815 -0500
@@ -1441,13 +1443,14 @@
 					ARCH_KMALLOC_FLAGS|SLAB_PANIC,
 					NULL, NULL);
 		}
-
+#ifdef CONFIG_ZONE_DMA
 		sizes->cs_dmacachep = kmem_cache_create(names->name_dma,
 					sizes->cs_size,
 					ARCH_KMALLOC_MINALIGN,
 					ARCH_KMALLOC_FLAGS|SLAB_CACHE_DMA|
 						SLAB_PANIC,
 					NULL, NULL);
+#endif
 		sizes++;
 		names++;
 	}
@@ -2275,8 +2278,10 @@
 	cachep->slab_size = slab_size;
 	cachep->flags = flags;
 	cachep->gfpflags = 0;
+#ifdef CONFIG_ZONE_DMA
 	if (flags & SLAB_CACHE_DMA)
 		cachep->gfpflags |= GFP_DMA;
+#endif
 	cachep->buffer_size = size;
 
 	if (flags & CFLGS_OFF_SLAB) {
Index: linux-2.6.18-rc6-mm1/include/linux/slab.h
=================================--- linux-2.6.18-rc6-mm1.orig/include/linux/slab.h	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/include/linux/slab.h	2006-09-11 16:45:01.931523561 -0500
@@ -72,7 +72,11 @@
 struct cache_sizes {
 	size_t		 cs_size;
 	kmem_cache_t	*cs_cachep;
+#ifdef CONFIG_ZONE_DMA
 	kmem_cache_t	*cs_dmacachep;
+#else
+#define cs_dmacachep cs_cachep
+#endif
 };
 extern struct cache_sizes malloc_sizes[];
 
Index: linux-2.6.18-rc6-mm1/mm/vmstat.c
=================================--- linux-2.6.18-rc6-mm1.orig/mm/vmstat.c	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/mm/vmstat.c	2006-09-11 16:45:01.940313371 -0500
@@ -437,6 +437,12 @@
 	.show	= frag_show,
 };
 
+#ifdef CONFIG_ZONE_DMA
+#define TEXT_FOR_DMA(xx) xx "_dma",
+#else
+#define TEXT_FOR_DMA(xx)
+#endif
+
 #ifdef CONFIG_ZONE_DMA32
 #define TEXT_FOR_DMA32(xx) xx "_dma32",
 #else
@@ -449,7 +455,7 @@
 #define TEXT_FOR_HIGHMEM(xx)
 #endif
 
-#define TEXTS_FOR_ZONES(xx) xx "_dma", TEXT_FOR_DMA32(xx) xx "_normal", \
+#define TEXTS_FOR_ZONES(xx) TEXT_FOR_DMA(xx) TEXT_FOR_DMA32(xx) xx "_normal", \
 					TEXT_FOR_HIGHMEM(xx)
 
 static char *vmstat_text[] = {
Index: linux-2.6.18-rc6-mm1/include/linux/vmstat.h
=================================--- linux-2.6.18-rc6-mm1.orig/include/linux/vmstat.h	2006-09-11 16:44:16.000000000 -0500
+++ linux-2.6.18-rc6-mm1/include/linux/vmstat.h	2006-09-11 16:45:01.949103180 -0500
@@ -17,6 +17,12 @@
  * generated will simply be the increment of a global address.
  */
 
+#ifdef CONFIG_ZONE_DMA
+#define DMA_ZONE(xx) xx##_DMA,
+#else
+#define DMA_ZONE(xx)
+#endif
+
 #ifdef CONFIG_ZONE_DMA32
 #define DMA32_ZONE(xx) xx##_DMA32,
 #else
@@ -29,7 +35,7 @@
 #define HIGHMEM_ZONE(xx)
 #endif
 
-#define FOR_ALL_ZONES(xx) xx##_DMA, DMA32_ZONE(xx) xx##_NORMAL HIGHMEM_ZONE(xx)
+#define FOR_ALL_ZONES(xx) DMA_ZONE(xx) DMA32_ZONE(xx) xx##_NORMAL HIGHMEM_ZONE(xx)
 
 enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT,
 		FOR_ALL_ZONES(PGALLOC),
@@ -88,7 +94,8 @@
 #endif /* CONFIG_VM_EVENT_COUNTERS */
 
 #define __count_zone_vm_events(item, zone, delta) \
-			__count_vm_events(item##_DMA + zone_idx(zone), delta)
+		__count_vm_events(item##_NORMAL - ZONE_NORMAL + \
+		zone_idx(zone), delta)
 
 /*
  * Zone based page accounting with per cpu differentials.
@@ -135,14 +142,16 @@
 	struct zone *zones = NODE_DATA(node)->node_zones;
 
 	return
+#ifdef CONFIG_ZONE_DMA
+		zone_page_state(&zones[ZONE_DMA], item) +
+#endif
 #ifdef CONFIG_ZONE_DMA32
 		zone_page_state(&zones[ZONE_DMA32], item) +
 #endif
-		zone_page_state(&zones[ZONE_NORMAL], item) +
 #ifdef CONFIG_HIGHMEM
 		zone_page_state(&zones[ZONE_HIGHMEM], item) +
 #endif
-		zone_page_state(&zones[ZONE_DMA], item);
+		zone_page_state(&zones[ZONE_NORMAL], item);
 }
 
 extern void zone_statistics(struct zonelist *, struct zone *);

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Nick Piggin]

Christoph Lameter wrote:
> Make ZONE_DMA optional in core code.
> 
> - ifdef definitions for ZONE_DMA and related code following
>   the example for ZONE_DMA32 and ZONE_HIGHMEM.
> 
> - Without ZONE_DMA, ZONE_HIGHMEM and ZONE_DMA32 we get to a
>   ZONES_SHIFT of 0.
> 
> - Modify the VM statistics to work correctly without a DMA
>   zone.
> 
> - Modify slab to not create DMA slabs if there is no
>   ZONE_DMA.
> 
> Signed-off-by: Christoph Lameter <clameter@sgi.com>

Nice. Looks like it should remove a bit of overhead.

I can't see from your patches, but what happens if someone asks
for a GFP_DMA page or dma slab allocation when there is no ZONE_DMA?

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Christoph Lameter]

On Tue, 12 Sep 2006, Nick Piggin wrote:

> I can't see from your patches, but what happens if someone asks
> for a GFP_DMA page or dma slab allocation when there is no ZONE_DMA?

The page/slab allocator will simply ignore the flag and return 
ZONE_NORMAL memory. See gfp_zone().

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Nick Piggin]

Christoph Lameter wrote:

>On Tue, 12 Sep 2006, Nick Piggin wrote:
>
>
>>I can't see from your patches, but what happens if someone asks
>>for a GFP_DMA page or dma slab allocation when there is no ZONE_DMA?
>>
>
>The page/slab allocator will simply ignore the flag and return 
>ZONE_NORMAL memory. See gfp_zone().
>

That's wrong, I think? It should return NULL surely? (and possibly do
a printk telling the user they should enable ZONE_DMA config, although
some users have valid fallback strategies).

--

Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Andi Kleen]

> So I totally agree, GFP_DMA allocations really ought to fail in this
> situation and we should fixup the users instead.

It's just impossible to fix them without having a bounce buffer.
The only way would be to not use the hardware. I guess 
we can find out how many people still use floppies and other
such hardware by having it for some time in -mm*.

But I reserve the right that if this thing becomes a FAQ - e.g. 
various people set the option wrong and break their systems
to rip it out again. I'm not interested at all in patches that increase
the bogus bug report rate. 

-Andi

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Jes Sorensen]

>>>>> "Nick" = Nick Piggin <nickpiggin@yahoo.com.au> writes:

Nick> Christoph Lameter wrote:
>> On Tue, 12 Sep 2006, Nick Piggin wrote:
>>> I can't see from your patches, but what happens if someone asks
>>> for a GFP_DMA page or dma slab allocation when there is no
>>> ZONE_DMA?
>>> 
>>  The page/slab allocator will simply ignore the flag and return
>> ZONE_NORMAL memory. See gfp_zone().

Nick> That's wrong, I think? It should return NULL surely? (and
Nick> possibly do a printk telling the user they should enable
Nick> ZONE_DMA config, although some users have valid fallback
Nick> strategies).

That was what hit my mind first as well when I saw these patches. The
other issue is what about 64 bit machines that are trying to allocate
32 bit addresses because they are either out of IOMMU slots or needs
to allocate bounce buffers for swiotlb support?  Silently trying to
feed a device a 64 bit address could have some 'interesting' side
effects.

So I totally agree, GFP_DMA allocations really ought to fail in this
situation and we should fixup the users instead.

Cheers,
Jes

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Jes Sorensen]

Andi Kleen wrote:
>> So I totally agree, GFP_DMA allocations really ought to fail in this
>> situation and we should fixup the users instead.
> 
> It's just impossible to fix them without having a bounce buffer.
> The only way would be to not use the hardware. I guess 
> we can find out how many people still use floppies and other
> such hardware by having it for some time in -mm*.

Actually I was somehow unclear in my mail. You are right about the 24bit
zone, but there is also the issue of the 32bit zone to deal with, which
is what I had in mind. This is still common, even if things like ISA and
floppies are history.

> But I reserve the right that if this thing becomes a FAQ - e.g. 
> various people set the option wrong and break their systems
> to rip it out again. I'm not interested at all in patches that increase
> the bogus bug report rate. 

Nod, I am worried if we get to a state with random bug reports or 32bit
cards not working because of this. You may be able to explain it to
users one at a time, but I really don't feel like spending my time doing
that :)

Cheers,
Jes

Re: [PATCH 3/6] Optional ZONE_DMA in the VM

[Christoph Lameter]

On Tue, 12 Sep 2006, Nick Piggin wrote:

> Christoph Lameter wrote:
> > On Tue, 12 Sep 2006, Nick Piggin wrote:
> > > I can't see from your patches, but what happens if someone asks
> > > for a GFP_DMA page or dma slab allocation when there is no ZONE_DMA?
> > The page/slab allocator will simply ignore the flag and return ZONE_NORMAL
> > memory. See gfp_zone().
> That's wrong, I think? It should return NULL surely? (and possibly do
> a printk telling the user they should enable ZONE_DMA config, although
> some users have valid fallback strategies).

Ignoring GFP_DMA is consistent with how GFP_DMA32 and GFP_HIGHMEM ar 
treated.

There are subsystems that sporadically set GFP_DMA and the earlier 
patchset was so full of #ifdefs because it attempted to cover all cases.

ZONE_DMA must be enabled if a device drivers relies on ISA DMA or 
GENERIC DMA. Otherwise we do not allow the selection of the driver.

We could add a BUG in gfp_zone which will give us all locations.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Arjan van de Ven]

On Mon, 2006-09-11 at 15:27 -0700, Christoph Lameter wrote:
> Optional ZONE_DMA
> 
> This patch follows up on the earlier work in Andrew's tree to reduce
> the number of zones. The patches allow to go to a minimum of 2 zones.
> This one allows also to make ZONE_DMA optional and therefore the
> number of zones can be reduced to one.
> 
> ZONE_DMA is usually used for ISA DMA devices. Typically modern hardware
> does not have any of these anymore. 

in the x86 / x86-64 world that statement is not correct ;(

while there is no physical bus there is still too much "be exactly as
if" stuff around.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jack Steiner]

On Mon, Sep 11, 2006 at 03:27:29PM -0700, Christoph Lameter wrote:
> Optional ZONE_DMA
> 
> This patch follows up on the earlier work in Andrew's tree to reduce
> the number of zones. The patches allow to go to a minimum of 2 zones.
> This one allows also to make ZONE_DMA optional and therefore the
> number of zones can be reduced to one.
> 
> ZONE_DMA is usually used for ISA DMA devices. Typically modern hardware
> does not have any of these anymore. So we frequently do not need
> the zone anymore. The presence of an additional zone unnecessarily
> complicates VM operations. It must be scanned and balancing logic
> must operate in it etc etc. If one has a 1-1 correspondence between
> zones and nodes in a NUMA system then various other optimizations
> become possible.
> 
> Many systems today (especially 64 bit but also 32 bit machines with less
> than 4G of memory) can therefore operate just fine with a single zone.
> With a single zone various loops can be optimized away by the
> compiler. Many system currently do not place anything in ZONE_DMA. On
> most of my systems ZONE_DMA is completely empty. Why constantly look
> at an empty zone in /proc/zoneinfo and empty slab in /proc/slabinfo?
> Non i386 also frequently have no need for ZONE_DMA and zones stay
> empty.
> 
I'm missing something here. On Altix, currently ALL of the memory is reported
as being in the DMA zone:

	% cat /proc/budd*
	Node 0, zone      DMA   3015    116      4      1    ...
	Node 1, zone      DMA   4243    355     15      3    ...
	Node 2, zone      DMA   4384    113      6      4    ...

	% cat /proc/zoneinfo
	Node 0, zone      DMA
	  pages free     5868
	  ...

The DMA slabs are empty, though.

I assume your patch will switch Altix memory to a different zone, but what happens when
drivers specify GFP_DMA. 

-- jack

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Tue, 12 Sep 2006, Arjan van de Ven wrote:

> > ZONE_DMA is usually used for ISA DMA devices. Typically modern hardware
> > does not have any of these anymore. 
> 
> in the x86 / x86-64 world that statement is not correct ;(

It is true for a large range of hardware. My x86 x64 systems do not need 
the DMA zone and I have been running them for awhile in this mode. Some 
drivers are not available as a result of not having ZONE_DMA.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Tue, 12 Sep 2006, Jack Steiner wrote:

> I'm missing something here. On Altix, currently ALL of the memory is reported
> as being in the DMA zone:
> 
> 	% cat /proc/budd*
> 	Node 0, zone      DMA   3015    116      4      1    ...
> 	Node 1, zone      DMA   4243    355     15      3    ...
> 	Node 2, zone      DMA   4384    113      6      4    ...
> 
> 	% cat /proc/zoneinfo
> 	Node 0, zone      DMA
> 	  pages free     5868
> 	  ...
> 
> The DMA slabs are empty, though.

This is wrong. All memory should be in ZONE_NORMAL since we have no DMA 
restrictions on Altix.

> I assume your patch will switch Altix memory to a different zone, but what happens when
> drivers specify GFP_DMA. 

GFP_DMA is ignored if there is no ZONE_DMA, same as GFP_DMA32 and 
GFP_HIGHMEM.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Martin Bligh]

Resending. Your outbound email address is invalid
(Christoph Lameter <christoph@engr.sgi.com>), as is
the address for linux-mm


Christoph Lameter wrote:
> On Tue, 12 Sep 2006, Jack Steiner wrote:
> 
> 
>>I'm missing something here. On Altix, currently ALL of the memory is reported
>>as being in the DMA zone:
>>
>>	% cat /proc/budd*
>>	Node 0, zone      DMA   3015    116      4      1    ...
>>	Node 1, zone      DMA   4243    355     15      3    ...
>>	Node 2, zone      DMA   4384    113      6      4    ...
>>
>>	% cat /proc/zoneinfo
>>	Node 0, zone      DMA
>>	  pages free     5868
>>	  ...
>>
>>The DMA slabs are empty, though.
> 
> 
> This is wrong. All memory should be in ZONE_NORMAL since we have no DMA 
> restrictions on Altix.

PPC64 works the same way, I believe. All memory is DMA'able, therefore
it all fits in ZONE_DMA.

The real problem is that there's no consistent definition of what the
zones actually mean.

1. Is it DMA'able (this is stupid, as it doesn't say 'for what device'
2. Is it permanently mapped into kernel address space.

Given an inconsistent set of questions, it is unsuprising that we come
up with an inconsistent set of answers. We're trying to answer a 2D
question with a 1D answer.

What is really needed is to pass a physical address limit from the
caller, together with a flag that says whether the memory needs to be
mapped into the permanent kernel address space or not. The allocator
then finds the set of zones that will fulfill this criteria.
But I suspect this level of change will cause too many people to squeak
loudly.

M.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Tue, 12 Sep 2006, Martin Bligh wrote:

> > This is wrong. All memory should be in ZONE_NORMAL since we have no DMA
> > restrictions on Altix.
> 
> PPC64 works the same way, I believe. All memory is DMA'able, therefore
> it all fits in ZONE_DMA.

ZONE_DMA is for broken/limited DMA controllers not for DMA controllers 
that can reach all of memory.
 
> The real problem is that there's no consistent definition of what the
> zones actually mean.

ZONE_DMA 	Special memory area for DMA controllers that can only
			do dma to a restricted memory area.

ZONE_DMA32	Second special memory area for DMA controllers that
		can only do dma to a restricted memory area that
		is different from ZONE_DMA

ZONE_NORMAL	Regular memory

ZONE_HIGHEM	Memory requires being mapped into kernel address space.


> 1. Is it DMA'able (this is stupid, as it doesn't say 'for what device'

That is *not* what ZONE_DMA means. We have always supported DMA to 
regular  memory.

> What is really needed is to pass a physical address limit from the
> caller, together with a flag that says whether the memory needs to be
> mapped into the permanent kernel address space or not. The allocator
> then finds the set of zones that will fulfill this criteria.
> But I suspect this level of change will cause too many people to squeak
> loudly.

Actually we could do this with the proposed change of passing an 
allocation_control struct instead of gfpflags to the allocator functions. 
See the discussion on linux-mm.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Martin Bligh]

Christoph Lameter wrote:
> On Tue, 12 Sep 2006, Jack Steiner wrote:
> 
> 
>>I'm missing something here. On Altix, currently ALL of the memory is reported
>>as being in the DMA zone:
>>
>>	% cat /proc/budd*
>>	Node 0, zone      DMA   3015    116      4      1    ...
>>	Node 1, zone      DMA   4243    355     15      3    ...
>>	Node 2, zone      DMA   4384    113      6      4    ...
>>
>>	% cat /proc/zoneinfo
>>	Node 0, zone      DMA
>>	  pages free     5868
>>	  ...
>>
>>The DMA slabs are empty, though.
> 
> 
> This is wrong. All memory should be in ZONE_NORMAL since we have no DMA 
> restrictions on Altix.

PPC64 works the same way, I believe. All memory is DMA'able, therefore 
it all fits in ZONE_DMA.

The real problem is that there's no consistent definition of what the
zones actually mean.

1. Is it DMA'able (this is stupid, as it doesn't say 'for what device'
2. Is it permanently mapped into kernel address space.

Given an inconsistent set of questions, it is unsuprising that we come
up with an inconsistent set of answers. We're trying to answer a 2D
question with a 1D answer.

What is really needed is to pass a physical address limit from the
caller, together with a flag that says whether the memory needs to be
mapped into the permanent kernel address space or not. The allocator
then finds the set of zones that will fulfill this criteria.
But I suspect this level of change will cause too many people to squeak
loudly.

M.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Nick Piggin]

Martin Bligh wrote:

> What is really needed is to pass a physical address limit from the
> caller, together with a flag that says whether the memory needs to be
> mapped into the permanent kernel address space or not. The allocator
> then finds the set of zones that will fulfill this criteria.
> But I suspect this level of change will cause too many people to squeak
> loudly.

That's a good point and it will be something to keep in mind if we're
going to rework the allocator API.

Whether or not people will squeak, having callers say explicitly what
type of memory they want will be much easier to audit/verify and more
flexible in the long term (eg. allow a best effort allocation for odd
sized masks, perhaps).

Some sort of compatiblity (gfp flag -> address) mapping would be
required to support legacy allocator APIs, but I don't think that
would be a problem.

-- 
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com 

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jes Sorensen]

>>>>> "Christoph" = Christoph Lameter <christoph@engr.sgi.com> writes:

Christoph> On Tue, 12 Sep 2006, Jack Steiner wrote:
>> I'm missing something here. On Altix, currently ALL of the memory
>> is reported as being in the DMA zone:
>> 
>> % cat /proc/budd* Node 0, zone DMA 3015 116 4 1 ...  Node 1, zone
>> DMA 4243 355 15 3 ...  Node 2, zone DMA 4384 113 6 4 ...
>> 
>> % cat /proc/zoneinfo Node 0, zone DMA pages free 5868 ...
>> 
>> The DMA slabs are empty, though.

Christoph> This is wrong. All memory should be in ZONE_NORMAL since we
Christoph> have no DMA restrictions on Altix.

This is harmless on Altix because we have an IOMMU - however we should
in theory put all memory below 4GB into the DMA32 zone (if there was
any memory there :) to use if we ever ran out of IOMMU slots when
running 32 bit only capable PCI devices. I've never seen this happen
so it's purely theoretical.

Cheers,
Jes

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jes Sorensen]

>>>>> "Christoph" = Christoph Lameter <clameter@sgi.com> writes:

Christoph> It is true for a large range of hardware. My x86 x64
Christoph> systems do not need the DMA zone and I have been running
Christoph> them for awhile in this mode. Some drivers are not
Christoph> available as a result of not having ZONE_DMA.

Any idea which ones?

Jes

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Wed, 13 Sep 2006, Jes Sorensen wrote:

> >>>>> "Christoph" = Christoph Lameter <clameter@sgi.com> writes:
> 
> Christoph> It is true for a large range of hardware. My x86 x64
> Christoph> systems do not need the DMA zone and I have been running
> Christoph> them for awhile in this mode. Some drivers are not
> Christoph> available as a result of not having ZONE_DMA.
> 
> Any idea which ones?

There was the floppy driver and one type of USB stick that I noticed 
during the work on the project. But other drivers may depend also depend 
indirectly on DMA functionality and may also be disabled.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jes Sorensen]

Christoph Lameter wrote:
> On Wed, 13 Sep 2006, Jes Sorensen wrote:
> 
>>>>>>> "Christoph" = Christoph Lameter <clameter@sgi.com> writes:
>> Christoph> It is true for a large range of hardware. My x86 x64
>> Christoph> systems do not need the DMA zone and I have been running
>> Christoph> them for awhile in this mode. Some drivers are not
>> Christoph> available as a result of not having ZONE_DMA.
>>
>> Any idea which ones?
> 
> There was the floppy driver and one type of USB stick that I noticed 
> during the work on the project. But other drivers may depend also depend 
> indirectly on DMA functionality and may also be disabled.

Ok, USB should ring alarm bells, floppy I think is less relevant these
days :)

Cheers,
Jes

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Wed, 13 Sep 2006, Jes Sorensen wrote:

> > There was the floppy driver and one type of USB stick that I noticed 
> > during the work on the project. But other drivers may depend also depend 
> > indirectly on DMA functionality and may also be disabled.
> 
> Ok, USB should ring alarm bells, floppy I think is less relevant these
> days :)

If you want all drivers then you must of course have ZONE_DMA. 
Distributions that want to cover all drivers will have it on by default 
and ZONE_DMA is available by default.

However, if you want to create a lean and mean kernel then you can switch 
ZONE_DMA off and if there is just one zone left then the VM can 
optimized much better because loops are avoided and some macros 
become constant etc etc.

Some architectures never need ZONE_DMA because all hardware supports DMA 
to all of memory. SGI Altix is one example. Carrying an additional 
useless zone around unecessarily bloats the kernel both in term of code 
and data. Data is a particular issue since zones contain per cpu elements. 
For a 1k cpu 1k node configuration this saves around 1 million per cpu 
structures (one zone per node with 1k per cpu pagesets).

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jack Steiner]

On Wed, Sep 13, 2006 at 10:23:10AM -0700, Christoph Lameter wrote:
> On Wed, 13 Sep 2006, Jes Sorensen wrote:
> 
> > > There was the floppy driver and one type of USB stick that I noticed 
> > > during the work on the project. But other drivers may depend also depend 
> > > indirectly on DMA functionality and may also be disabled.
> > 
> > Ok, USB should ring alarm bells, floppy I think is less relevant these
> > days :)
> 
> If you want all drivers then you must of course have ZONE_DMA. 
> Distributions that want to cover all drivers will have it on by default 
> and ZONE_DMA is available by default.
> 
> However, if you want to create a lean and mean kernel then you can switch 
> ZONE_DMA off and if there is just one zone left then the VM can 
> optimized much better because loops are avoided and some macros 
> become constant etc etc.
> 
> Some architectures never need ZONE_DMA because all hardware supports DMA 
> to all of memory. SGI Altix is one example. Carrying an additional 
> useless zone around unecessarily bloats the kernel both in term of code 
> and data. Data is a particular issue since zones contain per cpu elements. 
> For a 1k cpu 1k node configuration this saves around 1 million per cpu 
> structures (one zone per node with 1k per cpu pagesets).
> 

Most distros release GENERIC kernels for IA64. If _any_ IA64 platform requires 
ZONE_DMA, then it must be configured ON.

Two questions:

	- will any IA64 platform require that ZONE_DMA be enabled (I think
	  the answer is "yes")

	- if ZONE_DMA is enabled, ALTIX will still use only 1 zone. In your
	  statement above, you say that disabling ZONE_DMA save 1M cpu
	  structures. If ZONE_DMA is enabled, will these 1M structure be allocated
	  on SN even though they are not needed?

-- jack

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Wed, 13 Sep 2006, Jack Steiner wrote:

> 	- will any IA64 platform require that ZONE_DMA be enabled (I think
> 	  the answer is "yes")

I think that still remains to be discussed. Today there are various means
to get around address restrictions.

> 	- if ZONE_DMA is enabled, ALTIX will still use only 1 zone. In your
> 	  statement above, you say that disabling ZONE_DMA save 1M cpu
> 	  structures. If ZONE_DMA is enabled, will these 1M structure be allocated
> 	  on SN even though they are not needed?

That has always been the case. In SLES9 and SLES10 we are using 1/4th of the pagesets.

Having no ZONE_DMA provides additional benefits besides saving memory. The 
VM balancing of allocations within a node becomes not necessary and 
various VM optimization can be performed. Loops are unrolled. Less zones 
have to be processed for draining / vm counter updates etc etc.

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Jes Sorensen]

Christoph Lameter wrote:
> On Wed, 13 Sep 2006, Jack Steiner wrote:
> 
>> 	- will any IA64 platform require that ZONE_DMA be enabled (I think
>> 	  the answer is "yes")
> 
> I think that still remains to be discussed. Today there are various means
> to get around address restrictions.

I don't know about USB on ia64, but USB is an issue and we do support
it even on Altix, as crazy as it may seem (I use USB with my SGI Prism
foot-warmer in the office). Also take into account that some ia64 boxes
do not come with IOMMU's, DIG - be afraid, be very afraid. On those
machines you ideally want to have DMA32 zone for this stuff to support
32 bit PCI devices, even if the swiotlb can be used (bounce buffers for
all transactions is just a sick idea), and we get back to the issue of
using generic kernels.

>> 	- if ZONE_DMA is enabled, ALTIX will still use only 1 zone. In your
>> 	  statement above, you say that disabling ZONE_DMA save 1M cpu
>> 	  structures. If ZONE_DMA is enabled, will these 1M structure be allocated
>> 	  on SN even though they are not needed?
> 
> That has always been the case. In SLES9 and SLES10 we are using 1/4th of the pagesets.
> 
> Having no ZONE_DMA provides additional benefits besides saving memory. The 
> VM balancing of allocations within a node becomes not necessary and 
> various VM optimization can be performed. Loops are unrolled. Less zones 
> have to be processed for draining / vm counter updates etc etc.

I agree it sounds appealing, but if reality is that all distro kernels
will switch ZONE_DMA on, then having the option to switch it off is
going have little or zero impact on the end users.

In other words, will this really matter in end user situations?

Cheers,
Jes

Re: [PATCH 0/8] Optional ZONE_DMA V1

[Christoph Lameter]

On Thu, 14 Sep 2006, Jes Sorensen wrote:

> I don't know about USB on ia64, but USB is an issue and we do support
> it even on Altix, as crazy as it may seem (I use USB with my SGI Prism
> foot-warmer in the office). Also take into account that some ia64 boxes
> do not come with IOMMU's, DIG - be afraid, be very afraid. On those
> machines you ideally want to have DMA32 zone for this stuff to support
> 32 bit PCI devices, even if the swiotlb can be used (bounce buffers for
> all transactions is just a sick idea), and we get back to the issue of
> using generic kernels.

USB sticks that use ISA DMA is an issue but then IA64 does not 
support ISA DMA at all and would not even now support that USB stick type.

> I agree it sounds appealing, but if reality is that all distro kernels
> will switch ZONE_DMA on, then having the option to switch it off is
> going have little or zero impact on the end users.

I am sure that if we keep ZONE_DMA unconfigurable then the distros will 
never switch that off because they cannot. On the other hand if its 
optional then it can be switched off at some future date or special 
kernels can be build if this will turn out to be a big advantage.

Also not everyone (even we have the capability of generatic static SGI 
specific kernels) uses only distro kernels and this is a big memory saver 
and reduces complexity in the kernel with only a single zone.
 
> In other words, will this really matter in end user situations?

Certainly it will never have a change of mattering if we keep the 
chicken-and-egg argument going.