x86_64: 2.6.14-rc4 swiotlb broken

x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On x86_64 NUMA boxes, the revert
http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6e3254c4e2927c117044a02acf5f5b56e1373053
meant that swiotlb gets the IOTLB
memory from pages over 4G (if mem > 4G), which basically renders swiotlb useless, causing
breakage with devices not capable of DMA beyond 4G.  2.6.13 was (kinda) not
broken, although the patch titled "Reverse order of bootmem lists" was
not in 2.6.13, The reason is commit
http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6142891a0c0209c91aa4a98f725de0d6e2ed4918
was not in 2.6.13, PCI_DMA_BUS_IS_PHYS was 1 when no mmu was present, and the block layer did 
the bouncing, never using swiotlb.  I guess the right fix is to make sure
swiotlb gets the right memory.  Here is a patch doing that.  Tested on IBM
x460.  I hope the patch is ok for ia64s too.  I do not have access to ia64
boxen.

Thanks,
Kiran

Signed-off-by: Shai Fultheim <shai@scalex86.org>
Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>
 
Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-14 00:06:21.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 00:05:22.000000000 -0700
@@ -123,7 +123,7 @@
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
+	io_tlb_start = alloc_bootmem_node(NODE_DATA(0), io_tlb_nslabs *
 					       (1 << IO_TLB_SHIFT));
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andrew Morton]

Ravikiran G Thirumalai <kiran@scalex86.org> wrote:
>
> On x86_64 NUMA boxes, the revert
> http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6e3254c4e2927c117044a02acf5f5b56e1373053
> meant that swiotlb gets the IOTLB
> memory from pages over 4G (if mem > 4G), which basically renders swiotlb useless, causing
> breakage with devices not capable of DMA beyond 4G.  2.6.13 was (kinda) not
> broken, although the patch titled "Reverse order of bootmem lists" was
> not in 2.6.13, The reason is commit
> http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6142891a0c0209c91aa4a98f725de0d6e2ed4918
> was not in 2.6.13, PCI_DMA_BUS_IS_PHYS was 1 when no mmu was present, and the block layer did 
> the bouncing, never using swiotlb.  I guess the right fix is to make sure
> swiotlb gets the right memory.  Here is a patch doing that.  Tested on IBM
> x460.  I hope the patch is ok for ia64s too.  I do not have access to ia64
> boxen.
> 

This is an ia64 patch - what point was there in testing it on an x460?

Is something missing here?

>  
> Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
> ===================================================================
> --- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-14 00:06:21.000000000 -0700
> +++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 00:05:22.000000000 -0700
> @@ -123,7 +123,7 @@
>  	/*
>  	 * Get IO TLB memory from the low pages
>  	 */
> -	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
> +	io_tlb_start = alloc_bootmem_node(NODE_DATA(0), io_tlb_nslabs *
>  					       (1 << IO_TLB_SHIFT));
>  	if (!io_tlb_start)
>  		panic("Cannot allocate SWIOTLB buffer");

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 11:50, Andrew Morton wrote:

>
> This is an ia64 patch - what point was there in testing it on an x460?
>
> Is something missing here?

x86-64 shares that code with ia64.

The patch is actually not quite correct - in theory node 0 could be too small 
to contain the full swiotlb bounce buffers.

The real fix would be to get rid of the pgdata lists and just walk the 
node_online_map on bootmem.c. The memory hotplug guys have
a patch pending for this.

-Andi

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

Hello. Ravikiran-san.

> > This is an ia64 patch - what point was there in testing it on an x460?
> >
> > Is something missing here?
> 
> x86-64 shares that code with ia64.
> 
> The patch is actually not quite correct - in theory node 0 could be too small 
> to contain the full swiotlb bounce buffers.
> 
> The real fix would be to get rid of the pgdata lists and just walk the 
> node_online_map on bootmem.c. The memory hotplug guys have
> a patch pending for this.

Yeah!
I posted a patch for this problem to linux-mm ML. Could you try it? 
http://marc.theaimsgroup.com/?l=linux-mm&m=112791558527522&w=2

2.6.14-rc4-mm1 already has merged it. ;-)

Thanks.

-- 
Yasunori Goto 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Andi Kleen wrote:
> 
> The patch is actually not quite correct - in theory node 0 could be too small 
> to contain the full swiotlb bounce buffers.

Is node 0 guaranteed to be all low-memory? What if it allocates stuff at 
the end of memory on NODE(0)?

Anyway, it sounds like "alloc_bootmem_low_pages()" is seriously buggered 
if it allocates non-low pages, if only because of its name...

> The real fix would be to get rid of the pgdata lists and just walk the 
> node_online_map on bootmem.c. The memory hotplug guys have
> a patch pending for this.

Argh. Which one should I pick? The NODE(0) one looks simpler, but is it 
sufficient for now in practice (with the real one going into 2.6.14+)?

		Linus

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Mon, Oct 17, 2005 at 08:27:40AM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 17 Oct 2005, Andi Kleen wrote:
> > 
> > The patch is actually not quite correct - in theory node 0 could be too small 
> > to contain the full swiotlb bounce buffers.
> 
> Is node 0 guaranteed to be all low-memory? What if it allocates stuff at 
> the end of memory on NODE(0)?
> 
> Anyway, it sounds like "alloc_bootmem_low_pages()" is seriously buggered 
> if it allocates non-low pages, if only because of its name...
> 
> > The real fix would be to get rid of the pgdata lists and just walk the 
> > node_online_map on bootmem.c. The memory hotplug guys have
> > a patch pending for this.
> 
> Argh. Which one should I pick? The NODE(0) one looks simpler, but is it 
> sufficient for now in practice (with the real one going into 2.6.14+)?

That's the reason I made a small patch.  It does work on the boxes we use.
But I like Yasunori-san's patch.  Mine is just a chewing gum fix.

Thanks,
Kiran 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 17:27, Linus Torvalds wrote:
> On Mon, 17 Oct 2005, Andi Kleen wrote:
> > The patch is actually not quite correct - in theory node 0 could be too
> > small to contain the full swiotlb bounce buffers.
>
> Is node 0 guaranteed to be all low-memory? What if it allocates stuff at
> the end of memory on NODE(0)?

This is 64bit ... only low memory.

>
> Anyway, it sounds like "alloc_bootmem_low_pages()" is seriously buggered
> if it allocates non-low pages, if only because of its name...

The pages are all low, just at the wrong end of the memory.

> > The real fix would be to get rid of the pgdata lists and just walk the
> > node_online_map on bootmem.c. The memory hotplug guys have
> > a patch pending for this.
>
> Argh. Which one should I pick? The NODE(0) one looks simpler, but is it
> sufficient for now in practice (with the real one going into 2.6.14+)?
>
> 		Linus

First this problem is definitely not critical. AFAIK it only happens on 
scalex's unreleased machines. Intel NUMA x86 machines are really rare
and on AMD it doesn't happen because the swiotlb is not used there.
So just ignoring it is a quite reasonable option.

Both NODE(0) and node_online_map are risky. NODE(0) may break IA64
(they share this code) and node_online_map may break one of the weirder
ARM platforms again (for which the original revert was done)

For 2.6.14 I think it's best to ignore it and use node_online_map for 2.6.15.

-Andi 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Muli Ben-Yehuda]

On Mon, Oct 17, 2005 at 05:40:56PM +0200, Andi Kleen wrote:

> First this problem is definitely not critical. AFAIK it only happens on 
> scalex's unreleased machines. Intel NUMA x86 machines are really rare
> and on AMD it doesn't happen because the swiotlb is not used there.

It's not used by default, but there are cases where it's used and it
would be a shame to release a major kernel and knowingly break
them. For example, any setup that used iommu_force or any non-AMD
x86-64 machine with more than 4GB of memory and only 32-bit capable
DMA devices.

> Both NODE(0) and node_online_map are risky. NODE(0) may break IA64
> (they share this code) and node_online_map may break one of the weirder
> ARM platforms again (for which the original revert was done)

I don't have an IA64 machine, but if the NODE(0) fix is safe there, I
vote for it for 2.6.14-rc4. Another alternative is to temporarily
provide a different version of swiotlb_init() for x86-64 and IA64 -
I can whip up a patch if that's acceptable.

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 17:56, Muli Ben-Yehuda wrote:
> On Mon, Oct 17, 2005 at 05:40:56PM +0200, Andi Kleen wrote:
> > First this problem is definitely not critical. AFAIK it only happens on
> > scalex's unreleased machines. Intel NUMA x86 machines are really rare
> > and on AMD it doesn't happen because the swiotlb is not used there.
>
> It's not used by default, but there are cases where it's used and it
> would be a shame to release a major kernel and knowingly break
> them. For example, any setup that used iommu_force or any non-AMD
> x86-64 machine with more than 4GB of memory and only 32-bit capable
> DMA devices.

... but risk breaking other stuff. Unless you can get the ARM and/or IA64 
people to do some retesting with the proposed fixes it's quite risky.  
Sometimes you have to make compromises before releases.

> Another alternative is to temporarily
> provide a different version of swiotlb_init() for x86-64 and IA64 -
> I can whip up a patch if that's acceptable.

I don't want that.

-Andi

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Russell King]

On Mon, Oct 17, 2005 at 04:02:09PM +0200, Andi Kleen wrote:
> ... but risk breaking other stuff. Unless you can get the ARM and/or IA64 
> people to do some retesting with the proposed fixes it's quite risky.  
> Sometimes you have to make compromises before releases.

Since this issue came up, I have reworked ARM's memory initialisation
as below, which makes us independent of the order in which memory zones
are initialised.  However, this patch is more or less untested, and
does _not_ yet support XIP (which is pretty important for some people
on ARM.)  As such it isn't ready for mainline yet.

Please note that ARM has numerious differing memory setups, so it's
impossible for any one person to sufficiently test it - hence it's
definitely very early release cycle material only.

diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c
--- a/arch/arm/mm/init.c
+++ b/arch/arm/mm/init.c
@@ -86,15 +86,22 @@ void show_mem(void)
 	printk("%d pages swap cached\n", cached);
 }
 
-struct node_info {
-	unsigned int start;
-	unsigned int end;
-	int bootmap_pages;
-};
+static inline pmd_t *pmd_off(pgd_t *pgd, unsigned long virt)
+{
+	return pmd_offset(pgd, virt);
+}
+
+static inline pmd_t *pmd_off_k(unsigned long virt)
+{
+	return pmd_off(pgd_offset_k(virt), virt);
+}
 
-#define O_PFN_DOWN(x)	((x) >> PAGE_SHIFT)
 #define O_PFN_UP(x)	(PAGE_ALIGN(x) >> PAGE_SHIFT)
 
+#define for_each_nodebank(iter,mi,no)			\
+	for (iter = 0; iter < mi->nr_banks; iter++)	\
+		if (mi->bank[iter].node == no)
+
 /*
  * FIXME: We really want to avoid allocating the bootmap bitmap
  * over the top of the initrd.  Hopefully, this is located towards
@@ -109,12 +116,9 @@ find_bootmap_pfn(int node, struct meminf
 	start_pfn   = O_PFN_UP(__pa(&_end));
 	bootmap_pfn = 0;
 
-	for (bank = 0; bank < mi->nr_banks; bank ++) {
+	for_each_nodebank(bank, mi, node) {
 		unsigned int start, end;
 
-		if (mi->bank[bank].node != node)
-			continue;
-
 		start = mi->bank[bank].start >> PAGE_SHIFT;
 		end   = (mi->bank[bank].size +
 			 mi->bank[bank].start) >> PAGE_SHIFT;
@@ -140,92 +144,6 @@ find_bootmap_pfn(int node, struct meminf
 	return bootmap_pfn;
 }
 
-/*
- * Scan the memory info structure and pull out:
- *  - the end of memory
- *  - the number of nodes
- *  - the pfn range of each node
- *  - the number of bootmem bitmap pages
- */
-static unsigned int __init
-find_memend_and_nodes(struct meminfo *mi, struct node_info *np)
-{
-	unsigned int i, bootmem_pages = 0, memend_pfn = 0;
-
-	for (i = 0; i < MAX_NUMNODES; i++) {
-		np[i].start = -1U;
-		np[i].end = 0;
-		np[i].bootmap_pages = 0;
-	}
-
-	for (i = 0; i < mi->nr_banks; i++) {
-		unsigned long start, end;
-		int node;
-
-		if (mi->bank[i].size == 0) {
-			/*
-			 * Mark this bank with an invalid node number
-			 */
-			mi->bank[i].node = -1;
-			continue;
-		}
-
-		node = mi->bank[i].node;
-
-		/*
-		 * Make sure we haven't exceeded the maximum number of nodes
-		 * that we have in this configuration.  If we have, we're in
-		 * trouble.  (maybe we ought to limit, instead of bugging?)
-		 */
-		if (node >= MAX_NUMNODES)
-			BUG();
-		node_set_online(node);
-
-		/*
-		 * Get the start and end pfns for this bank
-		 */
-		start = mi->bank[i].start >> PAGE_SHIFT;
-		end   = (mi->bank[i].start + mi->bank[i].size) >> PAGE_SHIFT;
-
-		if (np[node].start > start)
-			np[node].start = start;
-
-		if (np[node].end < end)
-			np[node].end = end;
-
-		if (memend_pfn < end)
-			memend_pfn = end;
-	}
-
-	/*
-	 * Calculate the number of pages we require to
-	 * store the bootmem bitmaps.
-	 */
-	for_each_online_node(i) {
-		if (np[i].end == 0)
-			continue;
-
-		np[i].bootmap_pages = bootmem_bootmap_pages(np[i].end -
-							    np[i].start);
-		bootmem_pages += np[i].bootmap_pages;
-	}
-
-	high_memory = __va(memend_pfn << PAGE_SHIFT);
-
-	/*
-	 * This doesn't seem to be used by the Linux memory
-	 * manager any more.  If we can get rid of it, we
-	 * also get rid of some of the stuff above as well.
-	 *
-	 * Note: max_low_pfn and max_pfn reflect the number
-	 * of _pages_ in the system, not the maximum PFN.
-	 */
-	max_low_pfn = memend_pfn - O_PFN_DOWN(PHYS_OFFSET);
-	max_pfn = memend_pfn - O_PFN_DOWN(PHYS_OFFSET);
-
-	return bootmem_pages;
-}
-
 static int __init check_initrd(struct meminfo *mi)
 {
 	int initrd_node = -2;
@@ -266,9 +184,8 @@ static int __init check_initrd(struct me
 /*
  * Reserve the various regions of node 0
  */
-static __init void reserve_node_zero(unsigned int bootmap_pfn, unsigned int bootmap_pages)
+static __init void reserve_node_zero(pg_data_t *pgdat)
 {
-	pg_data_t *pgdat = NODE_DATA(0);
 	unsigned long res_size = 0;
 
 	/*
@@ -289,13 +206,6 @@ static __init void reserve_node_zero(uns
 			     PTRS_PER_PGD * sizeof(pgd_t));
 
 	/*
-	 * And don't forget to reserve the allocator bitmap,
-	 * which will be freed later.
-	 */
-	reserve_bootmem_node(pgdat, bootmap_pfn << PAGE_SHIFT,
-			     bootmap_pages << PAGE_SHIFT);
-
-	/*
 	 * Hmm... This should go elsewhere, but we really really need to
 	 * stop things allocating the low memory; ideally we need a better
 	 * implementation of GFP_DMA which does not assume that DMA-able
@@ -324,183 +234,276 @@ static __init void reserve_node_zero(uns
 		reserve_bootmem_node(pgdat, PHYS_OFFSET, res_size);
 }
 
-/*
- * Register all available RAM in this node with the bootmem allocator.
- */
-static inline void free_bootmem_node_bank(int node, struct meminfo *mi)
+void __init build_mem_type_table(void);
+void __init create_mapping(struct map_desc *md);
+
+static unsigned long __init
+bootmem_init_node(int node, int initrd_node, struct meminfo *mi)
 {
-	pg_data_t *pgdat = NODE_DATA(node);
-	int bank;
+	unsigned long zone_size[MAX_NR_ZONES], zhole_size[MAX_NR_ZONES];
+	unsigned long start_pfn, end_pfn, boot_pfn;
+	unsigned int boot_pages;
+	pg_data_t *pgdat;
+	int i;
 
-	for (bank = 0; bank < mi->nr_banks; bank++)
-		if (mi->bank[bank].node == node)
-			free_bootmem_node(pgdat, mi->bank[bank].start,
-					  mi->bank[bank].size);
-}
+	start_pfn = -1UL;
+	end_pfn = 0;
 
-/*
- * Initialise the bootmem allocator for all nodes.  This is called
- * early during the architecture specific initialisation.
- */
-static void __init bootmem_init(struct meminfo *mi)
-{
-	struct node_info node_info[MAX_NUMNODES], *np = node_info;
-	unsigned int bootmap_pages, bootmap_pfn, map_pg;
-	int node, initrd_node;
+	/*
+	 * Calculate the pfn range, and map the memory banks for this node.
+	 */
+	for_each_nodebank(i, mi, node) {
+		unsigned long start, end;
+		struct map_desc map;
 
-	bootmap_pages = find_memend_and_nodes(mi, np);
-	bootmap_pfn   = find_bootmap_pfn(0, mi, bootmap_pages);
-	initrd_node   = check_initrd(mi);
+		start = mi->bank[i].start >> PAGE_SHIFT;
+		end = (mi->bank[i].start + mi->bank[i].size) >> PAGE_SHIFT;
 
-	map_pg = bootmap_pfn;
+		if (start_pfn > start)
+			start_pfn = start;
+		if (end_pfn < end)
+			end_pfn = end;
+
+		map.physical = mi->bank[i].start;
+		map.virtual = __phys_to_virt(map.physical);
+		map.length = mi->bank[i].size;
+		map.type = MT_MEMORY;
+
+		create_mapping(&map);
+	}
 
 	/*
-	 * Initialise the bootmem nodes.
-	 *
-	 * What we really want to do is:
-	 *
-	 *   unmap_all_regions_except_kernel();
-	 *   for_each_node_in_reverse_order(node) {
-	 *     map_node(node);
-	 *     allocate_bootmem_map(node);
-	 *     init_bootmem_node(node);
-	 *     free_bootmem_node(node);
-	 *   }
-	 *
-	 * but this is a 2.5-type change.  For now, we just set
-	 * the nodes up in reverse order.
-	 *
-	 * (we could also do with rolling bootmem_init and paging_init
-	 * into one generic "memory_init" type function).
+	 * If there is no memory in this node, ignore it.
 	 */
-	np += num_online_nodes() - 1;
-	for (node = num_online_nodes() - 1; node >= 0; node--, np--) {
-		/*
-		 * If there are no pages in this node, ignore it.
-		 * Note that node 0 must always have some pages.
-		 */
-		if (np->end == 0 || !node_online(node)) {
-			if (node == 0)
-				BUG();
-			continue;
-		}
+	if (end_pfn == 0)
+		return end_pfn;
 
-		/*
-		 * Initialise the bootmem allocator.
-		 */
-		init_bootmem_node(NODE_DATA(node), map_pg, np->start, np->end);
-		free_bootmem_node_bank(node, mi);
-		map_pg += np->bootmap_pages;
+	/*
+	 * Allocate the bootmem bitmap page.
+	 */
+	boot_pages = bootmem_bootmap_pages(end_pfn - start_pfn);
+	boot_pfn = find_bootmap_pfn(node, mi, boot_pages);
 
-		/*
-		 * If this is node 0, we need to reserve some areas ASAP -
-		 * we may use bootmem on node 0 to setup the other nodes.
-		 */
-		if (node == 0)
-			reserve_node_zero(bootmap_pfn, bootmap_pages);
-	}
+	/*
+	 * Initialise the bootmem allocator for this node, handing the
+	 * memory banks over to bootmem.
+	 */
+	node_set_online(node);
+	pgdat = NODE_DATA(node);
+	init_bootmem_node(pgdat, boot_pfn, start_pfn, end_pfn);
 
+	for_each_nodebank(i, mi, node)
+		free_bootmem_node(pgdat, mi->bank[i].start, mi->bank[i].size);
+
+	/*
+	 * Reserve the bootmem bitmap for this node.
+	 */
+	reserve_bootmem_node(pgdat, boot_pfn << PAGE_SHIFT,
+			     boot_pages << PAGE_SHIFT);
 
 #ifdef CONFIG_BLK_DEV_INITRD
-	if (phys_initrd_size && initrd_node >= 0) {
-		reserve_bootmem_node(NODE_DATA(initrd_node), phys_initrd_start,
+	/*
+	 * If the initrd is in this node, reserve its memory.
+	 */
+	if (node == initrd_node) {
+		reserve_bootmem_node(pgdat, phys_initrd_start,
 				     phys_initrd_size);
 		initrd_start = __phys_to_virt(phys_initrd_start);
 		initrd_end = initrd_start + phys_initrd_size;
 	}
 #endif
 
-	BUG_ON(map_pg != bootmap_pfn + bootmap_pages);
+	/*
+	 * Finally, reserve any node zero regions.
+	 */
+	if (node == 0)
+		reserve_node_zero(pgdat);
+
+	/*
+	 * initialise the zones within this node.
+	 */
+	memset(zone_size, 0, sizeof(zone_size));
+	memset(zhole_size, 0, sizeof(zhole_size));
+
+	/*
+	 * The size of this node has already been determined.  If we need
+	 * to do anything fancy with the allocation of this memory to the
+	 * zones, now is the time to do it.
+	 */
+	zone_size[0] = end_pfn - start_pfn;
+
+	/*
+	 * For each bank in this node, calculate the size of the holes.
+	 *  holes = node_size - sum(bank_sizes_in_node)
+	 */
+	zhole_size[0] = zone_size[0];
+	for_each_nodebank(i, mi, node)
+		zhole_size[0] -= mi->bank[i].size >> PAGE_SHIFT;
+
+	/*
+	 * Adjust the sizes according to any special requirements for
+	 * this machine type.
+	 */
+	arch_adjust_zones(node, zone_size, zhole_size);
+
+	free_area_init_node(node, pgdat, zone_size, start_pfn, zhole_size);
+
+	return end_pfn;
 }
 
-/*
- * paging_init() sets up the page tables, initialises the zone memory
- * maps, and sets up the zero page, bad page and bad page tables.
- */
-void __init paging_init(struct meminfo *mi, struct machine_desc *mdesc)
+static void __init bootmem_init(struct meminfo *mi)
 {
-	void *zero_page;
-	int node;
+	unsigned long addr, memend_pfn = 0;
+	int node, initrd_node, i;
 
-	bootmem_init(mi);
+	/*
+	 * Invalidate the node number for empty or invalid memory banks
+	 */
+	for (i = 0; i < mi->nr_banks; i++)
+		if (mi->bank[i].size == 0 || mi->bank[i].node >= MAX_NUMNODES)
+			mi->bank[i].node = -1;
 
 	memcpy(&meminfo, mi, sizeof(meminfo));
 
+#ifdef CONFIG_XIP_KERNEL
+#error needs fixing
+	p->physical   = CONFIG_XIP_PHYS_ADDR & PMD_MASK;
+	p->virtual    = (unsigned long)&_stext & PMD_MASK;
+	p->length     = ((unsigned long)&_etext - p->virtual + ~PMD_MASK) & PMD_MASK;
+	p->type       = MT_ROM;
+	p ++;
+#endif
+
 	/*
-	 * allocate the zero page.  Note that we count on this going ok.
+	 * Clear out all the mappings below the kernel image.
+	 * FIXME: what about XIP?
 	 */
-	zero_page = alloc_bootmem_low_pages(PAGE_SIZE);
+	for (addr = 0; addr < PAGE_OFFSET; addr += PGDIR_SIZE)
+		pmd_clear(pmd_off_k(addr));
 
 	/*
-	 * initialise the page tables.
+	 * Clear out all the kernel space mappings, except for the first
+	 * memory bank, up to the end of the vmalloc region.
 	 */
-	memtable_init(mi);
-	if (mdesc->map_io)
-		mdesc->map_io();
-	local_flush_tlb_all();
+	for (addr = __phys_to_virt(mi->bank[0].start + mi->bank[0].size);
+	     addr < VMALLOC_END; addr += PGDIR_SIZE)
+		pmd_clear(pmd_off_k(addr));
 
 	/*
-	 * initialise the zones within each node
+	 * Locate which node contains the ramdisk image, if any.
 	 */
-	for_each_online_node(node) {
-		unsigned long zone_size[MAX_NR_ZONES];
-		unsigned long zhole_size[MAX_NR_ZONES];
-		struct bootmem_data *bdata;
-		pg_data_t *pgdat;
-		int i;
+	initrd_node = check_initrd(mi);
 
-		/*
-		 * Initialise the zone size information.
-		 */
-		for (i = 0; i < MAX_NR_ZONES; i++) {
-			zone_size[i]  = 0;
-			zhole_size[i] = 0;
-		}
+	/*
+	 * Run through each node initialising the bootmem allocator.
+	 */
+	for_each_node(node) {
+		unsigned long end_pfn;
 
-		pgdat = NODE_DATA(node);
-		bdata = pgdat->bdata;
+		end_pfn = bootmem_init_node(node, initrd_node, mi);
 
 		/*
-		 * The size of this node has already been determined.
-		 * If we need to do anything fancy with the allocation
-		 * of this memory to the zones, now is the time to do
-		 * it.
+		 * Remember the highest memory PFN.
 		 */
-		zone_size[0] = bdata->node_low_pfn -
-				(bdata->node_boot_start >> PAGE_SHIFT);
+		if (end_pfn > memend_pfn)
+			memend_pfn = end_pfn;
+	}
 
-		/*
-		 * If this zone has zero size, skip it.
-		 */
-		if (!zone_size[0])
-			continue;
+	high_memory = __va(memend_pfn << PAGE_SHIFT);
 
-		/*
-		 * For each bank in this node, calculate the size of the
-		 * holes.  holes = node_size - sum(bank_sizes_in_node)
-		 */
-		zhole_size[0] = zone_size[0];
-		for (i = 0; i < mi->nr_banks; i++) {
-			if (mi->bank[i].node != node)
-				continue;
+	/*
+	 * This doesn't seem to be used by the Linux memory manager any
+	 * more, but is used by ll_rw_block.  If we can get rid of it, we
+	 * also get rid of some of the stuff above as well.
+	 *
+	 * Note: max_low_pfn and max_pfn reflect the number of _pages_ in
+	 * the system, not the maximum PFN.
+	 */
+	max_pfn = max_low_pfn = memend_pfn - PHYS_PFN_OFFSET;
+}
 
-			zhole_size[0] -= mi->bank[i].size >> PAGE_SHIFT;
-		}
+/*
+ * Set up device the mappings.  Since we clear out the page tables for all
+ * mappings above VMALLOC_END, we will remove any debug device mappings.
+ * This means you have to be careful how you debug this function, or any
+ * called function.  (Do it by code inspection!)
+ */
+static void __init devicemaps_init(struct machine_desc *mdesc)
+{
+	struct map_desc map;
+	unsigned long addr;
+	void *vectors;
 
-		/*
-		 * Adjust the sizes according to any special
-		 * requirements for this machine type.
-		 */
-		arch_adjust_zones(node, zone_size, zhole_size);
+	for (addr = VMALLOC_END; addr; addr += PGDIR_SIZE)
+		pmd_clear(pmd_off_k(addr));
+
+	/*
+	 * Map the cache flushing regions.
+	 */
+#ifdef FLUSH_BASE
+	map.physical = FLUSH_BASE_PHYS;
+	map.virtual = FLUSH_BASE;
+	map.length = PGDIR_SIZE;
+	map.type = MT_CACHECLEAN;
+	create_mapping(&map);
+#endif
+#ifdef FLUSH_BASE_MINICACHE
+	map.physical = FLUSH_BASE_PHYS + PGDIR_SIZE;
+	map.virtual = FLUSH_BASE_MINICACHE;
+	map.length = PGDIR_SIZE;
+	map.type = MT_MINICLEAN;
+	create_mapping(&map);
+#endif
 
-		free_area_init_node(node, pgdat, zone_size,
-				bdata->node_boot_start >> PAGE_SHIFT, zhole_size);
+	flush_cache_all();
+	local_flush_tlb_all();
+
+	vectors = alloc_bootmem_low_pages(PAGE_SIZE);
+	BUG_ON(!vectors);
+
+	/*
+	 * Create a mapping for the machine vectors at the high-vectors
+	 * location (0xffff0000).  If we aren't using high-vectors, also
+	 * create a mapping at the low-vectors virtual address.
+	 */
+	map.physical = virt_to_phys(vectors);
+	map.virtual = 0xffff0000;
+	map.length = PAGE_SIZE;
+	map.type = MT_HIGH_VECTORS;
+	create_mapping(&map);
+
+	if (!vectors_high()) {
+		map.virtual = 0;
+		map.type = MT_LOW_VECTORS;
+		create_mapping(&map);
 	}
 
 	/*
-	 * finish off the bad pages once
-	 * the mem_map is initialised
+	 * Ask the machine support to map in the statically mapped devices.
+	 * After this point, we can start to touch devices again.
+	 */
+	if (mdesc->map_io)
+		mdesc->map_io();
+}
+
+/*
+ * paging_init() sets up the page tables, initialises the zone memory
+ * maps, and sets up the zero page, bad page and bad page tables.
+ */
+void __init paging_init(struct meminfo *mi, struct machine_desc *mdesc)
+{
+	void *zero_page;
+
+	build_mem_type_table();
+	bootmem_init(mi);
+	devicemaps_init(mdesc);
+
+	top_pmd = pmd_off_k(0xffff0000);
+
+	/*
+	 * allocate the zero page.  Note that we count on this going ok.
 	 */
+	zero_page = alloc_bootmem_low_pages(PAGE_SIZE);
 	memzero(zero_page, PAGE_SIZE);
 	empty_zero_page = virt_to_page(zero_page);
 	flush_dcache_page(empty_zero_page);
@@ -562,10 +565,7 @@ static void __init free_unused_memmap_no
 	 * may not be the case, especially if the user has provided the
 	 * information on the command line.
 	 */
-	for (i = 0; i < mi->nr_banks; i++) {
-		if (mi->bank[i].size == 0 || mi->bank[i].node != node)
-			continue;
-
+	for_each_nodebank(i, mi, node) {
 		bank_start = mi->bank[i].start >> PAGE_SHIFT;
 		if (bank_start < prev_bank_end) {
 			printk(KERN_ERR "MEM: unordered memory banks.  "
diff --git a/arch/arm/mm/mm-armv.c b/arch/arm/mm/mm-armv.c
--- a/arch/arm/mm/mm-armv.c
+++ b/arch/arm/mm/mm-armv.c
@@ -305,16 +305,6 @@ alloc_init_page(unsigned long virt, unsi
 	set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, prot));
 }
 
-/*
- * Clear any PGD mapping.  On a two-level page table system,
- * the clearance is done by the middle-level functions (pmd)
- * rather than the top-level (pgd) functions.
- */
-static inline void clear_mapping(unsigned long virt)
-{
-	pmd_clear(pmd_off_k(virt));
-}
-
 struct mem_types {
 	unsigned int	prot_pte;
 	unsigned int	prot_l1;
@@ -373,7 +363,7 @@ static struct mem_types mem_types[] __in
 /*
  * Adjust the PMD section entries according to the CPU in use.
  */
-static void __init build_mem_type_table(void)
+void __init build_mem_type_table(void)
 {
 	struct cachepolicy *cp;
 	unsigned int cr = get_cr();
@@ -483,7 +473,7 @@ static void __init build_mem_type_table(
  * offsets, and we take full advantage of sections and
  * supersections.
  */
-static void __init create_mapping(struct map_desc *md)
+void __init create_mapping(struct map_desc *md)
 {
 	unsigned long virt, length;
 	int prot_sect, prot_l1, domain;
@@ -601,100 +591,6 @@ void setup_mm_for_reboot(char mode)
 	}
 }
 
-extern void _stext, _etext;
-
-/*
- * Setup initial mappings.  We use the page we allocated for zero page to hold
- * the mappings, which will get overwritten by the vectors in traps_init().
- * The mappings must be in virtual address order.
- */
-void __init memtable_init(struct meminfo *mi)
-{
-	struct map_desc *init_maps, *p, *q;
-	unsigned long address = 0;
-	int i;
-
-	build_mem_type_table();
-
-	init_maps = p = alloc_bootmem_low_pages(PAGE_SIZE);
-
-#ifdef CONFIG_XIP_KERNEL
-	p->physical   = CONFIG_XIP_PHYS_ADDR & PMD_MASK;
-	p->virtual    = (unsigned long)&_stext & PMD_MASK;
-	p->length     = ((unsigned long)&_etext - p->virtual + ~PMD_MASK) & PMD_MASK;
-	p->type       = MT_ROM;
-	p ++;
-#endif
-
-	for (i = 0; i < mi->nr_banks; i++) {
-		if (mi->bank[i].size == 0)
-			continue;
-
-		p->physical   = mi->bank[i].start;
-		p->virtual    = __phys_to_virt(p->physical);
-		p->length     = mi->bank[i].size;
-		p->type       = MT_MEMORY;
-		p ++;
-	}
-
-#ifdef FLUSH_BASE
-	p->physical   = FLUSH_BASE_PHYS;
-	p->virtual    = FLUSH_BASE;
-	p->length     = PGDIR_SIZE;
-	p->type       = MT_CACHECLEAN;
-	p ++;
-#endif
-
-#ifdef FLUSH_BASE_MINICACHE
-	p->physical   = FLUSH_BASE_PHYS + PGDIR_SIZE;
-	p->virtual    = FLUSH_BASE_MINICACHE;
-	p->length     = PGDIR_SIZE;
-	p->type       = MT_MINICLEAN;
-	p ++;
-#endif
-
-	/*
-	 * Go through the initial mappings, but clear out any
-	 * pgdir entries that are not in the description.
-	 */
-	q = init_maps;
-	do {
-		if (address < q->virtual || q == p) {
-			clear_mapping(address);
-			address += PGDIR_SIZE;
-		} else {
-			create_mapping(q);
-
-			address = q->virtual + q->length;
-			address = (address + PGDIR_SIZE - 1) & PGDIR_MASK;
-
-			q ++;
-		}
-	} while (address != 0);
-
-	/*
-	 * Create a mapping for the machine vectors at the high-vectors
-	 * location (0xffff0000).  If we aren't using high-vectors, also
-	 * create a mapping at the low-vectors virtual address.
-	 */
-	init_maps->physical   = virt_to_phys(init_maps);
-	init_maps->virtual    = 0xffff0000;
-	init_maps->length     = PAGE_SIZE;
-	init_maps->type       = MT_HIGH_VECTORS;
-	create_mapping(init_maps);
-
-	if (!vectors_high()) {
-		init_maps->virtual = 0;
-		init_maps->type = MT_LOW_VECTORS;
-		create_mapping(init_maps);
-	}
-
-	flush_cache_all();
-	local_flush_tlb_all();
-
-	top_pmd = pmd_off_k(0xffff0000);
-}
-
 /*
  * Create the architecture specific mappings
  */

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:  2.6 Serial core

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Andi Kleen wrote:

> On Monday 17 October 2005 17:27, Linus Torvalds wrote:
> > On Mon, 17 Oct 2005, Andi Kleen wrote:
> > > The patch is actually not quite correct - in theory node 0 could be too
> > > small to contain the full swiotlb bounce buffers.
> >
> > Is node 0 guaranteed to be all low-memory? What if it allocates stuff at
> > the end of memory on NODE(0)?
> 
> This is 64bit ... only low memory.

Ehh.. No there isn't.

PCI DMA isn't magically 64-bit, even on your Opteron. 

So low memory in this case is anything < 32 bits. How many bits the CPU 
has is immaterial.

That's the whole _point_ of swtlb, after all, so I don't see why you 
argue.

		Linus

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 18:02, Linus Torvalds wrote:
> On Mon, 17 Oct 2005, Andi Kleen wrote:
> > On Monday 17 October 2005 17:27, Linus Torvalds wrote:
> > > On Mon, 17 Oct 2005, Andi Kleen wrote:
> > > > The patch is actually not quite correct - in theory node 0 could be
> > > > too small to contain the full swiotlb bounce buffers.
> > >
> > > Is node 0 guaranteed to be all low-memory? What if it allocates stuff
> > > at the end of memory on NODE(0)?
> >
> > This is 64bit ... only low memory.
>
> Ehh.. No there isn't.
>
> PCI DMA isn't magically 64-bit, even on your Opteron.
>
> So low memory in this case is anything < 32 bits. How many bits the CPU
> has is immaterial.

That's completely new terminology. We always called all of ZONE_NORMAL low 
memory.

The 32bit DMA zone had no special name before the ZONE_DMA32 patches. 
With that they are called dma32 zone.

-Andi

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Andi Kleen wrote:
> 
> That's completely new terminology. We always called all of ZONE_NORMAL low 
> memory.

We call it "low" memory because it happens to have "low" addresses. In 
other words, it's not "terminology", it's "English".

None of the allocators that allocate stuff in ZONE_NORMAL is called "low" 
normally. It's _normal_ memory. It's not ZONE_LOW. 

We don't say "kmalloc_low()". We say "kmalloc()". 

A function that is called "xyz_low()" means something else than normal to 
me. If it was normal memory, we'd call it just "xyz()". And if it did high 
memory, we'd call it "xyz_highmem()" (or, preferably, we'd just have a 
generic function that accepted GFP_HIGHMEM as a parameter).

			Linus

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 18:42, Linus Torvalds wrote:
> We call it "low" memory because it happens to have "low" addresses.

Well in NUMA bootmem it never was, unless you registered the nodes reversed.
It always starts with the highest node
(which I can't easily do, ARM does it so fixing it properly breaks them) 

-Andi

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Mon, Oct 17, 2005 at 05:40:56PM +0200, Andi Kleen wrote:
> On Monday 17 October 2005 17:27, Linus Torvalds wrote:
> > On Mon, 17 Oct 2005, Andi Kleen wrote:
> > ...
> > Argh. Which one should I pick? The NODE(0) one looks simpler, but is it
> > sufficient for now in practice (with the real one going into 2.6.14+)?
> >
> > 		Linus
> 
> First this problem is definitely not critical. AFAIK it only happens on 
> scalex's unreleased machines. Intel NUMA x86 machines are really rare
				^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
No they are not.  IBM X460s are generally available machines and  the bug
affects those boxes. How can there be a major kernel release which is known
to have breakage??  

Maybe someone with access to ia64 NUMA boxen can check if the NODE(0)
solution works (and does not break anything) on ia64?  Chrisoph, can you help?

Thanks,
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 19:52, Ravikiran G Thirumalai wrote:

> No they are not.  IBM X460s are generally available machines and  the bug
> affects those boxes. 

No reports from that front so far.

> How can there be a major kernel release which is known 
> to have breakage??

Welcome to the painful real world of software engineering.

Every software has bugs and if you want to ever get a release out you
have to make such decisions sometimes. 

As an alternative I can just backout the patch that enables the Intel
SRAT code. That is probably better for a short term fix and will
not regress anybody.

-Andi

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Muli Ben-Yehuda]

On Mon, Oct 17, 2005 at 08:08:24PM +0200, Andi Kleen wrote:
> On Monday 17 October 2005 19:52, Ravikiran G Thirumalai wrote:
> 
> > No they are not.  IBM X460s are generally available machines and  the bug
> > affects those boxes. 
> 
> No reports from that front so far.

We have such machines with >4GB memory and 32 bit DMA capable SCSI
controllers and would like to be able to run 2.6.14 on them when it
comes out...

> > How can there be a major kernel release which is known 
> > to have breakage??
> 
> Welcome to the painful real world of software engineering.
> 
> Every software has bugs and if you want to ever get a release out you
> have to make such decisions sometimes. 

Fair enough, but this is a regression for something that used to
work. If a painful choice is required, how about reverting the patch
that broke it and breaking something that used to be broken?

> As an alternative I can just backout the patch that enables the Intel
> SRAT code. That is probably better for a short term fix and will
> not regress anybody.

Sounds great!

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 20:27, Muli Ben-Yehuda wrote:
> On Mon, Oct 17, 2005 at 08:08:24PM +0200, Andi Kleen wrote:
> > On Monday 17 October 2005 19:52, Ravikiran G Thirumalai wrote:
> > > No they are not.  IBM X460s are generally available machines and  the
> > > bug affects those boxes.
> >
> > No reports from that front so far.
>
> We have such machines with >4GB memory and 32 bit DMA capable SCSI
> controllers 

32bit DMA SCSI controllers??? Where did you find such a beast?

> and would like to be able to run 2.6.14 on them when it 
> comes out...

So you're saying you tested it and it doesn't work? 

-Andi

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Muli Ben-Yehuda]

On Mon, Oct 17, 2005 at 08:32:45PM +0200, Andi Kleen wrote:

> > and would like to be able to run 2.6.14 on them when it 
> > comes out...
> 
> So you're saying you tested it and it doesn't work? 

Not quite; I'm saying that form the description up-thread it sounds
like there's a good chance it won't. Jon Mason (CC'd) has access to
such a  machine. Jon, can you please try the latest hg tree with and
without the patch and see how it fares?

Thanks,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Muli Ben-Yehuda wrote:
>
> On Mon, Oct 17, 2005 at 08:32:45PM +0200, Andi Kleen wrote:
> 
> > > and would like to be able to run 2.6.14 on them when it 
> > > comes out...
> > 
> > So you're saying you tested it and it doesn't work? 
> 
> Not quite; I'm saying that form the description up-thread it sounds
> like there's a good chance it won't. Jon Mason (CC'd) has access to
> such a  machine. Jon, can you please try the latest hg tree with and
> without the patch and see how it fares?

NOTE! Even if the machine has 4GB or more of memory, it's entirely likely 
that the quick "use NODE(0)" hack will work fine. 

Why? Because the bootmem memory should still be allocated low-to-high by 
default, which means that as logn as NODE(0) has _enough_ memory in the 
DMA range, we should be ok.

So I _think_ the simple one-liner NODE(0) patch is sufficient, and should 
work (and is a lot more acceptable for 2.6.14 than switching the node 
ordering around yet again, or doing bigger surgery on the bootmem code).

So the only thing that worried me (and made me ask whether there might be 
machines where it doesn't work) is if some machines might have their high 
memory (or no memory at all) on NODE(0). It does sound unlikely, but I 
simple don't know what kind of strange NUMA configs there are out there.

And I'm definitely only interested in machines that are out there, not 
some theoretical issues.

			Linus

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 21:04, Linus Torvalds wrote:

> So the only thing that worried me (and made me ask whether there might be
> machines where it doesn't work) is if some machines might have their high
> memory (or no memory at all) on NODE(0). It does sound unlikely, but I
> simple don't know what kind of strange NUMA configs there are out there.

It could happen in VirtualIron (they seem to interleave node 0 over many nodes 
to get equal use of lowmem in 32bit NUMA), but should not in x86-64..
 
> And I'm definitely only interested in machines that are out there, not
> some theoretical issues.

According to Alex W. it will break their sx1000 IA64 boxen.

-Andi (who still thinks it's best to just ignore it or disable Intel NUMA) 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Arjan van de Ven]

On Mon, 2005-10-17 at 21:09 +0200, Andi Kleen wrote:
> On Monday 17 October 2005 21:04, Linus Torvalds wrote:
> 
> > So the only thing that worried me (and made me ask whether there might be
> > machines where it doesn't work) is if some machines might have their high
> > memory (or no memory at all) on NODE(0). It does sound unlikely, but I
> > simple don't know what kind of strange NUMA configs there are out there.
> 
> It could happen in VirtualIron (they seem to interleave node 0 over many nodes 
> to get equal use of lowmem in 32bit NUMA), but should not in x86-64..
>  
does VirtualIron work with kernel.org kernels at all?

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Mon, Oct 17, 2005 at 09:09:52PM +0200, Andi Kleen wrote:
> On Monday 17 October 2005 21:04, Linus Torvalds wrote:
> 
> > So the only thing that worried me (and made me ask whether there might be
> > machines where it doesn't work) is if some machines might have their high
> > memory (or no memory at all) on NODE(0). It does sound unlikely, but I
> > simple don't know what kind of strange NUMA configs there are out there.
> 
> It could happen in VirtualIron (they seem to interleave node 0 over many nodes 
> to get equal use of lowmem in 32bit NUMA), but should not in x86-64..
>  
> > And I'm definitely only interested in machines that are out there, not
> > some theoretical issues.
> 
> According to Alex W. it will break their sx1000 IA64 boxen.

sx1000 is probably already broken;  Unless the last pgdat happens to be the
memory only node with 0-4G?  How about the fix I suggested which would iterate 
across all nodes until it found the right node for swiotlb?

Thanks,
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[David Lang]

On Mon, 17 Oct 2005, Linus Torvalds wrote:
>
> Why? Because the bootmem memory should still be allocated low-to-high by
> default, which means that as logn as NODE(0) has _enough_ memory in the
> DMA range, we should be ok.
>
> So I _think_ the simple one-liner NODE(0) patch is sufficient, and should
> work (and is a lot more acceptable for 2.6.14 than switching the node
> ordering around yet again, or doing bigger surgery on the bootmem code).
>
> So the only thing that worried me (and made me ask whether there might be
> machines where it doesn't work) is if some machines might have their high
> memory (or no memory at all) on NODE(0). It does sound unlikely, but I
> simple don't know what kind of strange NUMA configs there are out there.
>
> And I'm definitely only interested in machines that are out there, not
> some theoretical issues.

what about x86_64 machines with 8G of ram connected to each CPU (4x2G 
DIMMs), not exactly common, but also not that rare.

also if CPU's are registered from high to low would a 2 CPU box with 8G of 
ram connected to one CPU and no ram connected to the second CPU match your 
problem case (I know that's a sub-optimal layout, but it's legal)

David Lang


> 			Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

Hello. Linus-san.

> NOTE! Even if the machine has 4GB or more of memory, it's entirely likely 
> that the quick "use NODE(0)" hack will work fine. 
> 
> Why? Because the bootmem memory should still be allocated low-to-high by 
> default, which means that as logn as NODE(0) has _enough_ memory in the 
> DMA range, we should be ok.
> 
> So I _think_ the simple one-liner NODE(0) patch is sufficient, and should 
> work (and is a lot more acceptable for 2.6.14 than switching the node 
> ordering around yet again, or doing bigger surgery on the bootmem code).
> 
> So the only thing that worried me (and made me ask whether there might be 
> machines where it doesn't work) is if some machines might have their high 
> memory (or no memory at all) on NODE(0). It does sound unlikely, but I 
> simple don't know what kind of strange NUMA configs there are out there.
> 
> And I'm definitely only interested in machines that are out there, not 
> some theoretical issues.

In our making IA64 machine node 0 might not have any low-memory, and
another node can have low-memory instead.

This cause comes from hotplug whole of one node.
For example, please imagine following case.

1) In this case, firmware remembers pxm 1's node has low memory.

                 node 0             node 1 
               +--------------+  +-----------+
               |  pxm = 1     |  | pxm = 2   |
               |  low memory  |  |           |
               +--------------+  +-----------+


2) If one node is hot-added at pxm = 0 (pxm is decided from physical
   locate by firmware.), new node will be node 2.

  node 2          node 0          node 1 
+-----------+  +--------------+  +-----------+
| pxm = 0   |  |  pxm = 1     |  | pxm = 2   |
|           |  |  low memory  |  |           |
+-----------+  +--------------+  +-----------+

3) If user reboots the machine, Linux decides node id from pxm's order.
   But firmware still remembers which node has low memory.
   So, node 0 will not have any low memory.

  node 0          node 1          node 2
+-----------+  +--------------+  +-----------+
| pxm = 0   |  |  pxm = 1     |  | pxm = 2   |
|           |  |  low memory  |  |           |
+-----------+  +--------------+  +-----------+

So, just "use NODE(0)" is not enough hack for our machine.
If "use NODE(0)" is selected, kernel must sort pgdat link and
node id by memory address. I think that hot add code will be a 
bit messy instead.

Thanks.

-- 
Yasunori Goto 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 11:29:18AM +0900, Yasunori Goto wrote:
> Hello. Linus-san.
> ... 
> In our making IA64 machine node 0 might not have any low-memory, and
> another node can have low-memory instead.
> 
> This cause comes from hotplug whole of one node.
> For example, please imagine following case.
> 
> 1) In this case, firmware remembers pxm 1's node has low memory.
> 
>                  node 0             node 1 
>                +--------------+  +-----------+
>                |  pxm = 1     |  | pxm = 2   |
>                |  low memory  |  |           |
>                +--------------+  +-----------+
> 
> 
> 2) If one node is hot-added at pxm = 0 (pxm is decided from physical
>    locate by firmware.), new node will be node 2.
> 
>   node 2          node 0          node 1 
> +-----------+  +--------------+  +-----------+
> | pxm = 0   |  |  pxm = 1     |  | pxm = 2   |
> |           |  |  low memory  |  |           |
> +-----------+  +--------------+  +-----------+
> 
> 3) If user reboots the machine, Linux decides node id from pxm's order.
>    But firmware still remembers which node has low memory.
>    So, node 0 will not have any low memory.
> 
>   node 0          node 1          node 2
> +-----------+  +--------------+  +-----------+
> | pxm = 0   |  |  pxm = 1     |  | pxm = 2   |
> |           |  |  low memory  |  |           |
> +-----------+  +--------------+  +-----------+
> 
> So, just "use NODE(0)" is not enough hack for our machine.
> If "use NODE(0)" is selected, kernel must sort pgdat link and
> node id by memory address. I think that hot add code will be a 
> bit messy instead.

Yasunori-san,
Does this patch work on your boxes instead? (For 2.6.14)
http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2

Thanks,
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

> > So, just "use NODE(0)" is not enough hack for our machine.
> > If "use NODE(0)" is selected, kernel must sort pgdat link and
> > node id by memory address. I think that hot add code will be a 
> > bit messy instead.
> 
> Yasunori-san,
> Does this patch work on your boxes instead? (For 2.6.14)
> http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2

Not yet. But could you change this line at least?

+	
+	for_each_node(node) {
+		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);

for_each_node() loop walks around node_possible_map which includes
"offlined" node. 
Please use for_each_online_node() instead. Then, I'll check it. :-)



And I understand why my patch doesn't work on your box by
your patch. (Thanks!)
x86-64's DMA zone is smaller area than 16MB.
But swiotlb requires the area which is smaller than 4GB.
There is no interface to describe its difference.

I think your patch looks reasonable for 2.6.14 to be produced ASAP.
This is less impact than mine. 

However, kernel should have new interface like alloc_bootmmem_low32().
This problem will occur not only by swiotlb.

I'll rewrite my patch to make it. If it is applied at 2.6.15,
I'll be glad.

Thanks.

-- 
Yasunori Goto 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 01:28:20PM +0900, Yasunori Goto wrote:
> > > So, just "use NODE(0)" is not enough hack for our machine.
> > > If "use NODE(0)" is selected, kernel must sort pgdat link and
> > > node id by memory address. I think that hot add code will be a 
> > > bit messy instead.
> > 
> > Yasunori-san,
> > Does this patch work on your boxes instead? (For 2.6.14)
> > http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2
> 
> Not yet. But could you change this line at least?
> 
> +	
> +	for_each_node(node) {
> +		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
> 
> for_each_node() loop walks around node_possible_map which includes
> "offlined" node. 
> Please use for_each_online_node() instead. Then, I'll check it. :-)

Since swiotlb is is allocated even before APs are brought up, I thought,
if the node containing lowmem32 was not the boot node, it would not be
online.   But on closer look, my assumption was wrong.  Here is the patch
which iterates through online nodes to allocate lowmem32 bootmem for
swiotlb.

--

Patch to ensure low32 mem allocation for x86_64 swiotlb 

Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-17 13:27:35.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 16:00:44.000000000 -0700
@@ -106,6 +106,8 @@
 __setup("swiotlb=", setup_io_tlb_npages);
 /* make io_tlb_overflow tunable too? */
 
+#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
+
 /*
  * Statically reserve bounce buffer space and initialize bounce buffer data
  * structures for the software IO TLB used to implement the PCI DMA API.
@@ -114,17 +116,32 @@
 swiotlb_init_with_default_size (size_t default_size)
 {
 	unsigned long i;
+	unsigned long iotlbsz;
+	int node;
 
 	if (!io_tlb_nslabs) {
 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
 	}
 
+	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
+
 	/*
-	 * Get IO TLB memory from the low pages
+	 * Get IO TLB memory from the 0-4G range
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	
+	for_each_online_node(node) {
+		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
+		if (io_tlb_start) {
+			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
+				break;
+			free_bootmem_node(NODE_DATA(node), 
+					  virt_to_phys(io_tlb_start), iotlbsz);
+			io_tlb_start = NULL;
+		}
+	}
+
+	
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

I tested your patch, but unfortunately, it doesn't work IA64.
alloc_bootmem_node() requires bigger area than MAX_DMA_ADDRESS.
It is defined as 4GB for ia64. (arch/ia64/mm/init.c)
But this patch require smaller area than 4GB. 
So they are exclusive each other.

I'm convinced that a new interface like alloc_bootmem_low32() is
necessary after all. ;-)

Thanks.


> On Tue, Oct 18, 2005 at 01:28:20PM +0900, Yasunori Goto wrote:
> > > > So, just "use NODE(0)" is not enough hack for our machine.
> > > > If "use NODE(0)" is selected, kernel must sort pgdat link and
> > > > node id by memory address. I think that hot add code will be a 
> > > > bit messy instead.
> > > 
> > > Yasunori-san,
> > > Does this patch work on your boxes instead? (For 2.6.14)
> > > http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2
> > 
> > Not yet. But could you change this line at least?
> > 
> > +	
> > +	for_each_node(node) {
> > +		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
> > 
> > for_each_node() loop walks around node_possible_map which includes
> > "offlined" node. 
> > Please use for_each_online_node() instead. Then, I'll check it. :-)
> 
> Since swiotlb is is allocated even before APs are brought up, I thought,
> if the node containing lowmem32 was not the boot node, it would not be
> online.   But on closer look, my assumption was wrong.  Here is the patch
> which iterates through online nodes to allocate lowmem32 bootmem for
> swiotlb.
> 
> --
> 
> Patch to ensure low32 mem allocation for x86_64 swiotlb 
> 
> Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>
> 
> Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
> ===================================================================
> --- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-17 13:27:35.000000000 -0700
> +++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 16:00:44.000000000 -0700
> @@ -106,6 +106,8 @@
>  __setup("swiotlb=", setup_io_tlb_npages);
>  /* make io_tlb_overflow tunable too? */
>  
> +#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
> +
>  /*
>   * Statically reserve bounce buffer space and initialize bounce buffer data
>   * structures for the software IO TLB used to implement the PCI DMA API.
> @@ -114,17 +116,32 @@
>  swiotlb_init_with_default_size (size_t default_size)
>  {
>  	unsigned long i;
> +	unsigned long iotlbsz;
> +	int node;
>  
>  	if (!io_tlb_nslabs) {
>  		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
>  		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
>  	}
>  
> +	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
> +
>  	/*
> -	 * Get IO TLB memory from the low pages
> +	 * Get IO TLB memory from the 0-4G range
>  	 */
> -	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
> -					       (1 << IO_TLB_SHIFT));
> +	
> +	for_each_online_node(node) {
> +		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
> +		if (io_tlb_start) {
> +			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
> +				break;
> +			free_bootmem_node(NODE_DATA(node), 
> +					  virt_to_phys(io_tlb_start), iotlbsz);
> +			io_tlb_start = NULL;
> +		}
> +	}
> +
> +	
>  	if (!io_tlb_start)
>  		panic("Cannot allocate SWIOTLB buffer");
>  	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

-- 
Yasunori Goto 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 07:09:03PM +0900, Yasunori Goto wrote:
> 
> I tested your patch, but unfortunately, it doesn't work IA64.
> alloc_bootmem_node() requires bigger area than MAX_DMA_ADDRESS.
> It is defined as 4GB for ia64. (arch/ia64/mm/init.c)
> But this patch require smaller area than 4GB. 
> So they are exclusive each other.

No, alloc_bootmem_node should work for 4G allocations too; So my approach
should work unless there was some other bootmem request served out of that 
node earlier.  Btw, the default is to allocate 64MB for swiotlb.  
Do you modify that for your boxes?  IMHO, we should stick to fixing the 
stock kernel now for 2.6.14.  

However, IS_LOWPAGES macro in my patch enforces that iotlb start and end 
locations to be within 4G.  I can change that to within and upto 4G, and the 
patch should work for you too.

Thanks,
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Jon Mason]

I have run a few tests on the original code and the patches posted to the list,
and have some interesting results.  First, my system setup:  Dual Opteron, 8GB
RAM, SIL SATA controller (which is apparently 32bit), pcnet32 NIC (compiled as
a module) connected to the network.  The latter 2 should show any bounce
buffer problems.  

I get the following behavior: 
- With AMD HW IOMMU (AGP_GART), everything works perfectly.
- With IOMMU disabled (iommu=off), my kernel panics in SATA.
- With SW IOMMU (iommu=soft), my kernel is unable to find any partitions on my
SATA disk (and panics)
- With SW IOMMU and Ravikiran's original patch, kernel boots with no aperient
problems
- With SW IOMMU and Ravikiran's second patch, kernel boots with no aperient
problems
- With SW IOMMU and Yasunori's patch, kernel boots with no aperient problems

The edited dmesg output can be found below.  

>From the above, we can tell that my system requires either hardware IOMMU or
bounce buffers for the SATA disk.  We can also tell that the current
implimentation has a bug (where the actual cause lies is unknown).  All of the
patches in this e-mail thread seem to fix my problem and I had no noticeable
problems.

If anyone wants me to try anything on my system, I'll be happy to help.  

Thanks,
Jon

----------------------------------------------------

with Yasunori's last patch and with iommu=soft
[...]
Placing software IO TLB between 0x4821000 - 0x8821000
[...]
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: f9700000-f97fffff
  PREFETCH window: f9000000-f90fffff
PCI: Bridge: 0000:00:0a.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: f9800000-f98fffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:20000@f8000000 for 0000:09:00.0
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: f8000000-f8ffffff
  PREFETCH window: f0000000-f7ffffff
PCI: Bridge: 0000:08:03.0
  IO window: 3000-3fff
  MEM window: f9200000-f92fffff
  PREFETCH window: f9100000-f91fffff
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: f9300000-f93fffff
  PREFETCH window: f9500000-f95fffff
[...]
scsi2 : sata_sil
ata2: no device found (phy stat 00000000)
scsi3 : sata_sil
  Vendor: ATA       Model: Maxtor 6Y080M0    Rev: YAR5
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
[...]


with Ravikiran's second patch and with iommu=soft
[...]
Placing software IO TLB between 0x4821000 - 0x8821000
[...]
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: f9700000-f97fffff
  PREFETCH window: f9000000-f90fffff
PCI: Bridge: 0000:00:0a.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: f9800000-f98fffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:20000@f8000000 for 0000:09:00.0
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: f8000000-f8ffffff
  PREFETCH window: f0000000-f7ffffff
PCI: Bridge: 0000:08:03.0
  IO window: 3000-3fff
  MEM window: f9200000-f92fffff
  PREFETCH window: f9100000-f91fffff
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: f9300000-f93fffff
  PREFETCH window: f9500000-f95fffff
[...]
scsi2 : sata_sil
ata2: no device found (phy stat 00000000)
scsi3 : sata_sil
  Vendor: ATA       Model: Maxtor 6Y080M0    Rev: YAR5
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
[...]


with Ravikiran's first patch and with iommu=soft
[...]
Placing software IO TLB between 0x4820180 - 0x8820180
[...]
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: f9700000-f97fffff
  PREFETCH window: f9000000-f90fffff
PCI: Bridge: 0000:00:0a.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: f9800000-f98fffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:20000@f8000000 for 0000:09:00.0
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: f8000000-f8ffffff
  PREFETCH window: f0000000-f7ffffff
PCI: Bridge: 0000:08:03.0
  IO window: 3000-3fff
  MEM window: f9200000-f92fffff
  PREFETCH window: f9100000-f91fffff
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: f9300000-f93fffff
  PREFETCH window: f9500000-f95fffff
[...]
scsi2 : sata_sil
ata2: no device found (phy stat 00000000)
scsi3 : sata_sil
  Vendor: ATA       Model: Maxtor 6Y080M0    Rev: YAR5
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
 sda: sda1 sda2 sda3
[...]



without patch and with iommu=soft
[...]
Placing software IO TLB between 0x104466000 - 0x108466000
[...]
PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: f9700000-f97fffff
  PREFETCH window: f9000000-f90fffff
PCI: Bridge: 0000:00:0a.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: f9800000-f98fffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:20000@f8000000 for 0000:09:00.0
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: f8000000-f8ffffff
  PREFETCH window: f0000000-f7ffffff
PCI: Bridge: 0000:08:03.0
  IO window: 3000-3fff
  MEM window: f9200000-f92fffff
  PREFETCH window: f9100000-f91fffff
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: f9300000-f93fffff
  PREFETCH window: f9500000-f95fffff
[...]
scsi2 : sata_sil
ata2: no device found (phy stat 00000000)
scsi3 : sata_sil
  Vendor: ATA       Model: Maxtor 6Y080M0    Rev: YAR5
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
 sda: unknown partition table
[...]
VFS: Cannot open root device "sda2" or unknown-block(8,2)
Please append a correct "root=" boot option
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(8,2)


without patch and with iommu=off
[...]
PCI-DMA: Disabling IOMMU.
PCI: Bridge: 0000:00:06.0
  IO window: 2000-2fff
  MEM window: f9700000-f97fffff
  PREFETCH window: f9000000-f90fffff
PCI: Bridge: 0000:00:0a.0
  IO window: disabled.
  MEM window: disabled.
  PREFETCH window: disabled.
PCI: Bridge: 0000:00:0b.0
  IO window: disabled.
  MEM window: f9800000-f98fffff
  PREFETCH window: disabled.
PCI: Failed to allocate mem resource #6:20000@f8000000 for 0000:09:00.0
PCI: Bridge: 0000:08:01.0
  IO window: disabled.
  MEM window: f8000000-f8ffffff
  PREFETCH window: f0000000-f7ffffff
PCI: Bridge: 0000:08:03.0
  IO window: 3000-3fff
  MEM window: f9200000-f92fffff
  PREFETCH window: f9100000-f91fffff
PCI: Bridge: 0000:08:04.0
  IO window: 4000-4fff
  MEM window: f9300000-f93fffff
  PREFETCH window: f9500000-f95fffff
[...]
scsi2 : sata_sil
ata2: no device found (phy stat 00000000)
scsi3 : sata_sil
  Vendor: ATA       Model: Maxtor 6Y080M0    Rev: YAR5
  Type:   Direct-Access                      ANSI SCSI revision: 05
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
SCSI device sda: 156312576 512-byte hdwr sectors (80032 MB)
SCSI device sda: drive cache: write back
 sda:<0>Kernel panic - not syncing: PCI-DMA: high address but no IOMMU.

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Wednesday 19 October 2005 19:18, Jon Mason wrote:
> I have run a few tests on the original code and the patches posted to the
> list, and have some interesting results.  First, my system setup:  Dual
> Opteron, 8GB RAM, SIL SATA controller (which is apparently 32bit), pcnet32
> NIC (compiled as a module) connected to the network.  The latter 2 should
> show any bounce buffer problems.

We don't care about AMD systems for this problem because they
don't use swiotlb in normal operations, but the AGP remapping hardware (except 
on one particulr chipset, but people don't build servers from that) 

If anything then testing results from Summit3 based Intel x86 NUMA systems 
would be interesting.

-Andi
 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

On Mon, Oct 17, 2005 at 08:08:24PM +0200, Andi Kleen wrote:
> On Monday 17 October 2005 19:52, Ravikiran G Thirumalai wrote:
> 
> > No they are not.  IBM X460s are generally available machines and  the bug
> > affects those boxes. 
> 
> No reports from that front so far.

Attached is the dmesg on x460 which shows where swiotlb is allocated.

> 
> > How can there be a major kernel release which is known 
> > to have breakage??
> 
> Welcome to the painful real world of software engineering.
> 
> Every software has bugs and if you want to ever get a release out you
> have to make such decisions sometimes. 
> 
> As an alternative I can just backout the patch that enables the Intel
> SRAT code. That is probably better for a short term fix and will
> not regress anybody.

Intel SRAT code?  which patch are you referring to? How is that relevant here?  

Thanks,
Kiran

[-- Attachment #2: dmesg --]
[-- Type: text/plain, Size: 23518 bytes --]

[    0.000000] Bootdata ok (command line is root=LABEL=/  console=ttyS1,115200 console=ttyS0,115200 console=tty1 console=tty0)
[    0.000000] Linux version 2.6.14-rc4 (root@x460) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #3 SMP Thu Oct 13 13:37:51 PDT 2005
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 0000000000098000 (usable)
[    0.000000]  BIOS-e820: 0000000000098000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000e7f94d40 (usable)
[    0.000000]  BIOS-e820: 00000000e7f94d40 - 00000000e7fa6a40 (ACPI data)
[    0.000000]  BIOS-e820: 00000000e7fa6a40 - 00000000e8000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 00000003f8000000 (usable)
[    0.000000] ACPI: RSDP (v000 IBM                                   ) @ 0x00000000000fdcf0
[    0.000000] ACPI: RSDT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa69c0
[    0.000000] ACPI: FADT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6940
[    0.000000] ACPI: MADT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6840
[    0.000000] ACPI: SRAT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6700
[    0.000000] ACPI: HPET (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa66c0
[    0.000000] ACPI: SSDT (v001 IBM    VIGSSDT0 0x00001000 INTL 0x20030122) @ 0x00000000e7f9f0c0
[    0.000000] ACPI: SSDT (v001 IBM    VIGSSDT1 0x00001000 INTL 0x20030122) @ 0x00000000e7f97b00
[    0.000000] ACPI: DSDT (v001 IBM    EXA01ZEU 0x00001000 INTL 0x20030122) @ 0x0000000000000000
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 6 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 16 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 22 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 32 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 38 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 48 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 54 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-e7ffffff
[    0.000000] SRAT: Node 0 PXM 0 0-207ffffff
[    0.000000] SRAT: Node 1 PXM 1 208000000-3f7ffffff
[    0.000000] Using 22 for the hash shift. Max adder is 3f7ffffff 
[    0.000000] Bootmem setup node 0 0000000000000000-0000000207ffffff
[    0.000000] Bootmem setup node 1 0000000208000000-00000003f7ffffff
[    0.000000] On node 0 totalpages: 2031403
[    0.000000]   DMA zone: 3992 pages, LIFO batch:1
[    0.000000]   Normal zone: 2027411 pages, LIFO batch:31
[    0.000000]   HighMem zone: 0 pages, LIFO batch:1
[    0.000000] On node 1 totalpages: 2031615
[    0.000000]   DMA zone: 0 pages, LIFO batch:1
[    0.000000]   Normal zone: 2031615 pages, LIFO batch:31
[    0.000000]   HighMem zone: 0 pages, LIFO batch:1
[    0.000000] ACPI: PM-Timer IO Port: 0x9c
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] Processor #0 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
[    0.000000] Processor #6 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled)
[    0.000000] Processor #16 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x16] enabled)
[    0.000000] Processor #22 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
[    0.000000] Processor #32 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x26] enabled)
[    0.000000] Processor #38 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x30] enabled)
[    0.000000] Processor #48 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x36] enabled)
[    0.000000] Processor #54 15:4 APIC version 20
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] dfl dfl lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 15, version 17, address 0xfec00000, GSI 0-35
[    0.000000] ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[35])
[    0.000000] IOAPIC[1]: apic_id 14, version 17, address 0xfec01000, GSI 35-70
[    0.000000] ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[70])
[    0.000000] IOAPIC[2]: apic_id 13, version 17, address 0xfec02000, GSI 70-105
[    0.000000] ACPI: IOAPIC (id[0x0c] address[0xfec03000] gsi_base[105])
[    0.000000] IOAPIC[3]: apic_id 12, version 17, address 0xfec03000, GSI 105-140
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 8 global_irq 8 low edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 low edge)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ8 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] ACPI: IRQ14 used by override.
[    0.000000] Setting APIC routing to clustered
[    0.000000] ACPI: HPET id: 0x10142201 base: 0xfde84000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Allocating PCI resources starting at ea000000 (gap: e8000000:16c00000)
[    0.000000] Checking aperture...
[    0.000000] Built 2 zonelists
[    0.000000] Kernel command line: root=LABEL=/  console=ttyS1,115200 console=ttyS0,115200 console=tty1 console=tty0
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 131072 bytes)
[    0.000000] time.c: Using 3.579545 MHz PM timer.
[    0.000000] time.c: Detected 3336.330 MHz processor.
[  369.608912] Console: colour VGA+ 80x25
[  370.117124] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[  370.149658] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[  370.240852] Placing software IO TLB between 0x21051c000 - 0x21451c000
[  370.797462] Memory: 15927892k/16646144k available (2594k kernel code, 324188k reserved, 1356k data, 208k init)
[  370.951986] Calibrating delay using timer specific routine.. 6684.01 BogoMIPS (lpj=33420066)
[  370.962277] Mount-cache hash table entries: 256
[  370.967873] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  370.974137] CPU: L2 cache: 1024K
[  370.977990] CPU: L3 cache: 8192K
[  370.981864] CPU 0 -> Node 0
[  370.985220] using mwait in idle threads.
[  370.989926] CPU: Hyper-Threading is disabled
[  370.995063] CPU0: Thermal monitoring enabled (TM1)
[  371.000804] mtrr: v2.0 (20020519)
[  371.136909] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[  371.143884]  failed.
[  371.146519] timer doesn't work through the IO-APIC - disabling NMI Watchdog!
[  371.155928] Uhhuh. NMI received for unknown reason 35.
[  371.162067] Dazed and confused, but trying to continue
[  371.168210] Do you have a strange power saving mode enabled?
[  371.273826]  works.
[  371.276353] Using local APIC timer interrupts.
[  371.311561] Detected 10.425 MHz APIC timer.
[  371.316630] softlockup thread 0 started up.
[  371.321767] Booting processor 1/8 APIC 0x6
[  371.337804] Initializing CPU#1
[  371.480354] Calibrating delay using timer specific routine.. 6672.52 BogoMIPS (lpj=33362642)
[  371.480368] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  371.480370] CPU: L2 cache: 1024K
[  371.480372] CPU: L3 cache: 8192K
[  371.480375] CPU 1 -> Node 0
[  371.480377] CPU: Hyper-Threading is disabled
[  371.480389] CPU1: Thermal monitoring enabled (TM1)
[  371.481497]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  371.481508] APIC error on CPU1: 00(40)
[  371.481512] CPU 1: Syncing TSC to CPU 0.
[  371.481849] CPU 1: synchronized TSC with CPU 0 (last diff 10 cycles, maxerr 3060 cycles)
[  371.481873] softlockup thread 1 started up.
[  371.481968] Booting processor 2/8 APIC 0x10
[  371.493111] Initializing CPU#2
[  371.639864] Calibrating delay using timer specific routine.. 6672.50 BogoMIPS (lpj=33362516)
[  371.639880] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  371.639882] CPU: L2 cache: 1024K
[  371.639883] CPU: L3 cache: 8192K
[  371.639886] CPU 2 -> Node 0
[  371.639888] CPU: Hyper-Threading is disabled
[  371.639901] CPU2: Thermal monitoring enabled (TM1)
[  371.641006]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  371.641017] APIC error on CPU2: 00(40)
[  371.641021] CPU 2: Syncing TSC to CPU 0.
[  371.641460] CPU 2: synchronized TSC with CPU 0 (last diff -56 cycles, maxerr 3890 cycles)
[  371.641486] softlockup thread 2 started up.
[  371.641589] Booting processor 3/8 APIC 0x16
[  371.652751] Initializing CPU#3
[  371.799373] Calibrating delay using timer specific routine.. 6672.59 BogoMIPS (lpj=33362958)
[  371.799388] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  371.799390] CPU: L2 cache: 1024K
[  371.799392] CPU: L3 cache: 8192K
[  371.799394] CPU 3 -> Node 0
[  371.799397] CPU: Hyper-Threading is disabled
[  371.799409] CPU3: Thermal monitoring enabled (TM1)
[  371.800515]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  371.800526] APIC error on CPU3: 00(40)
[  371.800529] CPU 3: Syncing TSC to CPU 0.
[  371.800965] CPU 3: synchronized TSC with CPU 0 (last diff -45 cycles, maxerr 4040 cycles)
[  371.800992] softlockup thread 3 started up.
[  371.801087] Booting processor 4/8 APIC 0x20
[  242.970377] Initializing CPU#4
[  243.116990] Calibrating delay using timer specific routine.. 6672.69 BogoMIPS (lpj=33363488)
[  243.117013] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  243.117015] CPU: L2 cache: 1024K
[  243.117016] CPU: L3 cache: 8192K
[  243.117020] CPU 4 -> Node 1
[  243.117023] CPU: Hyper-Threading is disabled
[  243.117040] CPU4: Thermal monitoring enabled (TM1)
[  243.118176]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  243.118190] APIC error on CPU4: 00(40)
[  243.118195] CPU 4: Syncing TSC to CPU 0.
[  371.960880] CPU 4: synchronized TSC with CPU 0 (last diff -30 cycles, maxerr 6740 cycles)
[  371.960922] softlockup thread 4 started up.
[  371.961024] Booting processor 5/8 APIC 0x26
[  243.130286] Initializing CPU#5
[  243.276500] Calibrating delay using timer specific routine.. 6672.74 BogoMIPS (lpj=33363706)
[  243.276516] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  243.276518] CPU: L2 cache: 1024K
[  243.276519] CPU: L3 cache: 8192K
[  243.276522] CPU 5 -> Node 1
[  243.276525] CPU: Hyper-Threading is disabled
[  243.276537] CPU5: Thermal monitoring enabled (TM1)
[  243.277668]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  243.277679] APIC error on CPU5: 00(40)
[  243.277683] CPU 5: Syncing TSC to CPU 0.
[  372.120369] CPU 5: synchronized TSC with CPU 0 (last diff -73 cycles, maxerr 6700 cycles)
[  372.120407] softlockup thread 5 started up.
[  372.120504] Booting processor 6/8 APIC 0x30
[  243.289766] Initializing CPU#6
[  243.436010] Calibrating delay using timer specific routine.. 6672.98 BogoMIPS (lpj=33364932)
[  243.436026] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  243.436028] CPU: L2 cache: 1024K
[  243.436029] CPU: L3 cache: 8192K
[  243.436032] CPU 6 -> Node 1
[  243.436035] CPU: Hyper-Threading is disabled
[  243.436048] CPU6: Thermal monitoring enabled (TM1)
[  243.437176]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  243.437187] APIC error on CPU6: 00(40)
[  243.437191] CPU 6: Syncing TSC to CPU 0.
[  372.279874] CPU 6: synchronized TSC with CPU 0 (last diff -163 cycles, maxerr 6700 cycles)
[  372.279915] softlockup thread 6 started up.
[  372.280028] Booting processor 7/8 APIC 0x36
[  243.449291] Initializing CPU#7
[  243.595520] Calibrating delay using timer specific routine.. 6673.06 BogoMIPS (lpj=33365308)
[  243.595536] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  243.595538] CPU: L2 cache: 1024K
[  243.595539] CPU: L3 cache: 8192K
[  243.595543] CPU 7 -> Node 1
[  243.595545] CPU: Hyper-Threading is disabled
[  243.595558] CPU7: Thermal monitoring enabled (TM1)
[  243.596686]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  243.596697] APIC error on CPU7: 00(40)
[  243.596701] CPU 7: Syncing TSC to CPU 0.
[  372.439382] CPU 7: synchronized TSC with CPU 0 (last diff -25 cycles, maxerr 6700 cycles)
[  372.439406] Brought up 8 CPUs
[  372.439423] softlockup thread 7 started up.
[  372.508651] time.c: Using PIT/HPET based timekeeping.
[  372.514688] testing NMI watchdog ... CPU#0: NMI appears to be stuck (12->12)!
[  372.623502] checking if image is initramfs... it is
[  372.683597] NET: Registered protocol family 16
[  372.688970] ACPI: bus type pci registered
[  372.693789] PCI: Using configuration type 1
[  372.701977] ACPI: Subsystem revision 20050902
[  372.764709] ACPI: Interpreter enabled
[  372.769099] ACPI: Using IOAPIC for interrupt routing
[  372.777626] ACPI: PCI Root Bridge [VP00] (0000:00)
[  372.783347] PCI: Probing PCI hardware (bus 00)
[  372.791758] Boot video device is 0000:00:01.0
[  372.792283] PCI: Ignoring BAR0-3 of IDE controller 0000:00:0f.1
[  372.799534] ACPI: PCI Interrupt Routing Table [\_SB_.VP00._PRT]
[  372.805865] ACPI: PCI Root Bridge [VP01] (0000:01)
[  372.811612] PCI: Probing PCI hardware (bus 01)
[  372.820281] ACPI: PCI Interrupt Routing Table [\_SB_.VP01._PRT]
[  372.821600] ACPI: PCI Root Bridge [VP02] (0000:02)
[  372.827350] PCI: Probing PCI hardware (bus 02)
[  372.835635] ACPI: PCI Interrupt Routing Table [\_SB_.VP02._PRT]
[  372.838517] ACPI: PCI Root Bridge [VP03] (0000:04)
[  372.844260] PCI: Probing PCI hardware (bus 04)
[  372.852548] ACPI: PCI Interrupt Routing Table [\_SB_.VP03._PRT]
[  372.855032] ACPI: PCI Root Bridge [VP04] (0000:06)
[  372.860747] PCI: Probing PCI hardware (bus 06)
[  372.869001] ACPI: PCI Interrupt Routing Table [\_SB_.VP04._PRT]
[  372.871669] ACPI: PCI Root Bridge [VP05] (0000:08)
[  372.877417] PCI: Probing PCI hardware (bus 08)
[  372.885738] ACPI: PCI Interrupt Routing Table [\_SB_.VP05._PRT]
[  372.888439] ACPI: PCI Root Bridge [VP06] (0000:0a)
[  372.894163] PCI: Probing PCI hardware (bus 0a)
[  372.902467] ACPI: PCI Interrupt Routing Table [\_SB_.VP06._PRT]
[  372.905114] ACPI: PCI Root Bridge [VP07] (0000:0c)
[  372.910861] PCI: Probing PCI hardware (bus 0c)
[  372.919062] ACPI: PCI Interrupt Routing Table [\_SB_.VP07._PRT]
[  372.921803] ACPI: PCI Root Bridge [VP10] (0000:0e)
[  372.927544] PCI: Probing PCI hardware (bus 0e)
[  372.936358] ACPI: PCI Interrupt Routing Table [\_SB_.VP10._PRT]
[  372.937885] ACPI: PCI Root Bridge [VP11] (0000:0f)
[  372.943619] PCI: Probing PCI hardware (bus 0f)
[  372.952311] ACPI: PCI Interrupt Routing Table [\_SB_.VP11._PRT]
[  372.953723] ACPI: PCI Root Bridge [VP12] (0000:10)
[  372.959471] PCI: Probing PCI hardware (bus 10)
[  372.967770] ACPI: PCI Interrupt Routing Table [\_SB_.VP12._PRT]
[  372.970451] ACPI: PCI Root Bridge [VP13] (0000:12)
[  372.976209] PCI: Probing PCI hardware (bus 12)
[  372.984487] ACPI: PCI Interrupt Routing Table [\_SB_.VP13._PRT]
[  372.987187] ACPI: PCI Root Bridge [VP14] (0000:14)
[  372.992903] PCI: Probing PCI hardware (bus 14)
[  373.001171] ACPI: PCI Interrupt Routing Table [\_SB_.VP14._PRT]
[  373.003746] ACPI: PCI Root Bridge [VP15] (0000:16)
[  373.009481] PCI: Probing PCI hardware (bus 16)
[  373.017766] ACPI: PCI Interrupt Routing Table [\_SB_.VP15._PRT]
[  373.020441] ACPI: PCI Root Bridge [VP16] (0000:18)
[  373.026185] PCI: Probing PCI hardware (bus 18)
[  373.034475] ACPI: PCI Interrupt Routing Table [\_SB_.VP16._PRT]
[  373.037094] ACPI: PCI Root Bridge [VP17] (0000:1a)
[  373.042808] PCI: Probing PCI hardware (bus 1a)
[  373.052328] ACPI: PCI Interrupt Routing Table [\_SB_.VP17._PRT]
[  373.056903] SCSI subsystem initialized
[  373.061417] PCI: Using ACPI for IRQ routing
[  373.066423] PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
[  373.076690] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[  373.090986] PCI: Bridge: 0000:1a:01.0
[  373.095393]   IO window: f000-ffff
[  373.099472]   MEM window: f8600000-f86fffff
[  373.104475]   PREFETCH window: disabled.
[  373.114868] IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
[  373.134138] IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
[  373.149954] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[  373.157784] SGI XFS with large block/inode numbers, no debug enabled
[  373.167417] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[  373.347957] Real Time Clock Driver v1.12
[  373.352690] Linux agpgart interface v0.101 (c) Dave Jones
[  373.360764] serio: i8042 AUX port at 0x60,0x64 irq 12
[  373.366921] serio: i8042 KBD port at 0x60,0x64 irq 1
[  373.372911] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
[  373.382632] ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[  373.388853] ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[  373.396024] io scheduler noop registered
[  373.400804] io scheduler anticipatory registered
[  373.406410] io scheduler deadline registered
[  373.411662] io scheduler cfq registered
[  373.420116] RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
[  373.431148] loop: loaded (max 8 devices)
[  373.435886] Intel(R) PRO/1000 Network Driver - version 6.0.60-k2-NAPI
[  373.443638] Copyright (c) 1999-2005 Intel Corporation.
[  373.450091] ACPI: PCI Interrupt 0000:1b:04.0[A] -> GSI 124 (level, low) -> IRQ 169
[  373.798723] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[  373.806969] ACPI: PCI Interrupt 0000:1b:04.1[B] -> GSI 128 (level, low) -> IRQ 177
[  374.157872] e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
[  374.166146] ACPI: PCI Interrupt 0000:1b:06.0[A] -> GSI 132 (level, low) -> IRQ 185
[  374.516534] e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network Connection
[  374.526558] ACPI: PCI Interrupt 0000:1b:06.1[B] -> GSI 136 (level, low) -> IRQ 193
[  374.875427] e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network Connection
[  374.883807] tg3.c:v3.42 (Oct 3, 2005)
[  374.888214] ACPI: PCI Interrupt 0000:01:01.0[A] -> GSI 24 (level, low) -> IRQ 201
[  374.926685] eth4: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:7e
[  374.940607] eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[0] 
[  374.950619] eth4: dma_rwctrl[769f0000]
[  374.955132] ACPI: PCI Interrupt 0000:01:01.1[B] -> GSI 28 (level, low) -> IRQ 209
[  374.994160] eth5: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:7f
[  375.008044] eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  375.018045] eth5: dma_rwctrl[769f0000]
[  375.022610] ACPI: PCI Interrupt 0000:0f:01.0[A] -> GSI 94 (level, low) -> IRQ 217
[  375.055039] eth6: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:78
[  375.068919] eth6: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[0] 
[  375.078946] eth6: dma_rwctrl[769f0000]
[  375.083512] ACPI: PCI Interrupt 0000:0f:01.1[B] -> GSI 98 (level, low) -> IRQ 225
[  375.122777] eth7: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:79
[  375.136662] eth7: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  375.146689] eth7: dma_rwctrl[769f0000]
[  375.151353] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
[  375.158957] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
[  375.169595] libata version 1.12 loaded.
[  375.170516] mice: PS/2 mouse device common for all mice
[  375.176946] oprofile: using NMI interrupt.
[  375.182071] NET: Registered protocol family 2
[  375.213704] input: AT Translated Set 2 keyboard on isa0060/serio0
[  375.294629] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[  375.307385] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[  375.320764] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[  375.329743] TCP: Hash tables configured (established 262144 bind 65536)
[  375.337662] TCP reno registered
[  375.341583] TCP bic registered
[  375.345286] NET: Registered protocol family 1
[  375.350522] NET: Registered protocol family 17
[  375.356473] Freeing unused kernel memory: 208k freed
[  375.447440] Loading AIC-94xx Linux SAS/SATA Family Driver, Rev: 1.0.7-3
[  375.447444] 
[  375.457323] Probing Adaptec AIC-94xx Controller(s)...
[  375.463402] ACPI: PCI Interrupt 0000:01:02.0[A] -> GSI 25 (level, low) -> IRQ 233
[  375.495851] input: ImExPS/2 Generic Explorer Mouse on isa0060/serio1
[  375.602789] Probing Adaptec AIC-94xx Controller(s)...
[  375.608884] ACPI: PCI Interrupt 0000:0f:02.0[A] -> GSI 95 (level, low) -> IRQ 50
[  375.763377] scsi0 : Adaptec AIC-9410 SAS/SATA Host Adapter
[  375.976623]   Vendor: IBM-ESXS  Model: ST973401SS    F   Rev: B515
[  375.984267]   Type:   Direct-Access                      ANSI SCSI revision: 04
[  375.993059] adp94xx:1:128:0: Tagged Queuing enabled.  Depth 32
[  376.000815] SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
[  376.010208] SCSI device sda: drive cache: write through
[  376.017262] SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
[  376.026625] SCSI device sda: drive cache: write through
[  376.032871]  sda: sda1 sda2 sda3
[  376.045017] Attached scsi disk sda at scsi0, channel 1, id 128, lun 0
[  376.052886] Attached scsi generic sg0 at scsi0, channel 1, id 128, lun 0,  type 0
[  376.119922] scsi1 : Adaptec AIC-9410 SAS/SATA Host Adapter
[  376.327989]   Vendor: IBM-ESXS  Model: ST973401SS    F   Rev: B515
[  376.335661]   Type:   Direct-Access                      ANSI SCSI revision: 04
[  376.344446] adp94xx:0:128:0: Tagged Queuing enabled.  Depth 32
[  376.352208] SCSI device sdb: 143374000 512-byte hdwr sectors (73407 MB)
[  376.361579] SCSI device sdb: drive cache: write through
[  376.368576] SCSI device sdb: 143374000 512-byte hdwr sectors (73407 MB)
[  376.377947] SCSI device sdb: drive cache: write through
[  376.384185]  sdb: sdb1 sdb2 sdb3 < sdb5 >
[  376.403469] Attached scsi disk sdb at scsi1, channel 0, id 128, lun 0
[  376.411349] Attached scsi generic sg1 at scsi1, channel 0, id 128, lun 0,  type 0
[  376.484646] AIC-94xx controller(s) attached = 2.
[  376.484648] 
[  376.612494] kjournald starting.  Commit interval 5 seconds
[  376.612476] EXT3-fs: mounted filesystem with ordered data mode.
[  428.249374] EXT3 FS on sda2, internal journal
[  428.501350] kjournald starting.  Commit interval 5 seconds
[  428.510383] EXT3 FS on sda1, internal journal
[  428.510392] EXT3-fs: mounted filesystem with ordered data mode.
[  428.538172] kjournald starting.  Commit interval 5 seconds
[  428.548850] EXT3 FS on sdb2, internal journal
[  428.548857] EXT3-fs: mounted filesystem with ordered data mode.
[  428.568441] XFS mounting filesystem sdb5
[  428.661737] Ending clean XFS mount for filesystem: sdb5

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Christoph Lameter]

On Mon, 17 Oct 2005, Ravikiran G Thirumalai wrote:

> Maybe someone with access to ia64 NUMA boxen can check if the NODE(0)
> solution works (and does not break anything) on ia64?  Chrisoph, can you help?

Umm... SGI does not use the swiotlb and we do not have these issues. HP 
does use the swiotlb on IA64. CCing John and Alex.

For the newcomers: Thread is at 
http://marc.theaimsgroup.com/?t=112954203900001&r=1&w=2

Proposed patch by Kiran:

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-14 
00:06:21.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 
00:05:22.000000000 -0700
@@ -123,7 +123,7 @@
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
+	io_tlb_start = alloc_bootmem_node(NODE_DATA(0), io_tlb_nslabs *
 					       (1 << IO_TLB_SHIFT));
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Mon, 2005-10-17 at 11:20 -0700, Christoph Lameter wrote:
> On Mon, 17 Oct 2005, Ravikiran G Thirumalai wrote:
> 
> > Maybe someone with access to ia64 NUMA boxen can check if the NODE(0)
> > solution works (and does not break anything) on ia64?  Chrisoph, can you help?
> 
> Umm... SGI does not use the swiotlb and we do not have these issues. HP 
> does use the swiotlb on IA64. CCing John and Alex.
...
> @@ -123,7 +123,7 @@
>  	/*
>  	 * Get IO TLB memory from the low pages
>  	 */
> -	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
> +	io_tlb_start = alloc_bootmem_node(NODE_DATA(0), io_tlb_nslabs *

   HP ia64 boxes typically use a hardware I/O TLB, so this is not the
normal case.  However, the sx1000 boxes are exactly an example that will
break because of this assumption about memory layout.  These boxes can
be configured to have various ratios of node local memory and
interleaved memory.  Node local memory starts well above 4GB.
Interleaved memory is zero-based, and described in it's own proximity
domain.  It therefore looks like a memory-only node.  I believe the
above code change would cause us to allocate memory from the node local
range, way too high in the address space for bounce buffers.

   BTW, I've got a patch in Tony's testing branch that allows a late
initialization of the swiotlb.  This is currently only useful on ia64
since we have 4GB of ZONE_DMA, but may be useful on x86-ish archs when
the 4GB zone is introduced.

	Alex

-- 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Mon, Oct 17, 2005 at 01:04:01PM -0600, Alex Williamson wrote:
> On Mon, 2005-10-17 at 11:20 -0700, Christoph Lameter wrote:
> > On Mon, 17 Oct 2005, Ravikiran G Thirumalai wrote:
> > 
> > > Maybe someone with access to ia64 NUMA boxen can check if the NODE(0)
> > > solution works (and does not break anything) on ia64?  Chrisoph, can you help?
> > 
> > Umm... SGI does not use the swiotlb and we do not have these issues. HP 
> > does use the swiotlb on IA64. CCing John and Alex.
> ...
> > @@ -123,7 +123,7 @@
> >  	/*
> >  	 * Get IO TLB memory from the low pages
> >  	 */
> > -	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
> > +	io_tlb_start = alloc_bootmem_node(NODE_DATA(0), io_tlb_nslabs *
> 
>    HP ia64 boxes typically use a hardware I/O TLB, so this is not the
> normal case.  However, the sx1000 boxes are exactly an example that will
> break because of this assumption about memory layout.  These boxes can
> be configured to have various ratios of node local memory and
> interleaved memory.  Node local memory starts well above 4GB.
> Interleaved memory is zero-based, and described in it's own proximity
> domain.  It therefore looks like a memory-only node.  I believe the
> above code change would cause us to allocate memory from the node local
> range, way too high in the address space for bounce buffers.

This memory only node has a node id? Then how about a patch which iterates over 
nodes in swiotlb.c, trying to allocate DMA'ble memory from node 0 and above
until it gets proper memory for swiotlb?

Would that be accepatble?  I can quickly make a patch for that if it is
acceptable..

Thanks,
Kiran

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Mon, 2005-10-17 at 12:26 -0700, Ravikiran G Thirumalai wrote:

> This memory only node has a node id? Then how about a patch which iterates over 
> nodes in swiotlb.c, trying to allocate DMA'ble memory from node 0 and above
> until it gets proper memory for swiotlb?
> 
> Would that be accepatble?  I can quickly make a patch for that if it is
> acceptable..

   Yes, the memory-only node is just another node.  Iterating over all
nodes sounds a little brute force, but I guess it should work.  FWIW,
here's the results of the previous one-liner on an HP Superdome (booting
w/ machvec=dig to use the swiotlb instead of hardware iotlb):

2.6.14-rc4-mm1, before patch:
Placing software IO TLB between 0x4cdc000 - 0x8cdc000

after patch:
Placing software IO TLB between 0x74104e6b200 - 0x74108e6b200

Thanks,

	Alex

-- 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Mon, Oct 17, 2005 at 11:53:54AM +0200, Andi Kleen wrote:
> On Monday 17 October 2005 11:50, Andrew Morton wrote:
> 
> >
> > This is an ia64 patch - what point was there in testing it on an x460?
> >
> > Is something missing here?
> 
> x86-64 shares that code with ia64.
> 
> The patch is actually not quite correct - in theory node 0 could be too small 
> to contain the full swiotlb bounce buffers.

Good point. I missed that possibility. 

> 
> The real fix would be to get rid of the pgdata lists and just walk the 
> node_online_map on bootmem.c. The memory hotplug guys have
> a patch pending for this.

Yes, I just saw Yasunori-san's patch.  Would that be merged for 2.6.14?
'Cause 2.6.14 is broken as of now for x86_64 boxes with more than 4G ram.

Thanks,
Kiran

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Monday 17 October 2005 17:30, Ravikiran G Thirumalai wrote:

> Yes, I just saw Yasunori-san's patch.  Would that be merged for 2.6.14?

I think both are too risky at this point.

> 'Cause 2.6.14 is broken as of now for x86_64 boxes with more than 4G ram.

... that need swiotlb. Which makes that set much much smaller.
Sorry, but your scalex platform is definitely not anywhere release critical 
right now.

-Andi

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andrew Morton]

Andi Kleen <ak@suse.de> wrote:
>
> On Monday 17 October 2005 17:30, Ravikiran G Thirumalai wrote:
> 
> > Yes, I just saw Yasunori-san's patch.  Would that be merged for 2.6.14?
> 
> I think both are too risky at this point.
> 

Maybe.

There seem to be a lot of proposed solutions floating about and I fear that
different people will try to fix this in different ways.  Do we all agree
that this patch is the correct solution to this problem, or is something
more needed?



From: Yasunori Goto <y-goto@jp.fujitsu.com>

This is a patch to guarantee that alloc_bootmem_low() allocate DMA area.

Current alloc_bootmem_low() is just specify "goal=0".  And it is used for
__alloc_bootmem_core() to decide which address is better.  However, there
is no guarantee that __alloc_bootmem_core() allocate DMA area when goal=0
is specified.  Even if there is no DMA'ble area in searching node, it
allocates higher address than MAX_DMA_ADDRESS.

__alloc_bootmem_core() is called by order of for_each_pgdat() in
__alloc_bootmem().  So, if first node (node_id = 0) has DMA'ble area, no
trouble will occur.  However, our new Itanium2 server can change which node
has lower address.  And panic really occurred on it.  The message was
"bounce buffer is not DMA'ble" in swiothl_map_single().

To avoid this panic, following patch confirms allocated area, and retry if
it is not in DMA.  I tested this patch on my Tiger 4 and our new server.

Signed-off-by Yasunori Goto <y-goto@jp.fujitsu.com>

Signed-off-by: Andrew Morton <akpm@osdl.org>
---

 include/linux/bootmem.h |   16 ++++++++++++----
 mm/bootmem.c            |   43 +++++++++++++++++++++++++++++++++++++++----
 2 files changed, 51 insertions(+), 8 deletions(-)

diff -puN include/linux/bootmem.h~guarantee-dma-area-for-alloc_bootmem_low include/linux/bootmem.h
--- devel/include/linux/bootmem.h~guarantee-dma-area-for-alloc_bootmem_low	2005-10-11 00:34:32.000000000 -0700
+++ devel-akpm/include/linux/bootmem.h	2005-10-11 00:34:32.000000000 -0700
@@ -40,6 +40,14 @@ typedef struct bootmem_data {
 					 * up searching */
 } bootmem_data_t;
 
+static inline unsigned long max_dma_physaddr(void)
+{
+
+	if (MAX_DMA_ADDRESS == ~0UL)
+		return MAX_DMA_ADDRESS;
+	return __pa(MAX_DMA_ADDRESS);
+}
+
 extern unsigned long __init bootmem_bootmap_pages (unsigned long);
 extern unsigned long __init init_bootmem (unsigned long addr, unsigned long memend);
 extern void __init free_bootmem (unsigned long addr, unsigned long size);
@@ -47,11 +55,11 @@ extern void * __init __alloc_bootmem (un
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 extern void __init reserve_bootmem (unsigned long addr, unsigned long size);
 #define alloc_bootmem(x) \
-	__alloc_bootmem((x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem((x), SMP_CACHE_BYTES, max_dma_physaddr())
 #define alloc_bootmem_low(x) \
 	__alloc_bootmem((x), SMP_CACHE_BYTES, 0)
 #define alloc_bootmem_pages(x) \
-	__alloc_bootmem((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem((x), PAGE_SIZE, max_dma_physaddr())
 #define alloc_bootmem_low_pages(x) \
 	__alloc_bootmem((x), PAGE_SIZE, 0)
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
@@ -64,9 +72,9 @@ extern unsigned long __init free_all_boo
 extern void * __init __alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal);
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 #define alloc_bootmem_node(pgdat, x) \
-	__alloc_bootmem_node((pgdat), (x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_node((pgdat), (x), SMP_CACHE_BYTES, max_dma_physaddr())
 #define alloc_bootmem_pages_node(pgdat, x) \
-	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
+	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, max_dma_physaddr())
 #define alloc_bootmem_low_pages_node(pgdat, x) \
 	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, 0)
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
diff -puN mm/bootmem.c~guarantee-dma-area-for-alloc_bootmem_low mm/bootmem.c
--- devel/mm/bootmem.c~guarantee-dma-area-for-alloc_bootmem_low	2005-10-11 00:34:32.000000000 -0700
+++ devel-akpm/mm/bootmem.c	2005-10-11 00:34:32.000000000 -0700
@@ -382,19 +382,54 @@ unsigned long __init free_all_bootmem (v
 	return(free_all_bootmem_core(NODE_DATA(0)));
 }
 
+static int __init is_dma_required(unsigned long goal)
+{
+	return goal < max_dma_physaddr() ? 1 : 0;
+}
+
+static int __init unmatch_dma_required(void *ptr, unsigned long goal)
+{
+
+	if(is_dma_required(goal) && (unsigned long)ptr >= MAX_DMA_ADDRESS)
+		return 1;
+
+	return 0;
+}
+
 void * __init __alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal)
 {
 	pg_data_t *pgdat = pgdat_list;
 	void *ptr;
+	int retried = 0;
+
+retry:
+	for_each_pgdat(pgdat){
 
-	for_each_pgdat(pgdat)
-		if ((ptr = __alloc_bootmem_core(pgdat->bdata, size,
-						align, goal)))
-			return(ptr);
+		ptr = __alloc_bootmem_core(pgdat->bdata, size,
+					   align, goal);
+		if (!ptr)
+			continue;
+
+		if (unmatch_dma_required(ptr, goal) && !retried){
+			/* DMA is required, but normal area is allocated.
+			   Other node might have DMA, should try it. */
+			free_bootmem_core(pgdat->bdata, virt_to_phys(ptr), size);
+			continue;
+		}
+
+		return ptr;
+	}
 
 	/*
 	 * Whoops, we cannot satisfy the allocation request.
 	 */
+	if (is_dma_required(goal) && !retried){
+		printk(KERN_WARNING "bootmem alloc DMA of %lu bytes failed, retry normal area!\n", size);
+		dump_stack();
+		retried++;
+		goto retry;
+	}
+
 	printk(KERN_ALERT "bootmem alloc of %lu bytes failed!\n", size);
 	panic("Out of memory");
 	return NULL;
_

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Andrew Morton wrote:
> 
> There seem to be a lot of proposed solutions floating about and I fear that
> different people will try to fix this in different ways.  Do we all agree
> that this patch is the correct solution to this problem, or is something
> more needed?

I think this will fix it. 

The naming is horrible, though, and that whole "goal" parameter is 
senseless and ugly.

It should just be a flag on whether we want DMA'able memory or not. That's 
what it _is_, it's just strangely implemented, making the code less 
readable.

Since the patch changes all the users of that third parameter _anyway_, it 
should probably be fixed to just make the parameter sane instead.

		Linus

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

[-- Attachment #1: Type: text/plain, Size: 2545 bytes --]

On Mon, Oct 17, 2005 at 02:11:20PM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 17 Oct 2005, Andrew Morton wrote:
> > 
> > There seem to be a lot of proposed solutions floating about and I fear that
> > different people will try to fix this in different ways.  Do we all agree
> > that this patch is the correct solution to this problem, or is something
> > more needed?
> 
> I think this will fix it. 
> 

I just tried Yasunori-sans patch on our x460.  It doesn't fix the problem.  
Attaching the dmesg capture. However, the patch to iterate over nodes and 
allocate bootmem works, and should work for ia64s, boxes with insufficient 
memory on node 0, nodes with just cpus etc.  I have requested Alex to try this 
on the superdome (although the the use of swiotlb on superdomes seem to be
optional).  If this works on superdomes, then please apply.  This has been
tested on x460.

Thanks,
Kiran

Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-17 13:27:35.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-17 16:00:44.000000000 -0700
@@ -106,6 +106,8 @@
 __setup("swiotlb=", setup_io_tlb_npages);
 /* make io_tlb_overflow tunable too? */
 
+#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
+
 /*
  * Statically reserve bounce buffer space and initialize bounce buffer data
  * structures for the software IO TLB used to implement the PCI DMA API.
@@ -114,17 +116,32 @@
 swiotlb_init_with_default_size (size_t default_size)
 {
 	unsigned long i;
+	unsigned long iotlbsz;
+	int node;
 
 	if (!io_tlb_nslabs) {
 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
 	}
 
+	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
+
 	/*
-	 * Get IO TLB memory from the low pages
+	 * Get IO TLB memory from the 0-4G range
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	
+	for_each_node(node) {
+		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
+		if (io_tlb_start) {
+			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
+				break;
+			free_bootmem_node(NODE_DATA(node), 
+					  virt_to_phys(io_tlb_start), iotlbsz);
+			io_tlb_start = NULL;
+		}
+	}
+
+	
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

[-- Attachment #2: dmesg-x460-fail --]
[-- Type: text/plain, Size: 23878 bytes --]

[    0.000000] Bootdata ok (command line is root=LABEL=/  console=ttyS1,115200 console=ttyS0,115200 console=tty1 console=tty0)
[    0.000000] Linux version 2.6.14-rc4 (root@x460) (gcc version 3.4.3 20050227 (Red Hat 3.4.3-22.1)) #4 SMP Mon Oct 17 16:32:57 PDT 2005
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  BIOS-e820: 0000000000000000 - 0000000000098000 (usable)
[    0.000000]  BIOS-e820: 0000000000098000 - 00000000000a0000 (reserved)
[    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
[    0.000000]  BIOS-e820: 0000000000100000 - 00000000e7f94d40 (usable)
[    0.000000]  BIOS-e820: 00000000e7f94d40 - 00000000e7fa6a40 (ACPI data)
[    0.000000]  BIOS-e820: 00000000e7fa6a40 - 00000000e8000000 (reserved)
[    0.000000]  BIOS-e820: 00000000fec00000 - 0000000100000000 (reserved)
[    0.000000]  BIOS-e820: 0000000100000000 - 00000003f8000000 (usable)
[    0.000000] ACPI: RSDP (v000 IBM                                   ) @ 0x00000000000fdcf0
[    0.000000] ACPI: RSDT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa69c0
[    0.000000] ACPI: FADT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6940
[    0.000000] ACPI: MADT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6840
[    0.000000] ACPI: SRAT (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa6700
[    0.000000] ACPI: HPET (v001 IBM    EXA01ZEU 0x00001000 IBM  0x45444f43) @ 0x00000000e7fa66c0
[    0.000000] ACPI: SSDT (v001 IBM    VIGSSDT0 0x00001000 INTL 0x20030122) @ 0x00000000e7f9f0c0
[    0.000000] ACPI: SSDT (v001 IBM    VIGSSDT1 0x00001000 INTL 0x20030122) @ 0x00000000e7f97b00
[    0.000000] ACPI: DSDT (v001 IBM    EXA01ZEU 0x00001000 INTL 0x20030122) @ 0x0000000000000000
[    0.000000] SRAT: PXM 0 -> APIC 0 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 6 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 16 -> Node 0
[    0.000000] SRAT: PXM 0 -> APIC 22 -> Node 0
[    0.000000] SRAT: PXM 1 -> APIC 32 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 38 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 48 -> Node 1
[    0.000000] SRAT: PXM 1 -> APIC 54 -> Node 1
[    0.000000] SRAT: Node 0 PXM 0 0-e7ffffff
[    0.000000] SRAT: Node 0 PXM 0 0-207ffffff
[    0.000000] SRAT: Node 1 PXM 1 208000000-3f7ffffff
[    0.000000] Using 22 for the hash shift. Max adder is 3f7ffffff 
[    0.000000] Bootmem setup node 0 0000000000000000-0000000207ffffff
[    0.000000] Bootmem setup node 1 0000000208000000-00000003f7ffffff
[    0.000000] On node 0 totalpages: 2031403
[    0.000000]   DMA zone: 3992 pages, LIFO batch:1
[    0.000000]   Normal zone: 2027411 pages, LIFO batch:31
[    0.000000]   HighMem zone: 0 pages, LIFO batch:1
[    0.000000] On node 1 totalpages: 2031615
[    0.000000]   DMA zone: 0 pages, LIFO batch:1
[    0.000000]   Normal zone: 2031615 pages, LIFO batch:31
[    0.000000]   HighMem zone: 0 pages, LIFO batch:1
[    0.000000] ACPI: PM-Timer IO Port: 0x9c
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
[    0.000000] Processor #0 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x01] lapic_id[0x06] enabled)
[    0.000000] Processor #6 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x02] lapic_id[0x10] enabled)
[    0.000000] Processor #16 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x03] lapic_id[0x16] enabled)
[    0.000000] Processor #22 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
[    0.000000] Processor #32 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x11] lapic_id[0x26] enabled)
[    0.000000] Processor #38 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x12] lapic_id[0x30] enabled)
[    0.000000] Processor #48 15:4 APIC version 20
[    0.000000] ACPI: LAPIC (acpi_id[0x13] lapic_id[0x36] enabled)
[    0.000000] Processor #54 15:4 APIC version 20
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x00] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x01] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x02] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x03] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x10] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x11] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x12] dfl dfl lint[0x1])
[    0.000000] ACPI: LAPIC_NMI (acpi_id[0x13] dfl dfl lint[0x1])
[    0.000000] ACPI: IOAPIC (id[0x0f] address[0xfec00000] gsi_base[0])
[    0.000000] IOAPIC[0]: apic_id 15, version 17, address 0xfec00000, GSI 0-35
[    0.000000] ACPI: IOAPIC (id[0x0e] address[0xfec01000] gsi_base[35])
[    0.000000] IOAPIC[1]: apic_id 14, version 17, address 0xfec01000, GSI 35-70
[    0.000000] ACPI: IOAPIC (id[0x0d] address[0xfec02000] gsi_base[70])
[    0.000000] IOAPIC[2]: apic_id 13, version 17, address 0xfec02000, GSI 70-105
[    0.000000] ACPI: IOAPIC (id[0x0c] address[0xfec03000] gsi_base[105])
[    0.000000] IOAPIC[3]: apic_id 12, version 17, address 0xfec03000, GSI 105-140
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 8 global_irq 8 low edge)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 14 global_irq 14 low edge)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ2 used by override.
[    0.000000] ACPI: IRQ8 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] ACPI: IRQ14 used by override.
[    0.000000] Setting APIC routing to clustered
[    0.000000] ACPI: HPET id: 0x10142201 base: 0xfde84000
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] Allocating PCI resources starting at ea000000 (gap: e8000000:16c00000)
[    0.000000] Checking aperture...
[    0.000000] Built 2 zonelists
[    0.000000] Kernel command line: root=LABEL=/  console=ttyS1,115200 console=ttyS0,115200 console=tty1 console=tty0
[    0.000000] Initializing CPU#0
[    0.000000] PID hash table entries: 4096 (order: 12, 131072 bytes)
[    0.000000] time.c: Using 3.579545 MHz PM timer.
[    0.000000] time.c: Detected 3336.353 MHz processor.
[  331.470062] Console: colour VGA+ 80x25
[  331.978248] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[  332.010808] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[  332.163134] bootmem alloc DMA of 67108864 bytes failed, retry normal area!
[  332.171394] 
[  332.171395] Call Trace:<ffffffff8054d1b6>{__alloc_bootmem+227} <ffffffff8011c268>{swiotlb_init_with_default_size+51}
[  332.185629]        <ffffffff80549fac>{mem_init+76} <ffffffff8053a725>{start_kernel+320}
[  332.195279]        <ffffffff8053a237>{_sinittext+567} 
[  332.269214] Placing software IO TLB between 0x21051c000 - 0x21451c000
[  332.837489] Memory: 15927888k/16646144k available (2594k kernel code, 324192k reserved, 1356k data, 208k init)
[  332.992581] Calibrating delay using timer specific routine.. 6683.98 BogoMIPS (lpj=33419944)
[  333.002891] Mount-cache hash table entries: 256
[  333.008517] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  333.014787] CPU: L2 cache: 1024K
[  333.018668] CPU: L3 cache: 8192K
[  333.022565] CPU 0 -> Node 0
[  333.025890] using mwait in idle threads.
[  333.030578] CPU: Hyper-Threading is disabled
[  333.035732] CPU0: Thermal monitoring enabled (TM1)
[  333.041447] mtrr: v2.0 (20020519)
[  333.176831] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[  333.183779]  failed.
[  333.186407] timer doesn't work through the IO-APIC - disabling NMI Watchdog!
[  333.195849] Uhhuh. NMI received for unknown reason 35.
[  333.201975] Dazed and confused, but trying to continue
[  333.208117] Do you have a strange power saving mode enabled?
[  333.312735]  works.
[  333.315252] Using local APIC timer interrupts.
[  333.350450] Detected 10.426 MHz APIC timer.
[  333.355548] softlockup thread 0 started up.
[  333.360675] Booting processor 1/8 APIC 0x6
[  333.376738] Initializing CPU#1
[  333.520948] Calibrating delay using timer specific routine.. 6672.53 BogoMIPS (lpj=33362682)
[  333.520963] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  333.520965] CPU: L2 cache: 1024K
[  333.520966] CPU: L3 cache: 8192K
[  333.520969] CPU 1 -> Node 0
[  333.520971] CPU: Hyper-Threading is disabled
[  333.520983] CPU1: Thermal monitoring enabled (TM1)
[  333.522091]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  333.522102] APIC error on CPU1: 00(40)
[  333.522105] CPU 1: Syncing TSC to CPU 0.
[  333.522443] CPU 1: synchronized TSC with CPU 0 (last diff 10 cycles, maxerr 3060 cycles)
[  333.522468] softlockup thread 1 started up.
[  333.522561] Booting processor 2/8 APIC 0x10
[  333.533706] Initializing CPU#2
[  333.680458] Calibrating delay using timer specific routine.. 6672.80 BogoMIPS (lpj=33364014)
[  333.680474] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  333.680476] CPU: L2 cache: 1024K
[  333.680478] CPU: L3 cache: 8192K
[  333.680480] CPU 2 -> Node 0
[  333.680482] CPU: Hyper-Threading is disabled
[  333.680495] CPU2: Thermal monitoring enabled (TM1)
[  333.681600]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  333.681611] APIC error on CPU2: 00(40)
[  333.681614] CPU 2: Syncing TSC to CPU 0.
[  333.682057] CPU 2: synchronized TSC with CPU 0 (last diff 1 cycles, maxerr 3850 cycles)
[  333.682082] softlockup thread 2 started up.
[  333.682193] Booting processor 3/8 APIC 0x16
[  333.693347] Initializing CPU#3
[  333.839967] Calibrating delay using timer specific routine.. 6672.58 BogoMIPS (lpj=33362942)
[  333.839983] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  333.839985] CPU: L2 cache: 1024K
[  333.839986] CPU: L3 cache: 8192K
[  333.839989] CPU 3 -> Node 0
[  333.839991] CPU: Hyper-Threading is disabled
[  333.840004] CPU3: Thermal monitoring enabled (TM1)
[  333.841110]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  333.841121] APIC error on CPU3: 00(40)
[  333.841124] CPU 3: Syncing TSC to CPU 0.
[  333.841567] CPU 3: synchronized TSC with CPU 0 (last diff 56 cycles, maxerr 3850 cycles)
[  333.841593] softlockup thread 3 started up.
[  333.841689] Booting processor 4/8 APIC 0x20
[  205.772353] Initializing CPU#4
[  205.918959] Calibrating delay using timer specific routine.. 6672.85 BogoMIPS (lpj=33364294)
[  205.918982] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  205.918984] CPU: L2 cache: 1024K
[  205.918986] CPU: L3 cache: 8192K
[  205.918990] CPU 4 -> Node 1
[  205.918993] CPU: Hyper-Threading is disabled
[  205.919009] CPU4: Thermal monitoring enabled (TM1)
[  205.920145]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  205.920159] APIC error on CPU4: 00(40)
[  205.920163] CPU 4: Syncing TSC to CPU 0.
[  334.001485] CPU 4: synchronized TSC with CPU 0 (last diff -580 cycles, maxerr 8060 cycles)
[  334.001527] softlockup thread 4 started up.
[  334.001627] Booting processor 5/8 APIC 0x26
[  205.932263] Initializing CPU#5
[  206.078469] Calibrating delay using timer specific routine.. 6672.50 BogoMIPS (lpj=33362542)
[  206.078484] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  206.078486] CPU: L2 cache: 1024K
[  206.078488] CPU: L3 cache: 8192K
[  206.078491] CPU 5 -> Node 1
[  206.078493] CPU: Hyper-Threading is disabled
[  206.078506] CPU5: Thermal monitoring enabled (TM1)
[  206.079636]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  206.079648] APIC error on CPU5: 00(40)
[  206.079651] CPU 5: Syncing TSC to CPU 0.
[  334.160963] CPU 5: synchronized TSC with CPU 0 (last diff -95 cycles, maxerr 6740 cycles)
[  334.161001] softlockup thread 5 started up.
[  334.161101] Booting processor 6/8 APIC 0x30
[  206.091737] Initializing CPU#6
[  206.237979] Calibrating delay using timer specific routine.. 6673.07 BogoMIPS (lpj=33365394)
[  206.237995] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  206.237998] CPU: L2 cache: 1024K
[  206.237999] CPU: L3 cache: 8192K
[  206.238002] CPU 6 -> Node 1
[  206.238005] CPU: Hyper-Threading is disabled
[  206.238017] CPU6: Thermal monitoring enabled (TM1)
[  206.239145]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  206.239156] APIC error on CPU6: 00(40)
[  206.239159] CPU 6: Syncing TSC to CPU 0.
[  334.320471] CPU 6: synchronized TSC with CPU 0 (last diff 57 cycles, maxerr 6800 cycles)
[  334.320510] softlockup thread 6 started up.
[  334.320619] Booting processor 7/8 APIC 0x36
[  206.251256] Initializing CPU#7
[  206.397489] Calibrating delay using timer specific routine.. 6672.83 BogoMIPS (lpj=33364186)
[  206.397505] CPU: Trace cache: 12K uops, L1 D cache: 16K
[  206.397507] CPU: L2 cache: 1024K
[  206.397508] CPU: L3 cache: 8192K
[  206.397511] CPU 7 -> Node 1
[  206.397514] CPU: Hyper-Threading is disabled
[  206.397526] CPU7: Thermal monitoring enabled (TM1)
[  206.398656]                Intel(R) Xeon(TM) MP CPU 3.33GHz stepping 01
[  206.398666] APIC error on CPU7: 00(40)
[  206.398670] CPU 7: Syncing TSC to CPU 0.
[  334.479982] CPU 7: synchronized TSC with CPU 0 (last diff -760 cycles, maxerr 8020 cycles)
[  334.480006] Brought up 8 CPUs
[  334.480023] softlockup thread 7 started up.
[  334.549366] time.c: Using PIT/HPET based timekeeping.
[  334.555402] testing NMI watchdog ... CPU#0: NMI appears to be stuck (12->12)!
[  334.664231] checking if image is initramfs... it is
[  334.724472] NET: Registered protocol family 16
[  334.729831] ACPI: bus type pci registered
[  334.734651] PCI: Using configuration type 1
[  334.742672] ACPI: Subsystem revision 20050902
[  334.805446] ACPI: Interpreter enabled
[  334.809834] ACPI: Using IOAPIC for interrupt routing
[  334.818357] ACPI: PCI Root Bridge [VP00] (0000:00)
[  334.824082] PCI: Probing PCI hardware (bus 00)
[  334.832542] Boot video device is 0000:00:01.0
[  334.833064] PCI: Ignoring BAR0-3 of IDE controller 0000:00:0f.1
[  334.840283] ACPI: PCI Interrupt Routing Table [\_SB_.VP00._PRT]
[  334.847085] ACPI: PCI Root Bridge [VP01] (0000:01)
[  334.852820] PCI: Probing PCI hardware (bus 01)
[  334.861538] ACPI: PCI Interrupt Routing Table [\_SB_.VP01._PRT]
[  334.862989] ACPI: PCI Root Bridge [VP02] (0000:02)
[  334.868741] PCI: Probing PCI hardware (bus 02)
[  334.877037] ACPI: PCI Interrupt Routing Table [\_SB_.VP02._PRT]
[  334.879307] ACPI: PCI Root Bridge [VP03] (0000:04)
[  334.885052] PCI: Probing PCI hardware (bus 04)
[  334.893360] ACPI: PCI Interrupt Routing Table [\_SB_.VP03._PRT]
[  334.896057] ACPI: PCI Root Bridge [VP04] (0000:06)
[  334.901825] PCI: Probing PCI hardware (bus 06)
[  334.910117] ACPI: PCI Interrupt Routing Table [\_SB_.VP04._PRT]
[  334.912590] ACPI: PCI Root Bridge [VP05] (0000:08)
[  334.918326] PCI: Probing PCI hardware (bus 08)
[  334.926600] ACPI: PCI Interrupt Routing Table [\_SB_.VP05._PRT]
[  334.929213] ACPI: PCI Root Bridge [VP06] (0000:0a)
[  334.934929] PCI: Probing PCI hardware (bus 0a)
[  334.943195] ACPI: PCI Interrupt Routing Table [\_SB_.VP06._PRT]
[  334.945676] ACPI: PCI Root Bridge [VP07] (0000:0c)
[  334.951417] PCI: Probing PCI hardware (bus 0c)
[  334.959780] ACPI: PCI Interrupt Routing Table [\_SB_.VP07._PRT]
[  334.962958] ACPI: PCI Root Bridge [VP10] (0000:0e)
[  334.968707] PCI: Probing PCI hardware (bus 0e)
[  334.977583] ACPI: PCI Interrupt Routing Table [\_SB_.VP10._PRT]
[  334.979453] ACPI: PCI Root Bridge [VP11] (0000:0f)
[  334.985178] PCI: Probing PCI hardware (bus 0f)
[  334.993900] ACPI: PCI Interrupt Routing Table [\_SB_.VP11._PRT]
[  334.995202] ACPI: PCI Root Bridge [VP12] (0000:10)
[  335.000942] PCI: Probing PCI hardware (bus 10)
[  335.009184] ACPI: PCI Interrupt Routing Table [\_SB_.VP12._PRT]
[  335.011565] ACPI: PCI Root Bridge [VP13] (0000:12)
[  335.017305] PCI: Probing PCI hardware (bus 12)
[  335.025550] ACPI: PCI Interrupt Routing Table [\_SB_.VP13._PRT]
[  335.028008] ACPI: PCI Root Bridge [VP14] (0000:14)
[  335.033746] PCI: Probing PCI hardware (bus 14)
[  335.042015] ACPI: PCI Interrupt Routing Table [\_SB_.VP14._PRT]
[  335.044684] ACPI: PCI Root Bridge [VP15] (0000:16)
[  335.050424] PCI: Probing PCI hardware (bus 16)
[  335.058724] ACPI: PCI Interrupt Routing Table [\_SB_.VP15._PRT]
[  335.061213] ACPI: PCI Root Bridge [VP16] (0000:18)
[  335.066947] PCI: Probing PCI hardware (bus 18)
[  335.075231] ACPI: PCI Interrupt Routing Table [\_SB_.VP16._PRT]
[  335.077637] ACPI: PCI Root Bridge [VP17] (0000:1a)
[  335.083369] PCI: Probing PCI hardware (bus 1a)
[  335.092760] ACPI: PCI Interrupt Routing Table [\_SB_.VP17._PRT]
[  335.096962] SCSI subsystem initialized
[  335.101472] PCI: Using ACPI for IRQ routing
[  335.106488] PCI: If a device doesn't work, try "pci=routeirq".  If it helps, post a report
[  335.116783] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[  335.131226] PCI: Bridge: 0000:1a:01.0
[  335.135613]   IO window: f000-ffff
[  335.139707]   MEM window: f8600000-f86fffff
[  335.144732]   PREFETCH window: disabled.
[  335.154788] IA-32 Microcode Update Driver: v1.14 <tigran@veritas.com>
[  335.174698] IA32 emulation $Id: sys_ia32.c,v 1.32 2002/03/24 13:02:28 ak Exp $
[  335.192707] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[  335.200742] SGI XFS with large block/inode numbers, no debug enabled
[  335.210377] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[  335.383127] Real Time Clock Driver v1.12
[  335.387870] Linux agpgart interface v0.101 (c) Dave Jones
[  335.396008] serio: i8042 AUX port at 0x60,0x64 irq 12
[  335.402168] serio: i8042 KBD port at 0x60,0x64 irq 1
[  335.408211] Serial: 8250/16550 driver $Revision: 1.90 $ 4 ports, IRQ sharing disabled
[  335.417966] ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
[  335.424185] ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
[  335.431595] io scheduler noop registered
[  335.436386] io scheduler anticipatory registered
[  335.441989] io scheduler deadline registered
[  335.447246] io scheduler cfq registered
[  335.455801] RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
[  335.467104] loop: loaded (max 8 devices)
[  335.471840] Intel(R) PRO/1000 Network Driver - version 6.0.60-k2-NAPI
[  335.479602] Copyright (c) 1999-2005 Intel Corporation.
[  335.485973] ACPI: PCI Interrupt 0000:1b:04.0[A] -> GSI 124 (level, low) -> IRQ 169
[  335.837702] e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection
[  335.845989] ACPI: PCI Interrupt 0000:1b:04.1[B] -> GSI 128 (level, low) -> IRQ 177
[  336.196842] e1000: eth1: e1000_probe: Intel(R) PRO/1000 Network Connection
[  336.205097] ACPI: PCI Interrupt 0000:1b:06.0[A] -> GSI 132 (level, low) -> IRQ 185
[  336.555498] e1000: eth2: e1000_probe: Intel(R) PRO/1000 Network Connection
[  336.563729] ACPI: PCI Interrupt 0000:1b:06.1[B] -> GSI 136 (level, low) -> IRQ 193
[  336.914373] e1000: eth3: e1000_probe: Intel(R) PRO/1000 Network Connection
[  336.922749] tg3.c:v3.42 (Oct 3, 2005)
[  336.927149] ACPI: PCI Interrupt 0000:01:01.0[A] -> GSI 24 (level, low) -> IRQ 201
[  336.965752] eth4: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:7e
[  336.979682] eth4: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[0] 
[  336.989691] eth4: dma_rwctrl[769f0000]
[  336.994219] ACPI: PCI Interrupt 0000:01:01.1[B] -> GSI 28 (level, low) -> IRQ 209
[  337.033389] eth5: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:7f
[  337.047249] eth5: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  337.057270] eth5: dma_rwctrl[769f0000]
[  337.061834] ACPI: PCI Interrupt 0000:0f:01.0[A] -> GSI 94 (level, low) -> IRQ 217
[  337.094316] eth6: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:78
[  337.108200] eth6: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[1] Split[0] WireSpeed[1] TSOcap[0] 
[  337.118220] eth6: dma_rwctrl[769f0000]
[  337.122760] ACPI: PCI Interrupt 0000:0f:01.1[B] -> GSI 98 (level, low) -> IRQ 225
[  337.162010] eth7: Tigon3 [partno(BCM95704A6) rev 2100 PHY(5704)] (PCIX:66MHz:64-bit) 10/100/1000BaseT Ethernet 00:0d:60:98:3d:79
[  337.175900] eth7: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[1] 
[  337.185927] eth7: dma_rwctrl[769f0000]
[  337.190609] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
[  337.198195] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
[  337.209240] libata version 1.12 loaded.
[  337.210284] mice: PS/2 mouse device common for all mice
[  337.216669] oprofile: using NMI interrupt.
[  337.221799] NET: Registered protocol family 2
[  337.254300] input: AT Translated Set 2 keyboard on isa0060/serio0
[  337.343559] IP route cache hash table entries: 524288 (order: 10, 4194304 bytes)
[  337.356297] TCP established hash table entries: 262144 (order: 10, 4194304 bytes)
[  337.369703] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[  337.378656] TCP: Hash tables configured (established 262144 bind 65536)
[  337.386587] TCP reno registered
[  337.390511] TCP bic registered
[  337.394230] NET: Registered protocol family 1
[  337.399462] NET: Registered protocol family 17
[  337.405423] Freeing unused kernel memory: 208k freed
[  337.496659] Loading AIC-94xx Linux SAS/SATA Family Driver, Rev: 1.0.7-3
[  337.496663] 
[  337.506547] Probing Adaptec AIC-94xx Controller(s)...
[  337.512641] ACPI: PCI Interrupt 0000:01:02.0[A] -> GSI 25 (level, low) -> IRQ 233
[  337.521196] input: ImExPS/2 Generic Explorer Mouse on isa0060/serio1
[  337.659493] Probing Adaptec AIC-94xx Controller(s)...
[  337.665600] ACPI: PCI Interrupt 0000:0f:02.0[A] -> GSI 95 (level, low) -> IRQ 50
[  337.819855] scsi0 : Adaptec AIC-9410 SAS/SATA Host Adapter
[  338.035597]   Vendor: IBM-ESXS  Model: ST973401SS    F   Rev: B515
[  338.043272]   Type:   Direct-Access                      ANSI SCSI revision: 04
[  338.052045] adp94xx:1:128:0: Tagged Queuing enabled.  Depth 32
[  338.059835] SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
[  338.069220] SCSI device sda: drive cache: write through
[  338.076194] SCSI device sda: 143374000 512-byte hdwr sectors (73407 MB)
[  338.085563] SCSI device sda: drive cache: write through
[  338.091839]  sda: sda1 sda2 sda3
[  338.104332] Attached scsi disk sda at scsi0, channel 1, id 128, lun 0
[  338.112206] Attached scsi generic sg0 at scsi0, channel 1, id 128, lun 0,  type 0
[  338.179425] scsi1 : Adaptec AIC-9410 SAS/SATA Host Adapter
[  338.387094]   Vendor: IBM-ESXS  Model: ST973401SS    F   Rev: B515
[  338.394754]   Type:   Direct-Access                      ANSI SCSI revision: 04
[  338.403536] adp94xx:0:128:0: Tagged Queuing enabled.  Depth 32
[  338.411259] SCSI device sdb: 143374000 512-byte hdwr sectors (73407 MB)
[  338.420596] SCSI device sdb: drive cache: write through
[  338.427596] SCSI device sdb: 143374000 512-byte hdwr sectors (73407 MB)
[  338.436941] SCSI device sdb: drive cache: write through
[  338.443194]  sdb: sdb1 sdb2 sdb3 < sdb5 >
[  338.465813] Attached scsi disk sdb at scsi1, channel 0, id 128, lun 0
[  338.473691] Attached scsi generic sg1 at scsi1, channel 0, id 128, lun 0,  type 0
[  338.546790] AIC-94xx controller(s) attached = 2.
[  338.546792] 
[  338.671790] kjournald starting.  Commit interval 5 seconds
[  338.671771] EXT3-fs: mounted filesystem with ordered data mode.
[  390.457834] EXT3 FS on sda2, internal journal
[  390.721768] kjournald starting.  Commit interval 5 seconds
[  390.730787] EXT3 FS on sda1, internal journal
[  390.730796] EXT3-fs: mounted filesystem with ordered data mode.
[  390.745000] kjournald starting.  Commit interval 5 seconds
[  390.755673] EXT3 FS on sdb2, internal journal
[  390.755681] EXT3-fs: mounted filesystem with ordered data mode.
[  390.766741] XFS mounting filesystem sdb5
[  390.856620] Ending clean XFS mount for filesystem: sdb5

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Tuesday 18 October 2005 02:16, Ravikiran G Thirumalai wrote:
> On Mon, Oct 17, 2005 at 02:11:20PM -0700, Linus Torvalds wrote:
> > On Mon, 17 Oct 2005, Andrew Morton wrote:
> > > There seem to be a lot of proposed solutions floating about and I fear
> > > that different people will try to fix this in different ways.  Do we
> > > all agree that this patch is the correct solution to this problem, or
> > > is something more needed?
> >
> > I think this will fix it.
>
> I just tried Yasunori-sans patch on our x460.  It doesn't fix the problem.

That's surprising. Can you post the full boot log? The nodes should be really 
in order. Maybe we need to sort the SRAT first.

-Andi

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 10:23:51AM +0200, Andi Kleen wrote:
> On Tuesday 18 October 2005 02:16, Ravikiran G Thirumalai wrote:
> > On Mon, Oct 17, 2005 at 02:11:20PM -0700, Linus Torvalds wrote:
> > > On Mon, 17 Oct 2005, Andrew Morton wrote:
> > > > There seem to be a lot of proposed solutions floating about and I fear
> > > > that different people will try to fix this in different ways.  Do we
> > > > all agree that this patch is the correct solution to this problem, or
> > > > is something more needed?
> > >
> > > I think this will fix it.
> >
> > I just tried Yasunori-sans patch on our x460.  It doesn't fix the problem.
> 
> That's surprising. Can you post the full boot log? 

I'd already posted that... Here is the link
http://marc.theaimsgroup.com/?l=linux-kernel&m=112959469914681&w=2

> The nodes should be really in order. Maybe we need to sort the SRAT first.

I don't understand this comment.  I thought Yasunori-san's patch did not
require pgdats to be in any order..  Anywayz, the x460 nodes _are_ in proper
order.

Thanks,
Kiran

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Mon, 17 Oct 2005, Ravikiran G Thirumalai wrote:
>
> I just tried Yasunori-sans patch on our x460.  It doesn't fix the problem.  
> Attaching the dmesg capture. However, the patch to iterate over nodes and 
> allocate bootmem works, and should work for ia64s, boxes with insufficient 
> memory on node 0, nodes with just cpus etc.  I have requested Alex to try this 
> on the superdome (although the the use of swiotlb on superdomes seem to be
> optional).  If this works on superdomes, then please apply.  This has been
> tested on x460.

This version looks cleaner and smaller, and leaves the rest of the bootmem 
code alone. 

I vote for this one, assuming everybody who can test is happy.

		Linus

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Tue, 18 Oct 2005, Linus Torvalds wrote:
> 
> I vote for this one, assuming everybody who can test is happy.

Of course, just after sending the patch I noticed that there was a new 
version, even simpler. Can people test that one?

		Linus

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 08:50:18AM -0700, Linus Torvalds wrote:
> 
> 
> On Tue, 18 Oct 2005, Linus Torvalds wrote:
> > 
> > I vote for this one, assuming everybody who can test is happy.
> 
> Of course, just after sending the patch I noticed that there was a new 
> version, even simpler. Can people test that one?
> 

This version should work for everyone.  It falls back to the old 2.6.13
behaviour when it does not find suitable memory from any of the nodes.

Yasunori-san, Alex, can you confirm.  (Please use stock 2.6.14)

Thanks,
Kiran

--

Patch to ensure low32 mem allocation for x86_64 swiotlb

Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-17 22:48:25.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-18 12:44:17.000000000 -0700
@@ -106,6 +106,8 @@
 __setup("swiotlb=", setup_io_tlb_npages);
 /* make io_tlb_overflow tunable too? */
 
+#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
+
 /*
  * Statically reserve bounce buffer space and initialize bounce buffer data
  * structures for the software IO TLB used to implement the PCI DMA API.
@@ -114,17 +116,43 @@
 swiotlb_init_with_default_size (size_t default_size)
 {
 	unsigned long i;
+	unsigned long iotlbsz;
+	int node;
 
 	if (!io_tlb_nslabs) {
 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
 	}
 
+	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
+
 	/*
-	 * Get IO TLB memory from the low pages
+	 * Get IO TLB memory from the 0-4G range
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	
+	for_each_online_node(node) {
+		io_tlb_start = alloc_bootmem_node(NODE_DATA(node), iotlbsz);
+		if (io_tlb_start) {
+			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
+				break;
+			free_bootmem_node(NODE_DATA(node), 
+					  virt_to_phys(io_tlb_start), iotlbsz);
+			io_tlb_start = NULL;
+		}
+	}
+
+	/* 
+	 * FIXME: This should go away when the bootmem allocator is fixed to
+	 * guarantee lowmem32 allocations somehow, and the swiotlb mess is 
+	 * cleaned.  The alloc_bootmem_low_pages fall back is to ensure 
+         * boxes like amd64  which donot use swiotlb but still have 
+	 * swiotlb compiled in, falls back to the 2.6.13 behaviour instead
+	 * of panicking, when proper low32 pages are not available
+	 */
+	if (!io_tlb_start)
+		io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
+							(1 << IO_TLB_SHIFT));
+
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Tue, 2005-10-18 at 12:54 -0700, Ravikiran G Thirumalai wrote:
> On Tue, Oct 18, 2005 at 08:50:18AM -0700, Linus Torvalds wrote:
> > 
> > 
> > On Tue, 18 Oct 2005, Linus Torvalds wrote:
> > > 
> > > I vote for this one, assuming everybody who can test is happy.
> > 
> > Of course, just after sending the patch I noticed that there was a new 
> > version, even simpler. Can people test that one?
> > 
> 
> This version should work for everyone.  It falls back to the old 2.6.13
> behaviour when it does not find suitable memory from any of the nodes.
> 
> Yasunori-san, Alex, can you confirm.  (Please use stock 2.6.14)

   Oops, I used 2.6.14-rc4-mm1, I'll retest.  However, this does work on
the Superdome.  Not because of the iterating over the nodes code, but
because of the call to alloc_bootmem_low_pages() fallback case.  Adding
a printk(), I get this:

Node 0: 0xe000074104e6b200
Node 1: 0xe000082080723000
Node 2: 0xe000000101532000  *Note this is the interleaved memory node
Placing software IO TLB between 0x4cdc000 - 0x8cdc000

Looking at the memory map of the system, I see these major ranges:

Node 2:
0x00000000000 - 0x0007ffdefff (~2GB)
0x00100000000 - 0x0017fffffff (2GB)
0x04080000000 - 0x040f0000000 (2GB)
Node 0:
0x74100000000 - 0x741fbbfffff (~4GB)
Node 1:
0x82080000000 - 0x820fb453fff (~2GB)

So, it looks like we're iterating over the nodes, but
alloc_bootmem_node() isn't even guaranteed to try to get memory from the
low memory on that node.  Thanks,

	Alex

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 03:28:27PM -0600, Alex Williamson wrote:
> ... 
>    Oops, I used 2.6.14-rc4-mm1, I'll retest.  However, this does work on
> the Superdome.  Not because of the iterating over the nodes code, but
> because of the call to alloc_bootmem_low_pages() fallback case.  Adding
> a printk(), I get this:
> 
> Node 0: 0xe000074104e6b200
> Node 1: 0xe000082080723000
> Node 2: 0xe000000101532000  *Note this is the interleaved memory node
> Placing software IO TLB between 0x4cdc000 - 0x8cdc000
> 
> Looking at the memory map of the system, I see these major ranges:
> 
> Node 2:
> 0x00000000000 - 0x0007ffdefff (~2GB)
> 0x00100000000 - 0x0017fffffff (2GB)
> 0x04080000000 - 0x040f0000000 (2GB)
> Node 0:
> 0x74100000000 - 0x741fbbfffff (~4GB)
> Node 1:
> 0x82080000000 - 0x820fb453fff (~2GB)
> 
> So, it looks like we're iterating over the nodes, but
> alloc_bootmem_node() isn't even guaranteed to try to get memory from the
> low memory on that node.

Thanks Alex. 2.6.14-rc4-mm1 already has the 
guarantee-dma-area-for-alloc_bootmem_low.patch by Yasunori-san.  So it is 
safer to confirm results on latest 2.6.14 stock.

Could it also be that Node 2 is offline when swiotlb is allocated?

Thanks,
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Tue, 2005-10-18 at 14:53 -0700, Ravikiran G Thirumalai wrote:
> On Tue, Oct 18, 2005 at 03:28:27PM -0600, Alex Williamson wrote:
> > So, it looks like we're iterating over the nodes, but
> > alloc_bootmem_node() isn't even guaranteed to try to get memory from the
> > low memory on that node.
> 
> Thanks Alex. 2.6.14-rc4-mm1 already has the 
> guarantee-dma-area-for-alloc_bootmem_low.patch by Yasunori-san.  So it is 
> safer to confirm results on latest 2.6.14 stock.

   Ok. I'll need to build a stock tree then.

> Could it also be that Node 2 is offline when swiotlb is allocated?

   Nope.  Note that Node2 is iterated in the for_each_online_node, my
printk is within the body of the loop.  Also, the allocation it did get
is still from Node2.  My understanding is that goal for
alloc_bootmem_node is MAX_DMA_ADDRESS.  On ia64, that defaults to 4GB.
So it makes sense that we're using the second region in the
discontiguous space of that node.  This solution therefore relies on one
of the nodes having less than 4GB of zero base memory.  That's a pretty
weak assumption.  Thanks,

	Alex

-- 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Tue, 2005-10-18 at 16:04 -0600, Alex Williamson wrote:
> On Tue, 2005-10-18 at 14:53 -0700, Ravikiran G Thirumalai wrote:
> > On Tue, Oct 18, 2005 at 03:28:27PM -0600, Alex Williamson wrote:
> > > So, it looks like we're iterating over the nodes, but
> > > alloc_bootmem_node() isn't even guaranteed to try to get memory from the
> > > low memory on that node.
> > 
> > Thanks Alex. 2.6.14-rc4-mm1 already has the 
> > guarantee-dma-area-for-alloc_bootmem_low.patch by Yasunori-san.  So it is 
> > safer to confirm results on latest 2.6.14 stock.
> 
>    Ok. I'll need to build a stock tree then.
> 

   Nope, it breaks with a current git-2.6.14.  Here's what my extra
printk says:

Node 0: 0xe000074104e67200
Node 1: 0xe000082080722000
Node 2: 0xe000000101532000
Placing software IO TLB between 0x74108e68000 - 0x7410ce68000

So same scenario as on -mm w/ the iterating across nodes, but now
alloc_bootmem_low_pages() doesn't catch us in the end.  This will fail
in any case where MAX_DMA_ADDRESS is 4GB and every node has memory
sufficient for the swiotlb size above 4GB.  Thanks,

	Alex

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 04:37:03PM -0600, Alex Williamson wrote:
> On Tue, 2005-10-18 at 16:04 -0600, Alex Williamson wrote:
> 
>    Nope, it breaks with a current git-2.6.14.  Here's what my extra
> printk says:
> 
> Node 0: 0xe000074104e67200
> Node 1: 0xe000082080722000
> Node 2: 0xe000000101532000
> Placing software IO TLB between 0x74108e68000 - 0x7410ce68000
> 

Hope the following works.   Using __alloc_bootmem_node now with a hard coded
goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
<fingers crossed>. 

--

Patch to ensure low32 mem allocation for x86_64 swiotlb

Signed-off-by: Ravikiran Thirumalai <kiran@scalex86.org>

Index: linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c
===================================================================
--- linux-2.6.14-rc4.orig/arch/ia64/lib/swiotlb.c	2005-10-18 14:14:12.000000000 -0700
+++ linux-2.6.14-rc4/arch/ia64/lib/swiotlb.c	2005-10-18 16:09:51.000000000 -0700
@@ -106,6 +106,8 @@
 __setup("swiotlb=", setup_io_tlb_npages);
 /* make io_tlb_overflow tunable too? */
 
+#define IS_LOWPAGES(paddr, size) ((paddr < 0xffffffff) && ((paddr+size) < 0xffffffff))
+
 /*
  * Statically reserve bounce buffer space and initialize bounce buffer data
  * structures for the software IO TLB used to implement the PCI DMA API.
@@ -114,17 +116,46 @@
 swiotlb_init_with_default_size (size_t default_size)
 {
 	unsigned long i;
+	unsigned long iotlbsz;
+	int node;
 
 	if (!io_tlb_nslabs) {
 		io_tlb_nslabs = (default_size >> IO_TLB_SHIFT);
 		io_tlb_nslabs = ALIGN(io_tlb_nslabs, IO_TLB_SEGSIZE);
 	}
 
+	iotlbsz = io_tlb_nslabs * (1 << IO_TLB_SHIFT);	
+
 	/*
-	 * Get IO TLB memory from the low pages
+	 * Get IO TLB memory from the 0-4G range
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	
+	for_each_online_node(node) {
+		/* Ugly, hate it.  To be gone post 2.6.14 */
+		io_tlb_start = __alloc_bootmem_node(NODE_DATA(node), 
+						    iotlbsz, PAGE_SIZE, 
+						    0x1000000);
+		if (io_tlb_start) {
+			if (IS_LOWPAGES(virt_to_phys(io_tlb_start), iotlbsz))
+				break;
+			free_bootmem_node(NODE_DATA(node), 
+					  virt_to_phys(io_tlb_start), iotlbsz);
+			io_tlb_start = NULL;
+		}
+	}
+
+	/* 
+	 * FIXME: This should go away when the bootmem allocator is fixed to
+	 * guarantee lowmem32 allocations somehow, and the swiotlb mess is 
+	 * cleaned.  The alloc_bootmem_low_pages fall back is to ensure 
+         * boxes like amd64  which donot use swiotlb but still have 
+	 * swiotlb compiled in, falls back to the 2.6.13 behaviour instead
+	 * of panicking, when proper low32 pages are not available
+	 */
+	if (!io_tlb_start)
+		io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
+							(1 << IO_TLB_SHIFT));
+
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Tue, 2005-10-18 at 16:22 -0700, Ravikiran G Thirumalai wrote:

> Hope the following works.   Using __alloc_bootmem_node now with a hard coded
> goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
> <fingers crossed>. 

   Nope, it still gives me memory above 4GB.  If I change goal to 0x0 it
works.  One nit, shouldn't IS_LOW_AGES() be more like this:

#define IS_LOWPAGES(paddr, size) ((paddr+size-1) < 0x100000000UL)

Minor optimization not checking the start, but a 64MB swiotlb starting
at 4GB-64MB should be found as ok.  Thanks,

	Alex

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Tue, 2005-10-18 at 19:22 -0600, Alex Williamson wrote:
> On Tue, 2005-10-18 at 16:22 -0700, Ravikiran G Thirumalai wrote:
> 
> > Hope the following works.   Using __alloc_bootmem_node now with a hard coded
> > goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
> > <fingers crossed>. 
> 
>    Nope, it still gives me memory above 4GB.  If I change goal to 0x0 it
> works.

   BTW, the reason 16MB fails is this test in __alloc_bootmem_core():

     if (bdata->last_success >= preferred)
            preferred = bdata->last_success;

That pretty much negates most of usefulness of passing in a goal other
than 0 or !0.  If I comment out this test, the 16MB goal works as
expected, but I get an uninitialized timer in the sym53c8xx driver.  Not
sure what's happening there, but apparently the test has some purpose
other than optimization.  Thanks,

	Alex

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

> On Tue, 2005-10-18 at 16:22 -0700, Ravikiran G Thirumalai wrote:
> 
> > Hope the following works.   Using __alloc_bootmem_node now with a hard coded
> > goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
> > <fingers crossed>. 
> 
>    Nope, it still gives me memory above 4GB.  If I change goal to 0x0 it
> works. 
>  One nit, shouldn't IS_LOW_AGES() be more like this:
> 
> #define IS_LOWPAGES(paddr, size) ((paddr+size-1) < 0x100000000UL)
> 
> Minor optimization not checking the start, but a 64MB swiotlb starting
> at 4GB-64MB should be found as ok.  Thanks,
> 
> 	Alex
> 


Hmm.....
How is this patch? This is another way.

I think that true issue is there is no way for requester to
specify maxmum address at __alloc_bootmem_core().

"goal" is just to keep space lower address as much as possible.
and __alloc_bootmem_core() doesn't care about max address for requester.
I suppose it is a bit strange. The swiotlb's case is good example
by it.

So, I made a patch that __alloc_bootmem_core() cares it.

Thanks.


Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
-------

Index: 3rdbootmem/arch/ia64/lib/swiotlb.c
===================================================================
--- 3rdbootmem.orig/arch/ia64/lib/swiotlb.c	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/arch/ia64/lib/swiotlb.c	2005-10-19 21:05:18.000000000 +0900
@@ -123,8 +123,8 @@ swiotlb_init_with_default_size (size_t d
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	io_tlb_start = alloc_bootmem_low_pages_limit(io_tlb_nslabs *
+					     (1 << IO_TLB_SHIFT), 0x100000000);
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);
Index: 3rdbootmem/include/linux/bootmem.h
===================================================================
--- 3rdbootmem.orig/include/linux/bootmem.h	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/include/linux/bootmem.h	2005-10-19 21:09:21.000000000 +0900
@@ -43,7 +43,7 @@ typedef struct bootmem_data {
 extern unsigned long __init bootmem_bootmap_pages (unsigned long);
 extern unsigned long __init init_bootmem (unsigned long addr, unsigned long memend);
 extern void __init free_bootmem (unsigned long addr, unsigned long size);
-extern void * __init __alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal);
+extern void * __init __alloc_bootmem_limit (unsigned long size, unsigned long align, unsigned long goal, unsigned long limit);
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 extern void __init reserve_bootmem (unsigned long addr, unsigned long size);
 #define alloc_bootmem(x) \
@@ -54,6 +54,16 @@ extern void __init reserve_bootmem (unsi
 	__alloc_bootmem((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
 #define alloc_bootmem_low_pages(x) \
 	__alloc_bootmem((x), PAGE_SIZE, 0)
+
+#define alloc_bootmem_limit(x, limit)						\
+	__alloc_bootmem_limit((x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_limit(x, limit)			\
+	__alloc_bootmem_limit((x), SMP_CACHE_BYTES, 0, (limit))
+#define alloc_bootmem_pages_limit(x, limit)					\
+	__alloc_bootmem_limit((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_pages_limit(x, limit)		\
+	__alloc_bootmem_limit((x), PAGE_SIZE, 0, (limit))
+
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
 extern unsigned long __init free_all_bootmem (void);
 
@@ -61,7 +71,7 @@ extern unsigned long __init init_bootmem
 extern void __init reserve_bootmem_node (pg_data_t *pgdat, unsigned long physaddr, unsigned long size);
 extern void __init free_bootmem_node (pg_data_t *pgdat, unsigned long addr, unsigned long size);
 extern unsigned long __init free_all_bootmem_node (pg_data_t *pgdat);
-extern void * __init __alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal);
+extern void * __init __alloc_bootmem_node_limit (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal, unsigned long limit);
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 #define alloc_bootmem_node(pgdat, x) \
 	__alloc_bootmem_node((pgdat), (x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
@@ -69,6 +79,14 @@ extern void * __init __alloc_bootmem_nod
 	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
 #define alloc_bootmem_low_pages_node(pgdat, x) \
 	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, 0)
+
+#define alloc_bootmem_node_limit(pgdat, x, limit)				\
+	__alloc_bootmem_node_limit((pgdat), (x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_pages_node_limit(pgdat, x, limit)				\
+	__alloc_bootmem_node_limit((pgdat), (x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_pages_node_limit(pgdat, x, limit)		\
+	__alloc_bootmem_node_limit((pgdat), (x), PAGE_SIZE, 0, (limit))
+
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
 
 #ifdef CONFIG_HAVE_ARCH_ALLOC_REMAP
@@ -105,5 +123,15 @@ extern void *__init alloc_large_system_h
 #endif
 extern int __initdata hashdist;		/* Distribute hashes across NUMA nodes? */
 
+static inline void *__alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal)
+{
+	return __alloc_bootmem_limit(size, align, goal, 0);
+}
+
+static inline void *__alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align,
+				     unsigned long goal)
+{
+	return __alloc_bootmem_node_limit(pgdat, size, align, goal, 0);
+}
 
 #endif /* _LINUX_BOOTMEM_H */
Index: 3rdbootmem/mm/bootmem.c
===================================================================
--- 3rdbootmem.orig/mm/bootmem.c	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/mm/bootmem.c	2005-10-19 21:05:18.000000000 +0900
@@ -154,10 +154,10 @@ static void __init free_bootmem_core(boo
  */
 static void * __init
 __alloc_bootmem_core(struct bootmem_data *bdata, unsigned long size,
-		unsigned long align, unsigned long goal)
+	      unsigned long align, unsigned long goal, unsigned long limit)
 {
 	unsigned long offset, remaining_size, areasize, preferred;
-	unsigned long i, start = 0, incr, eidx;
+	unsigned long i, start = 0, incr, eidx, end_pfn = bdata->node_low_pfn;
 	void *ret;
 
 	if(!size) {
@@ -166,7 +166,14 @@ __alloc_bootmem_core(struct bootmem_data
 	}
 	BUG_ON(align & (align-1));
 
-	eidx = bdata->node_low_pfn - (bdata->node_boot_start >> PAGE_SHIFT);
+	if (limit && bdata->node_boot_start >= limit)
+		return NULL;
+
+        limit >>=PAGE_SHIFT;
+	if (limit && end_pfn > limit)
+		end_pfn = limit;
+
+	eidx = end_pfn - (bdata->node_boot_start >> PAGE_SHIFT);
 	offset = 0;
 	if (align &&
 	    (bdata->node_boot_start & (align - 1UL)) != 0)
@@ -178,11 +185,12 @@ __alloc_bootmem_core(struct bootmem_data
 	 * first, then we try to allocate lower pages.
 	 */
 	if (goal && (goal >= bdata->node_boot_start) && 
-	    ((goal >> PAGE_SHIFT) < bdata->node_low_pfn)) {
+	    ((goal >> PAGE_SHIFT) < end_pfn)) {
 		preferred = goal - bdata->node_boot_start;
 
 		if (bdata->last_success >= preferred)
-			preferred = bdata->last_success;
+			if (!limit || (limit && limit > bdata->last_success))
+				preferred = bdata->last_success;
 	} else
 		preferred = 0;
 
@@ -382,14 +390,15 @@ unsigned long __init free_all_bootmem (v
 	return(free_all_bootmem_core(NODE_DATA(0)));
 }
 
-void * __init __alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal)
+void * __init __alloc_bootmem_limit (unsigned long size, unsigned long align, unsigned long goal,
+				unsigned long limit)
 {
 	pg_data_t *pgdat = pgdat_list;
 	void *ptr;
 
 	for_each_pgdat(pgdat)
 		if ((ptr = __alloc_bootmem_core(pgdat->bdata, size,
-						align, goal)))
+						 align, goal, limit)))
 			return(ptr);
 
 	/*
@@ -400,14 +409,16 @@ void * __init __alloc_bootmem (unsigned 
 	return NULL;
 }
 
-void * __init __alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal)
+
+void * __init __alloc_bootmem_node_limit (pg_data_t *pgdat, unsigned long size, unsigned long align,
+				     unsigned long goal, unsigned long limit)
 {
 	void *ptr;
 
-	ptr = __alloc_bootmem_core(pgdat->bdata, size, align, goal);
+	ptr = __alloc_bootmem_core(pgdat->bdata, size, align, goal, limit);
 	if (ptr)
 		return (ptr);
 
-	return __alloc_bootmem(size, align, goal);
+	return __alloc_bootmem_limit(size, align, goal, limit);
 }
 


-- 
Yasunori Goto 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Alex Williamson]

On Wed, 2005-10-19 at 21:47 +0900, Yasunori Goto wrote:
> Hmm.....
> How is this patch? This is another way.
> 
> I think that true issue is there is no way for requester to
> specify maxmum address at __alloc_bootmem_core().
> 
> "goal" is just to keep space lower address as much as possible.
> and __alloc_bootmem_core() doesn't care about max address for requester.
> I suppose it is a bit strange. The swiotlb's case is good example
> by it.
> 
> So, I made a patch that __alloc_bootmem_core() cares it.

   This works on the Superdome.  The swiotlb is shown as:

Placing software IO TLB between 0x4d48000 - 0x8d48000

Thanks,

	Alex

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Wed, Oct 19, 2005 at 09:47:02PM +0900, Yasunori Goto wrote:
> > On Tue, 2005-10-18 at 16:22 -0700, Ravikiran G Thirumalai wrote:
> 
> 
> Hmm.....
> How is this patch? This is another way.
> 
> I think that true issue is there is no way for requester to
> specify maxmum address at __alloc_bootmem_core().
> 
> "goal" is just to keep space lower address as much as possible.
> and __alloc_bootmem_core() doesn't care about max address for requester.
> I suppose it is a bit strange. The swiotlb's case is good example
> by it.

This works for me too.  After this patch I see
[  400.495902] Placing software IO TLB between 0x722a000 - 0xb22a000
which means, the new patch is not digging into the 16MB x86_64 DMA area too. 

Linus would you apply this for 2.6.14?  This is the patch which works for
both x86_64 and ia64 boxes.  I limited my approaches not to touch the core
bootmem allocator for 2.6.14,  but that doesn't seem to work for ia64 boxes.  
Fixing the bootmem is the correct approach IMHO.  But in case you feel 
this is too intrusive for 2.6.14, we can whip up an ugly #ifdef CONFIG_X86_64 
patch which patches swiotlb.c only.

Thanks Yasunori-san for the patch, and Alex for testing out all the patches.

Thanks.
Kiran

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Linus Torvalds]

On Wed, 19 Oct 2005, Ravikiran G Thirumalai wrote:
> 
> Linus would you apply this for 2.6.14?  This is the patch which works for
> both x86_64 and ia64 boxes.

Ok, I'd love to have something that people finally agree on.

Can you re-post the final version as such, with explanations for the 
commit messages and the sign-off, and people who have issues with it 
_please_ speak up asap?

			Linus

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Wed, Oct 19, 2005 at 01:45:21PM -0700, Linus Torvalds wrote:
> ... 
> Can you re-post the final version as such, with explanations for the 
> commit messages and the sign-off, and people who have issues with it 
> _please_ speak up asap?

The final version which works for everyone is from Yasunori Goto. 

His patch introduces a limit parameter to the core bootmem allocator;  This
new parameter indicates that physical memory allocated by the bootmem allocator
should be within the requested limit.  The patch also introduces
alloc_bootmem_low_pages_limit(), alloc_bootmem_node_limit,
alloc_bootmem_low_pages_node_limit apis, but alloc_bootmem_low_pages_limit()
is the only api used for swiotlb.  IMO, instead of introducing xxx_limit apis,
the existing alloc_bootmem_low_pages() api could instead be changed and made 
to pass right limit to the core allocator.  But then that would make the 
patch more intrusive for 2.6.14, as other  arches use alloc_bootmem_low_pages().
(So maybe that can be done post 2.6.14 if Yasunori-san is OK with that)

With the patch, swiotlb gets memory within 4G for both x86_64 and ia64 arches.
Here's his post FYR

Thanks,
Kiran

--

From:	Yasunori Goto <y-goto@jp.fujitsu.com>
To:	Alex Williamson <alex.williamson@hp.com>,
	Ravikiran G Thirumalai <kiran@scalex86.org>
Subject: Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken
Cc:	Linus Torvalds <torvalds@osdl.org>, Andrew Morton <akpm@osdl.org>,
	Andi Kleen <ak@suse.de>, linux-kernel@vger.kernel.org,
	discuss@x86-64.org, tglx@linutronix.de, shai@scalex86.org

> On Tue, 2005-10-18 at 16:22 -0700, Ravikiran G Thirumalai wrote:
> 
> > Hope the following works.   Using __alloc_bootmem_node now with a hard coded
> > goal to avoid 16MB DMA zone.  It is ugly :( and hope it works this time
> > <fingers crossed>. 
> 
>    Nope, it still gives me memory above 4GB.  If I change goal to 0x0 it
> works. 
>  One nit, shouldn't IS_LOW_AGES() be more like this:
> 
> #define IS_LOWPAGES(paddr, size) ((paddr+size-1) < 0x100000000UL)
> 
> Minor optimization not checking the start, but a 64MB swiotlb starting
> at 4GB-64MB should be found as ok.  Thanks,
> 
> 	Alex
> 


Hmm.....
How is this patch? This is another way.

I think that true issue is there is no way for requester to
specify maxmum address at __alloc_bootmem_core().

"goal" is just to keep space lower address as much as possible.
and __alloc_bootmem_core() doesn't care about max address for requester.
I suppose it is a bit strange. The swiotlb's case is good example
by it.

So, I made a patch that __alloc_bootmem_core() cares it.

Thanks.


Signed-off-by: Yasunori Goto <y-goto@jp.fujitsu.com>
-------

Index: 3rdbootmem/arch/ia64/lib/swiotlb.c
===================================================================
--- 3rdbootmem.orig/arch/ia64/lib/swiotlb.c	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/arch/ia64/lib/swiotlb.c	2005-10-19 21:05:18.000000000 +0900
@@ -123,8 +123,8 @@ swiotlb_init_with_default_size (size_t d
 	/*
 	 * Get IO TLB memory from the low pages
 	 */
-	io_tlb_start = alloc_bootmem_low_pages(io_tlb_nslabs *
-					       (1 << IO_TLB_SHIFT));
+	io_tlb_start = alloc_bootmem_low_pages_limit(io_tlb_nslabs *
+					     (1 << IO_TLB_SHIFT), 0x100000000);
 	if (!io_tlb_start)
 		panic("Cannot allocate SWIOTLB buffer");
 	io_tlb_end = io_tlb_start + io_tlb_nslabs * (1 << IO_TLB_SHIFT);
Index: 3rdbootmem/include/linux/bootmem.h
===================================================================
--- 3rdbootmem.orig/include/linux/bootmem.h	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/include/linux/bootmem.h	2005-10-19 21:09:21.000000000 +0900
@@ -43,7 +43,7 @@ typedef struct bootmem_data {
 extern unsigned long __init bootmem_bootmap_pages (unsigned long);
 extern unsigned long __init init_bootmem (unsigned long addr, unsigned long memend);
 extern void __init free_bootmem (unsigned long addr, unsigned long size);
-extern void * __init __alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal);
+extern void * __init __alloc_bootmem_limit (unsigned long size, unsigned long align, unsigned long goal, unsigned long limit);
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 extern void __init reserve_bootmem (unsigned long addr, unsigned long size);
 #define alloc_bootmem(x) \
@@ -54,6 +54,16 @@ extern void __init reserve_bootmem (unsi
 	__alloc_bootmem((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
 #define alloc_bootmem_low_pages(x) \
 	__alloc_bootmem((x), PAGE_SIZE, 0)
+
+#define alloc_bootmem_limit(x, limit)						\
+	__alloc_bootmem_limit((x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_limit(x, limit)			\
+	__alloc_bootmem_limit((x), SMP_CACHE_BYTES, 0, (limit))
+#define alloc_bootmem_pages_limit(x, limit)					\
+	__alloc_bootmem_limit((x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_pages_limit(x, limit)		\
+	__alloc_bootmem_limit((x), PAGE_SIZE, 0, (limit))
+
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
 extern unsigned long __init free_all_bootmem (void);
 
@@ -61,7 +71,7 @@ extern unsigned long __init init_bootmem
 extern void __init reserve_bootmem_node (pg_data_t *pgdat, unsigned long physaddr, unsigned long size);
 extern void __init free_bootmem_node (pg_data_t *pgdat, unsigned long addr, unsigned long size);
 extern unsigned long __init free_all_bootmem_node (pg_data_t *pgdat);
-extern void * __init __alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal);
+extern void * __init __alloc_bootmem_node_limit (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal, unsigned long limit);
 #ifndef CONFIG_HAVE_ARCH_BOOTMEM_NODE
 #define alloc_bootmem_node(pgdat, x) \
 	__alloc_bootmem_node((pgdat), (x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS))
@@ -69,6 +79,14 @@ extern void * __init __alloc_bootmem_nod
 	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS))
 #define alloc_bootmem_low_pages_node(pgdat, x) \
 	__alloc_bootmem_node((pgdat), (x), PAGE_SIZE, 0)
+
+#define alloc_bootmem_node_limit(pgdat, x, limit)				\
+	__alloc_bootmem_node_limit((pgdat), (x), SMP_CACHE_BYTES, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_pages_node_limit(pgdat, x, limit)				\
+	__alloc_bootmem_node_limit((pgdat), (x), PAGE_SIZE, __pa(MAX_DMA_ADDRESS), (limit))
+#define alloc_bootmem_low_pages_node_limit(pgdat, x, limit)		\
+	__alloc_bootmem_node_limit((pgdat), (x), PAGE_SIZE, 0, (limit))
+
 #endif /* !CONFIG_HAVE_ARCH_BOOTMEM_NODE */
 
 #ifdef CONFIG_HAVE_ARCH_ALLOC_REMAP
@@ -105,5 +123,15 @@ extern void *__init alloc_large_system_h
 #endif
 extern int __initdata hashdist;		/* Distribute hashes across NUMA nodes? */
 
+static inline void *__alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal)
+{
+	return __alloc_bootmem_limit(size, align, goal, 0);
+}
+
+static inline void *__alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align,
+				     unsigned long goal)
+{
+	return __alloc_bootmem_node_limit(pgdat, size, align, goal, 0);
+}
 
 #endif /* _LINUX_BOOTMEM_H */
Index: 3rdbootmem/mm/bootmem.c
===================================================================
--- 3rdbootmem.orig/mm/bootmem.c	2005-10-19 21:05:15.000000000 +0900
+++ 3rdbootmem/mm/bootmem.c	2005-10-19 21:05:18.000000000 +0900
@@ -154,10 +154,10 @@ static void __init free_bootmem_core(boo
  */
 static void * __init
 __alloc_bootmem_core(struct bootmem_data *bdata, unsigned long size,
-		unsigned long align, unsigned long goal)
+	      unsigned long align, unsigned long goal, unsigned long limit)
 {
 	unsigned long offset, remaining_size, areasize, preferred;
-	unsigned long i, start = 0, incr, eidx;
+	unsigned long i, start = 0, incr, eidx, end_pfn = bdata->node_low_pfn;
 	void *ret;
 
 	if(!size) {
@@ -166,7 +166,14 @@ __alloc_bootmem_core(struct bootmem_data
 	}
 	BUG_ON(align & (align-1));
 
-	eidx = bdata->node_low_pfn - (bdata->node_boot_start >> PAGE_SHIFT);
+	if (limit && bdata->node_boot_start >= limit)
+		return NULL;
+
+        limit >>=PAGE_SHIFT;
+	if (limit && end_pfn > limit)
+		end_pfn = limit;
+
+	eidx = end_pfn - (bdata->node_boot_start >> PAGE_SHIFT);
 	offset = 0;
 	if (align &&
 	    (bdata->node_boot_start & (align - 1UL)) != 0)
@@ -178,11 +185,12 @@ __alloc_bootmem_core(struct bootmem_data
 	 * first, then we try to allocate lower pages.
 	 */
 	if (goal && (goal >= bdata->node_boot_start) && 
-	    ((goal >> PAGE_SHIFT) < bdata->node_low_pfn)) {
+	    ((goal >> PAGE_SHIFT) < end_pfn)) {
 		preferred = goal - bdata->node_boot_start;
 
 		if (bdata->last_success >= preferred)
-			preferred = bdata->last_success;
+			if (!limit || (limit && limit > bdata->last_success))
+				preferred = bdata->last_success;
 	} else
 		preferred = 0;
 
@@ -382,14 +390,15 @@ unsigned long __init free_all_bootmem (v
 	return(free_all_bootmem_core(NODE_DATA(0)));
 }
 
-void * __init __alloc_bootmem (unsigned long size, unsigned long align, unsigned long goal)
+void * __init __alloc_bootmem_limit (unsigned long size, unsigned long align, unsigned long goal,
+				unsigned long limit)
 {
 	pg_data_t *pgdat = pgdat_list;
 	void *ptr;
 
 	for_each_pgdat(pgdat)
 		if ((ptr = __alloc_bootmem_core(pgdat->bdata, size,
-						align, goal)))
+						 align, goal, limit)))
 			return(ptr);
 
 	/*
@@ -400,14 +409,16 @@ void * __init __alloc_bootmem (unsigned 
 	return NULL;
 }
 
-void * __init __alloc_bootmem_node (pg_data_t *pgdat, unsigned long size, unsigned long align, unsigned long goal)
+
+void * __init __alloc_bootmem_node_limit (pg_data_t *pgdat, unsigned long size, unsigned long align,
+				     unsigned long goal, unsigned long limit)
 {
 	void *ptr;
 
-	ptr = __alloc_bootmem_core(pgdat->bdata, size, align, goal);
+	ptr = __alloc_bootmem_core(pgdat->bdata, size, align, goal, limit);
 	if (ptr)
 		return (ptr);
 
-	return __alloc_bootmem(size, align, goal);
+	return __alloc_bootmem_limit(size, align, goal, limit);
 }

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Yasunori Goto]

Thanks a lot for your explanation!

> On Wed, Oct 19, 2005 at 01:45:21PM -0700, Linus Torvalds wrote:
> > ... 
> > Can you re-post the final version as such, with explanations for the 
> > commit messages and the sign-off, and people who have issues with it 
> > _please_ speak up asap?
> 
> The final version which works for everyone is from Yasunori Goto. 
> 
> His patch introduces a limit parameter to the core bootmem allocator;  This
> new parameter indicates that physical memory allocated by the bootmem allocator
> should be within the requested limit.  The patch also introduces
> alloc_bootmem_low_pages_limit(), alloc_bootmem_node_limit,
> alloc_bootmem_low_pages_node_limit apis, but alloc_bootmem_low_pages_limit()
> is the only api used for swiotlb.  IMO, instead of introducing xxx_limit apis,
> the existing alloc_bootmem_low_pages() api could instead be changed and made 
> to pass right limit to the core allocator.  But then that would make the 
> patch more intrusive for 2.6.14, as other  arches use alloc_bootmem_low_pages().

Yes. I worried a bit that I should replace all of the function or not.
But, its impact is too risky for 2.6.14. So, I wrote the patch to avoid
big impact as much as possible.

> (So maybe that can be done post 2.6.14 if Yasunori-san is OK with that)

Sure! 

-- 
Yasunori Goto 

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Andi Kleen]

On Wednesday 19 October 2005 14:47, Yasunori Goto wrote:

>
> Hmm.....
> How is this patch? This is another way.

That one looks good to me.

Thanks,
-Andi

Re: [discuss] Re: x86_64: 2.6.14-rc4 swiotlb broken

[Ravikiran G Thirumalai]

On Tue, Oct 18, 2005 at 04:04:00PM -0600, Alex Williamson wrote:
> On Tue, 2005-10-18 at 14:53 -0700, Ravikiran G Thirumalai wrote:
> > On Tue, Oct 18, 2005 at 03:28:27PM -0600, Alex Williamson wrote:
> > > So, it looks like we're iterating over the nodes, but
> > > alloc_bootmem_node() isn't even guaranteed to try to get memory from the
> > > low memory on that node.
> > 
> > Thanks Alex. 2.6.14-rc4-mm1 already has the 
> > guarantee-dma-area-for-alloc_bootmem_low.patch by Yasunori-san.  So it is 
> > safer to confirm results on latest 2.6.14 stock.
> 
>    Ok. I'll need to build a stock tree then.
> 
> > Could it also be that Node 2 is offline when swiotlb is allocated?
> 
>    Nope.  Note that Node2 is iterated in the for_each_online_node, my
> printk is within the body of the loop.  Also, the allocation it did get
> is still from Node2.  My understanding is that goal for
> alloc_bootmem_node is MAX_DMA_ADDRESS.  On ia64, that defaults to 4GB.

Ahhh... and it is 16MB on x86_64.  alloc_bootmem_node will never work here
for ia64 then, However, alloc_bootmem_low_pages_node will work, but then it will 
dig into 16MB DMA area of x86_64.... arrrrgh...

The first cleanup post 2.6.14 should be to seperate swiotlb for ia64 and
x86_64 IMHO. 

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Muli Ben-Yehuda]

On Mon, Oct 17, 2005 at 02:50:07AM -0700, Andrew Morton wrote:
> Ravikiran G Thirumalai <kiran@scalex86.org> wrote:
> >
> > On x86_64 NUMA boxes, the revert
> > http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6e3254c4e2927c117044a02acf5f5b56e1373053
> > meant that swiotlb gets the IOTLB
> > memory from pages over 4G (if mem > 4G), which basically renders swiotlb useless, causing
> > breakage with devices not capable of DMA beyond 4G.  2.6.13 was (kinda) not
> > broken, although the patch titled "Reverse order of bootmem lists" was
> > not in 2.6.13, The reason is commit
> > http://kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=6142891a0c0209c91aa4a98f725de0d6e2ed4918
> > was not in 2.6.13, PCI_DMA_BUS_IS_PHYS was 1 when no mmu was present, and the block layer did 
> > the bouncing, never using swiotlb.  I guess the right fix is to make sure
> > swiotlb gets the right memory.  Here is a patch doing that.  Tested on IBM
> > x460.  I hope the patch is ok for ia64s too.  I do not have access to ia64
> > boxen.
> > 
> 
> This is an ia64 patch - what point was there in testing it on an x460?
> 
> Is something missing here?

x86-64 uses swioltb as well, via arch/ia64 directly. John Linville has
a patch to move the swiotlb to lib/swiotlb.c that is waiting in an
IA64 for inclusion (post 2.6.14, I guess?)

Cheers,
Muli
-- 
Muli Ben-Yehuda
http://www.mulix.org | http://mulix.livejournal.com/

Re: x86_64: 2.6.14-rc4 swiotlb broken

[Tony Luck]

> x86-64 uses swioltb as well, via arch/ia64 directly. John Linville has
> a patch to move the swiotlb to lib/swiotlb.c that is waiting in an
> IA64 for inclusion (post 2.6.14, I guess?)

Yes.  John's patch is sitting in a "swiotlb" branch of my GIT tree.  I'd like to
hear some positive confirmation from Linus and/or Andrew that moving this
up to lib/ is ok ... I don't want to be accused of shovelling tasteless code up
into the base.

This is definitely a post 2.6.14 move.

-Tony