Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Srihari Vijayaraghavan]

[Sorry to reply to my own email thread]

Srihari Vijayaraghavan <sriharivijayaraghavan@yahoo.com.au> wrote:
...
> No problems. Here's the log of unworking kernel with IOMMU turned on.
> Basically it goes on reseting the SATA ports throwing many errors (none are
> present in 2.6.23 or on 2.6.24-rc with mem=3500M) for many minutes at which
> point I do a power reset :-(.
> 
> Also the log of the working kernel with IOMMU but with mem=3500M is also
> attached for the record. It's basically the same above kernel just with the
> added parameter.

Gentlemen,

This changeset has introduced a regression in 2.6.24-rc, such that my machine
boots no more:
http://www.kernel.org/hg/linux-2.6/rev/ddf8804136fb
changeset:   72064:ddf8804136fb
user:        Andi Kleen <ak@suse.de>
date:        Fri Oct 19 20:35:03 2007 +0200
files:       arch/x86/kernel/pci-dma_64.c
description:
x86: enable iommu_merge by default

[ tglx: arch/x86 adaptation ]

Signed-off-by: Andi Kleen <ak@suse.de>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

committer: Thomas Gleixner <tglx@linutronix.de>


diff -r 8c8683cbdc05 -r ddf8804136fb arch/x86/kernel/pci-dma_64.c
--- a/arch/x86/kernel/pci-dma_64.c      Fri Oct 19 20:35:03 2007 +0200
+++ b/arch/x86/kernel/pci-dma_64.c      Fri Oct 19 20:35:03 2007 +0200
@@ -11,7 +11,7 @@
 #include <asm/iommu.h>
 #include <asm/calgary.h>

-int iommu_merge __read_mostly = 0;
+int iommu_merge __read_mostly = 1;
 EXPORT_SYMBOL(iommu_merge);

 dma_addr_t bad_dma_address __read_mostly;

As a work-around, I can get it to boot with mem=3500M, but then it's ugly ;-)
& I lose some valuable memory I have.

Here's my email thread on linux-ide capturing the good & bad kernel behaviour
for reference:
http://marc.info/?t=119456213200005&r=1&w=2

Thanks

Hari

PS: Here's hoping for a kernel mem= parameter free bootable 2.6.24 ;-).



      Feel safe with award winning spam protection on Yahoo!7 Mail.  
www.yahoo.com.au/mail 

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Andi Kleen]

On Wednesday 14 November 2007 12:55, Srihari Vijayaraghavan wrote:
> [Sorry to reply to my own email thread]
>
> Srihari Vijayaraghavan <sriharivijayaraghavan@yahoo.com.au> wrote:
> ...
>
> > No problems. Here's the log of unworking kernel with IOMMU turned on.
> > Basically it goes on reseting the SATA ports throwing many errors (none
> > are present in 2.6.23 or on 2.6.24-rc with mem=3500M) for many minutes at
> > which point I do a power reset :-(.
> >
> > Also the log of the working kernel with IOMMU but with mem=3500M is also
> > attached for the record. It's basically the same above kernel just with
> > the added parameter.
>
> Gentlemen,
>
> This changeset has introduced a regression in 2.6.24-rc, such that my
> machine boots no more:

Hmm, you got an AHCI controller that does not do 64bit DMA masks?
Or do you have CONFIG_IOMMU_DEBUG enabled? 

Anyways, not being able to deal with merged SG lists must be some
driver or hardware bug. I would stick some printks into gart_map_sg()
and try to find out where the failing DMA is initiatiated and then
split it into multiple IO submissions at the caller level.

-Andi

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Thomas Gleixner]

On Wed, 14 Nov 2007, Andi Kleen wrote:
> On Wednesday 14 November 2007 12:55, Srihari Vijayaraghavan wrote:
> > [Sorry to reply to my own email thread]
> >
> > Srihari Vijayaraghavan <sriharivijayaraghavan@yahoo.com.au> wrote:
> > ...
> >
> > > No problems. Here's the log of unworking kernel with IOMMU turned on.
> > > Basically it goes on reseting the SATA ports throwing many errors (none
> > > are present in 2.6.23 or on 2.6.24-rc with mem=3500M) for many minutes at
> > > which point I do a power reset :-(.
> > >
> > > Also the log of the working kernel with IOMMU but with mem=3500M is also
> > > attached for the record. It's basically the same above kernel just with
> > > the added parameter.
> >
> > Gentlemen,
> >
> > This changeset has introduced a regression in 2.6.24-rc, such that my
> > machine boots no more:
> 
> Hmm, you got an AHCI controller that does not do 64bit DMA masks?
> Or do you have CONFIG_IOMMU_DEBUG enabled? 
> 
> Anyways, not being able to deal with merged SG lists must be some
> driver or hardware bug. I would stick some printks into gart_map_sg()
> and try to find out where the failing DMA is initiatiated and then
> split it into multiple IO submissions at the caller level.

64bit DMA on SB600 was disabled in May/07 due to a chip bug:
http://www.mail-archive.com/linux-ide@vger.kernel.org/msg06694.html

The AHCI code falls back to 32bit DMA in that case. Which in turn
causes the problem seen by Srihari. There is not much printk sticking
necessary, the code is simply not handling this. So the main option
right now seems to revert the iommu_merge patch.

Thanks,

	tglx

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Andi Kleen]

 
> The AHCI code falls back to 32bit DMA in that case. Which in turn
> causes the problem seen by Srihari. There is not much printk sticking
> necessary, the code is simply not handling this. 

What code is not handling what? 

IOMMU merging should be always safe. If it is not the driver should
not submit things in a single SG list.

> So the main option 
> right now seems to revert the iommu_merge patch.

I don't think that is the correct fix.

-Andi
 

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Srihari Vijayaraghavan]

Andi Kleen <ak@suse.de> wrote:
[...]
> 
> IOMMU merging should be always safe. If it is not the driver should
> not submit things in a single SG list.
> 
> > So the main option 
> > right now seems to revert the iommu_merge patch.
> 
> I don't think that is the correct fix.

Just to make sure this problem doesn't fall off the radar screen, I've filed
it on Kernel Bugzilla: 
http://bugzilla.kernel.org/show_bug.cgi?id=9412

Thanks



      Feel safe with award winning spam protection on Yahoo!7 Mail.  
www.yahoo.com.au/mail 

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Tejun Heo]

Andi Kleen wrote:
>  
>> The AHCI code falls back to 32bit DMA in that case. Which in turn
>> causes the problem seen by Srihari. There is not much printk sticking
>> necessary, the code is simply not handling this. 
> 
> What code is not handling what? 
> 
> IOMMU merging should be always safe. If it is not the driver should
> not submit things in a single SG list.

Yeap, a sg merged by IOMMU should be safe.  It's just another contiguous
memory area from the POV of the controller anyway.  I wonder what went
wrong here.  What has exactly changed with iommu_merge patch?

-- 
tejun

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Thomas Gleixner]

On Tue, 20 Nov 2007, Tejun Heo wrote:

> Andi Kleen wrote:
> >  
> >> The AHCI code falls back to 32bit DMA in that case. Which in turn
> >> causes the problem seen by Srihari. There is not much printk sticking
> >> necessary, the code is simply not handling this. 
> > 
> > What code is not handling what? 
> > 
> > IOMMU merging should be always safe. If it is not the driver should
> > not submit things in a single SG list.
> 
> Yeap, a sg merged by IOMMU should be safe.  It's just another contiguous
> memory area from the POV of the controller anyway.  I wonder what went
> wrong here.  What has exactly changed with iommu_merge patch?

Not much:

-int iommu_merge __read_mostly = 0;
+int iommu_merge __read_mostly = 1;
 EXPORT_SYMBOL(iommu_merge);

Which in turn enables the iommu_merge functionality in gart_map_sg().

	for_each_sg(sg, s, nents, i) {
		dma_addr_t addr = sg_phys(s);
		s->dma_address = addr;
		BUG_ON(s->length == 0);

		nextneed = need_iommu(dev, addr, s->length);

		/* Handle the previous not yet processed entries */
		if (i > start) {
			/* Can only merge when the last chunk ends on a page
			   boundary and the new one doesn't have an offset. */
			if (!iommu_merge || !nextneed || !need || s->offset ||
			    (ps->offset + ps->length) % PAGE_SIZE) {
				if (dma_map_cont(start_sg, i - start, sgmap,
						  pages, need) < 0)
					goto error;
				out++;
				sgmap = sg_next(sgmap);
				pages = 0;
				start = i;
				start_sg = s;
			}
		}

		need = nextneed;
		pages += to_pages(s->offset, s->length);
		ps = s;
	}
	if (dma_map_cont(start_sg, i - start, sgmap, pages, need) < 0)
		goto error;

So with iommu_merge off we map for each entry in the sg
list. iommu_merge concatenates into larger segments.

This requires propably working 64bit DMA, which is not possible with
the SB600 controller.

Thanks,

	tglx

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Andi Kleen]

> This requires propably working 64bit DMA, which is not possible with
> the SB600 controller.

It should not no. The remapping is done into the GART which is <4GB
and that is the address the SB600 sees.

-Andi

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM II

[Andi Kleen]

> Which in turn enables the iommu_merge functionality in gart_map_sg().

> 	for_each_sg(sg, s, nents, i) {

Hmm, another thought. Maybe this code just has trouble with the new 
linked SG lists and it's not really a SB600 problem?

I did a quick test on two ATI machines with older chipset and iommu=force,merge
and it didn't show a problem though.

-Andi

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Thomas Gleixner]

On Tue, 20 Nov 2007, Andi Kleen wrote:

> 
> > This requires propably working 64bit DMA, which is not possible with
> > the SB600 controller.
> 
> It should not no. The remapping is done into the GART which is <4GB
> and that is the address the SB600 sees.

Hmm, I just checked the boot logs of the failing 4GB kernel:

BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
...
CPU 0: aperture @ c000000 size 32 MB
Aperture too small (32 MB)
No AGP bridge found
Your BIOS doesn't leave a aperture memory hole
Please enable the IOMMU option in the BIOS setup
This costs you 64 MB of RAM
Mapping aperture over 65536 KB of RAM @ c000000
Memory: 4055984k/4718592k available (2146k kernel code, 136780k reserved, 1273k data, 296k init)

4718592k * 1024 == 0x120000000

So now we have addresses > 4G and I suspect that this is somehow
related to the problem. 

When mem=3500M is given on the kernel command line, we do not use this
address space.

Also is the aperture size of 32MB somehow related to this ?

     tglx

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM

[Andi Kleen]

On Tuesday 20 November 2007 19:29:56 Thomas Gleixner wrote:
> On Tue, 20 Nov 2007, Andi Kleen wrote:
> 
> > 
> > > This requires propably working 64bit DMA, which is not possible with
> > > the SB600 controller.
> > 
> > It should not no. The remapping is done into the GART which is <4GB
> > and that is the address the SB600 sees.
> 
> Hmm, I just checked the boot logs of the failing 4GB kernel:
> 
> BIOS-e820: 0000000100000000 - 0000000120000000 (usable)
> ...
> CPU 0: aperture @ c000000 size 32 MB
> Aperture too small (32 MB)
> No AGP bridge found
> Your BIOS doesn't leave a aperture memory hole
> Please enable the IOMMU option in the BIOS setup
> This costs you 64 MB of RAM
> Mapping aperture over 65536 KB of RAM @ c000000
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The aperture is mapped at c000000 and c000000 + 64MB < 4GB


> Memory: 4055984k/4718592k available (2146k kernel code, 136780k reserved, 1273k data, 296k init)
> 
> 4718592k * 1024 == 0x120000000
> 
> So now we have addresses > 4G and I suspect that this is somehow
> related to the problem. 

Yes of course -- without >4GB the PCI-GART would not be used at all
(unless you force it) and then no merging.

> 
> Also is the aperture size of 32MB somehow related to this ?

This just means the BIOS didn't initialize it properly (a lot of 
BIOS don't do anymore these days because they assume it's a AGP
only feature) -- that is why the kernel allocated its own over
memory.

I think we really have to find out which request freezes it.
Can you perhaps just apply this patch and post the output?

Index: linux-2.6.24-rc1-hack/arch/x86/kernel/pci-gart_64.c
===================================================================
--- linux-2.6.24-rc1-hack.orig/arch/x86/kernel/pci-gart_64.c
+++ linux-2.6.24-rc1-hack/arch/x86/kernel/pci-gart_64.c
@@ -385,13 +385,19 @@ static int gart_map_sg(struct device *de
 	unsigned long pages = 0;
 	int need = 0, nextneed;
 	struct scatterlist *s, *ps, *start_sg, *sgmap;
-
+	
 	if (nents == 0) 
 		return 0;
 
 	if (!dev)
 		dev = &fallback_dev;
 
+	if (*dev->dma_mask <= 0xffffffff) { 
+		for_each_sg(sg, s, nents, i) { 
+			printk("%d: map %lx len %u dir %d\n", i, sg_phys(s), s->length, dir);
+		}
+	}
+
 	out = 0;
 	start = 0;
 	start_sg = sgmap = sg;



Tejun can probably figure out from that output where it comes
from in libata :)

-Andi

Re: 2.6.24-rc & SB600 AHCI no go on >=4GB of RAM II

[Jens Axboe]

On Tue, Nov 20 2007, Andi Kleen wrote:
> 
> > Which in turn enables the iommu_merge functionality in gart_map_sg().
> 
> > 	for_each_sg(sg, s, nents, i) {
> 
> Hmm, another thought. Maybe this code just has trouble with the new 
> linked SG lists and it's not really a SB600 problem?
> 
> I did a quick test on two ATI machines with older chipset and
> iommu=force,merge and it didn't show a problem though.

chained sg lists aren't enabled on libata, so it should not affect
libata drivers.

-- 
Jens Axboe