What's the SB600 64-bit DMA problem?

What's the SB600 64-bit DMA problem?

[George Spelvin]

I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with
8 GiB of RAM each.  I noticed on boot that the SB600 AHCI implementation
had a 32-bit DMA limitation, and looking in the archives, it
appears that folks from AMD think it doesn't need to be there.

So, by way of cautious expoeriment, I removed that quirk from the
appropriate part of drivers/ata/ahci.c and did the following test:
- Place the system drives on a plug-in SiI3232 controller
- Install one scratch drive on the SB600
- Run "badblocks -s -b512 -w -t random -t 0 /dev/sda 9999999 0", and
  repeat for consecutive 1e7-sector ranges.

This produced no errors in an hour or so.

Next step:
- Install the system drives on the SB600 ports.
- boot "init=/bin/sh" so the root file system is read-only
- "echo check > /sys/block/mdX/md/sync_action"

This also reported no errors.  The next step is to run e2fsck on those
file systems.

Can anyone suggest any other tests to confirm the presence or absence
of the 64-bit DMA bug?  My impression from the previous discussion was
that it was pretty obvious.

Here's the lspci -nn output:

00:00.0 Host bridge [0600]: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part [1002:5956]
00:03.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port B) [1002:5979]
00:04.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port A) [1002:597a]
00:05.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port B) [1002:597b]
00:09.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e]
00:0b.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port A) [1002:5980]
00:0c.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port B) [1002:5981]
00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380]
00:13.0 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI0) [1002:4387]
00:13.1 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI1) [1002:4388]
00:13.2 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI2) [1002:4389]
00:13.3 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI3) [1002:438a]
00:13.4 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI4) [1002:438b]
00:13.5 USB Controller [0c03]: ATI Technologies Inc SB600 USB Controller (EHCI) [1002:4386]
00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller [1002:4385] (rev 14)
00:14.1 IDE interface [0101]: ATI Technologies Inc SB600 IDE [1002:438c]
00:14.2 Audio device [0403]: ATI Technologies Inc SBx00 Azalia [1002:4383]
00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d]
00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384]
00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration [1022:1200]
00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map [1022:1201]
00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller [1022:1202]
00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control [1022:1203]
00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control [1022:1204]
01:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
02:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01)
04:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20]
05:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)] [1002:5b60]
05:00.1 Display controller [0380]: ATI Technologies Inc RV370 [Radeon X300SE] [1002:5b70]
06:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
07:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. IEEE 1394 Host Controller [1106:3044] (rev c0)

Re: What's the SB600 64-bit DMA problem?

[Grant Grundler]

On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote:
> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with
> 8 GiB of RAM each.  I noticed on boot that the SB600 AHCI implementation
> had a 32-bit DMA limitation, and looking in the archives, it
> appears that folks from AMD think it doesn't need to be there.

URL?
Just want to be sure I'm looking at the same thread.

> So, by way of cautious expoeriment, I removed that quirk from the
> appropriate part of drivers/ata/ahci.c and did the following test:
> - Place the system drives on a plug-in SiI3232 controller
> - Install one scratch drive on the SB600
> - Run "badblocks -s -b512 -w -t random -t 0 /dev/sda 9999999 0", and
>  repeat for consecutive 1e7-sector ranges.
>
> This produced no errors in an hour or so.
>
> Next step:
> - Install the system drives on the SB600 ports.
> - boot "init=/bin/sh" so the root file system is read-only
> - "echo check > /sys/block/mdX/md/sync_action"
>
> This also reported no errors.  The next step is to run e2fsck on those
> file systems.
>
> Can anyone suggest any other tests to confirm the presence or absence
> of the 64-bit DMA bug?  My impression from the previous discussion was
> that it was pretty obvious.

It's obvious if the memory corruption causes a system crash or
data corruption visible to an application that is looking for it.

To be certain,  you just need to confirm that the IO buffers
are guaranteed to (a) not be mapped for DMA via the GART
(setting 64-bit DMA mask should have the result) and (b) the
physical addresses of the buffers are above 4GB.

(b) is easier to prove by using an application that consumes more
than 4GB of RAM and verifies the data is not corrupted at any time.
Oracle would be such an app but I'm hoping someone can point
at something simpler. (no, firefox doesn't count :)

hth,
grant

>
> Here's the lspci -nn output:
>
> 00:00.0 Host bridge [0600]: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part [1002:5956]
> 00:03.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port B) [1002:5979]
> 00:04.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port A) [1002:597a]
> 00:05.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port B) [1002:597b]
> 00:09.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e]
> 00:0b.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port A) [1002:5980]
> 00:0c.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port B) [1002:5981]
> 00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380]
> 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI0) [1002:4387]
> 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI1) [1002:4388]
> 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI2) [1002:4389]
> 00:13.3 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI3) [1002:438a]
> 00:13.4 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI4) [1002:438b]
> 00:13.5 USB Controller [0c03]: ATI Technologies Inc SB600 USB Controller (EHCI) [1002:4386]
> 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller [1002:4385] (rev 14)
> 00:14.1 IDE interface [0101]: ATI Technologies Inc SB600 IDE [1002:438c]
> 00:14.2 Audio device [0403]: ATI Technologies Inc SBx00 Azalia [1002:4383]
> 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d]
> 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384]
> 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration [1022:1200]
> 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map [1022:1201]
> 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller [1022:1202]
> 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control [1022:1203]
> 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control [1022:1204]
> 01:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
> 02:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
> 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01)
> 04:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20]
> 05:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)] [1002:5b60]
> 05:00.1 Display controller [0380]: ATI Technologies Inc RV370 [Radeon X300SE] [1002:5b70]
> 06:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01)
> 07:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. IEEE 1394 Host Controller [1106:3044] (rev c0)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ide" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

"Grant Grundler" <grundler@google.com> wrote:
> On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote:
>> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with
>> 8 GiB of RAM each.  I noticed on boot that the SB600 AHCI implementation
>> had a 32-bit DMA limitation, and looking in the archives, it
>> appears that folks from AMD think it doesn't need to be there.
>
> URL?
> Just want to be sure I'm looking at the same thread.

"About forcing 32bit DMA patch for AMD690G(SB600)"
http://marc.info/?t=120107450600001

Re: What's the SB600 64-bit DMA problem?

[Mark Nelson]

On Sun, Sep 21, 2008 at 10:15 AM, George Spelvin <linux@horizon.com> wrote:
> "Grant Grundler" <grundler@google.com> wrote:
>> On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote:
>>> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with
>>> 8 GiB of RAM each.  I noticed on boot that the SB600 AHCI implementation
>>> had a 32-bit DMA limitation, and looking in the archives, it
>>> appears that folks from AMD think it doesn't need to be there.
>>
>> URL?
>> Just want to be sure I'm looking at the same thread.
>
> "About forcing 32bit DMA patch for AMD690G(SB600)"
> http://marc.info/?t=120107450600001

It looks like the results from people testing it further never arrived...

>From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci:
work around ATI SB600 h/w quirk"), it looks like the initial
idea was to limit each command to a maximum of 255 sectors (but
to still allow 64bit DMA), but according to a later commit
(4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600
workaround is suspect... play it safe for now") this did not
solve the lockups. But there are references in the thread above
to broken MSI on the SB600 possibly being the cause of the problem,
and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327
("ahci: SB600 ahci can't do MSI, blacklist that capability") has gone in
maybe this is why you're able to get 64bit DMA working? (although
we do need more hints on how to conclusively test this)

Thanks!

Mark

Re: What's the SB600 64-bit DMA problem?

[Tejun Heo]

Mark Nelson wrote:
> On Sun, Sep 21, 2008 at 10:15 AM, George Spelvin <linux@horizon.com> wrote:
>> "Grant Grundler" <grundler@google.com> wrote:
>>> On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote:
>>>> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with
>>>> 8 GiB of RAM each.  I noticed on boot that the SB600 AHCI implementation
>>>> had a 32-bit DMA limitation, and looking in the archives, it
>>>> appears that folks from AMD think it doesn't need to be there.
>>> URL?
>>> Just want to be sure I'm looking at the same thread.
>> "About forcing 32bit DMA patch for AMD690G(SB600)"
>> http://marc.info/?t=120107450600001
> 
> It looks like the results from people testing it further never arrived...
> 
>>From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci:
> work around ATI SB600 h/w quirk"), it looks like the initial
> idea was to limit each command to a maximum of 255 sectors (but
> to still allow 64bit DMA), but according to a later commit
> (4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600
> workaround is suspect... play it safe for now") this did not
> solve the lockups. But there are references in the thread above
> to broken MSI on the SB600 possibly being the cause of the problem,
> and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327
> ("ahci: SB600 ahci can't do MSI, blacklist that capability") has gone in
> maybe this is why you're able to get 64bit DMA working? (although
> we do need more hints on how to conclusively test this)

Hmm... I doubt MSI has anything to do with 64bit DMA but then again
anything is possible.  Shane, anything came out of the testing?

Thanks.

-- 
tejun

RE: What's the SB600 64-bit DMA problem?

[Huang, Shane]

Hi Tejun and Mark,


> -----Original Message-----
> From: Tejun Heo [mailto:htejun@gmail.com] 
> > 
> >>From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci:
> > work around ATI SB600 h/w quirk"), it looks like the initial
> > idea was to limit each command to a maximum of 255 sectors (but
> > to still allow 64bit DMA), but according to a later commit
> > (4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600
> > workaround is suspect... play it safe for now") this did not
> > solve the lockups. But there are references in the thread above
> > to broken MSI on the SB600 possibly being the cause of the problem,
> > and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327
> > ("ahci: SB600 ahci can't do MSI, blacklist that 
> capability") has gone in
> > maybe this is why you're able to get 64bit DMA working? (although
> > we do need more hints on how to conclusively test this)
> 
> Hmm... I doubt MSI has anything to do with 64bit DMA but then again
> anything is possible.  Shane, anything came out of the testing?


SB600 255 sectors limit and SB600 64 bit DMA bug are two different bugs,
the first one appears on many boards while the latter appears on some of
them
(our reference boards can NOT reproduce it).

As to SB600 SATA MSI bug, since we found that there is such kind of bug,
all our SB600 BIOS for reference boards disabled SATA MSI capability
itself.
(we assume all the SB600 BIOS in the marketing should disable MSI too,
but,
in fact, some did not as you know).

So I'm not able to do the test so far due to the MSI disabled BIOS.
If there is some guy whose SB600 BIOS is enabled, he may do some
verification
for us.

http://marc.info/?l=linux-ide&m=117810966106759&w=2
I remember "pci=nomsi" did not work before the workaround of 32bit SB600
SATA DMA? Tejun, is that true?


Anyway, I suggest we keep current SB600 workarounds.


Thanks
Shane

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

Just a little followup... I currently have the machine with 8 GB RAM,
all 4 SB600 channels populated, and 64-bit DMA enabled, operating in
production.  No problems so far.

Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
64-bit DMA at all?

> PCI: Using ACPI for IRQ routing
> pci 0000:00:00.0: BAR 3: can't allocate resource
> PCI-DMA: Disabling AGP.
> PCI-DMA: aperture base @ 20000000 size 65536 KB
> PCI-DMA: using GART IOMMU.
> PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture
> hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0
> hpet0: 4 32-bit timers, 14318180 Hz
> ACPI: RTC can wake from S4
> Switched to high resolution mode on CPU 0
> Switched to high resolution mode on CPU 2
> Switched to high resolution mode on CPU 1
> Switched to high resolution mode on CPU 3


# lspci -s 00:12.0 -vvv -nn -xxx
00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380] (prog-if 01)
        Subsystem: Micro-Star International Co., Ltd. Device [1462:7378]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 22
        Region 0: I/O ports at 8000 [size=8]
        Region 1: I/O ports at 7000 [size=4]
        Region 2: I/O ports at 6000 [size=8]
        Region 3: I/O ports at 5000 [size=4]
        Region 4: I/O ports at 4000 [size=16]
        Region 5: Memory at fe7ff800 (32-bit, non-prefetchable) [size=1K]
        Capabilities: [60] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Kernel driver in use: ahci
00: 02 10 80 43 07 01 30 02 00 01 06 01 10 40 00 00
10: 01 80 00 00 01 70 00 00 01 60 00 00 01 50 00 00
20: 01 40 00 00 00 f8 7f fe 00 00 00 00 62 14 78 73
30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00
40: 10 00 80 02 01 00 10 00 01 00 00 00 00 00 00 00
50: 05 00 84 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 01 00 22 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 12 00 10 00 0f 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 06 00 00 2c d6 01 b4 00 d6 01 b4 00
90: d6 01 b4 00 d6 01 b4 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 78 00 00 00 00 00 00 00 78 00 00
b0: 00 00 00 00 00 78 00 00 00 00 00 00 00 78 00 00
c0: 00 20 00 00 80 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

# lspci -s 02:00.0 -vvv -nn -xxx
02:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20]
        Subsystem: Micro-Star International Co., Ltd. Device [1462:3716]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at a800 [size=128]
        Region 2: I/O ports at a400 [size=256]
        Region 3: Memory at fe9ff000 (32-bit, non-prefetchable) [size=4K]
        Region 4: Memory at fe9c0000 (32-bit, non-prefetchable) [size=128K]
        Region 5: Memory at fe9fc000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 2
                Flags: PMEClk- DSI+ D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [70] Express (v1) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Suprise- LLActRep- BwNot-
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
        Capabilities: [94] SATA HBA <?>
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Virtual Channel <?>
        Capabilities: [160] Device Serial Number 01-00-00-00-02-00-00-00
        Capabilities: [170] Power Budgeting <?>
        Kernel driver in use: ahci
00: 5a 10 20 3f 07 01 10 00 00 00 04 01 10 00 00 00
10: 01 a8 00 00 00 00 00 00 01 a4 00 00 00 f0 9f fe
20: 00 00 9c fe 00 c0 9f fe 00 00 00 00 62 14 16 37
30: 00 00 00 00 50 00 00 00 00 00 00 00 0a 01 00 00
40: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 01 70 22 02 00 00 00 00 00 00 00 00 00 00 00 00
60: 05 70 80 00 00 00 fa df 5a 00 00 00 00 00 00 00
70: 10 94 11 00 01 00 00 00 10 28 0a 00 11 0c 00 00
80: 42 00 11 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 12 00 10 00 46 02 00 00 09 00 4c 01
a0: 01 1c 00 00 fb ff ff 00 07 20 16 1b 38 81 0b 00
b0: de d0 0b 00 be 0d 00 00 02 28 11 00 00 00 00 00
c0: 00 02 00 00 03 00 03 00 00 00 00 00 10 00 20 09
d0: 0a 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


dmesg excerpt:
ahci 0000:00:12.0: version 3.0
ahci 0000:00:12.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22
ahci 0000:00:12.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA mode
ahci 0000:00:12.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
scsi0 : ahci
scsi1 : ahci
scsi2 : ahci
scsi3 : ahci
ata1: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ff900 irq 22
ata2: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ff980 irq 22
ata3: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ffa00 irq 22
ata4: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ffa80 irq 22
ata1: softreset failed (device not ready)
ata1: failed due to HW bug, retry pmp=0
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata1.00: ATA-7: ST3400832AS, 3.01, max UDMA/133
ata1.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata1.00: SB600 AHCI: limiting to 255 sectors per cmd
ata1.00: SB600 AHCI: limiting to 255 sectors per cmd
ata1.00: configured for UDMA/133
ata2: softreset failed (device not ready)
ata2: failed due to HW bug, retry pmp=0
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata2.00: ATA-7: ST3400832AS, 3.01, max UDMA/133
ata2.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata2.00: SB600 AHCI: limiting to 255 sectors per cmd
ata2.00: SB600 AHCI: limiting to 255 sectors per cmd
ata2.00: configured for UDMA/133
ata3: softreset failed (device not ready)
ata3: failed due to HW bug, retry pmp=0
ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata3.00: ATA-7: ST3400832AS, 3.01, max UDMA/133
ata3.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata3.00: SB600 AHCI: limiting to 255 sectors per cmd
ata3.00: SB600 AHCI: limiting to 255 sectors per cmd
ata3.00: configured for UDMA/133
ata4: softreset failed (device not ready)
ata4: failed due to HW bug, retry pmp=0
ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata4.00: ATA-7: ST3400832AS, 3.01, max UDMA/133
ata4.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32)
ata4.00: SB600 AHCI: limiting to 255 sectors per cmd
ata4.00: SB600 AHCI: limiting to 255 sectors per cmd
ata4.00: configured for UDMA/133
scsi 0:0:0:0: Direct-Access     ATA      ST3400832AS      3.01 PQ: 0 ANSI: 5
sd 0:0:0:0: [sda] 781422768 512-byte hardware sectors (400088 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 781422768 512-byte hardware sectors (400088 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4
sd 0:0:0:0: [sda] Attached SCSI disk
scsi 1:0:0:0: Direct-Access     ATA      ST3400832AS      3.01 PQ: 0 ANSI: 5
sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdb: sdb1 sdb2 sdb3 sdb4
sd 1:0:0:0: [sdb] Attached SCSI disk
scsi 2:0:0:0: Direct-Access     ATA      ST3400832AS      3.01 PQ: 0 ANSI: 5
sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB)
sd 2:0:0:0: [sdc] Write Protect is off
sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00
sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdc: sdc1 sdc2 sdc3 sdc4
sd 2:0:0:0: [sdc] Attached SCSI disk
scsi 3:0:0:0: Direct-Access     ATA      ST3400832AS      3.01 PQ: 0 ANSI: 5
sd 3:0:0:0: [sdd] 781422768 512-byte hardware sectors (400088 MB)
sd 3:0:0:0: [sdd] Write Protect is off
sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 3:0:0:0: [sdd] 781422768 512-byte hardware sectors (400088 MB)
sd 3:0:0:0: [sdd] Write Protect is off
sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00
sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdd: sdd1 sdd2 sdd3 sdd4
sd 3:0:0:0: [sdd] Attached SCSI disk
ahci 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
ahci 0000:02:00.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl RAID mode
ahci 0000:02:00.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part 
ahci 0000:02:00.0: setting latency timer to 64
scsi4 : ahci
scsi5 : ahci
scsi6 : ahci
scsi7 : ahci
ata5: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc100 irq 17
ata6: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc180 irq 17
ata7: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc200 irq 17
ata8: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc280 irq 17
ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata5.00: ATA-7: ST3400832AS, 3.03, max UDMA/133
ata5.00: 781422768 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata5.00: configured for UDMA/133
ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300)
ata6.00: ATA-7: ST3400832AS, 3.01, max UDMA/133
ata6.00: 781422768 sectors, multi 0: LBA48 NCQ (depth 31/32)
ata6.00: configured for UDMA/133
ata7: SATA link down (SStatus 0 SControl 300)
ata8: SATA link down (SStatus 0 SControl 300)
scsi 4:0:0:0: Direct-Access     ATA      ST3400832AS      3.03 PQ: 0 ANSI: 5
sd 4:0:0:0: [sde] 781422768 512-byte hardware sectors (400088 MB)
sd 4:0:0:0: [sde] Write Protect is off
sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO
 or FUA
sd 4:0:0:0: [sde] 781422768 512-byte hardware sectors (400088 MB)
sd 4:0:0:0: [sde] Write Protect is off
sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00
sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO
 or FUA
 sde: sde1 sde2 sde3 sde4
sd 4:0:0:0: [sde] Attached SCSI disk
scsi 5:0:0:0: Direct-Access     ATA      ST3400832AS      3.01 PQ: 0 ANSI: 5
sd 5:0:0:0: [sdf] 781422768 512-byte hardware sectors (400088 MB)
sd 5:0:0:0: [sdf] Write Protect is off
sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
sd 5:0:0:0: [sdf] 781422768 512-byte hardware sectors (400088 MB)
sd 5:0:0:0: [sdf] Write Protect is off
sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00
sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
 sdf: sdf1 sdf2 sdf3 sdf4
sd 5:0:0:0: [sdf] Attached SCSI disk

Re: What's the SB600 64-bit DMA problem?

[Tejun Heo]

George Spelvin wrote:
> Just a little followup... I currently have the machine with 8 GB RAM,
> all 4 SB600 channels populated, and 64-bit DMA enabled, operating in
> production.  No problems so far.
> 
> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
> 64-bit DMA at all?

Can you turn it off and see what happens?

-- 
tejun

Re: What's the SB600 64-bit DMA problem?

[Grant Grundler]

On Wed, Sep 24, 2008 at 8:24 PM, George Spelvin <linux@horizon.com> wrote:
> Just a little followup... I currently have the machine with 8 GB RAM,
> all 4 SB600 channels populated, and 64-bit DMA enabled, operating in
> production.  No problems so far.
>
> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
> 64-bit DMA at all?

If pci_dev->dma_mask has more bits set than the highest physical
memory address, then the GART will not be used for that device.

See "need_iommu()" in arch/x86/kernel/pci-gart_64.c and how that's
used in gart_map_single().

My assumption is libata will not override something the PCI device driver sets.

hth,
grant

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

Tejun Heo <htejun@gmail.com> wrote:
>> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
>> using 64-bit DMA at all?
>
> Can you turn it off and see what happens?

I can if I can verify that every other bus-master PCI device in the
system can do 64-bit DMA.

Do you know how to check that?  I haven't found a file under /sys or
/proc that tells me what the DMA masks are of various system
drivers, but that doesn't mean that it doesn't exist.

Thanks.

Re: What's the SB600 64-bit DMA problem?

[Tejun Heo]

George Spelvin wrote:
> Tejun Heo <htejun@gmail.com> wrote:
>>> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
>>> using 64-bit DMA at all?
>> Can you turn it off and see what happens?
> 
> I can if I can verify that every other bus-master PCI device in the
> system can do 64-bit DMA.
> 
> Do you know how to check that?  I haven't found a file under /sys or
> /proc that tells me what the DMA masks are of various system
> drivers, but that doesn't mean that it doesn't exist.

It doesn't really matter.  The kernel will bounce buffers as
necessary.

-- 
tejun

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

Tejun Heo <htejun@gmail.com> wrote:

> George Spelvin wrote:
>> Tejun Heo <htejun@gmail.com> wrote:
>>>> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using
>>>> using 64-bit DMA at all?
>>> Can you turn it off and see what happens?
>> 
>> I can if I can verify that every other bus-master PCI device in the
>> system can do 64-bit DMA.
>> 
>> Do you know how to check that?  I haven't found a file under /sys or
>> /proc that tells me what the DMA masks are of various system
>> drivers, but that doesn't mean that it doesn't exist.
>
> It doesn't really matter.  The kernel will bounce buffers as
> necessary.

Er, according to the docs I can find, that's iommu=soft, and still
doesn't guarantee 64-bit DMA.  I was thinking about using iommu=off,
which disables everything including bounce buffers, and thereby forces
64-bit DMA.

Um... actually, I just tried it, and when there's >4G of memory, the kernel
forces iommu=soft, and it works.  At least as far as e2fsck -n on every
file system (including a 1.7 TB RAID-5) can tell.

It still doesn't definitively tell me that a DMA to and from an address
>4G was done, but it's an awfully strong hint.  I was thinking of
inserting a debug message that logs every time the "highest DMA address
used" increases, than I can watch it to see for sure.

ATM I'm just recompiling the kernel from -rc6 to -rc7.

It's in single-user mode so I can't ssh over the boot messages, but
they definitely included "64bit" from ahci_print_info().

Speaking of anci_print_info(), sould something like the following be a good
thing?

diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
index 2e1a7cb..18137d2 100644
--- a/drivers/ata/ahci.c
+++ b/drivers/ata/ahci.c
@@ -2464,22 +2464,22 @@ static void ahci_print_info(struct ata_host *host)
 		"%s\n"
 		,
 
-		cap & (1 << 31) ? "64bit " : "",
-		cap & (1 << 30) ? "ncq " : "",
-		cap & (1 << 29) ? "sntf " : "",
-		cap & (1 << 28) ? "ilck " : "",
-		cap & (1 << 27) ? "stag " : "",
-		cap & (1 << 26) ? "pm " : "",
-		cap & (1 << 25) ? "led " : "",
-
-		cap & (1 << 24) ? "clo " : "",
-		cap & (1 << 19) ? "nz " : "",
-		cap & (1 << 18) ? "only " : "",
-		cap & (1 << 17) ? "pmp " : "",
-		cap & (1 << 15) ? "pio " : "",
-		cap & (1 << 14) ? "slum " : "",
-		cap & (1 << 13) ? "part " : "",
-		cap & (1 << 6) ? "ems ": ""
+		cap & HOST_CAP_64   ? "64bit " : "",
+		cap & HOST_CAP_NCQ  ? "ncq " : "",
+		cap & HOST_CAP_SNTF ? "sntf " : "",
+		cap & (1 << 28)     ? "ilck " : "",
+		cap & HOST_CAP_SSS  ? "stag " : "",
+		cap & HOST_CAP_ALPM ? "pm " : "",
+		cap & (1 << 25)     ? "led " : "",
+
+		cap & HOST_CAP_CLO  ? "clo " : "",
+		cap & (1 << 19)     ? "nz " : "",
+		cap & (1 << 18)     ? "only " : "",
+		cap & HOST_CAP_PMP  ? "pmp " : "",
+		cap & (1 << 15)     ? "pio " : "",
+		cap & HOST_CAP_SSC  ? "slum " : "",
+		cap & (1 << 13)     ? "part " : "",
+		cap & HOST_CAP_EMS  ? "ems ": ""
 		);
 }
 

Re: What's the SB600 64-bit DMA problem?

[Tejun Heo]

George Spelvin wrote:
> ATM I'm just recompiling the kernel from -rc6 to -rc7.
> 
> It's in single-user mode so I can't ssh over the boot messages, but
> they definitely included "64bit" from ahci_print_info().

Thanks for verifying.  I would love to lift the 32 restriction from
SB600 but it's been quite a rocky ride and I still can't tell what's
the determining factor.  :-(

> Speaking of anci_print_info(), sould something like the following be a good
> thing?
>
> diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
> index 2e1a7cb..18137d2 100644
> --- a/drivers/ata/ahci.c
> +++ b/drivers/ata/ahci.c
> @@ -2464,22 +2464,22 @@ static void ahci_print_info(struct ata_host *host)
>  		"%s\n"
>  		,
>  
> -		cap & (1 << 31) ? "64bit " : "",
> -		cap & (1 << 30) ? "ncq " : "",
> -		cap & (1 << 29) ? "sntf " : "",
> -		cap & (1 << 28) ? "ilck " : "",
> -		cap & (1 << 27) ? "stag " : "",
> -		cap & (1 << 26) ? "pm " : "",
> -		cap & (1 << 25) ? "led " : "",
> -
> -		cap & (1 << 24) ? "clo " : "",
> -		cap & (1 << 19) ? "nz " : "",
> -		cap & (1 << 18) ? "only " : "",
> -		cap & (1 << 17) ? "pmp " : "",
> -		cap & (1 << 15) ? "pio " : "",
> -		cap & (1 << 14) ? "slum " : "",
> -		cap & (1 << 13) ? "part " : "",
> -		cap & (1 << 6) ? "ems ": ""
> +		cap & HOST_CAP_64   ? "64bit " : "",
> +		cap & HOST_CAP_NCQ  ? "ncq " : "",
> +		cap & HOST_CAP_SNTF ? "sntf " : "",
> +		cap & (1 << 28)     ? "ilck " : "",
> +		cap & HOST_CAP_SSS  ? "stag " : "",
> +		cap & HOST_CAP_ALPM ? "pm " : "",
> +		cap & (1 << 25)     ? "led " : "",
> +
> +		cap & HOST_CAP_CLO  ? "clo " : "",
> +		cap & (1 << 19)     ? "nz " : "",
> +		cap & (1 << 18)     ? "only " : "",
> +		cap & HOST_CAP_PMP  ? "pmp " : "",
> +		cap & (1 << 15)     ? "pio " : "",
> +		cap & HOST_CAP_SSC  ? "slum " : "",
> +		cap & (1 << 13)     ? "part " : "",
> +		cap & HOST_CAP_EMS  ? "ems ": ""

Yeah, probably.  While at it, can you just add constants for other
bits too and send it as properly singed off patch?

Thanks.

-- 
tejun

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

Tejun Heo <htejun@gmail.com> wrote:

> George Spelvin wrote:
>> ATM I'm just recompiling the kernel from -rc6 to -rc7.
>> 
>> It's in single-user mode so I can't ssh over the boot messages, but
>> they definitely included "64bit" from ahci_print_info().
>
> Thanks for verifying.  I would love to lift the 32 restriction from
> SB600 but it's been quite a rocky ride and I still can't tell what's
> the determining factor.  :-(

H'm.... 64-bit support comes through in multiple places.
I wonder if it's just one of them?
- HBA memory-mapped registers (which are in fact limited to 32 bits)
- Port x command list base address
- Port x FIS base address
- Command table base address (command header, offset 8)
- Data base address (physical region descriptor, offset 0)

What we really care about is that the data physical regions can
be anywhere in 64-bit space.  If any of the others were limited to
32 bits, that would only be slightly annoying.

But...
The base addresses are already above 4G.  Aobve 8G, actually.

Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400
Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400
Port 2 base addresses: 02 1f98 0000, 02 1f98 0400
Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400

/proc/iomem looks like:
00000000-0009fbff : System RAM
0009fc00-0009ffff : reserved
000c0000-000cffff : pnp 00:0e
000e4000-000fffff : reserved
00100000-dffbffff : System RAM
  00200000-005187db : Kernel code
  005187dc-006c310f : Kernel data
  0073f000-007bb4cf : Kernel bss
  20000000-23ffffff : GART
dffc0000-dffcdfff : ACPI Tables
dffce000-dffeffff : ACPI Non-volatile Storage
dfff0000-dfffdfff : reserved
e0000000-efffffff : PCI MMCONFIG 0
fc000000-fdffffff : PCI Bus 0000:03
  fc000000-fdffffff : 0000:03:00.0
fe7f4000-fe7f7fff : 0000:00:14.2
fe7fa000-fe7fafff : 0000:00:13.4
  fe7fa000-fe7fafff : ohci_hcd
fe7fb000-fe7fbfff : 0000:00:13.3
  fe7fb000-fe7fbfff : ohci_hcd
fe7fc000-fe7fcfff : 0000:00:13.2
  fe7fc000-fe7fcfff : ohci_hcd
fe7fd000-fe7fdfff : 0000:00:13.1
  fe7fd000-fe7fdfff : ohci_hcd
fe7fe000-fe7fefff : 0000:00:13.0
  fe7fe000-fe7fefff : ohci_hcd
fe7ff000-fe7ff0ff : 0000:00:13.5
  fe7ff000-fe7ff0ff : ehci_hcd
fe7ff800-fe7ffbff : 0000:00:12.0
  fe7ff800-fe7ffbff : ahci
fe800000-fe8fffff : PCI Bus 0000:01
  fe8c0000-fe8dffff : 0000:01:00.0
  fe8ff000-fe8fffff : 0000:01:00.0
    fe8ff000-fe8fffff : r8169
fe900000-fe9fffff : PCI Bus 0000:02
  fe9c0000-fe9dffff : 0000:02:00.0
    fe9c0000-fe9dffff : ahci
  fe9fc000-fe9fdfff : 0000:02:00.0
    fe9fc000-fe9fdfff : ahci
  fe9ff000-fe9fffff : 0000:02:00.0
    fe9ff000-fe9fffff : ahci
fea00000-feafffff : PCI Bus 0000:03
  feac0000-feadffff : 0000:03:00.0
  feae0000-feaeffff : 0000:03:00.1
  feaf0000-feafffff : 0000:03:00.0
feb00000-febfffff : PCI Bus 0000:04
  febff800-febfffff : 0000:04:00.0
fec00000-fec00fff : IOAPIC 0
fed00000-fed003ff : HPET 2
fee00000-fee00fff : Local APIC
fff00000-ffffffff : reserved
100000000-21fffffff : System RAM

Re: What's the SB600 64-bit DMA problem?

[Tejun Heo]

George Spelvin wrote:
> Tejun Heo <htejun@gmail.com> wrote:
> 
>> George Spelvin wrote:
>>> ATM I'm just recompiling the kernel from -rc6 to -rc7.
>>>
>>> It's in single-user mode so I can't ssh over the boot messages, but
>>> they definitely included "64bit" from ahci_print_info().
>> Thanks for verifying.  I would love to lift the 32 restriction from
>> SB600 but it's been quite a rocky ride and I still can't tell what's
>> the determining factor.  :-(
> 
> H'm.... 64-bit support comes through in multiple places.
> I wonder if it's just one of them?
> - HBA memory-mapped registers (which are in fact limited to 32 bits)

This is determined by the BIOS and I don't think it ever puts it over
32bit boundary.

> - Port x command list base address
> - Port x FIS base address
> - Command table base address (command header, offset 8)
> - Data base address (physical region descriptor, offset 0)

The first three are controlled by consistent_dma_mask while the last
one is controlled by dma_mask, so maybe setting only
consistent_dma_mask to 32bit could help.  The problem is how to verify
them.  You'll need to dig the list and bugzilla to look for reporters
who reported failure on 64bit DMA and verify it actually fixes the
problem.  It can be tricky as the problem doesn't seem to appear for
all users.

> What we really care about is that the data physical regions can
> be anywhere in 64-bit space.  If any of the others were limited to
> 32 bits, that would only be slightly annoying.
> 
> But...
> The base addresses are already above 4G.  Aobve 8G, actually.
> 
> Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400
> Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400
> Port 2 base addresses: 02 1f98 0000, 02 1f98 0400
> Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400

Where are these numbers from?

-- 
tejun

Re: What's the SB600 64-bit DMA problem?

[George Spelvin]

Tejun Heo <htejun@gmail.com> wrote:
> George Spelvin wrote:
>> But...
>> The base addresses are already above 4G.  Aobve 8G, actually.
>> 
>> Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400
>> Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400
>> Port 2 base addresses: 02 1f98 0000, 02 1f98 0400
>> Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400
>
> Where are these numbers from?

A dump of the appropriate region of memory (1k @ 0xfe7ff800), offsets
0x100, 0x180, 0x200 and 0x280.  Here's a full dump (rebooted in between,
so the addresses might change).

dd if=/dev/mem bs=1024 skip=4169726 count=1 | od -Ax -tx4
1+0 records in
1+0 records out
1024 bytes (1.0 kB) copied, 0.000203726 s, 5.0 MB/s
000000 f722ff83 80000002 00000000 0000000f
000010 00010100 000101f8 00000000 00000000
000020 00000000 00000000 00000000 00000000
*
0000f0 00000000 00000000 00000000 03ef99fc
000100 1f1e0000 00000002 1f1e0400 00000002
000110 00000000 7840007f 00042017 00000000
000120 00000040 00000101 00000113 00000300
000130 00000000 00000000 00000000 00000000
000140 0001000f 00000000 00000000 00000000
000150 00000000 00000000 00000000 00000000
*
000170 00000000 00000000 00000000 08001c00
000180 1f1a0000 00000002 1f1a0400 00000002
000190 00000000 7840007f 0004a017 00000000
0001a0 00000040 00000101 00000113 00000300
0001b0 00000000 00000001 00000000 00000000
0001c0 0001000f 00000000 00000000 00000000
0001d0 00000000 00000000 00000000 00000000
*
0001f0 00000000 00000000 00000000 08001c00
000200 1f980000 00000002 1f980400 00000002
000210 00000000 7840007f 00042017 00000000
000220 00000040 00000101 00000113 00000300
000230 00000000 00000000 00000000 00000000
000240 0001000f 00000000 00000000 00000000
000250 00000000 00000000 00000000 00000000
*
000270 00000000 00000000 00000000 08001c00
000280 1f9e0000 00000002 1f9e0400 00000002
000290 00000000 7840007f 00042017 00000000
0002a0 00000040 00000101 00000113 00000300
0002b0 00000000 00000000 00000000 00000000
0002c0 0001000f 00000000 00000000 00000000
0002d0 00000000 00000000 00000000 00000000
*
0002f0 00000000 00000000 00000000 08001c00
000300 00000000 00000000 00000000 00000000
*
000400