* What's the SB600 64-bit DMA problem? @ 2008-09-20 15:17 George Spelvin 2008-09-20 22:28 ` Grant Grundler 0 siblings, 1 reply; 16+ messages in thread From: George Spelvin @ 2008-09-20 15:17 UTC (permalink / raw) To: linux-ide; +Cc: linux I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with 8 GiB of RAM each. I noticed on boot that the SB600 AHCI implementation had a 32-bit DMA limitation, and looking in the archives, it appears that folks from AMD think it doesn't need to be there. So, by way of cautious expoeriment, I removed that quirk from the appropriate part of drivers/ata/ahci.c and did the following test: - Place the system drives on a plug-in SiI3232 controller - Install one scratch drive on the SB600 - Run "badblocks -s -b512 -w -t random -t 0 /dev/sda 9999999 0", and repeat for consecutive 1e7-sector ranges. This produced no errors in an hour or so. Next step: - Install the system drives on the SB600 ports. - boot "init=/bin/sh" so the root file system is read-only - "echo check > /sys/block/mdX/md/sync_action" This also reported no errors. The next step is to run e2fsck on those file systems. Can anyone suggest any other tests to confirm the presence or absence of the 64-bit DMA bug? My impression from the previous discussion was that it was pretty obvious. Here's the lspci -nn output: 00:00.0 Host bridge [0600]: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part [1002:5956] 00:03.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port B) [1002:5979] 00:04.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port A) [1002:597a] 00:05.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port B) [1002:597b] 00:09.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e] 00:0b.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port A) [1002:5980] 00:0c.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port B) [1002:5981] 00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380] 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI0) [1002:4387] 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI1) [1002:4388] 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI2) [1002:4389] 00:13.3 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI3) [1002:438a] 00:13.4 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI4) [1002:438b] 00:13.5 USB Controller [0c03]: ATI Technologies Inc SB600 USB Controller (EHCI) [1002:4386] 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller [1002:4385] (rev 14) 00:14.1 IDE interface [0101]: ATI Technologies Inc SB600 IDE [1002:438c] 00:14.2 Audio device [0403]: ATI Technologies Inc SBx00 Azalia [1002:4383] 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d] 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384] 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration [1022:1200] 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map [1022:1201] 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller [1022:1202] 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control [1022:1203] 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control [1022:1204] 01:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) 02:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01) 04:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20] 05:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)] [1002:5b60] 05:00.1 Display controller [0380]: ATI Technologies Inc RV370 [Radeon X300SE] [1002:5b70] 06:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) 07:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. IEEE 1394 Host Controller [1106:3044] (rev c0) ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-20 15:17 What's the SB600 64-bit DMA problem? George Spelvin @ 2008-09-20 22:28 ` Grant Grundler 2008-09-21 0:15 ` George Spelvin 0 siblings, 1 reply; 16+ messages in thread From: Grant Grundler @ 2008-09-20 22:28 UTC (permalink / raw) To: George Spelvin; +Cc: linux-ide On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote: > I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with > 8 GiB of RAM each. I noticed on boot that the SB600 AHCI implementation > had a 32-bit DMA limitation, and looking in the archives, it > appears that folks from AMD think it doesn't need to be there. URL? Just want to be sure I'm looking at the same thread. > So, by way of cautious expoeriment, I removed that quirk from the > appropriate part of drivers/ata/ahci.c and did the following test: > - Place the system drives on a plug-in SiI3232 controller > - Install one scratch drive on the SB600 > - Run "badblocks -s -b512 -w -t random -t 0 /dev/sda 9999999 0", and > repeat for consecutive 1e7-sector ranges. > > This produced no errors in an hour or so. > > Next step: > - Install the system drives on the SB600 ports. > - boot "init=/bin/sh" so the root file system is read-only > - "echo check > /sys/block/mdX/md/sync_action" > > This also reported no errors. The next step is to run e2fsck on those > file systems. > > Can anyone suggest any other tests to confirm the presence or absence > of the 64-bit DMA bug? My impression from the previous discussion was > that it was pretty obvious. It's obvious if the memory corruption causes a system crash or data corruption visible to an application that is looking for it. To be certain, you just need to confirm that the IO buffers are guaranteed to (a) not be mapped for DMA via the GART (setting 64-bit DMA mask should have the result) and (b) the physical addresses of the buffers are above 4GB. (b) is easier to prove by using an application that consumes more than 4GB of RAM and verifies the data is not corrupted at any time. Oracle would be such an app but I'm hoping someone can point at something simpler. (no, firefox doesn't count :) hth, grant > > Here's the lspci -nn output: > > 00:00.0 Host bridge [0600]: ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part [1002:5956] > 00:03.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port B) [1002:5979] > 00:04.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port A) [1002:597a] > 00:05.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port B) [1002:597b] > 00:09.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (PCI express gpp port E) [1002:597e] > 00:0b.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port A) [1002:5980] > 00:0c.0 PCI bridge [0604]: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx1 port B) [1002:5981] > 00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380] > 00:13.0 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI0) [1002:4387] > 00:13.1 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI1) [1002:4388] > 00:13.2 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI2) [1002:4389] > 00:13.3 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI3) [1002:438a] > 00:13.4 USB Controller [0c03]: ATI Technologies Inc SB600 USB (OHCI4) [1002:438b] > 00:13.5 USB Controller [0c03]: ATI Technologies Inc SB600 USB Controller (EHCI) [1002:4386] > 00:14.0 SMBus [0c05]: ATI Technologies Inc SBx00 SMBus Controller [1002:4385] (rev 14) > 00:14.1 IDE interface [0101]: ATI Technologies Inc SB600 IDE [1002:438c] > 00:14.2 Audio device [0403]: ATI Technologies Inc SBx00 Azalia [1002:4383] > 00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d] > 00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384] > 00:18.0 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration [1022:1200] > 00:18.1 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map [1022:1201] > 00:18.2 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller [1022:1202] > 00:18.3 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control [1022:1203] > 00:18.4 Host bridge [0600]: Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control [1022:1204] > 01:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) > 02:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) > 03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller [10ec:8168] (rev 01) > 04:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20] > 05:00.0 VGA compatible controller [0300]: ATI Technologies Inc RV370 5B60 [Radeon X300 (PCIE)] [1002:5b60] > 05:00.1 Display controller [0380]: ATI Technologies Inc RV370 [Radeon X300SE] [1002:5b70] > 06:00.0 Mass storage controller [0180]: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller [1095:3132] (rev 01) > 07:00.0 FireWire (IEEE 1394) [0c00]: VIA Technologies, Inc. IEEE 1394 Host Controller [1106:3044] (rev c0) > -- > To unsubscribe from this list: send the line "unsubscribe linux-ide" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-20 22:28 ` Grant Grundler @ 2008-09-21 0:15 ` George Spelvin 2008-09-24 0:30 ` Mark Nelson 0 siblings, 1 reply; 16+ messages in thread From: George Spelvin @ 2008-09-21 0:15 UTC (permalink / raw) To: grundler; +Cc: linux, linux-ide "Grant Grundler" <grundler@google.com> wrote: > On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote: >> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with >> 8 GiB of RAM each. I noticed on boot that the SB600 AHCI implementation >> had a 32-bit DMA limitation, and looking in the archives, it >> appears that folks from AMD think it doesn't need to be there. > > URL? > Just want to be sure I'm looking at the same thread. "About forcing 32bit DMA patch for AMD690G(SB600)" http://marc.info/?t=120107450600001 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-21 0:15 ` George Spelvin @ 2008-09-24 0:30 ` Mark Nelson 2008-09-24 9:46 ` Tejun Heo 0 siblings, 1 reply; 16+ messages in thread From: Mark Nelson @ 2008-09-24 0:30 UTC (permalink / raw) To: George Spelvin; +Cc: grundler, linux-ide, shane.huang, htejun, jgarzik On Sun, Sep 21, 2008 at 10:15 AM, George Spelvin <linux@horizon.com> wrote: > "Grant Grundler" <grundler@google.com> wrote: >> On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote: >>> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with >>> 8 GiB of RAM each. I noticed on boot that the SB600 AHCI implementation >>> had a 32-bit DMA limitation, and looking in the archives, it >>> appears that folks from AMD think it doesn't need to be there. >> >> URL? >> Just want to be sure I'm looking at the same thread. > > "About forcing 32bit DMA patch for AMD690G(SB600)" > http://marc.info/?t=120107450600001 It looks like the results from people testing it further never arrived... >From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci: work around ATI SB600 h/w quirk"), it looks like the initial idea was to limit each command to a maximum of 255 sectors (but to still allow 64bit DMA), but according to a later commit (4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600 workaround is suspect... play it safe for now") this did not solve the lockups. But there are references in the thread above to broken MSI on the SB600 possibly being the cause of the problem, and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327 ("ahci: SB600 ahci can't do MSI, blacklist that capability") has gone in maybe this is why you're able to get 64bit DMA working? (although we do need more hints on how to conclusively test this) Thanks! Mark ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-24 0:30 ` Mark Nelson @ 2008-09-24 9:46 ` Tejun Heo 2008-09-24 10:19 ` Huang, Shane 0 siblings, 1 reply; 16+ messages in thread From: Tejun Heo @ 2008-09-24 9:46 UTC (permalink / raw) To: Mark Nelson; +Cc: George Spelvin, grundler, linux-ide, shane.huang, jgarzik Mark Nelson wrote: > On Sun, Sep 21, 2008 at 10:15 AM, George Spelvin <linux@horizon.com> wrote: >> "Grant Grundler" <grundler@google.com> wrote: >>> On Sat, Sep 20, 2008 at 8:17 AM, George Spelvin <linux@horizon.com> wrote: >>>> I just got a couple of MSI KA92 (AMD 790FX-based) motherboards with >>>> 8 GiB of RAM each. I noticed on boot that the SB600 AHCI implementation >>>> had a 32-bit DMA limitation, and looking in the archives, it >>>> appears that folks from AMD think it doesn't need to be there. >>> URL? >>> Just want to be sure I'm looking at the same thread. >> "About forcing 32bit DMA patch for AMD690G(SB600)" >> http://marc.info/?t=120107450600001 > > It looks like the results from people testing it further never arrived... > >>From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci: > work around ATI SB600 h/w quirk"), it looks like the initial > idea was to limit each command to a maximum of 255 sectors (but > to still allow 64bit DMA), but according to a later commit > (4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600 > workaround is suspect... play it safe for now") this did not > solve the lockups. But there are references in the thread above > to broken MSI on the SB600 possibly being the cause of the problem, > and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327 > ("ahci: SB600 ahci can't do MSI, blacklist that capability") has gone in > maybe this is why you're able to get 64bit DMA working? (although > we do need more hints on how to conclusively test this) Hmm... I doubt MSI has anything to do with 64bit DMA but then again anything is possible. Shane, anything came out of the testing? Thanks. -- tejun ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: What's the SB600 64-bit DMA problem? 2008-09-24 9:46 ` Tejun Heo @ 2008-09-24 10:19 ` Huang, Shane 2008-09-25 3:24 ` George Spelvin 0 siblings, 1 reply; 16+ messages in thread From: Huang, Shane @ 2008-09-24 10:19 UTC (permalink / raw) To: Tejun Heo, Mark Nelson Cc: George Spelvin, grundler, linux-ide, jgarzik, Huang, Shane Hi Tejun and Mark, > -----Original Message----- > From: Tejun Heo [mailto:htejun@gmail.com] > > > >>From git commit a878539ef994787c447a98c2e3ba0fe3dad984ec ("ahci: > > work around ATI SB600 h/w quirk"), it looks like the initial > > idea was to limit each command to a maximum of 255 sectors (but > > to still allow 64bit DMA), but according to a later commit > > (4cde32fc4b32e96a99063af3183acdfd54c563f0; "ahci: SB600 > > workaround is suspect... play it safe for now") this did not > > solve the lockups. But there are references in the thread above > > to broken MSI on the SB600 possibly being the cause of the problem, > > and now that commit 22b5e7a74280deae560c20ee1a9b502b35181327 > > ("ahci: SB600 ahci can't do MSI, blacklist that > capability") has gone in > > maybe this is why you're able to get 64bit DMA working? (although > > we do need more hints on how to conclusively test this) > > Hmm... I doubt MSI has anything to do with 64bit DMA but then again > anything is possible. Shane, anything came out of the testing? SB600 255 sectors limit and SB600 64 bit DMA bug are two different bugs, the first one appears on many boards while the latter appears on some of them (our reference boards can NOT reproduce it). As to SB600 SATA MSI bug, since we found that there is such kind of bug, all our SB600 BIOS for reference boards disabled SATA MSI capability itself. (we assume all the SB600 BIOS in the marketing should disable MSI too, but, in fact, some did not as you know). So I'm not able to do the test so far due to the MSI disabled BIOS. If there is some guy whose SB600 BIOS is enabled, he may do some verification for us. http://marc.info/?l=linux-ide&m=117810966106759&w=2 I remember "pci=nomsi" did not work before the workaround of 32bit SB600 SATA DMA? Tejun, is that true? Anyway, I suggest we keep current SB600 workarounds. Thanks Shane ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-24 10:19 ` Huang, Shane @ 2008-09-25 3:24 ` George Spelvin 2008-09-26 14:06 ` Tejun Heo 2008-09-26 16:04 ` Grant Grundler 0 siblings, 2 replies; 16+ messages in thread From: George Spelvin @ 2008-09-25 3:24 UTC (permalink / raw) To: mdnelson8, htejun; +Cc: Shane.Huang, linux, linux-ide, jgarzik, grundler Just a little followup... I currently have the machine with 8 GB RAM, all 4 SB600 channels populated, and 64-bit DMA enabled, operating in production. No problems so far. Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using 64-bit DMA at all? > PCI: Using ACPI for IRQ routing > pci 0000:00:00.0: BAR 3: can't allocate resource > PCI-DMA: Disabling AGP. > PCI-DMA: aperture base @ 20000000 size 65536 KB > PCI-DMA: using GART IOMMU. > PCI-DMA: Reserving 64MB of IOMMU area in the AGP aperture > hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0 > hpet0: 4 32-bit timers, 14318180 Hz > ACPI: RTC can wake from S4 > Switched to high resolution mode on CPU 0 > Switched to high resolution mode on CPU 2 > Switched to high resolution mode on CPU 1 > Switched to high resolution mode on CPU 3 # lspci -s 00:12.0 -vvv -nn -xxx 00:12.0 SATA controller [0106]: ATI Technologies Inc SB600 Non-Raid-5 SATA [1002:4380] (prog-if 01) Subsystem: Micro-Star International Co., Ltd. Device [1462:7378] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 64, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 22 Region 0: I/O ports at 8000 [size=8] Region 1: I/O ports at 7000 [size=4] Region 2: I/O ports at 6000 [size=8] Region 3: I/O ports at 5000 [size=4] Region 4: I/O ports at 4000 [size=16] Region 5: Memory at fe7ff800 (32-bit, non-prefetchable) [size=1K] Capabilities: [60] Power Management version 2 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Kernel driver in use: ahci 00: 02 10 80 43 07 01 30 02 00 01 06 01 10 40 00 00 10: 01 80 00 00 01 70 00 00 01 60 00 00 01 50 00 00 20: 01 40 00 00 00 f8 7f fe 00 00 00 00 62 14 78 73 30: 00 00 00 00 60 00 00 00 00 00 00 00 0b 01 00 00 40: 10 00 80 02 01 00 10 00 01 00 00 00 00 00 00 00 50: 05 00 84 00 00 00 00 00 00 00 00 00 00 00 00 00 60: 01 00 22 00 00 00 00 00 00 00 00 00 00 00 00 00 70: 12 00 10 00 0f 00 00 00 00 00 00 00 00 00 00 00 80: 00 00 00 00 06 00 00 2c d6 01 b4 00 d6 01 b4 00 90: d6 01 b4 00 d6 01 b4 00 00 00 00 00 00 00 00 00 a0: 00 00 00 00 00 78 00 00 00 00 00 00 00 78 00 00 b0: 00 00 00 00 00 78 00 00 00 00 00 00 00 78 00 00 c0: 00 20 00 00 80 00 00 00 00 00 00 00 00 00 00 00 d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 # lspci -s 02:00.0 -vvv -nn -xxx 02:00.0 RAID bus controller [0104]: Promise Technology, Inc. Device [105a:3f20] Subsystem: Micro-Star International Co., Ltd. Device [1462:3716] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 17 Region 0: I/O ports at a800 [size=128] Region 2: I/O ports at a400 [size=256] Region 3: Memory at fe9ff000 (32-bit, non-prefetchable) [size=4K] Region 4: Memory at fe9c0000 (32-bit, non-prefetchable) [size=128K] Region 5: Memory at fe9fc000 (32-bit, non-prefetchable) [size=8K] Capabilities: [50] Power Management version 2 Flags: PMEClk- DSI+ D1+ D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 PME-Enable- DSel=0 DScale=0 PME- Capabilities: [70] Express (v1) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency L0 <64ns, L1 <1us ClockPM- Suprise- LLActRep- BwNot- LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- Capabilities: [94] SATA HBA <?> Capabilities: [100] Advanced Error Reporting <?> Capabilities: [140] Virtual Channel <?> Capabilities: [160] Device Serial Number 01-00-00-00-02-00-00-00 Capabilities: [170] Power Budgeting <?> Kernel driver in use: ahci 00: 5a 10 20 3f 07 01 10 00 00 00 04 01 10 00 00 00 10: 01 a8 00 00 00 00 00 00 01 a4 00 00 00 f0 9f fe 20: 00 00 9c fe 00 c0 9f fe 00 00 00 00 62 14 16 37 30: 00 00 00 00 50 00 00 00 00 00 00 00 0a 01 00 00 40: 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 50: 01 70 22 02 00 00 00 00 00 00 00 00 00 00 00 00 60: 05 70 80 00 00 00 fa df 5a 00 00 00 00 00 00 00 70: 10 94 11 00 01 00 00 00 10 28 0a 00 11 0c 00 00 80: 42 00 11 00 00 00 00 00 00 00 00 00 00 00 00 00 90: 00 00 00 00 12 00 10 00 46 02 00 00 09 00 4c 01 a0: 01 1c 00 00 fb ff ff 00 07 20 16 1b 38 81 0b 00 b0: de d0 0b 00 be 0d 00 00 02 28 11 00 00 00 00 00 c0: 00 02 00 00 03 00 03 00 00 00 00 00 10 00 20 09 d0: 0a 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e0: 00 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 dmesg excerpt: ahci 0000:00:12.0: version 3.0 ahci 0000:00:12.0: PCI INT A -> GSI 22 (level, low) -> IRQ 22 ahci 0000:00:12.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl SATA mode ahci 0000:00:12.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part scsi0 : ahci scsi1 : ahci scsi2 : ahci scsi3 : ahci ata1: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ff900 irq 22 ata2: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ff980 irq 22 ata3: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ffa00 irq 22 ata4: SATA max UDMA/133 abar m1024@0xfe7ff800 port 0xfe7ffa80 irq 22 ata1: softreset failed (device not ready) ata1: failed due to HW bug, retry pmp=0 ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata1.00: ATA-7: ST3400832AS, 3.01, max UDMA/133 ata1.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32) ata1.00: SB600 AHCI: limiting to 255 sectors per cmd ata1.00: SB600 AHCI: limiting to 255 sectors per cmd ata1.00: configured for UDMA/133 ata2: softreset failed (device not ready) ata2: failed due to HW bug, retry pmp=0 ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata2.00: ATA-7: ST3400832AS, 3.01, max UDMA/133 ata2.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32) ata2.00: SB600 AHCI: limiting to 255 sectors per cmd ata2.00: SB600 AHCI: limiting to 255 sectors per cmd ata2.00: configured for UDMA/133 ata3: softreset failed (device not ready) ata3: failed due to HW bug, retry pmp=0 ata3: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata3.00: ATA-7: ST3400832AS, 3.01, max UDMA/133 ata3.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32) ata3.00: SB600 AHCI: limiting to 255 sectors per cmd ata3.00: SB600 AHCI: limiting to 255 sectors per cmd ata3.00: configured for UDMA/133 ata4: softreset failed (device not ready) ata4: failed due to HW bug, retry pmp=0 ata4: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata4.00: ATA-7: ST3400832AS, 3.01, max UDMA/133 ata4.00: 781422768 sectors, multi 16: LBA48 NCQ (depth 31/32) ata4.00: SB600 AHCI: limiting to 255 sectors per cmd ata4.00: SB600 AHCI: limiting to 255 sectors per cmd ata4.00: configured for UDMA/133 scsi 0:0:0:0: Direct-Access ATA ST3400832AS 3.01 PQ: 0 ANSI: 5 sd 0:0:0:0: [sda] 781422768 512-byte hardware sectors (400088 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 0:0:0:0: [sda] 781422768 512-byte hardware sectors (400088 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 sda3 sda4 sd 0:0:0:0: [sda] Attached SCSI disk scsi 1:0:0:0: Direct-Access ATA ST3400832AS 3.01 PQ: 0 ANSI: 5 sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB) sd 1:0:0:0: [sdb] Write Protect is off sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 1:0:0:0: [sdb] 781422768 512-byte hardware sectors (400088 MB) sd 1:0:0:0: [sdb] Write Protect is off sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00 sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdb: sdb1 sdb2 sdb3 sdb4 sd 1:0:0:0: [sdb] Attached SCSI disk scsi 2:0:0:0: Direct-Access ATA ST3400832AS 3.01 PQ: 0 ANSI: 5 sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB) sd 2:0:0:0: [sdc] Write Protect is off sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 2:0:0:0: [sdc] 781422768 512-byte hardware sectors (400088 MB) sd 2:0:0:0: [sdc] Write Protect is off sd 2:0:0:0: [sdc] Mode Sense: 00 3a 00 00 sd 2:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdc: sdc1 sdc2 sdc3 sdc4 sd 2:0:0:0: [sdc] Attached SCSI disk scsi 3:0:0:0: Direct-Access ATA ST3400832AS 3.01 PQ: 0 ANSI: 5 sd 3:0:0:0: [sdd] 781422768 512-byte hardware sectors (400088 MB) sd 3:0:0:0: [sdd] Write Protect is off sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00 sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 3:0:0:0: [sdd] 781422768 512-byte hardware sectors (400088 MB) sd 3:0:0:0: [sdd] Write Protect is off sd 3:0:0:0: [sdd] Mode Sense: 00 3a 00 00 sd 3:0:0:0: [sdd] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdd: sdd1 sdd2 sdd3 sdd4 sd 3:0:0:0: [sdd] Attached SCSI disk ahci 0000:02:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17 ahci 0000:02:00.0: AHCI 0001.0100 32 slots 4 ports 3 Gbps 0xf impl RAID mode ahci 0000:02:00.0: flags: 64bit ncq sntf ilck pm led clo pmp pio slum part ahci 0000:02:00.0: setting latency timer to 64 scsi4 : ahci scsi5 : ahci scsi6 : ahci scsi7 : ahci ata5: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc100 irq 17 ata6: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc180 irq 17 ata7: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc200 irq 17 ata8: SATA max UDMA/133 abar m8192@0xfe9fc000 port 0xfe9fc280 irq 17 ata5: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata5.00: ATA-7: ST3400832AS, 3.03, max UDMA/133 ata5.00: 781422768 sectors, multi 0: LBA48 NCQ (depth 31/32) ata5.00: configured for UDMA/133 ata6: SATA link up 1.5 Gbps (SStatus 113 SControl 300) ata6.00: ATA-7: ST3400832AS, 3.01, max UDMA/133 ata6.00: 781422768 sectors, multi 0: LBA48 NCQ (depth 31/32) ata6.00: configured for UDMA/133 ata7: SATA link down (SStatus 0 SControl 300) ata8: SATA link down (SStatus 0 SControl 300) scsi 4:0:0:0: Direct-Access ATA ST3400832AS 3.03 PQ: 0 ANSI: 5 sd 4:0:0:0: [sde] 781422768 512-byte hardware sectors (400088 MB) sd 4:0:0:0: [sde] Write Protect is off sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00 sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 4:0:0:0: [sde] 781422768 512-byte hardware sectors (400088 MB) sd 4:0:0:0: [sde] Write Protect is off sd 4:0:0:0: [sde] Mode Sense: 00 3a 00 00 sd 4:0:0:0: [sde] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sde: sde1 sde2 sde3 sde4 sd 4:0:0:0: [sde] Attached SCSI disk scsi 5:0:0:0: Direct-Access ATA ST3400832AS 3.01 PQ: 0 ANSI: 5 sd 5:0:0:0: [sdf] 781422768 512-byte hardware sectors (400088 MB) sd 5:0:0:0: [sdf] Write Protect is off sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 5:0:0:0: [sdf] 781422768 512-byte hardware sectors (400088 MB) sd 5:0:0:0: [sdf] Write Protect is off sd 5:0:0:0: [sdf] Mode Sense: 00 3a 00 00 sd 5:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sdf: sdf1 sdf2 sdf3 sdf4 sd 5:0:0:0: [sdf] Attached SCSI disk ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-25 3:24 ` George Spelvin @ 2008-09-26 14:06 ` Tejun Heo 2008-09-28 21:04 ` George Spelvin 2008-09-26 16:04 ` Grant Grundler 1 sibling, 1 reply; 16+ messages in thread From: Tejun Heo @ 2008-09-26 14:06 UTC (permalink / raw) To: George Spelvin; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler George Spelvin wrote: > Just a little followup... I currently have the machine with 8 GB RAM, > all 4 SB600 channels populated, and 64-bit DMA enabled, operating in > production. No problems so far. > > Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using > 64-bit DMA at all? Can you turn it off and see what happens? -- tejun ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-26 14:06 ` Tejun Heo @ 2008-09-28 21:04 ` George Spelvin 2008-09-28 23:20 ` Tejun Heo 0 siblings, 1 reply; 16+ messages in thread From: George Spelvin @ 2008-09-28 21:04 UTC (permalink / raw) To: linux, htejun; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler Tejun Heo <htejun@gmail.com> wrote: >> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using >> using 64-bit DMA at all? > > Can you turn it off and see what happens? I can if I can verify that every other bus-master PCI device in the system can do 64-bit DMA. Do you know how to check that? I haven't found a file under /sys or /proc that tells me what the DMA masks are of various system drivers, but that doesn't mean that it doesn't exist. Thanks. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-28 21:04 ` George Spelvin @ 2008-09-28 23:20 ` Tejun Heo 2008-09-29 4:08 ` George Spelvin 0 siblings, 1 reply; 16+ messages in thread From: Tejun Heo @ 2008-09-28 23:20 UTC (permalink / raw) To: George Spelvin; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler George Spelvin wrote: > Tejun Heo <htejun@gmail.com> wrote: >>> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using >>> using 64-bit DMA at all? >> Can you turn it off and see what happens? > > I can if I can verify that every other bus-master PCI device in the > system can do 64-bit DMA. > > Do you know how to check that? I haven't found a file under /sys or > /proc that tells me what the DMA masks are of various system > drivers, but that doesn't mean that it doesn't exist. It doesn't really matter. The kernel will bounce buffers as necessary. -- tejun ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-28 23:20 ` Tejun Heo @ 2008-09-29 4:08 ` George Spelvin 2008-09-29 4:25 ` Tejun Heo 0 siblings, 1 reply; 16+ messages in thread From: George Spelvin @ 2008-09-29 4:08 UTC (permalink / raw) To: linux, htejun; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler Tejun Heo <htejun@gmail.com> wrote: > George Spelvin wrote: >> Tejun Heo <htejun@gmail.com> wrote: >>>> Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using >>>> using 64-bit DMA at all? >>> Can you turn it off and see what happens? >> >> I can if I can verify that every other bus-master PCI device in the >> system can do 64-bit DMA. >> >> Do you know how to check that? I haven't found a file under /sys or >> /proc that tells me what the DMA masks are of various system >> drivers, but that doesn't mean that it doesn't exist. > > It doesn't really matter. The kernel will bounce buffers as > necessary. Er, according to the docs I can find, that's iommu=soft, and still doesn't guarantee 64-bit DMA. I was thinking about using iommu=off, which disables everything including bounce buffers, and thereby forces 64-bit DMA. Um... actually, I just tried it, and when there's >4G of memory, the kernel forces iommu=soft, and it works. At least as far as e2fsck -n on every file system (including a 1.7 TB RAID-5) can tell. It still doesn't definitively tell me that a DMA to and from an address >4G was done, but it's an awfully strong hint. I was thinking of inserting a debug message that logs every time the "highest DMA address used" increases, than I can watch it to see for sure. ATM I'm just recompiling the kernel from -rc6 to -rc7. It's in single-user mode so I can't ssh over the boot messages, but they definitely included "64bit" from ahci_print_info(). Speaking of anci_print_info(), sould something like the following be a good thing? diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c index 2e1a7cb..18137d2 100644 --- a/drivers/ata/ahci.c +++ b/drivers/ata/ahci.c @@ -2464,22 +2464,22 @@ static void ahci_print_info(struct ata_host *host) "%s\n" , - cap & (1 << 31) ? "64bit " : "", - cap & (1 << 30) ? "ncq " : "", - cap & (1 << 29) ? "sntf " : "", - cap & (1 << 28) ? "ilck " : "", - cap & (1 << 27) ? "stag " : "", - cap & (1 << 26) ? "pm " : "", - cap & (1 << 25) ? "led " : "", - - cap & (1 << 24) ? "clo " : "", - cap & (1 << 19) ? "nz " : "", - cap & (1 << 18) ? "only " : "", - cap & (1 << 17) ? "pmp " : "", - cap & (1 << 15) ? "pio " : "", - cap & (1 << 14) ? "slum " : "", - cap & (1 << 13) ? "part " : "", - cap & (1 << 6) ? "ems ": "" + cap & HOST_CAP_64 ? "64bit " : "", + cap & HOST_CAP_NCQ ? "ncq " : "", + cap & HOST_CAP_SNTF ? "sntf " : "", + cap & (1 << 28) ? "ilck " : "", + cap & HOST_CAP_SSS ? "stag " : "", + cap & HOST_CAP_ALPM ? "pm " : "", + cap & (1 << 25) ? "led " : "", + + cap & HOST_CAP_CLO ? "clo " : "", + cap & (1 << 19) ? "nz " : "", + cap & (1 << 18) ? "only " : "", + cap & HOST_CAP_PMP ? "pmp " : "", + cap & (1 << 15) ? "pio " : "", + cap & HOST_CAP_SSC ? "slum " : "", + cap & (1 << 13) ? "part " : "", + cap & HOST_CAP_EMS ? "ems ": "" ); } ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-29 4:08 ` George Spelvin @ 2008-09-29 4:25 ` Tejun Heo 2008-09-30 2:00 ` George Spelvin 0 siblings, 1 reply; 16+ messages in thread From: Tejun Heo @ 2008-09-29 4:25 UTC (permalink / raw) To: George Spelvin; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler George Spelvin wrote: > ATM I'm just recompiling the kernel from -rc6 to -rc7. > > It's in single-user mode so I can't ssh over the boot messages, but > they definitely included "64bit" from ahci_print_info(). Thanks for verifying. I would love to lift the 32 restriction from SB600 but it's been quite a rocky ride and I still can't tell what's the determining factor. :-( > Speaking of anci_print_info(), sould something like the following be a good > thing? > > diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c > index 2e1a7cb..18137d2 100644 > --- a/drivers/ata/ahci.c > +++ b/drivers/ata/ahci.c > @@ -2464,22 +2464,22 @@ static void ahci_print_info(struct ata_host *host) > "%s\n" > , > > - cap & (1 << 31) ? "64bit " : "", > - cap & (1 << 30) ? "ncq " : "", > - cap & (1 << 29) ? "sntf " : "", > - cap & (1 << 28) ? "ilck " : "", > - cap & (1 << 27) ? "stag " : "", > - cap & (1 << 26) ? "pm " : "", > - cap & (1 << 25) ? "led " : "", > - > - cap & (1 << 24) ? "clo " : "", > - cap & (1 << 19) ? "nz " : "", > - cap & (1 << 18) ? "only " : "", > - cap & (1 << 17) ? "pmp " : "", > - cap & (1 << 15) ? "pio " : "", > - cap & (1 << 14) ? "slum " : "", > - cap & (1 << 13) ? "part " : "", > - cap & (1 << 6) ? "ems ": "" > + cap & HOST_CAP_64 ? "64bit " : "", > + cap & HOST_CAP_NCQ ? "ncq " : "", > + cap & HOST_CAP_SNTF ? "sntf " : "", > + cap & (1 << 28) ? "ilck " : "", > + cap & HOST_CAP_SSS ? "stag " : "", > + cap & HOST_CAP_ALPM ? "pm " : "", > + cap & (1 << 25) ? "led " : "", > + > + cap & HOST_CAP_CLO ? "clo " : "", > + cap & (1 << 19) ? "nz " : "", > + cap & (1 << 18) ? "only " : "", > + cap & HOST_CAP_PMP ? "pmp " : "", > + cap & (1 << 15) ? "pio " : "", > + cap & HOST_CAP_SSC ? "slum " : "", > + cap & (1 << 13) ? "part " : "", > + cap & HOST_CAP_EMS ? "ems ": "" Yeah, probably. While at it, can you just add constants for other bits too and send it as properly singed off patch? Thanks. -- tejun ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-29 4:25 ` Tejun Heo @ 2008-09-30 2:00 ` George Spelvin 2008-10-01 3:57 ` Tejun Heo 0 siblings, 1 reply; 16+ messages in thread From: George Spelvin @ 2008-09-30 2:00 UTC (permalink / raw) To: linux, htejun; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler Tejun Heo <htejun@gmail.com> wrote: > George Spelvin wrote: >> ATM I'm just recompiling the kernel from -rc6 to -rc7. >> >> It's in single-user mode so I can't ssh over the boot messages, but >> they definitely included "64bit" from ahci_print_info(). > > Thanks for verifying. I would love to lift the 32 restriction from > SB600 but it's been quite a rocky ride and I still can't tell what's > the determining factor. :-( H'm.... 64-bit support comes through in multiple places. I wonder if it's just one of them? - HBA memory-mapped registers (which are in fact limited to 32 bits) - Port x command list base address - Port x FIS base address - Command table base address (command header, offset 8) - Data base address (physical region descriptor, offset 0) What we really care about is that the data physical regions can be anywhere in 64-bit space. If any of the others were limited to 32 bits, that would only be slightly annoying. But... The base addresses are already above 4G. Aobve 8G, actually. Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400 Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400 Port 2 base addresses: 02 1f98 0000, 02 1f98 0400 Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400 /proc/iomem looks like: 00000000-0009fbff : System RAM 0009fc00-0009ffff : reserved 000c0000-000cffff : pnp 00:0e 000e4000-000fffff : reserved 00100000-dffbffff : System RAM 00200000-005187db : Kernel code 005187dc-006c310f : Kernel data 0073f000-007bb4cf : Kernel bss 20000000-23ffffff : GART dffc0000-dffcdfff : ACPI Tables dffce000-dffeffff : ACPI Non-volatile Storage dfff0000-dfffdfff : reserved e0000000-efffffff : PCI MMCONFIG 0 fc000000-fdffffff : PCI Bus 0000:03 fc000000-fdffffff : 0000:03:00.0 fe7f4000-fe7f7fff : 0000:00:14.2 fe7fa000-fe7fafff : 0000:00:13.4 fe7fa000-fe7fafff : ohci_hcd fe7fb000-fe7fbfff : 0000:00:13.3 fe7fb000-fe7fbfff : ohci_hcd fe7fc000-fe7fcfff : 0000:00:13.2 fe7fc000-fe7fcfff : ohci_hcd fe7fd000-fe7fdfff : 0000:00:13.1 fe7fd000-fe7fdfff : ohci_hcd fe7fe000-fe7fefff : 0000:00:13.0 fe7fe000-fe7fefff : ohci_hcd fe7ff000-fe7ff0ff : 0000:00:13.5 fe7ff000-fe7ff0ff : ehci_hcd fe7ff800-fe7ffbff : 0000:00:12.0 fe7ff800-fe7ffbff : ahci fe800000-fe8fffff : PCI Bus 0000:01 fe8c0000-fe8dffff : 0000:01:00.0 fe8ff000-fe8fffff : 0000:01:00.0 fe8ff000-fe8fffff : r8169 fe900000-fe9fffff : PCI Bus 0000:02 fe9c0000-fe9dffff : 0000:02:00.0 fe9c0000-fe9dffff : ahci fe9fc000-fe9fdfff : 0000:02:00.0 fe9fc000-fe9fdfff : ahci fe9ff000-fe9fffff : 0000:02:00.0 fe9ff000-fe9fffff : ahci fea00000-feafffff : PCI Bus 0000:03 feac0000-feadffff : 0000:03:00.0 feae0000-feaeffff : 0000:03:00.1 feaf0000-feafffff : 0000:03:00.0 feb00000-febfffff : PCI Bus 0000:04 febff800-febfffff : 0000:04:00.0 fec00000-fec00fff : IOAPIC 0 fed00000-fed003ff : HPET 2 fee00000-fee00fff : Local APIC fff00000-ffffffff : reserved 100000000-21fffffff : System RAM ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-30 2:00 ` George Spelvin @ 2008-10-01 3:57 ` Tejun Heo 2008-10-01 13:26 ` George Spelvin 0 siblings, 1 reply; 16+ messages in thread From: Tejun Heo @ 2008-10-01 3:57 UTC (permalink / raw) To: George Spelvin; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler George Spelvin wrote: > Tejun Heo <htejun@gmail.com> wrote: > >> George Spelvin wrote: >>> ATM I'm just recompiling the kernel from -rc6 to -rc7. >>> >>> It's in single-user mode so I can't ssh over the boot messages, but >>> they definitely included "64bit" from ahci_print_info(). >> Thanks for verifying. I would love to lift the 32 restriction from >> SB600 but it's been quite a rocky ride and I still can't tell what's >> the determining factor. :-( > > H'm.... 64-bit support comes through in multiple places. > I wonder if it's just one of them? > - HBA memory-mapped registers (which are in fact limited to 32 bits) This is determined by the BIOS and I don't think it ever puts it over 32bit boundary. > - Port x command list base address > - Port x FIS base address > - Command table base address (command header, offset 8) > - Data base address (physical region descriptor, offset 0) The first three are controlled by consistent_dma_mask while the last one is controlled by dma_mask, so maybe setting only consistent_dma_mask to 32bit could help. The problem is how to verify them. You'll need to dig the list and bugzilla to look for reporters who reported failure on 64bit DMA and verify it actually fixes the problem. It can be tricky as the problem doesn't seem to appear for all users. > What we really care about is that the data physical regions can > be anywhere in 64-bit space. If any of the others were limited to > 32 bits, that would only be slightly annoying. > > But... > The base addresses are already above 4G. Aobve 8G, actually. > > Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400 > Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400 > Port 2 base addresses: 02 1f98 0000, 02 1f98 0400 > Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400 Where are these numbers from? -- tejun ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-10-01 3:57 ` Tejun Heo @ 2008-10-01 13:26 ` George Spelvin 0 siblings, 0 replies; 16+ messages in thread From: George Spelvin @ 2008-10-01 13:26 UTC (permalink / raw) To: linux, htejun; +Cc: Shane.Huang, mdnelson8, linux-ide, jgarzik, grundler Tejun Heo <htejun@gmail.com> wrote: > George Spelvin wrote: >> But... >> The base addresses are already above 4G. Aobve 8G, actually. >> >> Port 0 base addresses: 02 1f1e 0000, 02 1f1e 0400 >> Port 1 base addresses: 02 1f1a 0000, 02 1f1a 0400 >> Port 2 base addresses: 02 1f98 0000, 02 1f98 0400 >> Port 3 base addresses: 02 1f9e 0000, 02 1f9e 0400 > > Where are these numbers from? A dump of the appropriate region of memory (1k @ 0xfe7ff800), offsets 0x100, 0x180, 0x200 and 0x280. Here's a full dump (rebooted in between, so the addresses might change). dd if=/dev/mem bs=1024 skip=4169726 count=1 | od -Ax -tx4 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000203726 s, 5.0 MB/s 000000 f722ff83 80000002 00000000 0000000f 000010 00010100 000101f8 00000000 00000000 000020 00000000 00000000 00000000 00000000 * 0000f0 00000000 00000000 00000000 03ef99fc 000100 1f1e0000 00000002 1f1e0400 00000002 000110 00000000 7840007f 00042017 00000000 000120 00000040 00000101 00000113 00000300 000130 00000000 00000000 00000000 00000000 000140 0001000f 00000000 00000000 00000000 000150 00000000 00000000 00000000 00000000 * 000170 00000000 00000000 00000000 08001c00 000180 1f1a0000 00000002 1f1a0400 00000002 000190 00000000 7840007f 0004a017 00000000 0001a0 00000040 00000101 00000113 00000300 0001b0 00000000 00000001 00000000 00000000 0001c0 0001000f 00000000 00000000 00000000 0001d0 00000000 00000000 00000000 00000000 * 0001f0 00000000 00000000 00000000 08001c00 000200 1f980000 00000002 1f980400 00000002 000210 00000000 7840007f 00042017 00000000 000220 00000040 00000101 00000113 00000300 000230 00000000 00000000 00000000 00000000 000240 0001000f 00000000 00000000 00000000 000250 00000000 00000000 00000000 00000000 * 000270 00000000 00000000 00000000 08001c00 000280 1f9e0000 00000002 1f9e0400 00000002 000290 00000000 7840007f 00042017 00000000 0002a0 00000040 00000101 00000113 00000300 0002b0 00000000 00000000 00000000 00000000 0002c0 0001000f 00000000 00000000 00000000 0002d0 00000000 00000000 00000000 00000000 * 0002f0 00000000 00000000 00000000 08001c00 000300 00000000 00000000 00000000 00000000 * 000400 ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: What's the SB600 64-bit DMA problem? 2008-09-25 3:24 ` George Spelvin 2008-09-26 14:06 ` Tejun Heo @ 2008-09-26 16:04 ` Grant Grundler 1 sibling, 0 replies; 16+ messages in thread From: Grant Grundler @ 2008-09-26 16:04 UTC (permalink / raw) To: George Spelvin; +Cc: Shane.Huang, mdnelson8, htejun, linux-ide, jgarzik On Wed, Sep 24, 2008 at 8:24 PM, George Spelvin <linux@horizon.com> wrote: > Just a little followup... I currently have the machine with 8 GB RAM, > all 4 SB600 channels populated, and 64-bit DMA enabled, operating in > production. No problems so far. > > Q: Does the fact that I'm using the GART IOMMU mean that I'm not really using > 64-bit DMA at all? If pci_dev->dma_mask has more bits set than the highest physical memory address, then the GART will not be used for that device. See "need_iommu()" in arch/x86/kernel/pci-gart_64.c and how that's used in gart_map_single(). My assumption is libata will not override something the PCI device driver sets. hth, grant ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2008-10-01 15:13 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-09-20 15:17 What's the SB600 64-bit DMA problem? George Spelvin 2008-09-20 22:28 ` Grant Grundler 2008-09-21 0:15 ` George Spelvin 2008-09-24 0:30 ` Mark Nelson 2008-09-24 9:46 ` Tejun Heo 2008-09-24 10:19 ` Huang, Shane 2008-09-25 3:24 ` George Spelvin 2008-09-26 14:06 ` Tejun Heo 2008-09-28 21:04 ` George Spelvin 2008-09-28 23:20 ` Tejun Heo 2008-09-29 4:08 ` George Spelvin 2008-09-29 4:25 ` Tejun Heo 2008-09-30 2:00 ` George Spelvin 2008-10-01 3:57 ` Tejun Heo 2008-10-01 13:26 ` George Spelvin 2008-09-26 16:04 ` Grant Grundler
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.