BCM91x80A/B PCI DMA problems

BCM91x80A/B PCI DMA problems

[Martin Michlmayr]

Has anyone here successfully used a PCI IDE card on BCM91x80?
I immediately get lots of PCI DMA problems, e.g:


SiI680: IDE controller at PCI slot 0000:00:01.0
SiI680: chipset revision 2
SiI680: BASE CLOCK == 133
SiI680: 100% native mode on irq 8
    ide0: MMIO-DMA , BIOS settings: hda:pio, hdb:pio
    ide1: MMIO-DMA , BIOS settings: hdc:pio, hdd:pio
hda: FUJITSU MPB3043ATU E, ATA DISK drive
isa bounce pool size: 16 pages
ide0 at 0x9000000031080080-0x9000000031080087,0x900000003108008a on irq 8
hda: max request size: 64KiB
hda: 8448300 sectors (4325 MB), CHS=8940/15/63, UDMA(33)
 hda:<4>hda: dma_timer_expiry: dma status == 0x22
hda: DMA timeout error
hda: dma timeout error: status=0x01 { Error }
hda: dma timeout error: error=0x7f { DriveStatusError UncorrectableError SectorIdNotFound TrackZeroNotFound AddrMarkNotFound }, LBAsect=260013951, sector=0
ide: failed opcode was: unknown
end_request: I/O error, dev hda, sector 0

-- 
Martin Michlmayr
http://www.cyrius.com/

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello,

I have one in the slot closest to the CPU.  All of them are supposed to
work, but they're off of different controllers.

[4294667.639000] Uniform Multi-Platform E-IDE driver Revision:
7.00alpha2
[4294667.640000] ide: Assuming 33MHz system bus speed for PIO modes;
override with idebus=xx
[4294667.641000] CMD649: IDE controller at PCI slot 0000:00:01.0
[4294667.642000] CMD649: chipset revision 2
[4294667.643000] CMD649: 100% native mode on irq 8
[4294667.644000]     ide0: BM-DMA at 0x8000-0x8007, BIOS settings:
hda:pio, hdb:pio
[4294667.646000]     ide1: BM-DMA at 0x8008-0x800f, BIOS settings:
hdc:pio, hdd:pio
[4294667.912000] hda: WDC AC35100L, ATA DISK drive
[4294668.526000] ide0 at 0x8010-0x8017,0x8022 on irq 8
[4294669.566000] hda: max request size: 128KiB
[4294669.567000] hda: 10085040 sectors (5163 MB) w/256KiB Cache,
CHS=10672/15/63[4294669.570000]  hda: hda1 hda2 < hda5 >

Is this a 32-bit, or 64-bit kernel?  If 64-bit, do you have more than 1G
of DRAM installed in the system (DRAM above 1G is accessed at >32-bit
physical addresses).

Are you using CFE 1.2.5?  The PCI interrupt assignments in earlier
versions of CFE were not correct.

Thanks,
Mark

> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org 
> [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Martin 
> Michlmayr
> Sent: Tuesday, February 28, 2006 1:47 PM
> To: linux-mips@linux-mips.org
> Subject: BCM91x80A/B PCI DMA problems
> 
> Has anyone here successfully used a PCI IDE card on BCM91x80?
> I immediately get lots of PCI DMA problems, e.g:
> 
> 
> SiI680: IDE controller at PCI slot 0000:00:01.0
> SiI680: chipset revision 2
> SiI680: BASE CLOCK == 133
> SiI680: 100% native mode on irq 8
>     ide0: MMIO-DMA , BIOS settings: hda:pio, hdb:pio
>     ide1: MMIO-DMA , BIOS settings: hdc:pio, hdd:pio
> hda: FUJITSU MPB3043ATU E, ATA DISK drive isa bounce pool 
> size: 16 pages ide0 at 
> 0x9000000031080080-0x9000000031080087,0x900000003108008a on irq 8
> hda: max request size: 64KiB
> hda: 8448300 sectors (4325 MB), CHS=8940/15/63, UDMA(33)
>  hda:<4>hda: dma_timer_expiry: dma status == 0x22
> hda: DMA timeout error
> hda: dma timeout error: status=0x01 { Error }
> hda: dma timeout error: error=0x7f { DriveStatusError 
> UncorrectableError SectorIdNotFound TrackZeroNotFound 
> AddrMarkNotFound }, LBAsect=260013951, sector=0
> ide: failed opcode was: unknown
> end_request: I/O error, dev hda, sector 0
> 
> --
> Martin Michlmayr
> http://www.cyrius.com/
> 
> 
> 

Re: BCM91x80A/B PCI DMA problems

[Martin Michlmayr]

* Mark E Mason <mark.e.mason@broadcom.com> [2006-02-28 13:53]:
> Is this a 32-bit, or 64-bit kernel?  If 64-bit, do you have more than 1G
> of DRAM installed in the system (DRAM above 1G is accessed at >32-bit
> physical addresses).

Erm, I have 2 GB of RAM and used a 64-bit kernel.  Using a 32-bit
kernel works.

> Are you using CFE 1.2.5?  The PCI interrupt assignments in earlier

Yes.

> versions of CFE were not correct.
-- 
Martin Michlmayr
http://www.cyrius.com/

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello,

Is this driver known to work with 64-bit kernels (specifically: 64-bit
DMA addresses)?  It sounds like that might be the problem.

HTH,
Mark

> -----Original Message-----
> From: Martin Michlmayr [mailto:tbm@cyrius.com] 
> Sent: Tuesday, February 28, 2006 2:13 PM
> To: Mark E Mason
> Cc: linux-mips@linux-mips.org
> Subject: Re: BCM91x80A/B PCI DMA problems
> 
> * Mark E Mason <mark.e.mason@broadcom.com> [2006-02-28 13:53]:
> > Is this a 32-bit, or 64-bit kernel?  If 64-bit, do you have 
> more than 
> > 1G of DRAM installed in the system (DRAM above 1G is accessed at 
> > >32-bit physical addresses).
> 
> Erm, I have 2 GB of RAM and used a 64-bit kernel.  Using a 
> 32-bit kernel works.
> 
> > Are you using CFE 1.2.5?  The PCI interrupt assignments in earlier
> 
> Yes.
> 
> > versions of CFE were not correct.
> --
> Martin Michlmayr
> http://www.cyrius.com/
> 
> 

RE: BCM91x80A/B PCI DMA problems

[Alan Cox]

On Maw, 2006-02-28 at 14:15 -0800, Mark E Mason wrote:
> Hello,
> 
> Is this driver known to work with 64-bit kernels (specifically: 64-bit
> DMA addresses)?  It sounds like that might be the problem.

The standard ATA IDE hardware supports only 32bit addressing. However if
your I/O mapping logic is correctly implemented for the architecture
that should cause no problems as the buffers will be bounced.

The SIL680 hardware actually can support 64bit DMA using a private
non-standard PRD format and the data sheet is available if someone wants
to do the work. Probably best to do it for the new libata driver but its
doable for either

Re: BCM91x80A/B PCI DMA problems

[Martin Michlmayr]

* Alan Cox <alan@lxorguk.ukuu.org.uk> [2006-02-28 23:06]:
> > Is this driver known to work with 64-bit kernels (specifically: 64-bit
> > DMA addresses)?  It sounds like that might be the problem.
> 
> The standard ATA IDE hardware supports only 32bit addressing. However if
> your I/O mapping logic is correctly implemented for the architecture
> that should cause no problems as the buffers will be bounced.

Something is definitely broken in the implemention of this
architecture.  With a 64-bit kernel, my SII PCI card shows DMA errors.
With a 32-bit kernel I get some other weird errors.  What I managed to
do is to add the SII device ID to the generic IDE support.  This works
but is obviously quite slow...

> The SIL680 hardware actually can support 64bit DMA using a private
> non-standard PRD format and the data sheet is available if someone
> wants to do the work. Probably best to do it for the new libata
> driver but its doable for either

I just tried the SCSI PATA patch to see if that works.  Unfortunately,
it doesn't.  As I said above, something must be broken in the
architecture code of SB1.  However, Alan, I think you might be
interested in the oops I get anyway because I suspect you may want to
fail in a more gracious way.

BTW, does anyone have the knowledge and time to see what's broken in
the SB1 code?  I really need something better than generic IDE
support.  It's not fun having a quad-core CPU when your box has to
wait for IO all the time. :/

Alan, for you:

[4294667.296000] Linux version 2.6.16-rc6 (Debian 2.6.15+2.6.16-rc6-0experimental.1) (waldi@debian.org) (gcc version 4.1.0 (Debian 4.1.0-0)) #7 SMP Mon Mar 13 20:45:43 UTC 2006
[4294667.296000] CPU revision is: 01041100
[4294667.296000] FPU revision is: 000f0103
[4294667.296000] swarm setup: M41T81 RTC detected.
[4294667.296000] This kernel optimized for board runs with CFE
[4294667.296000] Determined physical RAM map:
[4294667.296000]  memory: 000000000fe91e00 @ 0000000000000000 (usable)
[4294667.296000]  memory: 000000001ffffe00 @ 0000000080000000 (usable)
[4294667.296000]  memory: 000000000ffffe00 @ 00000000c0000000 (usable)
[4294667.296000]  memory: 000000003ffffe00 @ 0000000140000000 (usable)
[4294667.296000] Detected 3 available secondary CPU(s)
[4294667.296000] Built 1 zonelists
[4294667.296000] Kernel command line: root=/dev/hda2
[4294667.296000] Primary instruction cache 32kB, 4-way, linesize 32 bytes.
[4294667.296000] Primary data cache 32kB, 4-way, linesize 32 bytes.
[4294667.296000] Synthesized TLB refill handler (35 instructions).
[4294667.296000] Synthesized TLB load handler fastpath (49 instructions).
[4294667.296000] Synthesized TLB store handler fastpath (49 instructions).
[4294667.296000] Synthesized TLB modify handler fastpath (48 instructions).
[4294667.296000] PID hash table entries: 4096 (order: 12, 131072 bytes)
[4294667.313000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[4294667.340000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
[4294667.653000] Memory: 1991912k/2095672k available (2374k kernel code, 103144k reserved, 643k data, 208k init, 0k highmem)
[4294667.674000] Mount-cache hash table entries: 256
[4294667.676000] Checking for 'wait' instruction...  unavailable.
[4294667.678000] Checking for the multiply/shift bug... no.
[4294667.680000] Checking for the daddi bug... no.
[4294667.681000] Checking for the daddiu bug... no.
[4294667.683000] CPU revision is: 03041100
[4294667.683000] FPU revision is: 000f0103
[4294667.683000] Primary instruction cache 32kB, 4-way, linesize 32 bytes.
[4294667.683000] Primary data cache 32kB, 4-way, linesize 32 bytes.
[4294667.683000] Synthesized TLB refill handler (35 instructions).
[4294667.703000] CPU revision is: 05041100
[4294667.703000] FPU revision is: 000f0103
[4294667.703000] Primary instruction cache 32kB, 4-way, linesize 32 bytes.
[4294667.703000] Primary data cache 32kB, 4-way, linesize 32 bytes.
[4294667.703000] Synthesized TLB refill handler (35 instructions).
[4294667.723000] CPU revision is: 07041100
[4294667.723000] FPU revision is: 000f0103
[4294667.723000] Primary instruction cache 32kB, 4-way, linesize 32 bytes.
[4294667.723000] Primary data cache 32kB, 4-way, linesize 32 bytes.
[4294667.723000] Synthesized TLB refill handler (35 instructions).
[4294667.743000] Brought up 4 CPUs
[4294667.757000] migration_cost=1000
[4294667.762000] NET: Registered protocol family 16
[4294667.765000] SCSI subsystem initialized
[4294667.769000] Initializing Cryptographic API
[4294667.770000] io scheduler noop registered
[4294667.771000] io scheduler anticipatory registered (default)
[4294667.773000] io scheduler deadline registered
[4294667.774000] io scheduler cfq registered
[4294667.787000] Generic RTC Driver v1.07
[4294667.788000] eth0: enabling TCP rcv checksum
[4294667.789000] eth0: SiByte Ethernet at 0x10064000, address: 00:02:4C:F5:2C:3C
[4294667.790000] eth1: enabling TCP rcv checksum
[4294667.791000] eth1: SiByte Ethernet at 0x10065000, address: 00:02:4C:F5:2C:3D
[4294667.792000] eth2: enabling TCP rcv checksum
[4294667.793000] eth2: SiByte Ethernet at 0x10066000, address: 00:02:4C:F5:2C:3E
[4294667.794000] eth3: enabling TCP rcv checksum
[4294667.795000] eth3: SiByte Ethernet at 0x10067000, address: 00:02:4C:F5:2C:3F
[4294667.797000] sil680: BA5_EN = 1 clock = 00
[4294667.798000] sil680: BA5_EN = 1 clock = 10
[4294667.799000] sil680: 133MHz clock.
[4294667.800000] ata1: PATA max UDMA/133 cmd 0x8010 ctl 0x8022 bmdma 0x8000 irq 8
[4294667.802000] ata2: PATA max UDMA/133 cmd 0x8018 ctl 0x8026 bmdma 0x8008 irq 8
[4294667.957000] ata1: dev 0 ATA-6, max UDMA/33, 19932192 sectors: LBA
[4294667.958000] ata1: dev 0 configured for UDMA/33
[4294667.959000] scsi0 : pata_sil680
[4294668.114000] ata2: dev 0 ATA-6, max UDMA/66, 19932192 sectors: LBA
[4294668.115000] ata2: dev 0 configured for UDMA/33
[4294668.116000] scsi1 : pata_sil680
[4294668.117000]   Vendor: ATA       Model: SAMSUNG SV1021H   Rev: PJ10
[4294668.122000]   Type:   Direct-Access                      ANSI SCSI revision: 05
[4294668.124000]   Vendor: ATA       Model: SAMSUNG SV1021H   Rev: PJ10
[4294668.129000]   Type:   Direct-Access                      ANSI SCSI revision: 05
[4294668.132000] SCSI device sda: 19932192 512-byte hdwr sectors (10205 MB)
[4294668.133000] sda: Write Protect is off
[4294668.134000] SCSI device sda: drive cache: write back
[4294668.135000] SCSI device sda: 19932192 512-byte hdwr sectors (10205 MB)
[4294668.136000] sda: Write Protect is off
[4294668.137000] SCSI device sda: drive cache: write back
[4294668.138000]  sda:DBE physical address: 002c008020
[4294698.139000] Data bus error, epc == ffffffff80293a48, ra == ffffffff8029b4d4
[4294698.139000] Oops[#1]:
[4294698.139000] Cpu 0
[4294698.139000] $ 0   : 0000000000000000 0000000014001fe0 00000000000000ff 900000002c008022
[4294698.139000] $ 4   : 900000002c000000 a80000017ff63458 ffffffff80360000 ffffffffffff00fe
[4294698.139000] $ 8   : 0000000000000001 ffffffffffffffc0 ffffffff80360000 ffffffff80360000
[4294698.139000] $12   : ffffffff80430000 0000000000001f00 0000000000000000 ffffffff803a0000
[4294698.139000] $16   : a80000017ff63000 a80000017ff63458 a80000017ff63990 a80000017ff21b00
[4294698.139000] $20   : 0000000000000002 0000000014001fe1 0000000000000000 0000000000000000
[4294698.139000] $24   : ffffffff80430000 ffffffff80430000                                  
[4294698.139000] $28   : a800000000504000 a800000000507ea0 0000000000000006 ffffffff8029b4d4
[4294698.139000] Hi    : 0000000000000000
[4294698.139000] Lo    : 0000000019643e80
[4294698.139000] epc   : ffffffff80293a48 ata_altstatus+0x68/0x88     Not tainted
[4294698.139000] ra    : ffffffff8029b4d4 ata_eng_timeout+0x15c/0x1b0
[4294698.139000] Status: 14001fe2    KX SX UX KERNEL EXL 
[4294698.139000] Cause : 8080801c
[4294698.139000] PrId  : 01041100
[4294698.139000] Modules linked in:
[4294698.139000] Process scsi_eh_0 (pid: 36, threadinfo=a800000000504000, task=a800000000503968)
[4294698.139000] Stack : a80000017ff63000 ffffffff8028ac68 a80000017ff63000 fffffffffffffffc
[4294698.139000]         0000000000000000 a800000000507f10 ffffffff8029b954 0000000000000000
[4294698.139000]         a80000017ff63000 ffffffff8028ad78 0000000000000003 0000000000000000
[4294698.139000]         ffffffff80422100 ffffffff80125a5c a80000009fffbb60 ffffffff803a4000
[4294698.139000]         ffffffff80422100 ffffffff80422100 a800000000507f20 0000000000000000
[4294698.139000]         a80000017ff63000 ffffffff8028ac68 a80000009fffbb50 fffffffffffffffc
[4294698.139000]         0000000000000000 0000000000000000 0000000000000000 0000000000000000
[4294698.139000]         0000000000000000 ffffffff8014fe90 ffffffffffffffff ffffffffffffffff
[4294698.139000]         0000000000000000 0000000000000000 0000000000000000 0000000000000000
[4294698.139000]         ffffffff801049b0 0000000000000000 0000000000000000 ffffffff801049a0
[4294698.139000]         ...
[4294698.139000] Call Trace:
[4294698.139000]  [<ffffffff8028ac68>] scsi_error_handler+0x0/0xa48
[4294698.139000]  [<ffffffff8029b954>] ata_scsi_error+0x24/0x50
[4294698.139000]  [<ffffffff8028ad78>] scsi_error_handler+0x110/0xa48
[4294698.139000]  [<ffffffff80125a5c>] __wake_up_common+0x7c/0xd8
[4294698.139000]  [<ffffffff8028ac68>] scsi_error_handler+0x0/0xa48
[4294698.139000]  [<ffffffff8014fe90>] kthread+0xf8/0x160
[4294698.139000]  [<ffffffff801049b0>] kernel_thread_helper+0x10/0x18
[4294698.139000]  [<ffffffff801049a0>] kernel_thread_helper+0x0/0x18
[4294698.139000] 
[4294698.139000] 
[4294698.139000] Code: dc448c50  0064182d  90620000 <03e00008> 304200ff  dc8200a0  67bd0010  90430000  03e00008 

-- 
Martin Michlmayr
http://www.cyrius.com/

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello,

Again -- which PCI slot are you seeing this with?  There's two PCI
controllers on this system -- and they work fairly differently.  The
slot closest to the CPU is direct PCI-X, and I believe the better tested
of the two.  The other slots are off of a HT/PCI-X bridge, and more
likely to be flakey.

Thanks,
Mark 

> -----Original Message-----
> From: Martin Michlmayr [mailto:tbm@cyrius.com] 
> Sent: Monday, March 13, 2006 12:57 PM
> To: Alan Cox
> Cc: Mark E Mason; linux-mips@linux-mips.org
> Subject: Re: BCM91x80A/B PCI DMA problems
> 
> * Alan Cox <alan@lxorguk.ukuu.org.uk> [2006-02-28 23:06]:
> > > Is this driver known to work with 64-bit kernels (specifically: 
> > > 64-bit DMA addresses)?  It sounds like that might be the problem.
> > 
> > The standard ATA IDE hardware supports only 32bit 
> addressing. However 
> > if your I/O mapping logic is correctly implemented for the 
> > architecture that should cause no problems as the buffers 
> will be bounced.
> 
> Something is definitely broken in the implemention of this 
> architecture.  With a 64-bit kernel, my SII PCI card shows DMA errors.
> With a 32-bit kernel I get some other weird errors.  What I 
> managed to do is to add the SII device ID to the generic IDE 
> support.  This works but is obviously quite slow...
> 
> > The SIL680 hardware actually can support 64bit DMA using a private 
> > non-standard PRD format and the data sheet is available if someone 
> > wants to do the work. Probably best to do it for the new 
> libata driver 
> > but its doable for either
> 
> I just tried the SCSI PATA patch to see if that works.  
> Unfortunately, it doesn't.  As I said above, something must 
> be broken in the architecture code of SB1.  However, Alan, I 
> think you might be interested in the oops I get anyway 
> because I suspect you may want to fail in a more gracious way.
> 
> BTW, does anyone have the knowledge and time to see what's 
> broken in the SB1 code?  I really need something better than 
> generic IDE support.  It's not fun having a quad-core CPU 
> when your box has to wait for IO all the time. :/
> 
> Alan, for you:
> 
> [4294667.296000] Linux version 2.6.16-rc6 (Debian 
> 2.6.15+2.6.16-rc6-0experimental.1) (waldi@debian.org) (gcc 
> version 4.1.0 (Debian 4.1.0-0)) #7 SMP Mon Mar 13 20:45:43 
> UTC 2006 [4294667.296000] CPU revision is: 01041100 
> [4294667.296000] FPU revision is: 000f0103 [4294667.296000] 
> swarm setup: M41T81 RTC detected.
> [4294667.296000] This kernel optimized for board runs with 
> CFE [4294667.296000] Determined physical RAM map:
> [4294667.296000]  memory: 000000000fe91e00 @ 0000000000000000 
> (usable) [4294667.296000]  memory: 000000001ffffe00 @ 
> 0000000080000000 (usable) [4294667.296000]  memory: 
> 000000000ffffe00 @ 00000000c0000000 (usable) [4294667.296000] 
>  memory: 000000003ffffe00 @ 0000000140000000 (usable) 
> [4294667.296000] Detected 3 available secondary CPU(s) 
> [4294667.296000] Built 1 zonelists [4294667.296000] Kernel 
> command line: root=/dev/hda2 [4294667.296000] Primary 
> instruction cache 32kB, 4-way, linesize 32 bytes.
> [4294667.296000] Primary data cache 32kB, 4-way, linesize 32 bytes.
> [4294667.296000] Synthesized TLB refill handler (35 instructions).
> [4294667.296000] Synthesized TLB load handler fastpath (49 
> instructions).
> [4294667.296000] Synthesized TLB store handler fastpath (49 
> instructions).
> [4294667.296000] Synthesized TLB modify handler fastpath (48 
> instructions).
> [4294667.296000] PID hash table entries: 4096 (order: 12, 
> 131072 bytes) [4294667.313000] Dentry cache hash table 
> entries: 1048576 (order: 11, 8388608 bytes) [4294667.340000] 
> Inode-cache hash table entries: 524288 (order: 10, 4194304 
> bytes) [4294667.653000] Memory: 1991912k/2095672k available 
> (2374k kernel code, 103144k reserved, 643k data, 208k init, 
> 0k highmem) [4294667.674000] Mount-cache hash table entries: 
> 256 [4294667.676000] Checking for 'wait' instruction...  unavailable.
> [4294667.678000] Checking for the multiply/shift bug... no.
> [4294667.680000] Checking for the daddi bug... no.
> [4294667.681000] Checking for the daddiu bug... no.
> [4294667.683000] CPU revision is: 03041100 [4294667.683000] 
> FPU revision is: 000f0103 [4294667.683000] Primary 
> instruction cache 32kB, 4-way, linesize 32 bytes.
> [4294667.683000] Primary data cache 32kB, 4-way, linesize 32 bytes.
> [4294667.683000] Synthesized TLB refill handler (35 instructions).
> [4294667.703000] CPU revision is: 05041100 [4294667.703000] 
> FPU revision is: 000f0103 [4294667.703000] Primary 
> instruction cache 32kB, 4-way, linesize 32 bytes.
> [4294667.703000] Primary data cache 32kB, 4-way, linesize 32 bytes.
> [4294667.703000] Synthesized TLB refill handler (35 instructions).
> [4294667.723000] CPU revision is: 07041100 [4294667.723000] 
> FPU revision is: 000f0103 [4294667.723000] Primary 
> instruction cache 32kB, 4-way, linesize 32 bytes.
> [4294667.723000] Primary data cache 32kB, 4-way, linesize 32 bytes.
> [4294667.723000] Synthesized TLB refill handler (35 instructions).
> [4294667.743000] Brought up 4 CPUs
> [4294667.757000] migration_cost=1000
> [4294667.762000] NET: Registered protocol family 16 
> [4294667.765000] SCSI subsystem initialized [4294667.769000] 
> Initializing Cryptographic API [4294667.770000] io scheduler 
> noop registered [4294667.771000] io scheduler anticipatory 
> registered (default) [4294667.773000] io scheduler deadline 
> registered [4294667.774000] io scheduler cfq registered 
> [4294667.787000] Generic RTC Driver v1.07 [4294667.788000] 
> eth0: enabling TCP rcv checksum [4294667.789000] eth0: SiByte 
> Ethernet at 0x10064000, address: 00:02:4C:F5:2C:3C 
> [4294667.790000] eth1: enabling TCP rcv checksum 
> [4294667.791000] eth1: SiByte Ethernet at 0x10065000, 
> address: 00:02:4C:F5:2C:3D [4294667.792000] eth2: enabling 
> TCP rcv checksum [4294667.793000] eth2: SiByte Ethernet at 
> 0x10066000, address: 00:02:4C:F5:2C:3E [4294667.794000] eth3: 
> enabling TCP rcv checksum [4294667.795000] eth3: SiByte 
> Ethernet at 0x10067000, address: 00:02:4C:F5:2C:3F 
> [4294667.797000] sil680: BA5_EN = 1 clock = 00 
> [4294667.798000] sil680: BA5_EN = 1 clock = 10 
> [4294667.799000] sil680: 133MHz clock.
> [4294667.800000] ata1: PATA max UDMA/133 cmd 0x8010 ctl 
> 0x8022 bmdma 0x8000 irq 8 [4294667.802000] ata2: PATA max 
> UDMA/133 cmd 0x8018 ctl 0x8026 bmdma 0x8008 irq 8 
> [4294667.957000] ata1: dev 0 ATA-6, max UDMA/33, 19932192 
> sectors: LBA [4294667.958000] ata1: dev 0 configured for 
> UDMA/33 [4294667.959000] scsi0 : pata_sil680 [4294668.114000] 
> ata2: dev 0 ATA-6, max UDMA/66, 19932192 sectors: LBA 
> [4294668.115000] ata2: dev 0 configured for UDMA/33 
> [4294668.116000] scsi1 : pata_sil680
> [4294668.117000]   Vendor: ATA       Model: SAMSUNG SV1021H   
> Rev: PJ10
> [4294668.122000]   Type:   Direct-Access                      
> ANSI SCSI revision: 05
> [4294668.124000]   Vendor: ATA       Model: SAMSUNG SV1021H   
> Rev: PJ10
> [4294668.129000]   Type:   Direct-Access                      
> ANSI SCSI revision: 05
> [4294668.132000] SCSI device sda: 19932192 512-byte hdwr 
> sectors (10205 MB) [4294668.133000] sda: Write Protect is off 
> [4294668.134000] SCSI device sda: drive cache: write back 
> [4294668.135000] SCSI device sda: 19932192 512-byte hdwr 
> sectors (10205 MB) [4294668.136000] sda: Write Protect is off 
> [4294668.137000] SCSI device sda: drive cache: write back 
> [4294668.138000]  sda:DBE physical address: 002c008020 
> [4294698.139000] Data bus error, epc == ffffffff80293a48, ra 
> == ffffffff8029b4d4 [4294698.139000] Oops[#1]:
> [4294698.139000] Cpu 0
> [4294698.139000] $ 0   : 0000000000000000 0000000014001fe0 
> 00000000000000ff 900000002c008022
> [4294698.139000] $ 4   : 900000002c000000 a80000017ff63458 
> ffffffff80360000 ffffffffffff00fe
> [4294698.139000] $ 8   : 0000000000000001 ffffffffffffffc0 
> ffffffff80360000 ffffffff80360000
> [4294698.139000] $12   : ffffffff80430000 0000000000001f00 
> 0000000000000000 ffffffff803a0000
> [4294698.139000] $16   : a80000017ff63000 a80000017ff63458 
> a80000017ff63990 a80000017ff21b00
> [4294698.139000] $20   : 0000000000000002 0000000014001fe1 
> 0000000000000000 0000000000000000
> [4294698.139000] $24   : ffffffff80430000 ffffffff80430000    
>                               
> [4294698.139000] $28   : a800000000504000 a800000000507ea0 
> 0000000000000006 ffffffff8029b4d4
> [4294698.139000] Hi    : 0000000000000000
> [4294698.139000] Lo    : 0000000019643e80
> [4294698.139000] epc   : ffffffff80293a48 
> ata_altstatus+0x68/0x88     Not tainted
> [4294698.139000] ra    : ffffffff8029b4d4 ata_eng_timeout+0x15c/0x1b0
> [4294698.139000] Status: 14001fe2    KX SX UX KERNEL EXL 
> [4294698.139000] Cause : 8080801c
> [4294698.139000] PrId  : 01041100
> [4294698.139000] Modules linked in:
> [4294698.139000] Process scsi_eh_0 (pid: 36, 
> threadinfo=a800000000504000, task=a800000000503968) 
> [4294698.139000] Stack : a80000017ff63000 ffffffff8028ac68 
> a80000017ff63000 fffffffffffffffc
> [4294698.139000]         0000000000000000 a800000000507f10 
> ffffffff8029b954 0000000000000000
> [4294698.139000]         a80000017ff63000 ffffffff8028ad78 
> 0000000000000003 0000000000000000
> [4294698.139000]         ffffffff80422100 ffffffff80125a5c 
> a80000009fffbb60 ffffffff803a4000
> [4294698.139000]         ffffffff80422100 ffffffff80422100 
> a800000000507f20 0000000000000000
> [4294698.139000]         a80000017ff63000 ffffffff8028ac68 
> a80000009fffbb50 fffffffffffffffc
> [4294698.139000]         0000000000000000 0000000000000000 
> 0000000000000000 0000000000000000
> [4294698.139000]         0000000000000000 ffffffff8014fe90 
> ffffffffffffffff ffffffffffffffff
> [4294698.139000]         0000000000000000 0000000000000000 
> 0000000000000000 0000000000000000
> [4294698.139000]         ffffffff801049b0 0000000000000000 
> 0000000000000000 ffffffff801049a0
> [4294698.139000]         ...
> [4294698.139000] Call Trace:
> [4294698.139000]  [<ffffffff8028ac68>] 
> scsi_error_handler+0x0/0xa48 [4294698.139000]  
> [<ffffffff8029b954>] ata_scsi_error+0x24/0x50 
> [4294698.139000]  [<ffffffff8028ad78>] 
> scsi_error_handler+0x110/0xa48 [4294698.139000]  
> [<ffffffff80125a5c>] __wake_up_common+0x7c/0xd8 
> [4294698.139000]  [<ffffffff8028ac68>] 
> scsi_error_handler+0x0/0xa48 [4294698.139000]  
> [<ffffffff8014fe90>] kthread+0xf8/0x160 [4294698.139000]  
> [<ffffffff801049b0>] kernel_thread_helper+0x10/0x18 
> [4294698.139000]  [<ffffffff801049a0>] 
> kernel_thread_helper+0x0/0x18 [4294698.139000] 
> [4294698.139000] [4294698.139000] Code: dc448c50  0064182d  
> 90620000 <03e00008> 304200ff  dc8200a0  67bd0010  90430000  03e00008 
> 
> --
> Martin Michlmayr
> http://www.cyrius.com/
> 
> 

Re: BCM91x80A/B PCI DMA problems

[Martin Michlmayr]

* Mark E Mason <mark.e.mason@broadcom.com> [2006-03-13 18:31]:
> Again -- which PCI slot are you seeing this with?

The one you recommended (i.e. the one closest to the CPU), but I see
the same with the other slots.

I have done some more research today with the help of Lennert
Buytenhek, an ARM hacker.  While we were not able to pin down the
exact problem, I think we have enough information so you can look into
it and hopefully suggest a fix.

Finding 1: while it fails with > 1 GB RAM, using 512 MB or 1 GB works.

Finding 2: with > 1 GB RAM, we're getting addresses over 4 GB in
sg->dma_address.  We put some printks into arch/mips/mm/dma-coherent.c
and while everything looks okay with 1 GB of RAM, with 2 GB I get e.g.
    map page a8000000063f36c0 to addr 000000017fc68000

That's an address > 4G.  Lennert thinks that it's "most likely going
to stuff it into a 32bit address register somewhere" and: "I can't see
how it can work at all if it passes the >32bit pci address to the
device.  it should detect, hey, this is above 4G, so then allocate a
buffer below 4G, copy it into there, and pass _that_ buffer to the
device.  that's called dma bounce buffering."

Finding 3: the memory layout is weird.
memory: 000000000fe91e00 @ 0000000000000000 (usable)
memory: 000000001ffffe00 @ 0000000080000000 (usable)
memory: 000000000ffffe00 @ 00000000c0000000 (usable)
memory: 000000003ffffe00 @ 0000000140000000 (usable)

Lennert, "if the pci controller doesn't compensate for such a weird
layout, you're bound to see pci issues."

Finding 4: the host bridge has some "unassigned" memory.  Why?

0001:00:04.0 Host bridge: Broadcom Corporation: Unknown device 0014 (rev 01)
         Flags: bus master, fast devsel, latency 0, IRQ 255
         Memory at 60000000 (32-bit, prefetchable) [size=16M]
         Memory at <unassigned> (32-bit, prefetchable)
         Memory at 70000000 (32-bit, prefetchable) [size=4K]
         Memory at <unassigned> (64-bit, prefetchable)

Does this information help?  Also, we were wondering how to find out
whether a driver is okay with 64-bit dma addresses.

Thanks a lot.
-- 
Martin Michlmayr
http://www.cyrius.com/

Re: BCM91x80A/B PCI DMA problems

[Alan Cox]

On Maw, 2006-03-14 at 03:34 +0000, Martin Michlmayr wrote:
> Does this information help?  Also, we were wondering how to find out
> whether a driver is okay with 64-bit dma addresses.

All drivers set a PCI DMA mask. If the kernel is not bouncing buffers
and ensuring the buffers are below the 32bit bus address limit by
default then the architecture kernel code needs fixing. The drivers
don't deal with this matter beyond setting their PCI DMA range mask.

Alan

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello, 

[snip]
> Finding 1: while it fails with > 1 GB RAM, using 512 MB or 1 GB works.
> 
> Finding 2: with > 1 GB RAM, we're getting addresses over 4 GB in
> sg->dma_address.  We put some printks into arch/mips/mm/dma-coherent.c
> and while everything looks okay with 1 GB of RAM, with 2 GB I get e.g.
>     map page a8000000063f36c0 to addr 000000017fc68000
> 
> That's an address > 4G.  Lennert thinks that it's "most 
> likely going to stuff it into a 32bit address register 
> somewhere" and: "I can't see how it can work at all if it 
> passes the >32bit pci address to the device.  it should 
> detect, hey, this is above 4G, so then allocate a buffer 
> below 4G, copy it into there, and pass _that_ buffer to the 
> device.  that's called dma bounce buffering."

Well, PCI supports both 64 and 32-bit addresses.  There's a lot of PCI
HW out there that's 32-bit only, but there's also a fair bit that isn't.
A 32-bit device in a 64-bit system is going to require bounce buffering
by the driver/OS.  But, a 64-bit device won't (given a reasonable PCI
controller implementation, which we have here).

> Finding 3: the memory layout is weird.
> memory: 000000000fe91e00 @ 0000000000000000 (usable)
> memory: 000000001ffffe00 @ 0000000080000000 (usable)
> memory: 000000000ffffe00 @ 00000000c0000000 (usable)
> memory: 000000003ffffe00 @ 0000000140000000 (usable)
> 
> Lennert, "if the pci controller doesn't compensate for such a 
> weird layout, you're bound to see pci issues."

The PCI controller is built into the 1250/1480, and compensates for
this.


> Finding 4: the host bridge has some "unassigned" memory.  Why?
> 
> 0001:00:04.0 Host bridge: Broadcom Corporation: Unknown 
> device 0014 (rev 01)
>          Flags: bus master, fast devsel, latency 0, IRQ 255
>          Memory at 60000000 (32-bit, prefetchable) [size=16M]
>          Memory at <unassigned> (32-bit, prefetchable)
>          Memory at 70000000 (32-bit, prefetchable) [size=4K]
>          Memory at <unassigned> (64-bit, prefetchable)

Chuckle, I have *no* idea what this corresponds to.

> 
> Does this information help?  Also, we were wondering how to 
> find out whether a driver is okay with 64-bit dma addresses.

This last part is the part I don't know about.  While I've spent the
last 12 years doing this sort of OS/driver programming, it wasn't under
Linux and I'm still coming up to speed on the way things are handled
there.

IIRC most of the testing of the PCI code on the 1480 was done with a
Broadcom PCI-X GigE ethernet card which does support 64-bit addresses,
so the bounce buffering simply wouldn't have been an issue.  I wouldn't
be surprised if there are a number of 32-bit drivers out there that
simply assume that they're always going to be in a 32-bit system.

Unfortunately, the engineer who developed and tested this has since
left, so I don't have a lot of details about what exactly was tested and
what wasn't.

HTH,
Mark

Re: BCM91x80A/B PCI DMA problems

[Martin Michlmayr]

* Mark E Mason <mark.e.mason@broadcom.com> [2006-03-14 08:21]:
> There's a lot of PCI HW out there that's 32-bit only, but there's
> also a fair bit that isn't.  A 32-bit device in a 64-bit system is
> going to require bounce buffering by the driver/OS.

If I understand you correctly, you're saying that you currently don't
supporting 32 bit PCI cards on the SB1 platform.  Is this something
you care about, or should I get a 64-bit PCI card instead?  How hard
would it be to implement bounce buffering anyway?
-- 
Martin Michlmayr
http://www.cyrius.com/

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello, 

> * Mark E Mason <mark.e.mason@broadcom.com> [2006-03-14 08:21]:
> > There's a lot of PCI HW out there that's 32-bit only, but 
> there's also 
> > a fair bit that isn't.  A 32-bit device in a 64-bit system 
> is going to 
> > require bounce buffering by the driver/OS.
> 
> If I understand you correctly, you're saying that you 
> currently don't supporting 32 bit PCI cards on the SB1 
> platform.

Um - not exactly.  A 32-bit device in a 64-bit system will require some
form of bounce buffering.   This is true regardless of what the platform
is, or what the OS/RTOS is.

Sun did some interesting stuff with this in the early UltraSPARC systems
with a 'reverse I/O MMU' which bypassed the need for bounce buffering
but imposed other requirements on the drivers and OS.  Of course, that
required specialized HW which doesn't seem to be in vogue.

>  Is this something you care about, or should I get 
> a 64-bit PCI card instead?  How hard would it be to implement 
> bounce buffering anyway?

This is something I care about.

How difficult it is under Linux - I (personally) don't know.  Based on
other posts, it sounds like Linux has a system for it... But like I said
earlier, I'm far more familiar with certain other RTOS/firmware
environments than I am Linux.

If someone more familiar with this area could point me in the right
direction though, I'll get someone to look at it (if I don't myself).

Thanks,
Mark

> --
> Martin Michlmayr
> http://www.cyrius.com/
> 
> 

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello Alan,
 
> All drivers set a PCI DMA mask. If the kernel is not bouncing 
> buffers and ensuring the buffers are below the 32bit bus 
> address limit by default then the architecture kernel code 
> needs fixing. The drivers don't deal with this matter beyond 
> setting their PCI DMA range mask.

Thanks!

I'm not that familiar with all parts of the Linux kernel.  What should I
be looking for in order to find the relevant bits in the arch-port code?

Thanks,
Mark

RE: BCM91x80A/B PCI DMA problems

[Alan Cox]

On Maw, 2006-03-14 at 08:55 -0800, Mark E Mason wrote:
> > needs fixing. The drivers don't deal with this matter beyond 
> > setting their PCI DMA range mask.
> 
> Thanks!
> 
> I'm not that familiar with all parts of the Linux kernel.  What should I
> be looking for in order to find the relevant bits in the arch-port code?

At the heart of it you need to look at the PCI DMA API. 

[Documentation/DMA-mapping.txt]

Linux uses three kinds of address: virtual and physical are the usual
expected CPU view, bus is the view from the I/O side. On most platforms
physical == bus but not all.

These are mapped onto the dma_* functions of the same name for most
platforms (see include/asm-generic/pci-dma-compat.h) but need not be.
The allocators then grab memory from the right memory pool (16Mb, 32bit,
highmem ..).

For mapping of pages this isn't so simple as the page might be above the
DMA limit of a device. dma_map_* gives you the bus address of a
something relative to the device. This is arch implemented and can
return an address out of the device range.

For a block device that is misbehaving you want to check

1. That the DMA allocations done by the driver (eg the IDE PRD) are
coming in below 4GB as expected

2. Take a look then at block/ll_rw_blk.c which deals with the block
layer. Dump the calls to blk_queue_block_limit and make sure the things
it does are looking sane. 

3. Take a look at what is going on in blk_rq_map_sg which deals with
mapping scatter gather lists to the device, while handling the bounce
limits. (dma_unmap_sg is the clean up end of this)

4. This lot then gets used by ide_build_sglist in drivers/ide/ide-dma.c
and the related ide_build_dmatable function. These might also be worth
dropping debug into to see what is cooking.

Alan

RE: BCM91x80A/B PCI DMA problems

[Mark E Mason]

Hello,

Unfortunately, I don't have the failing piece of HW.... So this is all
by code inspection.
 
> For a block device that is misbehaving you want to check
> 
> 1. That the DMA allocations done by the driver (eg the IDE 
> PRD) are coming in below 4GB as expected

Well, the 32bit DMA mask appears to be set correctly by the IDE driver
in drivers/ide/setup-pci.c (around line 312) -- so it would seem that if
large (>32-bit) addresses were being attempted by the driver, either the
code path including the call to pci_set_dma_mask() wasn't invoked, or
something within the driver or memory allocator went wrong.

What I'm not clear about is what part of this -- if any -- actually
involves the architecture specific code.  In particular, when you say
architecture specific here, do you mean "MIPS" or "SB1 on MIPS"?  It
seems like all of this is handled in the arch independent code.  FWIW,
the SB1 uses the fairly generic dma-coherent.c support under
arch/mips/mm.

The only troubling thing I saw in Documentation/DMA-mapping.txt was what
looked like an assumption that 64-bit systems with 32-bit devices would
have an IOMMU to redirect (arbitrary) 32-bit PCI bus addresses to
(arbitrary) 64-bit physical memory addresses.  I've seen (and
programmed) such things on UltraSPARC systems, but none exists on the
SB1.

But, the later comments about the automatic bounce buffering and the PCI
DMA mask seemed to imply that the IOMMU wasn't required/assumed in all
cases.

> 
> 2. Take a look then at block/ll_rw_blk.c which deals with the 
> block layer. Dump the calls to blk_queue_block_limit and make 
> sure the things it does are looking sane. 
> 
> 3. Take a look at what is going on in blk_rq_map_sg which 
> deals with mapping scatter gather lists to the device, while 
> handling the bounce limits. (dma_unmap_sg is the clean up end of this)
> 
> 4. This lot then gets used by ide_build_sglist in 
> drivers/ide/ide-dma.c and the related ide_build_dmatable 
> function. These might also be worth dropping debug into to 
> see what is cooking.

Unfortunately, 2-4 above requires that I have the failing HW on hand,
which I don't at the moment.  Martin - any chance you can take a look at
this?

64-bit capable devices are known to work correctly in this system (and
known to use >32-bit physical addresses for DMA targets), so it seems
like the issue would mostly lie with the allocation of DMA buffers from
the lower 4G of physical memory.

Of course, I'm probably a bit lost here.  Chuckle.

Thx,
Mark