* AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
@ 2009-03-16 19:15 Luis R. Rodriguez
2009-03-16 19:18 ` Luis R. Rodriguez
2009-03-17 0:35 ` FUJITA Tomonori
0 siblings, 2 replies; 6+ messages in thread
From: Luis R. Rodriguez @ 2009-03-16 19:15 UTC (permalink / raw)
To: wireless, joerg.roedel; +Cc: linux-kernel@vger.kernel.org, iommu
I've run into "PCI-DMA: Out of IOMMU space" messages after loading and
unloading a module 30 times. The interesting thing is this only
happens if I have > 4 GB of memory. The box this occurs has AMD Phenom
quad core CPU so I take it a harware IOMMU is being used. Below are
example relevant messages with > 4 GB and then < 4 GB of memory. The
driver I tested this with was ath9k. It could be an issue perhaps with
ath9k but I am unable to find an issue in our probe/removal.
Could this be an issue with the AMD IOMMU used? Or is it more likely a
driver issue?
With > 4 GB of RAM:
phy0: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011820000, irq=19
phy1: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20012520000, irq=19
phy2: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy3: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy4: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy5: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy6: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy7: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy8: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy9: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy10: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy11: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy12: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy13: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy14: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy15: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy16: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy17: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy18: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy19: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy20: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy21: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy22: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy23: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy24: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy25: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy26: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy27: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy28: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
phy29: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117c0000, irq=19
phy30: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200117a0000, irq=19
ath9k 0000:01:06.0: PCI-DMA: Out of IOMMU space for 3904 bytes
ath9k 0000:01:06.0: PCI-DMA: Out of IOMMU space for 3904 bytes
ath9k 0000:01:06.0: PCI-DMA: Out of IOMMU space for 3904 bytes
... etc
The full log can be seen at:
http://bombadil.infradead.org/~mcgrof/logs/log-out-of-mem-full
With < 4GB RAM:
phy0: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011200000, irq=19
phy1: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200125a0000, irq=19
phy2: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy3: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy4: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy5: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy6: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy7: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy8: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy9: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy10: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy11: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy12: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy13: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy14: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy15: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy16: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy17: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy18: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy19: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy20: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy21: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy22: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy23: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy24: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy25: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy26: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy27: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy28: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy29: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
phy30: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc200110a0000, irq=19
phy31: Atheros AR5416 MAC/BB Rev:2 AR2122 RF Rev:81:
mem=0xffffc20011140000, irq=19
..etc goes on and on
The full log can be seen at:
http://bombadil.infradead.org/~mcgrof/logs/log-out-of-mem-full-NOT
Luis
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
2009-03-16 19:15 AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM Luis R. Rodriguez
@ 2009-03-16 19:18 ` Luis R. Rodriguez
2009-03-17 0:35 ` FUJITA Tomonori
1 sibling, 0 replies; 6+ messages in thread
From: Luis R. Rodriguez @ 2009-03-16 19:18 UTC (permalink / raw)
To: wireless, joerg.roedel; +Cc: linux-kernel@vger.kernel.org, iommu
On Mon, Mar 16, 2009 at 12:15 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote:
> I've run into "PCI-DMA: Out of IOMMU space" messages after loading and
> unloading a module 30 times. The interesting thing is this only
> happens if I have > 4 GB of memory. The box this occurs has AMD Phenom
> quad core CPU so I take it a harware IOMMU is being used. Below are
> example relevant messages with > 4 GB and then < 4 GB of memory. The
> driver I tested this with was ath9k. It could be an issue perhaps with
> ath9k but I am unable to find an issue in our probe/removal.
>
> Could this be an issue with the AMD IOMMU used? Or is it more likely a
> driver issue?
I forgot to mention I'm using l Linus 2.6.29-rc8 with some
serialization patches applied for ath9k (which are unrelated to
DMA/IOMMU/probe) [1].
http://www.kernel.org/pub/linux/kernel/people/mcgrof/patches/ath9k/2009-03-12/serialization-v6/for-2.6.29/
Luis
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
2009-03-16 19:15 AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM Luis R. Rodriguez
2009-03-16 19:18 ` Luis R. Rodriguez
@ 2009-03-17 0:35 ` FUJITA Tomonori
2009-03-17 1:13 ` Luis R. Rodriguez
2009-03-17 7:45 ` Joerg Roedel
1 sibling, 2 replies; 6+ messages in thread
From: FUJITA Tomonori @ 2009-03-17 0:35 UTC (permalink / raw)
To: mcgrof; +Cc: linux-wireless, joerg.roedel, linux-kernel, iommu
On Mon, 16 Mar 2009 12:15:37 -0700
"Luis R. Rodriguez" <mcgrof@gmail.com> wrote:
> I've run into "PCI-DMA: Out of IOMMU space" messages after loading and
> unloading a module 30 times. The interesting thing is this only
> happens if I have > 4 GB of memory. The box this occurs has AMD Phenom
> quad core CPU so I take it a harware IOMMU is being used. Below are
> example relevant messages with > 4 GB and then < 4 GB of memory. The
> driver I tested this with was ath9k. It could be an issue perhaps with
> ath9k but I am unable to find an issue in our probe/removal.
>
> Could this be an issue with the AMD IOMMU used? Or is it more likely a
> driver issue?
ath9k wants DMA_32BIT_MASK, right? If so, GART IOMMU does nothing on a
system with < 4GB memory since ath9k can access to all the memory
addresses directly. With >4GB memory, GART needs to remap an address
higher than 4GB because ath9k wants DMA_32BIT_MASK.
>From a quick look, ath9k doesn't call pci_unmap_single for rx
buffers. Though I might be wrong because I don't know anything about
the driver.
=
From: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
Subject: [PATCH] ath9k: fix dma mapping leak of rx buffer
Signed-off-by: FUJITA Tomonori <fujita.tomonori@lab.ntt.co.jp>
---
drivers/net/wireless/ath9k/recv.c | 5 ++++-
1 files changed, 4 insertions(+), 1 deletions(-)
diff --git a/drivers/net/wireless/ath9k/recv.c b/drivers/net/wireless/ath9k/recv.c
index 462e08c..cd2a1f6 100644
--- a/drivers/net/wireless/ath9k/recv.c
+++ b/drivers/net/wireless/ath9k/recv.c
@@ -322,8 +322,11 @@ void ath_rx_cleanup(struct ath_softc *sc)
list_for_each_entry(bf, &sc->rx.rxbuf, list) {
skb = bf->bf_mpdu;
- if (skb)
+ if (skb) {
+ pci_unmap_single(sc->pdev, bf->bf_buf_addr,
+ sc->rx.bufsize, PCI_DMA_FROMDEVICE);
dev_kfree_skb(skb);
+ }
}
if (sc->rx.rxdma.dd_desc_len != 0)
--
1.6.0.6
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
2009-03-17 0:35 ` FUJITA Tomonori
@ 2009-03-17 1:13 ` Luis R. Rodriguez
2009-03-23 17:09 ` Luis R. Rodriguez
2009-03-17 7:45 ` Joerg Roedel
1 sibling, 1 reply; 6+ messages in thread
From: Luis R. Rodriguez @ 2009-03-17 1:13 UTC (permalink / raw)
To: FUJITA Tomonori; +Cc: linux-wireless, joerg.roedel, linux-kernel, iommu
On Mon, Mar 16, 2009 at 5:35 PM, FUJITA Tomonori
<fujita.tomonori@lab.ntt.co.jp> wrote:
> On Mon, 16 Mar 2009 12:15:37 -0700
> "Luis R. Rodriguez" <mcgrof@gmail.com> wrote:
>
>> I've run into "PCI-DMA: Out of IOMMU space" messages after loading and
>> unloading a module 30 times. The interesting thing is this only
>> happens if I have > 4 GB of memory. The box this occurs has AMD Phenom
>> quad core CPU so I take it a harware IOMMU is being used. Below are
>> example relevant messages with > 4 GB and then < 4 GB of memory. The
>> driver I tested this with was ath9k. It could be an issue perhaps with
>> ath9k but I am unable to find an issue in our probe/removal.
>>
>> Could this be an issue with the AMD IOMMU used? Or is it more likely a
>> driver issue?
>
> ath9k wants DMA_32BIT_MASK, right? If so, GART IOMMU does nothing on a
> system with < 4GB memory since ath9k can access to all the memory
> addresses directly. With >4GB memory, GART needs to remap an address
> higher than 4GB because ath9k wants DMA_32BIT_MASK.
>
> From a quick look, ath9k doesn't call pci_unmap_single for rx
> buffers. Though I might be wrong because I don't know anything about
> the driver.
You seem to be right actually, we were calling pci_unmap_single() for
rx buffers on the RX tasklet but never for the simple case of removing
the module. Will test it out in a bit.
Luis
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
2009-03-17 1:13 ` Luis R. Rodriguez
@ 2009-03-23 17:09 ` Luis R. Rodriguez
0 siblings, 0 replies; 6+ messages in thread
From: Luis R. Rodriguez @ 2009-03-23 17:09 UTC (permalink / raw)
To: FUJITA Tomonori; +Cc: linux-wireless, joerg.roedel, linux-kernel, iommu
On Mon, Mar 16, 2009 at 6:13 PM, Luis R. Rodriguez <mcgrof@gmail.com> wrote:
> On Mon, Mar 16, 2009 at 5:35 PM, FUJITA Tomonori
> <fujita.tomonori@lab.ntt.co.jp> wrote:
>> On Mon, 16 Mar 2009 12:15:37 -0700
>> "Luis R. Rodriguez" <mcgrof@gmail.com> wrote:
>>
>>> I've run into "PCI-DMA: Out of IOMMU space" messages after loading and
>>> unloading a module 30 times. The interesting thing is this only
>>> happens if I have > 4 GB of memory. The box this occurs has AMD Phenom
>>> quad core CPU so I take it a harware IOMMU is being used. Below are
>>> example relevant messages with > 4 GB and then < 4 GB of memory. The
>>> driver I tested this with was ath9k. It could be an issue perhaps with
>>> ath9k but I am unable to find an issue in our probe/removal.
>>>
>>> Could this be an issue with the AMD IOMMU used? Or is it more likely a
>>> driver issue?
>>
>> ath9k wants DMA_32BIT_MASK, right? If so, GART IOMMU does nothing on a
>> system with < 4GB memory since ath9k can access to all the memory
>> addresses directly. With >4GB memory, GART needs to remap an address
>> higher than 4GB because ath9k wants DMA_32BIT_MASK.
>>
>> From a quick look, ath9k doesn't call pci_unmap_single for rx
>> buffers. Though I might be wrong because I don't know anything about
>> the driver.
>
> You seem to be right actually, we were calling pci_unmap_single() for
> rx buffers on the RX tasklet but never for the simple case of removing
> the module. Will test it out in a bit.
That was it, thanks.. will propagate the patch.
Luis
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM
2009-03-17 0:35 ` FUJITA Tomonori
2009-03-17 1:13 ` Luis R. Rodriguez
@ 2009-03-17 7:45 ` Joerg Roedel
1 sibling, 0 replies; 6+ messages in thread
From: Joerg Roedel @ 2009-03-17 7:45 UTC (permalink / raw)
To: FUJITA Tomonori; +Cc: mcgrof, iommu, linux-wireless, linux-kernel
On Tue, Mar 17, 2009 at 09:35:45AM +0900, FUJITA Tomonori wrote:
> From a quick look, ath9k doesn't call pci_unmap_single for rx
> buffers. Though I might be wrong because I don't know anything about
> the driver.
In any case. This error description sounds very much like a dma memory
leak on module unload. So its very likely a driver issue.
Joerg
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2009-03-23 17:09 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2009-03-16 19:15 AMD Quad core - PCI-DMA: Out of IOMMU space on > 4 GB RAM Luis R. Rodriguez
2009-03-16 19:18 ` Luis R. Rodriguez
2009-03-17 0:35 ` FUJITA Tomonori
2009-03-17 1:13 ` Luis R. Rodriguez
2009-03-23 17:09 ` Luis R. Rodriguez
2009-03-17 7:45 ` Joerg Roedel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).