[PATCH] atl1: disable 64bit DMA

[PATCH] atl1: disable 64bit DMA

[Luca Tettamanti]

Il Mon, Jun 25, 2007 at 07:42:44AM -0500, Jay Cliburn ha scritto: 
> Jay L. T. Cornwall wrote:
> >Jay Cliburn wrote:
> >
> >>For reasons not yet clear to me, it appears the L1 driver has a bug or
> >>the device itself has trouble with DMA in high memory.  This patch,
> >>drafted by Luca Tettamanti, is being explored as a workaround.  I'd be
> >>interested to know if it fixes your problem.
> >
> >Yes, it certainly seems to. Now running with this patch and 4GB active,
> >I've transferred about 15GB with no problem so far. It usually oopses
> >after a GB or two.
> >
> >I guess it's not an ideal solution, architecturally speaking, but it's a
> >good deal better than an unstable driver. If there's any other patches
> >you'd like me to test or traces to capture, I'm happy to help out.
> >Otherwise I'll run with this one for now since it does the job!
> 
> Okay Jay, thanks.
> 
> Luca, would you please submit your patch to Jeff Garzik and netdev?

Hi Jeff,
a couple of users reported hard lockups when using L1 NICs on machines
with 4GB or more of RAM. We're still waiting official confirmation from
the vendor, but it seems that L1 has problems doing DMA to/from high
memory (physical address above the 4GB limit). Passing 32bit DMA mask
cures the problem.

Signed-Off-By: Luca Tettamanti <kronos.it@gmail.com>

---
I think that the patch should be included in 2.6.22.

 drivers/net/atl1/atl1_main.c |   15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/drivers/net/atl1/atl1_main.c b/drivers/net/atl1/atl1_main.c
index 6862c11..a730f15 100644
--- a/drivers/net/atl1/atl1_main.c
+++ b/drivers/net/atl1/atl1_main.c
@@ -2097,21 +2097,16 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
 	struct net_device *netdev;
 	struct atl1_adapter *adapter;
 	static int cards_found = 0;
-	bool pci_using_64 = true;
 	int err;
 
 	err = pci_enable_device(pdev);
 	if (err)
 		return err;
 
-	err = pci_set_dma_mask(pdev, DMA_64BIT_MASK);
+	err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
 	if (err) {
-		err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
-		if (err) {
-			dev_err(&pdev->dev, "no usable DMA configuration\n");
-			goto err_dma;
-		}
-		pci_using_64 = false;
+		dev_err(&pdev->dev, "no usable DMA configuration\n");
+		goto err_dma;
 	}
 	/* Mark all PCI regions associated with PCI device
 	 * pdev as being reserved by owner atl1_driver_name
@@ -2176,7 +2171,6 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
 
 	netdev->ethtool_ops = &atl1_ethtool_ops;
 	adapter->bd_number = cards_found;
-	adapter->pci_using_64 = pci_using_64;
 
 	/* setup the private structure */
 	err = atl1_sw_init(adapter);
@@ -2193,9 +2187,6 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
 	 */
 	/* netdev->features |= NETIF_F_TSO; */
 
-	if (pci_using_64)
-		netdev->features |= NETIF_F_HIGHDMA;
-
 	netdev->features |= NETIF_F_LLTX;
 
 	/*


Luca
-- 
Non ho ancora capito se il mio cane e` maschio o femmina:
quando fa la pipi` si chiude in bagno

Re: [PATCH] atl1: disable 64bit DMA

[Chris Snook]

Luca Tettamanti wrote:
> Il Mon, Jun 25, 2007 at 07:42:44AM -0500, Jay Cliburn ha scritto: 
>> Jay L. T. Cornwall wrote:
>>> Jay Cliburn wrote:
>>>
>>>> For reasons not yet clear to me, it appears the L1 driver has a bug or
>>>> the device itself has trouble with DMA in high memory.  This patch,
>>>> drafted by Luca Tettamanti, is being explored as a workaround.  I'd be
>>>> interested to know if it fixes your problem.
>>> Yes, it certainly seems to. Now running with this patch and 4GB active,
>>> I've transferred about 15GB with no problem so far. It usually oopses
>>> after a GB or two.
>>>
>>> I guess it's not an ideal solution, architecturally speaking, but it's a
>>> good deal better than an unstable driver. If there's any other patches
>>> you'd like me to test or traces to capture, I'm happy to help out.
>>> Otherwise I'll run with this one for now since it does the job!
>> Okay Jay, thanks.
>>
>> Luca, would you please submit your patch to Jeff Garzik and netdev?
> 
> Hi Jeff,
> a couple of users reported hard lockups when using L1 NICs on machines
> with 4GB or more of RAM. We're still waiting official confirmation from
> the vendor, but it seems that L1 has problems doing DMA to/from high
> memory (physical address above the 4GB limit). Passing 32bit DMA mask
> cures the problem.
> 
> Signed-Off-By: Luca Tettamanti <kronos.it@gmail.com>
> 
> ---
> I think that the patch should be included in 2.6.22.
> 
>  drivers/net/atl1/atl1_main.c |   15 +++------------
>  1 file changed, 3 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/net/atl1/atl1_main.c b/drivers/net/atl1/atl1_main.c
> index 6862c11..a730f15 100644
> --- a/drivers/net/atl1/atl1_main.c
> +++ b/drivers/net/atl1/atl1_main.c
> @@ -2097,21 +2097,16 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
>  	struct net_device *netdev;
>  	struct atl1_adapter *adapter;
>  	static int cards_found = 0;
> -	bool pci_using_64 = true;
>  	int err;
>  
>  	err = pci_enable_device(pdev);
>  	if (err)
>  		return err;
>  
> -	err = pci_set_dma_mask(pdev, DMA_64BIT_MASK);
> +	err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
>  	if (err) {
> -		err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
> -		if (err) {
> -			dev_err(&pdev->dev, "no usable DMA configuration\n");
> -			goto err_dma;
> -		}
> -		pci_using_64 = false;
> +		dev_err(&pdev->dev, "no usable DMA configuration\n");
> +		goto err_dma;
>  	}
>  	/* Mark all PCI regions associated with PCI device
>  	 * pdev as being reserved by owner atl1_driver_name
> @@ -2176,7 +2171,6 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
>  
>  	netdev->ethtool_ops = &atl1_ethtool_ops;
>  	adapter->bd_number = cards_found;
> -	adapter->pci_using_64 = pci_using_64;
>  
>  	/* setup the private structure */
>  	err = atl1_sw_init(adapter);
> @@ -2193,9 +2187,6 @@ static int __devinit atl1_probe(struct pci_dev *pdev,
>  	 */
>  	/* netdev->features |= NETIF_F_TSO; */
>  
> -	if (pci_using_64)
> -		netdev->features |= NETIF_F_HIGHDMA;
> -
>  	netdev->features |= NETIF_F_LLTX;
>  
>  	/*
> 
> 
> Luca

What boards have we seen this on?  It's quite possible this is:

a) an iommu-related problem specific to AMD or specific to Intel
b) a BIOS problem that atl1 happens to be a victim of

I'd rather not disable this unconditionally if we can get more 
information about why it's breaking.  Doing so might just end up 
covering up the most obvious manifestation of a larger problem.

	-- Chris

Re: [PATCH] atl1: disable 64bit DMA

[Jay L. T. Cornwall]

Chris Snook wrote:

> What boards have we seen this on?  It's quite possible this is:

I can reproduce on an Asus P5K with a Core 2 Duo E6600.

lspci identifies the controller as:
  02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
  Ethernet Adapter (rev b0)

dmesg notes the PCI-DMA mapping implementation:
  PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

-- 
Jay L. T. Cornwall, http://www.esuna.co.uk/~jay/
PhD Student
Imperial College London

Re: [PATCH] atl1: disable 64bit DMA

[Chris Snook]

Jay L. T. Cornwall wrote:
> Chris Snook wrote:
> 
>> What boards have we seen this on?  It's quite possible this is:
> 
> I can reproduce on an Asus P5K with a Core 2 Duo E6600.
> 
> lspci identifies the controller as:
>   02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
>   Ethernet Adapter (rev b0)
> 
> dmesg notes the PCI-DMA mapping implementation:
>   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> 

I had a hunch this was on Intel.  I'd rather just disable this when swiotlb is 
in use, unless we get more complaints.  It's probably ultimately a BIOS quirk 
anyway.

	-- Chris

Re: [PATCH] atl1: disable 64bit DMA

[Jay Cliburn]

On Mon, 25 Jun 2007 17:57:20 -0400
Chris Snook <csnook@redhat.com> wrote:

> Jay L. T. Cornwall wrote:
> > Chris Snook wrote:
> > 
> >> What boards have we seen this on?  It's quite possible this is:
> > 
> > I can reproduce on an Asus P5K with a Core 2 Duo E6600.
> > 
> > lspci identifies the controller as:
> >   02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
> >   Ethernet Adapter (rev b0)
> > 
> > dmesg notes the PCI-DMA mapping implementation:
> >   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> > 
> 
> I had a hunch this was on Intel.  I'd rather just disable this when
> swiotlb is in use, unless we get more complaints.  It's probably
> ultimately a BIOS quirk anyway.

So far we have reports from both camps:

Asus M2N8-VMX (AM2):	1 report of lockup
http://sourceforge.net/mailarchive/forum.php?thread_name=46780384.063603.26165%40m12-15.163.com&forum_name=atl1-devel

Asus P5K (LGA775):	2 reports of lockups
http://sourceforge.net/mailarchive/forum.php?thread_name=467E7E34.4010603%40gmail.com&forum_name=atl1-devel
http://lkml.org/lkml/2007/6/25/107

The common denominator in these reports is 4GB RAM.

Re: [PATCH] atl1: disable 64bit DMA

[Jeff Garzik]

Jay Cliburn wrote:
> On Mon, 25 Jun 2007 17:57:20 -0400
> Chris Snook <csnook@redhat.com> wrote:
> 
>> Jay L. T. Cornwall wrote:
>>> Chris Snook wrote:
>>>
>>>> What boards have we seen this on?  It's quite possible this is:
>>> I can reproduce on an Asus P5K with a Core 2 Duo E6600.
>>>
>>> lspci identifies the controller as:
>>>   02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
>>>   Ethernet Adapter (rev b0)
>>>
>>> dmesg notes the PCI-DMA mapping implementation:
>>>   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>>
>> I had a hunch this was on Intel.  I'd rather just disable this when
>> swiotlb is in use, unless we get more complaints.  It's probably
>> ultimately a BIOS quirk anyway.
> 
> So far we have reports from both camps:
> 
> Asus M2N8-VMX (AM2):	1 report of lockup
> http://sourceforge.net/mailarchive/forum.php?thread_name=46780384.063603.26165%40m12-15.163.com&forum_name=atl1-devel
> 
> Asus P5K (LGA775):	2 reports of lockups
> http://sourceforge.net/mailarchive/forum.php?thread_name=467E7E34.4010603%40gmail.com&forum_name=atl1-devel
> http://lkml.org/lkml/2007/6/25/107
> 
> The common denominator in these reports is 4GB RAM.

Although its possible this device doesn't really support 64-bit, it's 
more likely that this is a platform problem of some sort, or a driver 
bug of some sort.  In the driver, maybe it has a problem when you 
-cross- a 4GB boundary, which is not uncommon.

	Jeff

Re: [PATCH] atl1: disable 64bit DMA

[Chris Snook]

Jeff Garzik wrote:
> Jay Cliburn wrote:
>> On Mon, 25 Jun 2007 17:57:20 -0400
>> Chris Snook <csnook@redhat.com> wrote:
>>
>>> Jay L. T. Cornwall wrote:
>>>> Chris Snook wrote:
>>>>
>>>>> What boards have we seen this on?  It's quite possible this is:
>>>> I can reproduce on an Asus P5K with a Core 2 Duo E6600.
>>>>
>>>> lspci identifies the controller as:
>>>>   02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
>>>>   Ethernet Adapter (rev b0)
>>>>
>>>> dmesg notes the PCI-DMA mapping implementation:
>>>>   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
>>>>
>>> I had a hunch this was on Intel.  I'd rather just disable this when
>>> swiotlb is in use, unless we get more complaints.  It's probably
>>> ultimately a BIOS quirk anyway.
>>
>> So far we have reports from both camps:
>>
>> Asus M2N8-VMX (AM2):    1 report of lockup
>> http://sourceforge.net/mailarchive/forum.php?thread_name=46780384.063603.26165%40m12-15.163.com&forum_name=atl1-devel 
>>
>>
>> Asus P5K (LGA775):    2 reports of lockups
>> http://sourceforge.net/mailarchive/forum.php?thread_name=467E7E34.4010603%40gmail.com&forum_name=atl1-devel 
>>
>> http://lkml.org/lkml/2007/6/25/107
>>
>> The common denominator in these reports is 4GB RAM.
> 
> Although its possible this device doesn't really support 64-bit, it's 
> more likely that this is a platform problem of some sort, or a driver 
> bug of some sort.  In the driver, maybe it has a problem when you 
> -cross- a 4GB boundary, which is not uncommon.
> 
>     Jeff

I'm going on the record to say I don't trust the chipsets on these boards, and 
I'd like anyone having these problems to let us 
(atl1-devel@lists.sourceforge.net) know if they encounter similar problems with 
any other hardware.  That said, I'm not going to stand in the way of stability 
just because it *might* be someone else's fault.  I don't think limiting 
ourselves to dma32, at least while we track this down, is much of a loss on 
current hardware.

Acked-By: Chris Snook <csnook@redhat.com>

Re: [PATCH] atl1: disable 64bit DMA

[Luca]

On 6/26/07, Jeff Garzik <jgarzik@pobox.com> wrote:
> Jay Cliburn wrote:
> > On Mon, 25 Jun 2007 17:57:20 -0400
> > Chris Snook <csnook@redhat.com> wrote:
> >
> >> Jay L. T. Cornwall wrote:
> >>> Chris Snook wrote:
> >>>
> >>>> What boards have we seen this on?  It's quite possible this is:
> >>> I can reproduce on an Asus P5K with a Core 2 Duo E6600.
> >>>
> >>> lspci identifies the controller as:
> >>>   02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
> >>>   Ethernet Adapter (rev b0)
> >>>
> >>> dmesg notes the PCI-DMA mapping implementation:
> >>>   PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
> >>>
> >> I had a hunch this was on Intel.  I'd rather just disable this when
> >> swiotlb is in use, unless we get more complaints.  It's probably
> >> ultimately a BIOS quirk anyway.
> >
> > So far we have reports from both camps:
> >
> > Asus M2N8-VMX (AM2):  1 report of lockup
> > http://sourceforge.net/mailarchive/forum.php?thread_name=46780384.063603.26165%40m12-15.163.com&forum_name=atl1-devel
> >
> > Asus P5K (LGA775):    2 reports of lockups
> > http://sourceforge.net/mailarchive/forum.php?thread_name=467E7E34.4010603%40gmail.com&forum_name=atl1-devel
> > http://lkml.org/lkml/2007/6/25/107
> >
> > The common denominator in these reports is 4GB RAM.
>
> Although its possible this device doesn't really support 64-bit, it's
> more likely that this is a platform problem of some sort, or a driver
> bug of some sort.  In the driver, maybe it has a problem when you
> -cross- a 4GB boundary, which is not uncommon.

I don't follow you :| What kind "common" mistakes should we check for
in the driver?

Luca

Re: [PATCH] atl1: disable 64bit DMA

[Jay Cliburn]

On Mon, 25 Jun 2007 23:18:55 +0200
Luca Tettamanti <kronos.it@gmail.com> wrote:

> Il Mon, Jun 25, 2007 at 07:42:44AM -0500, Jay Cliburn ha scritto: 
> > Jay L. T. Cornwall wrote:
> > >Jay Cliburn wrote:
> > >
> > >>For reasons not yet clear to me, it appears the L1 driver has a
> > >>bug or the device itself has trouble with DMA in high memory.
> > >>This patch, drafted by Luca Tettamanti, is being explored as a
> > >>workaround.  I'd be interested to know if it fixes your problem.
> > >
> > >Yes, it certainly seems to. Now running with this patch and 4GB
> > >active, I've transferred about 15GB with no problem so far. It
> > >usually oopses after a GB or two.
> > >
> > >I guess it's not an ideal solution, architecturally speaking, but
> > >it's a good deal better than an unstable driver. If there's any
> > >other patches you'd like me to test or traces to capture, I'm
> > >happy to help out. Otherwise I'll run with this one for now since
> > >it does the job!
> > 
> > Okay Jay, thanks.
> > 
> > Luca, would you please submit your patch to Jeff Garzik and netdev?
> 
> Hi Jeff,
> a couple of users reported hard lockups when using L1 NICs on machines
> with 4GB or more of RAM. We're still waiting official confirmation
> from the vendor, but it seems that L1 has problems doing DMA to/from
> high memory (physical address above the 4GB limit). Passing 32bit DMA
> mask cures the problem.
> 
> Signed-Off-By: Luca Tettamanti <kronos.it@gmail.com>
> 
> ---
> I think that the patch should be included in 2.6.22.
> 
>  drivers/net/atl1/atl1_main.c |   15 +++------------
>  1 file changed, 3 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/net/atl1/atl1_main.c
> b/drivers/net/atl1/atl1_main.c index 6862c11..a730f15 100644
> --- a/drivers/net/atl1/atl1_main.c
> +++ b/drivers/net/atl1/atl1_main.c
> @@ -2097,21 +2097,16 @@ static int __devinit atl1_probe(struct
> pci_dev *pdev, struct net_device *netdev;
>  	struct atl1_adapter *adapter;
>  	static int cards_found = 0;
> -	bool pci_using_64 = true;
>  	int err;
>  
>  	err = pci_enable_device(pdev);
>  	if (err)
>  		return err;
>  
> -	err = pci_set_dma_mask(pdev, DMA_64BIT_MASK);
> +	err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
>  	if (err) {
> -		err = pci_set_dma_mask(pdev, DMA_32BIT_MASK);
> -		if (err) {
> -			dev_err(&pdev->dev, "no usable DMA
> configuration\n");
> -			goto err_dma;
> -		}
> -		pci_using_64 = false;
> +		dev_err(&pdev->dev, "no usable DMA configuration\n");
> +		goto err_dma;
>  	}
>  	/* Mark all PCI regions associated with PCI device
>  	 * pdev as being reserved by owner atl1_driver_name
> @@ -2176,7 +2171,6 @@ static int __devinit atl1_probe(struct pci_dev
> *pdev, 
>  	netdev->ethtool_ops = &atl1_ethtool_ops;
>  	adapter->bd_number = cards_found;
> -	adapter->pci_using_64 = pci_using_64;
>  
>  	/* setup the private structure */
>  	err = atl1_sw_init(adapter);
> @@ -2193,9 +2187,6 @@ static int __devinit atl1_probe(struct pci_dev
> *pdev, */
>  	/* netdev->features |= NETIF_F_TSO; */
>  
> -	if (pci_using_64)
> -		netdev->features |= NETIF_F_HIGHDMA;
> -
>  	netdev->features |= NETIF_F_LLTX;
>  
>  	/*

Acked-by: Jay Cliburn <jacliburn@bellsouth.net>