ASMedia ASM1083/1085 rev3 and Xen DMA Failure

ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Aaron Opfer]

First time submitting the kernel dev list, so if I have demonstrated
gross incompetence in some way, please cut me some slack. :)

I discussed with Clemens Ladisch an issue I was having that I thought
was related to the driver he authored for my soundcard, but on his
suggestion I experimented with other PCI devices and we have narrowed
the issue to the PCI Bridge.

What hints me toward DMA handling being the fault is the following
message I receive during kernel initialization after rebooting from an
Xen hypervisor into a baremetal-kernel, without a power-cycle:

[0.012815] dmar: DRHD: handling fault status reg 3
[0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
[0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

In addition to other hints, such as simple IO like changing the active
port on the soundcard worked (I could hear the relays clicking) but
not the functions that DMA would be used (streaming audio).

The PCI devices in my system are connected via this device, as listed by lspci:

06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
Bridge (rev 03)

This PCI bridge works under bare-metal Linux normally (w/ various
debian kernels stable/unstable/testing/backports), but under Xen
Hypervisor the PCI devices connected exhibit bad behavior. For
instance, aformentioned sound card outputs no sound and gets no
microphone input at all, and an ethernet card causes a system lockup
as soon as gnome's network manager attempts DHCP over it (presumably;
I did not test this as thoroughly as the sound card).

I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
above with both of them. I was briefly running 4.3 but I did not test
the device.

Rev1 of the ASM1083 was apparently buggy to the point of being
unusable, as Clemens pointed out. I would be disappointed if this
device is similarly unsalvageable,but would be happier if this buggy
hardware at the very least outputted warnings to users who attempt to
use it (in Xen).

All The Best,

Aaron Opfer

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Bjorn Helgaas]

[+cc Konrad, xen-devel]

On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
> First time submitting the kernel dev list, so if I have demonstrated
> gross incompetence in some way, please cut me some slack. :)
> 
> I discussed with Clemens Ladisch an issue I was having that I thought
> was related to the driver he authored for my soundcard, but on his
> suggestion I experimented with other PCI devices and we have narrowed
> the issue to the PCI Bridge.
> 
> What hints me toward DMA handling being the fault is the following
> message I receive during kernel initialization after rebooting from an
> Xen hypervisor into a baremetal-kernel, without a power-cycle:
> 
> [0.012815] dmar: DRHD: handling fault status reg 3
> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

Presumably 07:00.0 is your soundcard, and apparently it did a DMA
read to 0x7e00000, which wasn't mapped by the IOMMU.

> In addition to other hints, such as simple IO like changing the active
> port on the soundcard worked (I could hear the relays clicking) but
> not the functions that DMA would be used (streaming audio).
> 
> The PCI devices in my system are connected via this device, as listed by lspci:
> 
> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
> Bridge (rev 03)
> 
> This PCI bridge works under bare-metal Linux normally (w/ various
> debian kernels stable/unstable/testing/backports), but under Xen
> Hypervisor the PCI devices connected exhibit bad behavior.

So it fails with Xen but works otherwise?  I assume there are other
PCI devices, too?  Do they work?

If you want to open a report at http://bugzilla.kernel.org, that would
be a handy place to attach a complete dmesg log and "lspci -vv" output
for more details.

> For
> instance, aformentioned sound card outputs no sound and gets no
> microphone input at all, and an ethernet card causes a system lockup
> as soon as gnome's network manager attempts DHCP over it (presumably;
> I did not test this as thoroughly as the sound card).
> 
> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
> above with both of them. I was briefly running 4.3 but I did not test
> the device.
> 
> Rev1 of the ASM1083 was apparently buggy to the point of being
> unusable, as Clemens pointed out. I would be disappointed if this
> device is similarly unsalvageable,but would be happier if this buggy
> hardware at the very least outputted warnings to users who attempt to
> use it (in Xen).
> 
> All The Best,
> 
> Aaron Opfer
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Konrad Rzeszutek Wilk]

On Fri, Oct 04, 2013 at 11:29:58AM -0600, Bjorn Helgaas wrote:
> [+cc Konrad, xen-devel]

Hey Bjorn, Thanks!
> 
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
> > First time submitting the kernel dev list, so if I have demonstrated
> > gross incompetence in some way, please cut me some slack. :)
> > 
> > I discussed with Clemens Ladisch an issue I was having that I thought
> > was related to the driver he authored for my soundcard, but on his
> > suggestion I experimented with other PCI devices and we have narrowed
> > the issue to the PCI Bridge.
> > 
> > What hints me toward DMA handling being the fault is the following
> > message I receive during kernel initialization after rebooting from an
> > Xen hypervisor into a baremetal-kernel, without a power-cycle:
> > 
> > [0.012815] dmar: DRHD: handling fault status reg 3
> > [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
> > [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear
> 
> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
> read to 0x7e00000, which wasn't mapped by the IOMMU.
> 
> > In addition to other hints, such as simple IO like changing the active
> > port on the soundcard worked (I could hear the relays clicking) but
> > not the functions that DMA would be used (streaming audio).
> > 
> > The PCI devices in my system are connected via this device, as listed by lspci:
> > 
> > 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
> > Bridge (rev 03)
> > 
> > This PCI bridge works under bare-metal Linux normally (w/ various
> > debian kernels stable/unstable/testing/backports), but under Xen
> > Hypervisor the PCI devices connected exhibit bad behavior.
> 
> So it fails with Xen but works otherwise?  I assume there are other
> PCI devices, too?  Do they work?
> 
> If you want to open a report at http://bugzilla.kernel.org, that would
> be a handy place to attach a complete dmesg log and "lspci -vv" output
> for more details.
> 
> > For
> > instance, aformentioned sound card outputs no sound and gets no
> > microphone input at all, and an ethernet card causes a system lockup
> > as soon as gnome's network manager attempts DHCP over it (presumably;
> > I did not test this as thoroughly as the sound card).
> > 
> > I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
> > above with both of them. I was briefly running 4.3 but I did not test
> > the device.

So lets seperate some of these issues. The sound-card issue might be
related to the Linux kernel Xen-SWIOTLB (fixed at some point)- so could
you tell me what the kernel version you have?

Xen 4.3 also has numerous fixes in the IOMMU path.

> > 
> > Rev1 of the ASM1083 was apparently buggy to the point of being
> > unusable, as Clemens pointed out. I would be disappointed if this
> > device is similarly unsalvageable,but would be happier if this buggy
> > hardware at the very least outputted warnings to users who attempt to
> > use it (in Xen).
> > 
> > All The Best,
> > 
> > Aaron Opfer
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Boris Ostrovsky]

On 10/04/2013 02:34 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 04, 2013 at 11:29:58AM -0600, Bjorn Helgaas wrote:
>> [+cc Konrad, xen-devel]
> Hey Bjorn, Thanks!
>> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>>> First time submitting the kernel dev list, so if I have demonstrated
>>> gross incompetence in some way, please cut me some slack. :)
>>>
>>> I discussed with Clemens Ladisch an issue I was having that I thought
>>> was related to the driver he authored for my soundcard, but on his
>>> suggestion I experimented with other PCI devices and we have narrowed
>>> the issue to the PCI Bridge.
>>>
>>> What hints me toward DMA handling being the fault is the following
>>> message I receive during kernel initialization after rebooting from an
>>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>>>
>>> [0.012815] dmar: DRHD: handling fault status reg 3
>>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

But if you powercycle the system you don't see this error?

Have you tried booting your kernel with "intel_iommu=off"?

-boris

>> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
>> read to 0x7e00000, which wasn't mapped by the IOMMU.
>>
>>> In addition to other hints, such as simple IO like changing the active
>>> port on the soundcard worked (I could hear the relays clicking) but
>>> not the functions that DMA would be used (streaming audio).
>>>
>>> The PCI devices in my system are connected via this device, as listed by lspci:
>>>
>>> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
>>> Bridge (rev 03)
>>>
>>> This PCI bridge works under bare-metal Linux normally (w/ various
>>> debian kernels stable/unstable/testing/backports), but under Xen
>>> Hypervisor the PCI devices connected exhibit bad behavior.
>> So it fails with Xen but works otherwise?  I assume there are other
>> PCI devices, too?  Do they work?
>>
>> If you want to open a report at http://bugzilla.kernel.org, that would
>> be a handy place to attach a complete dmesg log and "lspci -vv" output
>> for more details.
>>
>>> For
>>> instance, aformentioned sound card outputs no sound and gets no
>>> microphone input at all, and an ethernet card causes a system lockup
>>> as soon as gnome's network manager attempts DHCP over it (presumably;
>>> I did not test this as thoroughly as the sound card).
>>>
>>> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
>>> above with both of them. I was briefly running 4.3 but I did not test
>>> the device.
> So lets seperate some of these issues. The sound-card issue might be
> related to the Linux kernel Xen-SWIOTLB (fixed at some point)- so could
> you tell me what the kernel version you have?
>
> Xen 4.3 also has numerous fixes in the IOMMU path.
>
>>> Rev1 of the ASM1083 was apparently buggy to the point of being
>>> unusable, as Clemens pointed out. I would be disappointed if this
>>> device is similarly unsalvageable,but would be happier if this buggy
>>> hardware at the very least outputted warnings to users who attempt to
>>> use it (in Xen).
>>>
>>> All The Best,
>>>
>>> Aaron Opfer
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [Xen-devel] ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Gordan Bobic]

On 10/04/2013 06:29 PM, Bjorn Helgaas wrote:
> [+cc Konrad, xen-devel]
>
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>> First time submitting the kernel dev list, so if I have demonstrated
>> gross incompetence in some way, please cut me some slack. :)
>>
>> I discussed with Clemens Ladisch an issue I was having that I thought
>> was related to the driver he authored for my soundcard, but on his
>> suggestion I experimented with other PCI devices and we have narrowed
>> the issue to the PCI Bridge.
>>
>> What hints me toward DMA handling being the fault is the following
>> message I receive during kernel initialization after rebooting from an
>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>>
>> [0.012815] dmar: DRHD: handling fault status reg 3
>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear
>
> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
> read to 0x7e00000, which wasn't mapped by the IOMMU.
>
>> In addition to other hints, such as simple IO like changing the active
>> port on the soundcard worked (I could hear the relays clicking) but
>> not the functions that DMA would be used (streaming audio).
>>
>> The PCI devices in my system are connected via this device, as listed by lspci:
>>
>> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
>> Bridge (rev 03)
>>
>> This PCI bridge works under bare-metal Linux normally (w/ various
>> debian kernels stable/unstable/testing/backports), but under Xen
>> Hypervisor the PCI devices connected exhibit bad behavior.
>
> So it fails with Xen but works otherwise?  I assume there are other
> PCI devices, too?  Do they work?
>
> If you want to open a report at http://bugzilla.kernel.org, that would
> be a handy place to attach a complete dmesg log and "lspci -vv" output
> for more details.
>
>> For
>> instance, aformentioned sound card outputs no sound and gets no
>> microphone input at all, and an ethernet card causes a system lockup
>> as soon as gnome's network manager attempts DHCP over it (presumably;
>> I did not test this as thoroughly as the sound card).
>>
>> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
>> above with both of them. I was briefly running 4.3 but I did not test
>> the device.
>>
>> Rev1 of the ASM1083 was apparently buggy to the point of being
>> unusable, as Clemens pointed out. I would be disappointed if this
>> device is similarly unsalvageable,but would be happier if this buggy
>> hardware at the very least outputted warnings to users who attempt to
>> use it (in Xen).

Just out of interest, have you tried without Xen, with a normal kernel 
with full virtualization support with intel_iommu=1 (or equivalent AMD 
option)? If you still get the same problem it is likely the same issue I 
have with phantom PCI bridges.

Gordan

Re: [Xen-devel] ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Jan Beulich]

>>> On 04.10.13 at 19:29, Bjorn Helgaas <bhelgaas@google.com> wrote:
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>> First time submitting the kernel dev list, so if I have demonstrated
>> gross incompetence in some way, please cut me some slack. :)
>> 
>> I discussed with Clemens Ladisch an issue I was having that I thought
>> was related to the driver he authored for my soundcard, but on his
>> suggestion I experimented with other PCI devices and we have narrowed
>> the issue to the PCI Bridge.
>> 
>> What hints me toward DMA handling being the fault is the following
>> message I receive during kernel initialization after rebooting from an
>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>> 
>> [0.012815] dmar: DRHD: handling fault status reg 3
>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

That's a firmware or native Linux problem then - even during a
"warm" boot all settings a prior OS may have effected to the
hardware must be erased by the firmware (i.e. all components
have to undergo a sufficient level of resetting) for the new OS
to be permitted to make assumptions of the system coming up
in a reasonably consistent state. IOW either some assumption
made by native Linux is invalid, or the firmware doesn't make
sure a valid assumption actually holds.

Jan