Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Bjorn Helgaas]

[+cc Konrad, xen-devel]

On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
> First time submitting the kernel dev list, so if I have demonstrated
> gross incompetence in some way, please cut me some slack. :)
> 
> I discussed with Clemens Ladisch an issue I was having that I thought
> was related to the driver he authored for my soundcard, but on his
> suggestion I experimented with other PCI devices and we have narrowed
> the issue to the PCI Bridge.
> 
> What hints me toward DMA handling being the fault is the following
> message I receive during kernel initialization after rebooting from an
> Xen hypervisor into a baremetal-kernel, without a power-cycle:
> 
> [0.012815] dmar: DRHD: handling fault status reg 3
> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

Presumably 07:00.0 is your soundcard, and apparently it did a DMA
read to 0x7e00000, which wasn't mapped by the IOMMU.

> In addition to other hints, such as simple IO like changing the active
> port on the soundcard worked (I could hear the relays clicking) but
> not the functions that DMA would be used (streaming audio).
> 
> The PCI devices in my system are connected via this device, as listed by lspci:
> 
> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
> Bridge (rev 03)
> 
> This PCI bridge works under bare-metal Linux normally (w/ various
> debian kernels stable/unstable/testing/backports), but under Xen
> Hypervisor the PCI devices connected exhibit bad behavior.

So it fails with Xen but works otherwise?  I assume there are other
PCI devices, too?  Do they work?

If you want to open a report at http://bugzilla.kernel.org, that would
be a handy place to attach a complete dmesg log and "lspci -vv" output
for more details.

> For
> instance, aformentioned sound card outputs no sound and gets no
> microphone input at all, and an ethernet card causes a system lockup
> as soon as gnome's network manager attempts DHCP over it (presumably;
> I did not test this as thoroughly as the sound card).
> 
> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
> above with both of them. I was briefly running 4.3 but I did not test
> the device.
> 
> Rev1 of the ASM1083 was apparently buggy to the point of being
> unusable, as Clemens pointed out. I would be disappointed if this
> device is similarly unsalvageable,but would be happier if this buggy
> hardware at the very least outputted warnings to users who attempt to
> use it (in Xen).
> 
> All The Best,
> 
> Aaron Opfer
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Konrad Rzeszutek Wilk]

On Fri, Oct 04, 2013 at 11:29:58AM -0600, Bjorn Helgaas wrote:
> [+cc Konrad, xen-devel]

Hey Bjorn, Thanks!
> 
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
> > First time submitting the kernel dev list, so if I have demonstrated
> > gross incompetence in some way, please cut me some slack. :)
> > 
> > I discussed with Clemens Ladisch an issue I was having that I thought
> > was related to the driver he authored for my soundcard, but on his
> > suggestion I experimented with other PCI devices and we have narrowed
> > the issue to the PCI Bridge.
> > 
> > What hints me toward DMA handling being the fault is the following
> > message I receive during kernel initialization after rebooting from an
> > Xen hypervisor into a baremetal-kernel, without a power-cycle:
> > 
> > [0.012815] dmar: DRHD: handling fault status reg 3
> > [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
> > [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear
> 
> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
> read to 0x7e00000, which wasn't mapped by the IOMMU.
> 
> > In addition to other hints, such as simple IO like changing the active
> > port on the soundcard worked (I could hear the relays clicking) but
> > not the functions that DMA would be used (streaming audio).
> > 
> > The PCI devices in my system are connected via this device, as listed by lspci:
> > 
> > 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
> > Bridge (rev 03)
> > 
> > This PCI bridge works under bare-metal Linux normally (w/ various
> > debian kernels stable/unstable/testing/backports), but under Xen
> > Hypervisor the PCI devices connected exhibit bad behavior.
> 
> So it fails with Xen but works otherwise?  I assume there are other
> PCI devices, too?  Do they work?
> 
> If you want to open a report at http://bugzilla.kernel.org, that would
> be a handy place to attach a complete dmesg log and "lspci -vv" output
> for more details.
> 
> > For
> > instance, aformentioned sound card outputs no sound and gets no
> > microphone input at all, and an ethernet card causes a system lockup
> > as soon as gnome's network manager attempts DHCP over it (presumably;
> > I did not test this as thoroughly as the sound card).
> > 
> > I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
> > above with both of them. I was briefly running 4.3 but I did not test
> > the device.

So lets seperate some of these issues. The sound-card issue might be
related to the Linux kernel Xen-SWIOTLB (fixed at some point)- so could
you tell me what the kernel version you have?

Xen 4.3 also has numerous fixes in the IOMMU path.

> > 
> > Rev1 of the ASM1083 was apparently buggy to the point of being
> > unusable, as Clemens pointed out. I would be disappointed if this
> > device is similarly unsalvageable,but would be happier if this buggy
> > hardware at the very least outputted warnings to users who attempt to
> > use it (in Xen).
> > 
> > All The Best,
> > 
> > Aaron Opfer
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Boris Ostrovsky]

On 10/04/2013 02:34 PM, Konrad Rzeszutek Wilk wrote:
> On Fri, Oct 04, 2013 at 11:29:58AM -0600, Bjorn Helgaas wrote:
>> [+cc Konrad, xen-devel]
> Hey Bjorn, Thanks!
>> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>>> First time submitting the kernel dev list, so if I have demonstrated
>>> gross incompetence in some way, please cut me some slack. :)
>>>
>>> I discussed with Clemens Ladisch an issue I was having that I thought
>>> was related to the driver he authored for my soundcard, but on his
>>> suggestion I experimented with other PCI devices and we have narrowed
>>> the issue to the PCI Bridge.
>>>
>>> What hints me toward DMA handling being the fault is the following
>>> message I receive during kernel initialization after rebooting from an
>>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>>>
>>> [0.012815] dmar: DRHD: handling fault status reg 3
>>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

But if you powercycle the system you don't see this error?

Have you tried booting your kernel with "intel_iommu=off"?

-boris

>> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
>> read to 0x7e00000, which wasn't mapped by the IOMMU.
>>
>>> In addition to other hints, such as simple IO like changing the active
>>> port on the soundcard worked (I could hear the relays clicking) but
>>> not the functions that DMA would be used (streaming audio).
>>>
>>> The PCI devices in my system are connected via this device, as listed by lspci:
>>>
>>> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
>>> Bridge (rev 03)
>>>
>>> This PCI bridge works under bare-metal Linux normally (w/ various
>>> debian kernels stable/unstable/testing/backports), but under Xen
>>> Hypervisor the PCI devices connected exhibit bad behavior.
>> So it fails with Xen but works otherwise?  I assume there are other
>> PCI devices, too?  Do they work?
>>
>> If you want to open a report at http://bugzilla.kernel.org, that would
>> be a handy place to attach a complete dmesg log and "lspci -vv" output
>> for more details.
>>
>>> For
>>> instance, aformentioned sound card outputs no sound and gets no
>>> microphone input at all, and an ethernet card causes a system lockup
>>> as soon as gnome's network manager attempts DHCP over it (presumably;
>>> I did not test this as thoroughly as the sound card).
>>>
>>> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
>>> above with both of them. I was briefly running 4.3 but I did not test
>>> the device.
> So lets seperate some of these issues. The sound-card issue might be
> related to the Linux kernel Xen-SWIOTLB (fixed at some point)- so could
> you tell me what the kernel version you have?
>
> Xen 4.3 also has numerous fixes in the IOMMU path.
>
>>> Rev1 of the ASM1083 was apparently buggy to the point of being
>>> unusable, as Clemens pointed out. I would be disappointed if this
>>> device is similarly unsalvageable,but would be happier if this buggy
>>> hardware at the very least outputted warnings to users who attempt to
>>> use it (in Xen).
>>>
>>> All The Best,
>>>
>>> Aaron Opfer
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Gordan Bobic]

On 10/04/2013 06:29 PM, Bjorn Helgaas wrote:
> [+cc Konrad, xen-devel]
>
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>> First time submitting the kernel dev list, so if I have demonstrated
>> gross incompetence in some way, please cut me some slack. :)
>>
>> I discussed with Clemens Ladisch an issue I was having that I thought
>> was related to the driver he authored for my soundcard, but on his
>> suggestion I experimented with other PCI devices and we have narrowed
>> the issue to the PCI Bridge.
>>
>> What hints me toward DMA handling being the fault is the following
>> message I receive during kernel initialization after rebooting from an
>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>>
>> [0.012815] dmar: DRHD: handling fault status reg 3
>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear
>
> Presumably 07:00.0 is your soundcard, and apparently it did a DMA
> read to 0x7e00000, which wasn't mapped by the IOMMU.
>
>> In addition to other hints, such as simple IO like changing the active
>> port on the soundcard worked (I could hear the relays clicking) but
>> not the functions that DMA would be used (streaming audio).
>>
>> The PCI devices in my system are connected via this device, as listed by lspci:
>>
>> 06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI
>> Bridge (rev 03)
>>
>> This PCI bridge works under bare-metal Linux normally (w/ various
>> debian kernels stable/unstable/testing/backports), but under Xen
>> Hypervisor the PCI devices connected exhibit bad behavior.
>
> So it fails with Xen but works otherwise?  I assume there are other
> PCI devices, too?  Do they work?
>
> If you want to open a report at http://bugzilla.kernel.org, that would
> be a handy place to attach a complete dmesg log and "lspci -vv" output
> for more details.
>
>> For
>> instance, aformentioned sound card outputs no sound and gets no
>> microphone input at all, and an ethernet card causes a system lockup
>> as soon as gnome's network manager attempts DHCP over it (presumably;
>> I did not test this as thoroughly as the sound card).
>>
>> I've used Xen Hypervisors 4.1 and 4.2 and had the issues I described
>> above with both of them. I was briefly running 4.3 but I did not test
>> the device.
>>
>> Rev1 of the ASM1083 was apparently buggy to the point of being
>> unusable, as Clemens pointed out. I would be disappointed if this
>> device is similarly unsalvageable,but would be happier if this buggy
>> hardware at the very least outputted warnings to users who attempt to
>> use it (in Xen).

Just out of interest, have you tried without Xen, with a normal kernel 
with full virtualization support with intel_iommu=1 (or equivalent AMD 
option)? If you still get the same problem it is likely the same issue I 
have with phantom PCI bridges.

Gordan

Re: ASMedia ASM1083/1085 rev3 and Xen DMA Failure

[Jan Beulich]

>>> On 04.10.13 at 19:29, Bjorn Helgaas <bhelgaas@google.com> wrote:
> On Wed, Oct 02, 2013 at 04:43:46PM -0400, Aaron Opfer wrote:
>> First time submitting the kernel dev list, so if I have demonstrated
>> gross incompetence in some way, please cut me some slack. :)
>> 
>> I discussed with Clemens Ladisch an issue I was having that I thought
>> was related to the driver he authored for my soundcard, but on his
>> suggestion I experimented with other PCI devices and we have narrowed
>> the issue to the PCI Bridge.
>> 
>> What hints me toward DMA handling being the fault is the following
>> message I receive during kernel initialization after rebooting from an
>> Xen hypervisor into a baremetal-kernel, without a power-cycle:
>> 
>> [0.012815] dmar: DRHD: handling fault status reg 3
>> [0.012868] dmar: DMAR:[DMA Read] Request device [07:00.0] fault addr 7e00000
>> [0.012868] DMAR:[fault reason 02] Present bit in context entry is clear

That's a firmware or native Linux problem then - even during a
"warm" boot all settings a prior OS may have effected to the
hardware must be erased by the firmware (i.e. all components
have to undergo a sufficient level of resetting) for the new OS
to be permitted to make assumptions of the system coming up
in a reasonably consistent state. IOW either some assumption
made by native Linux is invalid, or the firmware doesn't make
sure a valid assumption actually holds.

Jan