[PATCH V7 08/11] drivers: acpi: Handle IOMMU lookup failure with deferred probing or error

[lorenzo.pieralisi@arm.com (Lorenzo Pieralisi)]

On Mon, Jan 30, 2017 at 03:03:06PM -0500, Sinan Kaya wrote:
> On 1/30/2017 11:51 AM, Lorenzo Pieralisi wrote:
> > On Mon, Jan 30, 2017 at 10:46:39AM -0500, Sinan Kaya wrote:
> >> On 1/30/2017 9:54 AM, Nate Watterson wrote:
> >>> On 2017-01-30 09:38, Will Deacon wrote:
> >>>> On Mon, Jan 30, 2017 at 09:33:50AM -0500, Sinan Kaya wrote:
> >>>>> On 1/30/2017 9:23 AM, Nate Watterson wrote:
> >>>>>> On 2017-01-30 08:59, Sinan Kaya wrote:
> >>>>>>> On 1/30/2017 7:22 AM, Robin Murphy wrote:
> >>>>>>>> On 29/01/17 17:53, Sinan Kaya wrote:
> >>>>>>>>> On 1/24/2017 7:37 AM, Lorenzo Pieralisi wrote:
> >>>>>>>>>> [+hanjun, tomasz, sinan]
> >>>>>>>>>>
> >>>>>>>>>> It is quite a key patchset, I would be glad if they can test on their
> >>>>>>>>>> respective platforms with IORT.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Tested on top of 4.10-rc5.
> >>>>>>>>>
> >>>>>>>>> 1.    Platform Hidma device passed dmatest
> >>>>>>>>> 2.    Seeing some USB stalls on a platform USB device.
> >>>>>>>>> 3.    PCIe NVME drive probed and worked fine with MSI interrupts after boot.
> >>>>>>>>> 4.     NVMe driver didn't probe following a hotplug insertion and received an
> >>>>>>>>> SMMU error event during the insertion.
> >>>>>>>>
> >>>>>>>> What was the SMMU error - a translation/permission fault (implying the
> >>>>>>>> wrong DMA ops) or a bad STE fault (implying we totally failed to tell
> >>>>>>>> the SMMU about the device at all)?
> >>>>>>>>
> >>>>>>>
> >>>>>>> root at ubuntu:/sys/bus/pci/slots/4# echo 0 > power
> >>>>>>>
> >>>>>>> [__204.698522]_iommu:_Removing_device_0003:01:00.0_from_group_0
> >>>>>>> [  204.708704] pciehp 0003:00:00.0:pcie004: Slot(4): Link Down
> >>>>>>> [  204.708723] pciehp 0003:00:00.0:pcie004: Slot(4): Link Down event
> >>>>>>> ignored; already powering off
> >>>>>>>
> >>>>>>> root at ubuntu:/sys/bus/pci/slots/4#
> >>>>>>>
> >>>>>>> [__254.820440]_iommu:_Adding_device_0003:01:00.0_to_group_8
> >>>>>>> [  254.820599] nvme nvme0: pci function 0003:01:00.0
> >>>>>>> [  254.820621] nvme 0003:01:00.0: enabling device (0000 -> 0002)
> >>>>>>> [  261.948558] arm-smmu-v3 arm-smmu-v3.0.auto: event 0x0a received:
> >>>>>>> [  261.948561] arm-smmu-v3 arm-smmu-v3.0.auto:  0x000001000000000a
> >>>>>>> [  261.948563] arm-smmu-v3 arm-smmu-v3.0.auto:  0x0000000000000000
> >>>>>>> [  261.948564] arm-smmu-v3 arm-smmu-v3.0.auto:  0x0000000000000000
> >>>>>>> [  261.948566] arm-smmu-v3 arm-smmu-v3.0.auto:  0x0000000000000000
> >>>>>
> >>>>>> Looks like C_BAD_CD. Can you please try with:
> >>>>>> iommu/arm-smmu-v3: Clear prior settings when updating STEs
> >>>>>
> >>>>> This resolved the issue. Can we pull Nate's patch to 4.10 so that I don't see
> >>>>> this issue again.
> >>>>
> >>>> I already sent the pull request to Joerg for 4.11. Do you see this problem
> >>>> without Sricharan's patches (i.e. vanilla mainline)? If so, we'll need to
> >>>> send the patch to stable after -rc1.
> >>> Using vanilla mainline, I see it most commonly when directly assigning
> >>> a device to a guest machine. I think I've also seen it after removing then
> >>> re-adding a PCI device. Basically anytime an STE's CTX pointer is changed
> >>> from a non-NULL value and STE[CFG] indicates translation will be performed.
> >>>
> >>
> >> I was not able to reproduce the issue with Vanilla kernel. I only
> >> tested hotplug.
> > 
> > I would like to get the complete code path leading to this issue, it is
> > not clear to me why the probe deferral code triggers it and why we are
> > not able to trigger it with vanilla mainline, we must understand that first
> > before applying any fix to this series.
> > 
> > I do not have a platform to reproduce this issue I will send you a patch
> > to trace what's going on here please help us debug it.
> 
> Sure, send it to both Nate and me.

I debugged the issue and Nate's fix is correct, the fact that you
can't it hit it with mainline is just a matter of timing because it has
to do with the CTX pointer value (we OR it with the existing value), so
it may work or not depending on how the cdptr memory allocation
pattern turns out to be (which explains why Nate and I can hit it with
simple PCI device remove/add execution too).

So it is neither an ACPI nor an IOMMU probe deferral issue per-se,
fix is already queued, so it is all good.

What about USB stalls ?

Thanks !
Lorenzo