Re: [Bug 205701] New: Can't access RAM from PCIe

[Bjorn Helgaas <helgaas@kernel.org>]

On Fri, Nov 29, 2019 at 06:10:51PM +0200, Ranran wrote:
> On Fri, Nov 29, 2019 at 4:58 PM Bjorn Helgaas <helgaas@kernel.org> wrote:
> > On Fri, Nov 29, 2019 at 06:59:48AM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> > > https://bugzilla.kernel.org/show_bug.cgi?id=205701
> > > ...
> > >
> > > Using Intel Xeon computer with linux kernel 4.18.0 centos8.
> > > Trying to access RAM (with DMA) using FPGA  fails in this computer.
> > >
> > > 1. I tried to add intel_iommu=off - it did not help.
> > >
> > > 2. Installing windows on same PC - FPGA can access RAM using DMA without
> > > issues.
> > >
> > > 3. using another PC (Intel Duo) with same linux and OS - FPGA access works.
> > >
> > > FPGA access the RAM using a physical address provided by a kernel module which
> > > allocates physical continuous memory in PC. (the module works perfectly with
> > > Intel Duo on exactly same OS and kernel).
> >
> > Hi, thanks for the report!  Can you please attach the complete dmesg
> > and "sudo lspci -vv" output for the working and non-working v4.18
> > kernels to the bugzilla?
> >
> > Then please try to reproduce the problem on the current v5.4 kernel
> > and attach the v5.4 dmesg log.  If v5.4 fails, we'll have to debug it.
> > If v5.4 works, figure out what fixed it (by comparing dmesg logs or by
> > bisection) and backport it to v4.18.
> >
> > Bjorn
> 
> Hi,
> I've attached 2 files:
> 1. dmesg.log - is the dmesg you've requested.
> 2. dmesg_intel_iommu_off.log - dmesg when I added intel_iommu=off
> kernel parameter.

Thanks, I attached these to the bugzilla.  I think the linux-pci
mailing list rejected your mail since it wasn't plain-text.

Please also attach the "sudo lspci -vv" output to the bugzilla and
indicate which device is your FPGA.  I guess it might be 0000:20:00.0,
since it looks like it's being claimed by an out-of-tree module in
your dmesg_intel_iommu_off.log (but not dmesg.log).

Please also attach the driver source so we can see how it is obtaining
and using the DMA buffer address.

> I might try the new kernel, yet since we are required to use the
> installation of centos8  (centos8 was just published about 2 month ago
> and it comes with kernel 4.18.0), updating kernel might be
> problematic.

Even if you can't use the v5.4 kernel for your project, if you can
establish that it works, then you have a clear path to finding the
fix.  If v5.4 still *doesn't* work, then we'll be much more interested
in helping to fix that.

> I would please like to ask if there is some workaround you can think of ?
> For example, might it help if I disable iommu (VT-d) in BIOS ?

Usually when an IOMMU blocks a DMA, it seems like there's a note in
dmesg.  I don't see that in either of your logs, but I'm not an IOMMU
expert, so it does seem reasonable to try disabling the IOMMU.

Bjorn