Re: Report in downstream Debian: mpt3sas broken with xen dom0 with update to 5.10.149 in 5.10.y.

[Andrew Cooper <Andrew.Cooper3@citrix.com>]

On 24/10/2022 17:58, Christoph Hellwig wrote:
> On Mon, Oct 24, 2022 at 03:20:43PM +0200, Juergen Gross wrote:
>> Dom0 is (normally) a PV domain, so the physical memory can be still above
>> 4 GB even with dom0_mem set to 4GB.
> Which means that we need to ensure the DMA ops for Xen-PV (which is
> always xen-swiotlb I think?) need to return DMA_BIT_MASK(64) or whatever
> is the highest possible address.

Xen tries to allocate memory from the top down, in order to keep the low
memory free for special purses.  (32bit PV guests strictly need to live
below the 128G boundary in host physical space for ABI reasons, and
Xen's struct vcpu needs to be below the 4G boundary for compatibility
with some processors.)

This means that a 64bit PV dom0 ends up occupying the highest addresses
in RAM.  Naturally, this is a problem for devices which can't do 64bit DMA.

The xen-swiotlb is a region (default 128M iirc) which dom0 requests of
Xen during startup.  It is below the 4G boundary, and physically
contiguous (unlike most of the rest of PV guest memory).

Hence it's a suitable bounce buffer for any 32-bit-only devices.

I don't know exactly how this translates to Linux internals, but most
devices are fine and it's routinely the mpt2/3sas drivers which
encounter problems.  It would be lovely if we could get to the bottom of
this for once and for all.

~Andrew