Re: [PATCH rdma-next 1/2] arm64/io: add memcpy_toio_64

[Jason Gunthorpe <jgg@nvidia.com>]

On Fri, Jan 26, 2024 at 04:15:16PM +0000, Catalin Marinas wrote:
> > It looks to me like qemu turns on the KVM_CAP_ARM_NISV_TO_USER and
> > then when it gets a NISV it always converts it to a data abort to the
> > guest. See kvm_arm_handle_dabt_nisv() in qemu. So it is just a
> > correctness issue, not a 'VM userspace can crash the VMM' security
> > problem.
> 
> The VMM wasn't my concern but rather a guest getting killed or not
> functioning correctly (user app killed).

Right, hopefully it is the latter.

> > Thus, IMHO, doing IO emulation for VFIO that doesn't support all the
> > instructions actual existing SW uses to do IO is hard to justify. We
> > are already on a slow path that only exists for technical correctness,
> > it should be perfect. It is perfect on x86 because x86 KVM does SW
> > instruction decode and emulation. ARM could too, but doesn't.
> 
> It could fall back to instruction decode, either in KVM or the VMM
> (strong preference for the latter), but I'd only do this if it's
> justified.

From a performance perspective, if the VMM is doing pure emulation and
wants to memcpy lots of data to emulated vMMIO I'd look at it like this:

  1xST4 transfers 512 bits and requires one vmexit and one
  instruction parse.

  4xSTP is four instruction parses and four vmexits

  8xSTR is no instruction parses and eight vmexits

The instruction parse is not pure overhead, it saves on vmexit's which
are expensive things (at least on x86). I don't have a sense how this
stacks up on arm, but I wouldn't jump to it being horribly
non-performing.

> I don't think the issue here is VFIO, I doubt we'd ever see emulation
> for hardware like mlx5.

Sadly no :(

It can happen in non-production corner cases due to the VFIO MSI emulation.

There is a qemu bug prior to 8.something that causes it to happen at
random, with VFIO, rarely.

There is a non-prodcution debug mode in qemu where all VFIO MMIO is
trapped. The qemu expectation is that this is functionally identical
to non-trapping. (The E in qemu is emulation after all, kind of a core
reason it exists)

Finally, we do actually have an internal simulation tool that does
software emulate mlx5 HW without VFIO.

> But we are changing generic kernel functions
> like memcpy_toio/__iowrite64_copy() that end up being used in other
> drivers (e.g. USB, UART) for emulated devices. 

I didn't touch memcpy_toio, I think given this problem we shouldn't
touch it. I only touched __iowriteXX_copy() which did not look like it
is being used in any drivers with emulation.

Even if I got this wrong we can revert any impacted drivers to use
memcpy_toio() instead.

Jason