Re:Re: [RFC DISCUSSION] virtio-rdma: RDMA device emulation for QEMU

["Xiong Weimin" <15927021679@163.com>]

[-- Attachment #1.1: Type: text/plain, Size: 17384 bytes --]

Hi, all


Attached are three short documents:
(1) Validation Summary — dual-host VM topology, V1.5 TCP full-size
perftest results, and reproduction commands;
(2) Correctness Note — a V2 completion-semantics bug we fixed and our
policy not to report throughput without zero backend loss;
(3) Upstream Gap Checklist — IOTLB/ACCESS_PLATFORM and other blockers,
with priorities and a proposed upstream timeline.


Hi Serapheim, Stefan, and all,


Following up on the March thread — especially Stefan's points about
long-term maintenance and registered memory stepping outside the VIRTIO
device model — I wanted to share a concrete progress update from the
vhost-user-rdma / DPDK reference stack I mentioned in netdev [1].


Apologies for the quiet period since March. Rather than sending an
early "we have something" note, we spent the intervening months on
end-to-end bring-up and measurement: dual-host VM topology, full
guest/QEMU/rdma-core/DPDK stack integration, standard perftest
validation, and reproducible performance baselines. We wanted this
follow-up to come with evidence, not just intent — especially given
the history of virtio-rdma attempts that stalled after an initial RFC.


This is not an RFC patch series yet. The goal of this mail is to show
what we have validated today, what we deliberately invalidated, and
what we plan to change before sending driver patches.


= Maintenance commitment =


> I think this is a sign that virtual RDMA has a small community without
> someone willing to maintain it over the long term. Can you see
> yourself actively maintaining this over the coming years?


Yes. I (Weimin Xiong, xiongweimin@kylinos.cn) am actively maintaining
this stack at Kylinos/Kylinsoft, with product use cases behind it.
We are not asking for merge yet; we are asking for design feedback on
the IOTLB-based MR model before we start splitting patches.


We plan tagged reference releases with reproducible test bundles
(topology, perftest commands, CSV summaries).


= What is working today =


We now have an end-to-end stack:


  guest: linux drivers/infiniband/hw/virtio/
  userspace: rdma-core virtio provider
  VMM: QEMU vhost-user-rdma device
  backend: DPDK vhost-user backend (out-of-tree reference impl)


Validation topology:


  Host 222 (DPDK backend) -- VM 222 (vrdma0, 10.0.0.12)
          |
          | inter-host fabric (TCP in baseline; DPDK raw in experiments)
          |
  Host 223 (DPDK backend) -- VM 223 (vrdma0, 10.0.0.11)


All tests below use standard perftest tools (`ibv_devinfo`,
`ib_write_bw`, `ib_read_bw`) over RC QPs — not custom smoke tests.


Our virtqueue layout uses per-CQ completion virtqueues (not a single
shared completion VQ), which matches the direction several of us had
discussed for async CQ behavior.


Attached: `VIRTIO_RDMA_VALIDATION_SUMMARY.md` with topology, V1.5 TCP
full-size reference numbers, and copy-paste perftest commands.


= V1.5 TCP full-size reference (best credible run, not inflated) =


Our upstream-facing performance set is the V1.5 **TCP fabric** path
(`--vrdma-mode v15`), not the experimental V2 DPDK-raw backend.


Best full-size sweep (2026-06-17, 222 client -> 223 server, all rc=0):


  Size WRITE avg READ avg
  64 B 1.02 Gb/s 0.61 Gb/s
  512 B 5.94 Gb/s 4.46 Gb/s
  4 KB 17.51 Gb/s 13.88 Gb/s
  64 KB 27.50 Gb/s 19.99 Gb/s


Complete 64 B .. 64 KB table and CSV are in the attachment
(`V15_TCP_FULLSIZE_REFERENCE.csv`). We do **not** use V2 raw-fabric
numbers that showed 70-99 Gb/s with receiver-side fragment loss.


= Correctness work we want to highlight =


While tuning an experimental DPDK-raw fabric backend on 2026-06-25,
we found that 64 KB WRITE could report ~72 / ~99 Gb/s while the
receiver dropped millions of fragments (`imissed`, `frag_oom`,
`frag_drop`). Root cause: local completion before remote acknowledgment.


We fixed this in the reference backend — V2 WRITE now requires remote
response by default — and we **withdrew** the inflated numbers.


Post-fix trustworthy 64 KB WRITE (zero loss counters on both hosts):


  222 -> 223: 30.58 Gb/s
  223 -> 222: 16.48 Gb/s (bidirectional symmetry still being tuned)


Attached: `VIRTIO_RDMA_CORRECTNESS_NOTE.md`.


Policy going forward: we only report performance numbers accompanied
by backend loss counters at zero (`imissed`, `frag_drop`, `frag_oom`).


= VIRTIO device model / IOTLB =


> Accessing memory outside of virtqueue buffers is a violation of the
> VIRTIO device model.


Acknowledged. Our current reference implementation still registers MRs
via GPA direct mapping in the vhost-user backend. This is exactly why
we are **not** sending driver patches yet.


We agree with Serapheim's option [C] from the 2026-03-20 reply:


  REG_USER_MR sends GPA page list on command VQ
  -> device/backend establishes mappings via IOTLB
  -> WRITE/READ resolve through those mappings
  -> DEREG_MR invalidates


`VRDMA_VHOST_USER_IOTLB_MSG` is already defined in our backend headers
but not implemented. Guest `vrdma_ib.c` still uses DMA addresses today.


Attached: `VIRTIO_RDMA_UPSTREAM_GAP_CHECKLIST.md` with P0/P1 items and
target dates.


We would appreciate feedback on:


  1. Is option [C] the right v1 MR model for vhost-user backends?
  2. Should we pursue virtio-comment device ID reservation before or
     after the guest-driver RFC?
  3. Preferred patch order: spec draft first, or Linux guest driver
     prototype on an out-of-tree branch first?


= Scope boundary (what we will NOT upstream as-is) =


To avoid repeating the ByteDance effort's "big private stack" problem,
we are explicitly keeping these out of the first upstream series:


  - DPDK raw fabric (custom ethertype 0xA55A)
  - single-port paired-net product binding logic
  - cross-backend TCP broker modes
  - netdev same-slot fallback hack


The first upstream target is the **standard virtio-rdma core**:
config/queues/commands, guest driver, QEMU device model, and a minimal
reference backend path — with IOTLB + ACCESS_PLATFORM.


= Next steps from our side =


  - 2026-07: publish reproducibility bundle (commands + CSV + topology)
  - 2026-07 .. 08: IOTLB MR prototype branch (guest + vhost-user backend)
  - 2026-09 .. 10: RFC draft for virtio-comment review
  - 2026-11+: split patch series (Linux guest first)


Happy to post an out-of-tree branch link or walk through the test
setup on a call if useful.


Thanks,
Weimin Xiong
Kylinos / Kylinsoft
xiongweimin@kylinos.cn
At 2026-03-21 01:23:45, "Serapheim Dimitropoulos" <serapheimd@gmail.com> wrote:
>Hi Stefan,
>
>Thank you for the quick reply and thorough review! I waited a few days
>before replying to see if any of the folks from the previous virtio-rdma
>efforts would chime in but nothing so far. In any case, my responses
>inlined below.
>
>> I think this is a sign that virtual RDMA has a small community without
>> someone willing to maintain it over the long term. Can you see
>> yourself actively maintaining this over the coming years?
>>
>> If not, then it may be more appropriate to treat it as an experimental
>> and out-of-tree project. That way the spec and code can be shared in
>> case others want to build on it in the future without any commitment
>> or the overhead of going through the full process of getting a device
>> merged into the VIRTIO spec, QEMU emulating code merged, and Linux
>> guest code merged.
>>
>> If you are going to ship products that rely on this, then it's
>> probably necessary to go through the full process of getting
>> everything merged upstream.
>
>I understand the hesitation given the pvrdma precedent. I want to be
>explicit that this is not a side-project. I'm a kernel engineer at
>CoreWeave and having something like virtio-rdma is a requirement for
>some of our current projects. Two of them that I feel comfortable
>disclosing is our work with BlueField DPUs and Kata containers.
>
>For NVIDIA BlueFields we currently do most of our work with real
>hardware which for us developers is at times hard to come by as
>we want to make sure that our customers take priority. Being able
>to do emulation with QEMU means that every engineer on the team
>can iterate without a dedicated BlueField. Moreover, our test suite
>can run more often this way too.
>
>For Kata containers today, to get RDMA you have two options. Either
>use virtio-net and give up latency or vfio-pci passthrough pinning
>the pod to a NIC. The latter not only breaks the security model but
>also isolation (not to mention any prospect of live-migration).
>
>I'm ok keeping this as an out-of-tree project initially in the short
>term but would hate it if something else comes along later and we
>have to re-work everything on our end. As far as long-term commitment
>goes I commit to maintaining virtio-rdma for as long as it's upstream.
>If I ever leave my role at CoreWeave and my next role is not related
>to virtio-rdma, my team at CoreWeave will designate a successor
>maintainer as it has organizational interest in this work. I'm happy
>to formalize my commitment in a MAINTAINERS entry when/if the time
>comes.
>
>> Does this mean that memory is registered on VQ 0 and incoming RDMA
>> WRITE (without immediate) requests modify that memory directly without
>> virtqueue activity? I think this is necessary because registered
>> memory is available continuously and the virtqueue model doesn't
>> really work for this mode of operation.
>>
>> It's worth clarifying this because accessing memory outside of
>> virtqueue buffers is a violation of the VIRTIO device model. That's
>> okay, VIRTIO is pragmatic and some devices do this but it's worth
>> mentioning explicitly.
>>
>> Stepping outside the VIRTIO device model can create implementation
>> challenges because interfaces like vDPA/VDUSE may not be designed for
>> it though.
>
>Ok great point actually. Evaluating the potential paths forward I
>thought of the following (though I'm open to other ideas if you have
>them):
>
>A] Use a shared memory window (like virtio-fs DAX) - this wouldn't
>work because it changes the RDMA programming model. Real HCAs let
>you register *any* part of memory via ibv_reg_mr(). Restricting MRs
>to a pre-allocated window would thus break standard applications.
>
>B] Just acknowledge the deviation explicitly and move - as you said
>besides being a spec violation it doesn't concretely solve the
>vDPA/VDUSE case as they may want to enforce IOMMU boundaries.
>
>C] Go the IOTLB route - when the driver registers an MR, the device
>triggers IOTLB updates for every page in the MR giving the backend
>legal IOMMU mappings.
>
>Let me know if you can think of any other ways but I believe [C]
>may be the way to go as real HCAs do the same thing. In our case
>this would look like so:
>
>1. REG_USER_MR sends the page list (guest physical addresses) via
>   the command VQ (virtio-compliant command).
>
>2. The device uses the platform's DMA mapping mechanism to establish
>   mappings for each page in the MR. I believe for QEMU that would be
>   address_space_map() since it has full guest RAM access.
>   (VHOST_USER_IOTLB_MSG for vhost-user and VDUSE_IOTLB_REG_UMEM for
>   VDUSE).
>
>3. RDMA WRITE/READ resolve (remote_addr + rkey) through those mappings.
>
>4. DEREG_MR invalidates them.
>
>The above should require VIRTIO_F_ACCESS_PLATFORM when used with
>IOMMU-protected backends like vDPA/VDUSE. As for the per-page mapping
>cost at registration I'm open to ideas but I wonder if it is acceptable
>for the v1 pass as it is a one time cost.
>
>One potential future scalability issue with the flat list is that
>for very large MRs (128GB/32M page addresses) it can become too
>long/heavy in the command VQ which could be prohibitive for any
>potential hardware implementations (if there were to be any). For
>v1 maybe we could just reserve VIRTIO_RDMA_F_INDIRECT_MR and leave
>room for a future indirect page table model?
>
>Let me know how the above sound to you and I can make sure to document
>them more formally in the spec draft.
>
>> [...] check how registered memory can be implemented both in VDUSE
>> and in-kernel vDPA drivers.
>
>The IOTLB model above should cover both cases but I can double-check.
>For VDUSE, MR registration triggers VDUSE_IOTLB_REG_UMEM calls. For
>in-kernel vDPA, the vDPA bus provides DMA mapping APIs that map to the
>parent IOMMU. The guest driver should ideally be unaware of which
>backend is in use and just send REG_USER_MR with the device handling
>the rest.
>
>> For the userspace virtio-rdma device implementation I expected a new
>> UNIX domain socket protocol along the lines of vhost-user and
>> vfio-user. That's because sharing guest RAM is only part of the
>> communication that must happen between two QEMUs and I guess you'll
>> need to define your own protocol to coordinate RDMA between QEMU
>> processes anyway.
>>
>> When using vDPA or VDUSE, QEMU shares guest RAM with the device
>> through the /dev/vhost ioctls.
>>
>> In both cases, I'm not sure if ivshmem is necessary.
>
>Makes sense - thank you for the pointers! I'm almost done switching
>to domain sockets per your recommendation. The new scheme is currently
>peer-to-peer (not a re-use of vhost-user which is VMM-to-backend). As
>a first phase/stage each side exchanges MEM_REGIONS messages with
>memfd descriptors for guest RAM regions and the peer mmap()s them
>(handshake). Then we forward send/recv via framed messages on the
>socket. RDMA WRITE/READ operate directly on the mmap'd peer memory
>(no message nor remote CPU involvement).
>
>BTW I don't need to add that level of detail in my spec, correct?
>From what I can tell specs seem to define device-to-driver behavior
>only (e.g. virtio-net doesn't say anything about TAP/vhost-user,
>etc.)
>
>> It depends what you mean by async. Virtqueues can complete requests
>> out-of-order, so a separate completion virtqueue is not needed from
>> that perspective.
>>
>> There could be other reasons why a separate completion virtqueue makes
>> sense. If RDMA relies on the separate CQ design to emit multiple CQEs
>> for the same request or emits CQEs not associated with any request,
>> then a single virtqueue won't work. I don't know RDMA well enough to
>> say either way.
>
>ok looking more at RDMA CQ semantics and assuming I understand what you
>propose correclty, I do believe we need the separate queue for the
>following reasons:
>
>- An RDMA application may create one CQ and bind multiple QPs to it
>  (multi-QP fan-in). ibv_poll_cq() returns completions from all
>  associated QPs in one call. If completions live in per-VQ used
>  rings, then polling means scanning 2N VQs, O(n) per poll. It also
>  seems like a mismatch in the CQ abstraction being a single
>  aggregation point and virtio VQ a per-queue used ring. The dedicated
>  completion VQ gives you the fan-in O(1) for free.
>
>- The RDMA spec mandates that when a CQ overflows the device raises
>  IBV_EVENT_CQ_ERR, which cascades to IBV_EVENT_QP_FATAL on every QP
>  bound to that CQ. The device must be able to detect overflows to
>  trigger this. Detecting that with a dedicated completion VQ is
>  straightforward. The virtio used ring on the other hand doesn't have
>  any overflow semantics from what I can tell so we'd need to make
>  something ourselves.
>
>- ibv_req_notify_cq(solicited_only=1) fires only for CQEs with the
>  solicited flag set. I'm not sure how we could express this as-is
>  when virtio event index suppresses by count, rather than per-WR
>  flag. This is less critical though as we could work around it
>  by always notifying and having the driver filter in its interrupt
>  handler in order to be functionally correct.
>
>Let me know if the above reasons seem legitimate on your end. It
>generally seems to me the used ring has no place to put whatever
>fields are needed for an RDMA CQE (only handles generic completion
>metadata). So using a dedicated one to return buffers that contain
>CQEs makes more sense that trying to encode CQE data in the used
>ring itself.
>
>> Writing C devices is still perfectly acceptable in QEMU. With Rust you
>> are likely to have to work on bindings and may hit issues just because
>> Rust is new in QEMU, but it's there and you can absolutely use it.
>
>Yeah that's what it seemed like when I checked. I'd like to keep things
>simple as a first pass and make sure we get something working while
>monitoring for any Rust PCI+DMA bindings for future versions of this.
>
>> I think vhost-vpda-device-pci is good practice. It makes sure the
>> device is self-contained and doesn't rely on device-specific VMM
>> support. If it's possible to use just vhost-vdpa-device-pci, then
>> that's great. It scales better because it avoids the need to implement
>> a device in every VMM (like QEMU, Firecracker, etc).
>
>Great! Will proceed with that.
>
>> You can reserve a device ID from the VIRTIO Technical Committee
>> separately from getting the spec merged. Ask
>> virtio-comment@lists.linux.dev.
>
>Will do!
>
>> It is helpful to see the draft VIRTIO spec and RFC patches at the same
>> time. So as soon as you want to discuss the specifics of the VIRTIO
>> spec patches it would be a good time to send RFC patches showing how
>> the spec is implemented.
>
>Will do the above shortly. I definitely need to revise my spec and code
>first to incorporate some of the above. I really appreciate your
>feedback on this!
>
>Serapheim

[-- Attachment #1.2: Type: text/html, Size: 35293 bytes --]

[-- Attachment #2: VIRTIO_RDMA_UPSTREAM_GAP_CHECKLIST.pdf --]
[-- Type: application/pdf, Size: 12844 bytes --]

[-- Attachment #3: VIRTIO_RDMA_CORRECTNESS_NOTE.pdf --]
[-- Type: application/pdf, Size: 9785 bytes --]

[-- Attachment #4: VIRTIO_RDMA_VALIDATION_SUMMARY.pdf --]
[-- Type: application/pdf, Size: 13912 bytes --]