Re: [PATCH v14 29/44] arm64: RMI: Runtime faulting of memory

[Gavin Shan <gshan@redhat.com>]

On 6/26/26 1:58 AM, Suzuki K Poulose wrote:
> On 25/06/2026 14:53, Gavin Shan wrote:
>> On 6/6/26 12:35 AM, Lorenzo Pieralisi wrote:
>>> On Fri, Jun 05, 2026 at 06:11:11PM +1000, Gavin Shan wrote:
>>>> On 6/5/26 5:28 PM, Lorenzo Pieralisi wrote:
>>>>> On Fri, Jun 05, 2026 at 04:23:15PM +1000, Gavin Shan wrote:

[...]

>>>>
>>>> I tried to rebase Jean's latest QEMU series [1] to upstream QEMU, and found
>>>> that memory slots backed by THP are broken. With THP disabled on the host and
>>>> other fixes (mentioned in my prevous replies) applied on the top of this (v14)
>>>> series, I'm able to boot a realm guest with rebased QEMU series [2], plus more
>>>> fxies on the top.
>>>>
>>>> [1] https://git.codelinaro.org/linaro/dcap/qemu.git  (branch: cca/ latest)
>>>> [2] https://git.qemu.org/git/qemu.git                (branch: cca/gavin)
>>>>
>>>> Lorenzo, You may be saying there is someone making QEMU to support ARM/CCA?
>>>
>>> Mathieu and I are working on that yes and with Steven/Suzuki to fix the THP
>>> issues you pointed out above.
>>>
>>>> If so, I'm not sure if there is a QEMU repository for me to try?
>>>
>>> We should be able to submit patches by end of June - we shall let you know
>>> whether we can make something available earlier.
>>>
>>
>> Not sure if there are other known issues in this series. It seems the stage2
>> page fault handling on the shared space isn't working well. In my test, the
>> vring (struct vring_desc) of virtio-net-pci is updated by the guest, and the
>> data isn't seen by QEMU, I'm suspecting if the host-page-frame-number is properly
>> resolved in the s2 page fault handler for shared (unprotected) space.
>>
>> - I rebased Jean's latest qemu branch to the upstream qemu;
>>
>> - On the host, which is emulated by qemu/tcg, the THP (transparent huge page) is
>>    disabled.
>>
>> - On the guest, I can see the virtio vring (struct vring_desc) is updated. The
>>    S1 page-table entry looks correct because the corresponding physical address
>>    0x10046880000 is a sane shared (unprotected) space address.
>>
>>    [   52.094143] software IO TLB: Memory encryption is active and system is using DMA bounce buffers
>>    [   52.289746] virtqueue_add_desc_split: desc[0]@0xffff000006880000, [00000100b983f000  00000640  0002  0001]
>>    [   52.432150] PTE 0x00e8010046880707 at address 0xffff000006880000
>>
>> - On the host, the s2 page-table-entry is unmapped due to attribute transition (private -> shared).
>>    A subsequent S2 page fault is raised against the adress and the s2 page-table-entry is built.
>>
>>    [  109.259077] ====> realm_unmap_shared_range: tracked_unprot_addr=0x10046880000
>>    [  109.260249] realm_unmap_shared_range: unmapped shared range at 0x10046880000
>>    [  109.317786] realm_unmap_shared_range: unmapped shared range at 0x10046880000
>>    [  109.629939] ====> kvm_handle_guest_abort: fault_ipa=0x10046880000, esr=0x92000007
>>    [  109.630245] realm_map_non_secure: ipa=0x10046880000, pfn=0xb8b59, size=0x1000, prot=0xf
>>    [  109.630331] realm_map_non_secure: ipa=0x10046880000, ipa_top=0x10046881000, flags=0x1e0001, range_desc=0xb8b59004
> 
> Are you able to correlate the order of the transitions and the Guest
> access with RMM log ? We haven't seen this from our end. We are aware
> of permission fault issues with Unprotected IPA when backing the memslot
> with MAP_PRIVATE areas. But this looks different.
> 
> Lorenzo, have you run into this ?
> 

It's hard to correlate the order since the logs are collected from two separate
consoles. For the write permission, I add code to the host where the permission
is always added for all s2 page faults in the shared space. Otherwise, qemu can
be killed by -EFAULT or similar error.

There are more findings after more experiments: this virtio-net-pci device has 3
queues or vrings (Rx/Tx/Ctrl). The Rx/Tx/Ctrl queue are populated in order one after
one. In the guest kernel, I intentionally write fixed data (0x0123456789abcdef) to
the first 8 bytes of the queue when it gets populated, and stop the guest at random
points to see if the data is gone. I found that the data written to Rx/Tx queue are
lost after Ctrl queue is allocated.

The data written to Rx/Tx queue is lost if the guest stops (B). The data written to
Rx/Tx queue isn't lost if the guest stops at (A). I can see the pattern (0x0123...cdef)
by dumping the physcial memory through 'pmemsave' command in qemu.

DMA allocation
==============
dma_alloc_coherent
   dma_alloc_attrs
     dma_direct_alloc
       __dma_direct_alloc_pages
       dma_set_decrypted                    // (A) No data lost if being stopped here for the Ctrl queue
       memset(ret, 0, size)                 // (B) Data lost after being stopped after memset() for the Ctrl queue

The memset() on the Ctrl queue should trigger a stage2 page fault. It seems the page
fault enforces the shared pages for Rx/Tx queue to be dropped? I need to add more
debugging code and track it down.

> Suzuki
> 
> 
>>
>> - On QEMU, the updated vring (struct vring_desc) at GPA 0x46880000 isn't seen. All the
>>    data in that adress are zeros.
>>
>>    ====> virtqueue_split_pop: vdev=<virtio-net>, sz=0x38, queue_index=0x0, vq->vring.num=0x100
>>    virtqueue_split_pop: last_avail_idx=0x0, head=0x0
>>    address_space_read_cached_slow: cache@0xffff1c036440, addr=0x0, buf=0xffffeee34880, len=0x10
>>    address_space_read_cached_slow: cache: ptr=0x0, xlat=0x10046880000, len=0x1000, mrs=<realm-dma-region>, is_write=no
>>    address_space_read_cached_slow: translated to mr=<mach-virt.ram>, mr_addr=0x6880000, l=0x10
>>    flatview_read_continue_step: mr=<mach-virt.ram>, host=0xffff23e00000, mr_addr=0x6880000, ram_ptr=0xffff2a680000
>>    virtqueue_split_pop: desc: 0000000000000000 - 00000000 - 00000000 - 00000000
>>    qemu-system-aarch64: virtio: zero sized buffers are not allowed
>>
>>
Thanks,
Gavin