Re: [PATCH v14 29/44] arm64: RMI: Runtime faulting of memory

[Suzuki K Poulose <suzuki.poulose@arm.com>]

On 26/06/2026 08:43, Gavin Shan wrote:
> On 6/26/26 1:58 AM, Suzuki K Poulose wrote:
>> On 25/06/2026 14:53, Gavin Shan wrote:
>>> On 6/6/26 12:35 AM, Lorenzo Pieralisi wrote:
>>>> On Fri, Jun 05, 2026 at 06:11:11PM +1000, Gavin Shan wrote:
>>>>> On 6/5/26 5:28 PM, Lorenzo Pieralisi wrote:
>>>>>> On Fri, Jun 05, 2026 at 04:23:15PM +1000, Gavin Shan wrote:
> 
> [...]
> 
>>>>>
>>>>> I tried to rebase Jean's latest QEMU series [1] to upstream QEMU, 
>>>>> and found
>>>>> that memory slots backed by THP are broken. With THP disabled on 
>>>>> the host and
>>>>> other fixes (mentioned in my prevous replies) applied on the top of 
>>>>> this (v14)
>>>>> series, I'm able to boot a realm guest with rebased QEMU series 
>>>>> [2], plus more
>>>>> fxies on the top.
>>>>>
>>>>> [1] https://git.codelinaro.org/linaro/dcap/qemu.git  (branch: cca/ 
>>>>> latest)
>>>>> [2] https://git.qemu.org/git/qemu.git                (branch: cca/ 
>>>>> gavin)
>>>>>
>>>>> Lorenzo, You may be saying there is someone making QEMU to support 
>>>>> ARM/CCA?
>>>>
>>>> Mathieu and I are working on that yes and with Steven/Suzuki to fix 
>>>> the THP
>>>> issues you pointed out above.
>>>>
>>>>> If so, I'm not sure if there is a QEMU repository for me to try?
>>>>
>>>> We should be able to submit patches by end of June - we shall let 
>>>> you know
>>>> whether we can make something available earlier.
>>>>
>>>
>>> Not sure if there are other known issues in this series. It seems the 
>>> stage2
>>> page fault handling on the shared space isn't working well. In my 
>>> test, the
>>> vring (struct vring_desc) of virtio-net-pci is updated by the guest, 
>>> and the
>>> data isn't seen by QEMU, I'm suspecting if the host-page-frame-number 
>>> is properly
>>> resolved in the s2 page fault handler for shared (unprotected) space.
>>>
>>> - I rebased Jean's latest qemu branch to the upstream qemu;
>>>
>>> - On the host, which is emulated by qemu/tcg, the THP (transparent 
>>> huge page) is
>>>    disabled.
>>>
>>> - On the guest, I can see the virtio vring (struct vring_desc) is 
>>> updated. The
>>>    S1 page-table entry looks correct because the corresponding 
>>> physical address
>>>    0x10046880000 is a sane shared (unprotected) space address.
>>>
>>>    [   52.094143] software IO TLB: Memory encryption is active and 
>>> system is using DMA bounce buffers
>>>    [   52.289746] virtqueue_add_desc_split: 
>>> desc[0]@0xffff000006880000, [00000100b983f000  00000640  0002  0001]
>>>    [   52.432150] PTE 0x00e8010046880707 at address 0xffff000006880000
>>>
>>> - On the host, the s2 page-table-entry is unmapped due to attribute 
>>> transition (private -> shared).
>>>    A subsequent S2 page fault is raised against the adress and the s2 
>>> page-table-entry is built.
>>>
>>>    [  109.259077] ====> realm_unmap_shared_range: 
>>> tracked_unprot_addr=0x10046880000
>>>    [  109.260249] realm_unmap_shared_range: unmapped shared range at 
>>> 0x10046880000
>>>    [  109.317786] realm_unmap_shared_range: unmapped shared range at 
>>> 0x10046880000
>>>    [  109.629939] ====> kvm_handle_guest_abort: 
>>> fault_ipa=0x10046880000, esr=0x92000007
>>>    [  109.630245] realm_map_non_secure: ipa=0x10046880000, 
>>> pfn=0xb8b59, size=0x1000, prot=0xf
>>>    [  109.630331] realm_map_non_secure: ipa=0x10046880000, 
>>> ipa_top=0x10046881000, flags=0x1e0001, range_desc=0xb8b59004
>>
>> Are you able to correlate the order of the transitions and the Guest
>> access with RMM log ? We haven't seen this from our end. We are aware
>> of permission fault issues with Unprotected IPA when backing the memslot
>> with MAP_PRIVATE areas. But this looks different.
>>
>> Lorenzo, have you run into this ?
>>
> 
> It's hard to correlate the order since the logs are collected from two 
> separate
> consoles. For the write permission, I add code to the host where the 
> permission
> is always added for all s2 page faults in the shared space. Otherwise, 
> qemu can
> be killed by -EFAULT or similar error.

This is the problem. We can't add WRITE permission by default. I believe
you may have MAP_PRIVATE mapping and it has to be mapped as READ only
and on a permission fault, we replace it with a writable page. By
overriding the WRITE permission, you let the guest write to a page
that may not be seen by the VMM.

We identified this as a bug in the KVM driver in this series (reported
by Lorenzo) and there is a corresponding tf-RMM change that is required
to get this working. So, please could you wait until the next series
when this will be addressed ? Or you could switch to using MAP_SHARED
for the "shared" memory in the memslot.


Suzuki


> 
> There are more findings after more experiments: this virtio-net-pci 
> device has 3
> queues or vrings (Rx/Tx/Ctrl). The Rx/Tx/Ctrl queue are populated in 
> order one after
> one. In the guest kernel, I intentionally write fixed data 
> (0x0123456789abcdef) to
> the first 8 bytes of the queue when it gets populated, and stop the 
> guest at random
> points to see if the data is gone. I found that the data written to Rx/ 
> Tx queue are
> lost after Ctrl queue is allocated.
> 
> The data written to Rx/Tx queue is lost if the guest stops (B). The data 
> written to
> Rx/Tx queue isn't lost if the guest stops at (A). I can see the pattern 
> (0x0123...cdef)
> by dumping the physcial memory through 'pmemsave' command in qemu.
> 
> DMA allocation
> ==============
> dma_alloc_coherent
>    dma_alloc_attrs
>      dma_direct_alloc
>        __dma_direct_alloc_pages
>        dma_set_decrypted                    // (A) No data lost if being 
> stopped here for the Ctrl queue
>        memset(ret, 0, size)                 // (B) Data lost after being 
> stopped after memset() for the Ctrl queue
> 
> The memset() on the Ctrl queue should trigger a stage2 page fault. It 
> seems the page
> fault enforces the shared pages for Rx/Tx queue to be dropped? I need to 
> add more
> debugging code and track it down.
> 
>> Suzuki
>>
>>
>>>
>>> - On QEMU, the updated vring (struct vring_desc) at GPA 0x46880000 
>>> isn't seen. All the
>>>    data in that adress are zeros.
>>>
>>>    ====> virtqueue_split_pop: vdev=<virtio-net>, sz=0x38, 
>>> queue_index=0x0, vq->vring.num=0x100
>>>    virtqueue_split_pop: last_avail_idx=0x0, head=0x0
>>>    address_space_read_cached_slow: cache@0xffff1c036440, addr=0x0, 
>>> buf=0xffffeee34880, len=0x10
>>>    address_space_read_cached_slow: cache: ptr=0x0, 
>>> xlat=0x10046880000, len=0x1000, mrs=<realm-dma-region>, is_write=no
>>>    address_space_read_cached_slow: translated to mr=<mach-virt.ram>, 
>>> mr_addr=0x6880000, l=0x10
>>>    flatview_read_continue_step: mr=<mach-virt.ram>, 
>>> host=0xffff23e00000, mr_addr=0x6880000, ram_ptr=0xffff2a680000
>>>    virtqueue_split_pop: desc: 0000000000000000 - 00000000 - 00000000 
>>> - 00000000
>>>    qemu-system-aarch64: virtio: zero sized buffers are not allowed
>>>
>>>
> Thanks,
> Gavin
>