Re: MMIO through IOMMU from a TCG processor

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

Hi Mark,

"recorded section therefore seems to be incorrect".
do you observe a crash, or on assert failing at execution?

I don't know in details the code you mention, but after investigating 
and fixing https://gitlab.com/qemu-project/qemu/-/issues/3040, I can 
share a few things.

Overall, what you describe looks like a race condition exposing a 
lifetime issue, especially when saying "we 'loose' the address space 
that has been returned by the translate function".
A value was not updated as expected and is out of sync, or was freed too 
early. Memory regions lifetime is something definitely tricky in QEMU, 
and when you mix that with RCU, things can become very obscure in 
multithreaded scenarios.

In the bug above, the solution was to stop duplicating this information, 
and get it from the same source. The overhead to read such atomic data 
is quite small, thanks to use of RCU.
At KVM Forum, Paolo told me he introduced this copy precisely to avoid 
issues, but the opposite happened in reality, which we both found was 
quite funny.

Additional questions:
- At which time of execution does it happen? Is it during pci devices 
initialization, or when remapping specific memory sections?
- Is the bug deterministic or random? If random, does increasing the 
number of pci devices attached increase the probably to meet it?

Additional tools:
- If you observe a crash, build with asan. If you get a use-after-free 
error, it's probably an issue with RCU cleaning up things before you 
expect. This is what I had in the bug mentioned above.
- If your assert fail, I can recommend you capture execution through rr 
(https://github.com/rr-debugger/rr), using chaos mode rr record --chaos, 
which will randomize scheduling of threads. I don't know if you're 
familiar with it, but it allows you to debug your execution "backward".
Once you captured the faulty execution, you can reach the crash or 
faulty assert, then execute backward (reverse-continue) with a 
watchpoint set on the (correct) value that was updated meanwhile. This 
way, you'll find which sequence led to desynchronization, and then 
you'll have a good start to deduce what the root cause is.
- Spend some time making the crash/assert almost deterministic, it will 
save you time later, especially when implementing a possible fix and 
prove it works.

I hope it helps.

Regards,
Pierrick

On 10/9/25 2:10 AM, Mark Burton wrote:
> 
> (Adding Pierrick)
> Thanks for getting back to me Mark.
> 
> I initially thought the same, and I think I have seen that issue, I have also taken that patch, However …..
> 
> For MMIO access, as best I can tell, the initial calculation of the despatch is based on the iotlb reported by the translate function (correct), while the subsequent use of the section number uses the dispatch table from the CPU’s address space….. which gives you the wrong section.
> 
> I would very happily do a live debug with you (or anybody) if it would help… I’m more than willing to believe I’ve made a mistake, but I just don’t see how it’s supposed to work.
> 
> I have been looking at solutions, and right now, I don’t see anything obvious. As best I can tell, we “loose” the address space that has been returned by the translate function - so, either we would need a way to hold onto that, or, we would have to re-call the function, or….
> All of those options look really really nasty to me.
> 
> The issue is going to be systems where SMMU’s are used all over the place, specifically, in front of MMIO. (Memory works OK because we get the memory pointer itself, all is fine, the issue seems only be with MMIO accesses through IOMMU regions).
> 
> Cheers
> Mark.
> 
> 
>> On 9 Oct 2025, at 10:43, Mark Cave-Ayland <mark.caveayland@nutanix.com> wrote:
>>
>> On 08/10/2025 13:38, Mark Burton wrote:
>>
>>> All, sorry for the wide CC, I’m trying to find somebody who understands this corder of the code…. This is perhaps a obscure, but I think it should work.
>>> I am trying to access an MMIO region through an IOMMU, from TCG.
>>> The IOMMU translation has provided an address space that is different from the CPU’s own address space.
>>> In address_space_translate_for_iotlb the section is calculated using the address space provide by the IOMMU translation.
>>>> d = flatview_to_dispatch(address_space_to_flatview(iotlb.target_as));
>>>>
>>> Later, we come to do the actual access (via e.g. do_st_mmio_leN), and at this point we pick up the cpu’s address spaces in iotlb_to_section, which is different, and the recorded section therefore seems to be incorrect.
>>>> CPUAddressSpace *cpuas = &cpu->cpu_ases[asidx];
>>>> AddressSpaceDispatch *d = cpuas->memory_dispatch;
>>>> int section_index = index & ~TARGET_PAGE_MASK;
>>>> MemoryRegionSection *ret;
>>>>
>>>> assert(section_index < d->map.sections_nb);
>>>> ret = d->map.sections + section_index;
>>> What I don’t fully understand is how this is supposed to work….?
>>> Have I missed something obvious?
>>> Cheers
>>> Mark.
>>
>> What version of QEMU are you using? I'm wondering if you're getting caught out by a variant of this: https://gitlab.com/qemu-project/qemu/-/issues/3040.
>>
>>
>> ATB,
>>
>> Mark.
>>
>