Re: apparent race condition in mttcg memory handling

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

On 7/21/25 10:25 AM, Pierrick Bouvier wrote:
> On 7/21/25 10:14 AM, Michael Tokarev wrote:
>> On 21.07.2025 19:29, Pierrick Bouvier wrote:
>>> On 7/21/25 9:23 AM, Pierrick Bouvier wrote:
>> ..
>>>> looks like a good target for TSAN, which might expose the race without
>>>> really having to trigger it.
>>>> https://www.qemu.org/docs/master/devel/testing/main.html#building-and-
>>>> testing-with-tsan
>>
>> I think I tried with TSAN and it gave something useful even.
>> The prob now is to reproduce the thing by someone more familiar
>> with this stuff than me :)
>>
>>>> Else, you can reproduce your run using rr record -h (chaos mode) [1],
>>>> which randomly schedules threads, until it catches the segfault, and
>>>> then you'll have a reproducible case to debug.
>>>
>>> In case you never had opportunity to use rr, it is quite convenient,
>>> because you can set a hardware watchpoint on your faulty pointer (watch
>>> -l), do a reverse-continue, and in most cases, you'll directly reach
>>> where the bug happened. Feels like cheating.
>>
>> rr is the first thing I tried.  Nope, it's absolutely hopeless.   It
>> tried to boot just the kernel for over 30 minutes, after which I just
>> gave up.
>>
> 
> I had a similar thing to debug recently, and with a simple loop, I
> couldn't expose it easily. The bug I had was triggered with 3%
> probability, which seems close from yours.
> As rr record -h is single threaded, I found useful to write a wrapper
> script [1] to run one instance, and then run it in parallel using:
> ./run_one.sh | head -n 10000 | parallel --bar -j$(nproc)
> 
> With that, I could expose the bug in 2 minutes reliably (vs trying for
> more than one hour before). With your 64 cores, I'm sure it will quickly
> expose it.
> 
> Might be worth a try, as you need to only catch the bug once to be able
> to reproduce it.
> 
> [1] https://github.com/pbo-linaro/qemu/blob/master/try_rme.sh
>

In this script, I finally used qemu rr feature (as QEMU was working 
fine, but there was a bug in the software stack itself, that I wanted to 
investigate under gdbstub). But I was mentioning the same approach using 
rr (the tool).

>> Thanks,
>>
>> /mjt
>