Re: [PATCH 066/147] include/exec: Move TLB_MMIO, TLB_DISCARD_WRITE to slow flags

Re: [PATCH 066/147] include/exec: Move TLB_MMIO, TLB_DISCARD_WRITE to slow flags

[Jonathan Cameron]

On Tue, 29 Apr 2025 19:43:05 -0700
Richard Henderson <richard.henderson@linaro.org> wrote:

> On 4/29/25 14:35, Alistair Francis wrote:
> > On Sat, Apr 26, 2025 at 3:36 AM Jonathan Cameron via
> > <qemu-devel@nongnu.org> wrote:  
> >>
> >> On Tue, 22 Apr 2025 12:26:55 -0700
> >> Richard Henderson <richard.henderson@linaro.org> wrote:
> >>  
> >>> Recover two bits from the inline flags.  
> >>
> >>
> >> Hi Richard,
> >>
> >> Early days but something (I'm fairly sure in this patch) is tripping up my favourite
> >> TCG corner case of running code out of MMIO memory (interleaved CXL memory).
> >>
> >> Only seeing it on arm64 tests so far which isn't upstream yet..
> >> (guess what I was getting ready to post today)
> >>
> >> Back trace is:
> >>
> >> #0  0x0000555555fd4296 in cpu_atomic_fetch_andq_le_mmu (env=0x555557ee19b0, addr=18442241572520067072, val=18446744073701163007, oi=8244, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:140
> >> #1  0x00007fffb6894125 in code_gen_buffer ()
> >> #2  0x0000555555fc4c46 in cpu_tb_exec (cpu=cpu@entry=0x555557ededf0, itb=itb@entry=0x7fffb6894000 <code_gen_buffer+200511443>, tb_exit=tb_exit@entry=0x7ffff4bfb744) at ../../accel/tcg/cpu-exec.c:455
> >> #3  0x0000555555fc51c2 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfb744, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffb6894000 <code_gen_buffer+200511443>, cpu=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:904
> >> #4  cpu_exec_loop (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1018
> >> #5  0x0000555555fc58f1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1035
> >> #6  0x0000555555fc5f6c in cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:1061
> >> #7  0x0000555556146ac3 in tcg_cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops.c:81
> >> #8  0x0000555556146ee3 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
> >> #9  0x00005555561f6450 in qemu_thread_start (args=0x555557f8f430) at ../../util/qemu-thread-posix.c:541
> >> #10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
> >> #11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
> >>
> >> I haven't pushed out the rebased tree yet making this a truly awful bug report.
> >>
> >> The pull request you sent with this in wasn't bisectable so this was a bit of a guessing
> >> game. I see the seg fault only after this patch.  
> > 
> > I see the same thing with some RISC-V tests. I can provide the test
> > images if you want as well  
> 
> 
> Yes please.
> 
> 
> r~

I'm guessing Alastair is busy.

I got around to testing this on x86 and indeed blow up is the same.

0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
143     GEN_ATOMIC_HELPER(add_fetch)
(gdb) bt
#0  0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
#1  0x00007fffbc31c6f0 in code_gen_buffer ()
#2  0x0000555555e23aa6 in cpu_tb_exec (cpu=cpu@entry=0x555557369330, itb=itb@entry=0x7fffbc31c600 <code_gen_buffer+295441875>, tb_exit=tb_exit@entry=0x7ffff4bfd6ec) at ../../accel/tcg/cpu-exec.c:438
#3  0x0000555555e24025 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfd6ec, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffbc31c600 <code_gen_buffer+295441875>, cpu=0x555557369330) at ../../accel/tcg/cpu-exec.c:872
#4  cpu_exec_loop (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:982
#5  0x0000555555e247a1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:999
#6  0x0000555555e24e2c in cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/cpu-exec.c:1025
#7  0x0000555555e42c73 in tcg_cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops.c:81
#8  0x0000555555e43093 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
#9  0x0000555555ef2250 in qemu_thread_start (args=0x5555573e6e20) at ../../util/qemu-thread-posix.c:541
#10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

Need one patch for my particular setup to work around some DMA buffer issues in virtio (similar to
a patch for pci space last year).  I've been meaning to post an RFC to get feedback on how
to handle this but not gotten to it yet!

From 801e47897c5959a22ed050d7e7feebbbd3a12588 Mon Sep 17 00:00:00 2001
From: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Date: Mon, 22 Apr 2024 13:54:37 +0100
Subject: [PATCH] physmem: Increase bounce buffers for "memory" address space.

Doesn't need to be this big and should be configurable.

Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 system/physmem.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/system/physmem.c b/system/physmem.c
index 3f4fd69d9a..651b875827 100644
--- a/system/physmem.c
+++ b/system/physmem.c
@@ -2798,6 +2798,7 @@ static void memory_map_init(void)
     memory_region_init(system_memory, NULL, "system", UINT64_MAX);
     address_space_init(&address_space_memory, system_memory, "memory");
 
+    address_space_memory.max_bounce_buffer_size = 1024 * 1024 * 1024;
     system_io = g_malloc(sizeof(*system_io));
     memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io",
                           65536);
-- 
2.43.0


Anyhow, other than that you need any random distro image (I tend to use debian nocloud images)
and a recent kernel build (mainline is fine).

Then a config along the lines of (obviously this isn't minimal)

qemu-system-x86_64 -M q35,cxl=on,sata=off,smbus=off -m 4g,maxmem=8G,slots=4 -cpu max -smp 4 \
                                      -kernel bzImage \
                                      -bios bios \
 -drive if=none,file=/mnt/d/images/x86-full-big.qcow2,format=qcow2,id=hd \
 -device ioh3420,id=root_port1 -device virtio-blk-pci,drive=hd,bus=root_port1 \
 -netdev user,id=mynet,hostfwd=tcp::5553-:22 -device virtio-net-pci,netdev=mynet,id=bob \
 -nographic -no-reboot -append 'earlycon console=ttyS0 root=/dev/vda3 fsck.mode=skip tp_printk maxcpus=4' \
 -monitor telnet:127.0.0.1:1235,server,nowait \
 -object memory-backend-ram,size=4G,id=mem0 \
 -numa node,nodeid=0,cpus=0-3,memdev=mem0 \
 -numa node,nodeid=1 \
 -serial mon:stdio \
  -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=1M,align=1M \
 -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M,align=256M \
 -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=1M,align=1M \
 -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
 -device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=2 \
 -device cxl-rp,port=1,bus=cxl.1,id=root_port2,chassis=0,slot=3 \
 -device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem1,id=cxl-pmem0,lsa=cxl-lsa1,sn=3 \
 -device cxl-type3,bus=root_port2,volatile-memdev=cxl-mem3,id=cxl-pmem1,lsa=cxl-lsa2,sn=4 \
 -machine cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k

Then after booting into linux, bring up a cxl region with:

    cd /sys/bus/cxl/devices/decoder0.0/
    cat create_ram_region
    echo region0 > create_ram_region

    echo ram > /sys/bus/cxl/devices/decoder2.0/mode
    echo $((256 << 20)) > /sys/bus/cxl/devices/decoder2.0/dpa_size

    cd /sys/bus/cxl/devices/region0/
    echo 256 > interleave_granularity
    echo 1 > interleave_ways
    echo $((256 << 20)) > size 
    echo decoder2.0 > target0
    echo 1 > commit
    echo region0 > /sys/bus/cxl/drivers/cxl_region/bind

That should bring up a small amount of memory in node 2. Interleaving isn't actually
in use here but we haven't upstreamed the bypass optimizations so this is still
mmio space to QEMU.

Then numactl -m 2 ls 
boom.

A few relevant bits of kernel config (also not minimal)

//dax stuff to ensure we get memory as normal ram.
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_DEV_DAX_HMEM=y
CONFIG_DEV_DAX_CXL=m
CONFIG_DEV_DAX_HMEM_DEVICES=y
CONFIG_DEV_DAX_KMEM=m
//memory hotplug
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
CONFIG_ARCH_ENABLE_MEMORY_HOTREMOVE=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MHP_DEFAULT_ONLINE_TYPE_ONLINE_MOVABLE=y

Any hints welcome!  Also happy to provide any additional info as necessary.

Jonathan

Re: [PATCH 066/147] include/exec: Move TLB_MMIO, TLB_DISCARD_WRITE to slow flags

[Jonathan Cameron]

On Thu, 8 May 2025 14:29:18 +0100
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:

> On Tue, 29 Apr 2025 19:43:05 -0700
> Richard Henderson <richard.henderson@linaro.org> wrote:
> 
> > On 4/29/25 14:35, Alistair Francis wrote:  
> > > On Sat, Apr 26, 2025 at 3:36 AM Jonathan Cameron via
> > > <qemu-devel@nongnu.org> wrote:    
> > >>
> > >> On Tue, 22 Apr 2025 12:26:55 -0700
> > >> Richard Henderson <richard.henderson@linaro.org> wrote:
> > >>    
> > >>> Recover two bits from the inline flags.    
> > >>
> > >>
> > >> Hi Richard,
> > >>
> > >> Early days but something (I'm fairly sure in this patch) is tripping up my favourite
> > >> TCG corner case of running code out of MMIO memory (interleaved CXL memory).
> > >>
> > >> Only seeing it on arm64 tests so far which isn't upstream yet..
> > >> (guess what I was getting ready to post today)
> > >>
> > >> Back trace is:
> > >>
> > >> #0  0x0000555555fd4296 in cpu_atomic_fetch_andq_le_mmu (env=0x555557ee19b0, addr=18442241572520067072, val=18446744073701163007, oi=8244, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:140
> > >> #1  0x00007fffb6894125 in code_gen_buffer ()
> > >> #2  0x0000555555fc4c46 in cpu_tb_exec (cpu=cpu@entry=0x555557ededf0, itb=itb@entry=0x7fffb6894000 <code_gen_buffer+200511443>, tb_exit=tb_exit@entry=0x7ffff4bfb744) at ../../accel/tcg/cpu-exec.c:455
> > >> #3  0x0000555555fc51c2 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfb744, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffb6894000 <code_gen_buffer+200511443>, cpu=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:904
> > >> #4  cpu_exec_loop (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1018
> > >> #5  0x0000555555fc58f1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1035
> > >> #6  0x0000555555fc5f6c in cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:1061
> > >> #7  0x0000555556146ac3 in tcg_cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops.c:81
> > >> #8  0x0000555556146ee3 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
> > >> #9  0x00005555561f6450 in qemu_thread_start (args=0x555557f8f430) at ../../util/qemu-thread-posix.c:541
> > >> #10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
> > >> #11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
> > >>
> > >> I haven't pushed out the rebased tree yet making this a truly awful bug report.
> > >>
> > >> The pull request you sent with this in wasn't bisectable so this was a bit of a guessing
> > >> game. I see the seg fault only after this patch.    
> > > 
> > > I see the same thing with some RISC-V tests. I can provide the test
> > > images if you want as well    
> > 
> > 
> > Yes please.
> > 
> > 
> > r~  
> 
> I'm guessing Alastair is busy.
> 
> I got around to testing this on x86 and indeed blow up is the same.
> 
> 0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
> 143     GEN_ATOMIC_HELPER(add_fetch)
> (gdb) bt
> #0  0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
> #1  0x00007fffbc31c6f0 in code_gen_buffer ()
> #2  0x0000555555e23aa6 in cpu_tb_exec (cpu=cpu@entry=0x555557369330, itb=itb@entry=0x7fffbc31c600 <code_gen_buffer+295441875>, tb_exit=tb_exit@entry=0x7ffff4bfd6ec) at ../../accel/tcg/cpu-exec.c:438
> #3  0x0000555555e24025 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfd6ec, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffbc31c600 <code_gen_buffer+295441875>, cpu=0x555557369330) at ../../accel/tcg/cpu-exec.c:872
> #4  cpu_exec_loop (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:982
> #5  0x0000555555e247a1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:999
> #6  0x0000555555e24e2c in cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/cpu-exec.c:1025
> #7  0x0000555555e42c73 in tcg_cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops.c:81
> #8  0x0000555555e43093 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
> #9  0x0000555555ef2250 in qemu_thread_start (args=0x5555573e6e20) at ../../util/qemu-thread-posix.c:541
> #10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
> #11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
> 
> Need one patch for my particular setup to work around some DMA buffer issues in virtio (similar to
> a patch for pci space last year).  I've been meaning to post an RFC to get feedback on how
> to handle this but not gotten to it yet!
> 
> From 801e47897c5959a22ed050d7e7feebbbd3a12588 Mon Sep 17 00:00:00 2001
> From: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> Date: Mon, 22 Apr 2024 13:54:37 +0100
> Subject: [PATCH] physmem: Increase bounce buffers for "memory" address space.
> 
> Doesn't need to be this big and should be configurable.
> 
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
>  system/physmem.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/system/physmem.c b/system/physmem.c
> index 3f4fd69d9a..651b875827 100644
> --- a/system/physmem.c
> +++ b/system/physmem.c
> @@ -2798,6 +2798,7 @@ static void memory_map_init(void)
>      memory_region_init(system_memory, NULL, "system", UINT64_MAX);
>      address_space_init(&address_space_memory, system_memory, "memory");
>  
> +    address_space_memory.max_bounce_buffer_size = 1024 * 1024 * 1024;
>      system_io = g_malloc(sizeof(*system_io));
>      memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io",
>                            65536);

Hi Richard

As discussed on Friday, I've put test kernel up at https://gitlab.com/jic23/qemu-debug
It's just a build of mainline as checked out today. I'll commit the kernel config as well
for information.  Nothing particularly special just a lot of stuff built in so 
you don't need to fuss around with modules in the root fs / initrd etc.

The readme.md file in that repo has instructions to replicate with a typical setup +
shell scripts.  Only thing you'll need to install on the mentioned standard debian nocloud
image is numactl. Otherwise all cut and paste scripts.

Let me know if this either doesn't work for you (should segfault) on numctl -m 2 ls
or there is anything else I can do to help debug this one.

Thanks,

Jonathan

Re: [PATCH 066/147] include/exec: Move TLB_MMIO, TLB_DISCARD_WRITE to slow flags

[Richard Henderson]

On 5/20/25 18:01, Jonathan Cameron wrote:
> On Thu, 8 May 2025 14:29:18 +0100
> Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> 
>> On Tue, 29 Apr 2025 19:43:05 -0700
>> Richard Henderson <richard.henderson@linaro.org> wrote:
>>
>>> On 4/29/25 14:35, Alistair Francis wrote:
>>>> On Sat, Apr 26, 2025 at 3:36 AM Jonathan Cameron via
>>>> <qemu-devel@nongnu.org> wrote:
>>>>>
>>>>> On Tue, 22 Apr 2025 12:26:55 -0700
>>>>> Richard Henderson <richard.henderson@linaro.org> wrote:
>>>>>     
>>>>>> Recover two bits from the inline flags.
>>>>>
>>>>>
>>>>> Hi Richard,
>>>>>
>>>>> Early days but something (I'm fairly sure in this patch) is tripping up my favourite
>>>>> TCG corner case of running code out of MMIO memory (interleaved CXL memory).
>>>>>
>>>>> Only seeing it on arm64 tests so far which isn't upstream yet..
>>>>> (guess what I was getting ready to post today)
>>>>>
>>>>> Back trace is:
>>>>>
>>>>> #0  0x0000555555fd4296 in cpu_atomic_fetch_andq_le_mmu (env=0x555557ee19b0, addr=18442241572520067072, val=18446744073701163007, oi=8244, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:140
>>>>> #1  0x00007fffb6894125 in code_gen_buffer ()
>>>>> #2  0x0000555555fc4c46 in cpu_tb_exec (cpu=cpu@entry=0x555557ededf0, itb=itb@entry=0x7fffb6894000 <code_gen_buffer+200511443>, tb_exit=tb_exit@entry=0x7ffff4bfb744) at ../../accel/tcg/cpu-exec.c:455
>>>>> #3  0x0000555555fc51c2 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfb744, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffb6894000 <code_gen_buffer+200511443>, cpu=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:904
>>>>> #4  cpu_exec_loop (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1018
>>>>> #5  0x0000555555fc58f1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557ededf0, sc=sc@entry=0x7ffff4bfb7f0) at ../../accel/tcg/cpu-exec.c:1035
>>>>> #6  0x0000555555fc5f6c in cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/cpu-exec.c:1061
>>>>> #7  0x0000555556146ac3 in tcg_cpu_exec (cpu=cpu@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops.c:81
>>>>> #8  0x0000555556146ee3 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557ededf0) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
>>>>> #9  0x00005555561f6450 in qemu_thread_start (args=0x555557f8f430) at ../../util/qemu-thread-posix.c:541
>>>>> #10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
>>>>> #11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
>>>>>
>>>>> I haven't pushed out the rebased tree yet making this a truly awful bug report.
>>>>>
>>>>> The pull request you sent with this in wasn't bisectable so this was a bit of a guessing
>>>>> game. I see the seg fault only after this patch.
>>>>
>>>> I see the same thing with some RISC-V tests. I can provide the test
>>>> images if you want as well
>>>
>>>
>>> Yes please.
>>>
>>>
>>> r~
>>
>> I'm guessing Alastair is busy.
>>
>> I got around to testing this on x86 and indeed blow up is the same.
>>
>> 0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
>> 143     GEN_ATOMIC_HELPER(add_fetch)
>> (gdb) bt
>> #0  0x0000555555e3dd77 in cpu_atomic_add_fetchl_le_mmu (env=0x55555736bef0, addr=140271756837240, val=1, oi=34, retaddr=<optimized out>) at ../../accel/tcg/atomic_template.h:143
>> #1  0x00007fffbc31c6f0 in code_gen_buffer ()
>> #2  0x0000555555e23aa6 in cpu_tb_exec (cpu=cpu@entry=0x555557369330, itb=itb@entry=0x7fffbc31c600 <code_gen_buffer+295441875>, tb_exit=tb_exit@entry=0x7ffff4bfd6ec) at ../../accel/tcg/cpu-exec.c:438
>> #3  0x0000555555e24025 in cpu_loop_exec_tb (tb_exit=0x7ffff4bfd6ec, last_tb=<synthetic pointer>, pc=<optimized out>, tb=0x7fffbc31c600 <code_gen_buffer+295441875>, cpu=0x555557369330) at ../../accel/tcg/cpu-exec.c:872
>> #4  cpu_exec_loop (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:982
>> #5  0x0000555555e247a1 in cpu_exec_setjmp (cpu=cpu@entry=0x555557369330, sc=sc@entry=0x7ffff4bfd7b0) at ../../accel/tcg/cpu-exec.c:999
>> #6  0x0000555555e24e2c in cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/cpu-exec.c:1025
>> #7  0x0000555555e42c73 in tcg_cpu_exec (cpu=cpu@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops.c:81
>> #8  0x0000555555e43093 in mttcg_cpu_thread_fn (arg=arg@entry=0x555557369330) at ../../accel/tcg/tcg-accel-ops-mttcg.c:94
>> #9  0x0000555555ef2250 in qemu_thread_start (args=0x5555573e6e20) at ../../util/qemu-thread-posix.c:541
>> #10 0x00007ffff7750aa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
>> #11 0x00007ffff77ddc3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78
>>
>> Need one patch for my particular setup to work around some DMA buffer issues in virtio (similar to
>> a patch for pci space last year).  I've been meaning to post an RFC to get feedback on how
>> to handle this but not gotten to it yet!
>>
>>  From 801e47897c5959a22ed050d7e7feebbbd3a12588 Mon Sep 17 00:00:00 2001
>> From: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> Date: Mon, 22 Apr 2024 13:54:37 +0100
>> Subject: [PATCH] physmem: Increase bounce buffers for "memory" address space.
>>
>> Doesn't need to be this big and should be configurable.
>>
>> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>>   system/physmem.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/system/physmem.c b/system/physmem.c
>> index 3f4fd69d9a..651b875827 100644
>> --- a/system/physmem.c
>> +++ b/system/physmem.c
>> @@ -2798,6 +2798,7 @@ static void memory_map_init(void)
>>       memory_region_init(system_memory, NULL, "system", UINT64_MAX);
>>       address_space_init(&address_space_memory, system_memory, "memory");
>>   
>> +    address_space_memory.max_bounce_buffer_size = 1024 * 1024 * 1024;
>>       system_io = g_malloc(sizeof(*system_io));
>>       memory_region_init_io(system_io, NULL, &unassigned_io_ops, NULL, "io",
>>                             65536);
> 
> Hi Richard
> 
> As discussed on Friday, I've put test kernel up at https://gitlab.com/jic23/qemu-debug
> It's just a build of mainline as checked out today. I'll commit the kernel config as well
> for information.  Nothing particularly special just a lot of stuff built in so
> you don't need to fuss around with modules in the root fs / initrd etc.
> 
> The readme.md file in that repo has instructions to replicate with a typical setup +
> shell scripts.  Only thing you'll need to install on the mentioned standard debian nocloud
> image is numactl. Otherwise all cut and paste scripts.
> 
> Let me know if this either doesn't work for you (should segfault) on numctl -m 2 ls
> or there is anything else I can do to help debug this one.

Got it.  I failed to reconstruct tlb flags properly in the cmpxchg path,
which causes a crash for cmpxchg on mmio.

r~