* ASan reports use-after-free when running munmap-pthread
@ 2023-02-02 19:06 Anton Johansson via
2023-02-03 11:23 ` Alex Bennée
2023-02-03 21:18 ` Richard Henderson
0 siblings, 2 replies; 6+ messages in thread
From: Anton Johansson via @ 2023-02-02 19:06 UTC (permalink / raw)
To: qemu-devel; +Cc: ale@rev.ng, iii, Richard Henderson, Alex Bennée
[-- Attachment #1: Type: text/plain, Size: 3056 bytes --]
Hi,
I was running check-tcg with ASan enabled on master, and ran into
the following use-after-free. There appears to be a race between
jump cache invalidation and thread destruction (?)
I thought I'd post here since I noticed some previous discussion on the
topic, and I'm not sure myself what a proper fix would look like.
Tested on arm/aarch64/x86_64-linux-user.
Here's a snippet of the ASan output:
=================================================================
==187529==ERROR: AddressSanitizer: heap-use-after-free on address
0x62d000f433b0 at pc 0x55cfefe00246 bp 0x7f4725f400b0 sp 0x7f4725f400a0
READ of size 8 at 0x62d000f433b0 thread T2
#0 0x55cfefe00245 in tb_jmp_cache_inval_tb
/home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:861
#1 0x55cfefe00245 in do_tb_phys_invalidate
/home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:900
#2 0x55cfefe0088a in tb_phys_invalidate__locked
/home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:916
#3 0x55cfefe0088a in tb_invalidate_phys_range
/home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:1000
#4 0x55cfefe7ecf9 in target_munmap
/home/aj/git/qemu-upstream/build/../linux-user/mmap.c:766
#5 0x55cfefea5815 in do_syscall1
/home/aj/git/qemu-upstream/build/../linux-user/syscall.c:10105
#6 0x55cfefe9c950 in do_syscall
/home/aj/git/qemu-upstream/build/../linux-user/syscall.c:13329
#7 0x55cfefb97255 in cpu_loop
../linux-user/x86_64/../i386/cpu_loop.c:233
#8 0x55cfefec7af4 in clone_func
/home/aj/git/qemu-upstream/build/../linux-user/syscall.c:6633
#9 0x7f4726bbb8fc (/usr/lib/libc.so.6+0x868fc)
#10 0x7f4726c3da5f (/usr/lib/libc.so.6+0x108a5f)
0x62d000f433b0 is located 28592 bytes inside of 32768-byte region
[0x62d000f3c400,0x62d000f44400)
freed by thread T387 here:
#0 0x7f47270be672 in __interceptor_free
/usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52
#1 0x55cfefd071b8 in cpu_exec_unrealizefn
/home/aj/git/qemu-upstream/build/../cpu.c:180
#2 0x55cfefeea287 in property_set_bool
/home/aj/git/qemu-upstream/build/../qom/object.c:2285
#3 0x55cfefee603b in object_property_set
/home/aj/git/qemu-upstream/build/../qom/object.c:1420
#4 0x55cfefeef21c in object_property_set_qobject
/home/aj/git/qemu-upstream/build/../qom/qom-qobject.c:28
previously allocated by thread T0 here:
#0 0x7f47270bf411 in __interceptor_calloc
/usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
#1 0x7f4726e77681 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x53681)
#2 0x55cfefed7cfe in device_set_realized
/home/aj/git/qemu-upstream/build/../hw/core/qdev.c:510
#3 0x55cfefeea287 in property_set_bool
/home/aj/git/qemu-upstream/build/../qom/object.c:2285
#4 0x55cfefee603b in object_property_set
/home/aj/git/qemu-upstream/build/../qom/object.c:1420
#5 0x55cfefeef21c in object_property_set_qobject
/home/aj/git/qemu-upstream/build/../qom/qom-qobject.c:28
--
Anton Johansson,
rev.ng Labs Srl.
[-- Attachment #2: Type: text/html, Size: 4247 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ASan reports use-after-free when running munmap-pthread
2023-02-02 19:06 ASan reports use-after-free when running munmap-pthread Anton Johansson via
@ 2023-02-03 11:23 ` Alex Bennée
2023-02-03 17:34 ` Richard Henderson
2023-02-03 21:18 ` Richard Henderson
1 sibling, 1 reply; 6+ messages in thread
From: Alex Bennée @ 2023-02-03 11:23 UTC (permalink / raw)
To: anjo; +Cc: qemu-devel, ale@rev.ng, iii, Richard Henderson
Anton Johansson <anjo@rev.ng> writes:
> Hi,
>
> I was running check-tcg with ASan enabled on master, and ran into
> the following use-after-free. There appears to be a race between
> jump cache invalidation and thread destruction (?)
>
> I thought I'd post here since I noticed some previous discussion on the
> topic, and I'm not sure myself what a proper fix would look like.
Something like:
modified accel/tcg/tb-maint.c
@@ -858,9 +858,7 @@ static void tb_jmp_cache_inval_tb(TranslationBlock *tb)
CPU_FOREACH(cpu) {
CPUJumpCache *jc = cpu->tb_jmp_cache;
- if (qatomic_read(&jc->array[h].tb) == tb) {
- qatomic_set(&jc->array[h].tb, NULL);
- }
+ qatomic_cmpxchg(&jc->array[h].tb, tb, NULL);
}
}
?
>
> Tested on arm/aarch64/x86_64-linux-user.
>
> Here's a snippet of the ASan output:
> =================================================================
> ==187529==ERROR: AddressSanitizer: heap-use-after-free on address 0x62d000f433b0 at pc
> 0x55cfefe00246 bp 0x7f4725f400b0 sp 0x7f4725f400a0
> READ of size 8 at 0x62d000f433b0 thread T2
> #0 0x55cfefe00245 in tb_jmp_cache_inval_tb /home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:861
> #1 0x55cfefe00245 in do_tb_phys_invalidate /home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:900
> #2 0x55cfefe0088a in tb_phys_invalidate__locked
> /home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:916
> #3 0x55cfefe0088a in tb_invalidate_phys_range
> /home/aj/git/qemu-upstream/build/../accel/tcg/tb-maint.c:1000
> #4 0x55cfefe7ecf9 in target_munmap /home/aj/git/qemu-upstream/build/../linux-user/mmap.c:766
> #5 0x55cfefea5815 in do_syscall1 /home/aj/git/qemu-upstream/build/../linux-user/syscall.c:10105
> #6 0x55cfefe9c950 in do_syscall /home/aj/git/qemu-upstream/build/../linux-user/syscall.c:13329
> #7 0x55cfefb97255 in cpu_loop ../linux-user/x86_64/../i386/cpu_loop.c:233
> #8 0x55cfefec7af4 in clone_func /home/aj/git/qemu-upstream/build/../linux-user/syscall.c:6633
> #9 0x7f4726bbb8fc (/usr/lib/libc.so.6+0x868fc)
> #10 0x7f4726c3da5f (/usr/lib/libc.so.6+0x108a5f)
>
> 0x62d000f433b0 is located 28592 bytes inside of 32768-byte region [0x62d000f3c400,0x62d000f44400)
> freed by thread T387 here:
> #0 0x7f47270be672 in __interceptor_free /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:52
> #1 0x55cfefd071b8 in cpu_exec_unrealizefn /home/aj/git/qemu-upstream/build/../cpu.c:180
> #2 0x55cfefeea287 in property_set_bool /home/aj/git/qemu-upstream/build/../qom/object.c:2285
> #3 0x55cfefee603b in object_property_set /home/aj/git/qemu-upstream/build/../qom/object.c:1420
> #4 0x55cfefeef21c in object_property_set_qobject
> /home/aj/git/qemu-upstream/build/../qom/qom-qobject.c:28
>
> previously allocated by thread T0 here:
> #0 0x7f47270bf411 in __interceptor_calloc /usr/src/debug/gcc/libsanitizer/asan/asan_malloc_linux.cpp:77
> #1 0x7f4726e77681 in g_malloc0 (/usr/lib/libglib-2.0.so.0+0x53681)
> #2 0x55cfefed7cfe in device_set_realized /home/aj/git/qemu-upstream/build/../hw/core/qdev.c:510
> #3 0x55cfefeea287 in property_set_bool /home/aj/git/qemu-upstream/build/../qom/object.c:2285
> #4 0x55cfefee603b in object_property_set /home/aj/git/qemu-upstream/build/../qom/object.c:1420
> #5 0x55cfefeef21c in object_property_set_qobject
> /home/aj/git/qemu-upstream/build/../qom/qom-qobject.c:28
--
Alex Bennée
Virtualisation Tech Lead @ Linaro
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: ASan reports use-after-free when running munmap-pthread
2023-02-03 11:23 ` Alex Bennée
@ 2023-02-03 17:34 ` Richard Henderson
2023-02-03 18:08 ` Richard Henderson
0 siblings, 1 reply; 6+ messages in thread
From: Richard Henderson @ 2023-02-03 17:34 UTC (permalink / raw)
To: Alex Bennée, anjo; +Cc: qemu-devel, ale@rev.ng, iii
On 2/3/23 01:23, Alex Bennée wrote:
>
> Anton Johansson <anjo@rev.ng> writes:
>
>> Hi,
>>
>> I was running check-tcg with ASan enabled on master, and ran into
>> the following use-after-free. There appears to be a race between
>> jump cache invalidation and thread destruction (?)
>>
>> I thought I'd post here since I noticed some previous discussion on the
>> topic, and I'm not sure myself what a proper fix would look like.
>
> Something like:
>
> modified accel/tcg/tb-maint.c
> @@ -858,9 +858,7 @@ static void tb_jmp_cache_inval_tb(TranslationBlock *tb)
> CPU_FOREACH(cpu) {
> CPUJumpCache *jc = cpu->tb_jmp_cache;
>
> - if (qatomic_read(&jc->array[h].tb) == tb) {
> - qatomic_set(&jc->array[h].tb, NULL);
> - }
> + qatomic_cmpxchg(&jc->array[h].tb, tb, NULL);
> }
> }
No, this doesn't affect the use-after-free of jc itself.
I think CPUJumpCache needs to be freed with RCU.
r~
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: ASan reports use-after-free when running munmap-pthread
2023-02-03 17:34 ` Richard Henderson
@ 2023-02-03 18:08 ` Richard Henderson
0 siblings, 0 replies; 6+ messages in thread
From: Richard Henderson @ 2023-02-03 18:08 UTC (permalink / raw)
To: Alex Bennée, anjo; +Cc: qemu-devel, ale@rev.ng, iii
On 2/3/23 07:34, Richard Henderson wrote:
> On 2/3/23 01:23, Alex Bennée wrote:
>>
>> Anton Johansson <anjo@rev.ng> writes:
>>
>>> Hi,
>>>
>>> I was running check-tcg with ASan enabled on master, and ran into
>>> the following use-after-free. There appears to be a race between
>>> jump cache invalidation and thread destruction (?)
>>>
>>> I thought I'd post here since I noticed some previous discussion on the
>>> topic, and I'm not sure myself what a proper fix would look like.
>>
>> Something like:
>>
>> modified accel/tcg/tb-maint.c
>> @@ -858,9 +858,7 @@ static void tb_jmp_cache_inval_tb(TranslationBlock *tb)
>> CPU_FOREACH(cpu) {
>> CPUJumpCache *jc = cpu->tb_jmp_cache;
>> - if (qatomic_read(&jc->array[h].tb) == tb) {
>> - qatomic_set(&jc->array[h].tb, NULL);
>> - }
>> + qatomic_cmpxchg(&jc->array[h].tb, tb, NULL);
>> }
>> }
>
> No, this doesn't affect the use-after-free of jc itself.
> I think CPUJumpCache needs to be freed with RCU.
Bah, we already do that.
r~
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ASan reports use-after-free when running munmap-pthread
2023-02-02 19:06 ASan reports use-after-free when running munmap-pthread Anton Johansson via
2023-02-03 11:23 ` Alex Bennée
@ 2023-02-03 21:18 ` Richard Henderson
2023-02-04 12:33 ` Anton Johansson via
1 sibling, 1 reply; 6+ messages in thread
From: Richard Henderson @ 2023-02-03 21:18 UTC (permalink / raw)
To: anjo, qemu-devel; +Cc: ale@rev.ng, iii, Alex Bennée
On 2/2/23 09:06, Anton Johansson wrote:
> Hi,
>
> I was running check-tcg with ASan enabled on master, and ran into
> the following use-after-free. There appears to be a race between
> jump cache invalidation and thread destruction (?)
>
> I thought I'd post here since I noticed some previous discussion on the
> topic, and I'm not sure myself what a proper fix would look like.
>
> Tested on arm/aarch64/x86_64-linux-user.
>
> Here's a snippet of the ASan output:
> =================================================================
> ==187529==ERROR: AddressSanitizer: heap-use-after-free on address 0x62d000f433b0 at pc
> 0x55cfefe00246 bp 0x7f4725f400b0 sp 0x7f4725f400a0
The fix for this was merged today:
4731f89b3b cpu: free cpu->tb_jmp_cache with RCU
I'd forgotten about this, since the pull request was pending for some time, while we
waited for CI minutes to refresh.
r~
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: ASan reports use-after-free when running munmap-pthread
2023-02-03 21:18 ` Richard Henderson
@ 2023-02-04 12:33 ` Anton Johansson via
0 siblings, 0 replies; 6+ messages in thread
From: Anton Johansson via @ 2023-02-04 12:33 UTC (permalink / raw)
To: Richard Henderson, qemu-devel; +Cc: ale@rev.ng, iii, Alex Bennée
On 2/3/23 22:18, Richard Henderson wrote:
>
> The fix for this was merged today:
>
> 4731f89b3b cpu: free cpu->tb_jmp_cache with RCU
>
> I'd forgotten about this, since the pull request was pending for some
> time, while we waited for CI minutes to refresh.
>
>
> r~
Ah that's great, thanks for taking a look!
--
Anton Johansson,
rev.ng Labs Srl.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2023-02-04 12:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-02-02 19:06 ASan reports use-after-free when running munmap-pthread Anton Johansson via
2023-02-03 11:23 ` Alex Bennée
2023-02-03 17:34 ` Richard Henderson
2023-02-03 18:08 ` Richard Henderson
2023-02-03 21:18 ` Richard Henderson
2023-02-04 12:33 ` Anton Johansson via
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).