BUG: general protection fault in __free_object

BUG: general protection fault in __free_object

[Xingyu Li]

Hi,

We found a bug in Linux 6.10 using syzkaller. It is possibly a null
pointer dereference  bug.
The reproducer is
https://gist.github.com/freexxxyyy/5aefd53c6567415e9fe8c76cc2ad390c

The bug report is:

Syzkaller hit 'general protection fault in __free_object' bug.

Oops: general protection fault, probably for non-canonical address
0xdffffc0000000000: 0000 [#1] PREEMPT SMP KASAN PTI
KASAN: null-ptr-deref in range [0x0000000000000000-0x0000000000000007]
CPU: 0 PID: 8 Comm: kworker/0:0 Not tainted 6.10.0 #13
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: events kfree_rcu_work
RIP: 0010:hlist_add_head include/linux/list.h:1032 [inline]
RIP: 0010:__free_object+0x903/0xaa0 lib/debugobjects.c:396
Code: 24 18 48 8b 7c 24 38 74 05 e8 89 bf 95 fd 48 8b 44 24 28 49 89
45 08 eb 03 45 31 ed 48 8b 1d 34 95 61 0e 4c 89 e8 48 c1 e8 03 <42> 80
3c 30 00 74 08 4c 89 ef e8 5e bf 95 fd 49 89 5d 00 48 85 db
RSP: 0018:ffffc900000af740 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff888028f56cb0 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: 0000000000000004 RDI: ffff888028f56cb8
RBP: ffffc900000af920 R08: 0000000000000003 R09: fffff52000015ed8
R10: dffffc0000000000 R11: fffff52000015ed8 R12: ffffffff92c03280
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff888063a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555787e6868 CR3: 000000000d932000 CR4: 0000000000350ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 __debug_check_no_obj_freed lib/debugobjects.c:994 [inline]
 debug_check_no_obj_freed+0x135/0x530 lib/debugobjects.c:1019
 slab_free_hook mm/slub.c:2163 [inline]
 slab_free_freelist_hook mm/slub.c:2225 [inline]
 slab_free_bulk mm/slub.c:4462 [inline]
 kmem_cache_free_bulk+0x1bf/0x360 mm/slub.c:4676
 kfree_bulk include/linux/slab.h:568 [inline]
 kvfree_rcu_bulk+0x249/0x4d0 kernel/rcu/tree.c:3371
 kfree_rcu_work+0x443/0x500 kernel/rcu/tree.c:3450
 process_one_work kernel/workqueue.c:3248 [inline]
 process_scheduled_works+0x977/0x1410 kernel/workqueue.c:3329
 worker_thread+0xaa0/0x1020 kernel/workqueue.c:3409
 kthread+0x2eb/0x380 kernel/kthread.c:389
 ret_from_fork+0x49/0x80 arch/x86/kernel/process.c:147
 ret_from_fork_asm+0x11/0x20 arch/x86/entry/entry_64.S:244
 </TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
RIP: 0010:hlist_add_head include/linux/list.h:1032 [inline]
RIP: 0010:__free_object+0x903/0xaa0 lib/debugobjects.c:396
Code: 24 18 48 8b 7c 24 38 74 05 e8 89 bf 95 fd 48 8b 44 24 28 49 89
45 08 eb 03 45 31 ed 48 8b 1d 34 95 61 0e 4c 89 e8 48 c1 e8 03 <42> 80
3c 30 00 74 08 4c 89 ef e8 5e bf 95 fd 49 89 5d 00 48 85 db
RSP: 0018:ffffc900000af740 EFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff888028f56cb0 RCX: 0000000000000001
RDX: dffffc0000000000 RSI: 0000000000000004 RDI: ffff888028f56cb8
RBP: ffffc900000af920 R08: 0000000000000003 R09: fffff52000015ed8
R10: dffffc0000000000 R11: fffff52000015ed8 R12: ffffffff92c03280
R13: 0000000000000000 R14: dffffc0000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff888063a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555787e6868 CR3: 000000000d932000 CR4: 0000000000350ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
----------------
Code disassembly (best guess):
   0: 24 18                 and    $0x18,%al
   2: 48 8b 7c 24 38       mov    0x38(%rsp),%rdi
   7: 74 05                 je     0xe
   9: e8 89 bf 95 fd       call   0xfd95bf97
   e: 48 8b 44 24 28       mov    0x28(%rsp),%rax
  13: 49 89 45 08           mov    %rax,0x8(%r13)
  17: eb 03                 jmp    0x1c
  19: 45 31 ed             xor    %r13d,%r13d
  1c: 48 8b 1d 34 95 61 0e mov    0xe619534(%rip),%rbx        # 0xe619557
  23: 4c 89 e8             mov    %r13,%rax
  26: 48 c1 e8 03           shr    $0x3,%rax
* 2a: 42 80 3c 30 00       cmpb   $0x0,(%rax,%r14,1) <-- trapping instruction
  2f: 74 08                 je     0x39
  31: 4c 89 ef             mov    %r13,%rdi
  34: e8 5e bf 95 fd       call   0xfd95bf97
  39: 49 89 5d 00           mov    %rbx,0x0(%r13)
  3d: 48 85 db             test   %rbx,%rbx


-- 
Yours sincerely,
Xingyu

Re: BUG: general protection fault in __free_object

[Thomas Gleixner]

On Wed, Aug 28 2024 at 16:27, Xingyu Li wrote:
> We found a bug in Linux 6.10 using syzkaller. It is possibly a null
> pointer dereference  bug.
> The reproducer is
> https://gist.github.com/freexxxyyy/5aefd53c6567415e9fe8c76cc2ad390c

Reproducer alone is not really helpful. This needs the corresponding
kernel config, the exact kernel version and a description of the
reproduction environment (compiler, qemu command line ....)

It does not reproduce here at all.

Also if it really reproduces, then have you checked against current
mainline as well?

> RIP: 0010:hlist_add_head include/linux/list.h:1032 [inline]
> RIP: 0010:__free_object+0x903/0xaa0 lib/debugobjects.c:396
>  __debug_check_no_obj_freed lib/debugobjects.c:994 [inline]
>  debug_check_no_obj_freed+0x135/0x530 lib/debugobjects.c:1019
>  slab_free_hook mm/slub.c:2163 [inline]
>  slab_free_freelist_hook mm/slub.c:2225 [inline]
>  slab_free_bulk mm/slub.c:4462 [inline]
>  kmem_cache_free_bulk+0x1bf/0x360 mm/slub.c:4676

The code in question is serialized against objpool modifications with
pool_lock. So nothing can change any of the related data.

    if (obj_pool_free > debug_objects_pool_size &&
        obj_nr_tofree < ODEBUG_FREE_WORK_MAX) {

	for (i = 0; i < ODEBUG_BATCH_SIZE; i++) {
		obj = __alloc_object(&obj_pool);
		hlist_add_head(&obj->node, &obj_to_free); <- fail

debug_objects_pool_size is initialized to:

        ODEBUG_POOL_SIZE + num_possible_cpus() * ODEBUG_BATCH_SIZE;
=
        1024 + num_possible_cpus() * 16

The minimum size is 1040, so there are at least 1040 objects in
obj_pool. The loop allocates at max 16 objects, which means

             __alloc_object(&obj_pool);

is guaranteed to return an object and cannot return NULL.

So the only reason why this can result in a NULL pointer dereference is
that the obj_to_free list is corrupted. No idea how that should happen.

Maybe some proper reproducer instructions shed some light to it.

Thanks,

        tglx

Re: BUG: general protection fault in __free_object

[Xingyu Li]

This needs the corresponding
kernel config, the exact kernel version and a description of the
reproduction environment (compiler, qemu command line ....)

As mentioned, it is on Linux 6.10. And here is the config:
https://gist.github.com/TomAPU/64f5db0fe976a3e94a6dd2b621887cdd
It is compiled by clang version 14.0.0
(https://github.com/llvm/llvm-project.git
329fda39c507e8740978d10458451dcdb21563be).

On Mon, Sep 2, 2024 at 2:21 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Wed, Aug 28 2024 at 16:27, Xingyu Li wrote:
> > We found a bug in Linux 6.10 using syzkaller. It is possibly a null
> > pointer dereference  bug.
> > The reproducer is
> > https://gist.github.com/freexxxyyy/5aefd53c6567415e9fe8c76cc2ad390c
>
> Reproducer alone is not really helpful. This needs the corresponding
> kernel config, the exact kernel version and a description of the
> reproduction environment (compiler, qemu command line ....)
>
> It does not reproduce here at all.
>
> Also if it really reproduces, then have you checked against current
> mainline as well?
>
> > RIP: 0010:hlist_add_head include/linux/list.h:1032 [inline]
> > RIP: 0010:__free_object+0x903/0xaa0 lib/debugobjects.c:396
> >  __debug_check_no_obj_freed lib/debugobjects.c:994 [inline]
> >  debug_check_no_obj_freed+0x135/0x530 lib/debugobjects.c:1019
> >  slab_free_hook mm/slub.c:2163 [inline]
> >  slab_free_freelist_hook mm/slub.c:2225 [inline]
> >  slab_free_bulk mm/slub.c:4462 [inline]
> >  kmem_cache_free_bulk+0x1bf/0x360 mm/slub.c:4676
>
> The code in question is serialized against objpool modifications with
> pool_lock. So nothing can change any of the related data.
>
>     if (obj_pool_free > debug_objects_pool_size &&
>         obj_nr_tofree < ODEBUG_FREE_WORK_MAX) {
>
>         for (i = 0; i < ODEBUG_BATCH_SIZE; i++) {
>                 obj = __alloc_object(&obj_pool);
>                 hlist_add_head(&obj->node, &obj_to_free); <- fail
>
> debug_objects_pool_size is initialized to:
>
>         ODEBUG_POOL_SIZE + num_possible_cpus() * ODEBUG_BATCH_SIZE;
> =
>         1024 + num_possible_cpus() * 16
>
> The minimum size is 1040, so there are at least 1040 objects in
> obj_pool. The loop allocates at max 16 objects, which means
>
>              __alloc_object(&obj_pool);
>
> is guaranteed to return an object and cannot return NULL.
>
> So the only reason why this can result in a NULL pointer dereference is
> that the obj_to_free list is corrupted. No idea how that should happen.
>
> Maybe some proper reproducer instructions shed some light to it.
>
> Thanks,
>
>         tglx
>
>
>
>


-- 
Yours sincerely,
Xingyu

Re: BUG: general protection fault in __free_object

[Thomas Gleixner]

On Mon, Sep 02 2024 at 15:55, Xingyu Li wrote:
> This needs the corresponding
> kernel config, the exact kernel version and a description of the
> reproduction environment (compiler, qemu command line ....)
>
> As mentioned, it is on Linux 6.10. And here is the config:
> https://gist.github.com/TomAPU/64f5db0fe976a3e94a6dd2b621887cdd
> It is compiled by clang version 14.0.0
> (https://github.com/llvm/llvm-project.git 329fda39c507e8740978d10458451dcdb21563be).

You still fail to provide the reproduction environment:

    - qemu version
    - qemu invocation

This all matters and I don't have time to figure that out myself.

Thanks,

        tglx

Re: BUG: general protection fault in __free_object

[Xingyu Li]

Please see here: https://github.com/TomAPU/Linux610BugReort/tree/master


On Mon, Sep 2, 2024 at 4:22 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Mon, Sep 02 2024 at 15:55, Xingyu Li wrote:
> > This needs the corresponding
> > kernel config, the exact kernel version and a description of the
> > reproduction environment (compiler, qemu command line ....)
> >
> > As mentioned, it is on Linux 6.10. And here is the config:
> > https://gist.github.com/TomAPU/64f5db0fe976a3e94a6dd2b621887cdd
> > It is compiled by clang version 14.0.0
> > (https://github.com/llvm/llvm-project.git 329fda39c507e8740978d10458451dcdb21563be).
>
> You still fail to provide the reproduction environment:
>
>     - qemu version
>     - qemu invocation
>
> This all matters and I don't have time to figure that out myself.
>
> Thanks,
>
>         tglx
>
>


--
Yours sincerely,
Xingyu

Re: BUG: general protection fault in __free_object

[Thomas Gleixner]

On Tue, Sep 03 2024 at 01:49, Thomas Gleixner wrote:

> On Mon, Sep 02 2024 at 16:28, Xingyu Li wrote:
>
>> Please see here: https://github.com/TomAPU/Linux610BugReort/tree/master
>
> Can't you have the courtesy to provide me a consice anwser to my
> questions without making me chase a gazillion of http links?

And you still did not answer my question whether this is still relevant
to Linux mainline or to the latest linux-v6.10.y stable tree.

Thanks,

        tglx

Re: BUG: general protection fault in __free_object

[Xingyu Li]

it is linux mainline. We already provided a link to the linux that we
used, on https://github.com/TomAPU/Linux610BugReort/tree/master.

On Mon, Sep 2, 2024 at 5:08 PM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Tue, Sep 03 2024 at 01:49, Thomas Gleixner wrote:
>
> > On Mon, Sep 02 2024 at 16:28, Xingyu Li wrote:
> >
> >> Please see here: https://github.com/TomAPU/Linux610BugReort/tree/master
> >
> > Can't you have the courtesy to provide me a consice anwser to my
> > questions without making me chase a gazillion of http links?
>
> And you still did not answer my question whether this is still relevant
> to Linux mainline or to the latest linux-v6.10.y stable tree.
>
> Thanks,
>
>         tglx



-- 
Yours sincerely,
Xingyu

Re: BUG: general protection fault in __free_object

[Thomas Gleixner]

On Mon, Sep 02 2024 at 17:10, Xingyu Li wrote:
> it is linux mainline. We already provided a link to the linux that we
> used, on https://github.com/TomAPU/Linux610BugReort/tree/master.

No. This is not mainline. This is release v6.10, but mainline is already
at v6.11-rc6. The question is whether the problem is still there or has
been addressed by now.

Thanks,

        tglx

Re: BUG: general protection fault in __free_object

[Xingyu Li]

We also tried it on v6.11-rc6 and the bug can still be triggered.

On Tue, Sep 3, 2024 at 1:58 AM Thomas Gleixner <tglx@linutronix.de> wrote:
>
> On Mon, Sep 02 2024 at 17:10, Xingyu Li wrote:
> > it is linux mainline. We already provided a link to the linux that we
> > used, on https://github.com/TomAPU/Linux610BugReort/tree/master.
>
> No. This is not mainline. This is release v6.10, but mainline is already
> at v6.11-rc6. The question is whether the problem is still there or has
> been addressed by now.
>
> Thanks,
>
>         tglx



-- 
Yours sincerely,
Xingyu