[syzbot] [kvm?] WARNING in mmu_free_root_page

[syzbot] [kvm?] WARNING in mmu_free_root_page

[syzbot]

Hello,

syzbot found the following issue on:

HEAD commit:    928a87efa423 Merge tag 'gfs2-v6.8-fix' of git://git.kernel..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=127c0546180000
kernel config:  https://syzkaller.appspot.com/x/.config?x=f64ec427e98bccd7
dashboard link: https://syzkaller.appspot.com/bug?extid=dc308fcfcd53f987de73
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=110481f1180000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=177049a5180000

Downloadable assets:
disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-928a87ef.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/7979568a5a16/vmlinux-928a87ef.xz
kernel image: https://storage.googleapis.com/syzbot-assets/1bc6e1d480e3/bzImage-928a87ef.xz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+dc308fcfcd53f987de73@syzkaller.appspotmail.com

------------[ cut here ]------------
WARNING: CPU: 0 PID: 5187 at arch/x86/kvm/mmu/mmu.c:3579 mmu_free_root_page+0x36c/0x3f0 arch/x86/kvm/mmu/mmu.c:3579
Modules linked in:
CPU: 0 PID: 5187 Comm: syz-executor400 Not tainted 6.9.0-rc1-syzkaller-00005-g928a87efa423 #0
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
RIP: 0010:mmu_free_root_page+0x36c/0x3f0 arch/x86/kvm/mmu/mmu.c:3579
Code: 00 49 8d 7d 18 be 01 00 00 00 e8 8f 32 c0 09 31 ff 41 89 c6 89 c6 e8 13 e7 6f 00 45 85 f6 0f 85 5d fe ff ff e8 25 ec 6f 00 90 <0f> 0b 90 e9 4f fe ff ff e8 17 ec 6f 00 90 0f 0b 90 e9 79 fe ff ff
RSP: 0018:ffffc90002fb7700 EFLAGS: 00010293
RAX: 0000000000000000 RBX: ffff88801e0186c8 RCX: ffffffff811d855d
RDX: ffff888022f9a440 RSI: ffffffff811d856b RDI: 0000000000000005
RBP: ffff888024c50370 R08: 0000000000000005 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffff938d6090 R12: 1ffff920005f6ee1
R13: ffffc90000fae000 R14: 0000000000000000 R15: 0000000000000001
FS:  00007fe2bd3e76c0(0000) GS:ffff88806b000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ccab488ee8 CR3: 000000002d4ee000 CR4: 0000000000352ef0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 kvm_mmu_free_roots+0x621/0x710 arch/x86/kvm/mmu/mmu.c:3631
 kvm_mmu_unload+0x42/0x150 arch/x86/kvm/mmu/mmu.c:5638
 kvm_mmu_reset_context arch/x86/kvm/mmu/mmu.c:5596 [inline]
 kvm_mmu_after_set_cpuid+0x14d/0x300 arch/x86/kvm/mmu/mmu.c:5585
 kvm_vcpu_after_set_cpuid arch/x86/kvm/cpuid.c:386 [inline]
 kvm_set_cpuid+0x1ff1/0x3570 arch/x86/kvm/cpuid.c:460
 kvm_vcpu_ioctl_set_cpuid2+0xe7/0x160 arch/x86/kvm/cpuid.c:527
 kvm_arch_vcpu_ioctl+0x26b7/0x4310 arch/x86/kvm/x86.c:5946
 kvm_vcpu_ioctl+0xa2c/0x1090 arch/x86/kvm/../../../virt/kvm/kvm_main.c:4620
 vfs_ioctl fs/ioctl.c:51 [inline]
 __do_sys_ioctl fs/ioctl.c:904 [inline]
 __se_sys_ioctl fs/ioctl.c:890 [inline]
 __x64_sys_ioctl+0x193/0x220 fs/ioctl.c:890
 do_syscall_x64 arch/x86/entry/common.c:52 [inline]
 do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
 entry_SYSCALL_64_after_hwframe+0x6d/0x75
RIP: 0033:0x7fe2bd42e06b
Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
RSP: 002b:00007fe2bd3e5710 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007fe2bd3e5de0 RCX: 00007fe2bd42e06b
RDX: 00007fe2bd3e5de0 RSI: 000000004008ae90 RDI: 0000000000000006
RBP: 0000000000000000 R08: 0000000000000007 R09: 00000000000000eb
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000080
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000006
 </TASK>


---
This report is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this issue. See:
https://goo.gl/tpsmEJ#status for how to communicate with syzbot.

If the report is already addressed, let syzbot know by replying with:
#syz fix: exact-commit-title

If you want syzbot to run the reproducer, reply with:
#syz test: git://repo/address.git branch-or-commit-hash
If you attach or paste a git patch, syzbot will apply it before testing.

If you want to overwrite report's subsystems, reply with:
#syz set subsystems: new-subsystem
(See the list of subsystem names on the web dashboard)

If the report is a duplicate of another one, reply with:
#syz dup: exact-subject-of-another-report

If you want to undo deduplication, reply with:
#syz undup

Re: [syzbot] [kvm?] WARNING in mmu_free_root_page

[Phi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 5135 bytes --]

On 3/29/2024 11:55 AM, syzbot wrote:
> Hello,
> 
> syzbot found the following issue on:
> 
> HEAD commit:    928a87efa423 Merge tag 'gfs2-v6.8-fix' of git://git.kernel..
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=127c0546180000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=f64ec427e98bccd7
> dashboard link: https://syzkaller.appspot.com/bug?extid=dc308fcfcd53f987de73
> compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=110481f1180000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=177049a5180000
> 
> Downloadable assets:
> disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-928a87ef.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/7979568a5a16/vmlinux-928a87ef.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/1bc6e1d480e3/bzImage-928a87ef.xz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+dc308fcfcd53f987de73@syzkaller.appspotmail.com
> 
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 5187 at arch/x86/kvm/mmu/mmu.c:3579 mmu_free_root_page+0x36c/0x3f0 arch/x86/kvm/mmu/mmu.c:3579
> Modules linked in:
> CPU: 0 PID: 5187 Comm: syz-executor400 Not tainted 6.9.0-rc1-syzkaller-00005-g928a87efa423 #0
> Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> RIP: 0010:mmu_free_root_page+0x36c/0x3f0 arch/x86/kvm/mmu/mmu.c:3579
> Code: 00 49 8d 7d 18 be 01 00 00 00 e8 8f 32 c0 09 31 ff 41 89 c6 89 c6 e8 13 e7 6f 00 45 85 f6 0f 85 5d fe ff ff e8 25 ec 6f 00 90 <0f> 0b 90 e9 4f fe ff ff e8 17 ec 6f 00 90 0f 0b 90 e9 79 fe ff ff
> RSP: 0018:ffffc90002fb7700 EFLAGS: 00010293
> RAX: 0000000000000000 RBX: ffff88801e0186c8 RCX: ffffffff811d855d
> RDX: ffff888022f9a440 RSI: ffffffff811d856b RDI: 0000000000000005
> RBP: ffff888024c50370 R08: 0000000000000005 R09: 0000000000000000
> R10: 0000000000000000 R11: ffffffff938d6090 R12: 1ffff920005f6ee1
> R13: ffffc90000fae000 R14: 0000000000000000 R15: 0000000000000001
> FS:  00007fe2bd3e76c0(0000) GS:ffff88806b000000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 000055ccab488ee8 CR3: 000000002d4ee000 CR4: 0000000000352ef0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>   <TASK>
>   kvm_mmu_free_roots+0x621/0x710 arch/x86/kvm/mmu/mmu.c:3631
>   kvm_mmu_unload+0x42/0x150 arch/x86/kvm/mmu/mmu.c:5638
>   kvm_mmu_reset_context arch/x86/kvm/mmu/mmu.c:5596 [inline]
>   kvm_mmu_after_set_cpuid+0x14d/0x300 arch/x86/kvm/mmu/mmu.c:5585
>   kvm_vcpu_after_set_cpuid arch/x86/kvm/cpuid.c:386 [inline]
>   kvm_set_cpuid+0x1ff1/0x3570 arch/x86/kvm/cpuid.c:460
>   kvm_vcpu_ioctl_set_cpuid2+0xe7/0x160 arch/x86/kvm/cpuid.c:527
>   kvm_arch_vcpu_ioctl+0x26b7/0x4310 arch/x86/kvm/x86.c:5946
>   kvm_vcpu_ioctl+0xa2c/0x1090 arch/x86/kvm/../../../virt/kvm/kvm_main.c:4620
>   vfs_ioctl fs/ioctl.c:51 [inline]
>   __do_sys_ioctl fs/ioctl.c:904 [inline]
>   __se_sys_ioctl fs/ioctl.c:890 [inline]
>   __x64_sys_ioctl+0x193/0x220 fs/ioctl.c:890
>   do_syscall_x64 arch/x86/entry/common.c:52 [inline]
>   do_syscall_64+0xd2/0x260 arch/x86/entry/common.c:83
>   entry_SYSCALL_64_after_hwframe+0x6d/0x75
> RIP: 0033:0x7fe2bd42e06b
> Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
> RSP: 002b:00007fe2bd3e5710 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> RAX: ffffffffffffffda RBX: 00007fe2bd3e5de0 RCX: 00007fe2bd42e06b
> RDX: 00007fe2bd3e5de0 RSI: 000000004008ae90 RDI: 0000000000000006
> RBP: 0000000000000000 R08: 0000000000000007 R09: 00000000000000eb
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000080
> R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000006
>   </TASK>
> 
> 
> ---
> This report is generated by a bot. It may contain errors.
> See https://goo.gl/tpsmEJ for more information about syzbot.
> syzbot engineers can be reached at syzkaller@googlegroups.com.
> 
> syzbot will keep track of this issue. See:
> https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> 
> If the report is already addressed, let syzbot know by replying with:
> #syz fix: exact-commit-title
> 
> If you want syzbot to run the reproducer, reply with:
> #syz test: git://repo/address.git branch-or-commit-hash
> If you attach or paste a git patch, syzbot will apply it before testing.
> 
> If you want to overwrite report's subsystems, reply with:
> #syz set subsystems: new-subsystem
> (See the list of subsystem names on the web dashboard)
> 
> If the report is a duplicate of another one, reply with:
> #syz dup: exact-subject-of-another-report
> 
> If you want to undo deduplication, reply with:
> #syz undup
> 
Shadow TDP

#syz test:
git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master

[-- Attachment #2: free_page.patch --]
[-- Type: text/plain, Size: 558 bytes --]

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 992e651540e8..b4275dc22d21 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3591,7 +3591,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
 void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu,
                        ulong roots_to_free)
 {
-       bool is_tdp_mmu = tdp_mmu_enabled && mmu->root_role.direct;
+       bool is_tdp_mmu = tdp_mmu_enabled;
        int i;
        LIST_HEAD(invalid_list);
        bool free_active_root;

Re: [syzbot] [kvm?] WARNING in mmu_free_root_page

[syzbot]

Hello,

syzbot has tested the proposed patch and the reproducer did not trigger any issue:

Reported-and-tested-by: syzbot+dc308fcfcd53f987de73@syzkaller.appspotmail.com

Tested on:

commit:         a6bd6c93 Add linux-next specific files for 20240328
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=108a85b1180000
kernel config:  https://syzkaller.appspot.com/x/.config?x=45a8d76f0242b787
dashboard link: https://syzkaller.appspot.com/bug?extid=dc308fcfcd53f987de73
compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
patch:          https://syzkaller.appspot.com/x/patch.diff?x=10167af9180000

Note: testing is done by a robot and is best-effort only.

Re: [syzbot] [kvm?] WARNING in mmu_free_root_page

[Sean Christopherson]

On Mon, Apr 01, 2024, Phi Nguyen wrote:
> On 3/29/2024 11:55 AM, syzbot wrote:
> > Hello,
> > 
> > syzbot found the following issue on:
> > 
> > HEAD commit:    928a87efa423 Merge tag 'gfs2-v6.8-fix' of git://git.kernel..
> > git tree:       upstream
> > console output: https://syzkaller.appspot.com/x/log.txt?x=127c0546180000
> > kernel config:  https://syzkaller.appspot.com/x/.config?x=f64ec427e98bccd7
> > dashboard link: https://syzkaller.appspot.com/bug?extid=dc308fcfcd53f987de73
> > compiler:       gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40
> > syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=110481f1180000
> > C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=177049a5180000
> > 
> > Downloadable assets:
> > disk image (non-bootable): https://storage.googleapis.com/syzbot-assets/7bc7510fe41f/non_bootable_disk-928a87ef.raw.xz
> > vmlinux: https://storage.googleapis.com/syzbot-assets/7979568a5a16/vmlinux-928a87ef.xz
> > kernel image: https://storage.googleapis.com/syzbot-assets/1bc6e1d480e3/bzImage-928a87ef.xz
> > 
> > IMPORTANT: if you fix the issue, please add the following tag to the commit:
> > Reported-by: syzbot+dc308fcfcd53f987de73@syzkaller.appspotmail.com

...

> > This report is generated by a bot. It may contain errors.
> > See https://goo.gl/tpsmEJ for more information about syzbot.
> > syzbot engineers can be reached at syzkaller@googlegroups.com.
> > 
> > syzbot will keep track of this issue. See:
> > https://goo.gl/tpsmEJ#status for how to communicate with syzbot.
> > 
> > If the report is already addressed, let syzbot know by replying with:
> > #syz fix: exact-commit-title
> > 
> > If you want syzbot to run the reproducer, reply with:
> > #syz test: git://repo/address.git branch-or-commit-hash
> > If you attach or paste a git patch, syzbot will apply it before testing.
> > 
> > If you want to overwrite report's subsystems, reply with:
> > #syz set subsystems: new-subsystem
> > (See the list of subsystem names on the web dashboard)
> > 
> > If the report is a duplicate of another one, reply with:
> > #syz dup: exact-subject-of-another-report
> > 
> > If you want to undo deduplication, reply with:
> > #syz undup
> > 
> Shadow TDP
> 
> #syz test:
> git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master

> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index 992e651540e8..b4275dc22d21 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -3591,7 +3591,7 @@ static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
>  void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu,
>                         ulong roots_to_free)
>  {
> -       bool is_tdp_mmu = tdp_mmu_enabled && mmu->root_role.direct;
> +       bool is_tdp_mmu = tdp_mmu_enabled;

This isn't a proper fix.  It would actually make a relative benign bug (taking
mmu_lock for write instead of read) far worse (taking mmu_lock for read instead
of write).  The reproducer doesn't fail because it's not actually running nested
VMs, just doing fun things with KVM_SET_CPUID2.

The issue is that kvm_mmu_after_set_cpuid() clobbers the entire role, which
results in mmu->root_role.direct being garbage.  That results in a false negative,
but as above it's quite benign as it simply means KVM takes mmu_lock for write,
when acquiring for read would suffice.

kvm_mmu_page_role.invalid already exists, we just have never used it for the
root_role.  Unless I'm missing something, the below is the simplest fix.  I'll
post a patch tomorrow, assuming testing goes well.

diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 77d1072b130d..2a6c573e0c63 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -5540,9 +5540,9 @@ void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu)
         * that problem is swept under the rug; KVM's CPUID API is horrific and
         * it's all but impossible to solve it without introducing a new API.
         */
-       vcpu->arch.root_mmu.root_role.word = 0;
-       vcpu->arch.guest_mmu.root_role.word = 0;
-       vcpu->arch.nested_mmu.root_role.word = 0;
+       vcpu->arch.root_mmu.root_role.invalid = 1;
+       vcpu->arch.guest_mmu.root_role.invalid = 1;
+       vcpu->arch.nested_mmu.root_role.invalid = 1;
        vcpu->arch.root_mmu.cpu_role.ext.valid = 0;
        vcpu->arch.guest_mmu.cpu_role.ext.valid = 0;
        vcpu->arch.nested_mmu.cpu_role.ext.valid = 0;