[Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Pengfei Xu]

Hi Jason and kernel experts,

Greeting!

Platform: x86 platforms
There is "try_grab_folio" WARNING in v6.3-rc2 kernel:

All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230313_234302_try_grab_folio
Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/repro.c
v6.3-rc2 issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/eeac8ede17557680855031c6f305ece2378af326_dmesg.log
Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/bisect_info.log
Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/kconfig_origin
"
[   24.259581] memfd_create() without MFD_EXEC nor MFD_NOEXEC_SEAL, pid=327 'systemd'
[   30.909936] ------------[ cut here ]------------
[   30.910782] WARNING: CPU: 1 PID: 527 at mm/gup.c:75 try_grab_folio+0x503/0x740
[   30.911851] Modules linked in:
[   30.912325] CPU: 1 PID: 527 Comm: repro Not tainted 6.3.0-rc2-eeac8ede1755+ #1
[   30.913355] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
[   30.914936] RIP: 0010:try_grab_folio+0x503/0x740
[   30.915615] Code: e3 01 48 89 de e8 6d c1 dd ff 48 85 db 0f 84 7c fe ff ff e8 4f bf dd ff 49 8d 47 ff 48 89 45 d0 e9 73 fe ff ff e8 3d bf dd ff <0f> 0b 31 db e9 d0 fc ff ff e8 2f bf dd ff 48 8b 5d c8 31 ff 48 89
[   30.918172] RSP: 0018:ffffc90000f37908 EFLAGS: 00010046
[   30.918916] RAX: 0000000000000000 RBX: 00000000fffffc02 RCX: ffffffff81504c26
[   30.919886] RDX: 0000000000000000 RSI: ffff88800d030000 RDI: 0000000000000002
[   30.920863] RBP: ffffc90000f37948 R08: 000000000003ca24 R09: 0000000000000008
[   30.921865] R10: 000000000003ca00 R11: 0000000000000023 R12: ffffea000035d540
[   30.922842] R13: 0000000000000001 R14: 0000000000000000 R15: ffffea000035d540
[   30.923792] FS:  00007fecbf659740(0000) GS:ffff88807dd00000(0000) knlGS:0000000000000000
[   30.924873] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   30.925657] CR2: 00000000200011c3 CR3: 000000000ef66006 CR4: 0000000000770ee0
[   30.926604] PKRU: 55555554
[   30.926989] Call Trace:
[   30.927342]  <TASK>
[   30.927684]  internal_get_user_pages_fast+0xd32/0x2200
[   30.928474]  pin_user_pages_fast+0x65/0x90
[   30.929076]  pfn_reader_user_pin+0x376/0x390
[   30.929733]  pfn_reader_next+0x14a/0x7b0
[   30.930301]  ? interval_tree_double_span_iter_update+0x11a/0x140
[   30.931143]  pfn_reader_first+0x140/0x1b0
[   30.931722]  iopt_area_fill_domain+0x74/0x210
[   30.932411]  iopt_table_add_domain+0x30e/0x6e0
[   30.933070]  iommufd_device_selftest_attach+0x7f/0x140
[   30.933811]  iommufd_test+0x10ff/0x16f0
[   30.934371]  ? write_comp_data+0x2f/0x90
[   30.934972]  iommufd_fops_ioctl+0x206/0x330
[   30.935583]  __x64_sys_ioctl+0x10e/0x160
[   30.936161]  ? __pfx_iommufd_fops_ioctl+0x10/0x10
[   30.936828]  do_syscall_64+0x3b/0x90
[   30.937370]  entry_SYSCALL_64_after_hwframe+0x72/0xdc
[   30.938097] RIP: 0033:0x7fecbf77e59d
[   30.938616] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d3 08 0d 00 f7 d8 64 89 01 48
[   30.941034] RSP: 002b:00007ffda07340b8 EFLAGS: 00000217 ORIG_RAX: 0000000000000010
[   30.942065] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fecbf77e59d
[   30.943019] RDX: 00000000200001c0 RSI: 0000000000003ba0 RDI: 0000000000000003
[   30.943980] RBP: 00007ffda07340d0 R08: 00007ffda07340d0 R09: 00007ffda07340d0
[   30.944936] R10: 00007ffda07340d0 R11: 0000000000000217 R12: 0000000000401180
[   30.945894] R13: 00007ffda07341f0 R14: 0000000000000000 R15: 0000000000000000
[   30.946868]  </TASK>
[   30.947194] irq event stamp: 1304
[   30.947659] hardirqs last  enabled at (1303): [<ffffffff815f635b>] mod_objcg_state+0x16b/0x2f0
[   30.948851] hardirqs last disabled at (1304): [<ffffffff8150caee>] internal_get_user_pages_fast+0x205e/0x2200
[   30.950223] softirqs last  enabled at (0): [<ffffffff81117af8>] copy_process+0x1298/0x2d10
[   30.951386] softirqs last disabled at (0): [<0000000000000000>] 0x0
[   30.952232] ---[ end trace 0000000000000000 ]---
"

Bisected and found bad commit:
"
f4b20bb34c83dceade5470288f48f94ce3598ada
iommufd: Add kernel support for testing iommufd
"
It's just a  suspected commit, because reverted above commit on top of v6.3-rc2
and made kernel failed, could not double confirm the commit for this issue.

From reproduced code, it seems related to ioctl IOMMU_TEST_OP_MOCK_DOMAIN,
IOMMU_TEST_OP_CREATE_ACCESS and IOMMU_TEST_OP_ACCESS_PAGES related action.

I hope it's helpful.

Thanks!

---

If you don't need the following environment to reproduce the problem or if you
already have one, please ignore the following information.

How to reproduce:
git clone https://gitlab.com/xupengfe/repro_vm_env.git
cd repro_vm_env
tar -xvf repro_vm_env.tar.gz
cd repro_vm_env; ./start3.sh  // it needs qemu-system-x86_64 and I used v7.1.0
   // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
   // You could change the bzImage_xxx as you want
You could use below command to log in, there is no password for root.
ssh -p 10023 root@localhost

After login vm(virtual machine) successfully, you could transfer reproduced
binary to the vm by below way, and reproduce the problem in vm:
gcc -pthread -o repro repro.c
scp -P 10023 repro root@localhost:/root/

Get the bzImage for target kernel:
Please use target kconfig and copy it to kernel_src/.config
make olddefconfig
make -jx bzImage           //x should equal or less than cpu num your pc has

Fill the bzImage file into above start3.sh to load the target kernel in vm.

Tips:
If you already have qemu-system-x86_64, please ignore below info.
If you want to install qemu v7.1.0 version:
git clone https://github.com/qemu/qemu.git
cd qemu
git checkout -f v7.1.0
mkdir build
cd build
yum install -y ninja-build.x86_64
../configure --target-list=x86_64-softmmu --enable-kvm --enable-vnc --enable-gtk --enable-sdl
make
make install

Thanks!
BR.

Re: [Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Jason Gunthorpe]

On Wed, Mar 15, 2023 at 10:46:40AM +0800, Pengfei Xu wrote:
> Hi Jason and kernel experts,
> 
> Greeting!
> 
> Platform: x86 platforms
> There is "try_grab_folio" WARNING in v6.3-rc2 kernel:
> 
> All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230313_234302_try_grab_folio
> Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/repro.c
> v6.3-rc2 issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/eeac8ede17557680855031c6f305ece2378af326_dmesg.log
> Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/bisect_info.log
> Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/kconfig_origin

Do you have a syzkaller format reproducer for this? It is often
informative.

> If you don't need the following environment to reproduce the problem or if you
> already have one, please ignore the following information.
> 
> How to reproduce:
> git clone https://gitlab.com/xupengfe/repro_vm_env.git
> cd repro_vm_env
> tar -xvf repro_vm_env.tar.gz
> cd repro_vm_env; ./start3.sh  // it needs qemu-system-x86_64 and I used v7.1.0
>    // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
>    // You could change the bzImage_xxx as you want
> You could use below command to log in, there is no password for root.
> ssh -p 10023 root@localhost

These instructions did not result in a reproduction for me :(

Nor did the reproduction work in my usual test VM.

It must be timing sensitive?

The trace says it is touching a page with a corrupted refcount, which
suggests a double free, but it is weird that it would trigger here and
not at the point of the double free...

I don't have a guess what this is

Jason

Re: [Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Pengfei Xu]

Hi Jason,

On 2023-03-31 at 09:24:42 -0300, Jason Gunthorpe wrote:
> On Wed, Mar 15, 2023 at 10:46:40AM +0800, Pengfei Xu wrote:
> > Hi Jason and kernel experts,
> > 
> > Greeting!
> > 
> > Platform: x86 platforms
> > There is "try_grab_folio" WARNING in v6.3-rc2 kernel:
> > 
> > All detailed info: https://github.com/xupengfe/syzkaller_logs/tree/main/230313_234302_try_grab_folio
> > Reproduced code: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/repro.c
> > v6.3-rc2 issue dmesg: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/eeac8ede17557680855031c6f305ece2378af326_dmesg.log
> > Bisect info: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/bisect_info.log
> > Kconfig: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/kconfig_origin
> 
> Do you have a syzkaller format reproducer for this? It is often
> informative.
> 
  Thanks for your suggestion! I will add the repro.prog for new report.
  Updated repro.prog in link: https://github.com/xupengfe/syzkaller_logs/blob/main/230313_234302_try_grab_folio/repro.prog
  I also new added machineInfo0, log0, report0, repro.report, repro.stats in
  link: https://github.com/xupengfe/syzkaller_logs/tree/main/230313_234302_try_grab_folio

> > If you don't need the following environment to reproduce the problem or if you
> > already have one, please ignore the following information.
> > 
> > How to reproduce:
> > git clone https://gitlab.com/xupengfe/repro_vm_env.git
> > cd repro_vm_env
> > tar -xvf repro_vm_env.tar.gz
> > cd repro_vm_env; ./start3.sh  // it needs qemu-system-x86_64 and I used v7.1.0
> >    // start3.sh will load bzImage_2241ab53cbb5cdb08a6b2d4688feb13971058f65 v6.2-rc5 kernel
> >    // You could change the bzImage_xxx as you want
> > You could use below command to log in, there is no password for root.
> > ssh -p 10023 root@localhost
> 
> These instructions did not result in a reproduction for me :(
> 
> Nor did the reproduction work in my usual test VM.
> 
> It must be timing sensitive?
> 
> The trace says it is touching a page with a corrupted refcount, which
> suggests a double free, but it is weird that it would trigger here and
> not at the point of the double free...
> 
> I don't have a guess what this is
> 
  Ah, seems to be related to platform independence, I can reproduce this
  problem on RPL-P platform vm in almost 3 seconds after executing bianry.

  And I could reproduce this issue on ADL-S vm in 3 seconds also, seems it
  could be reproduced on ATOM and big cores platforms.

  Thanks!
  BR.
> Jason

Re: [Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Jason Gunthorpe]

On Fri, Mar 31, 2023 at 10:57:54PM +0800, Pengfei Xu wrote:
>   Ah, seems to be related to platform independence, I can reproduce this
>   problem on RPL-P platform vm in almost 3 seconds after executing bianry.

I think I found it anyhow

Please check the series I just sent against your reproduction for this

Thanks,
Jason

Re: [Syzkaller & bisect] There is "try_grab_folio" WARNING in v6.3-rc2 kernel

[Pengfei Xu]

Hi Jason,

On 2023-03-31 at 12:33:03 -0300, Jason Gunthorpe wrote:
> On Fri, Mar 31, 2023 at 10:57:54PM +0800, Pengfei Xu wrote:
> >   Ah, seems to be related to platform independence, I can reproduce this
> >   problem on RPL-P platform vm in almost 3 seconds after executing bianry.
> 
> I think I found it anyhow
> 
> Please check the series I just sent against your reproduction for this
> 
  Great! Thanks for your fixed patch series!
  I will verify it soon.

  Thanks!
  BR.

> Thanks,
> Jason