Re: INFO: task hung in vhost_net_stop_vq

Re: INFO: task hung in vhost_net_stop_vq

[Michael S. Tsirkin]

Looks like more iotlb locking mess?

On Tue, Mar 19, 2019 at 10:21:00PM -0700, syzbot wrote:
> syzbot has bisected this bug to:
> 
> commit 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c
> Author: Jason Wang <jasowang@redhat.com>
> Date:   Thu Jun 23 06:04:32 2016 +0000
> 
>     vhost: new device IOTLB API
> 
> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1486ad27200000
> start commit:   6b1e6cc7 vhost: new device IOTLB API
> git tree:       upstream
> final crash:    https://syzkaller.appspot.com/x/report.txt?x=1686ad27200000
> console output: https://syzkaller.appspot.com/x/log.txt?x=1286ad27200000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=c94f9f0c0363db4b
> dashboard link: https://syzkaller.appspot.com/bug?extid=d21e6e297322a900c128
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=141db34d400000
> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=108ef293400000
> 
> Reported-by: syzbot+d21e6e297322a900c128@syzkaller.appspotmail.com
> Fixes: 6b1e6cc7 ("vhost: new device IOTLB API")

Re: INFO: task hung in vhost_net_stop_vq

[Jason Wang]

On 2019/3/25 下午10:02, Michael S. Tsirkin wrote:
> Looks like more iotlb locking mess?


Looking at the calltrace:

[  221.743675] =============================================
[  221.744297] [ INFO: possible recursive locking detected ]
[  221.744944] 4.7.0+ #1 Not tainted
[  221.745326] ---------------------------------------------
[  221.746128] syz-executor1/6823 is trying to acquire lock:
[  221.746737]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
[  221.747789]
[  221.747789] but task is already holding lock:
[  221.748470]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
[  221.749535]
[  221.749535] other info that might help us debug this:
[  221.750280]  Possible unsafe locking scenario:
[  221.750280]
[  221.750946]        CPU0
[  221.751232]        ----
[  221.751523]   lock(&vq->mutex);
[  221.751922]   lock(&vq->mutex);
[  221.752339]
[  221.752339]  *** DEADLOCK ***
[  221.752339]

I could not think of a path that can hit this. And I could not reproduce with the reproducer in the link in net-next.

Thanks


>
> On Tue, Mar 19, 2019 at 10:21:00PM -0700, syzbot wrote:
>> syzbot has bisected this bug to:
>>
>> commit 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c
>> Author: Jason Wang <jasowang@redhat.com>
>> Date:   Thu Jun 23 06:04:32 2016 +0000
>>
>>      vhost: new device IOTLB API
>>
>> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1486ad27200000
>> start commit:   6b1e6cc7 vhost: new device IOTLB API
>> git tree:       upstream
>> final crash:    https://syzkaller.appspot.com/x/report.txt?x=1686ad27200000
>> console output: https://syzkaller.appspot.com/x/log.txt?x=1286ad27200000
>> kernel config:  https://syzkaller.appspot.com/x/.config?x=c94f9f0c0363db4b
>> dashboard link: https://syzkaller.appspot.com/bug?extid=d21e6e297322a900c128
>> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=141db34d400000
>> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=108ef293400000
>>
>> Reported-by: syzbot+d21e6e297322a900c128@syzkaller.appspotmail.com
>> Fixes: 6b1e6cc7 ("vhost: new device IOTLB API")
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: INFO: task hung in vhost_net_stop_vq

[Dmitry Vyukov via Virtualization]

On Tue, Mar 26, 2019 at 11:17 AM Jason Wang <jasowang@redhat.com> wrote:
>
>
> On 2019/3/25 下午10:02, Michael S. Tsirkin wrote:
> > Looks like more iotlb locking mess?
>
>
> Looking at the calltrace:
>
> [  221.743675] =============================================
> [  221.744297] [ INFO: possible recursive locking detected ]
> [  221.744944] 4.7.0+ #1 Not tainted
> [  221.745326] ---------------------------------------------
> [  221.746128] syz-executor1/6823 is trying to acquire lock:
> [  221.746737]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
> [  221.747789]
> [  221.747789] but task is already holding lock:
> [  221.748470]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
> [  221.749535]
> [  221.749535] other info that might help us debug this:
> [  221.750280]  Possible unsafe locking scenario:
> [  221.750280]
> [  221.750946]        CPU0
> [  221.751232]        ----
> [  221.751523]   lock(&vq->mutex);
> [  221.751922]   lock(&vq->mutex);
> [  221.752339]
> [  221.752339]  *** DEADLOCK ***
> [  221.752339]
>
> I could not think of a path that can hit this. And I could not reproduce with the reproducer in the link in net-next.


Looking at the bisection log, syzbot is able to reproduce this
super-reliably on multiple kernel revisions. Are you sure you are
using the right config/revision? What else can be in play? syzbot uses
VMs. The image is available.


> Thanks
>
>
> >
> > On Tue, Mar 19, 2019 at 10:21:00PM -0700, syzbot wrote:
> >> syzbot has bisected this bug to:
> >>
> >> commit 6b1e6cc7855b09a0a9bfa1d9f30172ba366f161c
> >> Author: Jason Wang <jasowang@redhat.com>
> >> Date:   Thu Jun 23 06:04:32 2016 +0000
> >>
> >>      vhost: new device IOTLB API
> >>
> >> bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=1486ad27200000
> >> start commit:   6b1e6cc7 vhost: new device IOTLB API
> >> git tree:       upstream
> >> final crash:    https://syzkaller.appspot.com/x/report.txt?x=1686ad27200000
> >> console output: https://syzkaller.appspot.com/x/log.txt?x=1286ad27200000
> >> kernel config:  https://syzkaller.appspot.com/x/.config?x=c94f9f0c0363db4b
> >> dashboard link: https://syzkaller.appspot.com/bug?extid=d21e6e297322a900c128
> >> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=141db34d400000
> >> C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=108ef293400000
> >>
> >> Reported-by: syzbot+d21e6e297322a900c128@syzkaller.appspotmail.com
> >> Fixes: 6b1e6cc7 ("vhost: new device IOTLB API")
>
> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/df4f2cf6-8469-f894-8f45-7c48a6a1801f%40redhat.com.
> For more options, visit https://groups.google.com/d/optout.
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: INFO: task hung in vhost_net_stop_vq

[Jason Wang]

On 2019/3/26 下午6:28, Dmitry Vyukov wrote:
> On Tue, Mar 26, 2019 at 11:17 AM Jason Wang<jasowang@redhat.com>  wrote:
>> On 2019/3/25 下午10:02, Michael S. Tsirkin wrote:
>>> Looks like more iotlb locking mess?
>> Looking at the calltrace:
>>
>> [  221.743675] =============================================
>> [  221.744297] [ INFO: possible recursive locking detected ]
>> [  221.744944] 4.7.0+ #1 Not tainted
>> [  221.745326] ---------------------------------------------
>> [  221.746128] syz-executor1/6823 is trying to acquire lock:
>> [  221.746737]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
>> [  221.747789]
>> [  221.747789] but task is already holding lock:
>> [  221.748470]  (&vq->mutex){+.+...}, at: [<ffffffff84484b70>] vhost_process_iotlb_msg+0xe0/0x9e0
>> [  221.749535]
>> [  221.749535] other info that might help us debug this:
>> [  221.750280]  Possible unsafe locking scenario:
>> [  221.750280]
>> [  221.750946]        CPU0
>> [  221.751232]        ----
>> [  221.751523]   lock(&vq->mutex);
>> [  221.751922]   lock(&vq->mutex);
>> [  221.752339]
>> [  221.752339]  *** DEADLOCK ***
>> [  221.752339]
>>
>> I could not think of a path that can hit this. And I could not reproduce with the reproducer in the link in net-next.
> Looking at the bisection log, syzbot is able to reproduce this
> super-reliably on multiple kernel revisions. Are you sure you are
> using the right config/revision? What else can be in play? syzbot uses
> VMs. The image is available.
>
>

Yes, looks like the reason is vhost accept zero size iova range which 
lead a infinite loop when trying to translate iova. Will post a patch to 
fix this.

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization