Re: netconsole deadlock with virtnet

[Leon Romanovsky <leon@kernel.org>]

On Thu, Nov 19, 2020 at 01:55:53PM +0100, Petr Mladek wrote:
> On Tue 2020-11-17 09:33:25, Steven Rostedt wrote:
> > On Tue, 17 Nov 2020 12:23:41 +0200
> > Leon Romanovsky <leon@kernel.org> wrote:
> >
> > > Hi,
> > >
> > > Approximately two weeks ago, our regression team started to experience those
> > > netconsole splats. The tested code is Linus's master (-rc4) + netdev net-next
> > > + netdev net-rc.
> > >
> > > Such splats are random and we can't bisect because there is no stable reproducer.
> > >
> > > Any idea, what is the root cause?
> > >
> > > [   21.149739]                       __do_sys_finit_module+0xbc/0x12c
> > > [   21.149740]                       __arm64_sys_finit_module+0x28/0x34
> > > [   21.149741]                       el0_svc_common.constprop.0+0x84/0x200
> > > [   21.149742]                       do_el0_svc+0x2c/0x90
> > > [   21.149743]                       el0_svc+0x18/0x50
> > > [   21.149744]                       el0_sync_handler+0xe0/0x350
> > > [   21.149745]                       el0_sync+0x158/0x180
> > > [   21.149746]  }
> > > [   21.149747]  ... key      at: [<ffff8000093d4018>] target_list_lock+0x18/0xfffffffffffff000 [netconsole]
> > > [   21.149748]  ..
> > > [   21.149750] Lost 190 message(s)!
> >
> > It really sucks that we lose 190 messages that would help to decipher this
> > more. :-p
>
> The message commes from the printk_safe code. The size can be
> increased by CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT.
>
> > Because I'm not sure where the xmit_lock is taken while holding the
> > target_list_lock. But the above does show that printk() calls write_msg()
> > while holding the console_lock, and write_msg() takes the target_list_lock.
> >
> > Thus, the fix would ether require disabling interrupts every time the
> > xmit_lock is taken, or to get it from being taken while holding the
> > target_list_lock.
>
> It seems that the missing messages might help to find the root of
> the problem.

Sorry for not being very responsive, I was in internet-free zone :).

I'll increase CONFIG_PRINTK_SAFE_LOG_BUF_SHIFT from 13 to be 26, let's
see what night run will give us.

Thanks

>
> Best Regards,
> Petr
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization