irqfd issue

irqfd issue

[Asias He]

Hi,

I am seeing this with linus/master. Any ideas?

[   34.168356] IPv6: ADDRCONF(NETDEV_UP): virbr0: link is not ready
[   36.743758] BUG: unable to handle kernel paging request at 0000000300000029
[   36.745177] IP: [<ffffffff81c08584>] __mutex_lock_slowpath+0x34/0x240
[   36.746576] PGD 0 
[   36.747962] Oops: 0000 [#1] SMP 
[   36.749343] Modules linked in: ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat vhost_net vhost
[   36.750753] CPU: 0 PID: 4260 Comm: qemu-kvm Not tainted 3.9.0+ #752
[   36.752130] Hardware name: Dell Inc. OptiPlex 790/0V5HMK, BIOS A11 12/30/2011
[   36.753495] task: ffff88021fba8000 ti: ffff88021f0c4000 task.ti: ffff88021f0c4000
[   36.754847] RIP: 0010:[<ffffffff81c08584>]  [<ffffffff81c08584>] __mutex_lock_slowpath+0x34/0x240
[   36.756228] RSP: 0018:ffff88021f0c5c88  EFLAGS: 00010202
[   36.757584] RAX: 0000000000000001 RBX: ffff880223ffb420 RCX: 0000000000000000
[   36.758926] RDX: 0000000300000001 RSI: ffff88021f0c5d60 RDI: ffff880223ffb420
[   36.760268] RBP: ffff88021f0c5cf8 R08: ffff88021f0c0000 R09: ffffffff00000000
[   36.761602] R10: ffff8802209c3f10 R11: 0000000000000000 R12: ffff880223ffb420
[   36.762924] R13: ffff88022236c000 R14: ffff8802236213b0 R15: ffff880223ffb420
[   36.764232] FS:  0000000000000000(0000) GS:ffff88022dc00000(0000) knlGS:0000000000000000
[   36.765543] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   36.766869] CR2: 0000000300000029 CR3: 000000000240b000 CR4: 00000000000427f0
[   36.768220] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   36.769565] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   36.770889] Stack:
[   36.772200]  0000000000000092 ffff88021fba8000 ffff88021f0c5cb8 ffffffff81c0b2c7
[   36.773551]  ffff8802210a4fc0 ffff8802210a4fc0 ffff88021f0c5d08 ffffffff810c0f2c
[   36.774884]  000000000000000e ffff880223ffb420 ffff88021f0c5d38 ffff88022236c000
[   36.776194] Call Trace:
[   36.777482]  [<ffffffff81c0b2c7>] ? _raw_spin_unlock_irqrestore+0x37/0x40
[   36.778789]  [<ffffffff810c0f2c>] ? try_to_wake_up+0x1ec/0x290
[   36.780107]  [<ffffffff81c0852b>] mutex_lock+0x2b/0x50
[   36.781420]  [<ffffffff810a9b6d>] flush_workqueue+0x9d/0x560
[   36.782729]  [<ffffffff8100933f>] kvm_irqfd_release+0x8f/0xa0
[   36.784046]  [<ffffffff8100456d>] kvm_vm_release+0x1d/0x30
[   36.785367]  [<ffffffff811a732a>] __fput+0xba/0x240
[   36.786693]  [<ffffffff811a751e>] ____fput+0xe/0x10
[   36.788007]  [<ffffffff810af685>] task_work_run+0xa5/0xe0
[   36.789317]  [<ffffffff81092cd7>] do_exit+0x2d7/0xac0
[   36.790622]  [<ffffffff811a4a04>] ? fsnotify_modify+0x64/0x80
[   36.791896]  [<ffffffff8140327a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[   36.793141]  [<ffffffff81093511>] do_group_exit+0x51/0xc0
[   36.794358]  [<ffffffff81093597>] SyS_exit_group+0x17/0x20
[   36.795547]  [<ffffffff81c13882>] system_call_fastpath+0x16/0x1b
[   36.796731] Code: 55 41 54 53 48 83 ec 48 66 66 66 66 90 65 48 8b 04 25 00 b8 00 00 49 89 fc 48 89 45 98 48 8b 57 18 b8 01 00 00 00 48 85 d2 74 03 <8b> 42 28 85 c0 0f 84 e6 00 00 00 65 48 8b 04 25 08 b8 00 00 48 
[   36.798194] RIP  [<ffffffff81c08584>] __mutex_lock_slowpath+0x34/0x240
[   36.799567]  RSP <ffff88021f0c5c88>
[   36.800943] CR2: 0000000300000029
[   36.813185] ---[ end trace 4877613defb9fc19 ]---
[   36.813188] Fixing recursive fault but reboot is needed!
[   37.011566] usb 2-1.1: link qh8-0601/ffff880223a9c600 start 3 [1/2 us]
[   70.539341] usb usb1: usb port1's DeviceRemovable is changed to 1 according to platform information.
[   70.539546] usb usb2: usb port1's DeviceRemovable is changed to 1 according to platform information.
[   70.862490] nr_pdflush_threads exported in /proc is scheduled for removal
[   70.862558] sysctl: The scan_unevictable_pages sysctl/node-interface has been disabled for lack of a legitimate use case.  If you have one, please send an email to linux-mm@kvack.org.

-- 
Asias

Re: irqfd issue

[Gleb Natapov]

On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> Hi,
> 
> I am seeing this with linus/master. Any ideas?
> 
How reproducible it this? What HEAD are you seeing this with?

--
			Gleb.

Re: irqfd issue

[Asias He]

On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > Hi,
> > 
> > I am seeing this with linus/master. Any ideas?
> > 
> How reproducible it this? What HEAD are you seeing this with?

Almost always. Start a guest with vhost-net on.

QEMU  : e3351000cd682200835763caca87adf708ed1c65
KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 

> --
> 			Gleb.

-- 
Asias

Re: irqfd issue

[Gleb Natapov]

On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > Hi,
> > > 
> > > I am seeing this with linus/master. Any ideas?
> > > 
> > How reproducible it this? What HEAD are you seeing this with?
> 
> Almost always. Start a guest with vhost-net on.
> 
It happens during start or on VM exit? The trace shows do_exit()

> QEMU  : e3351000cd682200835763caca87adf708ed1c65
> KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> 
Can you try with kvm.git next branch?

--
			Gleb.

Re: irqfd issue

[Asias He]

On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > Hi,
> > > > 
> > > > I am seeing this with linus/master. Any ideas?
> > > > 
> > > How reproducible it this? What HEAD are you seeing this with?
> > 
> > Almost always. Start a guest with vhost-net on.
> > 
> It happens during start or on VM exit? The trace shows do_exit()

Yes. It happens when you shutdown the VM.

> > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > 
> Can you try with kvm.git next branch?

/me compiling

> --
> 			Gleb.

-- 
Asias

Re: irqfd issue

[Asias He]

On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > Hi,
> > > > 
> > > > I am seeing this with linus/master. Any ideas?
> > > > 
> > > How reproducible it this? What HEAD are you seeing this with?
> > 
> > Almost always. Start a guest with vhost-net on.
> > 
> It happens during start or on VM exit? The trace shows do_exit()
> 
> > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > 
> Can you try with kvm.git next branch?

With next branch db6ae6158186a17165ef990bda2895ae7594b039,

[   53.386936] device tap0 entered promiscuous mode
[   53.386990] br0: port 2(tap0) entered forwarding state
[   53.386997] br0: port 2(tap0) entered forwarding state
[   80.096275] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   80.098592] IP: [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
[   80.100988] PGD 0 
[   80.103339] Oops: 0002 [#1] SMP 
[   80.105668] Modules linked in: vhost_net
[   80.108005] CPU 6 
[   80.108026] Pid: 4257, comm: qemu-system-x86 Not tainted 3.9.0-rc3+ #757 Dell Inc. OptiPlex 790/0V5HMK
[   80.112662] RIP: 0010:[<ffffffff81c0721e>]  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
[   80.115030] RSP: 0018:ffff880221721cc8  EFLAGS: 00010046
[   80.117365] RAX: 0000000000000100 RBX: ffff88022dcc003f RCX: ffff880221734950
[   80.119688] RDX: ffff8802208f6ca8 RSI: 000000007fffffff RDI: 0000000000000000
[   80.121982] RBP: ffff880221721cc8 R08: 0000000000000002 R09: 0000000000000002
[   80.124230] R10: 00007f7fd01087e0 R11: 0000000000000246 R12: ffff8802208f6ca8
[   80.126489] R13: 0000000000000080 R14: ffff880223e2a900 R15: 0000000000000000
[   80.128718] FS:  00007f7fd38488e0(0000) GS:ffff88022dcc0000(0000) knlGS:0000000000000000
[   80.130921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   80.133130] CR2: 0000000000000000 CR3: 000000022309f000 CR4: 00000000000427e0
[   80.135345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   80.137543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   80.139708] Process qemu-system-x86 (pid: 4257, threadinfo ffff880221720000, task ffff880222bd5640)
[   80.141887] Stack:
[   80.144064]  ffff880221721d08 ffffffff810ac5c5 ffff88022431dc00 0000000000000086
[   80.146316]  0000000000000080 ffff880223e2a900 ffff8802208f6ca8 0000000000000000
[   80.148558]  ffff880221721d48 ffffffff810ac8fe 0000000000000000 ffff880221734000
[   80.150784] Call Trace:
[   80.152975]  [<ffffffff810ac5c5>] __queue_work+0x45/0x2d0
[   80.155193]  [<ffffffff810ac8fe>] queue_work_on+0x8e/0xa0
[   80.157374]  [<ffffffff810ac949>] queue_work+0x19/0x20
[   80.159505]  [<ffffffff81009b6b>] irqfd_deactivate+0x4b/0x60
[   80.161621]  [<ffffffff8100a69d>] kvm_irqfd+0x39d/0x580
[   80.163717]  [<ffffffff81007a27>] kvm_vm_ioctl+0x207/0x5b0
[   80.165832]  [<ffffffff810c9545>] ? update_curr+0xf5/0x180
[   80.167931]  [<ffffffff811b66e8>] do_vfs_ioctl+0x98/0x550
[   80.170013]  [<ffffffff810c1f5e>] ? finish_task_switch+0x4e/0xe0
[   80.172109]  [<ffffffff81c054aa>] ? __schedule+0x2ea/0x710
[   80.174218]  [<ffffffff811b6bf7>] sys_ioctl+0x57/0x90
[   80.176338]  [<ffffffff8140ae9e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
[   80.178467]  [<ffffffff81c0f602>] system_call_fastpath+0x16/0x1b
[   80.180598] Code: c1 ea 08 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 0f b6 03 38 c2 75 f7 48 83 c4 08 5b c9 c3 55 48 89 e5 66 66 66 66 90 b8 00 01 00 00 <f0> 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 0c 0f 1f 00 f3 90 0f 
[   80.183098] RIP  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
[   80.185372]  RSP <ffff880221721cc8>
[   80.187584] CR2: 0000000000000000
[   80.208627] ---[ end trace 13fb1e4b6e5ab21f ]---

> --
> 			Gleb.

-- 
Asias

Re: irqfd issue

[Gleb Natapov]

On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > Hi,
> > > > > 
> > > > > I am seeing this with linus/master. Any ideas?
> > > > > 
> > > > How reproducible it this? What HEAD are you seeing this with?
> > > 
> > > Almost always. Start a guest with vhost-net on.
> > > 
> > It happens during start or on VM exit? The trace shows do_exit()
> > 
> > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > 
> > Can you try with kvm.git next branch?
> 
> With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> 
Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?

> [   53.386936] device tap0 entered promiscuous mode
> [   53.386990] br0: port 2(tap0) entered forwarding state
> [   53.386997] br0: port 2(tap0) entered forwarding state
> [   80.096275] BUG: unable to handle kernel NULL pointer dereference at           (null)
> [   80.098592] IP: [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> [   80.100988] PGD 0 
> [   80.103339] Oops: 0002 [#1] SMP 
> [   80.105668] Modules linked in: vhost_net
> [   80.108005] CPU 6 
> [   80.108026] Pid: 4257, comm: qemu-system-x86 Not tainted 3.9.0-rc3+ #757 Dell Inc. OptiPlex 790/0V5HMK
> [   80.112662] RIP: 0010:[<ffffffff81c0721e>]  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> [   80.115030] RSP: 0018:ffff880221721cc8  EFLAGS: 00010046
> [   80.117365] RAX: 0000000000000100 RBX: ffff88022dcc003f RCX: ffff880221734950
> [   80.119688] RDX: ffff8802208f6ca8 RSI: 000000007fffffff RDI: 0000000000000000
> [   80.121982] RBP: ffff880221721cc8 R08: 0000000000000002 R09: 0000000000000002
> [   80.124230] R10: 00007f7fd01087e0 R11: 0000000000000246 R12: ffff8802208f6ca8
> [   80.126489] R13: 0000000000000080 R14: ffff880223e2a900 R15: 0000000000000000
> [   80.128718] FS:  00007f7fd38488e0(0000) GS:ffff88022dcc0000(0000) knlGS:0000000000000000
> [   80.130921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   80.133130] CR2: 0000000000000000 CR3: 000000022309f000 CR4: 00000000000427e0
> [   80.135345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   80.137543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [   80.139708] Process qemu-system-x86 (pid: 4257, threadinfo ffff880221720000, task ffff880222bd5640)
> [   80.141887] Stack:
> [   80.144064]  ffff880221721d08 ffffffff810ac5c5 ffff88022431dc00 0000000000000086
> [   80.146316]  0000000000000080 ffff880223e2a900 ffff8802208f6ca8 0000000000000000
> [   80.148558]  ffff880221721d48 ffffffff810ac8fe 0000000000000000 ffff880221734000
> [   80.150784] Call Trace:
> [   80.152975]  [<ffffffff810ac5c5>] __queue_work+0x45/0x2d0
> [   80.155193]  [<ffffffff810ac8fe>] queue_work_on+0x8e/0xa0
> [   80.157374]  [<ffffffff810ac949>] queue_work+0x19/0x20
> [   80.159505]  [<ffffffff81009b6b>] irqfd_deactivate+0x4b/0x60
> [   80.161621]  [<ffffffff8100a69d>] kvm_irqfd+0x39d/0x580
> [   80.163717]  [<ffffffff81007a27>] kvm_vm_ioctl+0x207/0x5b0
> [   80.165832]  [<ffffffff810c9545>] ? update_curr+0xf5/0x180
> [   80.167931]  [<ffffffff811b66e8>] do_vfs_ioctl+0x98/0x550
> [   80.170013]  [<ffffffff810c1f5e>] ? finish_task_switch+0x4e/0xe0
> [   80.172109]  [<ffffffff81c054aa>] ? __schedule+0x2ea/0x710
> [   80.174218]  [<ffffffff811b6bf7>] sys_ioctl+0x57/0x90
> [   80.176338]  [<ffffffff8140ae9e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> [   80.178467]  [<ffffffff81c0f602>] system_call_fastpath+0x16/0x1b
> [   80.180598] Code: c1 ea 08 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 0f b6 03 38 c2 75 f7 48 83 c4 08 5b c9 c3 55 48 89 e5 66 66 66 66 90 b8 00 01 00 00 <f0> 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 0c 0f 1f 00 f3 90 0f 
> [   80.183098] RIP  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> [   80.185372]  RSP <ffff880221721cc8>
> [   80.187584] CR2: 0000000000000000
> [   80.208627] ---[ end trace 13fb1e4b6e5ab21f ]---
> 
> > --
> > 			Gleb.
> 
> -- 
> Asias

--
			Gleb.

Re: irqfd issue

[Asias He]

On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > Hi,
> > > > > > 
> > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > 
> > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > 
> > > > Almost always. Start a guest with vhost-net on.
> > > > 
> > > It happens during start or on VM exit? The trace shows do_exit()
> > > 
> > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > 
> > > Can you try with kvm.git next branch?
> > 
> > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > 
> Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?

The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
reverted.

> > [   53.386936] device tap0 entered promiscuous mode
> > [   53.386990] br0: port 2(tap0) entered forwarding state
> > [   53.386997] br0: port 2(tap0) entered forwarding state
> > [   80.096275] BUG: unable to handle kernel NULL pointer dereference at           (null)
> > [   80.098592] IP: [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > [   80.100988] PGD 0 
> > [   80.103339] Oops: 0002 [#1] SMP 
> > [   80.105668] Modules linked in: vhost_net
> > [   80.108005] CPU 6 
> > [   80.108026] Pid: 4257, comm: qemu-system-x86 Not tainted 3.9.0-rc3+ #757 Dell Inc. OptiPlex 790/0V5HMK
> > [   80.112662] RIP: 0010:[<ffffffff81c0721e>]  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > [   80.115030] RSP: 0018:ffff880221721cc8  EFLAGS: 00010046
> > [   80.117365] RAX: 0000000000000100 RBX: ffff88022dcc003f RCX: ffff880221734950
> > [   80.119688] RDX: ffff8802208f6ca8 RSI: 000000007fffffff RDI: 0000000000000000
> > [   80.121982] RBP: ffff880221721cc8 R08: 0000000000000002 R09: 0000000000000002
> > [   80.124230] R10: 00007f7fd01087e0 R11: 0000000000000246 R12: ffff8802208f6ca8
> > [   80.126489] R13: 0000000000000080 R14: ffff880223e2a900 R15: 0000000000000000
> > [   80.128718] FS:  00007f7fd38488e0(0000) GS:ffff88022dcc0000(0000) knlGS:0000000000000000
> > [   80.130921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [   80.133130] CR2: 0000000000000000 CR3: 000000022309f000 CR4: 00000000000427e0
> > [   80.135345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > [   80.137543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > [   80.139708] Process qemu-system-x86 (pid: 4257, threadinfo ffff880221720000, task ffff880222bd5640)
> > [   80.141887] Stack:
> > [   80.144064]  ffff880221721d08 ffffffff810ac5c5 ffff88022431dc00 0000000000000086
> > [   80.146316]  0000000000000080 ffff880223e2a900 ffff8802208f6ca8 0000000000000000
> > [   80.148558]  ffff880221721d48 ffffffff810ac8fe 0000000000000000 ffff880221734000
> > [   80.150784] Call Trace:
> > [   80.152975]  [<ffffffff810ac5c5>] __queue_work+0x45/0x2d0
> > [   80.155193]  [<ffffffff810ac8fe>] queue_work_on+0x8e/0xa0
> > [   80.157374]  [<ffffffff810ac949>] queue_work+0x19/0x20
> > [   80.159505]  [<ffffffff81009b6b>] irqfd_deactivate+0x4b/0x60
> > [   80.161621]  [<ffffffff8100a69d>] kvm_irqfd+0x39d/0x580
> > [   80.163717]  [<ffffffff81007a27>] kvm_vm_ioctl+0x207/0x5b0
> > [   80.165832]  [<ffffffff810c9545>] ? update_curr+0xf5/0x180
> > [   80.167931]  [<ffffffff811b66e8>] do_vfs_ioctl+0x98/0x550
> > [   80.170013]  [<ffffffff810c1f5e>] ? finish_task_switch+0x4e/0xe0
> > [   80.172109]  [<ffffffff81c054aa>] ? __schedule+0x2ea/0x710
> > [   80.174218]  [<ffffffff811b6bf7>] sys_ioctl+0x57/0x90
> > [   80.176338]  [<ffffffff8140ae9e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > [   80.178467]  [<ffffffff81c0f602>] system_call_fastpath+0x16/0x1b
> > [   80.180598] Code: c1 ea 08 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 0f b6 03 38 c2 75 f7 48 83 c4 08 5b c9 c3 55 48 89 e5 66 66 66 66 90 b8 00 01 00 00 <f0> 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 0c 0f 1f 00 f3 90 0f 
> > [   80.183098] RIP  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > [   80.185372]  RSP <ffff880221721cc8>
> > [   80.187584] CR2: 0000000000000000
> > [   80.208627] ---[ end trace 13fb1e4b6e5ab21f ]---
> > 
> > > --
> > > 			Gleb.
> > 
> > -- 
> > Asias
> 
> --
> 			Gleb.

-- 
Asias

Re: irqfd issue

[Gleb Natapov]

On Tue, May 07, 2013 at 05:10:41PM +0800, Asias He wrote:
> On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> > On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > > 
> > > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > > 
> > > > > Almost always. Start a guest with vhost-net on.
> > > > > 
> > > > It happens during start or on VM exit? The trace shows do_exit()
> > > > 
> > > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > > 
> > > > Can you try with kvm.git next branch?
> > > 
> > > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > > 
> > Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?
> 
> The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
> reverted.
> 
Cornelia, any ideas?

> > > [   53.386936] device tap0 entered promiscuous mode
> > > [   53.386990] br0: port 2(tap0) entered forwarding state
> > > [   53.386997] br0: port 2(tap0) entered forwarding state
> > > [   80.096275] BUG: unable to handle kernel NULL pointer dereference at           (null)
> > > [   80.098592] IP: [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > [   80.100988] PGD 0 
> > > [   80.103339] Oops: 0002 [#1] SMP 
> > > [   80.105668] Modules linked in: vhost_net
> > > [   80.108005] CPU 6 
> > > [   80.108026] Pid: 4257, comm: qemu-system-x86 Not tainted 3.9.0-rc3+ #757 Dell Inc. OptiPlex 790/0V5HMK
> > > [   80.112662] RIP: 0010:[<ffffffff81c0721e>]  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > [   80.115030] RSP: 0018:ffff880221721cc8  EFLAGS: 00010046
> > > [   80.117365] RAX: 0000000000000100 RBX: ffff88022dcc003f RCX: ffff880221734950
> > > [   80.119688] RDX: ffff8802208f6ca8 RSI: 000000007fffffff RDI: 0000000000000000
> > > [   80.121982] RBP: ffff880221721cc8 R08: 0000000000000002 R09: 0000000000000002
> > > [   80.124230] R10: 00007f7fd01087e0 R11: 0000000000000246 R12: ffff8802208f6ca8
> > > [   80.126489] R13: 0000000000000080 R14: ffff880223e2a900 R15: 0000000000000000
> > > [   80.128718] FS:  00007f7fd38488e0(0000) GS:ffff88022dcc0000(0000) knlGS:0000000000000000
> > > [   80.130921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > [   80.133130] CR2: 0000000000000000 CR3: 000000022309f000 CR4: 00000000000427e0
> > > [   80.135345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > [   80.137543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > [   80.139708] Process qemu-system-x86 (pid: 4257, threadinfo ffff880221720000, task ffff880222bd5640)
> > > [   80.141887] Stack:
> > > [   80.144064]  ffff880221721d08 ffffffff810ac5c5 ffff88022431dc00 0000000000000086
> > > [   80.146316]  0000000000000080 ffff880223e2a900 ffff8802208f6ca8 0000000000000000
> > > [   80.148558]  ffff880221721d48 ffffffff810ac8fe 0000000000000000 ffff880221734000
> > > [   80.150784] Call Trace:
> > > [   80.152975]  [<ffffffff810ac5c5>] __queue_work+0x45/0x2d0
> > > [   80.155193]  [<ffffffff810ac8fe>] queue_work_on+0x8e/0xa0
> > > [   80.157374]  [<ffffffff810ac949>] queue_work+0x19/0x20
> > > [   80.159505]  [<ffffffff81009b6b>] irqfd_deactivate+0x4b/0x60
> > > [   80.161621]  [<ffffffff8100a69d>] kvm_irqfd+0x39d/0x580
> > > [   80.163717]  [<ffffffff81007a27>] kvm_vm_ioctl+0x207/0x5b0
> > > [   80.165832]  [<ffffffff810c9545>] ? update_curr+0xf5/0x180
> > > [   80.167931]  [<ffffffff811b66e8>] do_vfs_ioctl+0x98/0x550
> > > [   80.170013]  [<ffffffff810c1f5e>] ? finish_task_switch+0x4e/0xe0
> > > [   80.172109]  [<ffffffff81c054aa>] ? __schedule+0x2ea/0x710
> > > [   80.174218]  [<ffffffff811b6bf7>] sys_ioctl+0x57/0x90
> > > [   80.176338]  [<ffffffff8140ae9e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > [   80.178467]  [<ffffffff81c0f602>] system_call_fastpath+0x16/0x1b
> > > [   80.180598] Code: c1 ea 08 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 0f b6 03 38 c2 75 f7 48 83 c4 08 5b c9 c3 55 48 89 e5 66 66 66 66 90 b8 00 01 00 00 <f0> 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 0c 0f 1f 00 f3 90 0f 
> > > [   80.183098] RIP  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > [   80.185372]  RSP <ffff880221721cc8>
> > > [   80.187584] CR2: 0000000000000000
> > > [   80.208627] ---[ end trace 13fb1e4b6e5ab21f ]---
> > > 
> > > > --
> > > > 			Gleb.
> > > 
> > > -- 
> > > Asias
> > 
> > --
> > 			Gleb.
> 
> -- 
> Asias

--
			Gleb.

Re: irqfd issue

[Cornelia Huck]

On Tue, 7 May 2013 12:12:09 +0300
Gleb Natapov <gleb@redhat.com> wrote:

> On Tue, May 07, 2013 at 05:10:41PM +0800, Asias He wrote:
> > On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> > > On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > > > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > > > 
> > > > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > > > 
> > > > > > Almost always. Start a guest with vhost-net on.
> > > > > > 
> > > > > It happens during start or on VM exit? The trace shows do_exit()
> > > > > 
> > > > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > > > 
> > > > > Can you try with kvm.git next branch?
> > > > 
> > > > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > > > 
> > > Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?
> > 
> > The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
> > reverted.
> > 
> Cornelia, any ideas?

irqfd_deactivate before kvm_init or after kvm_exit? Let me look...

> 
> > > > [   53.386936] device tap0 entered promiscuous mode
> > > > [   53.386990] br0: port 2(tap0) entered forwarding state
> > > > [   53.386997] br0: port 2(tap0) entered forwarding state
> > > > [   80.096275] BUG: unable to handle kernel NULL pointer dereference at           (null)
> > > > [   80.098592] IP: [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > > [   80.100988] PGD 0 
> > > > [   80.103339] Oops: 0002 [#1] SMP 
> > > > [   80.105668] Modules linked in: vhost_net
> > > > [   80.108005] CPU 6 
> > > > [   80.108026] Pid: 4257, comm: qemu-system-x86 Not tainted 3.9.0-rc3+ #757 Dell Inc. OptiPlex 790/0V5HMK
> > > > [   80.112662] RIP: 0010:[<ffffffff81c0721e>]  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > > [   80.115030] RSP: 0018:ffff880221721cc8  EFLAGS: 00010046
> > > > [   80.117365] RAX: 0000000000000100 RBX: ffff88022dcc003f RCX: ffff880221734950
> > > > [   80.119688] RDX: ffff8802208f6ca8 RSI: 000000007fffffff RDI: 0000000000000000
> > > > [   80.121982] RBP: ffff880221721cc8 R08: 0000000000000002 R09: 0000000000000002
> > > > [   80.124230] R10: 00007f7fd01087e0 R11: 0000000000000246 R12: ffff8802208f6ca8
> > > > [   80.126489] R13: 0000000000000080 R14: ffff880223e2a900 R15: 0000000000000000
> > > > [   80.128718] FS:  00007f7fd38488e0(0000) GS:ffff88022dcc0000(0000) knlGS:0000000000000000
> > > > [   80.130921] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   80.133130] CR2: 0000000000000000 CR3: 000000022309f000 CR4: 00000000000427e0
> > > > [   80.135345] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> > > > [   80.137543] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> > > > [   80.139708] Process qemu-system-x86 (pid: 4257, threadinfo ffff880221720000, task ffff880222bd5640)
> > > > [   80.141887] Stack:
> > > > [   80.144064]  ffff880221721d08 ffffffff810ac5c5 ffff88022431dc00 0000000000000086
> > > > [   80.146316]  0000000000000080 ffff880223e2a900 ffff8802208f6ca8 0000000000000000
> > > > [   80.148558]  ffff880221721d48 ffffffff810ac8fe 0000000000000000 ffff880221734000
> > > > [   80.150784] Call Trace:
> > > > [   80.152975]  [<ffffffff810ac5c5>] __queue_work+0x45/0x2d0
> > > > [   80.155193]  [<ffffffff810ac8fe>] queue_work_on+0x8e/0xa0
> > > > [   80.157374]  [<ffffffff810ac949>] queue_work+0x19/0x20
> > > > [   80.159505]  [<ffffffff81009b6b>] irqfd_deactivate+0x4b/0x60
> > > > [   80.161621]  [<ffffffff8100a69d>] kvm_irqfd+0x39d/0x580
> > > > [   80.163717]  [<ffffffff81007a27>] kvm_vm_ioctl+0x207/0x5b0
> > > > [   80.165832]  [<ffffffff810c9545>] ? update_curr+0xf5/0x180
> > > > [   80.167931]  [<ffffffff811b66e8>] do_vfs_ioctl+0x98/0x550
> > > > [   80.170013]  [<ffffffff810c1f5e>] ? finish_task_switch+0x4e/0xe0
> > > > [   80.172109]  [<ffffffff81c054aa>] ? __schedule+0x2ea/0x710
> > > > [   80.174218]  [<ffffffff811b6bf7>] sys_ioctl+0x57/0x90
> > > > [   80.176338]  [<ffffffff8140ae9e>] ? trace_hardirqs_on_thunk+0x3a/0x3c
> > > > [   80.178467]  [<ffffffff81c0f602>] system_call_fastpath+0x16/0x1b
> > > > [   80.180598] Code: c1 ea 08 38 c2 74 0f 66 0f 1f 44 00 00 f3 90 0f b6 03 38 c2 75 f7 48 83 c4 08 5b c9 c3 55 48 89 e5 66 66 66 66 90 b8 00 01 00 00 <f0> 66 0f c1 07 89 c2 66 c1 ea 08 38 c2 74 0c 0f 1f 00 f3 90 0f 
> > > > [   80.183098] RIP  [<ffffffff81c0721e>] _raw_spin_lock+0xe/0x30
> > > > [   80.185372]  RSP <ffff880221721cc8>
> > > > [   80.187584] CR2: 0000000000000000
> > > > [   80.208627] ---[ end trace 13fb1e4b6e5ab21f ]---
> > > > 

Re: irqfd issue

[Cornelia Huck]

On Tue, 7 May 2013 11:21:09 +0200
Cornelia Huck <cornelia.huck@de.ibm.com> wrote:

> On Tue, 7 May 2013 12:12:09 +0300
> Gleb Natapov <gleb@redhat.com> wrote:
> 
> > On Tue, May 07, 2013 at 05:10:41PM +0800, Asias He wrote:
> > > On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> > > > On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > > > > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > > > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > > > > Hi,
> > > > > > > > > 
> > > > > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > > > > 
> > > > > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > > > > 
> > > > > > > Almost always. Start a guest with vhost-net on.
> > > > > > > 
> > > > > > It happens during start or on VM exit? The trace shows do_exit()
> > > > > > 
> > > > > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > > > > 
> > > > > > Can you try with kvm.git next branch?
> > > > > 
> > > > > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > > > > 
> > > > Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?
> > > 
> > > The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
> > > reverted.
> > > 
> > Cornelia, any ideas?
> 
> irqfd_deactivate before kvm_init or after kvm_exit? Let me look...
> 

Hm, no idea.

Asias, could you try whether the patch below yields any interesting
backtraces?

diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 64ee720..59c3810 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -92,6 +92,8 @@ struct _irqfd {
 };
 
 static struct workqueue_struct *irqfd_cleanup_wq;
+static int irqfd_ready;
+static unsigned long irqfd_cnt;
 
 static void
 irqfd_inject(struct work_struct *work)
@@ -410,6 +412,8 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
 	 */
 	fput(file);
 
+	irqfd_cnt++;
+
 	return 0;
 
 fail:
@@ -483,12 +487,16 @@ kvm_irqfd_deassign(struct kvm *kvm, struct kvm_irqfd *args)
 	 */
 	flush_workqueue(irqfd_cleanup_wq);
 
+	irqfd_cnt--;
+
 	return 0;
 }
 
 int
 kvm_irqfd(struct kvm *kvm, struct kvm_irqfd *args)
 {
+	WARN_ON(!irqfd_ready);
+
 	if (args->flags & ~(KVM_IRQFD_FLAG_DEASSIGN | KVM_IRQFD_FLAG_RESAMPLE))
 		return -EINVAL;
 
@@ -548,15 +556,22 @@ void kvm_irq_routing_update(struct kvm *kvm,
  */
 int kvm_irqfd_init(void)
 {
+	WARN_ON(irqfd_cnt);
+
 	irqfd_cleanup_wq = create_singlethread_workqueue("kvm-irqfd-cleanup");
 	if (!irqfd_cleanup_wq)
 		return -ENOMEM;
 
+	irqfd_ready = 1;
+
 	return 0;
 }
 
 void kvm_irqfd_exit(void)
 {
+	WARN_ON(irqfd_cnt);
+	irqfd_ready = 0;
+
 	destroy_workqueue(irqfd_cleanup_wq);
 }
 #endif

Re: irqfd issue

[Gleb Natapov]

On Tue, May 07, 2013 at 01:07:06PM +0200, Cornelia Huck wrote:
> On Tue, 7 May 2013 11:21:09 +0200
> Cornelia Huck <cornelia.huck@de.ibm.com> wrote:
> 
> > On Tue, 7 May 2013 12:12:09 +0300
> > Gleb Natapov <gleb@redhat.com> wrote:
> > 
> > > On Tue, May 07, 2013 at 05:10:41PM +0800, Asias He wrote:
> > > > On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> > > > > On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > > > > > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > > > > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > > > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > > > > > Hi,
> > > > > > > > > > 
> > > > > > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > > > > > 
> > > > > > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > > > > > 
> > > > > > > > Almost always. Start a guest with vhost-net on.
> > > > > > > > 
> > > > > > > It happens during start or on VM exit? The trace shows do_exit()
> > > > > > > 
> > > > > > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > > > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > > > > > 
> > > > > > > Can you try with kvm.git next branch?
> > > > > > 
> > > > > > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > > > > > 
> > > > > Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?
> > > > 
> > > > The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
> > > > reverted.
> > > > 
> > > Cornelia, any ideas?
> > 
> > irqfd_deactivate before kvm_init or after kvm_exit? Let me look...
> > 
> 
> Hm, no idea.
> 
For am quick look is seems that after the patch irqfd_cleanup_wq is
initialized as part of kvm-intel/kvm-amd module, but before the patch is
was initialized as part of kvm module. The later is how it should be,
but I do not see why would it cause the problem.

> Asias, could you try whether the patch below yields any interesting
> backtraces?
> 
> diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
> index 64ee720..59c3810 100644
> --- a/virt/kvm/eventfd.c
> +++ b/virt/kvm/eventfd.c
> @@ -92,6 +92,8 @@ struct _irqfd {
>  };
>  
>  static struct workqueue_struct *irqfd_cleanup_wq;
> +static int irqfd_ready;
> +static unsigned long irqfd_cnt;
>  
>  static void
>  irqfd_inject(struct work_struct *work)
> @@ -410,6 +412,8 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
>  	 */
>  	fput(file);
>  
> +	irqfd_cnt++;
> +
>  	return 0;
>  
>  fail:
> @@ -483,12 +487,16 @@ kvm_irqfd_deassign(struct kvm *kvm, struct kvm_irqfd *args)
>  	 */
>  	flush_workqueue(irqfd_cleanup_wq);
>  
> +	irqfd_cnt--;
> +
>  	return 0;
>  }
>  
>  int
>  kvm_irqfd(struct kvm *kvm, struct kvm_irqfd *args)
>  {
> +	WARN_ON(!irqfd_ready);
> +
>  	if (args->flags & ~(KVM_IRQFD_FLAG_DEASSIGN | KVM_IRQFD_FLAG_RESAMPLE))
>  		return -EINVAL;
>  
> @@ -548,15 +556,22 @@ void kvm_irq_routing_update(struct kvm *kvm,
>   */
>  int kvm_irqfd_init(void)
>  {
> +	WARN_ON(irqfd_cnt);
> +
>  	irqfd_cleanup_wq = create_singlethread_workqueue("kvm-irqfd-cleanup");
>  	if (!irqfd_cleanup_wq)
>  		return -ENOMEM;
>  
> +	irqfd_ready = 1;
> +
>  	return 0;
>  }
>  
>  void kvm_irqfd_exit(void)
>  {
> +	WARN_ON(irqfd_cnt);
> +	irqfd_ready = 0;
> +
>  	destroy_workqueue(irqfd_cleanup_wq);
>  }
>  #endif

--
			Gleb.

Re: irqfd issue

[Asias He]

On Tue, May 07, 2013 at 05:29:09PM +0300, Gleb Natapov wrote:
> On Tue, May 07, 2013 at 01:07:06PM +0200, Cornelia Huck wrote:
> > On Tue, 7 May 2013 11:21:09 +0200
> > Cornelia Huck <cornelia.huck@de.ibm.com> wrote:
> > 
> > > On Tue, 7 May 2013 12:12:09 +0300
> > > Gleb Natapov <gleb@redhat.com> wrote:
> > > 
> > > > On Tue, May 07, 2013 at 05:10:41PM +0800, Asias He wrote:
> > > > > On Tue, May 07, 2013 at 11:42:07AM +0300, Gleb Natapov wrote:
> > > > > > On Tue, May 07, 2013 at 04:36:50PM +0800, Asias He wrote:
> > > > > > > On Tue, May 07, 2013 at 11:18:38AM +0300, Gleb Natapov wrote:
> > > > > > > > On Tue, May 07, 2013 at 04:14:50PM +0800, Asias He wrote:
> > > > > > > > > On Tue, May 07, 2013 at 10:55:36AM +0300, Gleb Natapov wrote:
> > > > > > > > > > On Tue, May 07, 2013 at 09:37:30AM +0800, Asias He wrote:
> > > > > > > > > > > Hi,
> > > > > > > > > > > 
> > > > > > > > > > > I am seeing this with linus/master. Any ideas?
> > > > > > > > > > > 
> > > > > > > > > > How reproducible it this? What HEAD are you seeing this with?
> > > > > > > > > 
> > > > > > > > > Almost always. Start a guest with vhost-net on.
> > > > > > > > > 
> > > > > > > > It happens during start or on VM exit? The trace shows do_exit()
> > > > > > > > 
> > > > > > > > > QEMU  : e3351000cd682200835763caca87adf708ed1c65
> > > > > > > > > KERNEL: 51a26ae7a14b85c99c9be470c2d28eeeba0f26a3 
> > > > > > > > > 
> > > > > > > > Can you try with kvm.git next branch?
> > > > > > > 
> > > > > > > With next branch db6ae6158186a17165ef990bda2895ae7594b039,
> > > > > > > 
> > > > > > Can you revert a0f155e9646d5f1c263f6f9aae880151100243bb and try again?
> > > > > 
> > > > > The issue is gone with a0f155e9646d5f1c263f6f9aae880151100243bb
> > > > > reverted.
> > > > > 
> > > > Cornelia, any ideas?
> > > 
> > > irqfd_deactivate before kvm_init or after kvm_exit? Let me look...
> > > 
> > 
> > Hm, no idea.
> > 
> For am quick look is seems that after the patch irqfd_cleanup_wq is
> initialized as part of kvm-intel/kvm-amd module, but before the patch is
> was initialized as part of kvm module. The later is how it should be,
> but I do not see why would it cause the problem.

I have a fix for this now. Will send out patch shortly.

> > Asias, could you try whether the patch below yields any interesting
> > backtraces?
> > 
> > diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
> > index 64ee720..59c3810 100644
> > --- a/virt/kvm/eventfd.c
> > +++ b/virt/kvm/eventfd.c
> > @@ -92,6 +92,8 @@ struct _irqfd {
> >  };
> >  
> >  static struct workqueue_struct *irqfd_cleanup_wq;
> > +static int irqfd_ready;
> > +static unsigned long irqfd_cnt;
> >  
> >  static void
> >  irqfd_inject(struct work_struct *work)
> > @@ -410,6 +412,8 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
> >  	 */
> >  	fput(file);
> >  
> > +	irqfd_cnt++;
> > +
> >  	return 0;
> >  
> >  fail:
> > @@ -483,12 +487,16 @@ kvm_irqfd_deassign(struct kvm *kvm, struct kvm_irqfd *args)
> >  	 */
> >  	flush_workqueue(irqfd_cleanup_wq);
> >  
> > +	irqfd_cnt--;
> > +
> >  	return 0;
> >  }
> >  
> >  int
> >  kvm_irqfd(struct kvm *kvm, struct kvm_irqfd *args)
> >  {
> > +	WARN_ON(!irqfd_ready);
> > +
> >  	if (args->flags & ~(KVM_IRQFD_FLAG_DEASSIGN | KVM_IRQFD_FLAG_RESAMPLE))
> >  		return -EINVAL;
> >  
> > @@ -548,15 +556,22 @@ void kvm_irq_routing_update(struct kvm *kvm,
> >   */
> >  int kvm_irqfd_init(void)
> >  {
> > +	WARN_ON(irqfd_cnt);
> > +
> >  	irqfd_cleanup_wq = create_singlethread_workqueue("kvm-irqfd-cleanup");
> >  	if (!irqfd_cleanup_wq)
> >  		return -ENOMEM;
> >  
> > +	irqfd_ready = 1;
> > +
> >  	return 0;
> >  }
> >  
> >  void kvm_irqfd_exit(void)
> >  {
> > +	WARN_ON(irqfd_cnt);
> > +	irqfd_ready = 0;
> > +
> >  	destroy_workqueue(irqfd_cleanup_wq);
> >  }
> >  #endif
> 
> --
> 			Gleb.

-- 
Asias