* pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
@ 2013-12-01 6:03 Elena Ufimtseva
2013-12-01 8:03 ` Dario Faggioli
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Elena Ufimtseva @ 2013-12-01 6:03 UTC (permalink / raw)
To: xen-devel@lists.xen.org
Hi
pv kernel crashes when starting guest with vcpus above 150 with latest
linux kernel. WIth increase of max_event_channel it boots (I tried
4096).
[ 0.387911] ------------[ cut here ]------------
[ 0.387918] kernel BUG at drivers/xen/events.c:909!
[ 0.387922] invalid opcode: 0000 [#1] SMP
[ 0.387928] Modules linked in:
[ 0.387934] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.12.0+ #95
[ 0.387940] task: ffff880088ecb080 ti: ffff880088ecc000 task.ti:
ffff880088ecc000
[ 0.387945] RIP: e030:[<ffffffff8126b310>] [<ffffffff8126b310>]
bind_ipi_to_irqhandler+0x9c/0x15d
[ 0.387959] RSP: e02b:ffff880088ecdd18 EFLAGS: 00010286
[ 0.387963] RAX: ffffffffffffffe4 RBX: 000000000000040d RCX: 0000000000000007
[ 0.387968] RDX: 0000000000000001 RSI: 00000000deadbeef RDI: 00000000deadbeef
[ 0.387972] RBP: 0000000000000091 R08: ffff880089003d70 R09: ffff880088ecdd50
[ 0.387977] R10: 0000000000001f00 R11: 0000000000000001 R12: 0000000000000091
[ 0.387982] R13: 0000000000000091 R14: 0000000000010b20 R15: 0000000000000004
[ 0.387990] FS: 0000000000000000(0000) GS:ffff880089400000(0000)
knlGS:0000000000000000
[ 0.387996] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.388000] CR2: 0000000000000000 CR3: 000000000160c000 CR4: 0000000000042660
[ 0.388000] Stack:
[ 0.388000] 0000000000000091 0000000000000091 ffffffff8100bc33
0000000000000c20
[ 0.388000] ffff880168c88760 0000000000000000 ffff880088ecdd98
ffff880000000091
[ 0.388000] ffff880088ecddb4 ffff88008c468000 0000000000000091
0000000000000091
[ 0.388000] Call Trace:
[ 0.388000] [<ffffffff8100bc33>] ? xen_hvm_cpu_up+0x53/0x53
[ 0.388000] [<ffffffff8100bba0>] ? xen_smp_intr_init+0x1be/0x1fe
[ 0.388000] [<ffffffff8100c17c>] ? xen_cpu_up+0x308/0x35d
[ 0.388000] [<ffffffff81047532>] ? _cpu_up+0xc1/0x120
[ 0.388000] [<ffffffff8104765f>] ? cpu_up+0xce/0xe2
[ 0.388000] [<ffffffff816e79f5>] ? smp_init+0x56/0x70
[ 0.388000] [<ffffffff816c8e22>] ? kernel_init_freeable+0xc4/0x1eb
[ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70
[ 0.388000] [<ffffffff813b34d2>] ? kernel_init+0x6/0xd3
[ 0.388000] [<ffffffff813c860c>] ? ret_from_fork+0x7c/0xb0
[ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70
[ 0.388000] Code: 81 48 c7 c2 bb 4c 08 81 48 c7 c6 20 d6 6a 81 e8
ad 96 e1 ff 48 8d 74 24 38 bf 07 00 00 00 89 6c 24 38 e8 c9 ea ff ff
85 c0 74 02 <0f> 0b 44 8b 64 24 3c 89 df e8 80 ea ff ff 89 de 48 89 c7
ba 03
[ 0.388000] RIP [<ffffffff8126b310>] bind_ipi_to_irqhandler+0x9c/0x15d
[ 0.388000] RSP <ffff880088ecdd18>
[ 0.388005] ---[ end trace 8347cf354b5857c5 ]---
[ 0.388026] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
[ 0.388026]
--
Elena
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel 2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva @ 2013-12-01 8:03 ` Dario Faggioli 2013-12-02 16:41 ` Elena Ufimtseva 2013-12-02 12:38 ` Wei Liu 2013-12-02 17:06 ` David Vrabel 2 siblings, 1 reply; 6+ messages in thread From: Dario Faggioli @ 2013-12-01 8:03 UTC (permalink / raw) To: Elena Ufimtseva; +Cc: xen-devel@lists.xen.org [-- Attachment #1.1: Type: text/plain, Size: 1116 bytes --] On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote: > Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. > Wow... >150 vcpus for a single guest? What is it that you are up to here?!? :-P :-P > WIth increase of max_event_channel it boots (I tried > 4096). > I don't remember the exact numbers. This: http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements says you should be able to reach 750 vcpus, but it's probably _in_total_, not necessarily within a single guest. Also, these talks should have some numbers (although the latter is about XenServer), but again, it's mostly about total number, I think: http://www.youtube.com/watch?v=uz52hxKbv5I http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1 Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) [-- Attachment #1.2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 198 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel 2013-12-01 8:03 ` Dario Faggioli @ 2013-12-02 16:41 ` Elena Ufimtseva 0 siblings, 0 replies; 6+ messages in thread From: Elena Ufimtseva @ 2013-12-02 16:41 UTC (permalink / raw) To: Dario Faggioli; +Cc: xen-devel@lists.xen.org On Sun, Dec 1, 2013 at 3:03 AM, Dario Faggioli <dario.faggioli@citrix.com> wrote: > On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote: >> Hi >> >> pv kernel crashes when starting guest with vcpus above 150 with latest >> linux kernel. >> > Wow... >150 vcpus for a single guest? What is it that you are up to > here?!? :-P :-P Was experimenting with 150 nodes :P > >> WIth increase of max_event_channel it boots (I tried >> 4096). >> > I don't remember the exact numbers. This: > http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements > says you should be able to reach 750 vcpus, but it's probably > _in_total_, not necessarily within a single guest. > > Also, these talks should have some numbers (although the latter is about > XenServer), but again, it's mostly about total number, I think: > http://www.youtube.com/watch?v=uz52hxKbv5I > http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1 Yes, I wanted to see if it should behave this way. > > Dario > > -- > <<This happens because I choose it to happen!>> (Raistlin Majere) > ----------------------------------------------------------------- > Dario Faggioli, Ph.D, http://about.me/dario.faggioli > Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK) > -- Elena ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel 2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva 2013-12-01 8:03 ` Dario Faggioli @ 2013-12-02 12:38 ` Wei Liu 2013-12-02 16:42 ` Elena Ufimtseva 2013-12-02 17:06 ` David Vrabel 2 siblings, 1 reply; 6+ messages in thread From: Wei Liu @ 2013-12-02 12:38 UTC (permalink / raw) To: Elena Ufimtseva; +Cc: wei.liu2, xen-devel@lists.xen.org On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote: > Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. WIth increase of max_event_channel it boots (I tried > 4096). > > > [ 0.387911] ------------[ cut here ]------------ > [ 0.387918] kernel BUG at drivers/xen/events.c:909! This is due to hypervisor failing to bind IPI for that guest. You're probably running out of resources. Try 'wc -l /proc/interrupts' 'grep ipi /proc/interrputs | wc -l' when the guest is running to have an idea how many it needs. ;-) Wei. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel 2013-12-02 12:38 ` Wei Liu @ 2013-12-02 16:42 ` Elena Ufimtseva 0 siblings, 0 replies; 6+ messages in thread From: Elena Ufimtseva @ 2013-12-02 16:42 UTC (permalink / raw) To: Wei Liu; +Cc: xen-devel@lists.xen.org On Mon, Dec 2, 2013 at 7:38 AM, Wei Liu <wei.liu2@citrix.com> wrote: > On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote: >> Hi >> >> pv kernel crashes when starting guest with vcpus above 150 with latest >> linux kernel. WIth increase of max_event_channel it boots (I tried >> 4096). >> >> >> [ 0.387911] ------------[ cut here ]------------ >> [ 0.387918] kernel BUG at drivers/xen/events.c:909! > > This is due to hypervisor failing to bind IPI for that guest. You're > probably running out of resources. > > Try 'wc -l /proc/interrupts' 'grep ipi /proc/interrputs | wc -l' when > the guest is running to have an idea how many it needs. ;-) > Yes, I am sure it is ) Thank you, will check it out. > Wei. -- Elena ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel 2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva 2013-12-01 8:03 ` Dario Faggioli 2013-12-02 12:38 ` Wei Liu @ 2013-12-02 17:06 ` David Vrabel 2 siblings, 0 replies; 6+ messages in thread From: David Vrabel @ 2013-12-02 17:06 UTC (permalink / raw) To: Elena Ufimtseva; +Cc: xen-devel@lists.xen.org On 01/12/13 06:03, Elena Ufimtseva wrote: > Hi > > pv kernel crashes when starting guest with vcpus above 150 with latest > linux kernel. WIth increase of max_event_channel it boots (I tried > 4096). The default for max_event_channels is 1023 which should be enough for a typical guest. 150 VCPUs is very much not a typical guest. We'll need to be sure to include a release note on this for the 4.4 release. David ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-12-02 17:06 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva 2013-12-01 8:03 ` Dario Faggioli 2013-12-02 16:41 ` Elena Ufimtseva 2013-12-02 12:38 ` Wei Liu 2013-12-02 16:42 ` Elena Ufimtseva 2013-12-02 17:06 ` David Vrabel
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).