* pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
@ 2013-12-01 6:03 Elena Ufimtseva
2013-12-01 8:03 ` Dario Faggioli
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Elena Ufimtseva @ 2013-12-01 6:03 UTC (permalink / raw)
To: xen-devel@lists.xen.org
Hi
pv kernel crashes when starting guest with vcpus above 150 with latest
linux kernel. WIth increase of max_event_channel it boots (I tried
4096).
[ 0.387911] ------------[ cut here ]------------
[ 0.387918] kernel BUG at drivers/xen/events.c:909!
[ 0.387922] invalid opcode: 0000 [#1] SMP
[ 0.387928] Modules linked in:
[ 0.387934] CPU: 0 PID: 1 Comm: swapper/0 Tainted: G W 3.12.0+ #95
[ 0.387940] task: ffff880088ecb080 ti: ffff880088ecc000 task.ti:
ffff880088ecc000
[ 0.387945] RIP: e030:[<ffffffff8126b310>] [<ffffffff8126b310>]
bind_ipi_to_irqhandler+0x9c/0x15d
[ 0.387959] RSP: e02b:ffff880088ecdd18 EFLAGS: 00010286
[ 0.387963] RAX: ffffffffffffffe4 RBX: 000000000000040d RCX: 0000000000000007
[ 0.387968] RDX: 0000000000000001 RSI: 00000000deadbeef RDI: 00000000deadbeef
[ 0.387972] RBP: 0000000000000091 R08: ffff880089003d70 R09: ffff880088ecdd50
[ 0.387977] R10: 0000000000001f00 R11: 0000000000000001 R12: 0000000000000091
[ 0.387982] R13: 0000000000000091 R14: 0000000000010b20 R15: 0000000000000004
[ 0.387990] FS: 0000000000000000(0000) GS:ffff880089400000(0000)
knlGS:0000000000000000
[ 0.387996] CS: e033 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 0.388000] CR2: 0000000000000000 CR3: 000000000160c000 CR4: 0000000000042660
[ 0.388000] Stack:
[ 0.388000] 0000000000000091 0000000000000091 ffffffff8100bc33
0000000000000c20
[ 0.388000] ffff880168c88760 0000000000000000 ffff880088ecdd98
ffff880000000091
[ 0.388000] ffff880088ecddb4 ffff88008c468000 0000000000000091
0000000000000091
[ 0.388000] Call Trace:
[ 0.388000] [<ffffffff8100bc33>] ? xen_hvm_cpu_up+0x53/0x53
[ 0.388000] [<ffffffff8100bba0>] ? xen_smp_intr_init+0x1be/0x1fe
[ 0.388000] [<ffffffff8100c17c>] ? xen_cpu_up+0x308/0x35d
[ 0.388000] [<ffffffff81047532>] ? _cpu_up+0xc1/0x120
[ 0.388000] [<ffffffff8104765f>] ? cpu_up+0xce/0xe2
[ 0.388000] [<ffffffff816e79f5>] ? smp_init+0x56/0x70
[ 0.388000] [<ffffffff816c8e22>] ? kernel_init_freeable+0xc4/0x1eb
[ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70
[ 0.388000] [<ffffffff813b34d2>] ? kernel_init+0x6/0xd3
[ 0.388000] [<ffffffff813c860c>] ? ret_from_fork+0x7c/0xb0
[ 0.388000] [<ffffffff813b34cc>] ? rest_init+0x70/0x70
[ 0.388000] Code: 81 48 c7 c2 bb 4c 08 81 48 c7 c6 20 d6 6a 81 e8
ad 96 e1 ff 48 8d 74 24 38 bf 07 00 00 00 89 6c 24 38 e8 c9 ea ff ff
85 c0 74 02 <0f> 0b 44 8b 64 24 3c 89 df e8 80 ea ff ff 89 de 48 89 c7
ba 03
[ 0.388000] RIP [<ffffffff8126b310>] bind_ipi_to_irqhandler+0x9c/0x15d
[ 0.388000] RSP <ffff880088ecdd18>
[ 0.388005] ---[ end trace 8347cf354b5857c5 ]---
[ 0.388026] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x0000000b
[ 0.388026]
--
Elena
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva
@ 2013-12-01 8:03 ` Dario Faggioli
2013-12-02 16:41 ` Elena Ufimtseva
2013-12-02 12:38 ` Wei Liu
2013-12-02 17:06 ` David Vrabel
2 siblings, 1 reply; 6+ messages in thread
From: Dario Faggioli @ 2013-12-01 8:03 UTC (permalink / raw)
To: Elena Ufimtseva; +Cc: xen-devel@lists.xen.org
[-- Attachment #1.1: Type: text/plain, Size: 1116 bytes --]
On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote:
> Hi
>
> pv kernel crashes when starting guest with vcpus above 150 with latest
> linux kernel.
>
Wow... >150 vcpus for a single guest? What is it that you are up to
here?!? :-P :-P
> WIth increase of max_event_channel it boots (I tried
> 4096).
>
I don't remember the exact numbers. This:
http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements
says you should be able to reach 750 vcpus, but it's probably
_in_total_, not necessarily within a single guest.
Also, these talks should have some numbers (although the latter is about
XenServer), but again, it's mostly about total number, I think:
http://www.youtube.com/watch?v=uz52hxKbv5I
http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1
Dario
--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
[-- Attachment #2: Type: text/plain, Size: 126 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva
2013-12-01 8:03 ` Dario Faggioli
@ 2013-12-02 12:38 ` Wei Liu
2013-12-02 16:42 ` Elena Ufimtseva
2013-12-02 17:06 ` David Vrabel
2 siblings, 1 reply; 6+ messages in thread
From: Wei Liu @ 2013-12-02 12:38 UTC (permalink / raw)
To: Elena Ufimtseva; +Cc: wei.liu2, xen-devel@lists.xen.org
On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote:
> Hi
>
> pv kernel crashes when starting guest with vcpus above 150 with latest
> linux kernel. WIth increase of max_event_channel it boots (I tried
> 4096).
>
>
> [ 0.387911] ------------[ cut here ]------------
> [ 0.387918] kernel BUG at drivers/xen/events.c:909!
This is due to hypervisor failing to bind IPI for that guest. You're
probably running out of resources.
Try 'wc -l /proc/interrupts' 'grep ipi /proc/interrputs | wc -l' when
the guest is running to have an idea how many it needs. ;-)
Wei.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
2013-12-01 8:03 ` Dario Faggioli
@ 2013-12-02 16:41 ` Elena Ufimtseva
0 siblings, 0 replies; 6+ messages in thread
From: Elena Ufimtseva @ 2013-12-02 16:41 UTC (permalink / raw)
To: Dario Faggioli; +Cc: xen-devel@lists.xen.org
On Sun, Dec 1, 2013 at 3:03 AM, Dario Faggioli
<dario.faggioli@citrix.com> wrote:
> On dom, 2013-12-01 at 01:03 -0500, Elena Ufimtseva wrote:
>> Hi
>>
>> pv kernel crashes when starting guest with vcpus above 150 with latest
>> linux kernel.
>>
> Wow... >150 vcpus for a single guest? What is it that you are up to
> here?!? :-P :-P
Was experimenting with 150 nodes :P
>
>> WIth increase of max_event_channel it boots (I tried
>> 4096).
>>
> I don't remember the exact numbers. This:
> http://wiki.xenproject.org/wiki/Xen_4.3_Feature_List#Performance_and_Scalability_Enhancements
> says you should be able to reach 750 vcpus, but it's probably
> _in_total_, not necessarily within a single guest.
>
> Also, these talks should have some numbers (although the latter is about
> XenServer), but again, it's mostly about total number, I think:
> http://www.youtube.com/watch?v=uz52hxKbv5I
> http://www.slideshare.net/xen_com_mgr/jonathandavies-xenservervmdensity20131024-1
Yes, I wanted to see if it should behave this way.
>
> Dario
>
> --
> <<This happens because I choose it to happen!>> (Raistlin Majere)
> -----------------------------------------------------------------
> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
--
Elena
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
2013-12-02 12:38 ` Wei Liu
@ 2013-12-02 16:42 ` Elena Ufimtseva
0 siblings, 0 replies; 6+ messages in thread
From: Elena Ufimtseva @ 2013-12-02 16:42 UTC (permalink / raw)
To: Wei Liu; +Cc: xen-devel@lists.xen.org
On Mon, Dec 2, 2013 at 7:38 AM, Wei Liu <wei.liu2@citrix.com> wrote:
> On Sun, Dec 01, 2013 at 01:03:08AM -0500, Elena Ufimtseva wrote:
>> Hi
>>
>> pv kernel crashes when starting guest with vcpus above 150 with latest
>> linux kernel. WIth increase of max_event_channel it boots (I tried
>> 4096).
>>
>>
>> [ 0.387911] ------------[ cut here ]------------
>> [ 0.387918] kernel BUG at drivers/xen/events.c:909!
>
> This is due to hypervisor failing to bind IPI for that guest. You're
> probably running out of resources.
>
> Try 'wc -l /proc/interrupts' 'grep ipi /proc/interrputs | wc -l' when
> the guest is running to have an idea how many it needs. ;-)
>
Yes, I am sure it is )
Thank you, will check it out.
> Wei.
--
Elena
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel
2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva
2013-12-01 8:03 ` Dario Faggioli
2013-12-02 12:38 ` Wei Liu
@ 2013-12-02 17:06 ` David Vrabel
2 siblings, 0 replies; 6+ messages in thread
From: David Vrabel @ 2013-12-02 17:06 UTC (permalink / raw)
To: Elena Ufimtseva; +Cc: xen-devel@lists.xen.org
On 01/12/13 06:03, Elena Ufimtseva wrote:
> Hi
>
> pv kernel crashes when starting guest with vcpus above 150 with latest
> linux kernel. WIth increase of max_event_channel it boots (I tried
> 4096).
The default for max_event_channels is 1023 which should be enough for a
typical guest. 150 VCPUs is very much not a typical guest.
We'll need to be sure to include a release note on this for the 4.4 release.
David
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2013-12-02 17:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-12-01 6:03 pv kernel panic xen 4.4 unstable with large number of vcpus and default max_event_channel Elena Ufimtseva
2013-12-01 8:03 ` Dario Faggioli
2013-12-02 16:41 ` Elena Ufimtseva
2013-12-02 12:38 ` Wei Liu
2013-12-02 16:42 ` Elena Ufimtseva
2013-12-02 17:06 ` David Vrabel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).