Re: credit2 BUG_ON triggered

[George Dunlap <george.dunlap@citrix.com>]

John, 

Thanks for your help looking into this.  Just to clarify, you're running
squeeze in HVM mode (not PV mode)?

 -George

On Tue, 2011-04-19 at 21:58 +0100, John Weekes wrote:
> I am testing credit2 on a dual Xeon L5640 machine. I have an HVM Debian 
> Squeeze domU that reliably leads to a panic when it's run with the 
> credit2 scheduler, but not with credit.
> 
> The reproduction steps on this machine are simple:
> 
> 1. Fully boot up the machine.
> 2. Enter commands that cause dom0 to use 100% CPU. For instance:
> 
> screen -AmdS burn1 perl -e 'while(1) {}'
> screen -AmdS burn2 perl -e 'while(1) {}'
> screen -AmdS burn3 perl -e 'while(1) {}'
> screen -AmdS burn4 perl -e 'while(1) {}'
> 
> 3. Start up the prepared Squeeze domU (which is a stock install), with 
> "xm" ("xl" doesn't work with debug=y because of a spurious assert, but 
> has the same problem with debug=n):
> 
> cd /servers/customers
> xm create testvds4.cfg
> 
> The serial console then shows this:
> 
> (XEN) irq.c:324: Dom1 callback via changed to Direct Vector 0xe9
> (XEN) Xen BUG at sched_credit2.c:1606
> (XEN) ----[ Xen-4.1.1-rc1-pre  x86_64  debug=y  Not tainted ]----
> (XEN) CPU:    12
> (XEN) RIP:    e008:[<ffff82c48011a383>] csched_schedule+0xdb/0xab1
> (XEN) RFLAGS: 0000000000010082   CONTEXT: hypervisor
> (XEN) rax: ffff830c2246c000   rbx: ffff830c2246bd10   rcx: 0000000000000000
> (XEN) rdx: 0000000000000001   rsi: ffff82c480241680   rdi: ffff8300bf74c000
> (XEN) rbp: ffff83043b28fe38   rsp: ffff83043b28fd58   r8:  0000000000000002
> (XEN) r9:  000000000000003e   r10: 0000000000000018   r11: 00000000000186a0
> (XEN) r12: 0000000000000000   r13: ffff83043ffe02d0   r14: 000000000000000c
> (XEN) r15: ffff83043ffe0010   cr0: 000000008005003b   cr4: 00000000000026f0
> (XEN) cr3: 0000000c22436000   cr2: 0000000000000000
> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
> (XEN) Xen stack trace from rsp=ffff83043b28fd58:
> (XEN)    ffff82c4801bc822 ffff83043b298040 0000000000000282 ffff83043b28fd88
> (XEN)    ffff82c48012248f ffff8300bf74c000 ffff83043b28fdb8 ffff82c4801b59bd
> (XEN)    00000014c77c5137 ffff83043b28fe68 ffff82c480241680 0000000000000001
> (XEN)    00007cfbc4d70217 ffff82c48014b2c0 ffff83043b298060 ffff83043b28fde8
> (XEN)    ffff82c480124345 ffff83043b298060 ffff83043b28fe38 0000000000000082
> (XEN)    00000000000186a0 0000000000000082 000000000000000c ffff8300bf74c000
> (XEN)    0000000000000000 ffff82c480241680 ffff83043b298040 ffff83043b298060
> (XEN)    ffff83043b28feb8 ffff82c48012061c ffff83043b28feb8 00000014c77c5137
> (XEN)    0000000000000293 ffff8300bf74d868 ffff82c48012248f ffff8300bf74c000
> (XEN)    ffff83043b28fe98 ffff82c4801b19d0 ffff8300bf74c000 ffff82c4802a8e80
> (XEN)    00000000ffffffff ffff82c4802a8880 ffff83043b28ff18 ffffffffffffffff
> (XEN)    ffff83043b28fef8 ffff82c480121caf 0440080000000001 ffff8300bf74c000
> (XEN)    0000000000000046 ffff8800018501a0 ffffffff81311470 0000000000000092
> (XEN)    ffff83043b28ff08 ffff82c480121d0c ffff88000184c600 ffff82c4801bb3f1
> (XEN)    0000000000000092 ffffffff81311470 ffff8800018501a0 0000000000000046
> (XEN)    ffff88000184c600 0000000000000001 00000000000186a0 0000000000000008
> (XEN)    0000000000000200 0000000000000008 0000000000000000 0000000000000002
> (XEN)    0000000000000000 0000000000000002 0000000000000007 0000beef0000beef
> (XEN)    ffffffff81009308 0000beef0000beef 0000000000000046 ffff880031e63e58
> (XEN)    000000000000beef 000000000000beef 000000000000beef 000000000000beef
> (XEN) Xen call trace:
> (XEN)    [<ffff82c48011a383>] csched_schedule+0xdb/0xab1
> (XEN)    [<ffff82c48012061c>] schedule+0x122/0x60c
> (XEN)    [<ffff82c480121caf>] __do_softirq+0x8d/0x9e
> (XEN)    [<ffff82c480121d0c>] do_softirq+0x4c/0x4e
> (XEN)
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 12:
> (XEN) Xen BUG at sched_credit2.c:1606
> (XEN) ****************************************
> (XEN)
> (XEN) Reboot in five seconds...
> 
> Where do we go from here?
> 
> -John
> 
> For reference, xl info output:
> 
> dallas-dodec226-5 ~ # xl info
> host                   : dallas-dodec226-5
> release                : 2.6.32.37-gbe57219
> version                : #1 SMP Tue Apr 19 00:14:46 CDT 2011
> machine                : x86_64
> nr_cpus                : 24
> nr_nodes               : 2
> cores_per_socket       : 6
> threads_per_core       : 2
> cpu_mhz                : 2266
> hw_caps                : 
> bfebfbff:2c100800:00000000:00003f40:009ee3fd:00000000:00000001:00000000
> virt_caps              : hvm hvm_directio
> total_memory           : 49143
> free_memory            : 47106
> free_cpus              : 0
> xen_major              : 4
> xen_minor              : 1
> xen_extra              : .1-rc1-pre
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit2
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : Thu Apr 07 15:26:58 2011 +0100 23025:dbf2ddf652dc
> xen_commandline        : dom0_mem=1500M dom0_max_vcpus=4 
> iommu=dom0-passthrough sched=credit2 loglvl=all guest_loglvl=all 
> com2=115200,8n1 console=com2
> cc_compiler            : gcc version 4.4.5 (Gentoo 4.4.5 p1.2, pie-0.4.5)
> cc_compile_by          : root
> cc_compile_domain      : nuclearfallout.net
> cc_compile_date        : Tue Apr 19 14:26:02 CDT 2011
> xend_config_format     : 4
>