From: Dion Kant <g.w.kant@hunenet.nl>
To: xen-users@lists.xen.org, xen-devel@lists.xen.org
Subject: Re: [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8
Date: Thu, 04 Jul 2013 15:43:59 +0200 [thread overview]
Message-ID: <51D57C1F.8070909@hunenet.nl> (raw)
In-Reply-To: <20130517085923.GC14401@zion.uk.xensource.com>
Hello Wei and all other interested people,
I saw this thread from around May. It got silent on this thread after
your post on May 31.
Is there any progress on this problem?
I am running into this issue as well with the openSUSE 12.3
distribution. This is with their 3.7.10-1.16-xen kernel and Xen version
4.2.1_12-1.12.10. On the net I see some discussion of people hitting
this issue but not that much. E.g., one of the symptoms is that a guest
crashes when running zypper install or zypper update when the Internet
connection is fast enough.
OpenSUSE 3.4.X kernels are running ok as guest on top of the openSUSE
12.3 Xen distribution, but apparently since 3.7.10 and higher there is
this issue.
I spent already quite some time in getting grip on the issue. I added a
bug to bugzilla.novell.com but no response. See
https://bugzilla.novell.com/show_bug.cgi?id=826374 for details.
Apparently for hitting this bug (i.e. make it all the way to the crash),
it is required to use some hardware which performs not too slow. With
this I mean it is easy to find hardware which is unable to reproduce the
issue.
In one of my recent experiments I changed the SLAB allocater to SLUB
which provides more detailed kernel logging. Here is the log output
after the first detected issue regarding xennet:
2013-07-03T23:51:16.560229+02:00 domUA kernel: [ 97.562370] netfront:
Too many frags
2013-07-03T23:51:17.228143+02:00 domUA kernel: [ 98.230466] netfront:
Too many frags
2013-07-03T23:51:17.596074+02:00 domUA kernel: [ 98.597300] netfront:
Too many frags
2013-07-03T23:51:18.740215+02:00 domUA kernel: [ 99.743080]
net_ratelimit: 2 callbacks suppressed
2013-07-03T23:51:18.740242+02:00 domUA kernel: [ 99.743084] netfront:
Too many frags
2013-07-03T23:51:19.104100+02:00 domUA kernel: [ 100.104281] netfront:
Too many frags
2013-07-03T23:51:19.760134+02:00 domUA kernel: [ 100.760594] netfront:
Too many frags
2013-07-03T23:51:21.820154+02:00 domUA kernel: [ 102.821202] netfront:
Too many frags
2013-07-03T23:51:22.192188+02:00 domUA kernel: [ 103.192655] netfront:
Too many frags
2013-07-03T23:51:26.060144+02:00 domUA kernel: [ 107.062447] netfront:
Too many frags
2013-07-03T23:51:26.412116+02:00 domUA kernel: [ 107.415165] netfront:
Too many frags
2013-07-03T23:51:27.092147+02:00 domUA kernel: [ 108.094615] netfront:
Too many frags
2013-07-03T23:51:27.492112+02:00 domUA kernel: [ 108.494255] netfront:
Too many frags
2013-07-03T23:51:27.520194+02:00 domUA kernel: [ 108.522445]
=============================================================================
2013-07-03T23:51:27.520206+02:00 domUA kernel: [ 108.522448] BUG
kmalloc-1024 (Tainted: G W ): Redzone overwritten
2013-07-03T23:51:27.520209+02:00 domUA kernel: [ 108.522450]
-----------------------------------------------------------------------------
2013-07-03T23:51:27.520212+02:00 domUA kernel: [ 108.522450]
2013-07-03T23:51:27.520215+02:00 domUA kernel: [ 108.522452] Disabling
lock debugging due to kernel taint
2013-07-03T23:51:27.520217+02:00 domUA kernel: [ 108.522454] INFO:
0xffff8800f66068f8-0xffff8800f66068ff. First byte 0x0 instead of 0xcc
2013-07-03T23:51:27.520220+02:00 domUA kernel: [ 108.522461] INFO:
Allocated in __alloc_skb+0x88/0x260 age=11 cpu=0 pid=1325
2013-07-03T23:51:27.520223+02:00 domUA kernel: [ 108.522466]
set_track+0x6c/0x190
2013-07-03T23:51:27.520225+02:00 domUA kernel: [ 108.522470]
alloc_debug_processing+0x83/0x109
2013-07-03T23:51:27.520228+02:00 domUA kernel: [ 108.522472]
__slab_alloc.constprop.48+0x523/0x593
2013-07-03T23:51:27.520231+02:00 domUA kernel: [ 108.522474]
__kmalloc_track_caller+0xb4/0x200
2013-07-03T23:51:27.520233+02:00 domUA kernel: [ 108.522477]
__kmalloc_reserve+0x3c/0xa0
2013-07-03T23:51:27.520236+02:00 domUA kernel: [ 108.522478]
__alloc_skb+0x88/0x260
2013-07-03T23:51:27.520239+02:00 domUA kernel: [ 108.522483]
network_alloc_rx_buffers+0x76/0x5f0 [xennet]
2013-07-03T23:51:27.520241+02:00 domUA kernel: [ 108.522486]
netif_poll+0xcf4/0xf30 [xennet]
2013-07-03T23:51:27.520243+02:00 domUA kernel: [ 108.522489]
net_rx_action+0xf0/0x2e0
2013-07-03T23:51:27.520246+02:00 domUA kernel: [ 108.522493]
__do_softirq+0x127/0x280
2013-07-03T23:51:27.520248+02:00 domUA kernel: [ 108.522496]
call_softirq+0x1c/0x30
2013-07-03T23:51:27.520251+02:00 domUA kernel: [ 108.522499]
do_softirq+0x56/0xd0
2013-07-03T23:51:27.520253+02:00 domUA kernel: [ 108.522501]
irq_exit+0x52/0xd0
2013-07-03T23:51:27.520256+02:00 domUA kernel: [ 108.522503]
evtchn_do_upcall+0x281/0x2e7
2013-07-03T23:51:27.520258+02:00 domUA kernel: [ 108.522505]
do_hypervisor_callback+0x1e/0x30
2013-07-03T23:51:27.520261+02:00 domUA kernel: [ 108.522507]
0x7f45f0a2f1e0
2013-07-03T23:51:27.520263+02:00 domUA kernel: [ 108.522509] INFO:
Freed in skb_free_head+0x5c/0x70 age=14 cpu=0 pid=1325
2013-07-03T23:51:27.520266+02:00 domUA kernel: [ 108.522512]
set_track+0x6c/0x190
2013-07-03T23:51:27.520269+02:00 domUA kernel: [ 108.522513]
free_debug_processing+0x151/0x201
2013-07-03T23:51:27.520271+02:00 domUA kernel: [ 108.522515]
__slab_free+0x47/0x499
2013-07-03T23:51:27.520274+02:00 domUA kernel: [ 108.522517]
kfree+0x1df/0x230
2013-07-03T23:51:27.520276+02:00 domUA kernel: [ 108.522519]
skb_free_head+0x5c/0x70
2013-07-03T23:51:27.520279+02:00 domUA kernel: [ 108.522521]
skb_release_data+0xea/0xf0
2013-07-03T23:51:27.520281+02:00 domUA kernel: [ 108.522522]
__kfree_skb+0x1e/0xb0
2013-07-03T23:51:27.520284+02:00 domUA kernel: [ 108.522524]
kfree_skb+0x80/0xc0
2013-07-03T23:51:27.520286+02:00 domUA kernel: [ 108.522527]
netif_poll+0x824/0xf30 [xennet]
2013-07-03T23:51:27.520289+02:00 domUA kernel: [ 108.522529]
net_rx_action+0xf0/0x2e0
2013-07-03T23:51:27.520291+02:00 domUA kernel: [ 108.522530]
__do_softirq+0x127/0x280
2013-07-03T23:51:27.520294+02:00 domUA kernel: [ 108.522532]
call_softirq+0x1c/0x30
2013-07-03T23:51:27.520296+02:00 domUA kernel: [ 108.522534]
do_softirq+0x56/0xd0
2013-07-03T23:51:27.520299+02:00 domUA kernel: [ 108.522536]
irq_exit+0x52/0xd0
2013-07-03T23:51:27.520302+02:00 domUA kernel: [ 108.522538]
evtchn_do_upcall+0x281/0x2e7
2013-07-03T23:51:27.520304+02:00 domUA kernel: [ 108.522539]
do_hypervisor_callback+0x1e/0x30
2013-07-03T23:51:27.520307+02:00 domUA kernel: [ 108.522541] INFO: Slab
0xffff8800ffd78100 objects=12 used=7 fp=0xffff8800f66074d0
flags=0x400000000000408
2013-07-03T23:51:27.520310+02:00 domUA kernel: [ 108.522543] INFO:
Object 0xffff8800f66064f8 @offset=9464 fp=0x0000018800000000
2013-07-03T23:51:27.520312+02:00 domUA kernel: [ 108.522543]
2013-07-03T23:51:27.520315+02:00 domUA kernel: [ 108.522546] Bytes b4
ffff8800f66064e8: 4a 40 ff ff 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a
J@......ZZZZZZZZ
2013-07-03T23:51:27.520318+02:00 domUA kernel: [ 108.522548] Object
ffff8800f66064f8: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
2013-07-03T23:51:27.520320+02:00 domUA kernel: [ 108.522549] Object
ffff8800f6606508: 00 16 3e 29 7e 3c 00 25 90 69 ea 4e 08 00 45 08
..>)~<.%.i.N..E.
2013-07-03T23:51:27.520323+02:00 domUA kernel: [ 108.522551] Object
ffff8800f6606518: fe bc 46 d7 40 00 40 06 d3 69 0a 57 06 91 0a 57
..F.@.@..i.W...W
2013-07-03T23:51:27.520326+02:00 domUA kernel: [ 108.522553] Object
ffff8800f6606528: 06 b4 9b 86 00 16 57 4d 5e bd 89 4c 40 ad 80 10
......WM^..L@...
2013-07-03T23:51:27.520329+02:00 domUA kernel: [ 108.522554] Object
ffff8800f6606538: 00 a6 20 a2 00 00 01 01 08 0a 01 eb 40 a7 ff ff ..
.........@...
2013-07-03T23:51:27.520332+02:00 domUA kernel: [ 108.522556] Object
ffff8800f6606548: 44 fa 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
D.kkkkkkkkkkkkkk
2013-07-03T23:51:27.520335+02:00 domUA kernel: [ 108.522557] Object
ffff8800f6606558: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
2013-07-03T23:51:27.520337+02:00 domUA kernel: [ 108.522559] Object
ffff8800f6606568: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
kkkkkkkkkkkkkkkk
Skipping some of the object dumping.......
2013-07-03T23:51:27.520583+02:00 domUA kernel: [ 108.522644] Object
ffff8800f66068d8: 00 d7 e4 ff 00 88 ff ff 00 00 00 00 00 10 00 00
................
2013-07-03T23:51:27.520586+02:00 domUA kernel: [ 108.522646] Object
ffff8800f66068e8: 00 92 dd ff 00 88 ff ff 00 00 00 00 88 01 00 00
................
2013-07-03T23:51:27.520588+02:00 domUA kernel: [ 108.522647] Redzone
ffff8800f66068f8: 00 92 dd ff 00 88 ff ff ........
2013-07-03T23:51:27.520591+02:00 domUA kernel: [ 108.522649] Padding
ffff8800f6606a38: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
2013-07-03T23:51:27.520594+02:00 domUA kernel: [ 108.522651] Pid: 1325,
comm: sshd Tainted: G B W 3.7.10-1.16-dbg-xen #3
2013-07-03T23:51:27.520597+02:00 domUA kernel: [ 108.522652] Call Trace:
2013-07-03T23:51:27.520599+02:00 domUA kernel: [ 108.522658]
[<ffffffff8000b097>] try_stack_unwind+0x87/0x1c0
2013-07-03T23:51:27.520602+02:00 domUA kernel: [ 108.522662]
[<ffffffff80008fa5>] dump_trace+0xd5/0x250
2013-07-03T23:51:27.520605+02:00 domUA kernel: [ 108.522665]
[<ffffffff8000b22c>] show_trace_log_lvl+0x5c/0x80
2013-07-03T23:51:27.520608+02:00 domUA kernel: [ 108.522668]
[<ffffffff8000b265>] show_trace+0x15/0x20
2013-07-03T23:51:27.520610+02:00 domUA kernel: [ 108.522672]
[<ffffffff80553a69>] dump_stack+0x77/0x80
2013-07-03T23:51:27.520612+02:00 domUA kernel: [ 108.522676]
[<ffffffff801491b1>] print_trailer+0x131/0x140
2013-07-03T23:51:27.520615+02:00 domUA kernel: [ 108.522680]
[<ffffffff80149709>] check_bytes_and_report+0xc9/0x120
2013-07-03T23:51:27.520617+02:00 domUA kernel: [ 108.522683]
[<ffffffff8014a7f6>] check_object+0x56/0x240
2013-07-03T23:51:27.520620+02:00 domUA kernel: [ 108.522687]
[<ffffffff805575b6>] free_debug_processing+0xc4/0x201
2013-07-03T23:51:27.520622+02:00 domUA kernel: [ 108.522690]
[<ffffffff8055773a>] __slab_free+0x47/0x499
2013-07-03T23:51:27.520625+02:00 domUA kernel: [ 108.522694]
[<ffffffff8014beff>] kfree+0x1df/0x230
2013-07-03T23:51:27.520627+02:00 domUA kernel: [ 108.522697]
[<ffffffff8044a8cc>] skb_free_head+0x5c/0x70
2013-07-03T23:51:27.520630+02:00 domUA kernel: [ 108.522701]
[<ffffffff8044a9ca>] skb_release_data+0xea/0xf0
2013-07-03T23:51:27.520632+02:00 domUA kernel: [ 108.522704]
[<ffffffff8044a9ee>] __kfree_skb+0x1e/0xb0
2013-07-03T23:51:27.520635+02:00 domUA kernel: [ 108.522709]
[<ffffffff8049fa2a>] tcp_recvmsg+0x99a/0xd50
2013-07-03T23:51:27.520637+02:00 domUA kernel: [ 108.522714]
[<ffffffff804c796d>] inet_recvmsg+0xed/0x110
2013-07-03T23:51:27.520640+02:00 domUA kernel: [ 108.522718]
[<ffffffff80440be8>] sock_aio_read+0x158/0x190
2013-07-03T23:51:27.520642+02:00 domUA kernel: [ 108.522722]
[<ffffffff8015cb68>] do_sync_read+0x98/0xf0
2013-07-03T23:51:27.520645+02:00 domUA kernel: [ 108.522726]
[<ffffffff8015d32d>] vfs_read+0xbd/0x180
2013-07-03T23:51:27.520647+02:00 domUA kernel: [ 108.522729]
[<ffffffff8015d442>] sys_read+0x52/0xa0
2013-07-03T23:51:27.520650+02:00 domUA kernel: [ 108.522733]
[<ffffffff8056ab3b>] system_call_fastpath+0x1a/0x1f
2013-07-03T23:51:27.520652+02:00 domUA kernel: [ 108.522736]
[<00007f45ef74c960>] 0x7f45ef74c95f
2013-07-03T23:51:27.520655+02:00 domUA kernel: [ 108.522738] FIX
kmalloc-1024: Restoring 0xffff8800f66068f8-0xffff8800f66068ff=0xcc
2013-07-03T23:51:27.520657+02:00 domUA kernel: [ 108.522738]
2013-07-03T23:51:27.679444+02:00 domUA kernel: [ 108.671750]
=============================================================================
2013-07-03T23:51:27.679454+02:00 domUA kernel: [ 108.671753] BUG
kmalloc-1024 (Tainted: G B W ): Redzone overwritten
2013-07-03T23:51:27.679456+02:00 domUA kernel: [ 108.671754]
-----------------------------------------------------------------------------
2013-07-03T23:51:27.679458+02:00 domUA kernel: [ 108.671754]
2013-07-03T23:51:27.679460+02:00 domUA kernel: [ 108.671757] INFO:
0xffff8800f66068f8-0xffff8800f66068ff. First byte 0xcc instead of 0xbb
2013-07-03T23:51:27.679462+02:00 domUA kernel: [ 108.671762] INFO:
Allocated in __alloc_skb+0x88/0x260 age=48 cpu=0 pid=1325
2013-07-03T23:51:27.679464+02:00 domUA kernel: [ 108.671765]
set_track+0x6c/0x190
2013-07-03T23:51:27.679466+02:00 domUA kernel: [ 108.671767]
alloc_debug_processing+0x83/0x109
2013-07-03T23:51:27.679468+02:00 domUA kernel: [ 108.671769]
__slab_alloc.constprop.48+0x523/0x593
2013-07-03T23:51:27.679469+02:00 domUA kernel: [ 108.671771]
__kmalloc_track_caller+0xb4/0x200
2013-07-03T23:51:27.679471+02:00 domUA kernel: [ 108.671773]
__kmalloc_reserve+0x3c/0xa0
2013-07-03T23:51:27.679473+02:00 domUA kernel: [ 108.671775]
__alloc_skb+0x88/0x260
2013-07-03T23:51:27.679475+02:00 domUA kernel: [ 108.671778]
network_alloc_rx_buffers+0x76/0x5f0 [xennet]
2013-07-03T23:51:27.679476+02:00 domUA kernel: [ 108.671781]
netif_poll+0xcf4/0xf30 [xennet]
2013-07-03T23:51:27.679478+02:00 domUA kernel: [ 108.671783]
net_rx_action+0xf0/0x2e0
I noticed that after turning on all this debugging stuff, a real panic
does not appear any more.
This happens while copying a file with scp from dom0 to the guest (scp
bigfile domu:/dev/null).
In my lab, I am currently experimenting with a SuperMicro based system
with Xen showing the following characteristics:
__ __ _ _ ____ _ _ ____ _ _ ____ _ ___
\ \/ /___ _ __ | || | |___ \ / | / |___ \ / | / |___ \ / |/ _ \
\ // _ \ '_ \ | || |_ __) | | | | | __) |__| | | | __) | | | | | |
/ \ __/ | | | |__ _| / __/ _| | | |/ __/|__| |_| |/ __/ _| | |_| |
/_/\_\___|_| |_| |_|(_)_____(_)_|___|_|_____| |_(_)_|_____(_)_|\___/
|_____|
(XEN) Xen version 4.2.1_12-1.12.10 (abuild@) (gcc (SUSE Linux) 4.7.2
20130108 [gcc-4_7-branch revision 195012]) Wed May 29 20:31:49 UTC 2013
(XEN) Latest ChangeSet: 25952
(XEN) Bootloader: GNU GRUB 0.97
(XEN) Command line: dom0_mem=2048M,max:2048M loglvl=all guest_loglvl=all
(XEN) Video information:
(XEN) VGA is text mode 80x25, font 8x16
(XEN) VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN) Found 4 MBR signatures
(XEN) Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN) 0000000000000000 - 0000000000096400 (usable)
(XEN) 0000000000096400 - 00000000000a0000 (reserved)
(XEN) 00000000000e0000 - 0000000000100000 (reserved)
(XEN) 0000000000100000 - 00000000bf780000 (usable)
(XEN) 00000000bf78e000 - 00000000bf790000 type 9
(XEN) 00000000bf790000 - 00000000bf79e000 (ACPI data)
(XEN) 00000000bf79e000 - 00000000bf7d0000 (ACPI NVS)
(XEN) 00000000bf7d0000 - 00000000bf7e0000 (reserved)
(XEN) 00000000bf7ec000 - 00000000c0000000 (reserved)
(XEN) 00000000e0000000 - 00000000f0000000 (reserved)
(XEN) 00000000fee00000 - 00000000fee01000 (reserved)
(XEN) 00000000ffc00000 - 0000000100000000 (reserved)
(XEN) 0000000100000000 - 0000000340000000 (usable)
Skipping ACPI en SRAT
(XEN) System RAM: 12279MB (12573784kB)
(XEN) NUMA: Allocated memnodemap from 33e38a000 - 33e38e000
(XEN) NUMA: Using 8 for the hash shift.
(XEN) Domain heap initialised DMA width 30 bits
(XEN) found SMP MP-table at 000ff780
(XEN) DMI present.
(XEN) Enabling APIC mode: Phys. Using 2 I/O APICs
(XEN) ACPI: HPET id: 0x8086a301 base: 0xfed00000
(XEN) Failed to get Error Log Address Range.
(XEN) Using ACPI (MADT) for SMP configuration information
(XEN) SMP: Allowing 24 CPUs (8 hotplug CPUs)
(XEN) IRQ limits: 48 GSI, 3040 MSI/MSI-X
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 2400.115 MHz processor.
(XEN) Initing memory sharing.
(XEN) mce_intel.c:1238: MCA Capability: BCAST 1 SER 0 CMCI 1 firstbank 0
extended MCE MSR 0
(XEN) Intel machine check reporting enabled
(XEN) PCI: MCFG configuration 0: base e0000000 segment 0000 buses 00 - ff
(XEN) PCI: MCFG area at e0000000 reserved in E820
(XEN) PCI: Using MCFG for segment 0000 bus 00-ff
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=-1 pin2=-1
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 128 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 16 CPUs
(XEN) ACPI sleep modes: S3
(XEN) mcheck_poll: Machine check polling timer started.
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, lsb, paddr 0x2000 -> 0xa65000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN) Dom0 alloc.: 0000000336000000->0000000337000000 (516915 pages
to be allocated)
(XEN) Init. ramdisk: 000000033f333000->0000000340000000
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN) Loaded kernel: ffffffff80002000->ffffffff80a65000
(XEN) Init. ramdisk: 0000000000000000->0000000000000000
(XEN) Phys-Mach map: ffffea0000000000->ffffea0000400000
(XEN) Start info: ffffffff80a65000->ffffffff80a654b4
(XEN) Page tables: ffffffff80a66000->ffffffff80a6f000
(XEN) Boot stack: ffffffff80a6f000->ffffffff80a70000
(XEN) TOTAL: ffffffff80000000->ffffffff80c00000
(XEN) ENTRY ADDRESS: ffffffff80002000
(XEN) Dom0 has maximum 16 VCPUs
(XEN) Scrubbing Free RAM:
.....................................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: All
(XEN) Guest Loglevel: All
(XEN) Xen is relinquishing VGA console.
(XEN) ACPI: RSDP 000FACE0, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT BF790100, 008C (r1 SMCI 20110827 MSFT 97)
(XEN) ACPI: FACP BF790290, 00F4 (r4 082711 FACP1638 20110827 MSFT 97)
(XEN) ACPI: DSDT BF7906A0, 6563 (r2 10600 10600000 0 INTL 20051117)
(XEN) ACPI: FACS BF79E000, 0040
(XEN) ACPI: APIC BF790390, 011E (r2 082711 APIC1638 20110827 MSFT 97)
(XEN) ACPI: MCFG BF7904B0, 003C (r1 082711 OEMMCFG 20110827 MSFT 97)
(XEN) ACPI: SLIT BF7904F0, 0030 (r1 082711 OEMSLIT 20110827 MSFT 97)
(XEN) ACPI: OEMB BF79E040, 0085 (r1 082711 OEMB1638 20110827 MSFT 97)
(XEN) ACPI: SRAT BF79A6A0, 01D0 (r2 082711 OEMSRAT 1 INTL 1)
(XEN) ACPI: HPET BF79A870, 0038 (r1 082711 OEMHPET 20110827 MSFT 97)
(XEN) ACPI: DMAR BF79E0D0, 0130 (r1 AMI OEMDMAR 1 MSFT 97)
(XEN) ACPI: SSDT BF7A1B30, 0363 (r1 DpgPmm CpuPm 12 INTL 20051117)
(XEN) ACPI: EINJ BF79A8B0, 0130 (r1 AMIER AMI_EINJ 20110827 MSFT 97)
(XEN) ACPI: BERT BF79AA40, 0030 (r1 AMIER AMI_BERT 20110827 MSFT 97)
(XEN) ACPI: ERST BF79AA70, 01B0 (r1 AMIER AMI_ERST 20110827 MSFT 97)
(XEN) ACPI: HEST BF79AC20, 00A8 (r1 AMIER ABC_HEST 20110827 MSFT 97)
(XEN) System RAM: 12279MB (12573784kB)
(XEN) SRAT: PXM 0 -> APIC 0 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 2 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 18 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 20 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 1 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 3 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 19 -> Node 0
(XEN) SRAT: PXM 0 -> APIC 21 -> Node 0
I am happy to assist in more kernel probing. It is even possible for me
to setup access for someone to this machine.
Best regards,
Dion Kant
On 05/17/2013 10:59 AM, Wei Liu wrote:
> Moving discussion to Xen-devel
>
> On Thu, May 16, 2013 at 10:29:56PM +0300, Eugene Istomin wrote:
>> Hello,
>>
>> I tried to use 3.9.2 kernel with xen 4.2.2/4.3rc1 and in both variants leads
>> to this error in network-intensive load (such as iperf, 100 nginx parallel
>> requests to 1M files and so on):
>>
> It would be more helpful if you can provide info on your configurations
> (Dom0 and DomU), your workload, how to reproduce the bug.
>
> I run iperf and NFS to test Xen network, but never see any crash like
> this myself.
>
>> BUG: unable to handle kernel paging request at ffff8800795a3000
>> [ 60.246945] IP: [<ffffffffa001a75c>] netif_poll+0x49c/0xe80 [xennet]
>> [ 60.246975] PGD a8a067 PUD a9a067 PMD 7fc27067 PTE
>> 80100000795a3065
>> [ 60.247004] Oops: 0003 [#1] SMP
>> [ 60.247020] Modules linked in: af_packet hwmon domctl crc32_pclmul
>> crc32c_intel ghash_clmulni_intel aesni_intel ablk_helper cryptd lrw
>> aes_x86_64 joydev xts gf128mul autofs4 scsi_dh_emc scsi_dh_alua
>> scsi_dh_rdac scsi_dh_hp_sw scsi_dh xenblk cdrom xennet ata_generic
>> ata_piix
>> [ 60.247144] CPU 0
>> [ 60.247154] Pid: 0, comm: swapper/0 Not tainted 3.9.2-1.g04040b9-xen
>> #1
>> [ 60.247179] RIP: e030:[<ffffffffa001a75c>] [<ffffffffa001a75c>]
>> netif_poll+0x49c/0xe80 [xennet]
>> ...
> Could you provide fuul stack trace? AFAICT there is no netif_poll in Xen
> netfront/back.
>
> Presumably this is Dom0 log? (from the domctl module)
>
>> We have couple of production hypervisors on 3.4 kernels with high-
>> throughput internal network (VM-to-VM in one Dom0), iperf on them is
>> working well:
>> [ 3] 0.0- 2.0 sec 3357 MBytes 14080 Mbits/sec
>> [ 3] 2.0- 4.0 sec 2880 MBytes 12077 Mbits/sec
>> [ 3] 4.0- 6.0 sec 2909 MBytes 12202 Mbits/sec
>> [ 3] 6.0- 8.0 sec 2552 MBytes 10702 Mbits/sec
>> [ 3] 8.0-10.0 sec 3616 MBytes 15166 Mbits/sec
>> [ 3] 10.0-12.0 sec 3415 MBytes 14324 Mbits/sec
>>
>>
>> Seems like a kernel bug, is this related to one of this fixes in linux-next or i
>> need to create new bugreport?
>>
>> 1) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=1aaf6d3d3d1e95f4be07e32dd84aa1c93855fbbd
>> 2) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=9ecd1a75d977e2e8c48139c7d3efed183f898d94
>> 3) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=2810e5b9a7731ca5fce22bfbe12c96e16ac44b6f
>> 4) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=03393fd5cc2b6cdeec32b704ecba64dbb0feae3c
>> 5) https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git/commit/?id=59ccb4ebbc35e36a3c143f2d1355deb75c2e628f
>>
> I don't think these patches can fix your problem at first glance.
>
>
> Wei.
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xen.org
> http://lists.xen.org/xen-users
next prev parent reply other threads:[~2013-07-04 13:43 UTC|newest]
Thread overview: 85+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <8511913.uMAmUdIO30@eistomin.edss.local>
2013-05-17 8:59 ` [Xen-users] kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Wei Liu
2013-05-17 9:08 ` Jan Beulich
2013-05-17 9:37 ` Eugene Istomin
2013-05-17 9:40 ` Eugene Istomin
2013-05-17 10:45 ` Jan Beulich
2013-05-17 12:03 ` Eugene Istomin
2013-05-17 12:25 ` Jan Beulich
2013-05-17 12:30 ` Eugene Istomin
2013-05-17 12:36 ` Jan Beulich
2013-05-17 12:52 ` Eugene Istomin
2013-05-17 13:00 ` Eugene Istomin
2013-05-18 10:06 ` Eugene Istomin
2013-05-20 6:52 ` Eugene Istomin
2013-05-20 8:58 ` Ian Campbell
2013-05-20 9:06 ` Wei Liu
2013-05-20 9:17 ` Eugene Istomin
2013-05-20 9:24 ` Wei Liu
2013-05-21 9:55 ` Jan Beulich
2013-05-21 11:09 ` Eugene Istomin
2013-05-21 11:49 ` Jan Beulich
2013-05-21 16:19 ` Eugene Istomin
2013-05-31 3:37 ` Eugene Istomin
2013-05-31 6:19 ` Jan Beulich
2013-05-31 6:22 ` Eugene Istomin
2013-05-31 8:20 ` Wei Liu
2013-05-21 15:28 ` Wei Liu
2013-05-31 15:47 ` Wei Liu
2013-05-31 16:00 ` Jan Beulich
2013-05-31 16:05 ` Wei Liu
2013-07-04 13:43 ` Dion Kant [this message]
2013-07-04 14:18 ` Jan Beulich
2013-07-04 15:01 ` Wei Liu
2013-07-05 9:32 ` [PATCH] xen-netfront: pull on receive skb may need to happen earlier Jan Beulich
2013-07-05 14:53 ` Wei Liu
2013-07-05 14:53 ` Wei Liu
2013-07-07 1:10 ` David Miller
2013-07-07 1:10 ` David Miller
2013-07-08 9:59 ` Jan Beulich
2013-07-08 9:59 ` Jan Beulich
2013-07-08 12:16 ` Dion Kant
2013-07-08 12:41 ` Jan Beulich
2013-07-08 12:41 ` Jan Beulich
2013-07-08 14:20 ` Jan Beulich
2013-07-08 14:20 ` [Xen-devel] " Jan Beulich
2013-07-08 15:22 ` Eric Dumazet
2013-07-08 15:22 ` [Xen-devel] " Eric Dumazet
2013-07-09 7:47 ` Jan Beulich
2013-07-09 7:47 ` [Xen-devel] " Jan Beulich
2013-07-08 15:48 ` Wei Liu
2013-07-09 6:52 ` Jan Beulich
2013-07-09 16:51 ` Wei Liu
2013-07-09 16:51 ` [Xen-devel] " Wei Liu
2013-07-10 6:58 ` Jan Beulich
2013-07-10 6:58 ` [Xen-devel] " Jan Beulich
2013-07-10 10:04 ` Wei Liu
2013-07-10 10:46 ` Jan Beulich
2013-07-10 12:50 ` Ian Campbell
2013-07-10 12:50 ` [Xen-devel] " Ian Campbell
2013-07-10 12:53 ` Wei Liu
2013-07-10 12:53 ` [Xen-devel] " Wei Liu
2013-07-10 13:58 ` Jan Beulich
2013-07-10 13:58 ` Jan Beulich
2013-07-10 10:46 ` Jan Beulich
2013-07-10 10:04 ` Wei Liu
2013-07-10 13:19 ` [Xen-devel] " Eric Dumazet
2013-07-10 13:19 ` Eric Dumazet
2013-07-09 6:52 ` Jan Beulich
2013-07-08 15:48 ` Wei Liu
2013-07-12 8:32 ` Wei Liu
2013-07-12 8:32 ` [Xen-devel] " Wei Liu
2013-07-12 8:56 ` Jan Beulich
2013-07-13 11:26 ` Dion Kant
2013-07-13 11:26 ` [Xen-devel] " Dion Kant
2013-07-13 20:17 ` Dion Kant
2013-07-15 6:37 ` Jan Beulich
2013-07-12 8:56 ` Jan Beulich
2013-07-05 9:32 ` Jan Beulich
2013-07-05 9:36 ` kernel 3.9.2 - xen 4.2.2/4.3rc1 => BUG unable to handle kernel paging request netif_poll+0x49c/0xe8 Jan Beulich
2013-07-05 10:40 ` [Xen-users] " Dion Kant
2013-07-05 10:54 ` Wei Liu
2013-07-05 19:11 ` Dion Kant
2013-07-05 10:56 ` Jan Beulich
2013-07-05 19:46 ` Dion Kant
2013-07-06 13:36 ` Wei Liu
2013-07-08 8:25 ` Jan Beulich
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=51D57C1F.8070909@hunenet.nl \
--to=g.w.kant@hunenet.nl \
--cc=xen-devel@lists.xen.org \
--cc=xen-users@lists.xen.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.