From: Andreas Olsowski <andreas.olsowski@leuphana.de>
To: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Cc: xen-devel@lists.xensource.com, giamteckchoon@gmail.com
Subject: Re: [SPAM] Re: kernel BUG at arch/x86/xen/mmu.c:1860! - ideas.
Date: Mon, 28 Mar 2011 13:37:56 +0200 [thread overview]
Message-ID: <4D907314.1040603@leuphana.de> (raw)
In-Reply-To: <20110316155220.GA15150@dumpdata.com>
[-- Attachment #1.1.1: Type: text/plain, Size: 1366 bytes --]
> - turn on CONFIG_DEBUG_PAGEALLOC
> - turn on CONFIG_DEBUG_LIST
> - turn on CONFIG_DEBUG_KMEMLEAK
> - turn on CONFIG_JBD_DEBUG, CONFIG_JBD2_DEBUG
> - turn on CONFIG_SLUB_DEBUG_ON
After i enabled those options (i dont use SLUB, i use SLAB) i do no
longer encounter any errors.
I completed 1000 loops of snapshot/mount/umoun/removesnapshot.
Without those options in 2.6.32.35 i hit a different bug earlier today:
But you really have to be patient to see some output, because lvremove
will hang quite a while:
(a "while" beeing the a a roughly the time it takes for: wait 5 min for
error, leave office, get coffee, smoke cigarette, goto restroom, return
to office, finally see error)
kernel: BUG: unable to handle kernel paging request
...
kernel: RIP [<ffffffff8100f2bf>] xen_set_pmd+0x2f/0xb0
syslog/dmesg output is attached as crash.2.6.32.35-xen_01 or available at:
http://pastebin.com/Ad8MhUzD
After that happened i did a kernel recompile without rebooting the
machine first and encoundeterd system_call_fastpath as last call once
more as shown in crash.2.6.32.35-xen_02 or http://pastebin.com/kB38W5mp
Maybe this helps, but i think, if anything, this makes it worse as the
debug options actually supressed the problem that needs to be debugged.
with best regards
--
Andreas "andiolsi "Olsowski
[-- Attachment #1.1.2: crash.2.6.32.35-xen_01 --]
[-- Type: text/plain, Size: 5292 bytes --]
Mar 28 08:07:59 netcatarina udevd[1019]: worker [5115] unexpectedly returned with status 0x0100
Mar 28 08:07:59 netcatarina udevd[1019]: worker [5115] failed while handling '/devices/virtual/block/dm-35'
Mar 28 08:08:02 netcatarina kernel: BUG: unable to handle kernel paging request at ffff8800c5d0d678
Mar 28 08:08:02 netcatarina kernel: IP: [<ffffffff8100f2bf>] xen_set_pmd+0x2f/0xb0
Mar 28 08:08:02 netcatarina kernel: PGD 1002067 PUD 6e8067 PMD 717067 PTE 100000c5d0d065
Mar 28 08:08:02 netcatarina kernel: Oops: 0003 [#1] SMP
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:Oops: 0003 [#1] SMP
Mar 28 08:08:02 netcatarina kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.1/host2/rport-2:0-2/target2:0:2/2:0:2:0/state
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:last sysfs file: /sys/devices/pci0000:00/0000:00:07.0/0000:04:00.1/host2/rport-2:0-2/target2:0:2/2:0:2:0/state
Mar 28 08:08:02 netcatarina kernel: CPU 2
Mar 28 08:08:02 netcatarina kernel: Modules linked in: iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi dm_round_robin dm_multipath qla2xxx
Mar 28 08:08:02 netcatarina kernel: Pid: 5182, comm: udevd Not tainted 2.6.32.35-xen #1 PowerEdge R610
Mar 28 08:08:02 netcatarina kernel: RIP: e030:[<ffffffff8100f2bf>] [<ffffffff8100f2bf>] xen_set_pmd+0x2f/0xb0
Mar 28 08:08:02 netcatarina kernel: RSP: e02b:ffff8800c280fd48 EFLAGS: 00010246
Mar 28 08:08:02 netcatarina kernel: RAX: 0000000000000000 RBX: ffff8800c5d0d678 RCX: ffff880000000558
Mar 28 08:08:02 netcatarina kernel: RDX: ffffea0000000000 RSI: 0000000000000000 RDI: ffff8800c5d0d678
Mar 28 08:08:02 netcatarina kernel: RBP: 0000000000000000 R08: 00007faadb57c000 R09: dead000000200200
Mar 28 08:08:02 netcatarina kernel: R10: dead000000100100 R11: 0000000000000000 R12: ffff8800c5d0d678
Mar 28 08:08:02 netcatarina kernel: R13: 00000000c2b01067 R14: ffff880028079fc0 R15: 00007faadb1fbfff
Mar 28 08:08:02 netcatarina kernel: FS: 00007faadafcd7a0(0000) GS:ffff88002806d000(0000) knlGS:0000000000000000
Mar 28 08:08:02 netcatarina kernel: CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 28 08:08:02 netcatarina kernel: CR2: ffff8800c5d0d678 CR3: 0000000001001000 CR4: 0000000000002660
Mar 28 08:08:02 netcatarina kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 28 08:08:02 netcatarina kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Mar 28 08:08:02 netcatarina kernel: Process udevd (pid: 5182, threadinfo ffff8800c280e000, task ffff8800c2b4e630)
Mar 28 08:08:02 netcatarina kernel: Stack:
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:Stack:
Mar 28 08:08:02 netcatarina kernel: 00007faad9e01000 00007faada000000 00007faadb1fc000 ffffffff8109e613
Mar 28 08:08:02 netcatarina kernel: <0> ffffffff81030dad ffffea0002a8c630 00007faad9e00000 ffff8800c285a558
Mar 28 08:08:02 netcatarina kernel: <0> 00007faadb1fc000 00007faad9e00000 00007faadb1fbfff ffff8800c52217f8
Mar 28 08:08:02 netcatarina kernel: Call Trace:
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:Call Trace:
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8109e613>] ? free_pgd_range+0x253/0x490
Mar 28 08:08:02 netcatarina kernel: [<ffffffff81030dad>] ? pvclock_clocksource_read+0x4d/0xb0
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8109e8e9>] ? free_pgtables+0x99/0xb0
Mar 28 08:08:02 netcatarina kernel: [<ffffffff810a3ca0>] ? exit_mmap+0xd0/0x170
Mar 28 08:08:02 netcatarina kernel: [<ffffffff81049902>] ? mmput+0x32/0x100
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8104de52>] ? exit_mm+0x102/0x140
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8150519c>] ? _spin_lock_irq+0xc/0x10
Mar 28 08:08:02 netcatarina kernel: [<ffffffff81078e31>] ? acct_collect+0x161/0x1b0
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8104fe2e>] ? do_exit+0x67e/0x750
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8103490b>] ? do_page_fault+0x10b/0x310
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8104ff50>] ? do_group_exit+0x50/0xc0
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8104ffd2>] ? sys_exit_group+0x12/0x20
Mar 28 08:08:02 netcatarina kernel: [<ffffffff8150587a>] ? error_exit+0x2a/0x60
Mar 28 08:08:02 netcatarina kernel: [<ffffffff81013f02>] ? system_call_fastpath+0x16/0x1b
Mar 28 08:08:02 netcatarina kernel: Code: 48 89 5c 24 08 48 89 6c 24 10 48 89 fb 48 89 f5 80 3d 01 af 7b 00 00 75 53 ff 05 45 ae 7b 00 48 89 df e8 55 d7 ff ff 84 c0 75 19 <48> 89 2b 48 8b 5c 24 08 48 8b 6c 24 10 48 83 c4 18 c3 0f 1f 80
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:Code: 48 89 5c 24 08 48 89 6c 24 10 48 89 fb 48 89 f5 80 3d 01 af 7b 00 00 75 53 ff 05 45 ae 7b 00 48 89 df e8 55 d7 ff ff 84 c0 75 19 <48> 89 2b 48 8b 5c 24 08 48 8b 6c 24 10 48 83 c4 18 c3 0f 1f 80
Mar 28 08:08:02 netcatarina kernel: RIP [<ffffffff8100f2bf>] xen_set_pmd+0x2f/0xb0
Mar 28 08:08:02 netcatarina kernel: RSP <ffff8800c280fd48>
Mar 28 08:08:02 netcatarina kernel: CR2: ffff8800c5d0d678
Message from syslogd@netcatarina at Mar 28 08:08:02 ...
kernel:CR2: ffff8800c5d0d678
Mar 28 08:08:02 netcatarina kernel: ---[ end trace cc5a29f516acc76b ]---
Mar 28 08:08:02 netcatarina kernel: Fixing recursive fault but reboot is needed!
[-- Attachment #1.1.3: crash.2.6.32.35-xen_02 --]
[-- Type: text/plain, Size: 4216 bytes --]
perl invoked oom-killer: gfp_mask=0xd0, order=1, oom_adj=0
Pid: 7146, comm: perl Tainted: G D 2.6.32.35-xen #1
Call Trace:
[<ffffffff810894cd>] ? T.437+0xcd/0x290
[<ffffffff810897a3>] ? __out_of_memory+0x113/0x180
[<ffffffff810898ea>] ? out_of_memory+0xda/0x160
[<ffffffff8108c790>] ? __alloc_pages_nodemask+0x680/0x690
[<ffffffff810b7d4b>] ? kmem_getpages+0x4b/0x140
[<ffffffff810b878b>] ? fallback_alloc+0x14b/0x1e0
[<ffffffff810b8e92>] ? kmem_cache_alloc+0xd2/0x110
[<ffffffff8104a328>] ? copy_process+0xb8/0x1170
[<ffffffff81030dad>] ? pvclock_clocksource_read+0x4d/0xb0
[<ffffffff8104b466>] ? do_fork+0x86/0x3b0
[<ffffffff81505235>] ? _spin_lock+0x5/0x10
[<ffffffff810c8d72>] ? do_pipe_flags+0xc2/0x110
[<ffffffff810141a3>] ? stub_clone+0x13/0x20
[<ffffffff81013f02>] ? system_call_fastpath+0x16/0x1b
Mem-Info:
Node 0 DMA per-cpu:
CPU 0: hi: 0, btch: 1 usd: 0
CPU 1: hi: 0, btch: 1 usd: 0
CPU 2: hi: 0, btch: 1 usd: 0
CPU 3: hi: 0, btch: 1 usd: 0
CPU 4: hi: 0, btch: 1 usd: 0
CPU 5: hi: 0, btch: 1 usd: 0
CPU 6: hi: 0, btch: 1 usd: 0
CPU 7: hi: 0, btch: 1 usd: 0
Node 0 DMA32 per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 0
CPU 2: hi: 186, btch: 31 usd: 0
CPU 3: hi: 186, btch: 31 usd: 0
CPU 4: hi: 186, btch: 31 usd: 0
CPU 5: hi: 186, btch: 31 usd: 0
CPU 6: hi: 186, btch: 31 usd: 0
CPU 7: hi: 186, btch: 31 usd: 0
Node 0 Normal per-cpu:
CPU 0: hi: 186, btch: 31 usd: 0
CPU 1: hi: 186, btch: 31 usd: 0
CPU 2: hi: 186, btch: 31 usd: 0
CPU 3: hi: 186, btch: 31 usd: 0
CPU 4: hi: 186, btch: 31 usd: 0
CPU 5: hi: 186, btch: 31 usd: 0
CPU 6: hi: 186, btch: 31 usd: 0
CPU 7: hi: 186, btch: 31 usd: 0
active_anon:269 inactive_anon:121 isolated_anon:96
active_file:0 inactive_file:8 isolated_file:33
unevictable:4282 dirty:0 writeback:73 unstable:0
free:79907 slab_reclaimable:32585 slab_unreclaimable:318961
mapped:990 shmem:6 pagetables:718 bounce:0
Node 0 DMA free:9416kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:9224kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 3243 39603 39603
Node 0 DMA32 free:300480kB min:2084kB low:2604kB high:3124kB active_anon:1076kB inactive_anon:484kB active_file:0kB inactive_file:32kB unevictable:17128kB isolated(anon):384kB isolated(file):132kB present:3321540kB mlocked:17128kB dirty:0kB writeback:292kB mapped:3960kB shmem:24kB slab_reclaimable:129184kB slab_unreclaimable:1275724kB kernel_stack:791040kB pagetables:2872kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:138 all_unreclaimable? no
lowmem_reserve[]: 0 0 36360 36360
Node 0 Normal free:9732kB min:23380kB low:29224kB high:35068kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:37232640kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:1156kB slab_unreclaimable:120kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 2*4kB 0*8kB 2*16kB 1*32kB 2*64kB 2*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 2*4096kB = 9416kB
Node 0 DMA32: 74880*4kB 141*8kB 0*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 301000kB
Node 0 Normal: 11*4kB 5*8kB 1*16kB 1*32kB 2*64kB 0*128kB 3*256kB 3*512kB 1*1024kB 3*2048kB 0*4096kB = 9732kB
1295 total pagecache pages
235 pages in swap cache
Swap cache stats: add 199389, delete 199154, find 38746/51504
Free swap = 3882788kB
Total swap = 3943416kB
10289152 pages RAM
9620405 pages reserved
9071 pages shared
458700 pages non-shared
Out of memory: kill process 2701 (dbus-daemon) score 5815 or a child
Killed process 2701 (dbus-daemon)
[-- Attachment #1.2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 6595 bytes --]
[-- Attachment #2: Type: text/plain, Size: 138 bytes --]
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
next prev parent reply other threads:[~2011-03-28 11:37 UTC|newest]
Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-26 8:16 kernel BUG at arch/x86/xen/mmu.c:1860! Teck Choon Giam
2010-12-27 15:53 ` Konrad Rzeszutek Wilk
2010-12-27 22:14 ` Teck Choon Giam
2010-12-28 10:42 ` Pasi Kärkkäinen
2010-12-28 18:01 ` Teck Choon Giam
2010-12-29 4:25 ` Teck Choon Giam
2010-12-29 4:58 ` Teck Choon Giam
2011-01-14 15:20 ` Konrad Rzeszutek Wilk
2011-01-14 19:25 ` Teck Choon Giam
2011-01-14 19:44 ` Konrad Rzeszutek Wilk
2011-01-14 20:09 ` Teck Choon Giam
2011-01-14 20:32 ` Teck Choon Giam
2011-01-24 1:42 ` Teck Choon Giam
2011-01-24 14:36 ` Konrad Rzeszutek Wilk
2011-01-24 15:56 ` Teck Choon Giam
2011-01-25 14:48 ` Konrad Rzeszutek Wilk
2011-01-26 14:31 ` Konrad Rzeszutek Wilk
2011-01-27 17:17 ` Teck Choon Giam
2011-01-27 20:32 ` Konrad Rzeszutek Wilk
2011-01-27 22:20 ` Teck Choon Giam
2011-02-26 12:03 ` Teck Choon Giam
2011-02-28 16:20 ` Konrad Rzeszutek Wilk
2011-03-01 9:59 ` Teck Choon Giam
2011-03-03 22:16 ` Konrad Rzeszutek Wilk
2011-03-04 5:30 ` Teck Choon Giam
2011-03-04 6:15 ` Fajar A. Nugraha
2011-03-04 6:33 ` Teck Choon Giam
2011-03-08 19:29 ` Konrad Rzeszutek Wilk
2011-03-08 20:10 ` Konrad Rzeszutek Wilk
2011-03-08 20:20 ` Teck Choon Giam
2011-03-08 20:45 ` Guido Hecken
2011-03-08 20:50 ` [SPAM] " tjaouen
2011-03-09 0:06 ` Andreas Olsowski
2011-03-11 18:38 ` benco
2011-03-11 19:59 ` Sander Eikelenboom
2011-03-11 20:29 ` Teck Choon Giam
2011-03-11 20:45 ` Teck Choon Giam
2011-03-11 21:02 ` Sander Eikelenboom
2011-03-11 21:15 ` Teck Choon Giam
2011-03-09 0:43 ` [SPAM] " Konrad Rzeszutek Wilk
2011-03-09 6:58 ` Andreas Olsowski
2011-03-09 15:00 ` Konrad Rzeszutek Wilk
2011-03-09 19:59 ` Andreas Olsowski
2011-03-10 7:20 ` Andreas Olsowski
2011-03-10 13:45 ` Andreas Olsowski
2011-03-11 18:05 ` Konrad Rzeszutek Wilk
2011-03-14 10:25 ` Ian Campbell
2011-03-14 10:36 ` Teck Choon Giam
2011-03-16 15:52 ` [SPAM] Re: kernel BUG at arch/x86/xen/mmu.c:1860! - ideas Konrad Rzeszutek Wilk
2011-03-16 16:26 ` Teck Choon Giam
2011-03-16 16:40 ` Konrad Rzeszutek Wilk
2011-03-24 11:57 ` Konrad Rzeszutek Wilk
2011-03-24 21:28 ` Teck Choon Giam
2011-03-25 3:57 ` Teck Choon Giam
2011-03-27 10:16 ` Teck Choon Giam
2011-03-28 11:37 ` Andreas Olsowski [this message]
2011-03-28 12:29 ` Teck Choon Giam
2011-04-05 22:01 ` Dave Hunter
2011-04-05 22:15 ` Teck Choon Giam
2011-04-05 23:20 ` Dave Hunter
2011-04-06 7:53 ` Ian Campbell
2011-04-06 21:52 ` Jeremy Fitzhardinge
2011-04-07 13:16 ` Teck Choon Giam
2011-03-09 0:41 ` kernel BUG at arch/x86/xen/mmu.c:1860! Konrad Rzeszutek Wilk
2011-01-04 15:10 ` Christophe Saout
2011-01-04 15:19 ` Christophe Saout
2011-01-04 15:37 ` benco
2011-01-04 18:40 ` Christophe Saout
2011-01-04 19:32 ` Teck Choon Giam
2011-01-04 19:56 ` benco
2011-01-14 15:22 ` Konrad Rzeszutek Wilk
2011-01-14 15:33 ` Christophe Saout
2011-01-04 23:10 ` Christophe Saout
2011-01-05 10:51 ` Pasi Kärkkäinen
2011-01-05 14:56 ` Teck Choon Giam
2011-01-14 15:24 ` Konrad Rzeszutek Wilk
2011-01-14 19:31 ` Teck Choon Giam
2011-01-04 13:48 ` Ian Campbell
2011-01-04 19:24 ` Teck Choon Giam
2011-01-05 15:30 ` Teck Choon Giam
2011-01-13 14:28 ` tjaouen
2011-01-14 14:47 ` Konrad Rzeszutek Wilk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4D907314.1040603@leuphana.de \
--to=andreas.olsowski@leuphana.de \
--cc=giamteckchoon@gmail.com \
--cc=konrad.wilk@oracle.com \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.