From: Dave Hunter <dave@ivt.com.au>
To: xen-devel@lists.xensource.com
Subject: Re: [SPAM] Re: kernel BUG at arch/x86/xen/mmu.c:1860! - ideas.
Date: Wed, 06 Apr 2011 08:01:55 +1000 [thread overview]
Message-ID: <1302040916.2042.108.camel@dave> (raw)
In-Reply-To: <AANLkTi=WtS4hgOjg63H1TU+89i03_cPnJzNgbTfgxqLo@mail.gmail.com>
Hi guys,
This thread has gone quiet for a while and I was wondering if a solution
had been found?
I'm currently running the packaged version of Xen 4.0.1 in Debian
Squeeze and everything runs well, except for the random crashing when
using LVM.
I use LVM for the disk partitions, and use live snapshots as part of our
backup routine. That is, create snapshot -> mount snapshot -> rsync ->
umount snapshot -> remove snapshot.
Cheers,
Dave Hunter.
On Mon, 2011-03-28 at 20:29 +0800, Teck Choon Giam wrote:
> On Mon, Mar 28, 2011 at 7:37 PM, Andreas Olsowski
> <andreas.olsowski@leuphana.de> wrote:
> >
> >> - turn on CONFIG_DEBUG_PAGEALLOC
> >> - turn on CONFIG_DEBUG_LIST
> >> - turn on CONFIG_DEBUG_KMEMLEAK
> >> - turn on CONFIG_JBD_DEBUG, CONFIG_JBD2_DEBUG
> >> - turn on CONFIG_SLUB_DEBUG_ON
> >
> > After i enabled those options (i dont use SLUB, i use SLAB) i do no longer
> > encounter any errors.
> >
> > I completed 1000 loops of snapshot/mount/umoun/removesnapshot.
>
> Did you try with just CONFIG_DEBUG_PAGEALLOC=y and leave the rest
> unchange of your config? My testing all narrow down to
> CONFIG_DEBUG_PAGEALLOC=y to prevent this BUG.
>
> >
> >
> > Without those options in 2.6.32.35 i hit a different bug earlier today:
> >
> > But you really have to be patient to see some output, because lvremove will
> > hang quite a while:
> > (a "while" beeing the a a roughly the time it takes for: wait 5 min for
> > error, leave office, get coffee, smoke cigarette, goto restroom, return to
> > office, finally see error)
> >
> > kernel: BUG: unable to handle kernel paging request
> > ...
> > kernel: RIP [<ffffffff8100f2bf>] xen_set_pmd+0x2f/0xb0
> > syslog/dmesg output is attached as crash.2.6.32.35-xen_01 or available at:
> > http://pastebin.com/Ad8MhUzD
>
> I hit this before:
>
> # grep 'xen_set_pmd' /var/log/messages*
> /var/log/messages:Mar 27 09:31:14 xen05 kernel: IP:
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 09:31:14 xen05 kernel: RIP:
> e030:[<ffffffff8100e2d4>] [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 09:31:14 xen05 kernel: RIP
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 09:06:10 xen05 kernel: IP:
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 09:06:10 xen05 kernel: RIP:
> e030:[<ffffffff8100e2d4>] [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 09:06:10 xen05 kernel: RIP
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 15:18:57 xen05 kernel: IP:
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 15:18:57 xen05 kernel: RIP:
> e030:[<ffffffff8100e2d4>] [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages:Mar 27 15:18:57 xen05 kernel: RIP
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages.1:Mar 23 11:00:16 xen05 kernel: IP:
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages.1:Mar 23 11:00:16 xen05 kernel: RIP:
> e030:[<ffffffff8100e2d4>] [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
> /var/log/messages.1:Mar 23 11:00:17 xen05 kernel: RIP
> [<ffffffff8100e2d4>] xen_set_pmd+0x16/0x2b
>
> But unable to reproduce when CONFIG_DEBUG_PAGEALLOC=y.
>
> >
> > After that happened i did a kernel recompile without rebooting the machine
> > first and encoundeterd system_call_fastpath as last call once more as shown
> > in crash.2.6.32.35-xen_02 or http://pastebin.com/kB38W5mp
>
> I hit this at least once but unable to when CONFIG_DEBUG_PAGEALLOC=y:
>
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: ------------[ cut here
> ]------------
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: kernel BUG at
> arch/x86/xen/mmu.c:1872!
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: invalid opcode: 0000 [#1] SMP
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: last sysfs file:
> /sys/block/sdd/dev
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: CPU 2
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Modules linked in:
> ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4
> xt_state nf_conntrack ipt_REJECT xt_tcpudp xt_physdev iptable_filter
> ip_tables x_tables bridge stp be2iscsi iscsi_tcp bnx2i cnic uio ipv6
> cxgb3i cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> dm_multipath scsi_dh video backlight output sbs sbshc power_meter
> hwmon battery acpi_memhotplug xen_acpi_memhotplug ac parport_pc lp
> parport tg3 libphy sg ide_cd_mod cdrom serio_raw button tpm_tis tpm
> tpm_bios i2c_i801 i2c_core shpchp iTCO_wdt pcspkr dm_snapshot dm_zero
> dm_mirror dm_region_hash dm_log dm_mod ata_piix libata sd_mod scsi_mod
> raid1 ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: microcode]
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Pid: 5874, comm:
> lvcreate Not tainted 2.6.32.35-4.xen.pvops.choon.centos5 #1 PowerEdge
> 860
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RIP:
> e030:[<ffffffff8100cb5b>] [<ffffffff8100cb5b>]
> pin_pagetable_pfn+0x53/0x59
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RSP:
> e02b:ffff8800303d1c28 EFLAGS: 00010282
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RAX: 00000000ffffffea
> RBX: 000000000003032d RCX: 0000000000000181
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RDX: 00000000deadbeef
> RSI: 00000000deadbeef RDI: 00000000deadbeef
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RBP: ffff8800303d1c48
> R08: 0000000000000968 R09: ffff880000000000
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: R10: 00000000deadbeef
> R11: ffff8800303d1d08 R12: 0000000000000003
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: R13: 000000000003032d
> R14: ffff880030360000 R15: 00007fd324a00000
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: FS:
> 00007fd327d2e710(0000) GS:ffff880028089000(0000)
> knlGS:0000000000000000
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: CS: e033 DS: 0000 ES:
> 0000 CR0: 000000008005003b
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: CR2: 00000000004612f0
> CR3: 000000003a025000 CR4: 0000000000002660
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: DR0: 0000000000000000
> DR1: 0000000000000000 DR2: 0000000000000000
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: DR3: 0000000000000000
> DR6: 00000000ffff0ff0 DR7: 0000000000000400
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Process lvcreate (pid:
> 5874, threadinfo ffff8800303d0000, task ffff880030360000)
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Stack:
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: 0000000000000000
> 00000000002027a9 000000013eb43318 000000000003032d
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: <0> ffff8800303d1c68
> ffffffff8100e07c ffff880032be05c0 ffff880032aa9928
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: <0> ffff8800303d1c78
> ffffffff8100e0af ffff8800303d1cb8 ffffffff810a4433
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Call Trace:
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff8100e07c>]
> xen_alloc_ptpage+0x64/0x69
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff8100e0af>]
> xen_alloc_pte+0xe/0x10
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a4433>]
> __pte_alloc+0x70/0xce
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a45d1>]
> handle_mm_fault+0x140/0x8b9
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a50c9>]
> __get_user_pages+0x37f/0x479
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a76ca>]
> __mlock_vma_pages_range+0xc0/0x16f
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff8131c03f>]
> ? _spin_unlock_irqrestore+0x11/0x13
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a78db>]
> mlock_fixup+0x162/0x199
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a7989>]
> do_mlockall+0x77/0x8d
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff81139016>]
> ? security_capable+0x27/0x29
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: [<ffffffff810a7ce2>]
> sys_mlockall+0x8f/0xb9
> /var/log/messages:Mar 27 17:04:39 xen05 kernel: [<ffffffff81012ac2>]
> system_call_fastpath+0x16/0x1b
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: Code: 48 b8 ff ff ff
> ff ff ff ff 7f 48 21 c2 48 89 55 e8 48 8d 7d e0 be 01 00 00 00 31 d2
> 41 ba f0 7f 00 00 e8 e9 c7 ff ff 85 c0 74 04 <0f> 0b eb fe c9 c3 55 40
> f6 c7 01 48 89 e5 53 48 89 fb 74 5b 48
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RIP
> [<ffffffff8100cb5b>] pin_pagetable_pfn+0x53/0x59
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: RSP <ffff8800303d1c28>
> /var/log/messages-Mar 27 17:04:39 xen05 kernel: ---[ end trace
> bf36c55d2ecd52e5 ]---
>
> >
> >
> > Maybe this helps, but i think, if anything, this makes it worse as the debug
> > options actually supressed the problem that needs to be debugged.
>
> True. At least now we know/narrow down to just related to
> CONFIG_DEBUG_PAGEALLOC. Maybe Konrad or Jeremy can have a closer look
> in the related codes... ...
>
> Thanks.
>
> Kindest regards,
> Giam Teck Choon
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
next prev parent reply other threads:[~2011-04-05 22:01 UTC|newest]
Thread overview: 82+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-12-26 8:16 kernel BUG at arch/x86/xen/mmu.c:1860! Teck Choon Giam
2010-12-27 15:53 ` Konrad Rzeszutek Wilk
2010-12-27 22:14 ` Teck Choon Giam
2010-12-28 10:42 ` Pasi Kärkkäinen
2010-12-28 18:01 ` Teck Choon Giam
2010-12-29 4:25 ` Teck Choon Giam
2010-12-29 4:58 ` Teck Choon Giam
2011-01-14 15:20 ` Konrad Rzeszutek Wilk
2011-01-14 19:25 ` Teck Choon Giam
2011-01-14 19:44 ` Konrad Rzeszutek Wilk
2011-01-14 20:09 ` Teck Choon Giam
2011-01-14 20:32 ` Teck Choon Giam
2011-01-24 1:42 ` Teck Choon Giam
2011-01-24 14:36 ` Konrad Rzeszutek Wilk
2011-01-24 15:56 ` Teck Choon Giam
2011-01-25 14:48 ` Konrad Rzeszutek Wilk
2011-01-26 14:31 ` Konrad Rzeszutek Wilk
2011-01-27 17:17 ` Teck Choon Giam
2011-01-27 20:32 ` Konrad Rzeszutek Wilk
2011-01-27 22:20 ` Teck Choon Giam
2011-02-26 12:03 ` Teck Choon Giam
2011-02-28 16:20 ` Konrad Rzeszutek Wilk
2011-03-01 9:59 ` Teck Choon Giam
2011-03-03 22:16 ` Konrad Rzeszutek Wilk
2011-03-04 5:30 ` Teck Choon Giam
2011-03-04 6:15 ` Fajar A. Nugraha
2011-03-04 6:33 ` Teck Choon Giam
2011-03-08 19:29 ` Konrad Rzeszutek Wilk
2011-03-08 20:10 ` Konrad Rzeszutek Wilk
2011-03-08 20:20 ` Teck Choon Giam
2011-03-08 20:45 ` Guido Hecken
2011-03-08 20:50 ` [SPAM] " tjaouen
2011-03-09 0:06 ` Andreas Olsowski
2011-03-11 18:38 ` benco
2011-03-11 19:59 ` Sander Eikelenboom
2011-03-11 20:29 ` Teck Choon Giam
2011-03-11 20:45 ` Teck Choon Giam
2011-03-11 21:02 ` Sander Eikelenboom
2011-03-11 21:15 ` Teck Choon Giam
2011-03-09 0:43 ` [SPAM] " Konrad Rzeszutek Wilk
2011-03-09 6:58 ` Andreas Olsowski
2011-03-09 15:00 ` Konrad Rzeszutek Wilk
2011-03-09 19:59 ` Andreas Olsowski
2011-03-10 7:20 ` Andreas Olsowski
2011-03-10 13:45 ` Andreas Olsowski
2011-03-11 18:05 ` Konrad Rzeszutek Wilk
2011-03-14 10:25 ` Ian Campbell
2011-03-14 10:36 ` Teck Choon Giam
2011-03-16 15:52 ` [SPAM] Re: kernel BUG at arch/x86/xen/mmu.c:1860! - ideas Konrad Rzeszutek Wilk
2011-03-16 16:26 ` Teck Choon Giam
2011-03-16 16:40 ` Konrad Rzeszutek Wilk
2011-03-24 11:57 ` Konrad Rzeszutek Wilk
2011-03-24 21:28 ` Teck Choon Giam
2011-03-25 3:57 ` Teck Choon Giam
2011-03-27 10:16 ` Teck Choon Giam
2011-03-28 11:37 ` Andreas Olsowski
2011-03-28 12:29 ` Teck Choon Giam
2011-04-05 22:01 ` Dave Hunter [this message]
2011-04-05 22:15 ` Teck Choon Giam
2011-04-05 23:20 ` Dave Hunter
2011-04-06 7:53 ` Ian Campbell
2011-04-06 21:52 ` Jeremy Fitzhardinge
2011-04-07 13:16 ` Teck Choon Giam
2011-03-09 0:41 ` kernel BUG at arch/x86/xen/mmu.c:1860! Konrad Rzeszutek Wilk
2011-01-04 15:10 ` Christophe Saout
2011-01-04 15:19 ` Christophe Saout
2011-01-04 15:37 ` benco
2011-01-04 18:40 ` Christophe Saout
2011-01-04 19:32 ` Teck Choon Giam
2011-01-04 19:56 ` benco
2011-01-14 15:22 ` Konrad Rzeszutek Wilk
2011-01-14 15:33 ` Christophe Saout
2011-01-04 23:10 ` Christophe Saout
2011-01-05 10:51 ` Pasi Kärkkäinen
2011-01-05 14:56 ` Teck Choon Giam
2011-01-14 15:24 ` Konrad Rzeszutek Wilk
2011-01-14 19:31 ` Teck Choon Giam
2011-01-04 13:48 ` Ian Campbell
2011-01-04 19:24 ` Teck Choon Giam
2011-01-05 15:30 ` Teck Choon Giam
2011-01-13 14:28 ` tjaouen
2011-01-14 14:47 ` Konrad Rzeszutek Wilk
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1302040916.2042.108.camel@dave \
--to=dave@ivt.com.au \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).