From: David Vrabel <david.vrabel@citrix.com>
To: "Christopher S. Aker" <caker@theshore.net>,
xen devel <xen-devel@lists.xensource.com>
Subject: Re: WARNING: at drivers/xen/gntdev.c:426 unmap_if_in_range+0x5d/0x60 [xen_gntdev]()
Date: Mon, 15 Dec 2014 11:11:23 +0000 [thread overview]
Message-ID: <548EC1DB.6070301@citrix.com> (raw)
In-Reply-To: <F8DF2DBE-7E43-4F1E-8303-6634693666F8@theshore.net>
On 11/12/14 15:12, Christopher S. Aker wrote:
> Xen: 4.4.2-pre (28573:f6f6236af933) + xsa111, xsa112, xsa114
> Dom0: 3.17.4
>
> Things go badly after a day or four. We've hit this on a number of previously healthy hosts, since moving from 3.10.x dom0 to 3.17.4:
>
> printk: 5441 messages suppressed.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
Can you provide more details about your networking and storage setup.
In particular, do you have a domU providing networked storage (iscsi for
example) to other domains on the same host?
David
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> grant_table.c:567:d0 Failed to obtain maptrack handle.
> (XEN) printk: 4857 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 4846 callbacks suppressed
> (XEN) printk: 4699 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1569 callbacks suppressed
> (XEN) printk: 1809 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 2327 callbacks suppressed
> (XEN) printk: 2779 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 2509 callbacks suppressed
> (XEN) printk: 2022 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 2282 callbacks suppressed
> (XEN) printk: 2778 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 2385 callbacks suppressed
> (XEN) printk: 1560 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1714 callbacks suppressed
> (XEN) printk: 1713 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1619 callbacks suppressed
> (XEN) printk: 1852 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1895 callbacks suppressed
> (XEN) printk: 2058 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1797 callbacks suppressed
> (XEN) printk: 1530 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1440 callbacks suppressed
> (XEN) printk: 1306 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
>
> (...this repeats a few hundred times over the course of 30 minutes...)
>
> net_ratelimit: 1221 callbacks suppressed
> (XEN) printk: 1719 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1747 callbacks suppressed
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 1496 callbacks suppressed
> br0: port 80(vif242.0) entered disabled state
> device vif242.0 left promiscuous mode
> br0: port 80(vif242.0) entered disabled state
> device vif249.0 entered promiscuous mode
> xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi) persistent grants
> xen-blkback:ring-ref 9, event-channel 10, protocol 1 (x86_64-abi) persistent grants
> (XEN) printk: 1107 messages suppressed.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle.
> net_ratelimit: 648 callbacks suppressed
> m2p_remove_override: pfn 10828f2 mfn 8000000005b4284e, failed to modify kernel mappings
> ------------[ cut here ]------------
> WARNING: CPU: 6 PID: 23911 at drivers/xen/gntdev.c:426 unmap_if_in_range+0x5d/0x60 [xen_gntdev]()
> Modules linked in: xt_u32 xt_physdev ebt_comment ebt_arp ebt_set ebt_limit ebt_ip6 ebt_ip ip_set_hash_net ip_set ip6table_mangle ip6_tables ebtable_nat xen_acpi_processor xen_pciback xen_gntalloc xen_gntdev bonding ebtable_filter 8021q mrp ixgbe mdio ptp pps_core
> CPU: 6 PID: 23911 Comm: qemu-dm Not tainted 3.17.4-1 #1
> Hardware name: Supermicro X9DRE-TF+/X9DR7-TF+/X9DRE-TF+/X9DR7-TF+, BIOS 3.0a 12/04/2013
> 0000000000000009 ffff880043dafcc8 ffffffff81876bcb 0000000000000001
> 0000000000000000 ffff880043dafd08 ffffffff81069777 ffff880043dafd18
> ffff880020154690 00007f8add804000 00007f8add80f000 ffff880020154660
> Call Trace:
> [<ffffffff81876bcb>] dump_stack+0x46/0x58
> [<ffffffff81069777>] warn_slowpath_common+0x87/0xb0
> [<ffffffff810697b5>] warn_slowpath_null+0x15/0x20
> [<ffffffffa012d29d>] unmap_if_in_range+0x5d/0x60 [xen_gntdev]
> [<ffffffffa012d46e>] mn_invl_range_start+0x4e/0xa0 [xen_gntdev]
> [<ffffffff811615cb>] __mmu_notifier_invalidate_range_start+0x5b/0x90
> [<ffffffff811469a9>] unmap_vmas+0x79/0x90
> [<ffffffff8114bb13>] unmap_region+0xa3/0x120
> [<ffffffff8116b339>] ? new_sync_read+0x79/0xb0
> [<ffffffff8114bfb1>] ? vma_rb_erase+0x121/0x210
> [<ffffffff8114dba0>] do_munmap+0x2a0/0x3b0
> [<ffffffff8114dcf9>] vm_munmap+0x49/0x70
> [<ffffffff8114ecd6>] SyS_munmap+0x26/0x40
> [<ffffffff81880169>] system_call_fastpath+0x16/0x1b
> ---[ end trace 25ca87f9adc0ad78 ]---
> INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 32, t=60002 jiffies, g=26177592, c=26177591, q=1229)
> Task dump for CPU 0:
> swapper/0 R running task 14072 0 0 0x00000008
> 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff
> ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30
> 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0
> Call Trace:
> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
> [<ffffffff8100a0c0>] ? xen_safe_halt+0x10/0x20
> [<ffffffff8101d73f>] ? default_idle+0x1f/0xb0
> [<ffffffff8101dfea>] ? arch_cpu_idle+0xa/0x10
> [<ffffffff8109ead4>] ? cpu_startup_entry+0x284/0x330
> [<ffffffff8186ec7d>] ? rest_init+0x6d/0x70
> [<ffffffff81eea081>] ? start_kernel+0x41d/0x42a
> [<ffffffff81ee9a51>] ? set_init_arg+0x58/0x58
> [<ffffffff81ee95f0>] ? x86_64_start_reservations+0x2a/0x2c
> [<ffffffff81eed774>] ? xen_start_kernel+0x540/0x542
> INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 34, t=240007 jiffies, g=26177592, c=26177591, q=4592)
> Task dump for CPU 0:
> swapper/0 R running task 14072 0 0 0x00000008
> 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff
> ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30
> 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0
> Call Trace:
> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
> [<ffffffff8100a0c0>] ? xen_safe_halt+0x10/0x20
> [<ffffffff8101d73f>] ? default_idle+0x1f/0xb0
> [<ffffffff8101dfea>] ? arch_cpu_idle+0xa/0x10
> [<ffffffff8109ead4>] ? cpu_startup_entry+0x284/0x330
> [<ffffffff8186ec7d>] ? rest_init+0x6d/0x70
> [<ffffffff81eea081>] ? start_kernel+0x41d/0x42a
> [<ffffffff81ee9a51>] ? set_init_arg+0x58/0x58
> [<ffffffff81ee95f0>] ? x86_64_start_reservations+0x2a/0x2c
> [<ffffffff81eed774>] ? xen_start_kernel+0x540/0x542
> INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 34, t=420012 jiffies, g=26177592, c=26177591, q=8255)
> Task dump for CPU 0:
> swapper/0 R running task 14072 0 0 0x00000008
> 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff
> ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30
> 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0
> Call Trace:
> [<ffffffff810013aa>] ? xen_hypercall_sched_op+0xa/0x20
> [<ffffffff8100a0c0>] ? xen_safe_halt+0x10/0x20
> [<ffffffff8101d73f>] ? default_idle+0x1f/0xb0
> [<ffffffff8101dfea>] ? arch_cpu_idle+0xa/0x10
> [<ffffffff8109ead4>] ? cpu_startup_entry+0x284/0x330
> [<ffffffff8186ec7d>] ? rest_init+0x6d/0x70
> [<ffffffff81eea081>] ? start_kernel+0x41d/0x42a
> [<ffffffff81ee9a51>] ? set_init_arg+0x58/0x58
> [<ffffffff81ee95f0>] ? x86_64_start_reservations+0x2a/0x2c
> [<ffffffff81eed774>] ? xen_start_kernel+0x540/0x542
>
> Then the dom0 is unresponsive, and requires a reboot.
>
> Any ideas?
>
> -Chris
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>
next prev parent reply other threads:[~2014-12-15 11:11 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-11 15:12 WARNING: at drivers/xen/gntdev.c:426 unmap_if_in_range+0x5d/0x60 [xen_gntdev]() Christopher S. Aker
2014-12-14 17:48 ` Christopher S. Aker
2014-12-15 11:11 ` David Vrabel [this message]
2014-12-16 23:04 ` Christopher S. Aker
2014-12-17 10:47 ` David Vrabel
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=548EC1DB.6070301@citrix.com \
--to=david.vrabel@citrix.com \
--cc=caker@theshore.net \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.