From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Vrabel Subject: Re: WARNING: at drivers/xen/gntdev.c:426 unmap_if_in_range+0x5d/0x60 [xen_gntdev]() Date: Mon, 15 Dec 2014 11:11:23 +0000 Message-ID: <548EC1DB.6070301@citrix.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Christopher S. Aker" , xen devel List-Id: xen-devel@lists.xenproject.org On 11/12/14 15:12, Christopher S. Aker wrote: > Xen: 4.4.2-pre (28573:f6f6236af933) + xsa111, xsa112, xsa114 > Dom0: 3.17.4 > > Things go badly after a day or four. We've hit this on a number of previously healthy hosts, since moving from 3.10.x dom0 to 3.17.4: > > printk: 5441 messages suppressed. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. Can you provide more details about your networking and storage setup. In particular, do you have a domU providing networked storage (iscsi for example) to other domains on the same host? David > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > grant_table.c:567:d0 Failed to obtain maptrack handle. > (XEN) printk: 4857 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 4846 callbacks suppressed > (XEN) printk: 4699 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1569 callbacks suppressed > (XEN) printk: 1809 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 2327 callbacks suppressed > (XEN) printk: 2779 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 2509 callbacks suppressed > (XEN) printk: 2022 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 2282 callbacks suppressed > (XEN) printk: 2778 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 2385 callbacks suppressed > (XEN) printk: 1560 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1714 callbacks suppressed > (XEN) printk: 1713 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1619 callbacks suppressed > (XEN) printk: 1852 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1895 callbacks suppressed > (XEN) printk: 2058 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1797 callbacks suppressed > (XEN) printk: 1530 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1440 callbacks suppressed > (XEN) printk: 1306 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > > (...this repeats a few hundred times over the course of 30 minutes...) > > net_ratelimit: 1221 callbacks suppressed > (XEN) printk: 1719 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1747 callbacks suppressed > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 1496 callbacks suppressed > br0: port 80(vif242.0) entered disabled state > device vif242.0 left promiscuous mode > br0: port 80(vif242.0) entered disabled state > device vif249.0 entered promiscuous mode > xen-blkback:ring-ref 8, event-channel 9, protocol 1 (x86_64-abi) persistent grants > xen-blkback:ring-ref 9, event-channel 10, protocol 1 (x86_64-abi) persistent grants > (XEN) printk: 1107 messages suppressed. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > (XEN) grant_table.c:567:d0 Failed to obtain maptrack handle. > net_ratelimit: 648 callbacks suppressed > m2p_remove_override: pfn 10828f2 mfn 8000000005b4284e, failed to modify kernel mappings > ------------[ cut here ]------------ > WARNING: CPU: 6 PID: 23911 at drivers/xen/gntdev.c:426 unmap_if_in_range+0x5d/0x60 [xen_gntdev]() > Modules linked in: xt_u32 xt_physdev ebt_comment ebt_arp ebt_set ebt_limit ebt_ip6 ebt_ip ip_set_hash_net ip_set ip6table_mangle ip6_tables ebtable_nat xen_acpi_processor xen_pciback xen_gntalloc xen_gntdev bonding ebtable_filter 8021q mrp ixgbe mdio ptp pps_core > CPU: 6 PID: 23911 Comm: qemu-dm Not tainted 3.17.4-1 #1 > Hardware name: Supermicro X9DRE-TF+/X9DR7-TF+/X9DRE-TF+/X9DR7-TF+, BIOS 3.0a 12/04/2013 > 0000000000000009 ffff880043dafcc8 ffffffff81876bcb 0000000000000001 > 0000000000000000 ffff880043dafd08 ffffffff81069777 ffff880043dafd18 > ffff880020154690 00007f8add804000 00007f8add80f000 ffff880020154660 > Call Trace: > [] dump_stack+0x46/0x58 > [] warn_slowpath_common+0x87/0xb0 > [] warn_slowpath_null+0x15/0x20 > [] unmap_if_in_range+0x5d/0x60 [xen_gntdev] > [] mn_invl_range_start+0x4e/0xa0 [xen_gntdev] > [] __mmu_notifier_invalidate_range_start+0x5b/0x90 > [] unmap_vmas+0x79/0x90 > [] unmap_region+0xa3/0x120 > [] ? new_sync_read+0x79/0xb0 > [] ? vma_rb_erase+0x121/0x210 > [] do_munmap+0x2a0/0x3b0 > [] vm_munmap+0x49/0x70 > [] SyS_munmap+0x26/0x40 > [] system_call_fastpath+0x16/0x1b > ---[ end trace 25ca87f9adc0ad78 ]--- > INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 32, t=60002 jiffies, g=26177592, c=26177591, q=1229) > Task dump for CPU 0: > swapper/0 R running task 14072 0 0 0x00000008 > 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff > ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30 > 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0 > Call Trace: > [] ? xen_hypercall_sched_op+0xa/0x20 > [] ? xen_safe_halt+0x10/0x20 > [] ? default_idle+0x1f/0xb0 > [] ? arch_cpu_idle+0xa/0x10 > [] ? cpu_startup_entry+0x284/0x330 > [] ? rest_init+0x6d/0x70 > [] ? start_kernel+0x41d/0x42a > [] ? set_init_arg+0x58/0x58 > [] ? x86_64_start_reservations+0x2a/0x2c > [] ? xen_start_kernel+0x540/0x542 > INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 34, t=240007 jiffies, g=26177592, c=26177591, q=4592) > Task dump for CPU 0: > swapper/0 R running task 14072 0 0 0x00000008 > 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff > ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30 > 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0 > Call Trace: > [] ? xen_hypercall_sched_op+0xa/0x20 > [] ? xen_safe_halt+0x10/0x20 > [] ? default_idle+0x1f/0xb0 > [] ? arch_cpu_idle+0xa/0x10 > [] ? cpu_startup_entry+0x284/0x330 > [] ? rest_init+0x6d/0x70 > [] ? start_kernel+0x41d/0x42a > [] ? set_init_arg+0x58/0x58 > [] ? x86_64_start_reservations+0x2a/0x2c > [] ? xen_start_kernel+0x540/0x542 > INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 34, t=420012 jiffies, g=26177592, c=26177591, q=8255) > Task dump for CPU 0: > swapper/0 R running task 14072 0 0 0x00000008 > 00000000ffffffed 0000000000000000 0000000000000001 ffffffffffffffff > ffffffff810013aa 000000000000e030 0000000000000246 ffffffff81e03e30 > 000000000000e02b 0000000000000000 0000000000000000 ffffffff8100a0c0 > Call Trace: > [] ? xen_hypercall_sched_op+0xa/0x20 > [] ? xen_safe_halt+0x10/0x20 > [] ? default_idle+0x1f/0xb0 > [] ? arch_cpu_idle+0xa/0x10 > [] ? cpu_startup_entry+0x284/0x330 > [] ? rest_init+0x6d/0x70 > [] ? start_kernel+0x41d/0x42a > [] ? set_init_arg+0x58/0x58 > [] ? x86_64_start_reservations+0x2a/0x2c > [] ? xen_start_kernel+0x540/0x542 > > Then the dom0 is unresponsive, and requires a reboot. > > Any ideas? > > -Chris > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel >