Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Armin Zentai]

Dear Xen Developers!


We're running Xen on multiple machines, most of them are Dell R410 or SM 
X8DTL, with one E5645 cpu, and 48 GB of RAM. We've update the kernel to 
3.15.4, after the some of our hypervisors started to rebooting at random 
times.

The logs were empty, and we have no information about the crashes, we've 
tried some tricks, and at the end the netconsole kernel modul helped, so 
we can do a very thin layer of remote kernel logging. We've found the 
following in the remote logs:

Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb: 
Trying to unmap invalid handle! pending_idx: c
Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here ]------------
Jul 13 00:46:58 node11 [157060.106546] kernel BUG at 
drivers/net/xen-netback/netback.c:998!
Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]
Jul 13 00:46:58 SMP
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.106765] Modules linked in:
Jul 13 00:46:58 node11  netconsole
Jul 13 00:46:58 node11  configfs
Jul 13 00:46:58 node11  nf_conntrack_ipv4
Jul 13 00:46:58 node11  nf_defrag_ipv4
Jul 13 00:46:58 node11  xt_multiport
Jul 13 00:46:58 node11  xt_physdev
Jul 13 00:46:58 node11  sch_tbf
Jul 13 00:46:58 node11  dm_snapshot
Jul 13 00:46:58 node11  dm_bufio
Jul 13 00:46:58 node11  arptable_filter
Jul 13 00:46:58 node11  arp_tables
Jul 13 00:46:58 node11  ip6t_REJECT
Jul 13 00:46:58 node11  ip6table_mangle
Jul 13 00:46:58 node11  ipt_REJECT
Jul 13 00:46:58 node11  iptable_filter
Jul 13 00:46:58 node11  ip_tables
Jul 13 00:46:58 node11  bridge
Jul 13 00:46:58 node11  xen_pciback
Jul 13 00:46:58 node11  xen_gntalloc
Jul 13 00:46:58 node11  autofs4
Jul 13 00:46:58 node11  dm_round_robin
Jul 13 00:46:58 node11  scsi_dh_alua
Jul 13 00:46:58 node11  8021q
Jul 13 00:46:58 node11  mrp
Jul 13 00:46:58 node11  garp
Jul 13 00:46:58 node11  stp
Jul 13 00:46:58 node11  llc
Jul 13 00:46:58 node11  bonding
Jul 13 00:46:58 node11  xt_tcpudp
Jul 13 00:46:58 node11  nf_conntrack_ipv6
Jul 13 00:46:58 node11  nf_defrag_ipv6
Jul 13 00:46:58 node11  xt_state
Jul 13 00:46:58 node11  nf_conntrack
Jul 13 00:46:58 node11  ip6table_filter
Jul 13 00:46:58 node11  ip6_tables
Jul 13 00:46:58 node11  x_tables
Jul 13 00:46:58 node11  bnx2i
Jul 13 00:46:58 node11  cnic
Jul 13 00:46:58 node11  uio
Jul 13 00:46:58 node11  cxgb4i
Jul 13 00:46:58 node11  cxgb4
Jul 13 00:46:58 node11  cxgb3i
Jul 13 00:46:58 node11  libcxgbi
Jul 13 00:46:58 node11  cxgb3
Jul 13 00:46:58 node11  mdio
Jul 13 00:46:58 node11  ib_iser
Jul 13 00:46:58 node11  rdma_cm
Jul 13 00:46:58 node11  ib_cm
Jul 13 00:46:58 node11  iw_cm
Jul 13 00:46:58 node11  ib_sa
Jul 13 00:46:58 node11  ib_mad
Jul 13 00:46:58 node11  ib_core
Jul 13 00:46:58 node11  ib_addr
Jul 13 00:46:58 node11  iscsi_tcp
Jul 13 00:46:58 node11  libiscsi_tcp
Jul 13 00:46:58 node11  binfmt_misc
Jul 13 00:46:58 node11  dm_mirror
Jul 13 00:46:58 node11  dm_region_hash
Jul 13 00:46:58 node11  dm_log
Jul 13 00:46:58 node11  dm_multipath
Jul 13 00:46:58 node11  scsi_dh
Jul 13 00:46:58 node11  xen_netback
Jul 13 00:46:58 node11  xen_blkback
Jul 13 00:46:58 node11  xen_gntdev
Jul 13 00:46:58 node11  xen_evtchn
Jul 13 00:46:58 node11  xenfs
Jul 13 00:46:58 node11  xen_privcmd
Jul 13 00:46:58 node11  ufs
Jul 13 00:46:58 node11  gpio_ich
Jul 13 00:46:58 node11  iTCO_wdt
Jul 13 00:46:58 node11  iTCO_vendor_support
Jul 13 00:46:58 node11  joydev
Jul 13 00:46:58 node11  psmouse
Jul 13 00:46:58 node11  serio_raw
Jul 13 00:46:58 node11  pcspkr
Jul 13 00:46:58 node11  tpm_infineon
Jul 13 00:46:58 node11  i2c_i801
Jul 13 00:46:58 node11  lpc_ich
Jul 13 00:46:58 node11  hid_generic
Jul 13 00:46:58 node11  e1000e
Jul 13 00:46:58 node11  ptp
Jul 13 00:46:58 node11  pps_core
Jul 13 00:46:58 node11  snd_hda_intel
Jul 13 00:46:58 node11  snd_hda_controller
Jul 13 00:46:58 node11  snd_hda_codec
Jul 13 00:46:58 node11  snd_hwdep
Jul 13 00:46:58 node11  snd_seq
Jul 13 00:46:58 node11  snd_seq_device
Jul 13 00:46:58 node11  snd_pcm
Jul 13 00:46:58 node11  snd_timer
Jul 13 00:46:58 node11  snd
Jul 13 00:46:58 node11  soundcore
Jul 13 00:46:58 node11  ioatdma
Jul 13 00:46:58 node11  dca
Jul 13 00:46:58 node11  mac_hid
Jul 13 00:46:58 node11  i7core_edac
Jul 13 00:46:58 node11  edac_core
Jul 13 00:46:58 node11  be2iscsi
Jul 13 00:46:58 node11  iscsi_boot_sysfs
Jul 13 00:46:58 node11  libiscsi
Jul 13 00:46:58 node11  scsi_transport_iscsi
Jul 13 00:46:58 node11  be2net
Jul 13 00:46:58 node11  vxlan
Jul 13 00:46:58 node11  ahci(E)
Jul 13 00:46:58 node11  libahci(E)
Jul 13 00:46:58 node11  usbhid(E)
Jul 13 00:46:58 node11  hid(E)
Jul 13 00:46:58 node11  [last unloaded: evbug]
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0 
Tainted: G            E 3.15.4 #1
Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro 
X8DTL/X8DTL, BIOS 1.1b    03/19/2010
Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti: 
ffffffff81c00000 task.ti: ffffffff81c00000
Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
Jul 13 00:46:58 node11  [<ffffffffa025f61d>] 
xenvif_idx_unmap+0x11d/0x130 [xen_netback]
Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48 
EFLAGS: 00010292
Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX: 
000000000000000c RCX: 0000000000000000
Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI: 
ffff88008ea03a18 RDI: 000000000000021b
Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08: 
0000000000000000 R09: ffff88008a40b600
Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11: 
00000000000006d8 R12: ffff8800569708c0
Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14: 
ffff8800569708c0 R15: 0000000000000001
Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000) 
GS:ffff88008ea00000(0000) knlGS:0000000000000000
Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0: 
000000008005003b
Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3: 
00000000049f3000 CR4: 0000000000002660
Jul 13 00:46:58 node11 [157060.113835] Stack:
Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
Jul 13 00:46:58 node11  ff00000000000001
Jul 13 00:46:58 node11  ffff880b0605e000
Jul 13 00:46:58 node11  0000000000000000
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
Jul 13 00:46:58 node11  00000000fffffff6
Jul 13 00:46:58 node11  0000000000000001
Jul 13 00:46:58 node11  ffff8800569769d0
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
Jul 13 00:46:58 node11  ffffffffa02622fc
Jul 13 00:46:58 node11  ffff88008ea03dd8
Jul 13 00:46:58 node11  ffffffff810b5223
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.114637] Call Trace:
Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.114750]
Jul 13 00:46:58 node11  [<ffffffffa02622fc>] 
xenvif_tx_action+0x27c/0x7f0 [xen_netback]
Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ? 
__wake_up+0x53/0x70
Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ? 
handle_irq_event_percpu+0xa7/0x1b0
Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>] 
xenvif_poll+0x31/0x64 [xen_netback]
Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>] 
net_rx_action+0x10b/0x290
Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>] 
__do_softirq+0x103/0x320
Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>] 
irq_exit+0x135/0x140
Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>] 
xen_evtchn_do_upcall+0x3c/0x50
Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>] 
xen_do_hypervisor_callback+0x1e/0x30
Jul 13 00:46:58 node11 [157060.115506]  <EOI>
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.115551]
Jul 13 00:46:58 node11  [<ffffffff810013aa>] ? 
xen_hypercall_sched_op+0xa/0x20
Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ? 
xen_hypercall_sched_op+0xa/0x20
Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ? 
xen_safe_halt+0x10/0x20
Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ? 
default_idle+0x1f/0xc0
Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ? 
arch_cpu_idle+0xf/0x20
Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ? 
cpu_startup_entry+0x201/0x360
Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ? 
rest_init+0x77/0x80
Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ? 
start_kernel+0x406/0x413
Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ? 
repair_env_string+0x5b/0x5b
Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ? 
x86_64_start_reservations+0x2a/0x2c
Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ? 
xen_start_kernel+0x584/0x586
Jul 13 00:46:58 node11 [157060.116446] Code:
Jul 13 00:46:58 41
Jul 13 00:46:58 5c
Jul 13 00:46:58 c9
Jul 13 00:46:58 c3
Jul 13 00:46:58 48
Jul 13 00:46:58 be
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 80
Jul 13 00:46:58 ff
Jul 13 00:46:58 77
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 e9
Jul 13 00:46:58 62
Jul 13 00:46:58 ff
Jul 13 00:46:58 ff
Jul 13 00:46:58 ff
Jul 13 00:46:58 49
Jul 13 00:46:58 8b
Jul 13 00:46:58 bc
Jul 13 00:46:58 24
Jul 13 00:46:58 78
Jul 13 00:46:58 ba
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 89
Jul 13 00:46:58 da
Jul 13 00:46:58 48
Jul 13 00:46:58 c7
Jul 13 00:46:58 c6
Jul 13 00:46:58 30
Jul 13 00:46:58 57
Jul 13 00:46:58 26
Jul 13 00:46:58 a0
Jul 13 00:46:58 31
Jul 13 00:46:58 c0
Jul 13 00:46:58 e8
Jul 13 00:46:58 73
Jul 13 00:46:58 df
Jul 13 00:46:58 3e
Jul 13 00:46:58 e1
Jul 13 00:46:58 node11 f>
Jul 13 00:46:58 0b
Jul 13 00:46:58 eb
Jul 13 00:46:58 fe
Jul 13 00:46:58 66
Jul 13 00:46:58 66
Jul 13 00:46:58 66
Jul 13 00:46:58 66
Jul 13 00:46:58 66
Jul 13 00:46:58 66
Jul 13 00:46:58 2e
Jul 13 00:46:58 0f
Jul 13 00:46:58 1f
Jul 13 00:46:58 84
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 00
Jul 13 00:46:58 55
Jul 13 00:46:58 48
Jul 13 00:46:58 node11
Jul 13 00:46:58 node11 [157060.119179] RIP
Jul 13 00:46:58 node11  [<ffffffffa025f61d>] 
xenvif_idx_unmap+0x11d/0x130 [xen_netback]
Jul 13 00:46:58 node11 [157060.119312]  RSP <ffff88008ea03d48>
Jul 13 00:46:58 node11 [157060.119395] ---[ end trace 7e021c96c8cfea53 ]---
Jul 13 00:46:58 node11 [157060.119465] Kernel panic - not syncing: Fatal 
exception in interrupt


h14z4mzbvfrrhb was a name of a VIF. This VIF belongs to a Windows Server 
2008 R2 X64 virtual machine. We had 6 random reboots until now, all of 
the VIFs are belonged to the same operating system, but different 
virtual machines. So only Windows Server 2008 R2 X64 system's virtual 
interfaces caused the crashes, these systems has been provisioned from 
different installs or templates. The GPLPV driver's versions are also 
different.

XM info output:
[root@c2-node11 ~]# xm info
host                   : c2-node11
release                : 3.15.4
version                : #1 SMP Tue Jul 8 17:58:26 CEST 2014
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2400
hw_caps                : 
bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 49143
free_memory            : 41830
free_cpus              : 0
xen_major              : 4
xen_minor              : 2
xen_extra              : .4-33.el6
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : dom0_mem=3145728 pcie_aspm=off noreboot=true
cc_compiler            : gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Mon Jun 16 17:22:14 UTC 2014
xend_config_format     : 4

[root@c2-node11 ~]# uname -a
Linux c2-node11 3.15.4 #1 SMP Tue Jul 8 17:58:26 CEST 2014 x86_64 x86_64 
x86_64 GNU/Linux


The xm create config file of the specified VM (the other VM's config 
files are the same):

kernel = "/usr/lib/xen/boot/hvmloader"
device_model = "/usr/lib64/xen/bin/qemu-dm"
builder = "hvm"
memory = "2000"
name = "vna3mhwnv9pn4m"
vcpus = "1"

timer_mode = "2"
viridian = "1"

vif = [ "type=ioemu, mac=00:16:3e:64:c8:ba, bridge=x0evss6g1ztoa4, 
ip=..., vifname=h14z4mzbvfrrhb, rate=100Mb/s" ]

disk = [ "phy:/dev/q7jiqc2gh02b2b/xz7wget4ycmp77,ioemu:hda,w" ]
vnc = 1
vncpasswd="aaaaa1"
usbdevice="tablet"


The HV's networking looks as the following:
We are using dual emulex 10gbit network adapters, with bonding (LACP), 
and on the top of the bond, we're using VLAN's for the VM, management 
and the iSCSI traffic.
We're tried to reproduce the error, but we couldn't, the crash/reboot 
happened randomly every time.

Thanks, for your help,

  - Armin Zentai

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Wei Liu]

Hello

On Mon, Jul 14, 2014 at 04:25:54AM +0200, Armin Zentai wrote:
> Dear Xen Developers!
> 
> 
> We're running Xen on multiple machines, most of them are Dell R410 or SM
> X8DTL, with one E5645 cpu, and 48 GB of RAM. We've update the kernel to
> 3.15.4, after the some of our hypervisors started to rebooting at random
> times.
> 
> The logs were empty, and we have no information about the crashes, we've
> tried some tricks, and at the end the netconsole kernel modul helped, so we
> can do a very thin layer of remote kernel logging. We've found the following
> in the remote logs:

It's good you've got netconsole working. I would still like to point out
that we have a wiki page on setting up serial console on Xen, which
might be helpful.

http://wiki.xen.org/wiki/Xen_Serial_Console

> 
> Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb: Trying to
> unmap invalid handle! pending_idx: c
> Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here ]------------
> Jul 13 00:46:58 node11 [157060.106546] kernel BUG at
> drivers/net/xen-netback/netback.c:998!
> Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]
> Jul 13 00:46:58 SMP
> Jul 13 00:46:58 node11
[...]
> Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0
> Tainted: G            E 3.15.4 #1
> Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro
> X8DTL/X8DTL, BIOS 1.1b    03/19/2010
> Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti:
> ffffffff81c00000 task.ti: ffffffff81c00000
> Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
> Jul 13 00:46:58 node11  [<ffffffffa025f61d>] xenvif_idx_unmap+0x11d/0x130
> [xen_netback]
> Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48 EFLAGS:
> 00010292
> Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX:
> 000000000000000c RCX: 0000000000000000
> Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI:
> ffff88008ea03a18 RDI: 000000000000021b
> Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08:
> 0000000000000000 R09: ffff88008a40b600
> Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11:
> 00000000000006d8 R12: ffff8800569708c0
> Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14:
> ffff8800569708c0 R15: 0000000000000001
> Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000)
> GS:ffff88008ea00000(0000) knlGS:0000000000000000
> Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3:
> 00000000049f3000 CR4: 0000000000002660
> Jul 13 00:46:58 node11 [157060.113835] Stack:
> Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
> Jul 13 00:46:58 node11  ff00000000000001
> Jul 13 00:46:58 node11  ffff880b0605e000
> Jul 13 00:46:58 node11  0000000000000000
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
> Jul 13 00:46:58 node11  00000000fffffff6
> Jul 13 00:46:58 node11  0000000000000001
> Jul 13 00:46:58 node11  ffff8800569769d0
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
> Jul 13 00:46:58 node11  ffffffffa02622fc
> Jul 13 00:46:58 node11  ffff88008ea03dd8
> Jul 13 00:46:58 node11  ffffffff810b5223
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114637] Call Trace:
> Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114750]
> Jul 13 00:46:58 node11  [<ffffffffa02622fc>] xenvif_tx_action+0x27c/0x7f0
> [xen_netback]
> Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ?
> __wake_up+0x53/0x70
> Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ?
> handle_irq_event_percpu+0xa7/0x1b0
> Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>]
> xenvif_poll+0x31/0x64 [xen_netback]
> Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>]
> net_rx_action+0x10b/0x290
> Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>]
> __do_softirq+0x103/0x320
> Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>]
> irq_exit+0x135/0x140
> Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>]
> xen_evtchn_do_upcall+0x3c/0x50
> Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>]
> xen_do_hypervisor_callback+0x1e/0x30
> Jul 13 00:46:58 node11 [157060.115506]  <EOI>
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.115551]
> Jul 13 00:46:58 node11  [<ffffffff810013aa>] ?
> xen_hypercall_sched_op+0xa/0x20
> Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ?
> xen_hypercall_sched_op+0xa/0x20
> Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ?
> xen_safe_halt+0x10/0x20
> Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ?
> default_idle+0x1f/0xc0
> Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ?
> arch_cpu_idle+0xf/0x20
> Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ?
> cpu_startup_entry+0x201/0x360
> Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ?
> rest_init+0x77/0x80
> Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ?
> start_kernel+0x406/0x413
> Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ?
> repair_env_string+0x5b/0x5b
> Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ?
> x86_64_start_reservations+0x2a/0x2c
> Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ?
> xen_start_kernel+0x584/0x586
[...]
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.119179] RIP
> Jul 13 00:46:58 node11  [<ffffffffa025f61d>] xenvif_idx_unmap+0x11d/0x130
> [xen_netback]
> Jul 13 00:46:58 node11 [157060.119312]  RSP <ffff88008ea03d48>
> Jul 13 00:46:58 node11 [157060.119395] ---[ end trace 7e021c96c8cfea53 ]---
> Jul 13 00:46:58 node11 [157060.119465] Kernel panic - not syncing: Fatal
> exception in interrupt
> 
> 
> h14z4mzbvfrrhb was a name of a VIF. This VIF belongs to a Windows Server
> 2008 R2 X64 virtual machine. We had 6 random reboots until now, all of the
> VIFs are belonged to the same operating system, but different virtual
> machines. So only Windows Server 2008 R2 X64 system's virtual interfaces
> caused the crashes, these systems has been provisioned from different
> installs or templates. The GPLPV driver's versions are also different.
> 

Unfortunately I don't have Windows server 2008 R2. :-(

This bug is in guest TX path. What's the workload of your guest? Is
there any pattern of its traffic?

I've checked changesets between 3.15.4 and 3.16-rc5 there's no fix for
this, so this is the first report of this issue.  If there's a reliable
reproduce then that would be great.

Zoltan, have you seen this before? Can your work on pktgen help?

> [root@c2-node11 ~]# uname -a
> Linux c2-node11 3.15.4 #1 SMP Tue Jul 8 17:58:26 CEST 2014 x86_64 x86_64
> x86_64 GNU/Linux
> 
> 
> The xm create config file of the specified VM (the other VM's config files
> are the same):
> 
> kernel = "/usr/lib/xen/boot/hvmloader"
> device_model = "/usr/lib64/xen/bin/qemu-dm"
> builder = "hvm"
> memory = "2000"
> name = "vna3mhwnv9pn4m"
> vcpus = "1"
> 
> timer_mode = "2"
> viridian = "1"
> 
> vif = [ "type=ioemu, mac=00:16:3e:64:c8:ba, bridge=x0evss6g1ztoa4, ip=...,
> vifname=h14z4mzbvfrrhb, rate=100Mb/s" ]
> 
> disk = [ "phy:/dev/q7jiqc2gh02b2b/xz7wget4ycmp77,ioemu:hda,w" ]
> vnc = 1
> vncpasswd="aaaaa1"
> usbdevice="tablet"
> 
> 
> The HV's networking looks as the following:
> We are using dual emulex 10gbit network adapters, with bonding (LACP), and
> on the top of the bond, we're using VLAN's for the VM, management and the
> iSCSI traffic.
> We're tried to reproduce the error, but we couldn't, the crash/reboot
> happened randomly every time.
> 

In that case you will need to instrument netback to spit out more
information. Zoltan, is there any other information that you would like
to know?

Wei.

> Thanks, for your help,
> 
>  - Armin Zentai
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Armin Zentai]

Hello!

On 14/07/14 11:52, Wei Liu wrote:
> Hello
>
> On Mon, Jul 14, 2014 at 04:25:54AM +0200, Armin Zentai wrote:
>> Dear Xen Developers!
>>
>>
>> We're running Xen on multiple machines, most of them are Dell R410 or SM
>> X8DTL, with one E5645 cpu, and 48 GB of RAM. We've update the kernel to
>> 3.15.4, after the some of our hypervisors started to rebooting at random
>> times.
>>
>> The logs were empty, and we have no information about the crashes, we've
>> tried some tricks, and at the end the netconsole kernel modul helped, so we
>> can do a very thin layer of remote kernel logging. We've found the following
>> in the remote logs:
>
> It's good you've got netconsole working. I would still like to point out
> that we have a wiki page on setting up serial console on Xen, which
> might be helpful.
>
> http://wiki.xen.org/wiki/Xen_Serial_Console
>

We've set up xen serial console, but we wanted to avoid about to reboot 
the hypervisors if it's not neccessary, so it will be activated on the 
systems on the next reboot.

(We have set up a system, that logs into every Dell iDRAC via telnet [we 
have 18 nodes, so we cannot plug every machine via a physical serial 
link], set up the SOL, and logs every output, but netconsole was a much 
more painless solution to gather the logs)

>>
>> Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb: Trying to
>> unmap invalid handle! pending_idx: c
>> Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here ]------------
>> Jul 13 00:46:58 node11 [157060.106546] kernel BUG at
>> drivers/net/xen-netback/netback.c:998!
>> Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]
>> Jul 13 00:46:58 SMP
>> Jul 13 00:46:58 node11
> [...]
>> Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0
>> Tainted: G            E 3.15.4 #1
>> Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro
>> X8DTL/X8DTL, BIOS 1.1b    03/19/2010
>> Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti:
>> ffffffff81c00000 task.ti: ffffffff81c00000
>> Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
>> Jul 13 00:46:58 node11  [<ffffffffa025f61d>] xenvif_idx_unmap+0x11d/0x130
>> [xen_netback]
>> Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48 EFLAGS:
>> 00010292
>> Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX:
>> 000000000000000c RCX: 0000000000000000
>> Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI:
>> ffff88008ea03a18 RDI: 000000000000021b
>> Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08:
>> 0000000000000000 R09: ffff88008a40b600
>> Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11:
>> 00000000000006d8 R12: ffff8800569708c0
>> Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14:
>> ffff8800569708c0 R15: 0000000000000001
>> Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000)
>> GS:ffff88008ea00000(0000) knlGS:0000000000000000
>> Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0:
>> 000000008005003b
>> Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3:
>> 00000000049f3000 CR4: 0000000000002660
>> Jul 13 00:46:58 node11 [157060.113835] Stack:
>> Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
>> Jul 13 00:46:58 node11  ff00000000000001
>> Jul 13 00:46:58 node11  ffff880b0605e000
>> Jul 13 00:46:58 node11  0000000000000000
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
>> Jul 13 00:46:58 node11  00000000fffffff6
>> Jul 13 00:46:58 node11  0000000000000001
>> Jul 13 00:46:58 node11  ffff8800569769d0
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
>> Jul 13 00:46:58 node11  ffffffffa02622fc
>> Jul 13 00:46:58 node11  ffff88008ea03dd8
>> Jul 13 00:46:58 node11  ffffffff810b5223
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114637] Call Trace:
>> Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114750]
>> Jul 13 00:46:58 node11  [<ffffffffa02622fc>] xenvif_tx_action+0x27c/0x7f0
>> [xen_netback]
>> Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ?
>> __wake_up+0x53/0x70
>> Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ?
>> handle_irq_event_percpu+0xa7/0x1b0
>> Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>]
>> xenvif_poll+0x31/0x64 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>]
>> net_rx_action+0x10b/0x290
>> Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>]
>> __do_softirq+0x103/0x320
>> Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>]
>> irq_exit+0x135/0x140
>> Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>]
>> xen_evtchn_do_upcall+0x3c/0x50
>> Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>]
>> xen_do_hypervisor_callback+0x1e/0x30
>> Jul 13 00:46:58 node11 [157060.115506]  <EOI>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.115551]
>> Jul 13 00:46:58 node11  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ?
>> xen_safe_halt+0x10/0x20
>> Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ?
>> default_idle+0x1f/0xc0
>> Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ?
>> arch_cpu_idle+0xf/0x20
>> Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ?
>> cpu_startup_entry+0x201/0x360
>> Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ?
>> rest_init+0x77/0x80
>> Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ?
>> start_kernel+0x406/0x413
>> Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ?
>> repair_env_string+0x5b/0x5b
>> Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ?
>> x86_64_start_reservations+0x2a/0x2c
>> Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ?
>> xen_start_kernel+0x584/0x586
> [...]
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.119179] RIP
>> Jul 13 00:46:58 node11  [<ffffffffa025f61d>] xenvif_idx_unmap+0x11d/0x130
>> [xen_netback]
>> Jul 13 00:46:58 node11 [157060.119312]  RSP <ffff88008ea03d48>
>> Jul 13 00:46:58 node11 [157060.119395] ---[ end trace 7e021c96c8cfea53 ]---
>> Jul 13 00:46:58 node11 [157060.119465] Kernel panic - not syncing: Fatal
>> exception in interrupt
>>
>>
>> h14z4mzbvfrrhb was a name of a VIF. This VIF belongs to a Windows Server
>> 2008 R2 X64 virtual machine. We had 6 random reboots until now, all of the
>> VIFs are belonged to the same operating system, but different virtual
>> machines. So only Windows Server 2008 R2 X64 system's virtual interfaces
>> caused the crashes, these systems has been provisioned from different
>> installs or templates. The GPLPV driver's versions are also different.
>>
>
> Unfortunately I don't have Windows server 2008 R2. :-(
>
> This bug is in guest TX path. What's the workload of your guest? Is
> there any pattern of its traffic?

It's not relevant, some of them uses one core at nearly 100%, some of 
them had 1-2% CPU and 5-10 mbps of networking and/or IO. I've tried to 
test the CPU with CPU burn, prime95, tried to stress the network with 
SYN flood, IIS stress testing with apache ab, stressing the throughput 
and bandwidth, but these attempts did not caused a reboot.

>
> I've checked changesets between 3.15.4 and 3.16-rc5 there's no fix for
> this, so this is the first report of this issue.  If there's a reliable
> reproduce then that would be great.
>
> Zoltan, have you seen this before? Can your work on pktgen help?
>
>> [root@c2-node11 ~]# uname -a
>> Linux c2-node11 3.15.4 #1 SMP Tue Jul 8 17:58:26 CEST 2014 x86_64 x86_64
>> x86_64 GNU/Linux
>>
>>
>> The xm create config file of the specified VM (the other VM's config files
>> are the same):
>>
>> kernel = "/usr/lib/xen/boot/hvmloader"
>> device_model = "/usr/lib64/xen/bin/qemu-dm"
>> builder = "hvm"
>> memory = "2000"
>> name = "vna3mhwnv9pn4m"
>> vcpus = "1"
>>
>> timer_mode = "2"
>> viridian = "1"
>>
>> vif = [ "type=ioemu, mac=00:16:3e:64:c8:ba, bridge=x0evss6g1ztoa4, ip=...,
>> vifname=h14z4mzbvfrrhb, rate=100Mb/s" ]
>>
>> disk = [ "phy:/dev/q7jiqc2gh02b2b/xz7wget4ycmp77,ioemu:hda,w" ]
>> vnc = 1
>> vncpasswd="aaaaa1"
>> usbdevice="tablet"
>>
>>
>> The HV's networking looks as the following:
>> We are using dual emulex 10gbit network adapters, with bonding (LACP), and
>> on the top of the bond, we're using VLAN's for the VM, management and the
>> iSCSI traffic.
>> We're tried to reproduce the error, but we couldn't, the crash/reboot
>> happened randomly every time.
>>
>
> In that case you will need to instrument netback to spit out more
> information. Zoltan, is there any other information that you would like
> to know?
>
> Wei.
>
>> Thanks, for your help,
>>
>>   - Armin Zentai
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Wei Liu]

On Mon, Jul 14, 2014 at 12:53:49PM +0200, Armin Zentai wrote:
[...]
> >>h14z4mzbvfrrhb was a name of a VIF. This VIF belongs to a Windows Server
> >>2008 R2 X64 virtual machine. We had 6 random reboots until now, all of the
> >>VIFs are belonged to the same operating system, but different virtual
> >>machines. So only Windows Server 2008 R2 X64 system's virtual interfaces
> >>caused the crashes, these systems has been provisioned from different
> >>installs or templates. The GPLPV driver's versions are also different.
> >>
> >
> >Unfortunately I don't have Windows server 2008 R2. :-(
> >
> >This bug is in guest TX path. What's the workload of your guest? Is
> >there any pattern of its traffic?
> 
> It's not relevant, some of them uses one core at nearly 100%, some of them
> had 1-2% CPU and 5-10 mbps of networking and/or IO. I've tried to test the
> CPU with CPU burn, prime95, tried to stress the network with SYN flood, IIS
> stress testing with apache ab, stressing the throughput and bandwidth, but
> these attempts did not caused a reboot.
> 

FYI I think the only thing that matters is the packets sent from the
guest, stressing CPU won't help. I could be wrong, but the tests you
mentioned seem to be sending packets to the guest. That's guest RX path
which is not very relevant to this bug. I think the stress test should
run in the guest, i.e. sending packets from guest to outside world. That
might help reproduce this bug. 

Wei.

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

Hi,

Based on the stack trace I think the guest sends packet with more than 
MAX_SKB_SLOTS slots, and one slot's grant mapping operation goes wrong. 
Then the error handling screws up the internal bookkeeping of pending 
slots, and tries to unmap something that is not mapped. Unfortunately it 
is quite complicated to emulate grant mapping failures in a predictable way.
Armin, if we provide some patches (hopefully with a solution, but at 
least with some additional logging), would you be able to compile and 
verify it?

Regards,

Zoltan Kiss

On 14/07/14 03:25, Armin Zentai wrote:
> Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb:
> Trying to unmap invalid handle! pending_idx: c
> Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here ]------------
> Jul 13 00:46:58 node11 [157060.106546] kernel BUG at
> drivers/net/xen-netback/netback.c:998!
> Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]

> Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0
> Tainted: G            E 3.15.4 #1
> Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro
> X8DTL/X8DTL, BIOS 1.1b    03/19/2010
> Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti:
> ffffffff81c00000 task.ti: ffffffff81c00000
> Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
> Jul 13 00:46:58 node11  [<ffffffffa025f61d>]
> xenvif_idx_unmap+0x11d/0x130 [xen_netback]
> Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48
> EFLAGS: 00010292
> Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX:
> 000000000000000c RCX: 0000000000000000
> Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI:
> ffff88008ea03a18 RDI: 000000000000021b
> Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08:
> 0000000000000000 R09: ffff88008a40b600
> Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11:
> 00000000000006d8 R12: ffff8800569708c0
> Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14:
> ffff8800569708c0 R15: 0000000000000001
> Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000)
> GS:ffff88008ea00000(0000) knlGS:0000000000000000
> Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0:
> 000000008005003b
> Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3:
> 00000000049f3000 CR4: 0000000000002660
> Jul 13 00:46:58 node11 [157060.113835] Stack:
> Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
> Jul 13 00:46:58 node11  ff00000000000001
> Jul 13 00:46:58 node11  ffff880b0605e000
> Jul 13 00:46:58 node11  0000000000000000
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
> Jul 13 00:46:58 node11  00000000fffffff6
> Jul 13 00:46:58 node11  0000000000000001
> Jul 13 00:46:58 node11  ffff8800569769d0
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
> Jul 13 00:46:58 node11  ffffffffa02622fc
> Jul 13 00:46:58 node11  ffff88008ea03dd8
> Jul 13 00:46:58 node11  ffffffff810b5223
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114637] Call Trace:
> Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.114750]
> Jul 13 00:46:58 node11  [<ffffffffa02622fc>]
> xenvif_tx_action+0x27c/0x7f0 [xen_netback]
> Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ?
> __wake_up+0x53/0x70
> Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ?
> handle_irq_event_percpu+0xa7/0x1b0
> Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>]
> xenvif_poll+0x31/0x64 [xen_netback]
> Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>]
> net_rx_action+0x10b/0x290
> Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>]
> __do_softirq+0x103/0x320
> Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>]
> irq_exit+0x135/0x140
> Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>]
> xen_evtchn_do_upcall+0x3c/0x50
> Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>]
> xen_do_hypervisor_callback+0x1e/0x30
> Jul 13 00:46:58 node11 [157060.115506]  <EOI>
> Jul 13 00:46:58 node11
> Jul 13 00:46:58 node11 [157060.115551]
> Jul 13 00:46:58 node11  [<ffffffff810013aa>] ?
> xen_hypercall_sched_op+0xa/0x20
> Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ?
> xen_hypercall_sched_op+0xa/0x20
> Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ?
> xen_safe_halt+0x10/0x20
> Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ?
> default_idle+0x1f/0xc0
> Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ?
> arch_cpu_idle+0xf/0x20
> Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ?
> cpu_startup_entry+0x201/0x360
> Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ?
> rest_init+0x77/0x80
> Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ?
> start_kernel+0x406/0x413
> Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ?
> repair_env_string+0x5b/0x5b
> Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ?
> x86_64_start_reservations+0x2a/0x2c
> Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ?
> xen_start_kernel+0x584/0x586

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

I forgot to mention this: Armin, can you set the loglevel to DBG, so we 
can see the grant op failure messages from xenvif_tx_check_gop next time?

Regards,

Zoltan

On 14/07/14 12:54, Zoltan Kiss wrote:
> Hi,
>
> Based on the stack trace I think the guest sends packet with more than
> MAX_SKB_SLOTS slots, and one slot's grant mapping operation goes wrong.
> Then the error handling screws up the internal bookkeeping of pending
> slots, and tries to unmap something that is not mapped. Unfortunately it
> is quite complicated to emulate grant mapping failures in a predictable
> way.
> Armin, if we provide some patches (hopefully with a solution, but at
> least with some additional logging), would you be able to compile and
> verify it?
>
> Regards,
>
> Zoltan Kiss
>
> On 14/07/14 03:25, Armin Zentai wrote:
>> Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb:
>> Trying to unmap invalid handle! pending_idx: c
>> Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here
>> ]------------
>> Jul 13 00:46:58 node11 [157060.106546] kernel BUG at
>> drivers/net/xen-netback/netback.c:998!
>> Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]
>
>> Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0
>> Tainted: G            E 3.15.4 #1
>> Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro
>> X8DTL/X8DTL, BIOS 1.1b    03/19/2010
>> Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti:
>> ffffffff81c00000 task.ti: ffffffff81c00000
>> Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
>> Jul 13 00:46:58 node11  [<ffffffffa025f61d>]
>> xenvif_idx_unmap+0x11d/0x130 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48
>> EFLAGS: 00010292
>> Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX:
>> 000000000000000c RCX: 0000000000000000
>> Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI:
>> ffff88008ea03a18 RDI: 000000000000021b
>> Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08:
>> 0000000000000000 R09: ffff88008a40b600
>> Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11:
>> 00000000000006d8 R12: ffff8800569708c0
>> Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14:
>> ffff8800569708c0 R15: 0000000000000001
>> Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000)
>> GS:ffff88008ea00000(0000) knlGS:0000000000000000
>> Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0:
>> 000000008005003b
>> Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3:
>> 00000000049f3000 CR4: 0000000000002660
>> Jul 13 00:46:58 node11 [157060.113835] Stack:
>> Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
>> Jul 13 00:46:58 node11  ff00000000000001
>> Jul 13 00:46:58 node11  ffff880b0605e000
>> Jul 13 00:46:58 node11  0000000000000000
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
>> Jul 13 00:46:58 node11  00000000fffffff6
>> Jul 13 00:46:58 node11  0000000000000001
>> Jul 13 00:46:58 node11  ffff8800569769d0
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
>> Jul 13 00:46:58 node11  ffffffffa02622fc
>> Jul 13 00:46:58 node11  ffff88008ea03dd8
>> Jul 13 00:46:58 node11  ffffffff810b5223
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114637] Call Trace:
>> Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114750]
>> Jul 13 00:46:58 node11  [<ffffffffa02622fc>]
>> xenvif_tx_action+0x27c/0x7f0 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ?
>> __wake_up+0x53/0x70
>> Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ?
>> handle_irq_event_percpu+0xa7/0x1b0
>> Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>]
>> xenvif_poll+0x31/0x64 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>]
>> net_rx_action+0x10b/0x290
>> Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>]
>> __do_softirq+0x103/0x320
>> Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>]
>> irq_exit+0x135/0x140
>> Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>]
>> xen_evtchn_do_upcall+0x3c/0x50
>> Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>]
>> xen_do_hypervisor_callback+0x1e/0x30
>> Jul 13 00:46:58 node11 [157060.115506]  <EOI>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.115551]
>> Jul 13 00:46:58 node11  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ?
>> xen_safe_halt+0x10/0x20
>> Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ?
>> default_idle+0x1f/0xc0
>> Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ?
>> arch_cpu_idle+0xf/0x20
>> Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ?
>> cpu_startup_entry+0x201/0x360
>> Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ?
>> rest_init+0x77/0x80
>> Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ?
>> start_kernel+0x406/0x413
>> Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ?
>> repair_env_string+0x5b/0x5b
>> Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ?
>> x86_64_start_reservations+0x2a/0x2c
>> Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ?
>> xen_start_kernel+0x584/0x586
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Armin Zentai]

Hi!

On 14/07/14 13:54, Zoltan Kiss wrote:
> Hi,
>
> Based on the stack trace I think the guest sends packet with more than
> MAX_SKB_SLOTS slots, and one slot's grant mapping operation goes wrong.
> Then the error handling screws up the internal bookkeeping of pending
> slots, and tries to unmap something that is not mapped. Unfortunately it
> is quite complicated to emulate grant mapping failures in a predictable
> way.
> Armin, if we provide some patches (hopefully with a solution, but at
> least with some additional logging), would you be able to compile and
> verify it?

I can compile and install it, the verification may take a few days. 
We're building up a test system, and try to test with outgoing traffic. 
If you have any advice, about how to test the new kernel, please share 
with me (I mean large pps, or large bandwidth is the better, or try to 
update/downgrade the GPLPV drivers to a specific version).

I'm setting the loglevel to DBG.

>
> Regards,
>
> Zoltan Kiss
>
> On 14/07/14 03:25, Armin Zentai wrote:
>> Jul 13 00:46:58 node11 [157060.106323] vif vif-2-0 h14z4mzbvfrrhb:
>> Trying to unmap invalid handle! pending_idx: c
>> Jul 13 00:46:58 node11 [157060.106476] ------------[ cut here
>> ]------------
>> Jul 13 00:46:58 node11 [157060.106546] kernel BUG at
>> drivers/net/xen-netback/netback.c:998!
>> Jul 13 00:46:58 node11 [157060.106616] invalid opcode: 0000 [#1]
>
>> Jul 13 00:46:58 node11 [157060.112705] CPU: 0 PID: 0 Comm: swapper/0
>> Tainted: G            E 3.15.4 #1
>> Jul 13 00:46:58 node11 [157060.112776] Hardware name: Supermicro
>> X8DTL/X8DTL, BIOS 1.1b    03/19/2010
>> Jul 13 00:46:58 node11 [157060.112848] task: ffffffff81c1b480 ti:
>> ffffffff81c00000 task.ti: ffffffff81c00000
>> Jul 13 00:46:58 node11 [157060.112936] RIP: e030:[<ffffffffa025f61d>]
>> Jul 13 00:46:58 node11  [<ffffffffa025f61d>]
>> xenvif_idx_unmap+0x11d/0x130 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.113078] RSP: e02b:ffff88008ea03d48
>> EFLAGS: 00010292
>> Jul 13 00:46:58 node11 [157060.113147] RAX: 000000000000004a RBX:
>> 000000000000000c RCX: 0000000000000000
>> Jul 13 00:46:58 node11 [157060.113234] RDX: ffff88008a40b600 RSI:
>> ffff88008ea03a18 RDI: 000000000000021b
>> Jul 13 00:46:58 node11 [157060.113321] RBP: ffff88008ea03d88 R08:
>> 0000000000000000 R09: ffff88008a40b600
>> Jul 13 00:46:58 node11 [157060.113408] R10: ffff88008a0004e8 R11:
>> 00000000000006d8 R12: ffff8800569708c0
>> Jul 13 00:46:58 node11 [157060.113495] R13: ffff88006558fec0 R14:
>> ffff8800569708c0 R15: 0000000000000001
>> Jul 13 00:46:58 node11 [157060.113589] FS:  00007f351684b700(0000)
>> GS:ffff88008ea00000(0000) knlGS:0000000000000000
>> Jul 13 00:46:58 node11 [157060.113679] CS:  e033 DS: 0000 ES: 0000 CR0:
>> 000000008005003b
>> Jul 13 00:46:58 node11 [157060.113747] CR2: 00007fc2a4372000 CR3:
>> 00000000049f3000 CR4: 0000000000002660
>> Jul 13 00:46:58 node11 [157060.113835] Stack:
>> Jul 13 00:46:58 node11 [157060.113896]  ffff880056979f90
>> Jul 13 00:46:58 node11  ff00000000000001
>> Jul 13 00:46:58 node11  ffff880b0605e000
>> Jul 13 00:46:58 node11  0000000000000000
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114143]  ffff0000ffffffff
>> Jul 13 00:46:58 node11  00000000fffffff6
>> Jul 13 00:46:58 node11  0000000000000001
>> Jul 13 00:46:58 node11  ffff8800569769d0
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114390]  ffff88008ea03e58
>> Jul 13 00:46:58 node11  ffffffffa02622fc
>> Jul 13 00:46:58 node11  ffff88008ea03dd8
>> Jul 13 00:46:58 node11  ffffffff810b5223
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114637] Call Trace:
>> Jul 13 00:46:58 node11 [157060.114700]  <IRQ>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.114750]
>> Jul 13 00:46:58 node11  [<ffffffffa02622fc>]
>> xenvif_tx_action+0x27c/0x7f0 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.114927]  [<ffffffff810b5223>] ?
>> __wake_up+0x53/0x70
>> Jul 13 00:46:58 node11 [157060.114998]  [<ffffffff810ca077>] ?
>> handle_irq_event_percpu+0xa7/0x1b0
>> Jul 13 00:46:58 node11 [157060.115073]  [<ffffffffa02647d1>]
>> xenvif_poll+0x31/0x64 [xen_netback]
>> Jul 13 00:46:58 node11 [157060.115147]  [<ffffffff81653d4b>]
>> net_rx_action+0x10b/0x290
>> Jul 13 00:46:58 node11 [157060.115221]  [<ffffffff81071c73>]
>> __do_softirq+0x103/0x320
>> Jul 13 00:46:58 node11 [157060.115292]  [<ffffffff81072015>]
>> irq_exit+0x135/0x140
>> Jul 13 00:46:58 node11 [157060.115363]  [<ffffffff8144759c>]
>> xen_evtchn_do_upcall+0x3c/0x50
>> Jul 13 00:46:58 node11 [157060.115436]  [<ffffffff8175c07e>]
>> xen_do_hypervisor_callback+0x1e/0x30
>> Jul 13 00:46:58 node11 [157060.115506]  <EOI>
>> Jul 13 00:46:58 node11
>> Jul 13 00:46:58 node11 [157060.115551]
>> Jul 13 00:46:58 node11  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115722]  [<ffffffff810013aa>] ?
>> xen_hypercall_sched_op+0xa/0x20
>> Jul 13 00:46:58 node11 [157060.115794]  [<ffffffff8100a200>] ?
>> xen_safe_halt+0x10/0x20
>> Jul 13 00:46:58 node11 [157060.115869]  [<ffffffff8101dbbf>] ?
>> default_idle+0x1f/0xc0
>> Jul 13 00:46:58 node11 [157060.115939]  [<ffffffff8101d38f>] ?
>> arch_cpu_idle+0xf/0x20
>> Jul 13 00:46:58 node11 [157060.116009]  [<ffffffff810b5aa1>] ?
>> cpu_startup_entry+0x201/0x360
>> Jul 13 00:46:58 node11 [157060.116084]  [<ffffffff817420a7>] ?
>> rest_init+0x77/0x80
>> Jul 13 00:46:58 node11 [157060.116156]  [<ffffffff81d3a156>] ?
>> start_kernel+0x406/0x413
>> Jul 13 00:46:58 node11 [157060.116227]  [<ffffffff81d39b6e>] ?
>> repair_env_string+0x5b/0x5b
>> Jul 13 00:46:58 node11 [157060.116298]  [<ffffffff81d39603>] ?
>> x86_64_start_reservations+0x2a/0x2c
>> Jul 13 00:46:58 node11 [157060.116373]  [<ffffffff81d3d5dc>] ?
>> xen_start_kernel+0x584/0x586
>


Thanks,
  - Armin

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

Hi,

I've already identified 2 issues in xenvif_tx_check_gop:

- (*gopp_copy)++ happens before we print the error message, though its 
only effect that the error message becomes incorrect
- a more severe issue is that if there is a mapping problem on the frags 
(or on the frag_list skb), AND the first TX request is entirely grant 
copied ( !(data_len < txreq.size) in build_gops), that pending_idx is 
never released. The frags are released but then the function returns, 
skb is freed and the while loop goes to next iteration. In the normal 
path it should be released a bit later in that loop, at the "Schedule a 
response immediately" comment.

However, I don't see yet how this second issue should cause this problem.

Zoltan

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

On 14/07/14 13:14, Armin Zentai wrote:
> Hi!
>
> On 14/07/14 13:54, Zoltan Kiss wrote:
>> Hi,
>>
>> Based on the stack trace I think the guest sends packet with more than
>> MAX_SKB_SLOTS slots, and one slot's grant mapping operation goes wrong.
>> Then the error handling screws up the internal bookkeeping of pending
>> slots, and tries to unmap something that is not mapped. Unfortunately it
>> is quite complicated to emulate grant mapping failures in a predictable
>> way.
>> Armin, if we provide some patches (hopefully with a solution, but at
>> least with some additional logging), would you be able to compile and
>> verify it?
>
> I can compile and install it, the verification may take a few days.
> We're building up a test system, and try to test with outgoing traffic.
> If you have any advice, about how to test the new kernel, please share
> with me (I mean large pps, or large bandwidth is the better, or try to
> update/downgrade the GPLPV drivers to a specific version).
>
> I'm setting the loglevel to DBG.

Also, if you have your vmlinux file, can you please run the following:

addr2line -e <your vmlinux file> ffffffffa02622fc

That would hopefully point out where xenvif_idx_unmap were called in 
check_gop, as it has 3 call sites.

Zoli

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Armin Zentai]

Hi!

On 14/07/14 17:30, Zoltan Kiss wrote:
> On 14/07/14 13:14, Armin Zentai wrote:
>> Hi!
>>
>> On 14/07/14 13:54, Zoltan Kiss wrote:
>>> Hi,
>>>
>>> Based on the stack trace I think the guest sends packet with more than
>>> MAX_SKB_SLOTS slots, and one slot's grant mapping operation goes wrong.
>>> Then the error handling screws up the internal bookkeeping of pending
>>> slots, and tries to unmap something that is not mapped. Unfortunately it
>>> is quite complicated to emulate grant mapping failures in a predictable
>>> way.
>>> Armin, if we provide some patches (hopefully with a solution, but at
>>> least with some additional logging), would you be able to compile and
>>> verify it?
>>
>> I can compile and install it, the verification may take a few days.
>> We're building up a test system, and try to test with outgoing traffic.
>> If you have any advice, about how to test the new kernel, please share
>> with me (I mean large pps, or large bandwidth is the better, or try to
>> update/downgrade the GPLPV drivers to a specific version).
>>
>> I'm setting the loglevel to DBG.
>
> Also, if you have your vmlinux file, can you please run the following:
>
> addr2line -e <your vmlinux file> ffffffffa02622fc
>
> That would hopefully point out where xenvif_idx_unmap were called in
> check_gop, as it has 3 call sites.
>

I've run it on the vmlinux file, that I've made with make vmlinux.

[root@c2-node16 linux-3.15.4]# addr2line -e vmlinux ffffffffa02622fc
??:0

I think it does not found anything.

At the time we've found one VM that crashes its hypervisor in every 
evening at the same time, this virtual machine is belongs to one of our 
clients, but I'll try to get some info about a scheduled job or 
something that starts in that time.

> Zoli
>

Thanks,
  - Armin

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Wei Liu]

On Mon, Jul 14, 2014 at 11:15:00PM +0200, Armin Zentai wrote:
[...]
> I've run it on the vmlinux file, that I've made with make vmlinux.
> 
> [root@c2-node16 linux-3.15.4]# addr2line -e vmlinux ffffffffa02622fc
> ??:0
> 
> I think it does not found anything.
> 

AFAICT your netback is a loadable kernel module. Addr2line won't work...

Use gdb instead

# gdb PATH_TO/xen-netback.ko
(gdb) list *(xenvif_tx_action+0x27c)


> At the time we've found one VM that crashes its hypervisor in every evening
> at the same time, this virtual machine is belongs to one of our clients, but
> I'll try to get some info about a scheduled job or something that starts in
> that time.
> 

"Every evening at the same time", how exciting...

Wei.

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

Hi,

I've just submitted 2 patch series to xen-devel, one is for the net tree 
(with multiqueue) and one for 3.15. I believe the first two patch in the 
series should address this problem, the another two is for related 
problems. Armin, can you test all four of them together?

Thanks,

Zoltan

p.s.: if you have the test setup running with increased loglevel, it 
would be good to see the result of that, as it can confirm that it is 
the problem I've solved in those patches

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Armin Zentai]

Hi!

On 17/07/14 21:17, Zoltan Kiss wrote:
> Hi,
>
> I've just submitted 2 patch series to xen-devel, one is for the net tree
> (with multiqueue) and one for 3.15. I believe the first two patch in the
> series should address this problem, the another two is for related
> problems. Armin, can you test all four of them together?
>
> Thanks,
>
> Zoltan
>
> p.s.: if you have the test setup running with increased loglevel, it
> would be good to see the result of that, as it can confirm that it is
> the problem I've solved in those patches

I'll compile the patches at next week, and I'll post the test results.

It's a live service, not a developers' playground, so its make the 
testing harder. We've moved back to 3.10.43-11 from the Xen4CentOS 
repository, instead of using a self-compiled kernel. The problem has 
been solved (I think), no crash over 5 days.

BEfore we're moving back the problematic VMs to the old kernel, we've 
cloned 2 of them, and running them on a HV, that runs the 3.15 kernel, 
with debug logging enabled. These VMs produced only one HV crash after 
the cloning, but we had no usable logs. It happened at night, and we 
were contiously getting very high load alerts from the HV, and after 
some minutes we've lost the connection to it, and to every VM on it. 
We've check it via KVM, and we found it continously posting the Draining 
TX queue message for all vifs. We only can do a hard reset on it, the 
logs were remain intact, no releated log entry have found in them. The 
netconsole module remained useless too, no messages has been forwarded 
after the _crash_.

So I'll apply your patches, and wait a week or two.

Re: Trying to unmap invalid handle! pending_idx: @ drivers/net/xen-netback/netback.c:998 causes kernel panic/reboot

[Zoltan Kiss]

On 20/07/14 18:20, Armin Zentai wrote:
> Hi!
>
> On 17/07/14 21:17, Zoltan Kiss wrote:
>> Hi,
>>
>> I've just submitted 2 patch series to xen-devel, one is for the net tree
>> (with multiqueue) and one for 3.15. I believe the first two patch in the
>> series should address this problem, the another two is for related
>> problems. Armin, can you test all four of them together?
>>
>> Thanks,
>>
>> Zoltan
>>
>> p.s.: if you have the test setup running with increased loglevel, it
>> would be good to see the result of that, as it can confirm that it is
>> the problem I've solved in those patches
>
> I'll compile the patches at next week, and I'll post the test results.
>
> It's a live service, not a developers' playground, so its make the
> testing harder. We've moved back to 3.10.43-11 from the Xen4CentOS
> repository, instead of using a self-compiled kernel. The problem has
> been solved (I think), no crash over 5 days.
>
> BEfore we're moving back the problematic VMs to the old kernel, we've
> cloned 2 of them, and running them on a HV, that runs the 3.15 kernel,
> with debug logging enabled. These VMs produced only one HV crash after
> the cloning, but we had no usable logs. It happened at night, and we
> were contiously getting very high load alerts from the HV, and after
> some minutes we've lost the connection to it, and to every VM on it.
> We've check it via KVM, and we found it continously posting the Draining
> TX queue message for all vifs. We only can do a hard reset on it, the
> logs were remain intact, no releated log entry have found in them. The
> netconsole module remained useless too, no messages has been forwarded
> after the _crash_.
That seems to be a different issue, might be not even netback related. 
The "Draining ..." messages means netback is still alive and purging the 
VMs queue out as it doesn't empty its ring at all. This usually means 
the guest OS is crashed in some way but haven't rebooted, so the domain 
is still there. Can you check the logs from the VM during the crash time?
If something similar happens next time, I recommend to check top/xentop, 
and the console of the guest.
>
> So I'll apply your patches, and wait a week or two.
>