netback oops in Linux 3.14 / 3.15

netback oops in Linux 3.14 / 3.15

[James Dingwall]

[-- Attachment #1: Type: text/plain, Size: 718 bytes --]

Hi,

Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in xenvif_rx_action().  Under 3.15 it is the line at 
629 which is triggering:

BUG_ON(ring_slots_used > max_slots_needed);

The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running the same 3.15.0 kernel, an rsync in the guest 
where the source filesystem is NFS mounted from the dom0.

I have added a printk to log the values which is seen as the first line in the attached trace.  Is there an off-by-one 
error in the calculation for max_slots_needed or is it worth adding some more printks to find out how the value was 
calculated?

Please let me know if any other information about my configuration is required.

Thanks,
James

[-- Attachment #2: netback-3.15.0.ooops --]
[-- Type: text/plain, Size: 3712 bytes --]

[  485.105896] ring_slots_used (19) > max_slots_needed (18)
[  485.105976] ------------[ cut here ]------------
[  485.107251] kernel BUG at drivers/net/xen-netback/netback.c:632!
[  485.108571] invalid opcode: 0000 [#1] SMP 
[  485.109849] Modules linked in: iptable_filter ip_tables x_tables lm85 dme1737 hwmon_vid autofs4 nfsd xen_gntalloc bridge stp llc ipv6 nls_iso8859_1 ipmi_si ipmi_devintf ipmi_msghandler openvswitch vxlan tun tmem xen_acpi_processor xen_gntdev xen_pciback xen_blkback xen_netback radeon fbcon bitblit softcursor font tileblit ttm drm_kms_helper drm microcode evdev radeonfb fb_ddc i2c_algo_bit e1000e cfbfillrect cfbimgblt k10temp cfbcopyarea backlight fb fbdev rtc_cmos floppy button i2c_nforce2 processor forcedeth ohci_pci i2c_core thermal_sys ata_generic dm_zero dm_thin_pool dm_persistent_data dm_bio_prison xts lrw gf128mul glue_helper ablk_helper cryptd aes_x86_64 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi tg3 ptp pps_core libphy hwmon e1000 fuse btrfs ext4 jbd2 linear raid0 dm_r
 aid raid1
[  485.121413]  raid10 dm_snapshot dm_bufio dm_crypt dm_mirror dm_region_hash dm_log firewire_core hid_sunplus hid_samsung hid_pl hid_petalynx hid_gyration usbhid ohci_hcd uhci_hcd usb_storage ehci_pci ehci_hcd megaraid_sas 3w_xxxx qla1280 aic7xxx scsi_transport_spi sr_mod cdrom sg ahci libahci sata_nv sata_sil pata_amd libata
[  485.127787] CPU: 1 PID: 8164 Comm: kdc0.0-guest-rx Not tainted 3.15.0 #103
[  485.129470] Hardware name: empty empty/S2932/S2932-E/S2932-SI, BIOS 'V2.06     ' 10/22/2009
[  485.131247] task: ffff880117394ac0 ti: ffff8800bf56c000 task.ti: ffff8800bf56c000
[  485.132989] RIP: e030:[<ffffffffa0815eb5>]  [<ffffffffa0815eb5>] xenvif_rx_action+0x496/0x830 [xen_netback]
[  485.134937] RSP: e02b:ffff8800bf56fd78  EFLAGS: 00010292
[  485.136621] RAX: 000000000000002c RBX: 0000000000000012 RCX: 0000000000000001
[  485.138434] RDX: ffff880126e2e8c8 RSI: ffff880126e2df78 RDI: ffff8800bf560238
[  485.140285] RBP: ffff8800bf56fe58 R08: 0000000000000001 R09: 0000000000000000
[  485.142147] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8800c8e88ee0
[  485.144041] R13: 0000000000000011 R14: 0000000000000000 R15: ffff8800c0a50800
[  485.145963] FS:  00007f55125c8740(0000) GS:ffff880126e20000(0000) knlGS:0000000000000000
[  485.147958] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[  485.149830] CR2: 0000000001760000 CR3: 00000000b9553000 CR4: 0000000000000660
[  485.151865] Stack:
[  485.153594]  ffff8800bf56fdd4 0000000000000000 ffff8800ffffffff ffffffff814ba418
[  485.155769]  ffff88011e72b230 000000111e72b85a ffff8800c0a5b148 ffff8800c0a50800
[  485.157911]  000000000000183b 0000000000000000 ffff8800bf56fdf8 000000001e72b230
[  485.160098] Call Trace:
[  485.161900]  [<ffffffff814ba418>] ? _raw_spin_unlock_irqrestore+0x47/0x62
[  485.163984]  [<ffffffffa081789a>] xenvif_kthread_guest_rx+0x12b/0x1ea [xen_netback]
[  485.166000]  [<ffffffff810753fc>] ? __wake_up_sync+0xd/0xd
[  485.168024]  [<ffffffffa081776f>] ? xenvif_stop_queue+0x53/0x53 [xen_netback]
[  485.170158]  [<ffffffff8105e1e0>] kthread+0xcd/0xd5
[  485.172166]  [<ffffffff8105e113>] ? kthread_freezable_should_stop+0x43/0x43
[  485.174323]  [<ffffffff814baf8c>] ret_from_fork+0x7c/0xb0
[  485.176385]  [<ffffffff8105e113>] ? kthread_freezable_should_stop+0x43/0x43
[  485.178676] Code: ff 41 89 44 24 28 41 8b b7 34 a9 00 00 2b b5 60 ff ff ff 39 de 0f 86 78 03 00 00 89 da 48 c7 c7 11 9d 81 a0 31 c0 e8 34 d1 c9 e0 <0f> 0b 48 8b bd 50 ff ff ff e8 8d b9 be e0 48 85 c0 49 89 c4 0f 
[  485.186512] RIP  [<ffffffffa0815eb5>] xenvif_rx_action+0x496/0x830 [xen_netback]
[  485.189164]  RSP <ffff8800bf56fd78>
[  485.191614] ---[ end trace f396ff56268b095a ]---

[-- Attachment #3: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: netback oops in Linux 3.14 / 3.15

[Zoltan Kiss]

Hi,

It is very likely to be a recent issue, please apply this fix onto your 
kernel:

http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee

Zoli

On 10/06/14 19:20, James Dingwall wrote:
> Hi,
>
> Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in xenvif_rx_action().  Under 3.15 it is the line at
> 629 which is triggering:
>
> BUG_ON(ring_slots_used > max_slots_needed);
>
> The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running the same 3.15.0 kernel, an rsync in the guest
> where the source filesystem is NFS mounted from the dom0.
>
> I have added a printk to log the values which is seen as the first line in the attached trace.  Is there an off-by-one
> error in the calculation for max_slots_needed or is it worth adding some more printks to find out how the value was
> calculated?
>
> Please let me know if any other information about my configuration is required.
>
> Thanks,
> James
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
>

Re: netback oops in Linux 3.14 / 3.15

[James Dingwall]

On Tue, Jun 10, 2014 at 07:42:57PM +0100, Zoltan Kiss wrote:
> It is very likely to be a recent issue, please apply this fix onto your 
> kernel:
> 
> http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee
> 
> Zoli

Thanks for the pointer.  I have tested with 3.15 and it seems to have solved the issue I was experiencing.

James

> 
> On 10/06/14 19:20, James Dingwall wrote:
> > Hi,
> >
> > Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in xenvif_rx_action().  Under 3.15 it is the line at
> > 629 which is triggering:
> >
> > BUG_ON(ring_slots_used > max_slots_needed);
> >
> > The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running the same 3.15.0 kernel, an rsync in the guest
> > where the source filesystem is NFS mounted from the dom0.
> >
> > I have added a printk to log the values which is seen as the first line in the attached trace.  Is there an off-by-one
> > error in the calculation for max_slots_needed or is it worth adding some more printks to find out how the value was
> > calculated?
> >
> > Please let me know if any other information about my configuration is required.
> >
> > Thanks,
> > James
> >
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > http://lists.xen.org/xen-devel
> >
> 

-- 
------------------------------------------------------------------------
James Dingwall
e: james@dingwall.me.uk
w: http://www.dingwall.me.uk/
------------------------------------------------------------------------

Re: netback oops in Linux 3.14 / 3.15

[Dennis Lan (dlan)]

Hello zoli:
 should we back port this to longterm maintain kernel? current the
most recent version is 3.14.17

Lan

On Wed, Jun 11, 2014 at 2:42 AM, Zoltan Kiss <zoltan.kiss@citrix.com> wrote:
> Hi,
>
> It is very likely to be a recent issue, please apply this fix onto your
> kernel:
>
> http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee
>
> Zoli
>
>
> On 10/06/14 19:20, James Dingwall wrote:
>>
>> Hi,
>>
>> Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in
>> xenvif_rx_action().  Under 3.15 it is the line at
>> 629 which is triggering:
>>
>> BUG_ON(ring_slots_used > max_slots_needed);
>>
>> The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running
>> the same 3.15.0 kernel, an rsync in the guest
>> where the source filesystem is NFS mounted from the dom0.
>>
>> I have added a printk to log the values which is seen as the first line in
>> the attached trace.  Is there an off-by-one
>> error in the calculation for max_slots_needed or is it worth adding some
>> more printks to find out how the value was
>> calculated?
>>
>> Please let me know if any other information about my configuration is
>> required.
>>
>> Thanks,
>> James
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: netback oops in Linux 3.14 / 3.15

[Zoltan Kiss]

Yes, it does worth to backport if the patch referred in the commit 
message is there.

Zoli

On 19/08/14 04:30, Dennis Lan (dlan) wrote:
> Hello zoli:
>   should we back port this to longterm maintain kernel? current the
> most recent version is 3.14.17
>
> Lan
>
> On Wed, Jun 11, 2014 at 2:42 AM, Zoltan Kiss <zoltan.kiss@citrix.com> wrote:
>> Hi,
>>
>> It is very likely to be a recent issue, please apply this fix onto your
>> kernel:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee
>>
>> Zoli
>>
>>
>> On 10/06/14 19:20, James Dingwall wrote:
>>> Hi,
>>>
>>> Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in
>>> xenvif_rx_action().  Under 3.15 it is the line at
>>> 629 which is triggering:
>>>
>>> BUG_ON(ring_slots_used > max_slots_needed);
>>>
>>> The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running
>>> the same 3.15.0 kernel, an rsync in the guest
>>> where the source filesystem is NFS mounted from the dom0.
>>>
>>> I have added a printk to log the values which is seen as the first line in
>>> the attached trace.  Is there an off-by-one
>>> error in the calculation for max_slots_needed or is it worth adding some
>>> more printks to find out how the value was
>>> calculated?
>>>
>>> Please let me know if any other information about my configuration is
>>> required.
>>>
>>> Thanks,
>>> James
>>>
>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel

Re: netback oops in Linux 3.14 / 3.15

[Dennis Lan (dlan)]

hi zoli:
  Can you help push this patch to stable kernel? eg, CC
stable@kerne.org, so greg-kh may include in the next release.
  Thanks.

Lan

On Wed, Aug 20, 2014 at 3:26 AM, Zoltan Kiss <zoltan.kiss@schaman.hu> wrote:
> Yes, it does worth to backport if the patch referred in the commit message
> is there.
>
> Zoli
>
>
> On 19/08/14 04:30, Dennis Lan (dlan) wrote:
>>
>> Hello zoli:
>>   should we back port this to longterm maintain kernel? current the
>> most recent version is 3.14.17
>>
>> Lan
>>
>> On Wed, Jun 11, 2014 at 2:42 AM, Zoltan Kiss <zoltan.kiss@citrix.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> It is very likely to be a recent issue, please apply this fix onto your
>>> kernel:
>>>
>>>
>>> http://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/drivers/net/xen-netback?id=59ae9fc67007da8b5aea7b0a31c3607745cfbfee
>>>
>>> Zoli
>>>
>>>
>>> On 10/06/14 19:20, James Dingwall wrote:
>>>>
>>>> Hi,
>>>>
>>>> Under Linux 3.14.6 and 3.15 I am getting a BUG_ON() from netback in
>>>> xenvif_rx_action().  Under 3.15 it is the line at
>>>> 629 which is triggering:
>>>>
>>>> BUG_ON(ring_slots_used > max_slots_needed);
>>>>
>>>> The conditions are a 3.15.0 dom0 with Xen 4.4.0, a guest domain running
>>>> the same 3.15.0 kernel, an rsync in the guest
>>>> where the source filesystem is NFS mounted from the dom0.
>>>>
>>>> I have added a printk to log the values which is seen as the first line
>>>> in
>>>> the attached trace.  Is there an off-by-one
>>>> error in the calculation for max_slots_needed or is it worth adding some
>>>> more printks to find out how the value was
>>>> calculated?
>>>>
>>>> Please let me know if any other information about my configuration is
>>>> required.
>>>>
>>>> Thanks,
>>>> James
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>>>>
>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>
>

Re: netback oops in Linux 3.14 / 3.15

[Wei Liu]

Please don't top post.

On Fri, Sep 19, 2014 at 04:44:39PM +0800, Dennis Lan (dlan) wrote:
> hi zoli:
>   Can you help push this patch to stable kernel? eg, CC
> stable@kerne.org, so greg-kh may include in the next release.
>   Thanks.
> 

The said patch was already tagged for "net" tree and can be found in
Greg's stable tree. It's just not yet in 3.14 LTS kernel. If you want it
to be included for next 3.14.y release you can send backport request to
stable@kernel.org.

Wei.