blktap2 bug:Assertion 'list_empty(&vreq->next)' failed

All of lore.kernel.org
 help / color / mirror / Atom feed

* blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
@ 2010-06-28  4:41 tsk
  2010-06-28  4:48 ` tsk
  2010-06-28  4:49 ` [Xen-devel] " Daniel Stodden
  0 siblings, 2 replies; 7+ messages in thread
From: tsk @ 2010-06-28  4:41 UTC (permalink / raw)
  To: xen-devel, xen-users


[-- Attachment #1.1: Type: text/plain, Size: 2187 bytes --]

Hi folks,

Yestoday, I run a testcase, 6 VMs restart every 10 minitues from 11:00 am to
16:30 pm until Dom0 crashed.

Assertion 'list_empty(&vreq->next)' failed will lead to tapdisk2 process
segfault, then Dom0 crashed!

Jun 27 16:31:05 r02k05014 kernel: device tap775.0 entered promiscuous mode
Jun 27 16:31:05 r02k05014 kernel: eth0: port 13(tap775.0) entering
forwarding state
Jun 27 16:31:15 r02k05014 tapdisk2[4341]: Assertion
'list_empty(&vreq->next)' failed, line 1822, file tapdisk-vbd.c
Jun 27 16:31:15 r02k05014 kernel: tapdisk2[4341]: segfault at 0 ip
000000000040a24f sp 00007fff0d8acb70 error 6 in tapdisk2[400000+39000]
Jun 28 18:58:09 r02k05014 syslogd 1.4.1: restart.

Any one who can give me some tips?

What will lead to   Assertion 'list_empty(&vreq->next)' failed   ?

xm info:
release                : 2.6.31.13
version                : #3 SMP Fri Apr 30 15:10:24 CST 2010
machine                : x86_64
nr_cpus                : 16
nr_nodes               : 2
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                :
bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm
total_memory           : 24544
free_memory            : 19693
node_to_cpu            : node0:0,2,4,6,8,10,12,14
                         node1:1,3,5,7,9,11,13,15
node_to_memory         : node0:7633
                         node1:12059
node_to_dma32_mem      : node0:2996
                         node1:0
max_node_id            : 1
xen_major              : 4
xen_minor              : 0
xen_extra              : .0
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
xen_commandline        : console=com1,vga com1=115200,8n1 msi=1
dom0_mem=6144M  dom0_max_vcpus=4  dom0_vcpus_pin iommu=off x2apic=off hap=0
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
cc_compile_by          : root
cc_compile_date        : Wed May 12 19:09:47 CST 2010
xend_config_format     : 4

tsk

[-- Attachment #1.2: Type: text/html, Size: 2831 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
  2010-06-28  4:41 blktap2 bug:Assertion 'list_empty(&vreq->next)' failed tsk
@ 2010-06-28  4:48 ` tsk
  2010-06-28  4:49 ` [Xen-devel] " Daniel Stodden
  1 sibling, 0 replies; 7+ messages in thread
From: tsk @ 2010-06-28  4:48 UTC (permalink / raw)
  To: xen-devel, xen-users


[-- Attachment #1.1: Type: text/plain, Size: 2518 bytes --]

If a tapdisk2 process segfault, the Dom0 will crash. So the robustness of
tapdisk2 process and blktap2 driver is very very important!

Is there any version to solve this problem?

2010/6/28 tsk <aixt2006@gmail.com>

> Hi folks,
>
> Yestoday, I run a testcase, 6 VMs restart every 10 minitues from 11:00 am
> to 16:30 pm until Dom0 crashed.
>
> Assertion 'list_empty(&vreq->next)' failed will lead to tapdisk2 process
> segfault, then Dom0 crashed!
>
> Jun 27 16:31:05 r02k05014 kernel: device tap775.0 entered promiscuous mode
> Jun 27 16:31:05 r02k05014 kernel: eth0: port 13(tap775.0) entering
> forwarding state
> Jun 27 16:31:15 r02k05014 tapdisk2[4341]: Assertion
> 'list_empty(&vreq->next)' failed, line 1822, file tapdisk-vbd.c
> Jun 27 16:31:15 r02k05014 kernel: tapdisk2[4341]: segfault at 0 ip
> 000000000040a24f sp 00007fff0d8acb70 error 6 in tapdisk2[400000+39000]
> Jun 28 18:58:09 r02k05014 syslogd 1.4.1: restart.
>
> Any one who can give me some tips?
>
> What will lead to   Assertion 'list_empty(&vreq->next)' failed   ?
>
> xm info:
> release                : 2.6.31.13
> version                : #3 SMP Fri Apr 30 15:10:24 CST 2010
> machine                : x86_64
> nr_cpus                : 16
> nr_nodes               : 2
> cores_per_socket       : 4
> threads_per_core       : 2
> cpu_mhz                : 2266
> hw_caps                :
> bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
> virt_caps              : hvm
> total_memory           : 24544
> free_memory            : 19693
> node_to_cpu            : node0:0,2,4,6,8,10,12,14
>                          node1:1,3,5,7,9,11,13,15
> node_to_memory         : node0:7633
>                          node1:12059
> node_to_dma32_mem      : node0:2996
>                          node1:0
> max_node_id            : 1
> xen_major              : 4
> xen_minor              : 0
> xen_extra              : .0
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : unavailable
> xen_commandline        : console=com1,vga com1=115200,8n1 msi=1
> dom0_mem=6144M  dom0_max_vcpus=4  dom0_vcpus_pin iommu=off x2apic=off hap=0
> cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)
> cc_compile_by          : root
> cc_compile_date        : Wed May 12 19:09:47 CST 2010
> xend_config_format     : 4
>
> tsk
>
>

[-- Attachment #1.2: Type: text/html, Size: 3356 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [Xen-devel] blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
  2010-06-28  4:41 blktap2 bug:Assertion 'list_empty(&vreq->next)' failed tsk
  2010-06-28  4:48 ` tsk
@ 2010-06-28  4:49 ` Daniel Stodden
  2010-06-28  4:58   ` tsk
  1 sibling, 1 reply; 7+ messages in thread
From: Daniel Stodden @ 2010-06-28  4:49 UTC (permalink / raw)
  To: tsk; +Cc: xen-devel, xen-users

On Mon, 2010-06-28 at 00:41 -0400, tsk wrote:
> Hi folks,
> 
> 
> Yestoday, I run a testcase, 6 VMs restart every 10 minitues from 11:00
> am to 16:30 pm until Dom0 crashed.
>  
> Assertion 'list_empty(&vreq->next)' failed will lead to tapdisk2
> process segfault, then Dom0 crashed!
> 
> 
> Jun 27 16:31:05 r02k05014 kernel: device tap775.0 entered promiscuous
> mode
> Jun 27 16:31:05 r02k05014 kernel: eth0: port 13(tap775.0) entering
> forwarding state
> Jun 27 16:31:15 r02k05014 tapdisk2[4341]: Assertion
> 'list_empty(&vreq->next)' failed, line 1822, file tapdisk-vbd.c
> Jun 27 16:31:15 r02k05014 kernel: tapdisk2[4341]: segfault at 0 ip
> 000000000040a24f sp 00007fff0d8acb70 error 6 in tapdisk2[400000+39000]
> Jun 28 18:58:09 r02k05014 syslogd 1.4.1: restart.
> 
> 
> Any one who can give me some tips?  

Semi-yes. Not entirely sure about the list_empty userland crasher, but
the kernel stuff was upstreamed with some minor exposed while unmapping
pending I/O. 

Which I didn't care yet to send patches for...

... Soon, real soon now...

Daniel

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
  2010-06-28  4:49 ` [Xen-devel] " Daniel Stodden
@ 2010-06-28  4:58   ` tsk
       [not found]     ` <1277701807.9747.1905.camel@ramone.somacoma.net>
  0 siblings, 1 reply; 7+ messages in thread
From: tsk @ 2010-06-28  4:58 UTC (permalink / raw)
  To: Daniel Stodden; +Cc: xen-devel, xen-users


[-- Attachment #1.1: Type: text/plain, Size: 1374 bytes --]

It is found that some assertion was in the tapdisk2 source, if anyone
failed, the tapdisk2 process will exit,
They are very dangerous...
How can the kernel handle this if the tapdisk2 process exit?



2010/6/28 Daniel Stodden <daniel.stodden@citrix.com>

> On Mon, 2010-06-28 at 00:41 -0400, tsk wrote:
> > Hi folks,
> >
> >
> > Yestoday, I run a testcase, 6 VMs restart every 10 minitues from 11:00
> > am to 16:30 pm until Dom0 crashed.
> >
> > Assertion 'list_empty(&vreq->next)' failed will lead to tapdisk2
> > process segfault, then Dom0 crashed!
> >
> >
> > Jun 27 16:31:05 r02k05014 kernel: device tap775.0 entered promiscuous
> > mode
> > Jun 27 16:31:05 r02k05014 kernel: eth0: port 13(tap775.0) entering
> > forwarding state
> > Jun 27 16:31:15 r02k05014 tapdisk2[4341]: Assertion
> > 'list_empty(&vreq->next)' failed, line 1822, file tapdisk-vbd.c
> > Jun 27 16:31:15 r02k05014 kernel: tapdisk2[4341]: segfault at 0 ip
> > 000000000040a24f sp 00007fff0d8acb70 error 6 in tapdisk2[400000+39000]
> > Jun 28 18:58:09 r02k05014 syslogd 1.4.1: restart.
> >
> >
> > Any one who can give me some tips?
>
> Semi-yes. Not entirely sure about the list_empty userland crasher, but
> the kernel stuff was upstreamed with some minor exposed while unmapping
> pending I/O.
>
> Which I didn't care yet to send patches for...
>
> ... Soon, real soon now...
>
> Daniel
>
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 1939 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 7+ messages in thread

[parent not found: <1277701807.9747.1905.camel@ramone.somacoma.net>]

[parent not found: <AANLkTikcN8oX-LG8bhsa8CQJTjn1CLFvNuwkThMSb5yR@mail.gmail.com>]

* Re: blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
       [not found]       ` <AANLkTikcN8oX-LG8bhsa8CQJTjn1CLFvNuwkThMSb5yR@mail.gmail.com>
@ 2010-07-05 21:25         ` Daniel Stodden
  2010-07-05 22:26           ` Jeremy Fitzhardinge
  0 siblings, 1 reply; 7+ messages in thread
From: Daniel Stodden @ 2010-07-05 21:25 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel@lists.xensource.com, tsk

On Thu, 2010-07-01 at 08:55 -0400, tsk wrote:
> Hi, will the patches be added to Xen-4.1？

About to prepare the next bunch of patches for xen.git.

Jeremy, is it okay to pull some of the changes made for 2.6.3x into the
xen/dom0/backend/blktap2 topic branch before moving on?

I'm mainly talking about this one:

>From ab77527f9a63a5c657c6d6a50e70a66adceaa3a0 Mon Sep 17 00:00:00 2001
From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
Date: Thu, 18 Feb 2010 17:22:09 -0800
Subject: [PATCH] xen/blktap2: make compile

and a couple related deltas which came later.

Except moving forward to newer blk queue interfaces, they don't make
much of a functional difference.

Would simplify merging quite a bit.

Daniel

> 
> 2010/6/28 Daniel Stodden <daniel.stodden@citrix.com>
>         On Mon, 2010-06-28 at 00:58 -0400, tsk wrote:
>         > It is found that some assertion was in the tapdisk2 source,
>         if anyone
>         > failed, the tapdisk2 process will exit,
>         > They are very dangerous...
>         > How can the kernel handle this if the tapdisk2 process exit?
>         
>         
>         There would be multiple ways.
>         
>         One is to recover in-flight I/O and wait for a new tapdisk to
>         start,
>         then reissue. There's not a real usecase for this, but think
>         watchdog.
>         Maybe someday.
>         
>         The present strategy is to:
>          - Fail pending I/O.
>          - Restart the disk I/O queue, thereby flushing everything
>         submitted
>           to the block device with EIO as well.
>          - Release the block device after the last opener finally gave
>         up.
>         
>         Works well. At least from a dom0 perspective ;)
>         
>         Daniel
>         
>         
>         >
>         >
>         > 2010/6/28 Daniel Stodden <daniel.stodden@citrix.com>
>         >         On Mon, 2010-06-28 at 00:41 -0400, tsk wrote:
>         >         > Hi folks,
>         >         >
>         >         >
>         >         > Yestoday, I run a testcase, 6 VMs restart every 10
>         minitues
>         >         from 11:00
>         >         > am to 16:30 pm until Dom0 crashed.
>         >         >
>         >         > Assertion 'list_empty(&vreq->next)' failed will
>         lead to
>         >         tapdisk2
>         >         > process segfault, then Dom0 crashed!
>         >         >
>         >         >
>         >         > Jun 27 16:31:05 r02k05014 kernel: device tap775.0
>         entered
>         >         promiscuous
>         >         > mode
>         >         > Jun 27 16:31:05 r02k05014 kernel: eth0: port
>         13(tap775.0)
>         >         entering
>         >         > forwarding state
>         >         > Jun 27 16:31:15 r02k05014 tapdisk2[4341]:
>         Assertion
>         >         > 'list_empty(&vreq->next)' failed, line 1822, file
>         >         tapdisk-vbd.c
>         >         > Jun 27 16:31:15 r02k05014 kernel: tapdisk2[4341]:
>         segfault
>         >         at 0 ip
>         >         > 000000000040a24f sp 00007fff0d8acb70 error 6 in
>         >         tapdisk2[400000+39000]
>         >         > Jun 28 18:58:09 r02k05014 syslogd 1.4.1: restart.
>         >         >
>         >         >
>         >         > Any one who can give me some tips?
>         >
>         >
>         >         Semi-yes. Not entirely sure about the list_empty
>         userland
>         >         crasher, but
>         >         the kernel stuff was upstreamed with some minor
>         exposed while
>         >         unmapping
>         >         pending I/O.
>         >
>         >         Which I didn't care yet to send patches for...
>         >
>         >         ... Soon, real soon now...
>         >
>         >         Daniel
>         >
>         >
>         >
>         >
>         >
>         >
>         
>         
>         
> 
> 

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
  2010-07-05 21:25         ` Daniel Stodden
@ 2010-07-05 22:26           ` Jeremy Fitzhardinge
  2010-07-05 23:25             ` Daniel Stodden
  0 siblings, 1 reply; 7+ messages in thread
From: Jeremy Fitzhardinge @ 2010-07-05 22:26 UTC (permalink / raw)
  To: Daniel Stodden; +Cc: xen-devel@lists.xensource.com, tsk

On 07/05/2010 02:25 PM, Daniel Stodden wrote:
> On Thu, 2010-07-01 at 08:55 -0400, tsk wrote:
>   
>> Hi, will the patches be added to Xen-4.1？
>>     
> About to prepare the next bunch of patches for xen.git.
>
> Jeremy, is it okay to pull some of the changes made for 2.6.3x into the
> xen/dom0/backend/blktap2 topic branch before moving on?
>
> I'm mainly talking about this one:
>
> >From ab77527f9a63a5c657c6d6a50e70a66adceaa3a0 Mon Sep 17 00:00:00 2001
> From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> Date: Thu, 18 Feb 2010 17:22:09 -0800
> Subject: [PATCH] xen/blktap2: make compile
>
> and a couple related deltas which came later.
>
> Except moving forward to newer blk queue interfaces, they don't make
> much of a functional difference.
>
> Would simplify merging quite a bit.
>   

Which kernel version has these interface updates?  Do they predate the
version the blktap2 branch is rooted on, or do the come later?  If
they're later, then merge the mainline kernel (ideally a tagged release
like v2.6.3[12]) into the blktap2 one and cherrypick the "make compile"
changes on top of the merge (and if they're already there, then just
cherrypick).

    J

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: blktap2 bug:Assertion 'list_empty(&vreq->next)' failed
  2010-07-05 22:26           ` Jeremy Fitzhardinge
@ 2010-07-05 23:25             ` Daniel Stodden
  0 siblings, 0 replies; 7+ messages in thread
From: Daniel Stodden @ 2010-07-05 23:25 UTC (permalink / raw)
  To: Jeremy Fitzhardinge; +Cc: xen-devel@lists.xensource.com, tsk

On Mon, 2010-07-05 at 18:26 -0400, Jeremy Fitzhardinge wrote:
> On 07/05/2010 02:25 PM, Daniel Stodden wrote:
> > On Thu, 2010-07-01 at 08:55 -0400, tsk wrote:
> >   
> >> Hi, will the patches be added to Xen-4.1？
> >>     
> > About to prepare the next bunch of patches for xen.git.
> >
> > Jeremy, is it okay to pull some of the changes made for 2.6.3x into the
> > xen/dom0/backend/blktap2 topic branch before moving on?
> >
> > I'm mainly talking about this one:
> >
> > >From ab77527f9a63a5c657c6d6a50e70a66adceaa3a0 Mon Sep 17 00:00:00 2001
> > From: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> > Date: Thu, 18 Feb 2010 17:22:09 -0800
> > Subject: [PATCH] xen/blktap2: make compile
> >
> > and a couple related deltas which came later.
> >
> > Except moving forward to newer blk queue interfaces, they don't make
> > much of a functional difference.
> >
> > Would simplify merging quite a bit.
> >   
> 
> Which kernel version has these interface updates?  Do they predate the
> version the blktap2 branch is rooted on, or do the come later?  If
> they're later, then merge the mainline kernel (ideally a tagged release
> like v2.6.3[12]) into the blktap2 one and cherrypick the "make compile"
> changes on top of the merge (and if they're already there, then just
> cherrypick).

Okay, the HG follower in me didn't quite follow yet the cherry-related
part of that procedure. But altogether it sounds quite promising.

Followup later. Thanks! :)

Daniel

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2010-07-05 23:25 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-06-28  4:41 blktap2 bug:Assertion 'list_empty(&vreq->next)' failed tsk
2010-06-28  4:48 ` tsk
2010-06-28  4:49 ` [Xen-devel] " Daniel Stodden
2010-06-28  4:58   ` tsk
     [not found]     ` <1277701807.9747.1905.camel@ramone.somacoma.net>
     [not found]       ` <AANLkTikcN8oX-LG8bhsa8CQJTjn1CLFvNuwkThMSb5yR@mail.gmail.com>
2010-07-05 21:25         ` Daniel Stodden
2010-07-05 22:26           ` Jeremy Fitzhardinge
2010-07-05 23:25             ` Daniel Stodden

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.