[Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Andrey Korolyov]

Hello,

2.1-stable is currently crashing with the

Co-routine re-entered recursively
2014-12-16 15:06:23.578+0000: shutting down

after execution of (for example) following when using virtio-dp as a
disk backend:

'{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
"inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
'{ "execute": "nbd-server-add", "arguments": {"device":
"drive-virtio-disk0","writable": false } }'

Should be easily reproduceable, args follows:

qemu-system-x86_64 -enable-kvm -name vm31148 -S -machine
pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+kvm_pv_eoi -m 512
-realtime mlock=off -smp 12,sockets=1,cores=12,threads=12 -uuid
fe590ff9-1f18-4bb1-9627-7d8c800a00fe -nographic -no-user-config
-nodefaults -device sga -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/vm31148.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
nec-usb-xhci,id=usb,bus=pci.0,addr=0x4 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=rbd:dev-rack2/vm31148-rjv:id=qemukvm:key=xxx:auth_supported=cephx\;none:mon_host=10.6.0.1\:6789\;10.6.0.3\:6789\;10.6.0.4\:6789,if=none,id=drive-virtio-disk0,format=raw
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/vm31148.sock,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.1
-object iothread,id=vm31148blk0 -set
device.virtio-disk0.config-wce=off -set device.virtio-disk0.scsi=off
-set device.virtio-disk0.iothread=vm31148blk0

Re: [Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 798 bytes --]

On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
> 2.1-stable is currently crashing with the
> 
> Co-routine re-entered recursively
> 2014-12-16 15:06:23.578+0000: shutting down
> 
> after execution of (for example) following when using virtio-dp as a
> disk backend:
> 
> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
> '{ "execute": "nbd-server-add", "arguments": {"device":
> "drive-virtio-disk0","writable": false } }'

Hi,
You need patches from Max Reitz that make the run-time NBD server
support dataplane:
f214928 nbd: Follow the BDS' AIO context
3338442 block: Add AIO context notifiers
958c717 nbd: Drop nbd_can_read()

Please try QEMU v2.2.0 or qemu.git.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Andrey Korolyov]

On Fri, Jan 2, 2015 at 4:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
>> 2.1-stable is currently crashing with the
>>
>> Co-routine re-entered recursively
>> 2014-12-16 15:06:23.578+0000: shutting down
>>
>> after execution of (for example) following when using virtio-dp as a
>> disk backend:
>>
>> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
>> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
>> '{ "execute": "nbd-server-add", "arguments": {"device":
>> "drive-virtio-disk0","writable": false } }'
>
> Hi,
> You need patches from Max Reitz that make the run-time NBD server
> support dataplane:
> f214928 nbd: Follow the BDS' AIO context
> 3338442 block: Add AIO context notifiers
> 958c717 nbd: Drop nbd_can_read()
>
> Please try QEMU v2.2.0 or qemu.git.
>
> Stefan

Thanks Stefan, those commits are enough to make dataplane drive work
well with runtime nbd server. By the way, can such functionality
improvements (not major improvements like discard support for scsi)
make their way as backports? To reword this, unexpected behavior like
sudden shutdown I described is better to be closed by later fixes than
by stubs or left as it currently behaving IMO.

Re: [Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]

On Mon, Jan 05, 2015 at 05:12:10PM +0400, Andrey Korolyov wrote:
> On Fri, Jan 2, 2015 at 4:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
> >> 2.1-stable is currently crashing with the
> >>
> >> Co-routine re-entered recursively
> >> 2014-12-16 15:06:23.578+0000: shutting down
> >>
> >> after execution of (for example) following when using virtio-dp as a
> >> disk backend:
> >>
> >> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
> >> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
> >> '{ "execute": "nbd-server-add", "arguments": {"device":
> >> "drive-virtio-disk0","writable": false } }'
> >
> > Hi,
> > You need patches from Max Reitz that make the run-time NBD server
> > support dataplane:
> > f214928 nbd: Follow the BDS' AIO context
> > 3338442 block: Add AIO context notifiers
> > 958c717 nbd: Drop nbd_can_read()
> >
> > Please try QEMU v2.2.0 or qemu.git.
> >
> > Stefan
> 
> Thanks Stefan, those commits are enough to make dataplane drive work
> well with runtime nbd server. By the way, can such functionality
> improvements (not major improvements like discard support for scsi)
> make their way as backports? To reword this, unexpected behavior like
> sudden shutdown I described is better to be closed by later fixes than
> by stubs or left as it currently behaving IMO.

It is a bug that NBD combined with virtio-blk dataplane aborted, it
should have produced an error message and simply refused to start the
NBD server.

Now that QEMU 2.2 stable has been released I'm not sure if additional
2.1.x releases will be made at all.  I have CCed qemu-stabled@nongnu.org
to see if there is interest.

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] 2.1 unexpected stop after exporting blockdev via nbd server

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1580 bytes --]

On Mon, Jan 05, 2015 at 05:12:10PM +0400, Andrey Korolyov wrote:
> On Fri, Jan 2, 2015 at 4:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
> >> 2.1-stable is currently crashing with the
> >>
> >> Co-routine re-entered recursively
> >> 2014-12-16 15:06:23.578+0000: shutting down
> >>
> >> after execution of (for example) following when using virtio-dp as a
> >> disk backend:
> >>
> >> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
> >> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
> >> '{ "execute": "nbd-server-add", "arguments": {"device":
> >> "drive-virtio-disk0","writable": false } }'
> >
> > Hi,
> > You need patches from Max Reitz that make the run-time NBD server
> > support dataplane:
> > f214928 nbd: Follow the BDS' AIO context
> > 3338442 block: Add AIO context notifiers
> > 958c717 nbd: Drop nbd_can_read()
> >
> > Please try QEMU v2.2.0 or qemu.git.
> >
> > Stefan
> 
> Thanks Stefan, those commits are enough to make dataplane drive work
> well with runtime nbd server. By the way, can such functionality
> improvements (not major improvements like discard support for scsi)
> make their way as backports? To reword this, unexpected behavior like
> sudden shutdown I described is better to be closed by later fixes than
> by stubs or left as it currently behaving IMO.

Oops, I typoed the qemu-stable@nongnu.org email address.  Trying
again...

QEMU stable folks: will there be another 2.1.x release?

Stefan

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]

Re: [Qemu-devel] [Qemu-stable] 2.1 unexpected stop after exporting blockdev via nbd server

[Michael Roth]

Quoting Stefan Hajnoczi (2015-01-06 05:49:11)
> On Mon, Jan 05, 2015 at 05:12:10PM +0400, Andrey Korolyov wrote:
> > On Fri, Jan 2, 2015 at 4:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
> > >> 2.1-stable is currently crashing with the
> > >>
> > >> Co-routine re-entered recursively
> > >> 2014-12-16 15:06:23.578+0000: shutting down
> > >>
> > >> after execution of (for example) following when using virtio-dp as a
> > >> disk backend:
> > >>
> > >> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
> > >> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
> > >> '{ "execute": "nbd-server-add", "arguments": {"device":
> > >> "drive-virtio-disk0","writable": false } }'
> > >
> > > Hi,
> > > You need patches from Max Reitz that make the run-time NBD server
> > > support dataplane:
> > > f214928 nbd: Follow the BDS' AIO context
> > > 3338442 block: Add AIO context notifiers
> > > 958c717 nbd: Drop nbd_can_read()
> > >
> > > Please try QEMU v2.2.0 or qemu.git.
> > >
> > > Stefan
> > 
> > Thanks Stefan, those commits are enough to make dataplane drive work
> > well with runtime nbd server. By the way, can such functionality
> > improvements (not major improvements like discard support for scsi)
> > make their way as backports? To reword this, unexpected behavior like
> > sudden shutdown I described is better to be closed by later fixes than
> > by stubs or left as it currently behaving IMO.
> 
> Oops, I typoed the qemu-stable@nongnu.org email address.  Trying
> again...
> 
> QEMU stable folks: will there be another 2.1.x release?

Yes, 2.1.3 was very poorly timed to go out before Christmas but obviously
that didn't happen. The patch round-up should go out tomorrow with the
freeze next Wednesday. The updated schedule is available here:

http://qemu-project.org/Planning/2.1#Release_Schedule

> 
> Stefan

Re: [Qemu-devel] [Qemu-stable] 2.1 unexpected stop after exporting blockdev via nbd server

[Stefan Hajnoczi]

On Thu, Jan 8, 2015 at 1:17 AM, Michael Roth <mdroth@linux.vnet.ibm.com> wrote:
> Quoting Stefan Hajnoczi (2015-01-06 05:49:11)
>> On Mon, Jan 05, 2015 at 05:12:10PM +0400, Andrey Korolyov wrote:
>> > On Fri, Jan 2, 2015 at 4:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> > > On Thu, Dec 18, 2014 at 07:38:59PM +0400, Andrey Korolyov wrote:
>> > >> 2.1-stable is currently crashing with the
>> > >>
>> > >> Co-routine re-entered recursively
>> > >> 2014-12-16 15:06:23.578+0000: shutting down
>> > >>
>> > >> after execution of (for example) following when using virtio-dp as a
>> > >> disk backend:
>> > >>
>> > >> '{ "execute": "nbd-server-start", "arguments": { "addr": { "type":
>> > >> "inet", "data": { "host": "10.6.0.1", "port": "3333" } } } }'
>> > >> '{ "execute": "nbd-server-add", "arguments": {"device":
>> > >> "drive-virtio-disk0","writable": false } }'
>> > >
>> > > Hi,
>> > > You need patches from Max Reitz that make the run-time NBD server
>> > > support dataplane:
>> > > f214928 nbd: Follow the BDS' AIO context
>> > > 3338442 block: Add AIO context notifiers
>> > > 958c717 nbd: Drop nbd_can_read()
>> > >
>> > > Please try QEMU v2.2.0 or qemu.git.
>> > >
>> > > Stefan
>> >
>> > Thanks Stefan, those commits are enough to make dataplane drive work
>> > well with runtime nbd server. By the way, can such functionality
>> > improvements (not major improvements like discard support for scsi)
>> > make their way as backports? To reword this, unexpected behavior like
>> > sudden shutdown I described is better to be closed by later fixes than
>> > by stubs or left as it currently behaving IMO.
>>
>> Oops, I typoed the qemu-stable@nongnu.org email address.  Trying
>> again...
>>
>> QEMU stable folks: will there be another 2.1.x release?
>
> Yes, 2.1.3 was very poorly timed to go out before Christmas but obviously
> that didn't happen. The patch round-up should go out tomorrow with the
> freeze next Wednesday. The updated schedule is available here:
>
> http://qemu-project.org/Planning/2.1#Release_Schedule

Excellent, please add the following commits to stable:
f214928 nbd: Follow the BDS' AIO context
3338442 block: Add AIO context notifiers
958c717 nbd: Drop nbd_can_read()

They prevent crashes when the run-time NBD server is used together
with virtio-blk dataplane.

They apply cleanly onto stable-2.1 and I have tested that the crash is solved.

Thanks,
Stefan

Re: [Qemu-devel] [Qemu-stable] 2.1 unexpected stop after exporting blockdev via nbd server

[Paolo Bonzini]

On 08/01/2015 11:31, Stefan Hajnoczi wrote:
> Excellent, please add the following commits to stable:
> f214928 nbd: Follow the BDS' AIO context
> 3338442 block: Add AIO context notifiers
> 958c717 nbd: Drop nbd_can_read()
> 
> They prevent crashes when the run-time NBD server is used together
> with virtio-blk dataplane.
> 
> They apply cleanly onto stable-2.1 and I have tested that the crash is solved.

I think they break Win32 compilation.  Didn't Max have to wait for me to
port socket AioContext handlers to Windows?

Paolo

Re: [Qemu-devel] [Qemu-stable] 2.1 unexpected stop after exporting blockdev via nbd server

[Stefan Hajnoczi]

On Thu, Jan 8, 2015 at 12:11 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 08/01/2015 11:31, Stefan Hajnoczi wrote:
>> Excellent, please add the following commits to stable:
>> f214928 nbd: Follow the BDS' AIO context
>> 3338442 block: Add AIO context notifiers
>> 958c717 nbd: Drop nbd_can_read()
>>
>> They prevent crashes when the run-time NBD server is used together
>> with virtio-blk dataplane.
>>
>> They apply cleanly onto stable-2.1 and I have tested that the crash is solved.
>
> I think they break Win32 compilation.  Didn't Max have to wait for me to
> port socket AioContext handlers to Windows?

You are right.  I remember that now.  Yikes, a 10 patch series.

Mike: Let's drop the idea.  QEMU 2.2 has been released and does not
have the bug.

Stefan