Re: [Qemu-devel] [PATCH v6 2/2] block: Support GlusterFS as a QEMU block backend

[Bharata B Rao <bharata@linux.vnet.ibm.com>]

On Wed, Aug 15, 2012 at 10:00:27AM +0200, Kevin Wolf wrote:
> Am 15.08.2012 07:21, schrieb Bharata B Rao:
> > On Tue, Aug 14, 2012 at 10:29:26AM +0200, Kevin Wolf wrote:
> >>>>> +static void gluster_finish_aiocb(struct glfs_fd *fd, ssize_t ret, void *arg)
> >>>>> +{
> >>>>> +    GlusterAIOCB *acb = (GlusterAIOCB *)arg;
> >>>>> +    BDRVGlusterState *s = acb->common.bs->opaque;
> >>>>> +
> >>>>> +    acb->ret = ret;
> >>>>> +    if (qemu_gluster_send_pipe(s, acb) < 0) {
> >>>>> +        /*
> >>>>> +         * Gluster AIO callback thread failed to notify the waiting
> >>>>> +         * QEMU thread about IO completion. Nothing much can be done
> >>>>> +         * here but to abruptly abort.
> >>>>> +         *
> >>>>> +         * FIXME: Check if the read side of the fd handler can somehow
> >>>>> +         * be notified of this failure paving the way for a graceful exit.
> >>>>> +         */
> >>>>> +        error_report("Gluster failed to notify QEMU about IO completion");
> >>>>> +        abort();
> >>>>
> >>>> In the extreme case you may choose to make this disk inaccessible
> >>>> (something like bs->drv = NULL), but abort() kills the whole VM and
> >>>> should only be called when there is a bug.
> >>>
> >>> There have been concerns raised about this earlier too. I settled for this
> >>> since I couldn't see a better way out and I could see the precedence
> >>> for this in posix-aio-compat.c
> >>>
> >>> So I could just do the necessary cleanup, set bs->drv to NULL and return from
> >>> here ? But how do I wake up the QEMU thread that is waiting on the read side
> >>> of the pipe ? W/o that, the QEMU thread that waits on the read side of the
> >>> pipe is still hung.
> >>
> >> There is no other thread. But you're right, you should probably
> >> unregister the aio_fd_handler and any other pending callbacks.
> > 
> > As I clarified in the other mail, this (gluster_finish_aiocb) is called
> > from gluster thread context and hence QEMU thread that raised the original
> > read/write request is still blocked on qemu_aio_wait().
> > 
> > I tried the following cleanup instead of abrupt abort:
> > 
> > close(read_fd); /* This will wake up the QEMU thread blocked on select(read_fd...) */
> > close(write_fd);
> > qemu_aio_set_fd_handler(read_fd, NULL, NULL, NULL, NULL);
> > qemu_aio_release(acb);
> > s->qemu_aio_count--;
> > bs->drv = NULL;
> > 
> > I tested this by manually injecting faults into qemu_gluster_send_pipe().
> > With the above cleanup, the guest kernel crashes with IO errors.
> 
> What does "crash" really mean? IO errors certainly shouldn't cause a
> kernel to crash?

Since an IO failed, it resulted in root file system corruption which
subsequently led to a panic.

[    1.529042] dracut: Switching root
qemu-system-x86_64: Gluster failed to notify QEMU about IO completion
qemu-system-x86_64: Gluster failed to notify QEMU about IO completion
qemu-system-x86_64: Gluster failed to notify QEMU about IO completion
qemu-system-x86_64: Gluster failed to notify QEMU about IO completion
[    1.584130] end_request: I/O error, dev vda, sector 13615224
[    1.585119] end_request: I/O error, dev vda, sector 13615344
[    1.585119] end_request: I/O error, dev vda, sector 13615352
[    1.585119] end_request: I/O error, dev vda, sector 13615360
[    1.593188] end_request: I/O error, dev vda, sector 1030144
[    1.594169] Buffer I/O error on device vda3, logical block 0
[    1.594169] lost page write due to I/O error on vda3
[    1.594169] EXT4-fs error (device vda3): __ext4_get_inode_loc:3539: inode #392441: block 1573135: comm systemd: unable to read itable block
[...]
[    1.620064] EXT4-fs error (device vda3): __ext4_get_inode_loc:3539: inode #392441: block 1573135: comm systemd: unable to read itable block
/usr/lib/systemd/systemd: error while loading shared libraries: libselinux.so.1: cannot open shared object file: Input/output error
[    1.626193] Kernel panic - not syncing: Attempted to kill init!
[    1.627789] Pid: 1, comm: systemd Not tainted 3.3.4-5.fc17.x86_64 #1
[    1.630063] Call Trace:
[    1.631120]  [<ffffffff815e21eb>] panic+0xba/0x1c6
[    1.632477]  [<ffffffff8105aff1>] do_exit+0x8b1/0x8c0
[    1.633851]  [<ffffffff8105b34f>] do_group_exit+0x3f/0xa0
[    1.635258]  [<ffffffff8105b3c7>] sys_exit_group+0x17/0x20
[    1.636619]  [<ffffffff815f38e9>] system_call_fastpath+0x16/0x1b

> 
> > Is there anything else that I need to do or do differently to retain the
> > VM running w/o disk access ?
> > 
> > I thought of completing the aio callback by doing
> > acb->common.cb(acb->common.opaque, -EIO);
> > but that would do a coroutine enter from gluster thread, which I don't think
> > should be done.
> 
> You would have to take the global qemu mutex at least. I agree it's not
> a good thing to do.

So is it really worth doing all this to handle this unlikely error ? The
chances of this error happening is quite remote I believe.

Regards,
Bharata.