[Qemu-devel] One more thing about block device locking

[Qemu-devel] One more thing about block device locking

[Michael Tokarev]

While I'm reviewing a thread about block device
locking, here's another data point which were
not touched before, as far as I remember.  It
is related.

What I'm talking is - when fsck/mkfs/... family
of programs are run against a mounted (or in use
by other means) device, they warn you about this,
or refuse to run.

This is done by opening the device in question
with O_EXCL flag - it is meaningless for an
existing block device so the meaning has been
overloaded on linux.

It is the _only_ way to ensure the device is
opened exclusively, and it covers such obscure
cases like opening whole disk when at least
one partition is in use by, say, an in-kernel
raid array or something like that.

So while it is the only way, it is also a
_reliable_ way too.

And while before, we were talked mostly about
inter-guest locking/protection, we also should
think about protecting guest and host from
each other.  O_EXCEL is exactly this case --
to ensure _host_ is not using the devie in
question when qemu is trying to open it.

Worth using IMHO :)  But it is a bit weird,
since it only works on linux (actually I've
no idea if it works on any other unix-like
system) and only on block devices, and only
at open(2) time.  So we'll have to either
trial and error, or open "normally", check
if it's a block device and re-open with that
flag set.  Both aren't exactly nice, but
should work.

I'll see what can be done there, but if
someone else who knows code better have
immediate ideas or implementation, welcome.

Thanks!

/mjt

Re: [Qemu-devel] One more thing about block device locking

[Richard W.M. Jones]

On Fri, Apr 23, 2010 at 09:07:28AM +0400, Michael Tokarev wrote:
> So we'll have to either
> trial and error, or open "normally", check
> if it's a block device and re-open with that
> flag set.

Perhaps I'm missing something, but why can't you stat(2) the name
first to see if it's a block device (ie. S_ISBLK(st_mode)) then add
the O_EXCL flag or not as appropriate?

Anyway, the problem with this is it's not helpful (in fact: actively
bad) for libguestfs.  We want to allow people to open existing devices
read-only, even if one qemu process has them open for writing.
Consider, for example, virt-df or virt-cat which allow you to read
information out from live VMs.

This is why we should have exclusive|shared modes for locking (or if
you prefer write|read -- it's the same thing with a different name
AFAICT).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into Xen guests.
http://et.redhat.com/~rjones/virt-p2v

Re: [Qemu-devel] One more thing about block device locking

[Michael Tokarev]

Richard W.M. Jones wrote:
> On Fri, Apr 23, 2010 at 09:07:28AM +0400, Michael Tokarev wrote:
>> So we'll have to either
>> trial and error, or open "normally", check
>> if it's a block device and re-open with that
>> flag set.
> 
> Perhaps I'm missing something, but why can't you stat(2) the name
> first to see if it's a block device (ie. S_ISBLK(st_mode)) then add
> the O_EXCL flag or not as appropriate?

Yes, that will work.  Still, it appears to be linux-specific (the
whole O_EXCL thing).

> Anyway, the problem with this is it's not helpful (in fact: actively
> bad) for libguestfs.  We want to allow people to open existing devices
> read-only, even if one qemu process has them open for writing.
> Consider, for example, virt-df or virt-cat which allow you to read
> information out from live VMs.

Actually it IS quite helpful, to stop kvm from opening a filesystem
mounted by the host or which is part of a raid array, or even if
it's the system hard drive (*).

And it looks like you didn't understand the semantics of O_EXCL: it
works for only one open(), namely the one which has O_EXCL flag set,
to mean "fail open if there are other users of the device at the
moment".  If a block device is open with O_EXCL flag, other processes
can open it (without O_EXCL) as before, so no harm is done for
virt-df or virt-cat which will still be able to do their tasks.

> This is why we should have exclusive|shared modes for locking (or if
> you prefer write|read -- it's the same thing with a different name
> AFAICT).

No, this is not the same thing.  Explicit locking (fcntl/etc) works
for plain files to prevent concurrent access by, say, two qemu
processes (two guests) or the like.  But O_EXCL on a block device
works for host kernel too, which ignores file locking.

One thing is to prevent two guests from opening the same device (or
file) simultaneously.  Another is to prevent corrupting a block
device used by _host_ (and thus probably crashing host as well).

That's why I said it's related but different topic, right at the
beginning of my email.  Both should be used as appropriate.  I'll
sum it all up when I'll finish with that long thread and after a
bit more thinking :)

Thanks!

/mjt

Re: [Qemu-devel] One more thing about block device locking

[Kevin Wolf]

Am 23.04.2010 10:00, schrieb Richard W.M. Jones:
> On Fri, Apr 23, 2010 at 09:07:28AM +0400, Michael Tokarev wrote:
>> So we'll have to either
>> trial and error, or open "normally", check
>> if it's a block device and re-open with that
>> flag set.
> 
> Perhaps I'm missing something, but why can't you stat(2) the name
> first to see if it's a block device (ie. S_ISBLK(st_mode)) then add
> the O_EXCL flag or not as appropriate?

We already have two different open functions for regular files and
devices (and yes, the detection uses stat).

Kevin