Re: [Qemu-devel] RFC: moving fsfreeze support from the userland guest agent to the guest kernel

[Michael Roth <mdroth@linux.vnet.ibm.com>]

On 07/28/2011 03:03 AM, Andrea Arcangeli wrote:
> On Thu, Jul 28, 2011 at 11:53:50AM +0900, Fernando Luis Vázquez Cao wrote:
>> On Wed, 2011-07-27 at 17:24 +0200, Andrea Arcangeli wrote:
>>> making
>>> sure no lib is calling any I/O function to be able to defreeze the
>>> filesystems later, making sure the oom killer or a wrong kill -9
>>> $RANDOM isn't killing the agent by mistake while the I/O is blocked
>>> and the copy is going.
>>
>> Yes with the current API if the agent is killed while the filesystems
>> are frozen we are screwed.
>>
>> I have just submitted patches that implement a new API that should make
>> the virtualization use case more reliable. Basically, I am adding a new
>> ioctl, FIGETFREEZEFD, which freezes the indicated filesystem and returns
>> a file descriptor; as long as that file descriptor is held open, the
>> filesystem remains open. If the freeze file descriptor is closed (be it
>> through a explicit call to close(2) or as part of process exit
>> housekeeping) the associated filesystem is automatically thawed.
>>
>> - fsfreeze: add ioctl to create a fd for freeze control
>>    http://marc.info/?l=linux-fsdevel&m=131175212512290&w=2
>> - fsfreeze: add freeze fd ioctls
>>    http://marc.info/?l=linux-fsdevel&m=131175220612341&w=2
>
> This is probably how the API should have been implemented originally
> instead of FIFREEZE/FITHAW.
>
> It looks a bit overkill though, I would think it'd be enough to have
> the fsfreeze forced at FIGETFREEZEFD, and the only way to thaw by
> closing the file without requiring any of the
> FS_FREEZE_FD/FS_THAW_FD/FS_ISFROZEN_FD. But I guess you have use cases

One of the crappy things about the current implementation is the 
inability to determine whether or not a filesystem is frozen. At least 
in the context of guest agent at least, it'd be nice if 
guest-fsfreeze-status checked the actual system state rather than some 
internal state that may not necessarily reflect reality (if we freeze, 
and some other application thaws, we currently still report the state as 
frozen).

Also in the context of the guest agent, we are indeed screwed if the 
agent gets killed while in a frozen state, and remain screwed even if 
it's restarted since we have no way of determining whether or not we're 
in a frozen state and thus should disable logging operations.

We could check status by looking for a failure from the freeze 
operation, but if you're just interested in getting the state, having to 
potentially induce a freeze just to get at the state is really heavy-handed.

So having an open operation that doesn't force a freeze/thaw/status 
operation serves some fairly common use cases I think.

> for those if you implemented it, maybe to check if root is stepping on
> its own toes by checking if the fs is already freezed before freezing
> it and returning failure if it is, running ioctl instead of opening
> closing the file isn't necessarily better. At the very least the
> get_user(should_freeze, argp) doesn't seem so necessary, it just
> complicates the ioctl API a bit without much gain, I think it'd be
> cleaner if the FS_FREEZE_FD was the only way to freeze then.
>
> It's certainly a nice reliability improvement and safer API.
>
> Now if you add a file descriptor to epoll/poll that userland can open
> and talk to, to know when a fsfreeze is asked on a certain fs, a
> fsfreeze userland agent (not virt related too) could open it and start
> the scripts if that filesystem is being fsfreezed before calling
> freeze_super().
>
> Then a PARAVIRT_FSFREEZE=y/m driver could just invoke the fsfreeze
> without any dependency on a virt specific guest agent.
>
> Maybe Christoph's right there are filesystems in userland (not sure
> how the storage is related, it's all about filesystems and apps as far
> I can see, and it's all blkdev agnostic) that may make things more
> complicated, but those usually have a kernel backend too (like
> fuse). I may not see the full picture of the filesystem in userland or
> how the storage agent in guest userland relates to this.
>
> If you believe having libvirt talking QMP/QAPI over a virtio-serial
> vmchannel with some virt specific guest userland agent bypassing qemu
> entirely is better, that's ok with me, but there should be a strong
> reason for it because the paravirt_fsfreeze.ko approach with a small
> qemu backend and a qemu monitor command that starts paravirt-fsfreeze
> in guest before going ahead blocking all I/O (to provide backwards
> compatibility and reliable snapshots to guest OS that won't have the
> paravirt fsfreeze too) looks more reliable, more compact and simpler
> to use to me. I'll be surely ok either ways though.
>
> Thanks,
> Andrea