From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52298) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aTIxb-0005Ih-1z for qemu-devel@nongnu.org; Tue, 09 Feb 2016 19:48:48 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aTIxa-0000j2-0y for qemu-devel@nongnu.org; Tue, 09 Feb 2016 19:48:46 -0500 From: John Snow Message-ID: <56BA88E7.7010104@redhat.com> Date: Tue, 9 Feb 2016 19:48:39 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] RFC: incremental backups: qmp-block-dirty-bitmap-diff List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Qemu-block Cc: Fam Zheng , qemu-devel , Vladimir Sementsov-Ogievskiy , Stefan Hajnoczi , "Denis V. Lunev" The thought was raised that it might be useful to create a "diff" command for bitmaps that allow us to populate a dirty bitmap based on the difference between a currently loaded disk image and some point-in-time stored on disk (an earlier full backup, incremental backup, etc). I originally voiced a desire for this command in response to how to recover from a failed post-copy bitmap migration (Talks ongoing for if we'll pursue that, but let's not worry about that here.) Another usage (apart from disaster recovery where we -- for whatever reason -- lost our bitmap or actually even lost an intermediate differential backup) for this kind of functionality would be to *start* a new incremental backup chain in a post-hoc fashion -- after we already made a full backup, but neglected to create a bitmap atomically. There are some pros and cons here; I wanted to get a general "Yes, that sounds good" or a "No way, don't bother" before I sunk some hours into it, at Stefan's suggestion. Cons: - Added maintenance burden of yet-another-blockjob that does a fairly esoteric operation - Catering to user carelessness. "You should have thought of that earlier!" - Reading an entire drive could take quite a long time -- Especially if both the source and the target are through network storage. - Reading an entire drive to populate a bitmap with the understanding that an incremental backup is soon to follow is inefficient if the drive is more than just a little dirty: it may have been quicker to just create a new full backup and bitmap. - Not strictly unlocking new functionality in the normative case. (We do add 'bitmap rebuilding after a freak accident', but this is not the normative case.) Pros: - Since bitmaps (and persistent bitmaps) are off by default, it's nice to allow users to enable the feature post-hoc. The chance for a misconfiguration here seems reasonably good, especially considering the feature hasn't really hit all of the stable distros yet. - Increases the flexibility overall of how to begin using the incremental backups feature. - Allows us to sync bitmaps against images created by external tools if needed or desired by external backup appliances. - If the image isn't that much different from the last good backup, reading the entire drive and writing out only a few changed blocks is likely to be faster than creating a new full backup, considering reads tend to be faster than writes -- reading the whole drive will be faster than writing a new full backup in many cases. - The fact that a diff+incremental backup may be slower in some cases than just simply creating a new full backup isn't a good rationale for why we can't offer this tool for the cases in which it may outperform. - Read of the last-known-good backup can be done in the background as a blockjob without impeding VM operation. Writes that occur during this process can be tracked in a temporary bitmap and merged after the diff completes.