Re: Feature requests: online backup - defrag - change RAID level

webmaster
2019-09-09T17:11:14Z

Quoting Qu Wenruo <quwenruo.btrfs@gmx.com>:

> On 2019/9/9 下午8:18, Qu Wenruo wrote:
>>
>>
>> On 2019/9/9 下午7:25, zedlryqc@server53.web-hosting.com wrote:
>>>
>>> Quoting Qu Wenruo <quwenruo.btrfs@gmx.com>:
>>>>> 1) Full online backup (or copy, whatever you want to call it)
>>>>> btrfs backup <filesystem name> <partition name> [-f]
>>>>> - backups a btrfs filesystem given by <filesystem name> to a partition
>>>>> <partition name> (with all subvolumes).
>>>>
>>>> Why not just btrfs send?
>>>>
>>>> Or you want to keep the whole subvolume structures/layout?
>>>
>>> Yes, I want to keep the whole subvolume structures/layout. I want to
>>> keep everything. Usually, when I want to backup a partition, I want to
>>> keep everything, and I suppose most other people have a similar idea.
>>>
>>>> I'd say current send/receive is more flex.
>>>
>>> Um, 'flexibility' has nothing to do with it. Send/receive is a
>>> completely different use case.
>>> So, each one has some benefits and some drawbacks, but 'send/receive'
>>> cannot replace 'full online backup'
>>>
>>> Here is where send/receive is lacking:
>>> - too complicated to do if many subvolumes are involved
>>> - may require recursive subvolume enumeration in order to emulate
>>> 'full online backup'
>>> - may require extra storage space
>>> - is not mountable, not easy to browse the backup contents
>>> - not easy to recover just a few selected files from backup
>>> There's probably more things where send/receive is lacking, but I think
>>> I have given sufficient number of important differences which show that
>>> send/receive cannot successfully replace the functionality of 'full
>>> online backup'.
>
> Forgot to mention this part.
>
> If your primary objective is to migrate your data to another device
> online (mounted, without unmount any of the fs).

This is not the primary objective. The primary objective is to produce
a full, online, easy-to-use, robust backup. But let's say we need to
do migration...
>
> Then I could say, you can still add a new device, then remove the old
> device to do that.

If the source filesystem already uses RAID1, then, yes, you could do
it, but it would be too slow, it would need a lot of user
intervention, so many commands typed, so many ways to do it wrong, to
make a mistake.

Too cumbersome. Too wastefull of time and resources.

> That would be even more efficient than LVM (not thin provisioned one),
> as we only move used space.

In fact, you can do this kind of full-online-backup with the help of
mdadm RAID, or some other RAID solution. It can already be done, no
need to add 'btrfs backup'.

But, again, to cumbersome, too inflexible, too many problems, and, the
user would have to setup a downgraded mdadm RAID in front and run with
a degraded mdadm RAID all the time (since btrfs RAID would be actually
protecting the data).

> If your objective is to create a full copy as backup, then I'd say my
> new patchset of btrfs-image data dump may be your best choice.

It should be mountable. It should be performed online. Never heard of
btrfs-image, i need the docs to see whether this btrfs-image is good
enough.

> The only down side is, you need to at least mount the source fs to RO mode.

No. That's not really an online backup. Not good enough.

> The true on-line backup is not that easy, especially any write can screw
> up your backup process, so it must be done unmounted.

Nope, I disagree.

First, there is the RAID1-alike solution, which is easy to perform
(just send all new writes to both source and destination). It's the
same thing that mdadm RAID1 would do (like I mentioned a few
paragraphs above).
But, this solution may have a performance concern, when the
destination drive is too slow.

Fortunately, with btrfs, an online backup is easier tha usual. To
produce a frozen snapshot of the entire filesystem, just create a
read-only snapshot of every subvolume (this is not 100% consistent, I
know, but it is good enough).

But I'm just repeating myself, I already wrote this in the first email.

So, in conclusion I disagree that true on-line backup is not easy.

> Even btrfs send handles this by forcing the source subvolume to be RO,
> so I can't find an easy solution to address that.

This is a digression, but I would say that you first make a temporary
RO snapshot of the source subvolume, then use 'btrfs send' on the
temporary snapshot, then delete the temporary snapshot.

Oh, my.