btrfs-image and btrfs send related queries

btrfs-image and btrfs send related queries

[sri]

Hi,

I have couple of queries related to btrfs-image, btrfs send and with 
combination of two.
1)
I would like to know if a btrfs source file system is spread across more 
than 1 disks, does btrfs-image require same number of disks to create 
empty file system without files content??


2) would btrfs-image can be modified to keep only given subvolume foot 
print and related meta data to bring back file system live on destination 
disk?

   To elaborate more on this, Lets say I have 5 subvolumes on source btrfs 
and i run btrfs-image written to destination disk say /dev/sdd. In this 
process, can btrfs-image modified to just have only 1 subvolume and skipp 
other 4 subvolumes and write to destination i.. /dev/sdd so that when I 
mount /dev/sdd , I will have btrfs with only 1 subvolume with no data.

3) If 3 can successful, can btrfs-image further changed to include data of 
selected subvolume which gives files data also written to /dev/sdd which 
would be kind of a backup of a subvolume taken out of a btrfs file system 
which is having more than 1 subvolumes.

Re: btrfs-image and btrfs send related queries

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2253 bytes --]

On Fri, Apr 15, 2016 at 12:41:36PM +0000, sri wrote:
> Hi,
> 
> I have couple of queries related to btrfs-image, btrfs send and with 
> combination of two.
> 1)
> I would like to know if a btrfs source file system is spread across more 
> than 1 disks, does btrfs-image require same number of disks to create 
> empty file system without files content??

   I don't _think_ you need as many devices as there were originally.

> 2) would btrfs-image can be modified to keep only given subvolume foot 
> print and related meta data to bring back file system live on destination 
> disk?
> 
>    To elaborate more on this, Lets say I have 5 subvolumes on source btrfs 
> and i run btrfs-image written to destination disk say /dev/sdd. In this 
> process, can btrfs-image modified to just have only 1 subvolume and skipp 
> other 4 subvolumes and write to destination i.. /dev/sdd so that when I 
> mount /dev/sdd , I will have btrfs with only 1 subvolume with no data.

   For a first approximation, you could just drop any FS tree from the
image which wasn't the target one. After that, it turns into a
complicated accounting exercise to drop all of the back-refs to the
missing FS trees, and to drop all the extent records for the
non-shared data and the metadata for the missing FS trees.

   It's probably going to be complicated, and will basically involve
rewriting most of the image to avoid the metadata you didn't want.

> 3) If 3 can successful, can btrfs-image further changed to include data of 
> selected subvolume which gives files data also written to /dev/sdd which 
> would be kind of a backup of a subvolume taken out of a btrfs file system 
> which is having more than 1 subvolumes.

   If you're going to do all the hard work of (2), then (3) is a
reasonable logical(?) extension.

   On the other hand, what's wrong with simply using send/receive? It
gives you a data structure (a FAR-format send stream) which contains
everything you need to reconstruct a subvolume on a btrfs different
to the original.

   Hugo.

-- 
Hugo Mills             | Mary had a little lamb
hugo@... carfax.org.uk | You've heard this tale before
http://carfax.org.uk/  | But did you know she passed her plate
PGP: E2AB1DE4          | And had a little more?

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: btrfs-image and btrfs send related queries

[Henk Slager]

On Fri, Apr 15, 2016 at 2:49 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Fri, Apr 15, 2016 at 12:41:36PM +0000, sri wrote:
>> Hi,
>>
>> I have couple of queries related to btrfs-image, btrfs send and with
>> combination of two.
>> 1)
>> I would like to know if a btrfs source file system is spread across more
>> than 1 disks, does btrfs-image require same number of disks to create
>> empty file system without files content??
>
>    I don't _think_ you need as many devices as there were originally.

Indeed, if you run    btrfs-image -r     on a dump from a multi device
fs, you get 1 big fs image. I once did that for a 4x4TB RAID10 fs
(tools v4.3.x at that time I believe), resulting in a 17TB (sparse)
file. I was expecting that option  -m would create multiple files,
however, scanning the source-code revealed that there are things not
implemented (it resulted in a 34T file that was simply not a valid
fs). Or I did something wrong or there is a bug.

For just my case, it was much quicker to patch the kernel so that it
worked with the 17T file. There are then still issues w.r.t. devices,
but data is missing so anyhow only a limited set of tool actions or
issues can be researched with such a generated image. But for a
multi-TB fs, the data volume is acceptable (roughly in 1G or 10G
order).

I think it would make sense that the btrfs-image restore output can be
split into multiple files, so that the multidevice aspects are better
represented (or modelled).

>> 2) would btrfs-image can be modified to keep only given subvolume foot
>> print and related meta data to bring back file system live on destination
>> disk?
>>
>>    To elaborate more on this, Lets say I have 5 subvolumes on source btrfs
>> and i run btrfs-image written to destination disk say /dev/sdd. In this
>> process, can btrfs-image modified to just have only 1 subvolume and skipp
>> other 4 subvolumes and write to destination i.. /dev/sdd so that when I
>> mount /dev/sdd , I will have btrfs with only 1 subvolume with no data.
>
>    For a first approximation, you could just drop any FS tree from the
> image which wasn't the target one. After that, it turns into a
> complicated accounting exercise to drop all of the back-refs to the
> missing FS trees, and to drop all the extent records for the
> non-shared data and the metadata for the missing FS trees.
>
>    It's probably going to be complicated, and will basically involve
> rewriting most of the image to avoid the metadata you didn't want.
>
>> 3) If 3 can successful, can btrfs-image further changed to include data of
>> selected subvolume which gives files data also written to /dev/sdd which
>> would be kind of a backup of a subvolume taken out of a btrfs file system
>> which is having more than 1 subvolumes.
>
>    If you're going to do all the hard work of (2), then (3) is a
> reasonable logical(?) extension.
>
>    On the other hand, what's wrong with simply using send/receive? It
> gives you a data structure (a FAR-format send stream) which contains
> everything you need to reconstruct a subvolume on a btrfs different
> to the original.
>
>    Hugo.
>
> --
> Hugo Mills             | Mary had a little lamb
> hugo@... carfax.org.uk | You've heard this tale before
> http://carfax.org.uk/  | But did you know she passed her plate
> PGP: E2AB1DE4          | And had a little more?

Re: btrfs-image and btrfs send related queries

[sri]

I tried btrfs-image and created image file and ran btrfs-image -r to a 
different disk. Once recovered and mounted, I can able to see data is 
not zeroed out as mentioned in btrfs-image man page.

I tried on same machine.

Re: btrfs-image and btrfs send related queries

[Henk Slager]

(your email keeps ending up in gmail spam folder)

On Mon, Apr 18, 2016 at 9:24 AM, sri <toyours_sridhar@yahoo.co.in> wrote:
> I tried btrfs-image and created image file and ran btrfs-image -r to a
> different disk. Once recovered and mounted, I can able to see data is
> not zeroed out as mentioned in btrfs-image man page.

"different disk"  you mention, that is important info. If you doe the
restore to a image file, that image file is sparse and all data blocks
are read as zeros.

However, if you restore to a block device, then you can assume it just
writes the device blocks for metadata and leaves the rest untouched.
So trim whole device first or brute-force overwrite completely with
zeros.

So maybe the man pages needs some correction / extra notes.

> I tried on same machine.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: btrfs-image and btrfs send related queries

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1200 bytes --]

On Mon, 18 Apr 2016 16:13:28 +0200
Henk Slager <eye1tm@gmail.com> wrote:

> (your email keeps ending up in gmail spam folder)
> 
> On Mon, Apr 18, 2016 at 9:24 AM, sri <toyours_sridhar@yahoo.co.in> wrote:
> > I tried btrfs-image and created image file and ran btrfs-image -r to a
> > different disk. Once recovered and mounted, I can able to see data is
> > not zeroed out as mentioned in btrfs-image man page.
> 
> "different disk"  you mention, that is important info. If you doe the
> restore to a image file, that image file is sparse and all data blocks
> are read as zeros.
> 
> However, if you restore to a block device, then you can assume it just
> writes the device blocks for metadata and leaves the rest untouched.
> So trim whole device first or brute-force overwrite completely with
> zeros.
> 
> So maybe the man pages needs some correction / extra notes.
> 
> > I tried on same machine.

Does btrfs-image store/restore the FS UUID? If it does, then potentially both
the source FS and the restored one were visible at the same time to the kernel
with identical UUIDs, and maybe it was actually accessing/mounting the source
one.

-- 
With respect,
Roman

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: btrfs-image and btrfs send related queries

[Henk Slager]

On Mon, Apr 18, 2016 at 4:26 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Mon, 18 Apr 2016 16:13:28 +0200
> Henk Slager <eye1tm@gmail.com> wrote:
>
>> (your email keeps ending up in gmail spam folder)
>>
>> On Mon, Apr 18, 2016 at 9:24 AM, sri <toyours_sridhar@yahoo.co.in> wrote:
>> > I tried btrfs-image and created image file and ran btrfs-image -r to a
>> > different disk. Once recovered and mounted, I can able to see data is
>> > not zeroed out as mentioned in btrfs-image man page.
>>
>> "different disk"  you mention, that is important info. If you doe the
>> restore to a image file, that image file is sparse and all data blocks
>> are read as zeros.
>>
>> However, if you restore to a block device, then you can assume it just
>> writes the device blocks for metadata and leaves the rest untouched.
>> So trim whole device first or brute-force overwrite completely with
>> zeros.
>>
>> So maybe the man pages needs some correction / extra notes.
>>
>> > I tried on same machine.
>
> Does btrfs-image store/restore the FS UUID? If it does, then potentially both
> the source FS and the restored one were visible at the same time to the kernel
> with identical UUIDs, and maybe it was actually accessing/mounting the source
> one.

Very good point! The UUID's are the same. I remember I used a VM to
separate the source FS from the restored FS

Also, the assumption I made about restoring to a block device is not
correct: If you restore to a loopdev that has a file with all
non-zeros as backing-store, the files in the mounted restored FS are
read as zeros.

Re: btrfs-image and btrfs send related queries

[sri]

Henk Slager <eye1tm <at> gmail.com> writes:

> 
> On Mon, Apr 18, 2016 at 4:26 PM, Roman Mamedov <rm <at> romanrm.net> 
wrote:
> > On Mon, 18 Apr 2016 16:13:28 +0200
> > Henk Slager <eye1tm <at> gmail.com> wrote:
> >
> >> (your email keeps ending up in gmail spam folder)
> >>
> >> On Mon, Apr 18, 2016 at 9:24 AM, sri <toyours_sridhar <at> 
yahoo.co.in> wrote:
> >> > I tried btrfs-image and created image file and ran btrfs-image -r 
to a
> >> > different disk. Once recovered and mounted, I can able to see 
data is
> >> > not zeroed out as mentioned in btrfs-image man page.
> >>
> >> "different disk"  you mention, that is important info. If you doe 
the
> >> restore to a image file, that image file is sparse and all data 
blocks
> >> are read as zeros.
> >>
> >> However, if you restore to a block device, then you can assume it 
just
> >> writes the device blocks for metadata and leaves the rest 
untouched.
> >> So trim whole device first or brute-force overwrite completely with
> >> zeros.
> >>
> >> So maybe the man pages needs some correction / extra notes.
> >>
> >> > I tried on same machine.
> >
> > Does btrfs-image store/restore the FS UUID? If it does, then 
potentially both
> > the source FS and the restored one were visible at the same time to 
the kernel
> > with identical UUIDs, and maybe it was actually accessing/mounting 
the source
> > one.
> 
> Very good point! The UUID's are the same. I remember I used a VM to
> separate the source FS from the restored FS
> 
> Also, the assumption I made about restoring to a block device is not
> correct: If you restore to a loopdev that has a file with all
> non-zeros as backing-store, the files in the mounted restored FS are
> read as zeros.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" 
in
> the body of a message to majordomo <at> vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
thank you for inputs. Actually I have tried in the following way.
I have /dev/sdb , /dev/sdc. Using wipefs -fa I cleared both devices and 
cre=
ated btrfs on /dev/sdb. Mounted and written some files and unmounted it.
Then I ran btrfs-image /dev/sdc /img1.img and got the dump.
Once image created I again ran wipefs -fa /dev/sdb.
Then I ran btrfs-image -r /img1.img /dev/sdc and mounted /dev/sdc.
ls to dumped filesystem shows the file size as original and no file 
content=
. I guess btrfs-image doesn't modify files stat so i feel it is showing 
siz=
e as original.
However running cat on some of files give i/o error
qd67:/btrfs1 # cat shadow.hcat: shadow.h: Input/output error
These errors are not on all files on other files, since dump doesn't 
contai=
ns any data it just prompts for cat as below.
qd67:/btrfs1 # cat stab.hqd67:/btrfs1 # cat stdio_ext.h
Not sure why i/o errors are coming.

Re: btrfs-image and btrfs send related queries

[Henk Slager]

> I have /dev/sdb , /dev/sdc. Using wipefs -fa I cleared both devices and
> created btrfs on /dev/sdb. Mounted and written some files and unmounted it.
>
> Then I ran btrfs-image /dev/sdc /img1.img and got the dump.

It looks like you imaged the wrong device, that might clarify the IO
errors later on

> Once image created I again ran wipefs -fa /dev/sdb.
>
> Then I ran btrfs-image -r /img1.img /dev/sdc and mounted /dev/sdc.
>
> ls to dumped filesystem shows the file size as original and no file content.
> I guess btrfs-image doesn't modify files stat so i feel it is showing size
> as original.
>
> However running cat on some of files give i/o error
>
> qd67:/btrfs1 # cat shadow.h
> cat: shadow.h: Input/output error
>
> These errors are not on all files on other files, since dump doesn't
> contains any data it just prompts for cat as below.
>
> qd67:/btrfs1 # cat stab.h
> qd67:/btrfs1 # cat stdio_ext.h
>
> Not sure why i/o errors are coming.

Re: btrfs-image and btrfs send related queries

[sri]

Sorry. Its typo I used original disk /dev/sdb where filesystem is 
created and seeing these errors.