multiple device usage

multiple device usage

[devzero]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

hello,

i have some difficulty in understanding multi-device handling in depth.

as http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices 
tells,

btrfs can span over multiple devices at the same time. (great feature, btw !)

ok then:
 
mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs spanning 
over /dev/sdb and sdc

mount /dev/sdb /btrfs - mounts it

btrfs-vol -b /btrfs - does a rebalancing of all data and metadata

btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes any 
extends in use on sdc to sdb (killer feature!!!)

but what if i want to remove /dev/sdb ? (as that one is in use for the 
mount)

regards
roland
__________________________________________________________________________
Verschicken Sie SMS direkt vom Postfach aus - in alle deutschen und viele 
ausländische Netze zum gleichen Preis! 
https://produkte.web.de/webde_sms/sms




[-- Attachment #2: Type: message/delivery-status, Size: 488 bytes --]

Re: multiple device usage

[Yan Zheng]

devzero@web.de wrote:
> hello,
> 
> i have some difficulty in understanding multi-device handling in depth.
> 
> as http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices 
> tells,
> 
> btrfs can span over multiple devices at the same time. (great feature, btw !)
> 
> ok then:
>  
> mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs spanning 
> over /dev/sdb and sdc
> 
> mount /dev/sdb /btrfs - mounts it
> 
> btrfs-vol -b /btrfs - does a rebalancing of all data and metadata
> 
> btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes any 
> extends in use on sdc to sdb (killer feature!!!)
> 
> but what if i want to remove /dev/sdb ? (as that one is in use for the 
> mount)
>  

Devices in btrfs are equal, so you can do this. The only glitch is
/proc/mounts and mount(8) get confused.

Regards
Yan Zheng 

Re: multiple device usage

[Chris Samuel]

[-- Attachment #1: Type: text/plain, Size: 847 bytes --]

On Sat, 27 Dec 2008 10:15:18 am devzero@web.de wrote:

> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices

I'll add two questions that're not answered by the Wiki too.. :-)

1) If you add a second disk to an existing btrfs filesystem, can you get it to 
set it up as a RAID-1 arrangement rather than just rebalancing the metadata 
and then striping ?

2) With the concerns that people have about SSD reliability (hi Val :-) !) 
would it make sense to set up two equal sized partitions on the SSD and use 
RAID-1 across them, or can you tell btrfs to keep multiple copies of the data, 
a la ZFS ?

cheers!
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 481 bytes --]

Re: multiple device usage

[Roland]

>> i have some difficulty in understanding multi-device handling in depth.
>>
>> as 
>> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
>> tells,
>>
>> btrfs can span over multiple devices at the same time. (great feature, 
>> btw !)
>>
>> ok then:
>>
>> mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs 
>> spanning
>> over /dev/sdb and sdc
>>
>> mount /dev/sdb /btrfs - mounts it
>>
>> btrfs-vol -b /btrfs - does a rebalancing of all data and metadata
>>
>> btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes any
>> extends in use on sdc to sdb (killer feature!!!)
>>
>> but what if i want to remove /dev/sdb ? (as that one is in use for the
>> mount)
>>
>
> Devices in btrfs are equal, so you can do this. The only glitch is
> /proc/mounts and mount(8) get confused.

so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk , 
replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk and 
then re-add - all while mount telling me, that /dev/sdb is still in use !? 

Re: multiple device usage

[yanhai zhu]

> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk ,
> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk and
                                                 ~~~~~~~~~~~~~~~~~~~~~~~
                                                This step will fail, you will
get a "/dev/sdb is mounted" by mkfs.btrfs, but for other slots it's ok.

> then re-add - all while mount telling me, that /dev/sdb is still in use !?

2008/12/27 Roland <devzero@web.de>:
>>> i have some difficulty in understanding multi-device handling in depth.
>>>
>>> as
>>> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
>>> tells,
>>>
>>> btrfs can span over multiple devices at the same time. (great feature,
>>> btw !)
>>>
>>> ok then:
>>>
>>> mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs
>>> spanning
>>> over /dev/sdb and sdc
>>>
>>> mount /dev/sdb /btrfs - mounts it
>>>
>>> btrfs-vol -b /btrfs - does a rebalancing of all data and metadata
>>>
>>> btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes any
>>> extends in use on sdc to sdb (killer feature!!!)
>>>
>>> but what if i want to remove /dev/sdb ? (as that one is in use for the
>>> mount)
>>>
>>
>> Devices in btrfs are equal, so you can do this. The only glitch is
>> /proc/mounts and mount(8) get confused.
>
> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk ,
> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk and
> then re-add - all while mount telling me, that /dev/sdb is still in use !?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Regards,
Zhu Yanhai

Re: multiple device usage

[Stephan von Krawczynski]

On Sun, 28 Dec 2008 21:26:11 +0800
"yanhai zhu" <zhu.yanhai@gmail.com> wrote:

> > so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk ,
> > replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk and
>                                                  ~~~~~~~~~~~~~~~~~~~~~~~
>                                                 This step will fail, you will
> get a "/dev/sdb is mounted" by mkfs.btrfs, but for other slots it's ok.

Really, what average-joe-user needs btrfs for his not-always-mounted
data-partition? Why shouldn't he just use any other fs for that?
All users needing _uptime_ of their fs cannot simply unmount and wait hours
for some cool feature to (hopefully) complete (without failures).
Why does nobody talk about the real world? What's the use of features that
cannot complete their job in users' lifetime due to everybody filling up TBs
of data on their harddisks. How long does it really take to replace a full 300
GB hd with a 800 GB in a multi volume btrfs ?
Please play with real amounts of data and judge for yourself the use of
offline-fs-features.
The times of classical PC use are gone, today people have just about
everything on their hds, music collection, dvd collection, mails, documents of
all kinds, sourcecode trees, name-one. 500 GB hds are very cheap. A _new_ fs
not able to cope with these new usage-patterns is in fact _useless_.

Regards,
Stephan


 
> > then re-add - all while mount telling me, that /dev/sdb is still in use !?
> 
> 2008/12/27 Roland <devzero@web.de>:
> >>> i have some difficulty in understanding multi-device handling in depth.
> >>>
> >>> as
> >>> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
> >>> tells,
> >>>
> >>> btrfs can span over multiple devices at the same time. (great feature,
> >>> btw !)
> >>>
> >>> ok then:
> >>>
> >>> mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs
> >>> spanning
> >>> over /dev/sdb and sdc
> >>>
> >>> mount /dev/sdb /btrfs - mounts it
> >>>
> >>> btrfs-vol -b /btrfs - does a rebalancing of all data and metadata
> >>>
> >>> btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes any
> >>> extends in use on sdc to sdb (killer feature!!!)
> >>>
> >>> but what if i want to remove /dev/sdb ? (as that one is in use for the
> >>> mount)
> >>>
> >>
> >> Devices in btrfs are equal, so you can do this. The only glitch is
> >> /proc/mounts and mount(8) get confused.
> >
> > so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk ,
> > replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk and
> > then re-add - all while mount telling me, that /dev/sdb is still in use !?
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> 
> -- 
> Regards,
> Zhu Yanhai
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: multiple device usage

[Chris Samuel]

[-- Attachment #1: Type: text/plain, Size: 1347 bytes --]

On Sat, 27 Dec 2008 5:45:42 pm Chris Samuel wrote:

> I'll add two questions that're not answered by the Wiki too.. :-)

Looking at the source implies to me:

> 1) If you add a second disk to an existing btrfs filesystem, can you get it
> to set it up as a RAID-1 arrangement rather than just rebalancing the
> metadata and then striping ?

No, you can't, the data stripe numbers seem to be set at mkfs time.

The rebalancing code does appear (from a naive read of the code) to be able to 
rebalance over stripes, but I have no idea if the disk format currently 
supports changing that on the fly.

> 2) With the concerns that people have about SSD reliability (hi Val :-) !)
> would it make sense to set up two equal sized partitions on the SSD and use
> RAID-1 across them, or can you tell btrfs to keep multiple copies of the
> data, a la ZFS ?

Again it would appear that you need to have two partitions and that btrfs 
cannot (at present) keep multiple data stripes on the same partition.

Could someone who actually knows the code to comment on whether my inferences 
are accurate or not please ?  :-)

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 481 bytes --]

Re: multiple device usage

[Roland]

>> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk 
>> ,
>> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk 
>> and
>                                                 ~~~~~~~~~~~~~~~~~~~~~~~
>                                                This step will fail, you 
> will
> get a "/dev/sdb is mounted" by mkfs.btrfs, but for other slots it's ok.

isn`t this sort of a design issue then?
no way to work around this ?

i think online data migration and disk replacement is a killer feature of 
btrfs
(even zfs doesn`t have this!), but  if you always have one out of X disks 
which
can`t be used for this, then it`s of no real value, imho.

regards
roland


>
>> then re-add - all while mount telling me, that /dev/sdb is still in use 
>> !?
>
> 2008/12/27 Roland <devzero@web.de>:
>>>> i have some difficulty in understanding multi-device handling in depth.
>>>>
>>>> as
>>>> http://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices
>>>> tells,
>>>>
>>>> btrfs can span over multiple devices at the same time. (great feature,
>>>> btw !)
>>>>
>>>> ok then:
>>>>
>>>> mkfs.btrfs -m single -d single  /dev/sdb /dev/sdc - creates a btrfs
>>>> spanning
>>>> over /dev/sdb and sdc
>>>>
>>>> mount /dev/sdb /btrfs - mounts it
>>>>
>>>> btrfs-vol -b /btrfs - does a rebalancing of all data and metadata
>>>>
>>>> btrfs-vol -r /dev/sdc - removes one of the volumes and redistributes 
>>>> any
>>>> extends in use on sdc to sdb (killer feature!!!)
>>>>
>>>> but what if i want to remove /dev/sdb ? (as that one is in use for the
>>>> mount)
>>>>
>>>
>>> Devices in btrfs are equal, so you can do this. The only glitch is
>>> /proc/mounts and mount(8) get confused.
>>
>> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk 
>> ,
>> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk 
>> and
>> then re-add - all while mount telling me, that /dev/sdb is still in use 
>> !?
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>
>
> -- 
> Regards,
> Zhu Yanhai 

Re: multiple device usage

[Yan Zheng]

2008/12/29 Chris Samuel <chris@csamuel.org>:
> On Sat, 27 Dec 2008 5:45:42 pm Chris Samuel wrote:
>
>> I'll add two questions that're not answered by the Wiki too.. :-)
>
> Looking at the source implies to me:
>
>> 1) If you add a second disk to an existing btrfs filesystem, can you get it
>> to set it up as a RAID-1 arrangement rather than just rebalancing the
>> metadata and then striping ?
>
> No, you can't, the data stripe numbers seem to be set at mkfs time.
>
> The rebalancing code does appear (from a naive read of the code) to be able to
> rebalance over stripes, but I have no idea if the disk format currently
> supports changing that on the fly.
>

The rebalancing moves data/metadata to newly created chunks. If there
are two devices, the new chunks will be set up as RAID-1 by default.

>> 2) With the concerns that people have about SSD reliability (hi Val :-) !)
>> would it make sense to set up two equal sized partitions on the SSD and use
>> RAID-1 across them, or can you tell btrfs to keep multiple copies of the
>> data, a la ZFS ?
>
> Again it would appear that you need to have two partitions and that btrfs
> cannot (at present) keep multiple data stripes on the same partition.
>

Yes. I think this is due to performance reason. Changing the code to
support data duplication in single spindle configuration is easy.

Regards
Yan Zheng

Re: multiple device usage

[Yan Zheng]

2008/12/29 Roland <devzero@web.de>:
>>> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk
>>> ,
>>> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk
>>> and
>>
>>                                                ~~~~~~~~~~~~~~~~~~~~~~~
>>                                               This step will fail, you
>> will
>> get a "/dev/sdb is mounted" by mkfs.btrfs, but for other slots it's ok.
>
> isn`t this sort of a design issue then?
> no way to work around this ?
>

No, this is a (easy to fix) bug.

Regards
Yan Zheng

Re: multiple device usage

[Chris Samuel]

[-- Attachment #1: Type: text/plain, Size: 1492 bytes --]

Hello Yan,

On Mon, 29 Dec 2008 11:33:18 pm Yan Zheng wrote:

> 2008/12/29 Chris Samuel <chris@csamuel.org>:
>
> > The rebalancing code does appear (from a naive read of the code) to be
> > able to rebalance over stripes, but I have no idea if the disk format
> > currently supports changing that on the fly.
>
> The rebalancing moves data/metadata to newly created chunks. If there
> are two devices, the new chunks will be set up as RAID-1 by default.

Very interesting!  I didn't realise that - I was presuming that the 
BTRFS_BLOCK_GROUP_RAID1 ioctl needed to be passed through (as at mkfs time) to 
change it into a RAID1 array.

Could you point me at the code where this change is detected please ?   Just 
so I can try and wrap my mind around it!

> > Again it would appear that you need to have two partitions and that btrfs
> > cannot (at present) keep multiple data stripes on the same partition.
>
> Yes. I think this is due to performance reason.

Understood.

> Changing the code to support data duplication in single spindle configuration
> is easy.

OK, that's encouraging (though for my immediate purposes creating two 
partitions is fine, but it might make life easier for others).

Again thanks so much for the response!

All the best,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 481 bytes --]

Re: multiple device usage

[Yan Zheng]

2008/12/29 Chris Samuel <chris@csamuel.org>:
> Hello Yan,
>
> On Mon, 29 Dec 2008 11:33:18 pm Yan Zheng wrote:
>
>> 2008/12/29 Chris Samuel <chris@csamuel.org>:
>>
>> > The rebalancing code does appear (from a naive read of the code) to be
>> > able to rebalance over stripes, but I have no idea if the disk format
>> > currently supports changing that on the fly.
>>
>> The rebalancing moves data/metadata to newly created chunks. If there
>> are two devices, the new chunks will be set up as RAID-1 by default.
>
> Very interesting!  I didn't realise that - I was presuming that the
> BTRFS_BLOCK_GROUP_RAID1 ioctl needed to be passed through (as at mkfs time) to
> change it into a RAID1 array.
>

See __btrfs_reserve_extent and __btrfs_alloc_chunk.  There is a mistake in
my previous reply. Only metadata chunks are set up as RAID-1 by default
when there are two devices.

Regards
Yan Zheng

Re: multiple device usage

[Chris Mason]

On Mon, 2008-12-29 at 23:16 +0800, Yan Zheng wrote:
> 2008/12/29 Chris Samuel <chris@csamuel.org>:
> > Hello Yan,
> >
> > On Mon, 29 Dec 2008 11:33:18 pm Yan Zheng wrote:
> >
> >> 2008/12/29 Chris Samuel <chris@csamuel.org>:
> >>
> >> > The rebalancing code does appear (from a naive read of the code) to be
> >> > able to rebalance over stripes, but I have no idea if the disk format
> >> > currently supports changing that on the fly.
> >>
> >> The rebalancing moves data/metadata to newly created chunks. If there
> >> are two devices, the new chunks will be set up as RAID-1 by default.
> >
> > Very interesting!  I didn't realise that - I was presuming that the
> > BTRFS_BLOCK_GROUP_RAID1 ioctl needed to be passed through (as at mkfs time) to
> > change it into a RAID1 array.
> >
> 
> See __btrfs_reserve_extent and __btrfs_alloc_chunk.  There is a mistake in
> my previous reply. Only metadata chunks are set up as RAID-1 by default
> when there are two devices.

This gets confusing in a hurry, but the idea is to duplicate metadata by
default.  So, if you're using the default mount options on a single
drive and add a second drive, it should switch metadata to raid1.

I've always planned on adding an option to btrfs-vol -b to have it
change the data or metadata allocation policies (raidX to raidY,
restripe etc).  The kernel side code is all there, we just need
something to wire it up to userland.

data duplication on a single spindle could work, I just haven't hooked
it up because I couldn't think of a really strong reason to do it ;)

-chris

Re: multiple device usage

[Chris Mason]

On Mon, 2008-12-29 at 20:35 +0800, Yan Zheng wrote:
> 2008/12/29 Roland <devzero@web.de>:
> >>> so i can do btrfs-vol -r /dev/sdb while it`s being mounted, pull the disk
> >>> ,
> >>> replace it with a bigger one, rescan-scsi-bus, mkfs.btrfs the new disk
> >>> and
> >>
> >>                                                ~~~~~~~~~~~~~~~~~~~~~~~
> >>                                               This step will fail, you
> >> will
> >> get a "/dev/sdb is mounted" by mkfs.btrfs, but for other slots it's ok.
> >
> > isn`t this sort of a design issue then?
> > no way to work around this ?
> >
> 
> No, this is a (easy to fix) bug.

Right, the check for a mounted FS came after we made the device
add/removal work ;)  mkfs.btrfs should be changed to do the mounted FS
check differently.

-chris

Re: multiple device usage

[Chris Samuel]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

On Wed, 31 Dec 2008 8:25:08 am Chris Mason wrote:

> This gets confusing in a hurry, but the idea is to duplicate metadata by
> default.  So, if you're using the default mount options on a single
> drive and add a second drive, it should switch metadata to raid1.

Yup, that's what I inferred from your earlier posts and the code. :-)

> I've always planned on adding an option to btrfs-vol -b to have it
> change the data or metadata allocation policies (raidX to raidY,
> restripe etc).  The kernel side code is all there, we just need
> something to wire it up to userland.

Understood.  If I get some Copious Spare Time (tm) I might take a look at that 
(though don't let that stop anyone else trying first!).

> data duplication on a single spindle could work, I just haven't hooked
> it up because I couldn't think of a really strong reason to do it ;)

I think for SSD's it could be really handy, especially for people who have a 
standard distro install and decide they want to convert their ext3 partition 
to btrfs using the tools.   It's just a bit easier than getting them to 
backup, repartition, make the filesystem and restore.

Downside of course is that those original blocks never get mirrored (unless 
something like btrfsck was extended to fix those things up).

cheers,
Chris
-- 
 Chris Samuel  :  http://www.csamuel.org/  :  Melbourne, VIC

This email may come with a PGP signature as a file. Do not panic.
For more info see: http://en.wikipedia.org/wiki/OpenPGP


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 481 bytes --]