LVM vs btrfs as a "volume manager" for SANs

LVM vs btrfs as a "volume manager" for SANs

[Tomasz Chmielewski]

Right now, the majority of Linux users probably have LVM on their SAN 
devices (i.e those being iSCSI targets).

Using LVM on a SAN device is easy: just create a new logical volume or 
its snapshot, make it a target to iSCSI initiators, done.

I was wondering how btrfs would fit here and if it could replace LVM.


I see the following benefits of using btrfs instead of LVM:

- you can create sparse files which would grow as iSCSI initators use 
more space (you can do it with ext3 now as well)

- you can use btrfs compression, to further reduce used space and 
perhaps increase speed (SANs are mostly IO bound, not CPU bound)

- LVM has a big performance hit when using snapshots; btrfs doesn't



However, with btrfs, I'm not sure about:

- what happens if SAN machine crashes while the iSCSI file images were 
being written to; with LVM and its block devices, I'm somehow more 
confident it wouldn't make more data loss than necessary

- taking snapshots of individual files (file images on SAN) is not 
possible with btrfs? Probably they would have to be placed in separate 
directories first to make snapshots - some minor manageability issue


-- 
Tomasz Chmielewski
http://wpkg.org

Re: LVM vs btrfs as a "volume manager" for SANs

[Dmitri Nikulin]

On Thu, Apr 23, 2009 at 5:20 AM, Tomasz Chmielewski <mangoo@wpkg.org> wrote:
> - what happens if SAN machine crashes while the iSCSI file images were being
> written to; with LVM and its block devices, I'm somehow more confident it
> wouldn't make more data loss than necessary

I would hope that COW applies to such a setting. It certainly does for
ZFS, making it an excellent backend for iSCSI. At least, having it on
btrfs shouldn't make it any less reliable than on LVM, as long as
btrfs does its job correctly.

-- 
Dmitri Nikulin

Centre for Synchrotron Science
Monash University
Victoria 3800, Australia

Re: LVM vs btrfs as a "volume manager" for SANs

[Chris Mason]

On Wed, 2009-04-22 at 21:20 +0200, Tomasz Chmielewski wrote:
> Right now, the majority of Linux users probably have LVM on their SAN 
> devices (i.e those being iSCSI targets).
> 
> Using LVM on a SAN device is easy: just create a new logical volume or 
> its snapshot, make it a target to iSCSI initiators, done.
> 

This is definitely one of the target use cases for btrfs.

> I was wondering how btrfs would fit here and if it could replace LVM.
> 
> 
> I see the following benefits of using btrfs instead of LVM:
> 
> - you can create sparse files which would grow as iSCSI initators use 
> more space (you can do it with ext3 now as well)
>
> - you can use btrfs compression, to further reduce used space and 
> perhaps increase speed (SANs are mostly IO bound, not CPU bound)
> 
> - LVM has a big performance hit when using snapshots; btrfs doesn't
> 
> 
> 
> However, with btrfs, I'm not sure about:
> 
> - what happens if SAN machine crashes while the iSCSI file images were 
> being written to; with LVM and its block devices, I'm somehow more 
> confident it wouldn't make more data loss than necessary

If iscsi is writing with O_DIRECT|O_SYNC it should work.  But, tuning
for this config is something we have to concentrate more on.

> 
> - taking snapshots of individual files (file images on SAN) is not 
> possible with btrfs? Probably they would have to be placed in separate 
> directories first to make snapshots - some minor manageability issue

Btrfs can't snapshot a single dir, but it can snapshot a single file.
See the bcp command included with btrfs-progs.

I'd also suggest using preallocated files (via fallocate) instead of
sparse files.  I will perform better in general.

-chris

Re: LVM vs btrfs as a "volume manager" for SANs

[Tomasz Chmielewski]

Chris Mason schrieb:

>> However, with btrfs, I'm not sure about:
>>
>> - what happens if SAN machine crashes while the iSCSI file images were 
>> being written to; with LVM and its block devices, I'm somehow more 
>> confident it wouldn't make more data loss than necessary
> 
> If iscsi is writing with O_DIRECT|O_SYNC it should work.

What if it doesn't?


> But, tuning
> for this config is something we have to concentrate more on.
> 
>> - taking snapshots of individual files (file images on SAN) is not 
>> possible with btrfs? Probably they would have to be placed in separate 
>> directories first to make snapshots - some minor manageability issue
> 
> Btrfs can't snapshot a single dir, but it can snapshot a single file.
> See the bcp command included with btrfs-progs.
> 
> I'd also suggest using preallocated files (via fallocate) instead of
> sparse files.  I will perform better in general.

When performance is needed - sure.


-- 
Tomasz Chmielewski
http://wpkg.org

Re: LVM vs btrfs as a "volume manager" for SANs

[Chris Mason]

On Fri, 2009-04-24 at 01:34 +0200, Tomasz Chmielewski wrote:
> Chris Mason schrieb:
> 
> >> However, with btrfs, I'm not sure about:
> >>
> >> - what happens if SAN machine crashes while the iSCSI file images were 
> >> being written to; with LVM and its block devices, I'm somehow more 
> >> confident it wouldn't make more data loss than necessary
> > 
> > If iscsi is writing with O_DIRECT|O_SYNC it should work.
> 
> What if it doesn't?
> 

Writes would go to the page cache only (not O_DIRECT) and the metadata
wouldn't get flushed with each write (not O_SYNC).

-chris

Re: LVM vs btrfs as a "volume manager" for SANs

[Tomasz Chmielewski]

Chris Mason wrote:
> On Fri, 2009-04-24 at 01:34 +0200, Tomasz Chmielewski wrote:
>> Chris Mason schrieb:
>>
>>>> However, with btrfs, I'm not sure about:
>>>>
>>>> - what happens if SAN machine crashes while the iSCSI file images were 
>>>> being written to; with LVM and its block devices, I'm somehow more 
>>>> confident it wouldn't make more data loss than necessary
>>> If iscsi is writing with O_DIRECT|O_SYNC it should work.
>> What if it doesn't?
>>
> 
> Writes would go to the page cache only (not O_DIRECT) and the metadata
> wouldn't get flushed with each write (not O_SYNC).

What about the "administrative" part of using btrfs as a volume manager?

For example, I can partition (fdisk + kpartx), extend volumes in LVM 
easily, on the host.


Are there any tools to "partition" or "extend" file images?

qemu-nbd and nbd-client come to mind, but it's a bit of an overhead (in 
typing, I mean).


-- 
Tomasz Chmielewski
http://wpkg.org

Re: LVM vs btrfs as a "volume manager" for SANs

[Chris Mason]

On Thu, Jul 02, 2009 at 05:22:24PM +0200, Tomasz Chmielewski wrote:
> Chris Mason wrote:
>> On Fri, 2009-04-24 at 01:34 +0200, Tomasz Chmielewski wrote:
>>> Chris Mason schrieb:
>>>
>>>>> However, with btrfs, I'm not sure about:
>>>>>
>>>>> - what happens if SAN machine crashes while the iSCSI file images 
>>>>> were being written to; with LVM and its block devices, I'm 
>>>>> somehow more confident it wouldn't make more data loss than 
>>>>> necessary
>>>> If iscsi is writing with O_DIRECT|O_SYNC it should work.
>>> What if it doesn't?
>>>
>>
>> Writes would go to the page cache only (not O_DIRECT) and the metadata
>> wouldn't get flushed with each write (not O_SYNC).
>
> What about the "administrative" part of using btrfs as a volume manager?

In this case it depends on exactly how you do it.  If you're just using
files and putting filesystems on those files (not partitioning inside
the image), then you can use truncate to shrink the image and fallocate
(or truncate if you want it sparse) to extend it.

If you want to make partition tables inside the volumes, then you need
to do the truncate/fallocate game and also change the partition table.

-chris