SSD caching an existing btrfs raid1

SSD caching an existing btrfs raid1

[Pat Sailor]

Hello,

I have a half-filled raid1 on top of six spinning devices. Now I have 
come into a spare SSD I'd like to use for caching, if possible without 
having to rebuild or, failing that, without having to renounce to btrfs 
and flexible reshaping.

I've been reading about the several options out there; I thought that 
EnhanceIO would be the simplest bet but unfortunately I couldn't get it 
to build with my recent kernel (last commits are from years ago).

Failing that, I read that lvmcache could be the way to go. However, I 
can't think of a way of setting it up in which I retain the ability to 
add/remove/replace drives as I can do now with pure btrfs; if I opted to 
drop btrfs to go to ext4 I still would have to offline the filesystem 
for downsizes. Not a frequent occurrence I hope, but now I'm used to 
keep working while I reshape things in btrfs, and it's better if I can 
avoid large downtimes.

Is what I want doable at all? Thanks in advance for any 
suggestions/experiences to proceed.

Re: SSD caching an existing btrfs raid1

[Austin S. Hemmelgarn]

On 2017-09-19 11:30, Pat Sailor wrote:
> Hello,
> 
> I have a half-filled raid1 on top of six spinning devices. Now I have 
> come into a spare SSD I'd like to use for caching, if possible without 
> having to rebuild or, failing that, without having to renounce to btrfs 
> and flexible reshaping.
> 
> I've been reading about the several options out there; I thought that 
> EnhanceIO would be the simplest bet but unfortunately I couldn't get it 
> to build with my recent kernel (last commits are from years ago).
As a general rule, avoid things that are out of tree if you want any 
chance of support here.  In the case of a third party module like 
EnhanceIO, you'll likely get asked to retest with a kernel without that 
module loaded if you run into a bug (because the module itself is suspect).
> 
> Failing that, I read that lvmcache could be the way to go. However, I 
> can't think of a way of setting it up in which I retain the ability to 
> add/remove/replace drives as I can do now with pure btrfs; if I opted to 
> drop btrfs to go to ext4 I still would have to offline the filesystem 
> for downsizes. Not a frequent occurrence I hope, but now I'm used to 
> keep working while I reshape things in btrfs, and it's better if I can 
> avoid large downtimes.
> 
> Is what I want doable at all? Thanks in advance for any 
> suggestions/experiences to proceed.
dm-cache (or lvmcache as the LVM2 developers want you to call it, 
despite it not being an LVM specific thing) would work fine, it won't 
prevent you from adding and removing devices, it would just make it more 
complicated.  Without it, you just need to issue a replace command (or 
add then remove).  With it, you need to set up the new cache device, 
bind that to the target, add the new device, and then delete the old 
device and remove the old cache target.  dm-cache managed through LVM 
also has the advantage that you can convert the existing FS trivially, 
although you will have to take it off-line for the conversion.

A better option if you can afford to remove a single device from that 
array temporarily is to use bcache.  Bcache has one specific advantage 
in this case, multiple backend devices can share the same cache device. 
This means you don't have to carve out dedicated cache space for each 
disk on the SSD and leave some unused space so that you can add new 
devices if needed.  The downside is that you can't convert each device 
in-place, but because you're using BTRFS, you can still convert the 
volume as a whole in-place.  The procedure for doing so looks like this:

1. Format the SSD as a bcache cache.
2. Use `btrfs device delete` to remove a single hard drive from the array.
3. Set up the drive you just removed as a bcache backing device bound to 
the cache you created in step 1.
4. Add the new bcache device to the array.
5. Repeat from step 2 until the whole array is converted.

A similar procedure can actually be used to do almost any underlying 
storage conversion (for example, switching to whole disk encryption, or 
adding LVM underneath BTRFS) provided all your data can fit on one less 
disk than you have.

Re: SSD caching an existing btrfs raid1

[Duncan]

Austin S. Hemmelgarn posted on Tue, 19 Sep 2017 11:47:24 -0400 as
excerpted:

> A better option if you can afford to remove a single device from that
> array temporarily is to use bcache.  Bcache has one specific advantage
> in this case, multiple backend devices can share the same cache device.
> This means you don't have to carve out dedicated cache space for each
> disk on the SSD and leave some unused space so that you can add new
> devices if needed.  The downside is that you can't convert each device
> in-place, but because you're using BTRFS, you can still convert the
> volume as a whole in-place.  The procedure for doing so looks like this:
> 
> 1. Format the SSD as a bcache cache.
> 2. Use `btrfs device delete` to remove a single hard drive from the
> array.
> 3. Set up the drive you just removed as a bcache backing device bound to
> the cache you created in step 1.
> 4. Add the new bcache device to the array.
> 5. Repeat from step 2 until the whole array is converted.
> 
> A similar procedure can actually be used to do almost any underlying
> storage conversion (for example, switching to whole disk encryption, or
> adding LVM underneath BTRFS) provided all your data can fit on one less
> disk than you have.

... And you're not already at the minimum operational for that btrfs 
array type.

For instance, I have lots of btrfs raid1s, two devices each.  I'd have 
trouble with the above, because I can't simply remove a device, despite 
all the data and metadata fitting on a single device.

I'd have three options that /would/ work, however.

1) If I was willing to risk having only a single checksummed copy of 
everything for the time it would take to process, I could do a btrfs 
balance convert from raid1 to single mode, then do a btrfs device remove, 
and go from there, converting back to raid1 after I was done.  Of course 
if anything goes wrong with that single copy and something fails 
checksum, I've potentially lost whatever failed the checksum unless I 
have it on backup, so converting to single mode is risky.

2) If the data and metadata will fit on /half/ of a single device, then I 
could convert to dup mode (since btrfs now allows dup data chunks) taking 
double the space on that single device, but keeping two checksummed 
copies of everything due to the dup mode, thus avoiding the risk of #1.

3) If #2 won't work due to size and I was unwilling to risk #1, I'd have 
to add a device (making it the first one converted to bcache before the 
add) before I could remove one of the others, in ordered to allow me to 
keep the checksummed redundancy of raid1 for the entire procedure.  After 
the bcache conversion and addition of the new device, and the removal, 
conversion, and readdition of one of the two existing devices, I could 
simply remove the third, giving me back my spare device, but I'd have to 
have and use it for the process.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: SSD caching an existing btrfs raid1

[Psalle]

On 19/09/17 17:47, Austin S. Hemmelgarn wrote:
(...)
>
> A better option if you can afford to remove a single device from that 
> array temporarily is to use bcache.  Bcache has one specific advantage 
> in this case, multiple backend devices can share the same cache 
> device. This means you don't have to carve out dedicated cache space 
> for each disk on the SSD and leave some unused space so that you can 
> add new devices if needed.  The downside is that you can't convert 
> each device in-place, but because you're using BTRFS, you can still 
> convert the volume as a whole in-place.  The procedure for doing so 
> looks like this:
>
> 1. Format the SSD as a bcache cache.
> 2. Use `btrfs device delete` to remove a single hard drive from the 
> array.
> 3. Set up the drive you just removed as a bcache backing device bound 
> to the cache you created in step 1.
> 4. Add the new bcache device to the array.
> 5. Repeat from step 2 until the whole array is converted.
>
> A similar procedure can actually be used to do almost any underlying 
> storage conversion (for example, switching to whole disk encryption, 
> or adding LVM underneath BTRFS) provided all your data can fit on one 
> less disk than you have.

Thanks Austin, that's just great. For some reason I had discarded bcache 
thinking that it would force me to rebuild from scratch, but this kind 
of incremental migration is exactly why I hoped was possible. I have 
plenty of space to replace the devices one by one.

I will report back my experience in a few days, I hope.

Re: SSD caching an existing btrfs raid1

[Kai Krakow]

Am Wed, 20 Sep 2017 17:51:15 +0200
schrieb Psalle <psalleetsile@gmail.com>:

> On 19/09/17 17:47, Austin S. Hemmelgarn wrote:
> (...)
> >
> > A better option if you can afford to remove a single device from
> > that array temporarily is to use bcache.  Bcache has one specific
> > advantage in this case, multiple backend devices can share the same
> > cache device. This means you don't have to carve out dedicated
> > cache space for each disk on the SSD and leave some unused space so
> > that you can add new devices if needed.  The downside is that you
> > can't convert each device in-place, but because you're using BTRFS,
> > you can still convert the volume as a whole in-place.  The
> > procedure for doing so looks like this:
> >
> > 1. Format the SSD as a bcache cache.
> > 2. Use `btrfs device delete` to remove a single hard drive from the 
> > array.
> > 3. Set up the drive you just removed as a bcache backing device
> > bound to the cache you created in step 1.
> > 4. Add the new bcache device to the array.
> > 5. Repeat from step 2 until the whole array is converted.
> >
> > A similar procedure can actually be used to do almost any
> > underlying storage conversion (for example, switching to whole disk
> > encryption, or adding LVM underneath BTRFS) provided all your data
> > can fit on one less disk than you have.  
> 
> Thanks Austin, that's just great. For some reason I had discarded
> bcache thinking that it would force me to rebuild from scratch, but
> this kind of incremental migration is exactly why I hoped was
> possible. I have plenty of space to replace the devices one by one.
> 
> I will report back my experience in a few days, I hope.

I've done it exactly that way in the past and it worked flawlessly (but
it took 24+ hours). But it was easy for me because I was also adding a
third disk to the pool, so existing stuff could easily move.

I suggest to initialize bcache to writearound mode while converting, so
your maybe terabytes of disk don't go through the SSD.

If you later decide to remove bcache or not sure about future bcache
usage, you can wrap any partition into a bcache container - just don't
connect it to a cache and it will work like a normal partition.


-- 
Regards,
Kai

Replies to list-only preferred.

RE: SSD caching an existing btrfs raid1

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 2475 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Kai Krakow
> Sent: Thursday, 21 September 2017 6:45 AM
> To: linux-btrfs@vger.kernel.org
> Subject: Re: SSD caching an existing btrfs raid1
> 
> Am Wed, 20 Sep 2017 17:51:15 +0200
> schrieb Psalle <psalleetsile@gmail.com>:
> 
> > On 19/09/17 17:47, Austin S. Hemmelgarn wrote:
> > (...)
> > >
> > > A better option if you can afford to remove a single device from
> > > that array temporarily is to use bcache.  Bcache has one specific
> > > advantage in this case, multiple backend devices can share the same
> > > cache device. This means you don't have to carve out dedicated cache
> > > space for each disk on the SSD and leave some unused space so that
> > > you can add new devices if needed.  The downside is that you can't
> > > convert each device in-place, but because you're using BTRFS, you
> > > can still convert the volume as a whole in-place.  The procedure for
> > > doing so looks like this:
> > >
> > > 1. Format the SSD as a bcache cache.
> > > 2. Use `btrfs device delete` to remove a single hard drive from the
> > > array.
> > > 3. Set up the drive you just removed as a bcache backing device
> > > bound to the cache you created in step 1.
> > > 4. Add the new bcache device to the array.
> > > 5. Repeat from step 2 until the whole array is converted.
> > >
> > > A similar procedure can actually be used to do almost any underlying
> > > storage conversion (for example, switching to whole disk encryption,
> > > or adding LVM underneath BTRFS) provided all your data can fit on
> > > one less disk than you have.
> >
> > Thanks Austin, that's just great. For some reason I had discarded
> > bcache thinking that it would force me to rebuild from scratch, but
> > this kind of incremental migration is exactly why I hoped was
> > possible. I have plenty of space to replace the devices one by one.
> >
> > I will report back my experience in a few days, I hope.
> 
> I've done it exactly that way in the past and it worked flawlessly (but it took
> 24+ hours). But it was easy for me because I was also adding a third disk to
> the pool, so existing stuff could easily move.

Device delete takes freaking ages! I would avoid using it if you can. Device replace is much faster.


Paul.




ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm

Re: SSD caching an existing btrfs raid1

[Psalle]

On 20/09/17 22:45, Kai Krakow wrote:
> Am Wed, 20 Sep 2017 17:51:15 +0200
> schrieb Psalle <psalleetsile@gmail.com>:
>
>> On 19/09/17 17:47, Austin S. Hemmelgarn wrote:
>> (...)
>>> A better option if you can afford to remove a single device from
>>> that array temporarily is to use bcache.  Bcache has one specific
>>> advantage in this case, multiple backend devices can share the same
>>> cache device. This means you don't have to carve out dedicated
>>> cache space for each disk on the SSD and leave some unused space so
>>> that you can add new devices if needed.  The downside is that you
>>> can't convert each device in-place, but because you're using BTRFS,
>>> you can still convert the volume as a whole in-place.  The
>>> procedure for doing so looks like this:
>>>
>>> 1. Format the SSD as a bcache cache.
>>> 2. Use `btrfs device delete` to remove a single hard drive from the
>>> array.
>>> 3. Set up the drive you just removed as a bcache backing device
>>> bound to the cache you created in step 1.
>>> 4. Add the new bcache device to the array.
>>> 5. Repeat from step 2 until the whole array is converted.
>>>
>>> A similar procedure can actually be used to do almost any
>>> underlying storage conversion (for example, switching to whole disk
>>> encryption, or adding LVM underneath BTRFS) provided all your data
>>> can fit on one less disk than you have.
>> Thanks Austin, that's just great. For some reason I had discarded
>> bcache thinking that it would force me to rebuild from scratch, but
>> this kind of incremental migration is exactly why I hoped was
>> possible. I have plenty of space to replace the devices one by one.
>>
>> I will report back my experience in a few days, I hope.
> I've done it exactly that way in the past and it worked flawlessly (but
> it took 24+ hours). But it was easy for me because I was also adding a
> third disk to the pool, so existing stuff could easily move.
>
> I suggest to initialize bcache to writearound mode while converting, so
> your maybe terabytes of disk don't go through the SSD.
>
> If you later decide to remove bcache or not sure about future bcache
> usage, you can wrap any partition into a bcache container - just don't
> connect it to a cache and it will work like a normal partition.

Those are good advices. I've finished now and it seems to have gone 
without a hitch. Thanks!

>
>

RE: SSD caching an existing btrfs raid1

[Paul Jones]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="utf-8", Size: 1867 bytes --]

> -----Original Message-----
> From: linux-btrfs-owner@vger.kernel.org [mailto:linux-btrfs-
> owner@vger.kernel.org] On Behalf Of Pat Sailor
> Sent: Wednesday, 20 September 2017 1:31 AM
> To: Btrfs BTRFS <linux-btrfs@vger.kernel.org>
> Subject: SSD caching an existing btrfs raid1
> 
> Hello,
> 
> I have a half-filled raid1 on top of six spinning devices. Now I have come into
> a spare SSD I'd like to use for caching, if possible without having to rebuild or,
> failing that, without having to renounce to btrfs and flexible reshaping.
> 
> I've been reading about the several options out there; I thought that
> EnhanceIO would be the simplest bet but unfortunately I couldn't get it to
> build with my recent kernel (last commits are from years ago).
> 
> Failing that, I read that lvmcache could be the way to go. However, I can't
> think of a way of setting it up in which I retain the ability to
> add/remove/replace drives as I can do now with pure btrfs; if I opted to drop
> btrfs to go to ext4 I still would have to offline the filesystem for downsizes.
> Not a frequent occurrence I hope, but now I'm used to keep working while I
> reshape things in btrfs, and it's better if I can avoid large downtimes.
> 
> Is what I want doable at all? Thanks in advance for any
> suggestions/experiences to proceed.

When I did mine I used a spare disk to create the initial LVM filesystem, then used btrfs dev replace to swap one of the raid1 mirrors to the new lvm device. Then repeat for the other mirror.
My suggestion for lvm setup is to use two different pools, one for each btrfs mirror. That ensures you don't accidently have btrfs sharing the one physical disk by mistake, or lvm using the same SSD to cache the two discs.

Paul.
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±ý»k~ÏâžØ^n‡r¡ö¦zË\x1aëh™¨è­Ú&£ûàz¿äz¹Þ—ú+€Ê+zf£¢·hšˆ§~†­†Ûiÿÿïêÿ‘êçz_è®\x0fæj:+v‰¨þ)ߣøm