Re: Suggestions for building new 44TB Raid5 array

[Marc MERLIN <marc@merlins.org>]

Thanks to you both for your kind help. If I'm rebulding everything,
might as well future-proof it as well as possible.

On Mon, Jun 13, 2022 at 11:06:25PM +0500, Roman Mamedov wrote:
> What I mean is bcache in this way stays bcache-without-a-cache forever, which
> feels odd; it still goes through the bcache code, has the module loaded, keeps
> the device name, etc;
 
Fair point. I have done that, but I see what you're saying.

> Whereas in LVM caching is a completely optional side-feature, and many people
> would just run LVM in any case, not even thinking about enabling cache. LVM is
> basically "the next generation" of disk partitions, with way more features,
> but not much more overhead.

Fair enough. I have used LVM for many years, since the now defunct lvm1,
and I've run through a fair amount of issues, some reliability, some
performance. that was many many years ago though, so I'll take your word
for it that it's a lot more lightweight and safe now.

Actually I think I stopped using LVM the same time I started using
btrfs, because effectively btrfs subvolumes were close enough to LVM LVs
for my use, but yes I understand that different LVs are actually
different filesystems and you can do extra stuff like caching.

Actually I have another array where there were so many files and
snapshots that I split it into different LVs with dm-thin so that I
didn't stress the btrfs code too much (which I'm told gets unhappy when
you have hundreds of snapshots).

On Mon, Jun 13, 2022 at 02:10:56PM -0400, Zygo Blaxell wrote:
> You can trivially convert from lvmcache to plain LV on the fly.  It's a
> pretty essential capability for long-term maintenance, since you can't
> move or resize the LV while it's cached.
> 
> If you have a LV and you want it to be cached with bcache, you can hack
> up the LVM configuration after the fact with https://github.com/g2p/blocks
 
Got it, thanks much.

On Mon, Jun 13, 2022 at 11:29:07PM +0500, Roman Mamedov wrote:
> It is a question of whether you want to cache encrypted, or plain-text data. I
> guess the former should be preferable, for a complete peace-of-mind against
> data forensics vs the cache device, but with a toll on performance, due to the
> need to re-decrypt even the cache hits each time.
 
Right, I know that tradeoff. Also, LUKS makes things a bit more complicated
if you want to grow the FS.

> In case of caching encrypted, it's:
> 
> mdraid => PV => LV => LUKS
>                 |
>              (cache)
> 
> Otherwise:
> 
> mdraid => LUKS => PV => LV
>                         |
>                      (cache)

Right. I'll probably do that.

On Mon, Jun 13, 2022 at 04:08:07PM -0400, Zygo Blaxell wrote:
> Add a cache LV to an existing LV with:
> 
> 	lvcreate $vg -n $meta -L 1G $device
> 	lvcreate $vg -n $pool -l 90%PVS $device
> 	lvconvert -f --type cache-pool --poolmetadata $vg/$meta $vg/$pool
> 	lvconvert -f --type cache --cachepool $vg/$pool $vg/$data --cachemode writethrough
> 
> Uncache with:
> 
> 	lvconvert -f --uncache $vg/$data
> 
> Note that 'lvconvert' will flush the entire cache back to the backing
> store during uncache at minimum IO priority, so it will take some time
> and can be prolonged indefinitely by a continuous IO workload on top.
> Also, the uncache operation will propagate any corruption in the SSD
> cache back to the HDD LV, even in writethrough mode.

Thanks much for the heads up.

Best,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
 
Home page: http://marc.merlins.org/                       | PGP 7F55D5F27AAF9D08