bcache and LVM

bcache and LVM

[Nikolaus Rath]

Hello,

The internet claims that using bcache with LVM is not a good idea
(eg. on https://wiki.archlinux.org/index.php/Bcache )- but I wasn't able
to find any substantial information other than this general
recommendation.

Is this still (or has ever been) correct? If so, what issues can arise?
And does this happen only when using bcache on top of an LVM LV, or also
when using a bcache device as an LVM PV?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: bcache and LVM

[Jens-U. Mozdzen]

Hi Nikolaus,

Zitat von Nikolaus Rath <Nikolaus@rath.org>:
> Hello,
>
> The internet claims that using bcache with LVM is not a good idea
> (eg. on https://wiki.archlinux.org/index.php/Bcache )- but I wasn't able
> to find any substantial information other than this general
> recommendation.

looking at that article, it only references possibly ill-handled  
"discard" operations, when running bcache on top of LVM.

The article itself does positively mention sub-dividing bcache devices  
using LVM.

> Is this still (or has ever been) correct? If so, what issues can arise?
> And does this happen only when using bcache on top of an LVM LV, or also
> when using a bcache device as an LVM PV?

We're running bcache between MD-RAID (RAID6 for the HDD backing store  
and RAID1 for caching SSDs) and LVM2 (using bcache0 as the only PV for  
the volume group) without any noticeable problem, for moderate to  
significant load (SAN/NAS servers with NFS, Samba, and virtual  
machines' storage via SCST/FC and SCST/iSCSI).

Regards,
Jens

Re: bcache and LVM

[Nikolaus Rath]

On Feb 08 2016, "Jens-U. Mozdzen" <jmozdzen@nde.ag> wrote:
> Hi Nikolaus,
>
> Zitat von Nikolaus Rath <Nikolaus@rath.org>:
>> Hello,
>>
>> The internet claims that using bcache with LVM is not a good idea
>> (eg. on https://wiki.archlinux.org/index.php/Bcache )- but I wasn't able
>> to find any substantial information other than this general
>> recommendation.
>
> looking at that article, it only references possibly ill-handled
> "discard" operations, when running bcache on top of LVM.

Well, it says:

| Warning:
| 
|     it is widely recommended to use Bcache underneath any other block
|    layer.


To me this implies that running it on top of another block layer is
dangerous (why warn about it otherwise?). 

> The article itself does positively mention sub-dividing bcache devices
> using LVM.

Yep, but the talk page then says:

| Initially, LVM did not recognize my /dev/bcache0 when I wanted to
| create a physical volume on it. For anyone else who has that issue,
| this may be relevant:
| http://www.redhat.com/archives/linux-lvm/2012-March/msg00005.html.


>> Is this still (or has ever been) correct? If so, what issues can arise?
>> And does this happen only when using bcache on top of an LVM LV, or also
>> when using a bcache device as an LVM PV?
>
> We're running bcache between MD-RAID (RAID6 for the HDD backing store
> and RAID1 for caching SSDs) and LVM2 (using bcache0 as the only PV for
> the volume group) without any noticeable problem, for moderate to
> significant load (SAN/NAS servers with NFS, Samba, and virtual
> machines' storage via SCST/FC and SCST/iSCSI).

Thanks for the datapoint! What kernel version do you use?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: bcache and LVM

[Jens-U. Mozdzen]

Hi Nikolaus,

Zitat von Nikolaus Rath <Nikolaus@rath.org>:
> On Feb 08 2016, "Jens-U. Mozdzen" <jmozdzen@nde.ag> wrote:
>> [...]
>> The article itself does positively mention sub-dividing bcache devices
>> using LVM.
>
> Yep, but the talk page then says:
>
> | Initially, LVM did not recognize my /dev/bcache0 when I wanted to
> | create a physical volume on it. For anyone else who has that issue,
> | this may be relevant:
> | http://www.redhat.com/archives/linux-lvm/2012-March/msg00005.html.

this still holds true even with LVM on OpenSUSE Leap 42.1 (which is  
what we're using ATM) - but that's no "stability issue", but rather an  
installation nuisance. You only need to tell LVM to consider bcache  
devices at all, by extending /etc/lvm/lvm.conf.

> Thanks for the datapoint! What kernel version do you use?

4.1.13-5-default #1 SMP PREEMPT Thu Nov 26 16:35:17 UTC 2015 (49475c3)  
x86_64 x86_64 x86_64 GNU/Linux

It's the kernel from the OpenSUSE project repository, no further  
patches applied by me:

Source Timestamp: 2015-11-26 17:35:17 +0100
GIT Revision: 49475c39bfd2ee30d145707df8a17419822ba804
GIT Branch: users/tiwai/openSUSE-42.1/for-next
Distribution: openSUSE Leap 42.1

Regards,
Jens

Re: bcache and LVM

[Nikolaus Rath]

On Feb 09 2016, "Jens-U. Mozdzen" <jmozdzen@nde.ag> wrote:
> Hi Nikolaus,
>
> Zitat von Nikolaus Rath <Nikolaus@rath.org>:
>> On Feb 08 2016, "Jens-U. Mozdzen" <jmozdzen@nde.ag> wrote:
>>> [...]
>>> The article itself does positively mention sub-dividing bcache devices
>>> using LVM.
>>
>> Yep, but the talk page then says:
>>
>> | Initially, LVM did not recognize my /dev/bcache0 when I wanted to
>> | create a physical volume on it. For anyone else who has that issue,
>> | this may be relevant:
>> | http://www.redhat.com/archives/linux-lvm/2012-March/msg00005.html.
>
> this still holds true even with LVM on OpenSUSE Leap 42.1 (which is
> what we're using ATM) - but that's no "stability issue", but rather an
> installation nuisance. You only need to tell LVM to consider bcache
> devices at all, by extending /etc/lvm/lvm.conf.

Indeed. But it's also yet another slightly worrying tidbit (maybe LVM is
not considering it by default for a good reason).

Also, just now someone on -btrfs advised:

| > Otherwise I'll give bcache a shot. I've avoided it so far because of the
| > need to reformat and because of rumours that it doesn't work well with
| > LVM or BTRFS. But it sounds as if that's not the case..
|
| It should work fine with _just_ BTRFS, but don't put any other layers
| into the storage system like LVM or dmcrypt or mdraid, it's got some
| pretty pathological interactions with the device mapper and md
| frameworks still.


Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: bcache and LVM

[Jens-U. Mozdzen]

Hi Nikolas,

Zitat von Nikolaus Rath <Nikolaus@rath.org>:
> [...]
> Indeed. But it's also yet another slightly worrying tidbit (maybe LVM is
> not considering it by default for a good reason).

Agreed, but from what I read, newer versions of LVM were said to  
support bcache devices out of the box  
(https://www.redhat.com/archives/linux-lvm/2012-March/msg00007.html).

> Also, just now someone on -btrfs advised:
>
> | > Otherwise I'll give bcache a shot. I've avoided it so far because of the
> | > need to reformat and because of rumours that it doesn't work well with
> | > LVM or BTRFS. But it sounds as if that's not the case..
> |
> | It should work fine with _just_ BTRFS, but don't put any other layers
> | into the storage system like LVM or dmcrypt or mdraid, it's got some
> | pretty pathological interactions with the device mapper and md
> | frameworks still.

It might be that this is a bit oldish info. BTW, I read that BTRFS on  
LVM is bad and causes problems ;)

We run MD-RAID (R6 for HDDs, R1 for SSDs) beneath bcache. We've run  
thousands of Gigabytes of read and write traffic through this,  
successfully. (And of course - YMMV :-/ ). We've used SATA disks  
(RAID-compatible WD Red) & SSDs (when not running via Supermicro's  
servers with SAS Extender chassis) and SAS disks & SSDs (on those  
servers with SAS extender), all fine.

 From my point of view, bcache with latest bug fixes is a helpful and  
(operationally) stable solution.

Regards,
Jens

Re: bcache and LVM

[Nikolaus Rath]

On Feb 07 2016, Nikolaus Rath <Nikolaus@rath.org> wrote:
> Hello,
>
> The internet claims that using bcache with LVM is not a good idea
> (eg. on https://wiki.archlinux.org/index.php/Bcache )- but I wasn't able
> to find any substantial information other than this general
> recommendation.
>
> Is this still (or has ever been) correct? If so, what issues can arise?
> And does this happen only when using bcache on top of an LVM LV, or also
> when using a bcache device as an LVM PV?

I now have the following stack:

btrfs on LUKS on LVM on bcache

The VG contains two bcache PVs with backing devices on different
spinning disks, and a shared cache device on SSD. I'm using Kernel 4.3.

I'm super happy with the performance, boot times increased from 1:30
minutes to X11 and 2:00 to Firefox to roughly 0:10 to X11 and 0:30 to
Firefox.

Time will tell if it also keeps my data intact, but I hope btrfs would
at least detect any corruption. 


Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: bcache and LVM --- works great.

[Eric Wheeler]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3158 bytes --]

On Tue, 9 Feb 2016, Nikolaus Rath wrote:
> On Feb 07 2016, Nikolaus Rath <Nikolaus@rath.org> wrote:
> > Hello,
> >
> > The internet claims that using bcache with LVM is not a good idea
> > (eg. on https://wiki.archlinux.org/index.php/Bcache )- but I wasn't able
> > to find any substantial information other than this general
> > recommendation.
> >
> > Is this still (or has ever been) correct? If so, what issues can arise?
> > And does this happen only when using bcache on top of an LVM LV, or also
> > when using a bcache device as an LVM PV?

We use bcache0 as the PV for our LVs and our caching device is actually an 
LV sliced from an SSD PV backed by hardware raid10 (Areca).  We have 
hot-detached the cache, lvresized it, re-make-bcache'd the cache, and 
re-attached it without needing a reboot (backing dev resizes can't be hot 
though---also not sure what would happen if we re-attached after resize 
without make-bcache'ing the cache).  We haven't tried making the backing 
device an LV, but the caching device works great as an LV.

With LVM on both top and bottom of our bcache stack we have been stable 
for a year with 100's of TBs written.  We burned out the Samsung 840's 
after about 150 TB written and replaced them hot.  Even though they 
started to stutter, no data lost and no issue.  In fact we replaced the 
aging SSDs without rebooting the hardware (and moved from raid1 to raid10 
at the same time).

> I now have the following stack:
> 
> btrfs on LUKS on LVM on bcache
> 
> The VG contains two bcache PVs with backing devices on different
> spinning disks, and a shared cache device on SSD. I'm using Kernel 4.3.

4.3 is missing the bcache stability patches, so be sure you have 
pulled them in.  (4.3 is EOL too, so change if you can.)

Be sure to cherry-pick these from linux 4.5-rc1:
        git cherry-pick 2ef9ccbf~1..627ccd20

or use one of the 4.1 or 3.18 longterm kernels.  4.1.17 is rock solid.  
Haven't tried 4.1.18 yet, but it has the stability patches now and I 
expect 4.1.18 will be a great kernel too.

> I'm super happy with the performance, boot times increased from 1:30
> minutes to X11 and 2:00 to Firefox to roughly 0:10 to X11 and 0:30 to
> Firefox.
> 
> Time will tell if it also keeps my data intact, but I hope btrfs would
> at least detect any corruption. 

Excellent!  

We have great success with bcache as well with possibly PB's of data 
written and well over a year of use. Our VM stack looks something like 
this:
   KVM{btrfs -> lvm -> virtio-scsi+unmap} -> 
	LV -> PV -> bcache0(writeback) -> 
		backing device: hardware raid5 (full blockdev)
		caching device: lvm->pv->(hardware raid10 ssd)
	  

-Eric
 
> 
> Best,
> -Nikolaus
> 
> (No Cc on replies please, I'm reading the list)
> -- 
> GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
> Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F
> 
>              »Time flies like an arrow, fruit flies like a Banana.«
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>