Re: bcache vs enhanceio?

Re: bcache vs enhanceio?

[Roy Sigurd Karlsbakk]

----- Opprinnelig melding -----
> ZFS is not just a filesystem. It is a complete block device storage
> management stack that has a filesystem component.

I know ZFS quite well, so I'm aware of the comparison is somewhat unfair. 

> The Linux tool chain can be more complicated but it is vastly more
> flexible. With the addition of a caching target all of the pieces are
> there to be able to build a tiering storage system in any way your use
> case needs it to be.

Still, flashcache/enhanceio seems to be able to handle this in a very flexible way. I really don't want to recreate my home RAID (7,2TiB) just to add cache to it…

Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy-ooPBL11mRiZbRRN4PJnoQQ@public.gmane.org
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

Re: bcache vs enhanceio?

[Jason Warr]

On 02/19/2013 06:23 AM, Roy Sigurd Karlsbakk wrote:
> ----- Opprinnelig melding -----
>> ZFS is not just a filesystem. It is a complete block device storage
>> management stack that has a filesystem component.
> I know ZFS quite well, so I'm aware of the comparison is somewhat unfair. 
Unfair is an understatement.  But I also think it is quite a sham how
many things that Sun and now Oracle has been bromising to put in for
years that seem now to be fairy tails.  The two biggest ones that I can
think of are VDEV removal and RAIDZ[2,3] stripe expansion.

I've been working with ZFS since I was able to get the first internal
alpha from the team at Sun that would apply to the Nevada dev tree.  I
think that was about build 17.  I believe it was 27a that it finally
made it in to the ON consolidation.  Really smart bunch of guys working
on that at the time.  Almost all of them have jumped ship now though.
>
>> The Linux tool chain can be more complicated but it is vastly more
>> flexible. With the addition of a caching target all of the pieces are
>> there to be able to build a tiering storage system in any way your use
>> case needs it to be.
> Still, flashcache/enhanceio seems to be able to handle this in a very flexible way. I really don't want to recreate my home RAID (7,2TiB) just to add cache to it…
I can agree with this on principal but having to re-do it for cache can
have some other positive effects.  Some that I can think of is it give
you an opportunity to get LV's re-aligned if your like most and have
extended them one or more times, Perhaps your volume layout is not ideal
and you have been putting off fixing that, etc...

Chances are that if your wanting to add a cache the filesystems are
relatively busy and may benefit from being re-created to reduce file
fragmentation.

It would be very nice to be able to do but in order for it to happen
your entire block tool chain would have to be prepared for it to be a
possibility for it to happen.  That's why ZFS can do that.

A question for Kent, once you have bcache and it's tools built,
installed and running, is there anything to stop a user from always
tagging devices of whatever type you choose from having the superblock
info to accept a cache dynamically?  Example, if I create an MD RAID
device and before I pvcreate or anything else with it I prep it for
bcache but don't actually attach a cache device, is there any negative
effects that can come from that?  Can I then at anytime attach a cache
device to it?  I realize that once attached in writeback it becomes
non-detachable.  Same question for raw sd devices and LVM volumes.

> Vennlige hilsener / Best regards
>
> roy
> --
> Roy Sigurd Karlsbakk
> (+47) 98013356
> roy-ooPBL11mRiZbRRN4PJnoQQ@public.gmane.org
> http://blogg.karlsbakk.net/
> GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
> --
> I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

Re: bcache vs enhanceio?

[Joseph Glanville]

I am not Kent.. but I can answer your questions.

8<--- snip ---->8
> A question for Kent, once you have bcache and it's tools built,
> installed and running, is there anything to stop a user from always
> tagging devices of whatever type you choose from having the superblock
> info to accept a cache dynamically?  Example, if I create an MD RAID
> device and before I pvcreate or anything else with it I prep it for
> bcache but don't actually attach a cache device, is there any negative
> effects that can come from that?  Can I then at anytime attach a cache
> device to it?  I realize that once attached in writeback it becomes
> non-detachable.  Same question for raw sd devices and LVM volumes.

In short yes, there are no detrimental effects for having backing
devices with superblocks that don't have associated cache sets.

To touch on the second point about writeback - it's not so much that
it's non-detachable it's that you don't want the backing device to be
used while the cache is not attached and is dirty (contains unflushed
data).

You can detach the cache safely from a writeback device by first
switching the cache to writethrough (or none from memory) and waiting
for the data to flush to the backing device.
Once that is done you can either continue to use it in writethrough
mode or you can detach it completely.

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: bcache vs enhanceio?

[Jason Warr]

On 02/19/2013 11:17 AM, Joseph Glanville wrote:
> I am not Kent.. but I can answer your questions.
>
> 8<--- snip ---->8
>> A question for Kent, once you have bcache and it's tools built,
>> installed and running, is there anything to stop a user from always
>> tagging devices of whatever type you choose from having the superblock
>> info to accept a cache dynamically?  Example, if I create an MD RAID
>> device and before I pvcreate or anything else with it I prep it for
>> bcache but don't actually attach a cache device, is there any negative
>> effects that can come from that?  Can I then at anytime attach a cache
>> device to it?  I realize that once attached in writeback it becomes
>> non-detachable.  Same question for raw sd devices and LVM volumes.
> In short yes, there are no detrimental effects for having backing
> devices with superblocks that don't have associated cache sets.

That's what I thought.  This could be an argument for integration with
DM or MD.  Up-rev the superblock or metadata version and have the bcache
bits in it by default.

>
> To touch on the second point about writeback - it's not so much that
> it's non-detachable it's that you don't want the backing device to be
> used while the cache is not attached and is dirty (contains unflushed
> data).
>
> You can detach the cache safely from a writeback device by first
> switching the cache to writethrough (or none from memory) and waiting
> for the data to flush to the backing device.
> Once that is done you can either continue to use it in writethrough
> mode or you can detach it completely.
>
:) Typing faster than I am thinking.  I should have said non-detachable
while in writeback mode, or rather while it contains "dirty" blocks.

Re: bcache vs enhanceio?

[Kent Overstreet]

On Tue, Feb 19, 2013 at 11:42:14AM -0600, Jason Warr wrote:
> 
> On 02/19/2013 11:17 AM, Joseph Glanville wrote:
> > I am not Kent.. but I can answer your questions.
> >
> > 8<--- snip ---->8
> >> A question for Kent, once you have bcache and it's tools built,
> >> installed and running, is there anything to stop a user from always
> >> tagging devices of whatever type you choose from having the superblock
> >> info to accept a cache dynamically?  Example, if I create an MD RAID
> >> device and before I pvcreate or anything else with it I prep it for
> >> bcache but don't actually attach a cache device, is there any negative
> >> effects that can come from that?  Can I then at anytime attach a cache
> >> device to it?  I realize that once attached in writeback it becomes
> >> non-detachable.  Same question for raw sd devices and LVM volumes.
> > In short yes, there are no detrimental effects for having backing
> > devices with superblocks that don't have associated cache sets.
> 
> That's what I thought.  This could be an argument for integration with
> DM or MD.  Up-rev the superblock or metadata version and have the bcache
> bits in it by default.

Yeah, that idea's been kicked around before. A couple people have said
they were going to work on it, but nothing's happened.

> >
> > To touch on the second point about writeback - it's not so much that
> > it's non-detachable it's that you don't want the backing device to be
> > used while the cache is not attached and is dirty (contains unflushed
> > data).
> >
> > You can detach the cache safely from a writeback device by first
> > switching the cache to writethrough (or none from memory) and waiting
> > for the data to flush to the backing device.
> > Once that is done you can either continue to use it in writethrough
> > mode or you can detach it completely.
> >
> :) Typing faster than I am thinking.  I should have said non-detachable
> while in writeback mode, or rather while it contains "dirty" blocks.

You actually don't have to switch out of writeback mode to safely detach
- if you detach in writeback mode, it detaches as soon as it's finished
flushing all the dirty data.

bcache vs enhanceio?

[Roy Sigurd Karlsbakk]

Hi

Seems old flashcache and the new enhanceio can add flash drives on the run, while bcache needs the filsystem to be built with them.

What is the future of bcache in this setting?

-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy-ooPBL11mRiZbRRN4PJnoQQ@public.gmane.org
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

Re: bcache vs enhanceio?

[Andrew Thrift]

Hi Roy,

This is by design, and minimizes the potential for data corruption.

If you have a backing device with an existing filesystem, and are 
caching this in writeback mode, then reboot the system and the cache 
device does not come up, the backing device could still come up and 
could potentially be out of sync with the writes that were buffered to 
the cache device.

With bcache in writeback mode, if it cannot attach the cache device it 
will not register the backing device.


I am sure Kent will provide a much better answer than I have  :)




Regards,




Andrew

On 2/8/2013 10:25 AM, Roy Sigurd Karlsbakk wrote:
> Hi
>
> Seems old flashcache and the new enhanceio can add flash drives on the run, while bcache needs the filsystem to be built with them.
>
> What is the future of bcache in this setting?
>

Re: bcache vs enhanceio?

[Roy Sigurd Karlsbakk]

Wouldn't it be better to allow the sysadmin to determine the safety?
Also, if running in writethrough, why shouldn't an SSD be allowed to be added in realtime?
All of this works well with systems like ZFS. I really don't see a reason for a filesystem being created to allow caching

roy

----- Opprinnelig melding -----
> Hi Roy,
> 
> This is by design, and minimizes the potential for data corruption.
> 
> If you have a backing device with an existing filesystem, and are
> caching this in writeback mode, then reboot the system and the cache
> device does not come up, the backing device could still come up and
> could potentially be out of sync with the writes that were buffered to
> the cache device.
> 
> With bcache in writeback mode, if it cannot attach the cache device it
> will not register the backing device.
> 
> 
> I am sure Kent will provide a much better answer than I have :)
> 
> 
> 
> 
> Regards,
> 
> 
> 
> 
> Andrew
> 
> On 2/8/2013 10:25 AM, Roy Sigurd Karlsbakk wrote:
> > Hi
> >
> > Seems old flashcache and the new enhanceio can add flash drives on
> > the run, while bcache needs the filsystem to be built with them.
> >
> > What is the future of bcache in this setting?
> >

-- 
Vennlige hilsener / Best regards

roy
--
Roy Sigurd Karlsbakk
(+47) 98013356
roy-ooPBL11mRiZbRRN4PJnoQQ@public.gmane.org
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.

Re: bcache vs enhanceio?

[Kent Overstreet]

On Tue, Feb 12, 2013 at 09:41:20PM +0100, Roy Sigurd Karlsbakk wrote:
> Wouldn't it be better to allow the sysadmin to determine the safety?

The sysadmin can override, but - say you're using bcache for your root
filesystem and after a reboot the SSD doesn't come up, for whatever
reason.

How's the sysadmin supposed to know for sure whether there was dirty
data in the cache? There's no way to reliably track that without bcache
tracking it itself in the backing device's superblock.

Say there wasn't any dirty data in the cache, so you can safely run
without the cache device - so you do, so you can boot up and use your
machine.

Then later you figure out what's wrong with the SSD (cable got
unplugged?), so you reenable caching.

But the cache is now inconsistent - you _cannot_ use that cached data.
With the backing device superblock, bcache can trivially note that the
cache is out of sync and make sure that that cached data isn't used.

If we didn't have that, the sysadmin would have to make _sure_ to use
the right flag when reattaching to specify cached data shouldn't be
used, otherwise he just corrupted all his data.


> Also, if running in writethrough, why shouldn't an SSD be allowed to be added in realtime?

If you're running in writethrough, you'd already be caching... not sure
what you mean?

> All of this works well with systems like ZFS. I really don't see a reason for a filesystem being created to allow caching

ZFS is a filesystem that also does caching, bcache isn't a filesystem.
Did you get something backwards...?

Re: bcache vs enhanceio?

[Joseph Glanville]

On 13 February 2013 08:56, Kent Overstreet <koverstreet-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org> wrote:
> On Tue, Feb 12, 2013 at 09:41:20PM +0100, Roy Sigurd Karlsbakk wrote:
>> Wouldn't it be better to allow the sysadmin to determine the safety?
>
> The sysadmin can override, but - say you're using bcache for your root
> filesystem and after a reboot the SSD doesn't come up, for whatever
> reason.
>
> How's the sysadmin supposed to know for sure whether there was dirty
> data in the cache? There's no way to reliably track that without bcache
> tracking it itself in the backing device's superblock.
>
> Say there wasn't any dirty data in the cache, so you can safely run
> without the cache device - so you do, so you can boot up and use your
> machine.
>
> Then later you figure out what's wrong with the SSD (cable got
> unplugged?), so you reenable caching.
>
> But the cache is now inconsistent - you _cannot_ use that cached data.
> With the backing device superblock, bcache can trivially note that the
> cache is out of sync and make sure that that cached data isn't used.
>
> If we didn't have that, the sysadmin would have to make _sure_ to use
> the right flag when reattaching to specify cached data shouldn't be
> used, otherwise he just corrupted all his data.

Looking through the enhanceio repo it seems possible to do it without
a superblock by utilizing customized udev rules.
Said rules prevent the backing device from being using to mount the
root filesystem before the cache is attached.

>
>
>> Also, if running in writethrough, why shouldn't an SSD be allowed to be added in realtime?
>
> If you're running in writethrough, you'd already be caching... not sure
> what you mean?
>
>> All of this works well with systems like ZFS. I really don't see a reason for a filesystem being created to allow caching
>
> ZFS is a filesystem that also does caching, bcache isn't a filesystem.
> Did you get something backwards...?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bcache" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: bcache vs enhanceio?

[Joseph Glanville]

> Looking through the enhanceio repo it seems possible to do it without
> a superblock by utilizing customized udev rules.
> Said rules prevent the backing device from being using to mount the
> root filesystem before the cache is attached.

Just to folow up with the udev config mentioned:
https://github.com/stec-inc/EnhanceIO/tree/master/Documents

-- 
CTO | Orion Virtualisation Solutions | www.orionvm.com.au
Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: bcache vs enhanceio?

[Jason Warr]

>>> Also, if running in writethrough, why shouldn't an SSD be allowed to be added in realtime?
>> If you're running in writethrough, you'd already be caching... not sure
>> what you mean?

Perhaps he means being restricted to only being able to add a write-through cache without the superblock prep?  On a reboot or cache device fault event it would not cause any corruption, just performance impact back to where it was before the device was added.