bcache not working on large files?

bcache not working on large files?

[Dirk Geschke]

Hi all,

I was just playing a little bit with bcache and it works fine. But
if I try random IOPS writes (writeback) on a file larger than the
cache, it seems not to work? At least I get a performance as without
bcache.

I used a RAID-6 of 8 SSDs (each 238 GiB) as a cache and a RAID-6 of
10 HDDs (2794 Gib). For my test I did random writes of 4k blocks.
With 8 threads writing on one 1000 GB file, I get IOPS in the range
of 20000, but with a file of 10000 GB I end up with about 400 IOPS.
That is the same as without cache. 

Did I miss something? Is caching disabled in such cases?

I would expect, that even in this case (writeback and random writes)
it should work as well. Even after a few minutes I get these results,
so it is not a problem of a full cache.

Has anyone a hint for me, what is going wrong?

Best regards

Dirk

-- 
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| dirk-tpR6ahGJfjwEZ6m2XrtfILNAH6kLmebB@public.gmane.org / dirk-WMiem5eIfR0n5izryJqWLw@public.gmane.org  / kontakt-WMiem5eIfR0n5izryJqWLw@public.gmane.org |
+----------------------------------------------------------------------+

Re: bcache not working on large files?

[Rolf Fokkens]

Hi Dirk,

On 10/22/2013 09:08 AM, Dirk Geschke wrote:
> I was just playing a little bit with bcache and it works fine. But
> if I try random IOPS writes (writeback) on a file larger than the
> cache, it seems not to work? At least I get a performance as without
> bcache.
>
> Did I miss something? Is caching disabled in such cases?
>
> Has anyone a hint for me, what is going wrong?
Bcache has some specific handling of sequential I/O:

http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt

Could this explain what you're seeing?

Rolf

Re: bcache not working on large files?

[Dirk Geschke]

Hi Rolf,

> >I was just playing a little bit with bcache and it works fine. But
> >if I try random IOPS writes (writeback) on a file larger than the
> >cache, it seems not to work? At least I get a performance as without
> >bcache.
> >
> >Did I miss something? Is caching disabled in such cases?
> >
> >Has anyone a hint for me, what is going wrong?
> Bcache has some specific handling of sequential I/O:
> 
> http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt
> 
> Could this explain what you're seeing?

no, I was explicitly testing random I/O writes of 4k blocks, 
no sequential writing. With a file of 1000 GB it does work, but
if I use a 10000 GB file, it seems to fail. I would expect, that
the size should not really matter here, at least until the cache
is filled up.

The only thing I can imagine is a problem with the RAID controller.
Both RAIDs (HDDs and SSDs) are on the same controller. Maybe the
controller slows down the SSD cache if it writes to the HDDs?

Hmm, maybe I should do two test prarallel, benchmarking random
writes of 4k blocks to a 1000 GB file on the SSD RAID and random
writes on the HDD RAID. Maybe with random I/O on a 10000 GB file 
would take too long and slow down the SSD RAID?

I will see if I can test it. So maybe it is not an issue of BCache
at all...

Best regards

Dirk
-- 
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| dirk-tpR6ahGJfjwEZ6m2XrtfILNAH6kLmebB@public.gmane.org / dirk-WMiem5eIfR0n5izryJqWLw@public.gmane.org  / kontakt-WMiem5eIfR0n5izryJqWLw@public.gmane.org |
+----------------------------------------------------------------------+

Re: bcache not working on large files?

[matthew patton]

>>  http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt

> 
> no, I was explicitly testing random I/O writes of 4k blocks, 
> no sequential writing. With a file of 1000 GB it does work, but
> if I use a 10000 GB file, it seems to fail. I would expect, that
> the size should not really matter here, at least until the cache
> is filled up.


When the SSD gets too slow it will get by-passed by bcache. The tunable is in the document. Though if memory serves it's a latency of 20ms for writes which is probably way too short since SSDs can easily take 1-5 seconds when they have to resort to heavy lifting.



What you should do is turn off RAID controller READ caching entirely. And turn OFF writeBACK-caching for the SSD-based LUN(s) at said controller that are being used as bcache caching devices.

It would be helpful is you elaborated as to the HBA and SSD drives you are using. BTW doing RAID across the SSDs being used for cache is rather pointless IMO. You're shortening their life, adding unnecessary write IOPs. It's a cache. It's supposed to fail. Bcache will(?) properly handle a busted SSD.

Cache is only useful for absorbing sudden spikes in IOPs or for highly localized and frequently re-used blocks. It's not intended to magically improve the underlying storage by 10-100x under a load that can't be sustained by said layer. Big $$$ SANs have lots of cache but at some point you will reach the saturation point and everything slows to HDD speed.

I also wouldn't futz with the EXT4 settings you had posted previously while doing  benchmark runs because I expect it gets in the way and makes performance worse than if the block stream was less chunky. Only once you have defined a realistic sustained workload and know how often and how fast journaled-to-SSD writes (bcache write-back) can reasonably be de-staged would I revisit those tuning parameters and see if there is any merit to them.

Personally I put NVRAM boards in my servers to be filesystem journals and MD mirror maps. They're incredibly cheap.

Re: bcache not working on large files?

[Dirk Geschke]

Hi Matthew,

> > no, I was explicitly testing random I/O writes of 4k blocks, 
> > no sequential writing. With a file of 1000 GB it does work, but
> > if I use a 10000 GB file, it seems to fail. I would expect, that
> > the size should not really matter here, at least until the cache
> > is filled up.
> 
> 
> When the SSD gets too slow it will get by-passed by bcache. The tunable is in the document. Though if memory serves it's a latency of 20ms for writes which is probably way too short since SSDs can easily take 1-5 seconds when they have to resort to heavy lifting.

ah, that could be an explanation for this effect. But I do not find
an apropriate parameter in the documentation. Do you have a hint?

> What you should do is turn off RAID controller READ caching entirely. And turn OFF writeBACK-caching for the SSD-based LUN(s) at said controller that are being used as bcache caching devices.

hmm, yes read cache for SSDs is a kind of overhead. You mean, this may
have an impact, too? Actually I am not sure about the settings, it is
likely the default...

However, it worked with smaller files like 1000 GB. There is no change
at all in random I/O but the size of the file I am using.

> It would be helpful is you elaborated as to the HBA and SSD drives you are using. BTW doing RAID across the SSDs being used for cache is rather pointless IMO. You're shortening their life, adding unnecessary write IOPs. It's a cache. It's supposed to fail. Bcache will(?) properly handle a busted SSD.

The doumentation says no and recommends a RAID. That's the reason 
why it is used in WriteThrough mode per default. Probably a RAID 5/6
is not a good idea for a Caching SSD-RAID, especially when the HDDs
form a RAID 5/6 on the same controller, too.

Probably a RAID 10 would be better...

> Cache is only useful for absorbing sudden spikes in IOPs or for highly localized and frequently re-used blocks. It's not intended to magically improve the underlying storage by 10-100x under a load that can't be sustained by said layer. Big $$$ SANs have lots of cache but at some point you will reach the saturation point and everything slows to HDD speed.

Yes, I was just testing random I/O. And I had expected, that bcache
will work with large files, too.

So with 8 threads doing random I/O of 4k blocks in a 1000 GB file
results in IOPS in the range of about 20,000. The same setup with
a file of 10,000 GB results in an IOPS of about 400. That is about
the same rate the HDDs have without bcache. This is something I did
not expect. It looks like bcache stopped to work at all...

Maybe your first comment explains this, so that the SSDs get too
lazy. Maybe this is due to the raid controller working with the
HDDs. 

> I also wouldn't futz with the EXT4 settings you had posted previously while doing  benchmark runs because I expect it gets in the way and makes performance worse than if the block stream was less chunky. Only once you have defined a realistic sustained workload and know how often and how fast journaled-to-SSD writes (bcache write-back) can reasonably be de-staged would I revisit those tuning parameters and see if there is any merit to them.

Oh, that must be another thread, I am using XFS... ;-)

> Personally I put NVRAM boards in my servers to be filesystem journals and MD mirror maps. They're incredibly cheap.

I am just curious, where the limits are and in which ranges on can 
use bcache. But I did not expect this result, therefore I asked for
a hint. I would have expected, it would work as before. The random
writes would entirely fit in the SSD cache even if the HDDs would
sleep all the time...

And yes, I am not convinced that a RAID 5/6 for SSD caches is really a
good idea. Hmm, maybe I can use a RAID 10 instead...

Best regards

Dirk

-- 
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| dirk-tpR6ahGJfjwEZ6m2XrtfILNAH6kLmebB@public.gmane.org / dirk-WMiem5eIfR0n5izryJqWLw@public.gmane.org  / kontakt-WMiem5eIfR0n5izryJqWLw@public.gmane.org |
+----------------------------------------------------------------------+

Re: bcache not working on large files?

[matthew patton]

>> cache. It's supposed to fail. Bcache will(?) properly handle a busted SSD.

> 
> The doumentation says no and recommends a RAID. That's the reason 
> why it is used in WriteThrough mode per default.

'busted' in this case means errors on write. If the device disappears and is no longer readable, then sure any un-committed blocks are gone and your filesystem/file is broken.

On write failure (how many?) I would sure hope the code-path marks the malfunctioning SSD as disabled and just sends the writes to the HDD. Whether it then gets more aggressive about de-staging previously staged writes I don't know (probably not) but just as long as the SSD is readable the dirty blocks can be written out.

If you're not running kernel 3.11.5 (or 3.11.4 with *all* the bcache fixes) you're wasting your time.

RE: bcache not working on large files?

[Paul B. Henson]

> From: matthew patton
> Sent: Tuesday, October 22, 2013 3:27 PM
>
> 'busted' in this case means errors on write. If the device disappears and
is no
> longer readable, then sure any un-committed blocks are gone and your
> filesystem/file is broken.

In other words, if you care about your data, don't use a nonredundant cache
device in writeback mode 8-/. I'm tentatively planning to use an md mirror
of two 256G SSD's to front end a raid10 of 4 2TB HD's, I'm still in the
investigate stage but hopefully will get to prototyping soon.
 
> If you're not running kernel 3.11.5 (or 3.11.4 with *all* the bcache
fixes)
> you're wasting your time.

Is there an intention to port all of these fixes back to an LTS 3.10 kernel,
or if you're going to run in production you're pretty much committed to the
latest stable?

Thanks.

Re: bcache not working on large files?

[Dirk Geschke]

Hi all,

> So with 8 threads doing random I/O of 4k blocks in a 1000 GB file
> results in IOPS in the range of about 20,000. The same setup with
> a file of 10,000 GB results in an IOPS of about 400. That is about
> the same rate the HDDs have without bcache. This is something I did
> not expect. It looks like bcache stopped to work at all...

I have to correct it: I rebuild the SSD-RAID and used now an RAID 0
instead of RAID 6 for caching. So now I get about 10.000 IOPS with
writing 4k blocks . I think this is 

   a) ok 

and 

   b) it is not a bcache problem

I suspect it is a problem of the raid controller, but I have no clue 
which one...

Probably it is a better decision to use a separate raid controller
for SSD (b)cache...

Best regards

Dirk

-- 
+----------------------------------------------------------------------+
| Dr. Dirk Geschke       / Plankensteinweg 61    / 85435 Erding        |
| Telefon: 08122-559448  / Mobil: 0176-96906350 / Fax: 08122-9818106   |
| dirk-tpR6ahGJfjwEZ6m2XrtfILNAH6kLmebB@public.gmane.org / dirk-WMiem5eIfR0n5izryJqWLw@public.gmane.org  / kontakt-WMiem5eIfR0n5izryJqWLw@public.gmane.org |
+----------------------------------------------------------------------+

Re: bcache not working on large files?

[matthew patton]

> http://evilpiepirate.org/git/linux-bcache.git/tree/Documentation/bcache.txt

>
> When the SSD gets too slow it will get by-passed by bcache. The tunable is in 
> the document. Though if memory serves it's a latency of 20ms for writes 
> which is probably way too short since SSDs can easily take 1-5 seconds when they 
> have to resort to heavy lifting.


- Traffic's still going to the spindle/still getting cache misses In the real world, SSDs don't always keep up with disks - particularly with slower SSDs, many disks being cached by one SSD, or mostly sequential IO. So you want to avoid being bottlenecked by the SSD and having it slow everything down. To avoid that bcache tracks latency to the cache device, and gradually throttles traffic if the latency exceeds a threshold (it does this by cranking down the sequential bypass). You can disable this if you need to by setting the thresholds to 0: # echo 0 > /sys/fs/bcache/<cache set>/congested_read_threshold_us # echo 0 > /sys/fs/bcache/<cache set>/congested_write_threshold_us