how does the caching works in bcachefs

how does the caching works in bcachefs

[Stefan K]

Hello,

short question: how does the caching works with bcachefs? Is it like  "first in first out" or is it more complex like the ARC system in zfs? The same with the write-cache,  will be everything written to the SSD/NVMe (Cache) and then to the HDD? When will will the filesystem say "its written to disk"? And what happens with the data on the write cache if we have a powerfail?

And can I say have this file/folder always in the cache, while it works "normal" ?

Thanks in advance!
best regards
Stefan

Re: how does the caching works in bcachefs

[kent.overstreet]

On Wed, Jul 08, 2020 at 11:46:00PM +0200, Stefan K wrote:
> Hello,
> 
> short question: how does the caching works with bcachefs? Is it like  "first
> in first out" or is it more complex like the ARC system in zfs?

LRU, same as bcache.

> The same with the write-cache,  will be everything written to the SSD/NVMe
> (Cache) and then to the HDD? When will will the filesystem say "its written to
> disk"? And what happens with the data on the write cache if we have a
> powerfail?

Disks that are used as caches are treated no differently from other disks by the
filesystem. If you want bcachefs to not rely on a specific disk, you can set its
durability to 0, and then it'll basically only be used as a writethrough cache.

> 
> And can I say have this file/folder always in the cache, while it works "normal" ?

Yes.

So caching is configured differently, specifically so that it can be configured
on a per file/directory basis. Instead of having a notion of "cache device",
there are options for
 - foregroud target: which device or group of devices are used for foreground
   writes
 - background target: if enabled, the rebalance thread will in the background
   move data to this target in the background, leaving a cached copy on the
   foreground target
 - promote target: if enabled, when data is read and it doesn't exist in this
   target, a cached copy will be added there

So these options can be set to get you writeback mode, by setting foreground
target and promote target to your SSD and background target to your HDD.

And you can pin specific files/folders to a device, by setting foreground target
to that device and setting background target and promote target to nothing.

Re: how does the caching works in bcachefs

[Stefan K]

Hi and thanks!

> LRU, same as bcache.
do you plan to change this , since LRU is not very efficient (in comparison to other), maybe 2Q or ARC[1-4]

> [...]
> And you can pin specific files/folders to a device, by setting foreground target
> to that device and setting background target and promote target to nothing.
ok thank you very much! That must be documented somewhere ;-)


[1] https://en.ru.is/media/skjol-td/cache_comparison.pdf
[2] https://dbs.uni-leipzig.de/file/ARC.pdf
[3] http://people.cs.vt.edu/~butta/docs/sigmetrics05_kernelPrefetch.pdf
[4] http://www.vldb.org/conf/1994/P439.PDF

On Thursday, July 9, 2020 12:37:29 AM CEST kent.overstreet@gmail.com wrote:
> On Wed, Jul 08, 2020 at 11:46:00PM +0200, Stefan K wrote:
> > Hello,
> >
> > short question: how does the caching works with bcachefs? Is it like  "first
> > in first out" or is it more complex like the ARC system in zfs?
>
> LRU, same as bcache.
>
> > The same with the write-cache,  will be everything written to the SSD/NVMe
> > (Cache) and then to the HDD? When will will the filesystem say "its written to
> > disk"? And what happens with the data on the write cache if we have a
> > powerfail?
>
> Disks that are used as caches are treated no differently from other disks by the
> filesystem. If you want bcachefs to not rely on a specific disk, you can set its
> durability to 0, and then it'll basically only be used as a writethrough cache.
>
> >
> > And can I say have this file/folder always in the cache, while it works "normal" ?
>
> Yes.
>
> So caching is configured differently, specifically so that it can be configured
> on a per file/directory basis. Instead of having a notion of "cache device",
> there are options for
>  - foregroud target: which device or group of devices are used for foreground
>    writes
>  - background target: if enabled, the rebalance thread will in the background
>    move data to this target in the background, leaving a cached copy on the
>    foreground target
>  - promote target: if enabled, when data is read and it doesn't exist in this
>    target, a cached copy will be added there
>
> So these options can be set to get you writeback mode, by setting foreground
> target and promote target to your SSD and background target to your HDD.
>
> And you can pin specific files/folders to a device, by setting foreground target
> to that device and setting background target and promote target to nothing.
>

Re: how does the caching works in bcachefs

[kent.overstreet]

On Thu, Jul 09, 2020 at 03:20:14PM +0200, Stefan K wrote:
> Hi and thanks!
> 
> > LRU, same as bcache.
> do you plan to change this , since LRU is not very efficient (in comparison to
> other), maybe 2Q or ARC[1-4]

No, but I'd be happy to help if someone else wanted to implement a new caching
algorithm :)

In real world mixed workloads LRU is fine, it's not that much of a difference
vs. the more sophisticated algorithms. More important is the stuff like
sequential_bypass or some other kind of knob to ensure your backup process
doesn't blow away the entire cache.

> 
> > [...]
> > And you can pin specific files/folders to a device, by setting foreground target
> > to that device and setting background target and promote target to nothing.
> ok thank you very much! That must be documented somewhere ;-)

I just write code, not documentation :)

Re: how does the caching works in bcachefs

[Nix]

On 9 Jul 2020, kent overstreet told this:

> In real world mixed workloads LRU is fine, it's not that much of a difference
> vs. the more sophisticated algorithms. More important is the stuff like
> sequential_bypass or some other kind of knob to ensure your backup process
> doesn't blow away the entire cache.

The ioprio thing that never got integrated into bcache (but is still
available as out-of-tree patches that work fine) is even better for
this: yes, it might not quite be what ioprio was meant for, but it means
the user has the equivalent of 'nocache' that they can apply to entire
process hierarchies that they don't want to pollute the cache, just by
using ionice. Run your backup with ionice -c 3 and now everything, even
the metadata reads, comes from the cache if it's already in the cache
but otherwise stays out of cache.

(This matters if you have a huge amount of metadata that is rarely read
except when the backup sweeps over it, and a not-too-huge cache device.
You don't want to waste cache space on stuff that only the backup is
accessing...)