Tiering in bcachefs

[Kent Overstreet <kent.overstreet@gmail.com>]

Just wrote a long post explaining all about tiering in bcachefs:

https://www.patreon.com/posts/all-about-6751413

Also, tiering should now be working! I'd appreciate help testing it.

Notes:

Chris Halse Rogers (RAOF in the #bcache IRC channel) has been testing it. He has
been seeing an intermittent deadlock while copying large amounts of data, which
may or may not be tiering related: if anyone else hits it, I'd really appreciate
if you could grab backtraces. Do a "echo t > /proc/sysrq-trigger" and then grab
the full dmesg log - that should be enough to figure out what the deadlock is.

I haven't gotten to device add/remove yet, don't try it - you'll need to create
a new filesystem:

bcache format --tier 0 <SSD> --tier 1 <HDD>

Also, if you've already been playing with tiering, I highly recommend you create
a new filesystem. One of the bugs was an issue with which pointers are
considered dirty - in an extent with a tier 0 pointer and a tier 1 pointer, the
tier 0 pointer was being considered dirty instead of the tier 1 pointer (with
predictably bad consequences; tier 0 would fill up because the tiering thread
wasn't actually making anything on tier 0 clean). On a fixed kernel, as extents
are migrated it'll end up fixing which pointers are considered dirty which may
or may not trigger assertions as which pointers are considered dirty changes.