[PATCH 0/5] improve journal pipelining

[Kent Overstreet <kent.overstreet@linux.dev>]

this series makes the journal code even more complicated for the
purposes of improving journal pipelining, which yet again is turning up
as an issue in performance testing

previously, there could be at most 4 journal writes in flight at a time
(each up to ~2MB); the new limit is 16.

there are two different "outstanding journal buf" limits which now have
to be split: journal bufs with outstanding reservations, and journal
bufs with writes in flight. journal reservation state can only address
up to 4 journal bufs - that's the max for "bufs with outstanding
reservations" (and 2 would likely be fine there, now that they're split,
but it's not necessary).

with JOURNAL_BUF_NR going up, they can't be hanging onto data buffers
when not in use anymore - that would pin 16 * 2 = 32MB of ram; so we add
a single element open coded mempool (because the element size is not
fixed), which also will help performance because it'll cut out a lot of
system allocator calls and high order page allocations can be slow.

re: journal pipelining being a performance limit - the reason this is
coming up is because the amount of data we're journalling for a simple
extents btree update has been going up, and up, and up. nowadays, there
are backpointer updates and a whole bunch of accounting updates - check
__trigger_extent() for the full list or look at 'bcachefs list_journal
-a' output, there's compression type accounting, snapshot accounting,
per inode accounting, per replicas accounting...

so, one thing I've been considering is not journalling
backpointers/accounting updates since they're strictly dependent on keys
in the same transaction, i.e. journal replay can correctly regenerate
them. have not seen a perf increase from it in testing yet though, may
have to kill other bottlenecks first.

Kent Overstreet (5):
  bcachefs: Kill journal_res_state.unwritten_idx
  bcachefs: Kill journal_res.idx
  bcachefs: Don't touch journal_buf->data->seq in journal_res_get
  bcachefs: Free journal bufs when not in use
  bcachefs: Increase JOURNAL_BUF_NR

 fs/bcachefs/bcachefs.h      |  1 +
 fs/bcachefs/journal.c       | 97 ++++++++++++++++++++++++++++++-------
 fs/bcachefs/journal.h       | 38 ++++++++++-----
 fs/bcachefs/journal_io.c    | 28 ++++++-----
 fs/bcachefs/journal_types.h | 19 +++++---
 5 files changed, 135 insertions(+), 48 deletions(-)

-- 
2.45.2