[RFC] [PATCH 00/48] Validate journal log items during log recovery.

[Dave Chinner <dgc@kernel.org>]

Log recovery reads log items straight from the journal and acts on
them with very little checking. Validation of each item type is
scattered and much of it runs only as the item is being applied to
the filesystem. Corrupt or malicious log contents can therefore be
partially acted on before anything notices they are wrong, and the
checks that do exist are inconsistent from one item type to the
next.

This series makes log item validation a first-class, up-front step.
As each item is decoded from the journal it is fully validated
before any recovery code acts on it, so downstream recovery can
trust the item's structure and contents. The validation is driven by
per-item-type callbacks on the recovery item operations, so every
item type is checked the same way: the region count is checked
against what the item type and its logged fields require, each
region is bounds checked, and the whole item is checked for internal
consistency. Existing validation that was scattered through the
commit handlers is brought forward into these callbacks, and the
now-redundant checks are removed so the commit handlers are left to
do only replay.

The operation header decode and the transaction header are validated
the same way, so the record framing is checked before the items
within it are assembled.

The scope is the on-disk log format and the in-core structures
recovery builds from it. Every log item type is covered. The
validation is what can be checked from the log item alone together
with stable mount and superblock geometry.

Some checks cannot be made at decode time and are deliberately left
to the first runtime access after recovery. Realtime group bounds
are not available because the realtime inodes are not set up until
after recovery. Filesystem block number bounds cannot be trusted at
decode time because a growfs superblock update is not replayed into
the in-core superblock until pass two. Hence extent and pointer block
range validation will need to be reworked before it can be relied
on. More detail is below under known issues.

The known issues will take a little time to be addressed, so I'd
prefer to get feedback on the rest of the patchset before I spend
time reworking that aspect of it.  Early(-ish) feedback on the
structure and implementation as it currently stands is more
important right now.

The patchset passes all of fstests on 1k and 4k block sizes (except
for the couple of growfs/recovery stress tests) without regressions,
so it largely works for known good journals.  Anyone who has been
fuzzing journal structures with LLMs and/or modified syzbots are
welcome to try it and see what holes they can find in it.

The entire series can be pulled from here:

https://git.kernel.org/pub/scm/linux/kernel/git/dgc/linux-xfs.git log-verification-1

Known issues:

1. Validation runs repeatedly on every recovery pass

Except for the CRC validation pass.

Pass 1 needs to validate all the transaction/checkpoint structures
as we walk the entire active journal to find cancellation items. We
also need to validate buffer and quotaoff items as tehy are recorded
in cancelation tables.

Pass 2 readahead needs items that are read ahead to be validated
sufficiently to ensure the location being read from disk is likely
to be OK.

Pass 2 needs to fully validate all the items being recovered, before
they are recovered.

Some of the item recovery also reformats the items being recovered;
things like 32bit vs 64bit incore structure differences are being
resolved at item validation time so they don't need to be handled
differently by validation or recovery code. This means that
validation needs to run on those items on every pass that they will
have something run against them. This needs more thought; inodes are
the main issue here, and largely they are only parsed in pass 2.

Hence we might be able to split up validation into pass-1 + pass-2
variants. pass-1 needs to validate the number and size of regions
matches the format header to ensure checkpoint decoding is always
valid. pass-2 can then do the full item validation.


2. Geometry validation versus growfs and shrink

This is an extension of issue 1, in that validation occurs before
replay.  The geometry items validate against is the geometry in
effect at the start of recovery. A geometry change only takes effect
when the primary superblock buffer is replayed, which happens during
transaction recovery in the second pass, not during checkpoint
region decoding.

A growfs operation logs the superblock geometry change ahead of any
metadata that uses the new geometry. A shrink also logs the
superblock geometry change after relevant internal metadata has been
updated and/or invalidated. Hence if the superblock geometry changes
during recovery, item validation at region decode time will fail to
validate geometry-based fields correctly.

This affects any item that carries a type that encodes geometry
information, such as inode numbers, disk addresses, and extents,
including the realtime variants.  Structural validation that does
not depend on geometry is unaffected.

As the geometry change only takes effect during transaction
recovery, any validation that depends on it must run after the
superblock buffer has been replayed.

This causes issues for readahead validation, which decodes
addresses and submits IO, potentially for addresses outside the
range of the initial superblock geometry. Readahead silently
discards reads beyond the end of the filesystem, but we shouldn't
issue them in the first place.

Even in pass 2, the checkpoint where the geometry change is
recorded cannot be validated until recovery even though items in the
checkpoint are correctly ordered either side of the change.

Hence it is likely that we need two layers of item validation for
pass 2 - the decode time validation of everything except disk
addresses, and recovery time validation of addresses. This will
order all disk address based validation correctly against
grow/shrink operations.

3. fold ->complete() into ->validate_item()

The xlog_recover_item_ops ->complete() callback was added early on,
when it was not entirely clear how item validation would end u
plooking. In the end, it is used by only one special item type:
the transaction header.

->validate_item() already runs on every item at that same point, so
->complete() probably should be folded into it. ->validate_item()
would perform the header copy and signal that the item is consumed,
removing the need for ->complete() altogether.

Commits:
28f113a   xfs: set the unmount record operation header length
a61b098   xfs: add log recovery item validation design document
06c7593   xfs: lift transaction header parsing out of the region assembly
a68100c   xfs: track transaction header decode state explicitly
24e21ca   xfs: track the log item being rebuilt with an explicit pointer
d91765d   xfs: complete log items explicitly during recovery
1927333   xfs: add log item validation infrastructure to log recovery
cacb405   xfs: recover the transaction header as a validated item type
0703ad7   xfs: factor out xlog_recover_validate_ophdr()
fc50f95   xfs: validate operation header length and alignment during recovery
3d16b7b   xfs: validate the operation header padding during recovery
fe25cc9   xfs: validate the operation header flags during recovery
c2268a5   xfs: enforce zero length control records during recovery
7ea3fb8   xfs: validate the unmount record during recovery
dfd687d   xfs: handle zero length continuation op headers during recovery
74f2613   xfs: bound the assembled region size during recovery
1de7ba8   xfs: dump the corrupt region when log item validation fails
3c87e627   xfs: validate fixed size log items during recovery
8a79bbf   xfs: validate extent free log items during recovery
1bb8381   xfs: validate and convert extent free intents during recovery
426c695   xfs: validate and convert extent free done items during recovery
425e959   xfs: validate rmap update intents during recovery
2549fef   xfs: validate rmap update done items during recovery
80bb17f   xfs: validate refcount update intents during recovery
d10d3f8   xfs: validate refcount update done items during recovery
6061b3b   xfs: validate bmap update intents during recovery
aa3c496   xfs: validate bmap update done items during recovery
27521ef   xfs: validate mapping exchange intents during recovery
3773819   xfs: validate mapping exchange done items during recovery
e3b1f31   xfs: validate dquot items during recovery
96008da   xfs: validate attr done items during recovery
b21e835   xfs: validate attr intents during recovery
e91bfa9   xfs: convert inode items to host format during recovery
2a6df3a   xfs: allow xfs_imap() to map without an inode btree lookup
c661e55   xfs: validate the inode item format header during recovery
71dd594   xfs: validate the logged inode core during recovery
0a30aae   xfs: validate the logged inode forks during recovery
e1d4af6   xfs: validate logged local fork contents during recovery
88362e8   xfs: validate logged extent fork contents during recovery
1f1bb06   xfs: allow a long format btree root to be checked without a cursor
7e1242f   xfs: validate logged btree fork roots during recovery
34f073c   xfs: validate logged metadata btree fork roots during recovery
4a383f7   xfs: validate buffer item region count during recovery
9503237   xfs: validate buffer item regions during recovery
cc5eed7   xfs: validate the whole buffer item during recovery
f5855c0   xfs: remove redundant buffer checks from recovery replay
e0a8bd9   xfs: fail recovery on a buffer type mismatch
44cad35   xfs: validate logged dquot buffers during recovery

Diffstat:
 Documentation/filesystems/xfs/index.rst                              |   1 +-
 Documentation/filesystems/xfs/xfs-log-recovery-validation-design.rst | 514 ++++-
 fs/xfs/libxfs/xfs_bmap.c                                             |  48 +-
 fs/xfs/libxfs/xfs_bmap.h                                             |   8 +-
 fs/xfs/libxfs/xfs_btree.c                                            |  50 +-
 fs/xfs/libxfs/xfs_btree.h                                            |   4 +-
 fs/xfs/libxfs/xfs_dquot_buf.c                                        |  64 +-
 fs/xfs/libxfs/xfs_format.h                                           |   8 +-
 fs/xfs/libxfs/xfs_ialloc.c                                           |   7 +-
 fs/xfs/libxfs/xfs_inode_buf.c                                        |  19 +-
 fs/xfs/libxfs/xfs_inode_buf.h                                        |   2 +-
 fs/xfs/libxfs/xfs_inode_fork.c                                       |  25 +-
 fs/xfs/libxfs/xfs_log_format.h                                       |   8 +-
 fs/xfs/libxfs/xfs_log_recover.h                                      |  64 +-
 fs/xfs/libxfs/xfs_quota_defs.h                                       |   2 +-
 fs/xfs/scrub/inode_repair.c                                          |  14 +-
 fs/xfs/xfs_attr_item.c                                               | 283 +-
 fs/xfs/xfs_bmap_item.c                                               | 109 +-
 fs/xfs/xfs_buf_item_recover.c                                        | 349 +-
 fs/xfs/xfs_dquot_item_recover.c                                      | 118 +-
 fs/xfs/xfs_exchmaps_item.c                                           |  46 +-
 fs/xfs/xfs_extfree_item.c                                            | 368 ++-
 fs/xfs/xfs_icache.h                                                  |   2 +-
 fs/xfs/xfs_icreate_item.c                                            |  18 +-
 fs/xfs/xfs_inode.h                                                   |  43 +-
 fs/xfs/xfs_inode_item_recover.c                                      | 979 +++++--
 fs/xfs/xfs_log.c                                                     |  16 +-
 fs/xfs/xfs_log_recover.c                                             | 764 ++++-
 fs/xfs/xfs_refcount_item.c                                           | 163 +-
 fs/xfs/xfs_rmap_item.c                                               | 163 +-
 30 files changed, 3410 insertions(+), 849 deletions(-)