Metadata rereplication not triggering

[Chris Webb <chris@arachsys.com>]

[Spotted while double-checking c85e27c45512 fixed the rereplicate BUG_ON at
fs/bcachefs/btree_iter.c:2475 the other day.]

If I create a filesystem with --replicas=2, fail a component drive and
replace with a new one, then use bcachefs data rereplicate, metadata
doesn't seem to get copied to the new drive.

bcachefs fs usage suggests that all the data contents of the filesystem do
get correctly rereplicated; it's just the metadata that's missing.

Here's a minimal ktest to reproduce - I'll follow up with a patch to
tests/bcachefs/replication.ktest:

  test_replace_replica()
  {
      bcachefs format --errors=panic --replicas=2 /dev/sd[bc]
      mount -t bcachefs -o degraded /dev/sdb /mnt

      bcachefs device add -f /mnt /dev/sdd
      bcachefs data rereplicate /mnt
      umount /mnt

      mount -t bcachefs -o degraded /dev/sdd /mnt
      umount /mnt
  }

The second mount (of the new replacement drive alone) fails:

  00016 bcachefs (dev-1): btree write error: device removed
  00016 bcachefs (a5f018b0-5a18-40b9-9539-b8d2325a4dc4): insufficient devices online (0) for replicas entry btree: 1/1 [0]
  00016 bcachefs: bch2_fs_open() bch_fs_open err opening /dev/sdd: insufficient devices
  00016 mount: /mnt: wrong fs type, bad option, bad superblock on /dev/sdd, missing codepage or helper program, or other error.

Mounting /dev/sdb solo at this point will work fine, as that does contain the
metadata + data, as does /dev/sdc, albeit for an older version of the fs.

The specific procedure here don't seem to matter: you can mount the
filesystem non-degraded, force remove a drive then add a new one, etc. I've
not found a scenario where I'm able to successfully trigger metadata
rereplication.

/sys/fs/bcachefs/*/options/data_replicas and metadata_replicas are both
still 2 at the point the new device is added and following data replication.
(Setting them explicitly 'just in case' doesn't change the behaviour.)

I'm still learning my way around fs/bcachefs, but there seems to be code to
replicate metadata already in tree - looks like it should be automatic - so I
wonder if it's not getting triggered when the replacement is added any more?

Best wishes,

Chris.