All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFC PATCH 0/2] dm-verity: don't let FEC decode into a dm_bufio page that is already in the cache
@ 2026-11-10 13:06 周伟江 (Weijiang Zhou)
  2026-11-10 13:06 ` [RFC PATCH 1/2] dm-verity: don't decode FEC " 周伟江 (Weijiang Zhou)
  2026-11-10 13:06 ` [RFC PATCH 2/2] dm-verity: don't restart or panic on errors during system shutdown 周伟江 (Weijiang Zhou)
  0 siblings, 2 replies; 12+ messages in thread
From: 周伟江 (Weijiang Zhou) @ 2026-11-10 13:06 UTC (permalink / raw)
  To: dm-devel@lists.linux.dev
  Cc: Alasdair Kergon, Mike Snitzer, Mikulas Patocka,
	Benjamin Marzinski, Sami Tolvanen,
	金红宇 (Hongyu Jin),
	吴祥禄 (Xianglu Wu),
	吴周镜 (Justin Wu)

Hi,

We would like to ask for your help with something we ran into on dm-verity.

On 6.18 we are seeing a hash block being reported as corrupted while FEC is
in the middle of recovering it, and we suspect the reason is that the block
becomes visible in the dm_bufio cache before FEC has written it. It looks to
us like a regression from 51ba14fb368e (6.15), but we would very much like a
second opinion before going any further.

Two patches follow, as an RFC: they have not been built or run on a device
yet, and we would rather agree on the direction with you first.

1. Root cause

As far as we can tell, verity_verify_level() handles a failed hash block
read by taking a fresh buffer with dm_bufio_new() and decoding into it with
verity_fec_decode():

    data = dm_bufio_new(v->bufio, hash_block, &buf);        /* <-- */

Our understanding is that dm_bufio_new() inserts that buffer into v->bufio
immediately, with no B_READING and no read_error, and never submits I/O for
it. While FEC runs, the cache would then hold a page that cannot be told
apart from a block which was read successfully and simply has not been hash-
checked yet, while its contents are whatever the allocator returned.

FEC then walks the RS codeword and, for the message blocks that live on the
data device, calls verity_hash_for_block() to decide which of them are
erasures - through the same v->bufio. So if one of those data blocks happens
to be covered by the hash block being recovered, the nested
verity_verify_level() would take that placeholder out of the cache, hash
zeros, get a mismatch and, since nested FEC is refused (fio->level != 0 ->
-EIO), end up in verity_handle_err(), which reports corruption.

Both the publication and the read-back happen in the same thread, so it does
not look like a race to us. Whether it triggers seems to depend only on the
geometry.

2. Symptom

Unisoc ums9635, Android 17, kernel 6.18.21, dm-verity over a virtual A/B
system partition: 4K blocks, sha256, fec_roots=2, created with
restart_on_corruption. Rebooting a slot whose snapshot had not been merged
yet, we got:

device-mapper: verity: 254:3: metadata block 229730 is corrupted
reboot: Restarting system with command 'dm-verity device corrupted'

The hash read did fail with -EIO, because the UFS was already offline. The
media itself is fine: we rebuilt that slot from the COW and verified both
slots' hash trees offline, zero bad blocks, and block 229730 contains
exactly the digest the tree expects.

If our reading is right, any failed hash read can trigger this, not just one
during shutdown. With 228497 data blocks, 251 of the 1786 level-0 hash
blocks, all 14 level-1 blocks and the root block have a codeword data block
underneath them, so an I/O error on any of those would be reported as
corruption. android16-6.12 has no 51ba14fb368e, and there it simply returns
-EIO.

3. Code evidence

    /* drivers/md/dm-verity-target.c, verity_verify_level() */
    if (IS_ERR(data)) {
            if (skip_unverified)
                    return 1;
            r = PTR_ERR(data);
            data = dm_bufio_new(v->bufio, hash_block, &buf);        /* <-- */

    /* drivers/md/dm-verity-fec.c, fec_read_bufs() */
    /* locate erasures if the block is on the data device */
    if (bufio == v->fec->data_bufio &&
        verity_hash_for_block(v, io, block, want_digest, &is_zero) == 0) {

What we can add from the minidump, all of it in thread T329 (kverityd):

- block 229730 was read from the device twice; the nested read issued no I/O
  and was served from the cache
- T329 was preempted 171 times between the failed read and the report,
  matching the 171 successful reads of blocks i = 0..170 before the nested
  one
- kprobe: real digest 8b1f331f... equals sha256(salt || 4096 * 0x00), wanted
  digest bac80063... equals offset 0x840 of L1 block 228507 in the image we
  rebuilt offline
- dump, report and restart all carry TID 329

In our case it happens at i = 171: message block 171 * 911 + 158 = 155939,
whose leaf hash is 228512 + (155939 >> 7) = 229730 (region_blocks 911 from
fec_blocks 230298 and roots 2, index_in_region 158).

The guards we looked at do not seem to cover this path: 252bd1256396 in
verity_end_io applies to the outer bio, which succeeded; verity_finish_io
checks bio completion, while verity_handle_err() has no system_state check;
and dm_bufio_release only drops buffers with read_error set, which an
NF_FRESH buffer does not have. If we are missing one, we would be glad to be
corrected.

4. Reproducer, no device and no shutdown needed

1. put a dm-dust layer below the verity hash area and mark one level-1 block
   bad
2. drop the page cache and let the bufio cache go
3. read a data block that has that level-1 block as an ancestor

In our hands an unmodified 6.18 reports "metadata block N is corrupted"
every time, single threaded. We would be happy to turn this into a proper
reproducer if it is useful.

5. Patch 1

What we have tried so far is to decode into a private buffer, so that
nothing enters v->bufio before it holds recovered data:

    fec_buf = kmalloc(1 << v->hash_dev_block_bits, GFP_NOIO);
    if (!fec_buf)
            return r;
    if (verity_fec_decode(v, io, DM_VERITY_BLOCK_TYPE_METADATA,
                          want_digest, hash_block, fec_buf) == 0) {
            memcpy(want_digest, fec_buf + offset, v->digest_size);
            r = 0;
    }
    kfree(fec_buf);
    return r;

We believe this closes the window rather than narrowing it. A recovered hash
block is no longer cached, so a later access reads it again and, if the
device is still failing, recovers it again - which costs something only on
media that already returns errors, and keeps the change inside the function
that got us here. We also wondered about the other end, having
dm_bufio_new() refuse to publish a buffer before it is filled: that would
hold for every caller rather than just verity, but NF_FRESH exists precisely
so a caller can fill the buffer itself and other targets rely on it, so it
looked like a larger change than this needs. We are quite happy to redo it
there if you prefer.

Patch 2 is unrelated to cache visibility and came out of the same dump:
verity_handle_err() does not consult system_state, so during a shutdown it
still restarts or panics for an error that cannot mean corruption, and
overwrites the reason of the reboot already in progress. It reuses
verity_is_system_shutting_down() and fails the I/O instead. Please look at
the two separately.

6. Questions

1. Have you seen this before, or is a fix already queued somewhere?
   verity_verify_level() and fec_read_bufs() look unchanged in master
   (05441d697f7f), and the v2 review of 51ba14fb368e does not mention cache
   visibility, but you would know better than we do.
2. Would you be willing to check whether our reading of the dm_bufio_new() +
   FEC interaction is right, and whether the shape of patch 1 makes sense?
   If we have it wrong, we would much rather hear it now than later.

The dump comes from a product device so it is not public, but we can extract
anything you would like to see: the raw log_buf, the sched_trace window
around T329, the kprobe output. And of course we are glad to test anything
you suggest on the device.

Thank you for your time, and for the work on dm-verity.

BRs
weijiang.zhou
Unisoc ota team
________________________________
 This email (including its attachments) is intended only for the person or entity to which it is addressed and may contain information that is privileged, confidential or otherwise protected from disclosure. Unauthorized use, dissemination, distribution or copying of this email or the information herein or taking any action in reliance on the contents of this email or the information herein, by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is strictly prohibited. If you are not the intended recipient, please do not read, copy, use or disclose any part of this e-mail to others. Please notify the sender immediately and permanently delete this e-mail and any attachments if you received it in error. Internet communications cannot be guaranteed to be timely, secure, error-free or virus-free. The sender does not accept liability for any errors or omissions.
本邮件及其附件具有保密性质,受法律保护不得泄露,仅发送给本邮件所指特定收件人。严禁非经授权使用、宣传、发布或复制本邮件或其内容。若非该特定收件人,请勿阅读、复制、 使用或披露本邮件的任何内容。若误收本邮件,请从系统中永久性删除本邮件及所有附件,并以回复邮件的方式即刻告知发件人。无法保证互联网通信及时、安全、无误或防毒。发件人对任何错漏均不承担责任。

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2026-11-13  5:55 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-10 13:06 [RFC PATCH 0/2] dm-verity: don't let FEC decode into a dm_bufio page that is already in the cache 周伟江 (Weijiang Zhou)
2026-11-10 13:06 ` [RFC PATCH 1/2] dm-verity: don't decode FEC " 周伟江 (Weijiang Zhou)
2026-11-10 21:08   ` Mikulas Patocka
2026-11-11  3:14     ` 周伟江 (Weijiang Zhou)
2026-11-11 23:33       ` Mikulas Patocka
2026-11-12  0:38     ` Sami Tolvanen
2026-11-13  5:36     ` Eric Biggers
2026-11-10 13:06 ` [RFC PATCH 2/2] dm-verity: don't restart or panic on errors during system shutdown 周伟江 (Weijiang Zhou)
2026-11-10 21:13   ` Mikulas Patocka
2026-11-11  3:14     ` 周伟江 (Weijiang Zhou)
2026-11-11 22:48       ` Mikulas Patocka
2026-11-13  5:55   ` Eric Biggers

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.