* "netfs: Can't donate prior to front" @ 2025-02-07 18:40 Max Kellermann 2025-02-10 14:07 ` David Howells 2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann 0 siblings, 2 replies; 13+ messages in thread From: Max Kellermann @ 2025-02-07 18:40 UTC (permalink / raw) To: David Howells, netfs, LKML, linux-nfs Hi, the following crash occurs with 6.13.1 on our servers every 20 minutes or so: netfs: Can't donate prior to front R=00070d30[3] s=9a000-9bfff 0/2000/2000 folio: 98000-9bfff donated: prev=0 next=0 s=9a000 av=2000 part=2000 ------------[ cut here ]------------ kernel BUG at fs/netfs/read_collect.c:315! Oops: invalid opcode: 0000 [#1] SMP PTI CPU: 7 UID: 0 PID: 0 Comm: swapper/7 Not tainted 6.13.1-cm4all2-hp #416 Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 11/23/2021 RIP: 0010:netfs_consume_read_data.isra.0+0xa72/0xab0 Code: 48 89 ea 31 f6 48 c7 c7 bb 7a d0 ae e8 b7 d2 d1 ff 48 8b 4c 24 20 4c 89 e2 48 c7 c7 d7 7a d0 ae 48 8b 74 24 18 e8 9e d2 d1 ff <0f> 0b 4c 89 ef 48 89 54 24 10 4c 89 44 24 08 e8 1a 4e b5 00 48 c7 RSP: 0018:ffffb434cc448db0 EFLAGS: 00010246 RAX: 0000000000000019 RBX: ffff8fa63d9cbec0 RCX: 0000000000000027 RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8fbb1f9db840 RBP: 0000000000000000 R08: 00000000ffffbfff R09: 0000000000000001 R10: 0000000000000003 R11: ffff8fd31f6a0000 R12: 0000000000002000 R13: ffff8fa5350aaee8 R14: 0000000000004000 R15: ffff8fa5350aaee8 FS: 0000000000000000(0000) GS:ffff8fbb1f9c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f9c5000ef48 CR3: 0000000bcee2e001 CR4: 00000000001706f0 Call Trace: <IRQ> ? die+0x32/0x80 ? do_trap+0xd8/0x100 ? do_error_trap+0x65/0x80 ? netfs_consume_read_data.isra.0+0xa72/0xab0 ? exc_invalid_op+0x4c/0x60 ? netfs_consume_read_data.isra.0+0xa72/0xab0 ? asm_exc_invalid_op+0x16/0x20 ? netfs_consume_read_data.isra.0+0xa72/0xab0 ? __pfx_cachefiles_read_complete+0x10/0x10 netfs_read_subreq_terminated+0x22d/0x370 cachefiles_read_complete+0x48/0xf0 iomap_dio_bio_end_io+0x125/0x160 blk_update_request+0xea/0x3e0 scsi_end_request+0x27/0x190 scsi_io_completion+0x43/0x6c0 blk_complete_reqs+0x40/0x50 handle_softirqs+0xd1/0x280 irq_exit_rcu+0x91/0xb0 common_interrupt+0x79/0xa0 </IRQ> <TASK> asm_common_interrupt+0x22/0x40 RIP: 0010:cpuidle_enter_state+0xba/0x3b0 Code: 00 e8 ea 86 1c ff e8 45 f7 ff ff 8b 53 04 49 89 c5 0f 1f 44 00 00 31 ff e8 73 b9 1b ff 45 84 ff 0f 85 f8 01 00 00 fb 45 85 f6 <0f> 88 46 01 00 00 48 8b 04 24 49 63 ce 48 6b d1 68 49 29 c5 48 89 RSP: 0018:ffffb434c018be98 EFLAGS: 00000202 RAX: ffff8fbb1f9c0000 RBX: ffffd41cbe7e3448 RCX: 000000000000001f RDX: 0000000000000007 RSI: 000000003149acb2 RDI: 0000000000000000 RBP: 0000000000000004 R08: 0000000000000002 R09: 0000000000000000 R10: 0000000000000004 R11: 000000000000001f R12: ffffffffaf660060 R13: 0000030f6179fa73 R14: 0000000000000004 R15: 0000000000000000 ? cpuidle_enter_state+0xad/0x3b0 cpuidle_enter+0x29/0x40 do_idle+0x19c/0x200 cpu_startup_entry+0x25/0x30 start_secondary+0xf3/0x100 common_startup_64+0x13e/0x148 </TASK> Modules linked in: ---[ end trace 0000000000000000 ]--- This is a server with heavy NFS traffic (with fscache enabled). Please help - and let me know if you need more information. Max ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: "netfs: Can't donate prior to front" 2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann @ 2025-02-10 14:07 ` David Howells 2025-02-10 17:41 ` Max Kellermann 2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann 1 sibling, 1 reply; 13+ messages in thread From: David Howells @ 2025-02-10 14:07 UTC (permalink / raw) To: Max Kellermann; +Cc: dhowells, netfs, LKML, linux-nfs Can you apply the attached patch to your kernel, and then run with: echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_read/enable echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_rreq/enable echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_sreq/enable echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_failure/enable echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_donate/enable echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_progress/enable enabled. If you can capture the trace output (and compress it!), that should hopefully help debug this. David diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c index e8624f5c7fcc..8b78c1ec0677 100644 --- a/fs/netfs/read_collect.c +++ b/fs/netfs/read_collect.c @@ -312,6 +312,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was printk("folio: %llx-%llx\n", fpos, fend - 1); printk("donated: prev=%zx next=%zx\n", prev_donated, next_donated); printk("s=%llx av=%zx part=%zx\n", start, avail, part); + tracing_off(); BUG(); } ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: "netfs: Can't donate prior to front" 2025-02-10 14:07 ` David Howells @ 2025-02-10 17:41 ` Max Kellermann 2025-02-10 17:45 ` Max Kellermann 0 siblings, 1 reply; 13+ messages in thread From: Max Kellermann @ 2025-02-10 17:41 UTC (permalink / raw) To: David Howells; +Cc: netfs, LKML, linux-nfs [-- Attachment #1: Type: text/plain, Size: 1751 bytes --] On Mon, Feb 10, 2025 at 3:08 PM David Howells <dhowells@redhat.com> wrote: > Can you apply the attached patch to your kernel, and then run with: > > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_read/enable > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_rreq/enable > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_sreq/enable > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_failure/enable > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_donate/enable > echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_progress/enable > > enabled. If you can capture the trace output (and compress it!), that should > hopefully help debug this. I needed some more tricks to be able to capture the trace of a crashing kernel, but here it is (a minimal trace because I now figured out how to reproduce the bug with a single command on a disabled machine). Additionally, here you have a file with a dump of *subreq and *req (of R=8). I have added additional fields "start0" and "len0" which contain the values initially set on these. Kernel output just before the crash: 3,1644,103679537,-;netfs: Can't donate prior to front 4,1645,103679550,-;R=00000007[6] s=187000-1bffff 0/39000/39000 4,1646,103679562,-;folio: 180000-1bffff 4,1647,103679568,-;donated: prev=0 next=0 4,1648,103679574,-;s=187000 av=39000 part=39000 3,1649,103708094,-;netfs: Can't donate prior to front 4,1650,103708105,-;R=00000008[7] s=1c6000-1fffff 0/3a000/3a000 4,1651,103708117,-;folio: 1c0000-1fffff 4,1652,103708123,-;donated: prev=0 next=0 4,1653,103708129,-;s=1c6000 av=3a000 part=3a000 The trace file contains just the R=8 request, but I guess this one is enough for you. Max [-- Attachment #2: trace8.txt --] [-- Type: text/plain, Size: 6737 bytes --] kworker/u193:5-3716 [003] ..... 103.651806: netfs_sreq: R=00000008[1] WRIT PREP f=00 s=c0000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.651808: netfs_sreq: R=00000008[1] WRIT SUBMT f=100 s=c0000 0/40000 e=0 kworker/3:2-4269 [003] ..... 103.652314: netfs_sreq: R=00000008[1] WRIT TERM f=100 s=c0000 40000/40000 e=0 kworker/u193:5-3716 [003] ..... 103.652343: netfs_rreq: R=00000008 2C COLLECT f=2020 kworker/u193:5-3716 [003] ..... 103.652349: netfs_sreq: R=00000008[1] WRIT FREE f=00 s=c0000 40000/40000 e=0 kworker/u193:5-3716 [003] ..... 103.652357: netfs_rreq: R=00000008 2C WR-DONE f=2020 kworker/u193:5-3716 [003] ..... 103.652360: netfs_rreq: R=00000008 2C WAKE-IP f=2020 kworker/u193:5-3716 [003] ..... 103.652362: netfs_rreq: R=00000008 2C FREE f=2000 cp-4373 [038] ..... 103.724132: netfs_read: R=00000008 READAHEAD c=00000002 ni=76464c3 s=1c0000 l=40000 sz=776749 cp-4373 [038] b.... 103.724170: netfs_sreq: R=00000008[1] ---- ADD f=00 s=1c0000 0/40000 e=0 cp-4373 [038] b.... 103.724470: netfs_sreq: R=00000008[2] ---- ADD f=00 s=1d0000 0/30000 e=0 cp-4373 [038] b.... 103.724753: netfs_sreq: R=00000008[3] ---- ADD f=00 s=1d4000 0/2c000 e=0 cp-4373 [038] ..... 103.724767: netfs_sreq: R=00000008[3] READ SUBMT f=00 s=1d4000 0/3000 e=0 cp-4373 [038] b.... 103.724845: netfs_sreq: R=00000008[4] ---- ADD f=00 s=1d7000 0/29000 e=0 cp-4373 [038] b.... 103.725119: netfs_sreq: R=00000008[5] ---- ADD f=00 s=1e7000 0/19000 e=0 cp-4373 [038] b.... 103.725389: netfs_sreq: R=00000008[6] ---- ADD f=00 s=1f1000 0/f000 e=0 cp-4373 [038] ..... 103.725403: netfs_sreq: R=00000008[6] READ SUBMT f=00 s=1f1000 0/3000 e=0 cp-4373 [038] b.... 103.725478: netfs_sreq: R=00000008[7] ---- ADD f=00 s=1f4000 0/c000 e=0 cp-4373 [038] b.... 103.725751: netfs_sreq: R=00000008[8] ---- ADD f=00 s=1fa000 0/6000 e=0 cp-4373 [038] ..... 103.725767: netfs_sreq: R=00000008[8] READ SUBMT f=00 s=1fa000 0/6000 e=0 <idle>-0 [038] ..s.. 103.729472: netfs_progress: R=00000008[06] s=1f1000 ct=0/3000 pa=3000/3000 sl=0 <idle>-0 [038] b.s.. 103.729482: netfs_donate: R=00000008[06] -> [07] to-next am=3000 <idle>-0 [038] b.s.. 103.729483: netfs_sreq: R=00000008[6] READ DON-N f=00 s=1f1000 3000/3000 e=0 <idle>-0 [038] ..s.. 103.729486: netfs_sreq: R=00000008[6] READ TERM f=10 s=1f1000 0/0 e=0 <idle>-0 [038] ..s.. 103.729487: netfs_sreq: R=00000008[6] READ FREE f=10 s=1f1000 0/0 e=0 <idle>-0 [038] ..s.. 103.735784: netfs_progress: R=00000008[03] s=1d4000 ct=0/3000 pa=3000/3000 sl=0 <idle>-0 [038] b.s.. 103.735792: netfs_donate: R=00000008[03] -> [04] to-next am=3000 <idle>-0 [038] b.s.. 103.735793: netfs_sreq: R=00000008[3] READ DON-N f=00 s=1d4000 3000/3000 e=0 <idle>-0 [038] ..s.. 103.735796: netfs_sreq: R=00000008[3] READ TERM f=10 s=1d4000 0/0 e=0 <idle>-0 [038] ..s.. 103.735798: netfs_sreq: R=00000008[3] READ FREE f=10 s=1d4000 0/0 e=0 <idle>-0 [038] ..s.. 103.736603: netfs_progress: R=00000008[08] s=1fa000 ct=0/6000 pa=6000/6000 sl=0 <idle>-0 [038] b.s.. 103.736612: netfs_donate: R=00000008[08] -> [07] tail-to-prev am=6000 <idle>-0 [038] b.s.. 103.736613: netfs_sreq: R=00000008[8] READ DON-P f=00 s=200000 0/0 e=0 <idle>-0 [038] ..s.. 103.736616: netfs_sreq: R=00000008[8] READ TERM f=10 s=200000 0/0 e=0 <idle>-0 [038] ..s.. 103.736618: netfs_sreq: R=00000008[8] READ FREE f=10 s=200000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.780260: netfs_progress: R=00000008[01] s=1c0000 ct=0/10000 pa=10000/10000 sl=0 kworker/u193:5-3716 [003] b.... 103.780269: netfs_donate: R=00000008[01] -> [02] to-next am=10000 kworker/u193:5-3716 [003] b.... 103.780271: netfs_sreq: R=00000008[1] DOWN DON-N f=01 s=1c0000 10000/10000 e=0 kworker/u193:5-3716 [003] ..... 103.780272: netfs_sreq: R=00000008[1] DOWN TERM f=11 s=1c0000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.780273: netfs_sreq: R=00000008[1] DOWN FREE f=11 s=1c0000 0/0 e=0 kworker/u193:5-3716 [003] b.... 103.781527: netfs_sreq: R=00000008[2] DOWN +DON f=01 s=1c0000 14000/14000 e=0 kworker/u193:5-3716 [003] ..... 103.781536: netfs_progress: R=00000008[02] s=1c0000 ct=0/14000 pa=14000/14000 sl=0 kworker/u193:5-3716 [003] b.... 103.781538: netfs_donate: R=00000008[02] -> [04] to-next am=14000 kworker/u193:5-3716 [003] b.... 103.781539: netfs_sreq: R=00000008[2] DOWN DON-N f=01 s=1c0000 14000/14000 e=0 kworker/u193:5-3716 [003] ..... 103.781540: netfs_sreq: R=00000008[2] DOWN TERM f=11 s=1c0000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.781541: netfs_sreq: R=00000008[2] DOWN FREE f=11 s=1c0000 0/0 e=0 kworker/u193:5-3716 [003] b.... 103.783277: netfs_sreq: R=00000008[4] DOWN +DON f=01 s=1c3000 24000/24000 e=0 kworker/u193:5-3716 [003] ..... 103.783286: netfs_progress: R=00000008[04] s=1c3000 ct=0/24000 pa=24000/24000 sl=0 kworker/u193:5-3716 [003] b.... 103.783288: netfs_donate: R=00000008[04] -> [05] to-next am=24000 kworker/u193:5-3716 [003] b.... 103.783289: netfs_sreq: R=00000008[4] DOWN DON-N f=01 s=1c3000 24000/24000 e=0 kworker/u193:5-3716 [003] ..... 103.783290: netfs_sreq: R=00000008[4] DOWN TERM f=11 s=1c3000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.783291: netfs_sreq: R=00000008[4] DOWN FREE f=11 s=1c3000 0/0 e=0 kworker/u193:5-3716 [003] b.... 103.784366: netfs_sreq: R=00000008[5] DOWN +DON f=01 s=1c3000 2e000/2e000 e=0 kworker/u193:5-3716 [003] ..... 103.784372: netfs_progress: R=00000008[05] s=1c3000 ct=0/2e000 pa=2e000/2e000 sl=0 kworker/u193:5-3716 [003] b.... 103.784374: netfs_donate: R=00000008[05] -> [07] to-next am=2e000 kworker/u193:5-3716 [003] b.... 103.784375: netfs_sreq: R=00000008[5] DOWN DON-N f=01 s=1c3000 2e000/2e000 e=0 kworker/u193:5-3716 [003] ..... 103.784376: netfs_sreq: R=00000008[5] DOWN TERM f=11 s=1c3000 0/0 e=0 kworker/u193:5-3716 [003] ..... 103.784377: netfs_sreq: R=00000008[5] DOWN FREE f=11 s=1c3000 0/0 e=0 kworker/u193:5-3716 [003] b.... 103.785425: netfs_sreq: R=00000008[7] DOWN +DON f=01 s=1c6000 3a000/3a000 e=0 kworker/u193:5-3716 [003] ..... 103.785432: netfs_progress: R=00000008[07] s=1c6000 ct=0/3a000 pa=3a000/3a000 sl=0 [-- Attachment #3: gdb.txt --] [-- Type: text/plain, Size: 5412 bytes --] (gdb) p/x *rreq $3 = { { work = { data = { counter = 0xfffffffe00000 }, entry = { next = 0xffff8898b7df6908, prev = 0xffff8898b7df6908 }, func = 0xffffffff81b44e90 }, rcu = { next = 0xfffffffe00000, func = 0xffff8898b7df6908 } }, inode = 0xffff889886939518, mapping = 0xffff889886939670, iocb = 0x0, cache_resources = { ops = 0xffffffff845fd480, cache_priv = 0xffff8898b876ed20, cache_priv2 = 0xffff8898a7b29700, debug_id = 0x2, inval_counter = 0x0 }, ractl = 0xffff8898b88b7980, proc_link = { next = 0xffffffff857668a0, prev = 0xffff8898b7df3c60 }, subrequests = { next = 0xffff8898b87fd568, prev = 0xffff8898b87fd568 }, io_streams = {{ construct = 0x0, sreq_max_len = 0x10000, sreq_max_segs = 0x0, submit_off = 0x0, submit_len = 0x0, submit_extendable_to = 0x0, prepare_write = 0x0, issue_write = 0x0, subrequests = { next = 0xffff8898b7df69b0, prev = 0xffff8898b7df69b0 }, front = 0x0, collected_to = 0x0, transferred = 0x0, source = 0x0, error = 0x0, stream_nr = 0x0, avail = 0x0, active = 0x0, need_retry = 0x0, failed = 0x0 }, { construct = 0x0, sreq_max_len = 0x0, sreq_max_segs = 0x0, submit_off = 0x0, submit_len = 0x0, submit_extendable_to = 0x0, prepare_write = 0x0, issue_write = 0x0, subrequests = { next = 0xffff8898b7df6a18, prev = 0xffff8898b7df6a18 }, front = 0x0, collected_to = 0x0, transferred = 0x0, source = 0x0, error = 0x0, stream_nr = 0x0, avail = 0x0, active = 0x0, need_retry = 0x0, failed = 0x0 }}, group = 0x0, buffer = 0xffff8898b8000000, buffer_tail = 0xffff8898b8000000, iter = { iter_type = 0x4, nofault = 0x0, data_source = 0x0, iov_offset = 0x0, { __ubuf_iovec = { iov_base = 0xffff8898b8000000, iov_len = 0x0 }, { { __iov = 0xffff8898b8000000, kvec = 0xffff8898b8000000, bvec = 0xffff8898b8000000, folioq = 0xffff8898b8000000, xarray = 0xffff8898b8000000, ubuf = 0xffff8898b8000000 }, count = 0x0 } }, { nr_segs = 0x1, folioq_slot = 0x1, xarray_start = 0x1 } }, io_iter = { iter_type = 0x0, nofault = 0x0, data_source = 0x0, iov_offset = 0x0, { __ubuf_iovec = { iov_base = 0x0, iov_len = 0x0 }, { { __iov = 0x0, kvec = 0x0, bvec = 0x0, folioq = 0x0, xarray = 0x0, ubuf = 0x0 }, count = 0x0 } }, { nr_segs = 0x0, folioq_slot = 0x0, xarray_start = 0x0 } }, netfs_priv = 0xffff8898b041fa00, netfs_priv2 = 0x0, direct_bv = 0x0, direct_bv_count = 0x0, debug_id = 0x8, rsize = 0x0, wsize = 0x7fffffff, subreq_counter = { counter = 0x8 }, nr_group_rel = 0x0, lock = { { rlock = { raw_lock = { { val = { counter = 0x0 }, { locked = 0x0, pending = 0x0 }, { locked_pending = 0x0, tail = 0x0 } } } } } }, nr_outstanding = { counter = 0x1 }, submitted = 0x200000, len0 = 0x40000, len = 0x40000, transferred = 0x0, error = 0x0, origin = 0x0, direct_bv_unpin = 0x0, buffer_head_slot = 0x0, buffer_tail_slot = 0x0, i_size = 0x776749, start0 = 0x1c0000, start = 0x1c0000, issued_to = { counter = 0x0 }, collected_to = 0x0, cleaned_to = 0x0, no_unlock_folio = 0x0, prev_donated = 0x0, ref = { refs = { counter = 0x1 } }, flags = 0x80000020, netfs_ops = 0xffffffff8459dba0, cleanup = 0x0 } (gdb) p/x *subreq $4 = { rreq = 0xffff8898b7df6900, work = { data = { counter = 0xfffffffe00000 }, entry = { next = 0xffff8898b87fd550, prev = 0xffff8898b87fd550 }, func = 0x0 }, rreq_link = { next = 0xffff8898b7df6970, prev = 0xffff8898b7df6970 }, io_iter = { iter_type = 0x4, nofault = 0x0, data_source = 0x0, iov_offset = 0x34000, { __ubuf_iovec = { iov_base = 0xffff8898b8000000, iov_len = 0x6000 }, { { __iov = 0xffff8898b8000000, kvec = 0xffff8898b8000000, bvec = 0xffff8898b8000000, folioq = 0xffff8898b8000000, xarray = 0xffff8898b8000000, ubuf = 0xffff8898b8000000 }, count = 0x6000 } }, { nr_segs = 0x0, folioq_slot = 0x0, xarray_start = 0x0 } }, start0 = 0x1f4000, start = 0x1c6000, len0 = 0xc000, len = 0x3a000, transferred = 0x3a000, consumed = 0x0, prev_donated = 0x0, next_donated = 0x0, ref = { refs = { counter = 0x2 } }, error = 0x0, debug_index = 0x7, nr_segs = 0x0, retry_count = 0x0, source = 0x2, stream_nr = 0x0, curr_folioq_slot = 0x0, curr_folio_order = 0x6, curr_folioq = 0xffff8898b8000000, flags = 0x1 } ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: "netfs: Can't donate prior to front" 2025-02-10 17:41 ` Max Kellermann @ 2025-02-10 17:45 ` Max Kellermann 0 siblings, 0 replies; 13+ messages in thread From: Max Kellermann @ 2025-02-10 17:45 UTC (permalink / raw) To: David Howells; +Cc: netfs, LKML, linux-nfs On Mon, Feb 10, 2025 at 6:41 PM Max Kellermann <max.kellermann@ionos.com> wrote: > Additionally, here you have a file with a dump of *subreq and *req (of > R=8). I have added additional fields "start0" and "len0" which contain > the values initially set on these. btw. this happens because there is only one subrequest (R=8[7]), but the start offset is being rounded down (folio_order=2), thus "fpos<start": (gdb) p/x fpos $5 = 0x1c0000 (gdb) p/x fend $6 = 0x200000 (gdb) p/x start $7 = 0x1c6000 ^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann 2025-02-10 14:07 ` David Howells @ 2025-02-10 19:11 ` Max Kellermann 2025-02-14 12:47 ` David Howells 1 sibling, 1 reply; 13+ messages in thread From: Max Kellermann @ 2025-02-10 19:11 UTC (permalink / raw) To: dhowells, netfs, linux-kernel, linux-nfs; +Cc: Max Kellermann If multiple subrequests donate data to the same "next" request (depending on the subrequest completion order), each of them would overwrite the `prev_donated` field, causing data corruption and a BUG() crash ("Can't donate prior to front"). Fixes: ee4cdf7ba857 ("netfs: Speed up buffered reading") Closes: https://lore.kernel.org/netfs/CAKPOu+_4mUwYgQtRTbXCmi+-k3PGvLysnPadkmHOyB7Gz0iSMA@mail.gmail.com/ Signed-off-by: Max Kellermann <max.kellermann@ionos.com> --- David, this seems to fix the bug for me. Please also check if we need a "donation_changed" check. --- fs/netfs/read_collect.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c index 0d95cdbe5611..681b630b4f06 100644 --- a/fs/netfs/read_collect.c +++ b/fs/netfs/read_collect.c @@ -284,7 +284,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was netfs_trace_donate_to_deferred_next); } else { next = list_next_entry(subreq, rreq_link); - WRITE_ONCE(next->prev_donated, excess); + WRITE_ONCE(next->prev_donated, next->prev_donated + excess); trace_netfs_donate(rreq, subreq, next, excess, netfs_trace_donate_to_next); } -- 2.47.2 ^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann @ 2025-02-14 12:47 ` David Howells 2025-02-20 13:09 ` Max Kellermann 0 siblings, 1 reply; 13+ messages in thread From: David Howells @ 2025-02-14 12:47 UTC (permalink / raw) To: Max Kellermann; +Cc: dhowells, netfs, linux-kernel, linux-nfs Max Kellermann <max.kellermann@ionos.com> wrote: > David, this seems to fix the bug for me. Please also check if we need > a "donation_changed" check. Shouldn't do since at that point we've been holding rreq->lock since the last check. > If multiple subrequests donate data to the same "next" request > (depending on the subrequest completion order), each of them would > overwrite the `prev_donated` field, causing data corruption and a > BUG() crash ("Can't donate prior to front"). > > Fixes: ee4cdf7ba857 ("netfs: Speed up buffered reading") > Closes: https://lore.kernel.org/netfs/CAKPOu+_4mUwYgQtRTbXCmi+-k3PGvLysnPadkmHOyB7Gz0iSMA@mail.gmail.com/ > Signed-off-by: Max Kellermann <max.kellermann@ionos.com> Signed-off-by: David Howells <dhowells@redhat.com> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-14 12:47 ` David Howells @ 2025-02-20 13:09 ` Max Kellermann 2025-02-20 14:17 ` Greg Kroah-Hartman 0 siblings, 1 reply; 13+ messages in thread From: Max Kellermann @ 2025-02-20 13:09 UTC (permalink / raw) To: David Howells, Greg Kroah-Hartman; +Cc: netfs, linux-kernel, linux-nfs On Fri, Feb 14, 2025 at 1:47 PM David Howells <dhowells@redhat.com> wrote: > Signed-off-by: David Howells <dhowells@redhat.com> Greg, you merged my other two netfs fixes into 6.13.3, but omitted this one. Did you miss it, or was there another reason? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-20 13:09 ` Max Kellermann @ 2025-02-20 14:17 ` Greg Kroah-Hartman 2025-02-20 15:00 ` Max Kellermann 0 siblings, 1 reply; 13+ messages in thread From: Greg Kroah-Hartman @ 2025-02-20 14:17 UTC (permalink / raw) To: Max Kellermann; +Cc: David Howells, netfs, linux-kernel, linux-nfs On Thu, Feb 20, 2025 at 02:09:17PM +0100, Max Kellermann wrote: > On Fri, Feb 14, 2025 at 1:47 PM David Howells <dhowells@redhat.com> wrote: > > Signed-off-by: David Howells <dhowells@redhat.com> > > Greg, you merged my other two netfs fixes into 6.13.3, but omitted > this one. Did you miss it, or was there another reason? It wasn't sent to me or to the stable list, so how could have I seen it? confused, greg k-h ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-20 14:17 ` Greg Kroah-Hartman @ 2025-02-20 15:00 ` Max Kellermann 2025-02-20 15:10 ` Greg Kroah-Hartman 2025-03-01 14:17 ` Salvatore Bonaccorso 0 siblings, 2 replies; 13+ messages in thread From: Max Kellermann @ 2025-02-20 15:00 UTC (permalink / raw) To: Greg Kroah-Hartman; +Cc: David Howells, netfs, linux-kernel, linux-nfs On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman <gregkh@linuxfoundation.org> wrote: > It wasn't sent to me or to the stable list, so how could have I seen it? Oh, of course, I forgot to add stable. How shall we proceed? Do you want me to resend to you with David's Signed-off-by? ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-20 15:00 ` Max Kellermann @ 2025-02-20 15:10 ` Greg Kroah-Hartman 2025-03-01 14:17 ` Salvatore Bonaccorso 1 sibling, 0 replies; 13+ messages in thread From: Greg Kroah-Hartman @ 2025-02-20 15:10 UTC (permalink / raw) To: Max Kellermann; +Cc: David Howells, netfs, linux-kernel, linux-nfs On Thu, Feb 20, 2025 at 04:00:49PM +0100, Max Kellermann wrote: > On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > It wasn't sent to me or to the stable list, so how could have I seen it? > > Oh, of course, I forgot to add stable. How shall we proceed? Do you > want me to resend to you with David's Signed-off-by? Please do, and cc: stable. thanks, greg k-h ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-02-20 15:00 ` Max Kellermann 2025-02-20 15:10 ` Greg Kroah-Hartman @ 2025-03-01 14:17 ` Salvatore Bonaccorso 2025-03-01 16:51 ` Max Kellermann 1 sibling, 1 reply; 13+ messages in thread From: Salvatore Bonaccorso @ 2025-03-01 14:17 UTC (permalink / raw) To: Max Kellermann Cc: Greg Kroah-Hartman, David Howells, netfs, linux-kernel, linux-nfs Hi (btw side question from a bystander looking a the bug) On Thu, Feb 20, 2025 at 04:00:49PM +0100, Max Kellermann wrote: > On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman > <gregkh@linuxfoundation.org> wrote: > > It wasn't sent to me or to the stable list, so how could have I seen it? > > Oh, of course, I forgot to add stable. How shall we proceed? Do you > want me to resend to you with David's Signed-off-by? Do I see it correctly that this will be 6.12.y and 6.13.y specific backports since the code in mainline changed substantially with e2d46f2ec332 ("netfs: Change the read result collector to only use one work item") and so your change does not apply there anymore? I'm asking since we got a bug report in Debian which seems to idicate it might have the same root cause as you report, and it is at https://bugs.debian.org/1098698 . Regards, Salvatore ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated 2025-03-01 14:17 ` Salvatore Bonaccorso @ 2025-03-01 16:51 ` Max Kellermann 0 siblings, 0 replies; 13+ messages in thread From: Max Kellermann @ 2025-03-01 16:51 UTC (permalink / raw) To: Salvatore Bonaccorso Cc: Greg Kroah-Hartman, David Howells, netfs, linux-kernel, linux-nfs On Sat, Mar 1, 2025 at 3:17 PM Salvatore Bonaccorso <carnil@debian.org> wrote: > Do I see it correctly that this will be 6.12.y and 6.13.y specific > backports since the code in mainline changed substantially with > e2d46f2ec332 ("netfs: Change the read result collector to only use one > work item") and so your change does not apply there anymore? Correct. > I'm asking since we got a bug report in Debian which seems to idicate > it might have the same root cause as you report, and it is at > https://bugs.debian.org/1098698 . This indeed looks similar. Note that this is one of four netfs crash fixes I posted recently. Two other fixes have been released with 6.13.4 already: https://lore.kernel.org/netfs/20250211093432.3524035-1-max.kellermann@ionos.com/ https://lore.kernel.org/netfs/20250210223144.3481766-1-max.kellermann@ionos.com/ The fourth bug (that got no reviews so far) is here (this one affects master as well): https://lore.kernel.org/netfs/20250228123602.2140459-1-max.kellermann@ionos.com/ Max ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated @ 2025-03-07 16:09 Norbert Lange 0 siblings, 0 replies; 13+ messages in thread From: Norbert Lange @ 2025-03-07 16:09 UTC (permalink / raw) To: max.kellermann Cc: Salvatore Bonaccorso, dhowells, gregkh, linux-kernel, linux-nfs, netfs I reproduced with the available versions in debian: linux-image-6.12.12-amd64 6.12.12-1 -> segfault linux-image-6.12.17-amd64 6.12.17-1 -> segfault linux-image-6.13-amd64 6.13.5-1~exp1 -> 'kernel BUG at fs/netfs/read_collect.c:316!' Then I took the debian 6.12.17-1 kernel (latest LTS), added those 3 patches: https://lore.kernel.org/netfs/20250211093432.3524035-1-max.kellermann@ionos.com/ https://lore.kernel.org/netfs/20250210223144.3481766-1-max.kellermann@ionos.com/ https://lore.kernel.org/netfs/20250210191118.3444416-1-max.kellermann@ionos.com/ The resulting kernel apparently fixed the issue, I just testet in Qemu so far (no signed kernel for secure boot). Tested-by: Norbert Lange <nolange79@gmail.com> ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-03-07 16:09 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann 2025-02-10 14:07 ` David Howells 2025-02-10 17:41 ` Max Kellermann 2025-02-10 17:45 ` Max Kellermann 2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann 2025-02-14 12:47 ` David Howells 2025-02-20 13:09 ` Max Kellermann 2025-02-20 14:17 ` Greg Kroah-Hartman 2025-02-20 15:00 ` Max Kellermann 2025-02-20 15:10 ` Greg Kroah-Hartman 2025-03-01 14:17 ` Salvatore Bonaccorso 2025-03-01 16:51 ` Max Kellermann -- strict thread matches above, loose matches on Subject: below -- 2025-03-07 16:09 Norbert Lange
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox