* "netfs: Can't donate prior to front"
@ 2025-02-07 18:40 Max Kellermann
2025-02-10 14:07 ` David Howells
2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann
0 siblings, 2 replies; 12+ messages in thread
From: Max Kellermann @ 2025-02-07 18:40 UTC (permalink / raw)
To: David Howells, netfs, LKML, linux-nfs
Hi,
the following crash occurs with 6.13.1 on our servers every 20 minutes or so:
netfs: Can't donate prior to front
R=00070d30[3] s=9a000-9bfff 0/2000/2000
folio: 98000-9bfff
donated: prev=0 next=0
s=9a000 av=2000 part=2000
------------[ cut here ]------------
kernel BUG at fs/netfs/read_collect.c:315!
Oops: invalid opcode: 0000 [#1] SMP PTI
CPU: 7 UID: 0 PID: 0 Comm: swapper/7 Not tainted 6.13.1-cm4all2-hp #416
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 11/23/2021
RIP: 0010:netfs_consume_read_data.isra.0+0xa72/0xab0
Code: 48 89 ea 31 f6 48 c7 c7 bb 7a d0 ae e8 b7 d2 d1 ff 48 8b 4c 24
20 4c 89 e2 48 c7 c7 d7 7a d0 ae 48 8b 74 24 18 e8 9e d2 d1 ff <0f> 0b
4c 89 ef 48 89 54 24 10 4c 89 44 24 08 e8 1a 4e b5 00 48 c7
RSP: 0018:ffffb434cc448db0 EFLAGS: 00010246
RAX: 0000000000000019 RBX: ffff8fa63d9cbec0 RCX: 0000000000000027
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff8fbb1f9db840
RBP: 0000000000000000 R08: 00000000ffffbfff R09: 0000000000000001
R10: 0000000000000003 R11: ffff8fd31f6a0000 R12: 0000000000002000
R13: ffff8fa5350aaee8 R14: 0000000000004000 R15: ffff8fa5350aaee8
FS: 0000000000000000(0000) GS:ffff8fbb1f9c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9c5000ef48 CR3: 0000000bcee2e001 CR4: 00000000001706f0
Call Trace:
<IRQ>
? die+0x32/0x80
? do_trap+0xd8/0x100
? do_error_trap+0x65/0x80
? netfs_consume_read_data.isra.0+0xa72/0xab0
? exc_invalid_op+0x4c/0x60
? netfs_consume_read_data.isra.0+0xa72/0xab0
? asm_exc_invalid_op+0x16/0x20
? netfs_consume_read_data.isra.0+0xa72/0xab0
? __pfx_cachefiles_read_complete+0x10/0x10
netfs_read_subreq_terminated+0x22d/0x370
cachefiles_read_complete+0x48/0xf0
iomap_dio_bio_end_io+0x125/0x160
blk_update_request+0xea/0x3e0
scsi_end_request+0x27/0x190
scsi_io_completion+0x43/0x6c0
blk_complete_reqs+0x40/0x50
handle_softirqs+0xd1/0x280
irq_exit_rcu+0x91/0xb0
common_interrupt+0x79/0xa0
</IRQ>
<TASK>
asm_common_interrupt+0x22/0x40
RIP: 0010:cpuidle_enter_state+0xba/0x3b0
Code: 00 e8 ea 86 1c ff e8 45 f7 ff ff 8b 53 04 49 89 c5 0f 1f 44 00
00 31 ff e8 73 b9 1b ff 45 84 ff 0f 85 f8 01 00 00 fb 45 85 f6 <0f> 88
46 01 00 00 48 8b 04 24 49 63 ce 48 6b d1 68 49 29 c5 48 89
RSP: 0018:ffffb434c018be98 EFLAGS: 00000202
RAX: ffff8fbb1f9c0000 RBX: ffffd41cbe7e3448 RCX: 000000000000001f
RDX: 0000000000000007 RSI: 000000003149acb2 RDI: 0000000000000000
RBP: 0000000000000004 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000004 R11: 000000000000001f R12: ffffffffaf660060
R13: 0000030f6179fa73 R14: 0000000000000004 R15: 0000000000000000
? cpuidle_enter_state+0xad/0x3b0
cpuidle_enter+0x29/0x40
do_idle+0x19c/0x200
cpu_startup_entry+0x25/0x30
start_secondary+0xf3/0x100
common_startup_64+0x13e/0x148
</TASK>
Modules linked in:
---[ end trace 0000000000000000 ]---
This is a server with heavy NFS traffic (with fscache enabled).
Please help - and let me know if you need more information.
Max
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: "netfs: Can't donate prior to front"
2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann
@ 2025-02-10 14:07 ` David Howells
2025-02-10 17:41 ` Max Kellermann
2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann
1 sibling, 1 reply; 12+ messages in thread
From: David Howells @ 2025-02-10 14:07 UTC (permalink / raw)
To: Max Kellermann; +Cc: dhowells, netfs, LKML, linux-nfs
Can you apply the attached patch to your kernel, and then run with:
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_read/enable
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_rreq/enable
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_sreq/enable
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_failure/enable
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_donate/enable
echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_progress/enable
enabled. If you can capture the trace output (and compress it!), that should
hopefully help debug this.
David
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index e8624f5c7fcc..8b78c1ec0677 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -312,6 +312,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was
printk("folio: %llx-%llx\n", fpos, fend - 1);
printk("donated: prev=%zx next=%zx\n", prev_donated, next_donated);
printk("s=%llx av=%zx part=%zx\n", start, avail, part);
+ tracing_off();
BUG();
}
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: "netfs: Can't donate prior to front"
2025-02-10 14:07 ` David Howells
@ 2025-02-10 17:41 ` Max Kellermann
2025-02-10 17:45 ` Max Kellermann
0 siblings, 1 reply; 12+ messages in thread
From: Max Kellermann @ 2025-02-10 17:41 UTC (permalink / raw)
To: David Howells; +Cc: netfs, LKML, linux-nfs
[-- Attachment #1: Type: text/plain, Size: 1751 bytes --]
On Mon, Feb 10, 2025 at 3:08 PM David Howells <dhowells@redhat.com> wrote:
> Can you apply the attached patch to your kernel, and then run with:
>
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_read/enable
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_rreq/enable
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_sreq/enable
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_failure/enable
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_donate/enable
> echo 1 >/sys/kernel/debug/tracing/events/netfs/netfs_progress/enable
>
> enabled. If you can capture the trace output (and compress it!), that should
> hopefully help debug this.
I needed some more tricks to be able to capture the trace of a
crashing kernel, but here it is (a minimal trace because I now figured
out how to reproduce the bug with a single command on a disabled
machine).
Additionally, here you have a file with a dump of *subreq and *req (of
R=8). I have added additional fields "start0" and "len0" which contain
the values initially set on these.
Kernel output just before the crash:
3,1644,103679537,-;netfs: Can't donate prior to front
4,1645,103679550,-;R=00000007[6] s=187000-1bffff 0/39000/39000
4,1646,103679562,-;folio: 180000-1bffff
4,1647,103679568,-;donated: prev=0 next=0
4,1648,103679574,-;s=187000 av=39000 part=39000
3,1649,103708094,-;netfs: Can't donate prior to front
4,1650,103708105,-;R=00000008[7] s=1c6000-1fffff 0/3a000/3a000
4,1651,103708117,-;folio: 1c0000-1fffff
4,1652,103708123,-;donated: prev=0 next=0
4,1653,103708129,-;s=1c6000 av=3a000 part=3a000
The trace file contains just the R=8 request, but I guess this one is
enough for you.
Max
[-- Attachment #2: trace8.txt --]
[-- Type: text/plain, Size: 6737 bytes --]
kworker/u193:5-3716 [003] ..... 103.651806: netfs_sreq: R=00000008[1] WRIT PREP f=00 s=c0000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.651808: netfs_sreq: R=00000008[1] WRIT SUBMT f=100 s=c0000 0/40000 e=0
kworker/3:2-4269 [003] ..... 103.652314: netfs_sreq: R=00000008[1] WRIT TERM f=100 s=c0000 40000/40000 e=0
kworker/u193:5-3716 [003] ..... 103.652343: netfs_rreq: R=00000008 2C COLLECT f=2020
kworker/u193:5-3716 [003] ..... 103.652349: netfs_sreq: R=00000008[1] WRIT FREE f=00 s=c0000 40000/40000 e=0
kworker/u193:5-3716 [003] ..... 103.652357: netfs_rreq: R=00000008 2C WR-DONE f=2020
kworker/u193:5-3716 [003] ..... 103.652360: netfs_rreq: R=00000008 2C WAKE-IP f=2020
kworker/u193:5-3716 [003] ..... 103.652362: netfs_rreq: R=00000008 2C FREE f=2000
cp-4373 [038] ..... 103.724132: netfs_read: R=00000008 READAHEAD c=00000002 ni=76464c3 s=1c0000 l=40000 sz=776749
cp-4373 [038] b.... 103.724170: netfs_sreq: R=00000008[1] ---- ADD f=00 s=1c0000 0/40000 e=0
cp-4373 [038] b.... 103.724470: netfs_sreq: R=00000008[2] ---- ADD f=00 s=1d0000 0/30000 e=0
cp-4373 [038] b.... 103.724753: netfs_sreq: R=00000008[3] ---- ADD f=00 s=1d4000 0/2c000 e=0
cp-4373 [038] ..... 103.724767: netfs_sreq: R=00000008[3] READ SUBMT f=00 s=1d4000 0/3000 e=0
cp-4373 [038] b.... 103.724845: netfs_sreq: R=00000008[4] ---- ADD f=00 s=1d7000 0/29000 e=0
cp-4373 [038] b.... 103.725119: netfs_sreq: R=00000008[5] ---- ADD f=00 s=1e7000 0/19000 e=0
cp-4373 [038] b.... 103.725389: netfs_sreq: R=00000008[6] ---- ADD f=00 s=1f1000 0/f000 e=0
cp-4373 [038] ..... 103.725403: netfs_sreq: R=00000008[6] READ SUBMT f=00 s=1f1000 0/3000 e=0
cp-4373 [038] b.... 103.725478: netfs_sreq: R=00000008[7] ---- ADD f=00 s=1f4000 0/c000 e=0
cp-4373 [038] b.... 103.725751: netfs_sreq: R=00000008[8] ---- ADD f=00 s=1fa000 0/6000 e=0
cp-4373 [038] ..... 103.725767: netfs_sreq: R=00000008[8] READ SUBMT f=00 s=1fa000 0/6000 e=0
<idle>-0 [038] ..s.. 103.729472: netfs_progress: R=00000008[06] s=1f1000 ct=0/3000 pa=3000/3000 sl=0
<idle>-0 [038] b.s.. 103.729482: netfs_donate: R=00000008[06] -> [07] to-next am=3000
<idle>-0 [038] b.s.. 103.729483: netfs_sreq: R=00000008[6] READ DON-N f=00 s=1f1000 3000/3000 e=0
<idle>-0 [038] ..s.. 103.729486: netfs_sreq: R=00000008[6] READ TERM f=10 s=1f1000 0/0 e=0
<idle>-0 [038] ..s.. 103.729487: netfs_sreq: R=00000008[6] READ FREE f=10 s=1f1000 0/0 e=0
<idle>-0 [038] ..s.. 103.735784: netfs_progress: R=00000008[03] s=1d4000 ct=0/3000 pa=3000/3000 sl=0
<idle>-0 [038] b.s.. 103.735792: netfs_donate: R=00000008[03] -> [04] to-next am=3000
<idle>-0 [038] b.s.. 103.735793: netfs_sreq: R=00000008[3] READ DON-N f=00 s=1d4000 3000/3000 e=0
<idle>-0 [038] ..s.. 103.735796: netfs_sreq: R=00000008[3] READ TERM f=10 s=1d4000 0/0 e=0
<idle>-0 [038] ..s.. 103.735798: netfs_sreq: R=00000008[3] READ FREE f=10 s=1d4000 0/0 e=0
<idle>-0 [038] ..s.. 103.736603: netfs_progress: R=00000008[08] s=1fa000 ct=0/6000 pa=6000/6000 sl=0
<idle>-0 [038] b.s.. 103.736612: netfs_donate: R=00000008[08] -> [07] tail-to-prev am=6000
<idle>-0 [038] b.s.. 103.736613: netfs_sreq: R=00000008[8] READ DON-P f=00 s=200000 0/0 e=0
<idle>-0 [038] ..s.. 103.736616: netfs_sreq: R=00000008[8] READ TERM f=10 s=200000 0/0 e=0
<idle>-0 [038] ..s.. 103.736618: netfs_sreq: R=00000008[8] READ FREE f=10 s=200000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.780260: netfs_progress: R=00000008[01] s=1c0000 ct=0/10000 pa=10000/10000 sl=0
kworker/u193:5-3716 [003] b.... 103.780269: netfs_donate: R=00000008[01] -> [02] to-next am=10000
kworker/u193:5-3716 [003] b.... 103.780271: netfs_sreq: R=00000008[1] DOWN DON-N f=01 s=1c0000 10000/10000 e=0
kworker/u193:5-3716 [003] ..... 103.780272: netfs_sreq: R=00000008[1] DOWN TERM f=11 s=1c0000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.780273: netfs_sreq: R=00000008[1] DOWN FREE f=11 s=1c0000 0/0 e=0
kworker/u193:5-3716 [003] b.... 103.781527: netfs_sreq: R=00000008[2] DOWN +DON f=01 s=1c0000 14000/14000 e=0
kworker/u193:5-3716 [003] ..... 103.781536: netfs_progress: R=00000008[02] s=1c0000 ct=0/14000 pa=14000/14000 sl=0
kworker/u193:5-3716 [003] b.... 103.781538: netfs_donate: R=00000008[02] -> [04] to-next am=14000
kworker/u193:5-3716 [003] b.... 103.781539: netfs_sreq: R=00000008[2] DOWN DON-N f=01 s=1c0000 14000/14000 e=0
kworker/u193:5-3716 [003] ..... 103.781540: netfs_sreq: R=00000008[2] DOWN TERM f=11 s=1c0000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.781541: netfs_sreq: R=00000008[2] DOWN FREE f=11 s=1c0000 0/0 e=0
kworker/u193:5-3716 [003] b.... 103.783277: netfs_sreq: R=00000008[4] DOWN +DON f=01 s=1c3000 24000/24000 e=0
kworker/u193:5-3716 [003] ..... 103.783286: netfs_progress: R=00000008[04] s=1c3000 ct=0/24000 pa=24000/24000 sl=0
kworker/u193:5-3716 [003] b.... 103.783288: netfs_donate: R=00000008[04] -> [05] to-next am=24000
kworker/u193:5-3716 [003] b.... 103.783289: netfs_sreq: R=00000008[4] DOWN DON-N f=01 s=1c3000 24000/24000 e=0
kworker/u193:5-3716 [003] ..... 103.783290: netfs_sreq: R=00000008[4] DOWN TERM f=11 s=1c3000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.783291: netfs_sreq: R=00000008[4] DOWN FREE f=11 s=1c3000 0/0 e=0
kworker/u193:5-3716 [003] b.... 103.784366: netfs_sreq: R=00000008[5] DOWN +DON f=01 s=1c3000 2e000/2e000 e=0
kworker/u193:5-3716 [003] ..... 103.784372: netfs_progress: R=00000008[05] s=1c3000 ct=0/2e000 pa=2e000/2e000 sl=0
kworker/u193:5-3716 [003] b.... 103.784374: netfs_donate: R=00000008[05] -> [07] to-next am=2e000
kworker/u193:5-3716 [003] b.... 103.784375: netfs_sreq: R=00000008[5] DOWN DON-N f=01 s=1c3000 2e000/2e000 e=0
kworker/u193:5-3716 [003] ..... 103.784376: netfs_sreq: R=00000008[5] DOWN TERM f=11 s=1c3000 0/0 e=0
kworker/u193:5-3716 [003] ..... 103.784377: netfs_sreq: R=00000008[5] DOWN FREE f=11 s=1c3000 0/0 e=0
kworker/u193:5-3716 [003] b.... 103.785425: netfs_sreq: R=00000008[7] DOWN +DON f=01 s=1c6000 3a000/3a000 e=0
kworker/u193:5-3716 [003] ..... 103.785432: netfs_progress: R=00000008[07] s=1c6000 ct=0/3a000 pa=3a000/3a000 sl=0
[-- Attachment #3: gdb.txt --]
[-- Type: text/plain, Size: 5412 bytes --]
(gdb) p/x *rreq
$3 = {
{
work = {
data = {
counter = 0xfffffffe00000
},
entry = {
next = 0xffff8898b7df6908,
prev = 0xffff8898b7df6908
},
func = 0xffffffff81b44e90
},
rcu = {
next = 0xfffffffe00000,
func = 0xffff8898b7df6908
}
},
inode = 0xffff889886939518,
mapping = 0xffff889886939670,
iocb = 0x0,
cache_resources = {
ops = 0xffffffff845fd480,
cache_priv = 0xffff8898b876ed20,
cache_priv2 = 0xffff8898a7b29700,
debug_id = 0x2,
inval_counter = 0x0
},
ractl = 0xffff8898b88b7980,
proc_link = {
next = 0xffffffff857668a0,
prev = 0xffff8898b7df3c60
},
subrequests = {
next = 0xffff8898b87fd568,
prev = 0xffff8898b87fd568
},
io_streams = {{
construct = 0x0,
sreq_max_len = 0x10000,
sreq_max_segs = 0x0,
submit_off = 0x0,
submit_len = 0x0,
submit_extendable_to = 0x0,
prepare_write = 0x0,
issue_write = 0x0,
subrequests = {
next = 0xffff8898b7df69b0,
prev = 0xffff8898b7df69b0
},
front = 0x0,
collected_to = 0x0,
transferred = 0x0,
source = 0x0,
error = 0x0,
stream_nr = 0x0,
avail = 0x0,
active = 0x0,
need_retry = 0x0,
failed = 0x0
}, {
construct = 0x0,
sreq_max_len = 0x0,
sreq_max_segs = 0x0,
submit_off = 0x0,
submit_len = 0x0,
submit_extendable_to = 0x0,
prepare_write = 0x0,
issue_write = 0x0,
subrequests = {
next = 0xffff8898b7df6a18,
prev = 0xffff8898b7df6a18
},
front = 0x0,
collected_to = 0x0,
transferred = 0x0,
source = 0x0,
error = 0x0,
stream_nr = 0x0,
avail = 0x0,
active = 0x0,
need_retry = 0x0,
failed = 0x0
}},
group = 0x0,
buffer = 0xffff8898b8000000,
buffer_tail = 0xffff8898b8000000,
iter = {
iter_type = 0x4,
nofault = 0x0,
data_source = 0x0,
iov_offset = 0x0,
{
__ubuf_iovec = {
iov_base = 0xffff8898b8000000,
iov_len = 0x0
},
{
{
__iov = 0xffff8898b8000000,
kvec = 0xffff8898b8000000,
bvec = 0xffff8898b8000000,
folioq = 0xffff8898b8000000,
xarray = 0xffff8898b8000000,
ubuf = 0xffff8898b8000000
},
count = 0x0
}
},
{
nr_segs = 0x1,
folioq_slot = 0x1,
xarray_start = 0x1
}
},
io_iter = {
iter_type = 0x0,
nofault = 0x0,
data_source = 0x0,
iov_offset = 0x0,
{
__ubuf_iovec = {
iov_base = 0x0,
iov_len = 0x0
},
{
{
__iov = 0x0,
kvec = 0x0,
bvec = 0x0,
folioq = 0x0,
xarray = 0x0,
ubuf = 0x0
},
count = 0x0
}
},
{
nr_segs = 0x0,
folioq_slot = 0x0,
xarray_start = 0x0
}
},
netfs_priv = 0xffff8898b041fa00,
netfs_priv2 = 0x0,
direct_bv = 0x0,
direct_bv_count = 0x0,
debug_id = 0x8,
rsize = 0x0,
wsize = 0x7fffffff,
subreq_counter = {
counter = 0x8
},
nr_group_rel = 0x0,
lock = {
{
rlock = {
raw_lock = {
{
val = {
counter = 0x0
},
{
locked = 0x0,
pending = 0x0
},
{
locked_pending = 0x0,
tail = 0x0
}
}
}
}
}
},
nr_outstanding = {
counter = 0x1
},
submitted = 0x200000,
len0 = 0x40000,
len = 0x40000,
transferred = 0x0,
error = 0x0,
origin = 0x0,
direct_bv_unpin = 0x0,
buffer_head_slot = 0x0,
buffer_tail_slot = 0x0,
i_size = 0x776749,
start0 = 0x1c0000,
start = 0x1c0000,
issued_to = {
counter = 0x0
},
collected_to = 0x0,
cleaned_to = 0x0,
no_unlock_folio = 0x0,
prev_donated = 0x0,
ref = {
refs = {
counter = 0x1
}
},
flags = 0x80000020,
netfs_ops = 0xffffffff8459dba0,
cleanup = 0x0
}
(gdb) p/x *subreq
$4 = {
rreq = 0xffff8898b7df6900,
work = {
data = {
counter = 0xfffffffe00000
},
entry = {
next = 0xffff8898b87fd550,
prev = 0xffff8898b87fd550
},
func = 0x0
},
rreq_link = {
next = 0xffff8898b7df6970,
prev = 0xffff8898b7df6970
},
io_iter = {
iter_type = 0x4,
nofault = 0x0,
data_source = 0x0,
iov_offset = 0x34000,
{
__ubuf_iovec = {
iov_base = 0xffff8898b8000000,
iov_len = 0x6000
},
{
{
__iov = 0xffff8898b8000000,
kvec = 0xffff8898b8000000,
bvec = 0xffff8898b8000000,
folioq = 0xffff8898b8000000,
xarray = 0xffff8898b8000000,
ubuf = 0xffff8898b8000000
},
count = 0x6000
}
},
{
nr_segs = 0x0,
folioq_slot = 0x0,
xarray_start = 0x0
}
},
start0 = 0x1f4000,
start = 0x1c6000,
len0 = 0xc000,
len = 0x3a000,
transferred = 0x3a000,
consumed = 0x0,
prev_donated = 0x0,
next_donated = 0x0,
ref = {
refs = {
counter = 0x2
}
},
error = 0x0,
debug_index = 0x7,
nr_segs = 0x0,
retry_count = 0x0,
source = 0x2,
stream_nr = 0x0,
curr_folioq_slot = 0x0,
curr_folio_order = 0x6,
curr_folioq = 0xffff8898b8000000,
flags = 0x1
}
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: "netfs: Can't donate prior to front"
2025-02-10 17:41 ` Max Kellermann
@ 2025-02-10 17:45 ` Max Kellermann
0 siblings, 0 replies; 12+ messages in thread
From: Max Kellermann @ 2025-02-10 17:45 UTC (permalink / raw)
To: David Howells; +Cc: netfs, LKML, linux-nfs
On Mon, Feb 10, 2025 at 6:41 PM Max Kellermann <max.kellermann@ionos.com> wrote:
> Additionally, here you have a file with a dump of *subreq and *req (of
> R=8). I have added additional fields "start0" and "len0" which contain
> the values initially set on these.
btw. this happens because there is only one subrequest (R=8[7]), but
the start offset is being rounded down (folio_order=2), thus
"fpos<start":
(gdb) p/x fpos
$5 = 0x1c0000
(gdb) p/x fend
$6 = 0x200000
(gdb) p/x start
$7 = 0x1c6000
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann
2025-02-10 14:07 ` David Howells
@ 2025-02-10 19:11 ` Max Kellermann
2025-02-14 12:47 ` David Howells
1 sibling, 1 reply; 12+ messages in thread
From: Max Kellermann @ 2025-02-10 19:11 UTC (permalink / raw)
To: dhowells, netfs, linux-kernel, linux-nfs; +Cc: Max Kellermann
If multiple subrequests donate data to the same "next" request
(depending on the subrequest completion order), each of them would
overwrite the `prev_donated` field, causing data corruption and a
BUG() crash ("Can't donate prior to front").
Fixes: ee4cdf7ba857 ("netfs: Speed up buffered reading")
Closes: https://lore.kernel.org/netfs/CAKPOu+_4mUwYgQtRTbXCmi+-k3PGvLysnPadkmHOyB7Gz0iSMA@mail.gmail.com/
Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
---
David, this seems to fix the bug for me. Please also check if we need
a "donation_changed" check.
---
fs/netfs/read_collect.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fs/netfs/read_collect.c b/fs/netfs/read_collect.c
index 0d95cdbe5611..681b630b4f06 100644
--- a/fs/netfs/read_collect.c
+++ b/fs/netfs/read_collect.c
@@ -284,7 +284,7 @@ static bool netfs_consume_read_data(struct netfs_io_subrequest *subreq, bool was
netfs_trace_donate_to_deferred_next);
} else {
next = list_next_entry(subreq, rreq_link);
- WRITE_ONCE(next->prev_donated, excess);
+ WRITE_ONCE(next->prev_donated, next->prev_donated + excess);
trace_netfs_donate(rreq, subreq, next, excess,
netfs_trace_donate_to_next);
}
--
2.47.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann
@ 2025-02-14 12:47 ` David Howells
2025-02-20 13:09 ` Max Kellermann
0 siblings, 1 reply; 12+ messages in thread
From: David Howells @ 2025-02-14 12:47 UTC (permalink / raw)
To: Max Kellermann; +Cc: dhowells, netfs, linux-kernel, linux-nfs
Max Kellermann <max.kellermann@ionos.com> wrote:
> David, this seems to fix the bug for me. Please also check if we need
> a "donation_changed" check.
Shouldn't do since at that point we've been holding rreq->lock since the last
check.
> If multiple subrequests donate data to the same "next" request
> (depending on the subrequest completion order), each of them would
> overwrite the `prev_donated` field, causing data corruption and a
> BUG() crash ("Can't donate prior to front").
>
> Fixes: ee4cdf7ba857 ("netfs: Speed up buffered reading")
> Closes: https://lore.kernel.org/netfs/CAKPOu+_4mUwYgQtRTbXCmi+-k3PGvLysnPadkmHOyB7Gz0iSMA@mail.gmail.com/
> Signed-off-by: Max Kellermann <max.kellermann@ionos.com>
Signed-off-by: David Howells <dhowells@redhat.com>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-14 12:47 ` David Howells
@ 2025-02-20 13:09 ` Max Kellermann
2025-02-20 14:17 ` Greg Kroah-Hartman
0 siblings, 1 reply; 12+ messages in thread
From: Max Kellermann @ 2025-02-20 13:09 UTC (permalink / raw)
To: David Howells, Greg Kroah-Hartman; +Cc: netfs, linux-kernel, linux-nfs
On Fri, Feb 14, 2025 at 1:47 PM David Howells <dhowells@redhat.com> wrote:
> Signed-off-by: David Howells <dhowells@redhat.com>
Greg, you merged my other two netfs fixes into 6.13.3, but omitted
this one. Did you miss it, or was there another reason?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-20 13:09 ` Max Kellermann
@ 2025-02-20 14:17 ` Greg Kroah-Hartman
2025-02-20 15:00 ` Max Kellermann
0 siblings, 1 reply; 12+ messages in thread
From: Greg Kroah-Hartman @ 2025-02-20 14:17 UTC (permalink / raw)
To: Max Kellermann; +Cc: David Howells, netfs, linux-kernel, linux-nfs
On Thu, Feb 20, 2025 at 02:09:17PM +0100, Max Kellermann wrote:
> On Fri, Feb 14, 2025 at 1:47 PM David Howells <dhowells@redhat.com> wrote:
> > Signed-off-by: David Howells <dhowells@redhat.com>
>
> Greg, you merged my other two netfs fixes into 6.13.3, but omitted
> this one. Did you miss it, or was there another reason?
It wasn't sent to me or to the stable list, so how could have I seen it?
confused,
greg k-h
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-20 14:17 ` Greg Kroah-Hartman
@ 2025-02-20 15:00 ` Max Kellermann
2025-02-20 15:10 ` Greg Kroah-Hartman
2025-03-01 14:17 ` Salvatore Bonaccorso
0 siblings, 2 replies; 12+ messages in thread
From: Max Kellermann @ 2025-02-20 15:00 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: David Howells, netfs, linux-kernel, linux-nfs
On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman
<gregkh@linuxfoundation.org> wrote:
> It wasn't sent to me or to the stable list, so how could have I seen it?
Oh, of course, I forgot to add stable. How shall we proceed? Do you
want me to resend to you with David's Signed-off-by?
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-20 15:00 ` Max Kellermann
@ 2025-02-20 15:10 ` Greg Kroah-Hartman
2025-03-01 14:17 ` Salvatore Bonaccorso
1 sibling, 0 replies; 12+ messages in thread
From: Greg Kroah-Hartman @ 2025-02-20 15:10 UTC (permalink / raw)
To: Max Kellermann; +Cc: David Howells, netfs, linux-kernel, linux-nfs
On Thu, Feb 20, 2025 at 04:00:49PM +0100, Max Kellermann wrote:
> On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > It wasn't sent to me or to the stable list, so how could have I seen it?
>
> Oh, of course, I forgot to add stable. How shall we proceed? Do you
> want me to resend to you with David's Signed-off-by?
Please do, and cc: stable.
thanks,
greg k-h
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-02-20 15:00 ` Max Kellermann
2025-02-20 15:10 ` Greg Kroah-Hartman
@ 2025-03-01 14:17 ` Salvatore Bonaccorso
2025-03-01 16:51 ` Max Kellermann
1 sibling, 1 reply; 12+ messages in thread
From: Salvatore Bonaccorso @ 2025-03-01 14:17 UTC (permalink / raw)
To: Max Kellermann
Cc: Greg Kroah-Hartman, David Howells, netfs, linux-kernel, linux-nfs
Hi
(btw side question from a bystander looking a the bug)
On Thu, Feb 20, 2025 at 04:00:49PM +0100, Max Kellermann wrote:
> On Thu, Feb 20, 2025 at 3:17 PM Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> > It wasn't sent to me or to the stable list, so how could have I seen it?
>
> Oh, of course, I forgot to add stable. How shall we proceed? Do you
> want me to resend to you with David's Signed-off-by?
Do I see it correctly that this will be 6.12.y and 6.13.y specific
backports since the code in mainline changed substantially with
e2d46f2ec332 ("netfs: Change the read result collector to only use one
work item") and so your change does not apply there anymore?
I'm asking since we got a bug report in Debian which seems to idicate
it might have the same root cause as you report, and it is at
https://bugs.debian.org/1098698 .
Regards,
Salvatore
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH] fs/netfs/read_collect: add to next->prev_donated
2025-03-01 14:17 ` Salvatore Bonaccorso
@ 2025-03-01 16:51 ` Max Kellermann
0 siblings, 0 replies; 12+ messages in thread
From: Max Kellermann @ 2025-03-01 16:51 UTC (permalink / raw)
To: Salvatore Bonaccorso
Cc: Greg Kroah-Hartman, David Howells, netfs, linux-kernel, linux-nfs
On Sat, Mar 1, 2025 at 3:17 PM Salvatore Bonaccorso <carnil@debian.org> wrote:
> Do I see it correctly that this will be 6.12.y and 6.13.y specific
> backports since the code in mainline changed substantially with
> e2d46f2ec332 ("netfs: Change the read result collector to only use one
> work item") and so your change does not apply there anymore?
Correct.
> I'm asking since we got a bug report in Debian which seems to idicate
> it might have the same root cause as you report, and it is at
> https://bugs.debian.org/1098698 .
This indeed looks similar.
Note that this is one of four netfs crash fixes I posted recently.
Two other fixes have been released with 6.13.4 already:
https://lore.kernel.org/netfs/20250211093432.3524035-1-max.kellermann@ionos.com/
https://lore.kernel.org/netfs/20250210223144.3481766-1-max.kellermann@ionos.com/
The fourth bug (that got no reviews so far) is here (this one affects
master as well):
https://lore.kernel.org/netfs/20250228123602.2140459-1-max.kellermann@ionos.com/
Max
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2025-03-01 16:51 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-07 18:40 "netfs: Can't donate prior to front" Max Kellermann
2025-02-10 14:07 ` David Howells
2025-02-10 17:41 ` Max Kellermann
2025-02-10 17:45 ` Max Kellermann
2025-02-10 19:11 ` [PATCH] fs/netfs/read_collect: add to next->prev_donated Max Kellermann
2025-02-14 12:47 ` David Howells
2025-02-20 13:09 ` Max Kellermann
2025-02-20 14:17 ` Greg Kroah-Hartman
2025-02-20 15:00 ` Max Kellermann
2025-02-20 15:10 ` Greg Kroah-Hartman
2025-03-01 14:17 ` Salvatore Bonaccorso
2025-03-01 16:51 ` Max Kellermann
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox