Re: [REGRESSION] 9pfs issues on 6.12-rc1

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[David Howells]

Hi Antony,

I think the attached should fix it properly rather than working around it as
the previous patch did.  If you could give it a whirl?

Thanks,
David
---
commit 68dddbfdf45e8f176cc8556a3db69af24dfb8519
Author: David Howells <dhowells@redhat.com>
Date:   Wed Oct 23 10:24:12 2024 +0100

    iov_iter: Fix iov_iter_get_pages*() for folio_queue
    
    p9_get_mapped_pages() uses iov_iter_get_pages_alloc2() to extract pages
    from an iterator when performing a zero-copy request and under some
    circumstances, this crashes with odd page errors[1], for example, I see:
    
        page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0xbcf0
        flags: 0x2000000000000000(zone=1)
        ...
        page dumped because: VM_BUG_ON_FOLIO(((unsigned int) folio_ref_count(folio) + 127u <= 127u))
        ------------[ cut here ]------------
        kernel BUG at include/linux/mm.h:1444!
    
    This is because, unlike in iov_iter_extract_folioq_pages(), the
    iter_folioq_get_pages() helper function doesn't skip the current folio when
    iov_offset points to the end of it, but rather extracts the next page
    beyond the end of the folio and adds it to the list.  Reading will then
    clobber the contents of this page, leading to system corruption, and if the
    page is not in use, put_page() may try to clean up the unused page.
    
    This can be worked around by copying the iterator before each extraction[2]
    and using iov_iter_advance() on the original as the advance function steps
    over the page we're at the end of.
    
    Fix this by skipping the page extraction if we're at the end of the folio.
    
    This was reproduced in the ktest environment[3] by forcing 9p to use the
    fscache caching mode and then reading a file through 9p.
    
    Fixes: db0aa2e9566f ("mm: Define struct folio_queue and ITER_FOLIOQ to handle a sequence of folios")
    Reported-by: Antony Antony <antony@phenome.org>
    Closes: https://lore.kernel.org/r/ZxFQw4OI9rrc7UYc@Antony2201.local/
    Signed-off-by: David Howells <dhowells@redhat.com>
    cc: Eric Van Hensbergen <ericvh@kernel.org>
    cc: Latchesar Ionkov <lucho@ionkov.net>
    cc: Dominique Martinet <asmadeus@codewreck.org>
    cc: Christian Schoenebeck <linux_oss@crudebyte.com>
    cc: v9fs@lists.linux.dev
    cc: netfs@lists.linux.dev
    cc: linux-fsdevel@vger.kernel.org
    Link: https://lore.kernel.org/r/ZxFEi1Tod43pD6JC@moon.secunet.de/ [1]
    Link: https://lore.kernel.org/r/2299159.1729543103@warthog.procyon.org.uk/ [2]
    Link: https://github.com/koverstreet/ktest.git [3]

diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 1abb32c0da50..cc4b5541eef8 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -1021,15 +1021,18 @@ static ssize_t iter_folioq_get_pages(struct iov_iter *iter,
 		size_t offset = iov_offset, fsize = folioq_folio_size(folioq, slot);
 		size_t part = PAGE_SIZE - offset % PAGE_SIZE;
 
-		part = umin(part, umin(maxsize - extracted, fsize - offset));
-		count -= part;
-		iov_offset += part;
-		extracted += part;
-
-		*pages = folio_page(folio, offset / PAGE_SIZE);
-		get_page(*pages);
-		pages++;
-		maxpages--;
+		if (offset < fsize) {
+			part = umin(part, umin(maxsize - extracted, fsize - offset));
+			count -= part;
+			iov_offset += part;
+			extracted += part;
+
+			*pages = folio_page(folio, offset / PAGE_SIZE);
+			get_page(*pages);
+			pages++;
+			maxpages--;
+		}
+
 		if (maxpages == 0 || extracted >= maxsize)
 			break;
 

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Antony Antony]

On Wed, Oct 23, 2024 at 11:07:05 +0100, David Howells wrote:
> Hi Antony,
> 
> I think the attached should fix it properly rather than working around it as
> the previous patch did.  If you could give it a whirl?

Yes this also fix the crash.

Tested-by: Antony Antony <antony.antony@secunet.com>

thanks,
-antony

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Ryan Lahfa]

Hi everyone,

Le Wed, Oct 23, 2024 at 09:38:39PM +0200, Antony Antony a écrit :
> On Wed, Oct 23, 2024 at 11:07:05 +0100, David Howells wrote:
> > Hi Antony,
> > 
> > I think the attached should fix it properly rather than working around it as
> > the previous patch did.  If you could give it a whirl?
> 
> Yes this also fix the crash.
> 
> Tested-by: Antony Antony <antony.antony@secunet.com>

I cannot confirm this fixes the crash for me. My reproducer is slightly
more complicated than Max's original one, albeit, still on NixOS and
probably uses 9p more intensively than the automated NixOS testings
workload.

Here is how to reproduce it:

$ git clone https://gerrit.lix.systems/lix
$ cd lix
$ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD
$ nix-build -A hydraJobs.tests.local-releng

I suspect the reason for why Antony considers the crash to be fixed is
that the workload used to test it requires a significant amount of
chance and retries to trigger the bug.

On my end, you can see our CI showing the symptoms:
https://buildkite.com/organizations/lix-project/pipelines/lix/builds/2357/jobs/019761e7-784e-4790-8c1b-f609270d9d19/log.

We retried probably hundreds of times and saw different corruption
patterns, Python getting confused, ld.so getting confused, systemd
sometimes too. Python had a much higher chance of crashing in many of
our tests. We reproduced it over aarch64-linux (Ampere Altra Q80-30) but
also Intel and AMD CPUs (~5 different systems).

As soon as we reverted to Linux 6.6 series, the bug went away.

We bisected but we started to have weirder problems, this is because we
encountered the original regression mentioned in October 2024 and for a
certain range of commits, we were unable to bisect anything further.

So I switched my bisection strategy to understand when the bug was
fixed, this lead me on the commit
e65a0dc1cabe71b91ef5603e5814359451b74ca7 which is the proper fix
mentioned here and on this discussion.

Reverting this on the top of 6.12 cause indeed a massive amount of
traces, see this gist [1] for examples.

Applying the "workaround patch" aka "[PATCH] 9p: Don't revert the I/O
iterator after reading" after reverting e65a0dc1cabe makes the problem
go away after 5 tries (5 tries were sufficient to trigger with the
proper fix).

If this can be helpful, the nature of the test above is to copy a
significant amount of assets to an S3 implementation (Garage) running
inside of the VM. Many of these assets comes from the Nix store which
sits over 9p.

Anyhow, I see three patterns:

- Kernel panic when starting the /init, this is the crash Max reported
  back in October 2024 and the one we started to encounter while
  bisecting this problem in the range between v6.11 and v6.12.
- systemd crashing very quickly, 
  this is what we see when reverting e65a0dc1cabe71b91ef5603e5814359451b74ca7
  on the top of v6.12 *OR* when we are around v6.12rc5.
- what the CI above shows which are userspace programs crashing after
  some serious I/O exercising has been done, which happens on the top of
  v6.12, v6.14, v6.15 (incl. stable kernels).

If you need me to test things, please let me know.

[1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd
-- 
Ryan Lahfa

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Christian Theune]

Hi,

we’re experiencing the same issue with a number of NixOS tests that are heavy in operations copying from the v9fs mounted nix store.

> On 13. Jun 2025, at 00:24, Ryan Lahfa <ryan@lahfa.xyz> wrote:
> 
> Hi everyone,
> 
> Le Wed, Oct 23, 2024 at 09:38:39PM +0200, Antony Antony a écrit :
>> On Wed, Oct 23, 2024 at 11:07:05 +0100, David Howells wrote:
>>> Hi Antony,
>>> 
>>> I think the attached should fix it properly rather than working around it as
>>> the previous patch did.  If you could give it a whirl?
>> 
>> Yes this also fix the crash.
>> 
>> Tested-by: Antony Antony <antony.antony@secunet.com>
> 
> I cannot confirm this fixes the crash for me. My reproducer is slightly
> more complicated than Max's original one, albeit, still on NixOS and
> probably uses 9p more intensively than the automated NixOS testings
> workload.
> 
> Here is how to reproduce it:
> 
> $ git clone https://gerrit.lix.systems/lix
> $ cd lix
> $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD
> $ nix-build -A hydraJobs.tests.local-releng
> 
> I suspect the reason for why Antony considers the crash to be fixed is
> that the workload used to test it requires a significant amount of
> chance and retries to trigger the bug.
> 
> On my end, you can see our CI showing the symptoms:
> https://buildkite.com/organizations/lix-project/pipelines/lix/builds/2357/jobs/019761e7-784e-4790-8c1b-f609270d9d19/log.
> 
> We retried probably hundreds of times and saw different corruption
> patterns, Python getting confused, ld.so getting confused, systemd
> sometimes too. Python had a much higher chance of crashing in many of
> our tests. We reproduced it over aarch64-linux (Ampere Altra Q80-30) but
> also Intel and AMD CPUs (~5 different systems).

Yeah. We’re on AMD CPUs and it wasn’t hardware-bound. 

The errors we saw where: 

- malloc(): unaligned tcache chunk detected
- segfaulting java processes
- misbehaving filesystems (errors about internal structures in ext4, incorrect file content in xfs)
- crashing kernels when dealing with the outfall of those errors

> As soon as we reverted to Linux 6.6 series, the bug went away.

Same here, the otherway around: we came from 6.6.94 and updated to 6.12.34 and immediately saw a number of tests failing, all of which were heavy in copying data from v9fs to the root filesystem in the VM.

> We bisected but we started to have weirder problems, this is because we
> encountered the original regression mentioned in October 2024 and for a
> certain range of commits, we were unable to bisect anything further.

I had already found the issue from last October when started bisecting, I later got in touch with Ryan who recognized that we were chasing the same issue. I stopped bisecting at that point - the bisect was already homing in around the time of the changes in last October.

> So I switched my bisection strategy to understand when the bug was
> fixed, this lead me on the commit
> e65a0dc1cabe71b91ef5603e5814359451b74ca7 which is the proper fix
> mentioned here and on this discussion.
> 
> Reverting this on the top of 6.12 cause indeed a massive amount of
> traces, see this gist [1] for examples.

Yeah. During bisect I noticed it flapping around with the original October issues crashing immediately during boot.

> Applying the "workaround patch" aka "[PATCH] 9p: Don't revert the I/O
> iterator after reading" after reverting e65a0dc1cabe makes the problem
> go away after 5 tries (5 tries were sufficient to trigger with the
> proper fix).

Yup, I applied the revert and workaround patch on top of 6.12.34 and the reliably broken test became reliably green again.

Our test can be reproduced, too:

$ git clone https://github.com/flyingcircusio/fc-nixos.git
$ cd fc-nixos
$ eval $(./dev-setup)
$ nix-build tests/matomo.nix

The test will fail with ext4 complaining something like this:

machine # [ 42.596728] vn2haz1283lxz6iy0rai850a7jlgxbja-matomo-setup-update-pre[1233]: Copied files, updating package link in /var/lib/matomo/current-package.
machine # [ 42.788956] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm setfacl: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 42.958590] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm chown: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 43.068003] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm chmod: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0
machine # [ 43.004098] vn2haz1283lxz6iy0rai850a7jlgxbja-matomo-setup-update-pre[1233]: Giving matomo read+write access to /var/lib/matomo/share/matomo.js, /var/lib/matomo/share/piwik.js, /var/lib/matomo/share/config, /var/lib/matomo/share/misc/user, /var/lib/matomo/share/js, /var/lib/matomo/share/tmp, /var/lib/matomo/share/misc
machine # [ 43.201319] EXT4-fs error (device vda): htree_dirblock_to_tree:1109: inode #13138: block 5883: comm setfacl: bad entry in directory: rec_len % 4 != 0 - offset=0, inode=606087968, rec_len=31074, size=4096 fake=0

I’m also available for testing and further diagnosis.

Christian

-- 
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Dominique Martinet]

Hi all,

sorry for the slow reply; I wasn't in Cc of most of the mails back in
October so this is a pain to navigate... Let me recap a bit:
- stuff started failing in 6.12-rc1
- David first posted "9p: Don't revert the I/O iterator after
reading"[1], which fixed the bug, but then found a "better" fix as
"iov_iter: Fix iov_iter_get_pages*() for folio_queue" [2] which was
merged instead (so the first patch was not merged)

But it turns out the second patch is not enough (or causes another
issue?), and the reverting it + applying first one works, is that
correct?
What happens if you keep [2] and just apply [1], does that still bug?

(I've tried reading through the thread now and I don't even see what was
the "bad" patch in the first place, although I assume it's ee4cdf7ba857
("netfs: Speed up buffered reading") -- was that confirmed?)


David, as you worked on this at the time it'd be great if you could have
another look, I have no idea what made you try [1] in the first place
but unless you think 9p is doing something wrong like double-reverting
on error or something like that I'd like to understand a bit more what
happens... Although given 6.12 is getting used more now it could make
sense to just apply [1] first until we understand, and have a proper fix
come second -- if someone can confirm we don't need to revert [2].


[1] https://lore.kernel.org/all/3327438.1729678025@warthog.procyon.org.uk/T/#mc97a248b0f673dff6dc8613b508ca4fd45c4fefe
[2] https://lore.kernel.org/all/3327438.1729678025@warthog.procyon.org.uk/T/#m89597a1144806db4ae89992953031cdffa0b0bf9


Thanks,
-- 
Dominique

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Christian Theune]

Hi,

> On 27. Jun 2025, at 08:44, Dominique Martinet <asmadeus@codewreck.org> wrote:
> 
> Hi all,
> 
> sorry for the slow reply; I wasn't in Cc of most of the mails back in
> October so this is a pain to navigate... Let me recap a bit:
> - stuff started failing in 6.12-rc1

yes, to my knowledge and interpretation of this thread.

> - David first posted "9p: Don't revert the I/O iterator after
> reading"[1], which fixed the bug, but then found a "better" fix as
> "iov_iter: Fix iov_iter_get_pages*() for folio_queue" [2] which was
> merged instead (so the first patch was not merged)
> 
> But it turns out the second patch is not enough (or causes another
> issue?), and the reverting it + applying first one works, is that
> correct?
> What happens if you keep [2] and just apply [1], does that still bug?

I tried that and the test that so far under all the variations reliably
crashed (or not) is not crashing in this case.

> (I've tried reading through the thread now and I don't even see what was
> the "bad" patch in the first place, although I assume it's ee4cdf7ba857
> ("netfs: Speed up buffered reading") -- was that confirmed?)

I was late to the party, to, so I’ll defer to the others.

> David, as you worked on this at the time it'd be great if you could have
> another look, I have no idea what made you try [1] in the first place
> but unless you think 9p is doing something wrong like double-reverting
> on error or something like that I'd like to understand a bit more what
> happens... Although given 6.12 is getting used more now it could make
> sense to just apply [1] first until we understand, and have a proper fix
> come second -- if someone can confirm we don't need to revert [2].

I guess I confirmed this. However, I’m just barely better than a monkey
here so I can’t tell whether this makes sense from the internal logic of
things.

To repeat, for safety: my test case worked with the situation you described and suggested:
[1] applied on top of 6.12.34 and *not* having [2] reverted.

Christian

-- 
Christian Theune · ct@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · https://flyingcircus.io
Leipziger Str. 70/71 · 06108 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[David Howells]

Ryan Lahfa <ryan@lahfa.xyz> wrote:

> Here is how to reproduce it:
> 
> $ git clone https://gerrit.lix.systems/lix
> $ cd lix
> $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD
> $ nix-build -A hydraJobs.tests.local-releng

How do I build and run this on Fedora is the problem :-/

> [1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd

Looking at this, it looks very much like a page may have been double-freed.

Just to check, what are you using 9p for?  Containers?  And which transport is
being used, the virtio one?

David

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Ryan Lahfa]

Hi David,

Le Fri, Jun 27, 2025 at 11:00:06AM +0100, David Howells a écrit :
> Ryan Lahfa <ryan@lahfa.xyz> wrote:
> 
> > Here is how to reproduce it:
> > 
> > $ git clone https://gerrit.lix.systems/lix
> > $ cd lix
> > $ git fetch https://gerrit.lix.systems/lix refs/changes/29/3329/8 && git checkout FETCH_HEAD
> > $ nix-build -A hydraJobs.tests.local-releng
> 
> How do I build and run this on Fedora is the problem :-/

This may introduce another layer but you could use a Docker container
(Lix has http://ghcr.io/lix-project/lix) and run these instructions
inside that context.

Alternatives are the following:

- static binary for Nix, I can build one for you and make it available.
- the Lix installer, https://lix.systems/install/ (curl | sh but it does
  prompt you for any step and tell you what it does, it should also be
  very easy to uninstall!).
- Debian has Nix packaged: https://packages.debian.org/sid/nix-bin (not
  Lix, but doesn't matter for this reproducer).
- Can install a remote VM for you with Fedora with one of the previous
  option and give you root@ over there.

Let me know how I can help.

> > [1]: https://gist.dgnum.eu/raito/3d1fa61ebaf642218342ffe644fb6efd
> 
> Looking at this, it looks very much like a page may have been double-freed.
> 
> Just to check, what are you using 9p for?  Containers?  And which transport is
> being used, the virtio one?

9p is used in QEMU in this context.
NixOS has a framework of end to end testing à la OpenQA from OpenSUSE
that makes use of 9pfs to mount the host Nix store inside the guest VM
to avoid copying back'n'forth things that are not under test.

Yep, transport is virtio.

Kind regards,
-- 
Ryan Lahfa

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Arnout Engelen]

On Fri, 13 Jun 2025 00:24:13 +0200, Ryan Lahfa wrote:
> Le Wed, Oct 23, 2024 at 09:38:39PM +0200, Antony Antony a écrit :
> > On Wed, Oct 23, 2024 at 11:07:05 +0100, David Howells wrote:
> > > Hi Antony,
> > > 
> > > I think the attached should fix it properly rather than working around it as
> > > the previous patch did.  If you could give it a whirl?
> > 
> > Yes this also fix the crash.
> > 
> > Tested-by: Antony Antony <antony.antony@secunet.com>
> 
> I cannot confirm this fixes the crash for me. My reproducer is slightly
> more complicated than Max's original one, albeit, still on NixOS and
> probably uses 9p more intensively than the automated NixOS testings
> workload.

I'm seeing a problem in the same area - the symptom is slightly different,
but the location seems very similar. I'm also running a NixOS image.
Mounting a 9p filesystem in qemu with `cache=readahead`, reading a
12943-byte file, in the guest I do see a 12943-byte file, but only
the first 12288 bytes are populated: the rest are zero. This also
reproduces (most but not all of the time) on 6.16-rc7, but not on all host
machines I've tried.

After applying a simplified version of [1] (i.e. [2]), the problem does not
reproduce anymore. It seems something in `p9_client_read_once` somehow
leaves the iov_iter in an unhealthy state. It would be good to understand
exactly what, but I haven't been able to figure that out yet.

I have a smallish nix-based reproducer at [3], and a more involved setup
with a lot of logging enabled and a convenient way to attach gdb at [4].
You start the VM and then 'cat /repro/default.json' manually, and see if
it looks 'truncated'.

Interestingly, the file is read in two p9 read calls: one of 12288 bytes and
one of 655 bytes. The first read is a zero-copy one, the second is not
zero-copy (because it is smaller than 1024). I've also tried with a slightly
larger version of the file, that is read as 2 zero-copy reads, and I have not
been able to reproduce the problem with that. From my (admittedly limited)
understanding the non-zerocopy code path looks fine, though.

I hope this is helpful - I'd be happy to keep looking into this further,
but any help pointing me in the right direction would be much appreciated :)


Kind regards,

Arnout

[1] https://lore.kernel.org/all/3327438.1729678025@warthog.procyon.org.uk/T/#mc97a248b0f673dff6dc8613b508ca4fd45c4fefe
[2] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging/kernel-use-copied-iov_iter.patch
[3] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/small-reproducer
[4] https://codeberg.org/raboof/nextcloud-onlyoffice-test-vm/src/branch/reproducer-with-debugging

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[asmadeus]

Arnout Engelen wrote on Sun, Aug 10, 2025 at 07:57:11PM +0200:
> I have a smallish nix-based reproducer at [3], and a more involved setup
> with a lot of logging enabled and a convenient way to attach gdb at [4].
> You start the VM and then 'cat /repro/default.json' manually, and see if
> it looks 'truncated'.

Thank you!!! I was able to reproduce with this!

(well, `nix -L build .#nixosConfigurations.default.config.system.build.vm`
to build the VM as this machine isn't running nixos and doesn't have
nixos-rebuild...)

> Interestingly, the file is read in two p9 read calls: one of 12288 bytes and
> one of 655 bytes. The first read is a zero-copy one, the second is not
> zero-copy (because it is smaller than 1024).

Yes, your msize is set to 16k but with the 9p overhead the largest,
4k-aligned read that can be done is 12k, so that's coherent.
(Changing the msize to 32k so it's read in a single zero-copy read,
obviously makes this particular error go away, but it's a huge hint)

Removing readahead also makes the problem go away, which is also
surprising because from looking at traces it's only calling into
p9_client_read() once (which forks the two p9_client_read_once, one with
zc and the other without), so readahead shouldn't matter at all but it
obviously does...

Also I haven't been able to reproduce it with a kernel I built myself/my
environment, but it reproduces reliably 99% of the times in the nixos
VM, so we're missing a last piece for a "simple" (non-nix) reproducer,
but I think it's good enough for me to dig into this;
I'll try to find time to check in details this afternoon...
Basically "just" have to follow where the data is written and why it
doesn't end up in the iov and fix that, but I'll need to reproduce on a
kernel I built first to be able to validate the fix.


Anyway this is a huge leap forward (hopeful it's the same problem and we
don't have two similar issues lurking here...), we can't thank you
enough.

I'll report back ASAP.
-- 
Dominique Martinet | Asmadeus

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Dominique Martinet]

asmadeus@codewreck.org wrote on Mon, Aug 11, 2025 at 09:57:51AM +0900:
> Also I haven't been able to reproduce it with a kernel I built myself/my
> environment, but it reproduces reliably 99% of the times in the nixos
> VM, so we're missing a last piece for a "simple" (non-nix) reproducer,
> but I think it's good enough for me to dig into this;

(okay, I got this to work wedging my kernel into the nixos initrd; this
requires the iov to be built off non-contiguous pages and so having
systemd thrash around was apparently a required step...)

So that wasn't a 9p bug, I'm not sure if I should be happy or not?

I've sent "proper-ish" patches at [1] which most concerned people should
be in Cc; I'm fairly confident this will make the bug go away but any
testing is appreciated, please reply to the patches with a Tested-by if
you have time.

[1] https://lkml.kernel.org/r/20250811-iot_iter_folio-v1-0-d9c223adf93c@codewreck.org


Thank you all again, and sorry I haven't had the time to look further
into this without a clean reproducer, this really shouldn't have taken
that long...
-- 
Dominique Martinet | Asmadeus

Re: [REGRESSION] 9pfs issues on 6.12-rc1

[Arnout Engelen]

On Mon, Aug 11, 2025, at 02:57, asmadeus@codewreck.org wrote:
> Arnout Engelen wrote on Sun, Aug 10, 2025 at 07:57:11PM +0200:
> > I have a smallish nix-based reproducer at [3], and a more involved setup
> > with a lot of logging enabled and a convenient way to attach gdb at [4].
> > You start the VM and then 'cat /repro/default.json' manually, and see if
> > it looks 'truncated'.
> 
> Thank you!!! I was able to reproduce with this!
>
> Anyway this is a huge leap forward (hopeful it's the same problem and we
> don't have two similar issues lurking here...), we can't thank you
> enough.

Great - that means a lot ;)

On Mon, Aug 11, 2025, at 09:43, Dominique Martinet wrote:
> So that wasn't a 9p bug, I'm not sure if I should be happy or not?

:D

> I've sent "proper-ish" patches at [1] which most concerned people should
> be in Cc; I'm fairly confident this will make the bug go away but any
> testing is appreciated, please reply to the patches with a Tested-by if
> you have time.
> 
> [1] https://lkml.kernel.org/r/20250811-iot_iter_folio-v1-0-d9c223adf93c@codewreck.org

Awesome!


Kind regards,

-- 
Arnout Engelen
Engelen Open Source
https://engelen.eu