9p regression linux-next next-20230327

9p regression linux-next next-20230327

[Luis Chamberlain]

kdevops uses 9p for its client / host setup to enable rapid kernel
development on guests using the host to compile / guest to install
a read-only-mount.

I updated the kernel today to next-20230327 and see the 9p mount won't
come up and so boot fails. I'm starting to bisect and see next-20230301
was OK. So at least have:

next-20230301: GOOD                                                             
next-20230315: GOOD
next-20230327: BAD

Once I narrow this down a bit further I'll poke back.

What sort of tests are run for 9p before some development changes get
pushed into a tree that lands on linux-next? If none exists, simply
testing 'make linux' works with kdevops could perhaps be a start.

  Luis

Re: 9p regression linux-next next-20230327

[asmadeus]

Luis Chamberlain wrote on Mon, Mar 27, 2023 at 06:12:13PM -0700:
> kdevops uses 9p for its client / host setup to enable rapid kernel
> development on guests using the host to compile / guest to install
> a read-only-mount.

Thanks for the head's up!


> I updated the kernel today to next-20230327 and see the 9p mount won't
> come up and so boot fails. I'm starting to bisect and see next-20230301
> was OK. So at least have:
> 
> next-20230301: GOOD                                                             
> next-20230315: GOOD
> next-20230327: BAD
> 
> Once I narrow this down a bit further I'll poke back.
> 
> What sort of tests are run for 9p before some development changes get
> pushed into a tree that lands on linux-next? If none exists, simply
> testing 'make linux' works with kdevops could perhaps be a start.

I generally boot a qemu VM and run some semi-automated tests, and Eric
has his own set of automated testing, so we should both be testing stuff
that's being sent to -next, so I'm surprised it wouldn't mount at all...
(and I haven't pushed anything there recently)

There might be some unrelated change that impacts the way we're
mounting?
I've just built Eric's for-next branch and I'm not seeing any issue
there, I'll be checking the next tag you pointed at next.


I'm going to rant a bit here, but my main problem with testing is that
there are plenty of tools, but I have very little compute available for
this (just a small machine at home that's getting close to 10 years
old...), as this is really a free time activity which isn't getting any
funding.
[read: happy to spend a bit more time on 9p if it's getting paid :P]

(Linaro offered me some resources to build over different arches a while
ago and I've maybe used it once, but I never got to the end of their
LKFT (Linux Kernel Functional Testing, https://lkft.linaro.org/ )
framework to see if I am allowed to run the kernel that was just built
so it's a bit pointless if it doesn't go all the way...)


Eric brought up that we should share more tests and I definitely agree;
my setup is closely tied to my machine (mounts /nix/store over nfs and
runs tests on 9p from there as my host is running nixos), but if we
could setup some piece of shared infra to work on tests together I'd be
happy to add some of my tests there.

-- 
Dominique

Re: [V9fs-developer] 9p regression linux-next next-20230327

[asmadeus]

asmadeus@codewreck.org wrote on Tue, Mar 28, 2023 at 10:44:46AM +0900:
> I've just built Eric's for-next branch and I'm not seeing any issue
> there, I'll be checking the next tag you pointed at next.

Well, at the very least I can confirm mount hangs there :)

[<0>] p9_client_rpc+0xf1/0x380 [9pnet]
[<0>] p9_client_attach+0x8d/0x1d0 [9pnet]
[<0>] v9fs_session_init+0x4ad/0x810 [9p]
[<0>] v9fs_mount+0x6a/0x420 [9p]
[<0>] legacy_get_tree+0x28/0x50
[<0>] vfs_get_tree+0x1a/0x90
[<0>] path_mount+0x746/0x9b0
[<0>] __x64_sys_mount+0x153/0x1b0
[<0>] do_syscall_64+0x3c/0x80
[<0>] entry_SYSCALL_64_after_hwframe+0x63/0xcd


And my async flush hasn't gotten fixed yet, so that mount cannot be
killed either.


I can mount over tcp so it's a virtio change, I'll dig a bit more and
report back with you and cc
-- 
Dominique

Re: 9p regression linux-next next-20230327

[Luis Chamberlain]

On Tue, Mar 28, 2023 at 10:44:46AM +0900, asmadeus@codewreck.org wrote:
> I'm going to rant a bit here, but my main problem with testing is that
> there are plenty of tools, but I have very little compute available for
> this (just a small machine at home that's getting close to 10 years
> old...), as this is really a free time activity which isn't getting any
> funding.
> [read: happy to spend a bit more time on 9p if it's getting paid :P]

I think we can likely help with at least hardware access to help test 9p well.
If you're interested let me know!

  Luis

Re: 9p regression linux-next next-20230327

[Eric Van Hensbergen]

So, I'm still mucking about - but I actually have a threadripper and
an Asahi-M1 here that I'm dedicating to regression testing but haven't
quite finished setting it up.  If you look in
https://github.com/v9fs/test you can see the simple tests I'm running
(dbench, fsx, postmark basically) with different configs specified by
fstabs -- I'm using cpu in my automation (the docker container which
it is supposed to run in is in https://github.com/v9fs/docker).  I
started playing with github actions, but quickly realized it was going
to be an nightmare to manage the .github/workflow embedded in linux
trees so am rethinking that strategy.

But, in order to move past the simple virtio tests and incorporate the
network servers, I'm going to probably rework the performance
framework I have in https://github.com/v9fs/notebook to be able to
cover all the different configs and options in the tests and then have
some shortcuts for smoke test, short tests, and long regression tests.
Then I can look at broadening the test programs and also look at
automating nightly regressions (including performance tracking)
against linux-next and mainline.

It has taken me longer than I would have liked, but I'm getting closer
-- and i have to look at kdevops as well!

        -eric

On Mon, Mar 27, 2023 at 11:46 PM Luis Chamberlain <mcgrof@kernel.org> wrote:
>
> On Tue, Mar 28, 2023 at 10:44:46AM +0900, asmadeus@codewreck.org wrote:
> > I'm going to rant a bit here, but my main problem with testing is that
> > there are plenty of tools, but I have very little compute available for
> > this (just a small machine at home that's getting close to 10 years
> > old...), as this is really a free time activity which isn't getting any
> > funding.
> > [read: happy to spend a bit more time on 9p if it's getting paid :P]
>
> I think we can likely help with at least hardware access to help test 9p well.
> If you're interested let me know!
>
>   Luis