Re: kunit stopped working

Re: kunit stopped working

[David Gow]

On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:
> > +Cc people from culprit commit
>
> Guys, revert helps. I am open to test any solution you may propose, thanks!
>
> ...
>
> > # first bad commit: [757055ae8dedf5333af17b3b5b4b70ba9bc9da4e] init/console: Use ttynull as a fallback when there is no console
>
> --

+CC linux-um

There appear to be two problems here:
1. UML now no longer has console output by default (which KUnit needs
to get results)
2. UML now seems to crash on startup when ttynull is used (which is now default)

This can be worked around for KUnit by passing console=tty to the
kernel. I don't think this is a "correct" fix, as UML seems to be
crashing out-of-the-box anyway (see below), but it may be worth us
forcing this as we require the console output as well.

In any case, this patch fixes it in kunit_tool for now. I may submit
this as a proper patch anyway, but that won't fix UML in general:

diff --git a/tools/testing/kunit/kunit_kernel.py
b/tools/testing/kunit/kunit_kernel.py
index 57c1724b7e5d..698358c9c0d6 100644
--- a/tools/testing/kunit/kunit_kernel.py
+++ b/tools/testing/kunit/kunit_kernel.py
@@ -198,7 +198,7 @@ class LinuxSourceTree(object):
               return self.validate_config(build_dir)

       def run_kernel(self, args=[], build_dir='', timeout=None):
-               args.extend(['mem=1G'])
+               args.extend(['mem=1G', 'console=tty'])
               self._ops.linux_bin(args, timeout, build_dir)
               outfile = get_outfile_path(build_dir)
               subprocess.call(['stty', 'sane'])

---

It looks like this is breaking UML entirely by default, though: if I
start ./vmlinux, I get a SIGSEGV in n_tty_open() early on. It looks
like this is due to the ttynull driver trying to be opened and, for
whatever reason, `ldata' is pointing to invalid memory. I'm not sure
why this is broken but it definitely looks like ttynull is just
completely crashing UML at the moment.
The kernel output and stacktrace look like this:
Core dump limits :
       soft - 0
       hard - NONE
Checking that ptrace can change system call numbers...OK
Checking syscall emulation patch for ptrace...OK
Checking advanced syscall emulation patch for ptrace...OK
Checking environment variables for a tempdir...none found
Checking if /dev/shm is on tmpfs...OK
Checking PROT_EXEC mmap in /dev/shm...OK
Program received signal SIGSEGV, Segmentation fault.
0x00000000601bf17c in n_tty_open (tty=0x6064d800) at
../drivers/tty/n_tty.c:1920
1920            ldata->overrun_time = jiffies;
(gdb) bt
#0  0x00000000601bf17c in n_tty_open (tty=0x6064d800) at
../drivers/tty/n_tty.c:1920
#1  0x00000000601c54e3 in tty_ldisc_open (ld=0x6041a020,
tty=0x6064d800) at ../drivers/tty/tty_ldisc.c:463
#2  tty_ldisc_setup (tty=tty@entry=0x6064d800, o_tty=0x0) at
../drivers/tty/tty_ldisc.c:773
#3  0x00000000601bcb36 in tty_init_dev
(driver=driver@entry=0x6040fb40, idx=0) at
../drivers/tty/tty_io.c:1366
#4  0x00000000601bd822 in tty_open_by_driver (filp=0x60407200,
device=5242881) at ../drivers/tty/tty_io.c:1987
#5  tty_open (inode=0x608050c0, filp=0x60407200) at ../drivers/tty/tty_io.c:2035
#6  0x00000000600ee087 in chrdev_open (inode=inode@entry=0x608050c0,
filp=filp@entry=0x60407200) at ../fs/char_dev.c:414
#7  0x00000000600e40c9 in do_dentry_open (f=f@entry=0x60407200,
inode=0x608050c0, open=0x600ee030 <chrdev_open>, open@entry=0x0) at
../fs/open.c:817
#8  0x00000000600e59d8 in vfs_open (path=path@entry=0x60433db0,
file=file@entry=0x60407200) at ../include/linux/dcache.h:552
#9  0x00000000600fa371 in do_open (op=0x60433ed0, file=0x60407200,
nd=0x60433db0) at ../fs/namei.c:3252
#10 path_openat (nd=nd@entry=0x60433db0, op=op@entry=0x60433ed0,
flags=flags@entry=65) at ../fs/namei.c:3369
#11 0x00000000600fa7ec in do_filp_open (dfd=dfd@entry=-100,
pathname=pathname@entry=0x60498000, op=op@entry=0x60433ed0) at
../fs/namei.c:3396
#12 0x00000000600e5f31 in file_open_name (name=name@entry=0x60498000,
flags=flags@entry=2, mode=mode@entry=0) at ../fs/open.c:1117
#13 0x00000000600e605e in filp_open
(filename=filename@entry=0x6025e26f "/dev/console",
flags=flags@entry=2, mode=mode@entry=0) at ../fs/open.c:1137
#14 0x0000000060001f03 in console_on_rootfs () at ../init/main.c:1480
#15 0x00000000600021b4 in kernel_init_freeable () at ../init/main.c:1536
#16 0x0000000060218bde in kernel_init (unused=<optimized out>) at
../init/main.c:1415
#17 0x00000000600184a1 in new_thread_handler () at
../arch/um/kernel/process.c:136
#18 0x0000000000000000 in ?? ()

Does anyone know why ttynull is being used by default on UML even when
there's the better 'tty' console driver to use? Either way, it'd be
nice if it didn't crash things, too.

Cheers,
-- David

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[David Gow]

On Tue, Dec 22, 2020 at 9:43 AM David Gow <davidgow@google.com> wrote:
>
> On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> >
> > On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:
> > > +Cc people from culprit commit
> >
> > Guys, revert helps. I am open to test any solution you may propose, thanks!
> >
> > ...
> >
> > > # first bad commit: [757055ae8dedf5333af17b3b5b4b70ba9bc9da4e] init/console: Use ttynull as a fallback when there is no console
> >
> > --
>
> +CC linux-um
>
> There appear to be two problems here:
> 1. UML now no longer has console output by default (which KUnit needs
> to get results)
> 2. UML now seems to crash on startup when ttynull is used (which is now default)

Whoops. I was wrong about number 2 here (thanks Johannes for pointing
that out). The segfault in n_tty_open() actually happens normally as
part of UML startup, and it's recovered from perfectly.

So the only real issue is that this is changing the default console
output to ttynull, and hence hiding the output.

>
> This can be worked around for KUnit by passing console=tty to the
> kernel. I don't think this is a "correct" fix, as UML seems to be
> crashing out-of-the-box anyway (see below), but it may be worth us
> forcing this as we require the console output as well.
>
> In any case, this patch fixes it in kunit_tool for now. I may submit
> this as a proper patch anyway, but that won't fix UML in general:
>
> diff --git a/tools/testing/kunit/kunit_kernel.py
> b/tools/testing/kunit/kunit_kernel.py
> index 57c1724b7e5d..698358c9c0d6 100644
> --- a/tools/testing/kunit/kunit_kernel.py
> +++ b/tools/testing/kunit/kunit_kernel.py
> @@ -198,7 +198,7 @@ class LinuxSourceTree(object):
>                return self.validate_config(build_dir)
>
>        def run_kernel(self, args=[], build_dir='', timeout=None):
> -               args.extend(['mem=1G'])
> +               args.extend(['mem=1G', 'console=tty'])
>                self._ops.linux_bin(args, timeout, build_dir)
>                outfile = get_outfile_path(build_dir)
>                subprocess.call(['stty', 'sane'])
>

I'll send this out properly as a patch to kunit_tool: while I still
think that the default console on UML shouldn't change, it probably
makes sense for KUnit to not rely on the default.

Cheers,
-- David

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Andy Shevchenko]

On Tue, Dec 22, 2020 at 03:26:24PM +0800, David Gow wrote:
> On Tue, Dec 22, 2020 at 9:43 AM David Gow <davidgow@google.com> wrote:
> > On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
> > <andriy.shevchenko@linux.intel.com> wrote:
> > > On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:

...

> > > Guys, revert helps. I am open to test any solution you may propose, thanks!

...

> I'll send this out properly as a patch to kunit_tool: while I still
> think that the default console on UML shouldn't change, it probably
> makes sense for KUnit to not rely on the default.

Thanks for fast response. I have tested and answered to the patch.

-- 
With Best Regards,
Andy Shevchenko



_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Brendan Higgins]

On Tue, Dec 22, 2020 at 5:33 AM Andy Shevchenko
<andriy.shevchenko@linux.intel.com> wrote:
>
> On Tue, Dec 22, 2020 at 03:26:24PM +0800, David Gow wrote:
> > On Tue, Dec 22, 2020 at 9:43 AM David Gow <davidgow@google.com> wrote:
> > > On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
> > > <andriy.shevchenko@linux.intel.com> wrote:
> > > > On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:
>
> ...
>
> > > > Guys, revert helps. I am open to test any solution you may propose, thanks!
>
> ...
>
> > I'll send this out properly as a patch to kunit_tool: while I still
> > think that the default console on UML shouldn't change, it probably
> > makes sense for KUnit to not rely on the default.
>
> Thanks for fast response. I have tested and answered to the patch.

Sorry all, I was and still am on vacation.

Looks like this was taken care of, nevertheless, I will make sure to
go and ACK David's fix.

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Petr Mladek]

On Tue 2020-12-22 09:43:48, David Gow wrote:
> On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
> <andriy.shevchenko@linux.intel.com> wrote:
> >
> > On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:
> > > +Cc people from culprit commit
> >
> > Guys, revert helps. I am open to test any solution you may propose, thanks!
> >
> > ...
> >
> > > # first bad commit: [757055ae8dedf5333af17b3b5b4b70ba9bc9da4e] init/console: Use ttynull as a fallback when there is no console
> >
> > --
> 
> +CC linux-um
> 
> There appear to be two problems here:
> 1. UML now no longer has console output by default (which KUnit needs
> to get results)

> This can be worked around for KUnit by passing console=tty to the
> kernel. I don't think this is a "correct" fix

It is rather a workaround. ttynull was supposed to be an ultimate
fallback to provide a "valid" stdin, stdout, and stderr for
the init process. ttyX still should be used by default when
there is no console defined on the command line.

So the question is why ttyX was not registered with this patch.

I see the problem even when I revert the commit
757055ae8dedf5333af ("init/console: Use ttynull as a fallback when
there is no console") and enable the ttynull driver as built in:

     CONFIG_NULL_TTY=y

By other words, the problem existed even before. The commit only
made it visible by default.

I am still trying to understand arch/um and kunit code. I wonder
if it is somehow related to stdiocons implemented in
arch/um/drivers/stdio_console.c.

Best Regards,
Petr

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Petr Mladek]

On Tue 2021-01-05 17:17:08, Petr Mladek wrote:
> On Tue 2020-12-22 09:43:48, David Gow wrote:
> > On Tue, Dec 22, 2020 at 4:02 AM Andy Shevchenko
> > <andriy.shevchenko@linux.intel.com> wrote:
> > >
> > > On Mon, Dec 21, 2020 at 09:40:08PM +0200, Andy Shevchenko wrote:
> > > > +Cc people from culprit commit
> > >
> > > Guys, revert helps. I am open to test any solution you may propose, thanks!
> > >
> > > ...
> > >
> > > > # first bad commit: [757055ae8dedf5333af17b3b5b4b70ba9bc9da4e] init/console: Use ttynull as a fallback when there is no console
> > >
> > > --
> > 
> > +CC linux-um
> > 
> > There appear to be two problems here:
> > 1. UML now no longer has console output by default (which KUnit needs
> > to get results)
> 
> > This can be worked around for KUnit by passing console=tty to the
> > kernel. I don't think this is a "correct" fix
> 
> It is rather a workaround. ttynull was supposed to be an ultimate
> fallback to provide a "valid" stdin, stdout, and stderr for
> the init process. ttyX still should be used by default when
> there is no console defined on the command line.
> 
> So the question is why ttyX was not registered with this patch.
> 
> I see the problem even when I revert the commit
> 757055ae8dedf5333af ("init/console: Use ttynull as a fallback when
> there is no console") and enable the ttynull driver as built in:
> 
>      CONFIG_NULL_TTY=y
> 
> By other words, the problem existed even before. The commit only
> made it visible by default.
> 
> I am still trying to understand arch/um and kunit code. I wonder
> if it is somehow related to stdiocons implemented in
> arch/um/drivers/stdio_console.c.

The following change solved the problem for me as well. It causes
that ttynull is initialized after stdiocons console.

diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
index eced70ec54e1..602af4d30bd4 100644
--- a/drivers/tty/ttynull.c
+++ b/drivers/tty/ttynull.c
@@ -121,7 +121,6 @@ static void __exit ttynull_exit(void)
 	tty_port_destroy(&ttynull_port);
 }
 
-module_init(ttynull_init);
-module_exit(ttynull_exit);
+late_initcall_sync(ttynull_init);
 
 MODULE_LICENSE("GPL v2");

But I am not completely sure that it is the right solution.

It is strange. Console should get registered only when
it was added by add_preferred_console(). It means that
ttynull_init() should not register by default.

Some clue might be in stderr_console. It has
to be explicitly unregistered to avoid staying as
the default console, see unregister_stderr() in
arch/um/drivers/stderr_console.c

I am going to dig more into it.

Best Regards,
Petr

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Sergey Senozhatsky]

On (21/01/05 17:49), Petr Mladek wrote:
> The following change solved the problem for me as well. It causes
> that ttynull is initialized after stdiocons console.
> 
> diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
> index eced70ec54e1..602af4d30bd4 100644
> --- a/drivers/tty/ttynull.c
> +++ b/drivers/tty/ttynull.c
> @@ -121,7 +121,6 @@ static void __exit ttynull_exit(void)
>  	tty_port_destroy(&ttynull_port);
>  }
>  
> -module_init(ttynull_init);
> -module_exit(ttynull_exit);
> +late_initcall_sync(ttynull_init);
>  
>  MODULE_LICENSE("GPL v2");
> 
> But I am not completely sure that it is the right solution.

Wow, hmm, puzzled. Why does it help?

> It is strange. Console should get registered only when
> it was added by add_preferred_console(). It means that
> ttynull_init() should not register by default.
[..]
> Some clue might be in stderr_console. It has
> to be explicitly unregistered to avoid staying as
> the default console, see unregister_stderr() in
> arch/um/drivers/stderr_console.c

Hmm... Some random thoughts:

Looking at arch/um/drivers/stderr_console.c - it doesn't have tty
driver and it doesn't register one. So as far as console_device()
concerned we still don't have a workable console - it will return
NULL to tty_lookup_driver(), which will eventually return an error
to filp_open("/dev/console"); hence we'd call register_ttynull_console()
from console_on_rootfs(). So now we register ttynull as preferred
console; hence when another console attempts to register itself we
don't set CON_CONSDEV on it, because of `has_preferred_console`.

But I still don't understand why the initcall patch helped.
Can you shed some light on it?

	-ss

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Petr Mladek]

On Wed 2021-01-06 13:04:57, Sergey Senozhatsky wrote:
> On (21/01/05 17:49), Petr Mladek wrote:
> > The following change solved the problem for me as well. It causes
> > that ttynull is initialized after stdiocons console.
> > 
> > diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
> > index eced70ec54e1..602af4d30bd4 100644
> > --- a/drivers/tty/ttynull.c
> > +++ b/drivers/tty/ttynull.c
> > @@ -121,7 +121,6 @@ static void __exit ttynull_exit(void)
> >  	tty_port_destroy(&ttynull_port);
> >  }
> >  
> > -module_init(ttynull_init);
> > -module_exit(ttynull_exit);
> > +late_initcall_sync(ttynull_init);
> >  
> >  MODULE_LICENSE("GPL v2");
> > 
> > But I am not completely sure that it is the right solution.
> 
> Wow, hmm, puzzled. Why does it help?

I have been as well. But it seems that I got it, see below.

> > It is strange. Console should get registered only when
> > it was added by add_preferred_console(). It means that
> > ttynull_init() should not register by default.
> [..]
> > Some clue might be in stderr_console. It has
> > to be explicitly unregistered to avoid staying as
> > the default console, see unregister_stderr() in
> > arch/um/drivers/stderr_console.c
> 
> Hmm... Some random thoughts:
> 
> Looking at arch/um/drivers/stderr_console.c - it doesn't have tty
> driver and it doesn't register one.

stderr_console.c is used only during early boot.

stdio_console.c is the console that is supposed to print kunit
test results. And it has tty driver.

> But I still don't understand why the initcall patch helped.
> Can you shed some light on it?

The trick is that both stdio_init() and ttynull_init():

  + call register_console()
  + the console has tty driver

The first one is registered as a fallback when there is
no preferred console (has_preferred_console()).

It means that late_initcall_sync(ttynull_init) makes sense.
We need to call register_console() from ttynull_init() so that
it is registered when defined on the command line. But it should
be the last chance to register a fallback console with tty binding.

Alternative solution is to ignore ttynull as the fallback console
in register_console(). I mean the following:

diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index ffdd0dc7ec6d..cdb77903b0af 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -2816,8 +2816,12 @@ void register_console(struct console *newcon)
 	 *	See if we want to use this console driver. If we
 	 *	didn't select a console we take the first one
 	 *	that registers here.
+	 *
+	 *	Ignore ttynull console. It should be used only
+	 *	when explicitly configured or as an ultimate
+	 *	fallback when no better console gets registered at all.
 	 */
-	if (!has_preferred_console) {
+	if (!has_preferred_console && strcmp(newcon->name, "ttynull") != 0) {
 		if (newcon->index < 0)
 			newcon->index = 0;
 		if (newcon->setup == NULL ||


Best Regards,
Petr

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: kunit stopped working

[Sergey Senozhatsky]

On (21/01/06 14:10), Petr Mladek wrote:
> > > 
> > > But I am not completely sure that it is the right solution.
> > 
> > Wow, hmm, puzzled. Why does it help?
> 
> I have been as well. But it seems that I got it, see below.

Thanks!

[..]
> 
> Alternative solution is to ignore ttynull as the fallback console
> in register_console(). I mean the following:

I personally would prefer a very explicit fix (IOW, the patch below),
rather than relying on some initcall trickery (which has already
failed on us)

> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index ffdd0dc7ec6d..cdb77903b0af 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -2816,8 +2816,12 @@ void register_console(struct console *newcon)
>  	 *	See if we want to use this console driver. If we
>  	 *	didn't select a console we take the first one
>  	 *	that registers here.
> +	 *
> +	 *	Ignore ttynull console. It should be used only
> +	 *	when explicitly configured or as an ultimate
> +	 *	fallback when no better console gets registered at all.
>  	 */
> -	if (!has_preferred_console) {
> +	if (!has_preferred_console && strcmp(newcon->name, "ttynull") != 0) {
>  		if (newcon->index < 0)
>  			newcon->index = 0;
>  		if (newcon->setup == NULL ||

So IIUC in case of ttynull fallback (console= ) we still end up setting
CON_CONSDEV on nulltty console, but we do it in try_enable_new_console().

Feel free to add
Acked-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

	-ss

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um