[PATCH v4 0/2] Optionally allow ttynull to be selected as a default console

[PATCH v4 0/2] Optionally allow ttynull to be selected as a default console

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

When switching to a CONFIG_VT=n world, at least on x86 systems,
/dev/console becomes /dev/ttyS0. This can cause some undesired effects.
/dev/console's behavior is now tied to the physical /dev/ttyS0, which when
disconnected can cause isatty() to fail when /dev/ttyS0 is disconnected,
and users who upgrade to a theoretical vt-less kernel from their
distribution who have a device such as a science instrument connected to
their /dev/ttyS0 port will suddenly see it receive kernel log messages.

When the new CONFIG_NULL_TTY_CONSOLE option is turned on, this will allow
the ttynull device to be leveraged as the default console. Distributions
that had CONFIG_VT turned on before will be able to leverage this option
to where /dev/console is still backed by a psuedo device, avoiding these
issues, without needing to enable the entire VT subsystem.

v2:
    rebase

v3:
    Clarify commit messages.

    Guard the all the register_console()s in ttynull to prevent it from being
    registered twice.

    Only change the link order if CONFIG_NULL_TTY_CONSOLE is enabled, otherwise
    use the existing order for ttynull if only CONFIG_NULL_TTY is enabled.

    Document why the link order changes in the drivers/tty/Makefile file.

    Replace #ifdefs

v4:
    Remember to actually include the changes to v3 in the cover letter.

Adam Simonelli (2):
  ttynull: Add an option to allow ttynull to be used as a console device
  tty: Change order of ttynull to be linked sooner if enabled as a
    console.

 drivers/tty/Kconfig   | 15 ++++++++++++++-
 drivers/tty/Makefile  | 10 ++++++++++
 drivers/tty/ttynull.c | 20 +++++++++++++++++++-
 3 files changed, 43 insertions(+), 2 deletions(-)

-- 
2.45.2

[PATCH v4 1/2] ttynull: Add an option to allow ttynull to be used as a console device

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

The new config option, CONFIG_NULL_TTY_CONSOLE will allow ttynull to be
initialized by console_initcall() and selected as a possible console
device.

Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
---
 drivers/tty/Kconfig   | 15 ++++++++++++++-
 drivers/tty/ttynull.c | 20 +++++++++++++++++++-
 2 files changed, 33 insertions(+), 2 deletions(-)

diff --git a/drivers/tty/Kconfig b/drivers/tty/Kconfig
index 63a494d36a1f..b4afae8b0e74 100644
--- a/drivers/tty/Kconfig
+++ b/drivers/tty/Kconfig
@@ -383,7 +383,20 @@ config NULL_TTY
 	  available or desired.
 
 	  In order to use this driver, you should redirect the console to this
-	  TTY, or boot the kernel with console=ttynull.
+	  TTY, boot the kernel with console=ttynull, or enable
+	  CONFIG_NULL_TTY_CONSOLE.
+
+	  If unsure, say N.
+
+config NULL_TTY_CONSOLE
+        bool "Support for console on ttynull"
+        depends on NULL_TTY=y && !VT_CONSOLE
+	help
+	  Say Y here if you want the NULL TTY to be used as a /dev/console
+	  device.
+
+	  This is similar to CONFIG_VT_CONSOLE, but without the dependency on
+	  CONFIG_VT. It uses the ttynull driver as the system console.
 
 	  If unsure, say N.
 
diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
index 6b2f7208b564..ec3dd3fd41c0 100644
--- a/drivers/tty/ttynull.c
+++ b/drivers/tty/ttynull.c
@@ -57,6 +57,13 @@ static struct tty_driver *ttynull_device(struct console *c, int *index)
 static struct console ttynull_console = {
 	.name = "ttynull",
 	.device = ttynull_device,
+
+	/*
+	 * Match the index and flags from other boot consoles when CONFIG_NULL_TTY_CONSOLE is
+	 * enabled, otherwise, use the default values for the index and flags.
+	*/
+	.index = IS_ENABLED(CONFIG_NULL_TTY_CONSOLE) ? -1 : 0,
+	.flags = IS_ENABLED(CONFIG_NULL_TTY_CONSOLE) ? CON_PRINTBUFFER : 0,
 };
 
 static int __init ttynull_init(void)
@@ -90,11 +97,22 @@ static int __init ttynull_init(void)
 	}
 
 	ttynull_driver = driver;
-	register_console(&ttynull_console);
+	if (!console_is_registered(&ttynull_console))
+		register_console(&ttynull_console);
 
 	return 0;
 }
 
+#ifdef CONFIG_NULL_TTY_CONSOLE
+static int __init ttynull_register(void)
+{
+	if (!console_is_registered(&ttynull_console))
+		register_console(&ttynull_console);
+	return 0;
+}
+console_initcall(ttynull_register);
+#endif
+
 static void __exit ttynull_exit(void)
 {
 	unregister_console(&ttynull_console);
-- 
2.45.2

[PATCH v4 2/2] tty: Change order of ttynull to be linked sooner if enabled as a console.

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

If CONFIG_NULL_TTY_CONSOLE is enabled, and CONFIG_VT is disabled, ttynull
will become the default primary console device, based on the link order.

Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
if CONFIG_VT is disabled, the default console device falls back to
/dev/ttyS0 instead of /dev/tty.

This could cause issues in user space, and hardware problems:

1. The user space issues include the case where  /dev/ttyS0 is
disconnected, and the TCGETS ioctl, which some user space libraries use
as a probe to determine if a file is a tty, is called on /dev/console and
fails. Programs that call isatty() on /dev/console and get an incorrect
false value may skip expected logging to /dev/console

2. The hardware issues include the case if a user has a science instrument
or other device connected to the /dev/ttyS0 port, and they were to upgrade
to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
sent to the device connected to /dev/ttyS0 unless they edit their kernel
command line manually.

The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
maintainers an option to avoid this. Disabling CONFIG_VT and enabling
CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
is not dependant on hardware configuration by default, and avoid
unexpected new behavior on devices connected to the /dev/ttyS0 serial
port.

Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
---
 drivers/tty/Makefile | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/drivers/tty/Makefile b/drivers/tty/Makefile
index 07aca5184a55..1a1051ecb1af 100644
--- a/drivers/tty/Makefile
+++ b/drivers/tty/Makefile
@@ -11,6 +11,10 @@ obj-$(CONFIG_N_HDLC)		+= n_hdlc.o
 obj-$(CONFIG_N_GSM)		+= n_gsm.o
 
 obj-y				+= vt/
+#If ttynull is configured to be a console by default, ensure that it is linked
+#earlier before a real one is selected.
+obj-$(CONFIG_NULL_TTY_CONSOLE)	+= ttynull.o
+
 obj-$(CONFIG_HVC_DRIVER)	+= hvc/
 obj-y				+= serial/
 obj-$(CONFIG_SERIAL_DEV_BUS)	+= serdev/
@@ -20,7 +24,13 @@ obj-$(CONFIG_AMIGA_BUILTIN_SERIAL) += amiserial.o
 obj-$(CONFIG_MOXA_INTELLIO)	+= moxa.o
 obj-$(CONFIG_MOXA_SMARTIO)	+= mxser.o
 obj-$(CONFIG_NOZOMI)		+= nozomi.o
+
+#If ttynull is enabled, but not as a boot console, it is linked and used later
+#after the real ones.
+ifneq ($(CONFIG_NULL_TTY_CONSOLE),y)
 obj-$(CONFIG_NULL_TTY)	        += ttynull.o
+endif
+
 obj-$(CONFIG_SYNCLINK_GT)	+= synclink_gt.o
 obj-$(CONFIG_PPC_EPAPR_HV_BYTECHAN) += ehv_bytechan.o
 obj-$(CONFIG_GOLDFISH_TTY)	+= goldfish.o
-- 
2.45.2

Re: [PATCH v4 0/2] Optionally allow ttynull to be selected as a default console

[Andy Shevchenko]

Sun, Feb 23, 2025 at 03:44:54PM -0500, adamsimonelli@gmail.com kirjoitti:
> From: Adam Simonelli <adamsimonelli@gmail.com>
> 
> When switching to a CONFIG_VT=n world, at least on x86 systems,
> /dev/console becomes /dev/ttyS0. This can cause some undesired effects.
> /dev/console's behavior is now tied to the physical /dev/ttyS0, which when
> disconnected can cause isatty() to fail when /dev/ttyS0 is disconnected,
> and users who upgrade to a theoretical vt-less kernel from their
> distribution who have a device such as a science instrument connected to
> their /dev/ttyS0 port will suddenly see it receive kernel log messages.
> 
> When the new CONFIG_NULL_TTY_CONSOLE option is turned on, this will allow
> the ttynull device to be leveraged as the default console. Distributions
> that had CONFIG_VT turned on before will be able to leverage this option
> to where /dev/console is still backed by a psuedo device, avoiding these
> issues, without needing to enable the entire VT subsystem.

This rings a bell of the following

https://lore.kernel.org/all/20201111135450.11214-1-pmladek@suse.com/
https://lore.kernel.org/all/20210107164400.17904-1-pmladek@suse.com/
https://lore.kernel.org/all/20210108114847.23469-1-pmladek@suse.com/

I don't see any mention in the commit message about these, have you studied the
cases? Will your change anyhow affect the described there?

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v4 0/2] Optionally allow ttynull to be selected as a default console

[Adam Simonelli]

On Sunday, February 23, 2025 4:10:10 PM EST Andy Shevchenko wrote:
> Sun, Feb 23, 2025 at 03:44:54PM -0500, adamsimonelli@gmail.com kirjoitti:
> > From: Adam Simonelli <adamsimonelli@gmail.com>
> > 
> > When switching to a CONFIG_VT=n world, at least on x86 systems,
> > /dev/console becomes /dev/ttyS0. This can cause some undesired effects.
> > /dev/console's behavior is now tied to the physical /dev/ttyS0, which when
> > disconnected can cause isatty() to fail when /dev/ttyS0 is disconnected,
> > and users who upgrade to a theoretical vt-less kernel from their
> > distribution who have a device such as a science instrument connected to
> > their /dev/ttyS0 port will suddenly see it receive kernel log messages.
> > 
> > When the new CONFIG_NULL_TTY_CONSOLE option is turned on, this will allow
> > the ttynull device to be leveraged as the default console. Distributions
> > that had CONFIG_VT turned on before will be able to leverage this option
> > to where /dev/console is still backed by a psuedo device, avoiding these
> > issues, without needing to enable the entire VT subsystem.
> 
> This rings a bell of the following
> 
> https://lore.kernel.org/all/20201111135450.11214-1-pmladek@suse.com/
> https://lore.kernel.org/all/20210107164400.17904-1-pmladek@suse.com/
> https://lore.kernel.org/all/20210108114847.23469-1-pmladek@suse.com/
> 
> I don't see any mention in the commit message about these, have you studied the
> cases? Will your change anyhow affect the described there?
> 
> 
I did see that sifting through commits, it looks kind of different though, as
that was to make ttynull more always on, and if I am understanding it correctly
it looks more of a last resort? I could be wrong, but I see it was attempted
and reverted because of conflicts on some hardware platforms?

The scope with this new patch set is much different, as it has to be manually
enabled on top of CONFIG_NULL_TTY, rather than assuming CONFIG_NULL_TTY is
enabled, and adding it to the list of preferred consoles. It is more for
Desktop configs that are looking to disable CONFIG_VT, but still want
/dev/console to not be a physical device by default.

Re: [PATCH v4 0/2] Optionally allow ttynull to be selected as a default console

[Andy Shevchenko]

On Mon, Feb 24, 2025 at 2:23 AM Adam Simonelli <adamsimonelli@gmail.com> wrote:
> On Sunday, February 23, 2025 4:10:10 PM EST Andy Shevchenko wrote:
> > Sun, Feb 23, 2025 at 03:44:54PM -0500, adamsimonelli@gmail.com kirjoitti:
> > > From: Adam Simonelli <adamsimonelli@gmail.com>
> > >
> > > When switching to a CONFIG_VT=n world, at least on x86 systems,
> > > /dev/console becomes /dev/ttyS0. This can cause some undesired effects.
> > > /dev/console's behavior is now tied to the physical /dev/ttyS0, which when
> > > disconnected can cause isatty() to fail when /dev/ttyS0 is disconnected,
> > > and users who upgrade to a theoretical vt-less kernel from their
> > > distribution who have a device such as a science instrument connected to
> > > their /dev/ttyS0 port will suddenly see it receive kernel log messages.
> > >
> > > When the new CONFIG_NULL_TTY_CONSOLE option is turned on, this will allow
> > > the ttynull device to be leveraged as the default console. Distributions
> > > that had CONFIG_VT turned on before will be able to leverage this option
> > > to where /dev/console is still backed by a psuedo device, avoiding these
> > > issues, without needing to enable the entire VT subsystem.
> >
> > This rings a bell of the following
> >
> > https://lore.kernel.org/all/20201111135450.11214-1-pmladek@suse.com/
> > https://lore.kernel.org/all/20210107164400.17904-1-pmladek@suse.com/
> > https://lore.kernel.org/all/20210108114847.23469-1-pmladek@suse.com/
> >
> > I don't see any mention in the commit message about these, have you studied the
> > cases? Will your change anyhow affect the described there?
> >
> I did see that sifting through commits, it looks kind of different though, as
> that was to make ttynull more always on, and if I am understanding it correctly
> it looks more of a last resort? I could be wrong, but I see it was attempted
> and reverted because of conflicts on some hardware platforms?
>
> The scope with this new patch set is much different, as it has to be manually
> enabled on top of CONFIG_NULL_TTY, rather than assuming CONFIG_NULL_TTY is
> enabled, and adding it to the list of preferred consoles. It is more for
> Desktop configs that are looking to disable CONFIG_VT, but still want
> /dev/console to not be a physical device by default.

Thank you for elaboration. Even if this may be a different case, I
think it would be nice to have the PRINTK maintainers (who are also
involved in the console code) blessing before going in.

In case of a new version, please summarize above in the commit message.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v4 2/2] tty: Change order of ttynull to be linked sooner if enabled as a console.

[Greg Kroah-Hartman]

On Sun, Feb 23, 2025 at 03:44:56PM -0500, adamsimonelli@gmail.com wrote:
> From: Adam Simonelli <adamsimonelli@gmail.com>
> 
> If CONFIG_NULL_TTY_CONSOLE is enabled, and CONFIG_VT is disabled, ttynull
> will become the default primary console device, based on the link order.
> 
> Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
> if CONFIG_VT is disabled, the default console device falls back to
> /dev/ttyS0 instead of /dev/tty.
> 
> This could cause issues in user space, and hardware problems:
> 
> 1. The user space issues include the case where  /dev/ttyS0 is
> disconnected, and the TCGETS ioctl, which some user space libraries use
> as a probe to determine if a file is a tty, is called on /dev/console and
> fails. Programs that call isatty() on /dev/console and get an incorrect
> false value may skip expected logging to /dev/console
> 
> 2. The hardware issues include the case if a user has a science instrument
> or other device connected to the /dev/ttyS0 port, and they were to upgrade
> to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
> sent to the device connected to /dev/ttyS0 unless they edit their kernel
> command line manually.
> 
> The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
> maintainers an option to avoid this. Disabling CONFIG_VT and enabling
> CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
> is not dependant on hardware configuration by default, and avoid
> unexpected new behavior on devices connected to the /dev/ttyS0 serial
> port.
> 
> Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
> ---
>  drivers/tty/Makefile | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/drivers/tty/Makefile b/drivers/tty/Makefile
> index 07aca5184a55..1a1051ecb1af 100644
> --- a/drivers/tty/Makefile
> +++ b/drivers/tty/Makefile
> @@ -11,6 +11,10 @@ obj-$(CONFIG_N_HDLC)		+= n_hdlc.o
>  obj-$(CONFIG_N_GSM)		+= n_gsm.o
>  
>  obj-y				+= vt/
> +#If ttynull is configured to be a console by default, ensure that it is linked
> +#earlier before a real one is selected.
> +obj-$(CONFIG_NULL_TTY_CONSOLE)	+= ttynull.o
> +
>  obj-$(CONFIG_HVC_DRIVER)	+= hvc/
>  obj-y				+= serial/
>  obj-$(CONFIG_SERIAL_DEV_BUS)	+= serdev/
> @@ -20,7 +24,13 @@ obj-$(CONFIG_AMIGA_BUILTIN_SERIAL) += amiserial.o
>  obj-$(CONFIG_MOXA_INTELLIO)	+= moxa.o
>  obj-$(CONFIG_MOXA_SMARTIO)	+= mxser.o
>  obj-$(CONFIG_NOZOMI)		+= nozomi.o
> +
> +#If ttynull is enabled, but not as a boot console, it is linked and used later
> +#after the real ones.
> +ifneq ($(CONFIG_NULL_TTY_CONSOLE),y)
>  obj-$(CONFIG_NULL_TTY)	        += ttynull.o
> +endif

Nit, a " " needs to be after the "#" character, right?

And ick, this is going to be tricky, changing the link order depending
on the configuration option setting?  This feels wrong, and messy, and
very fragile.

thanks,

greg k-h

Re: [PATCH v4 2/2] tty: Change order of ttynull to be linked sooner if enabled as a console.

[Adam Simonelli]

On Monday, February 24, 2025 2:23:09 AM EST Greg Kroah-Hartman wrote:
> On Sun, Feb 23, 2025 at 03:44:56PM -0500, adamsimonelli@gmail.com wrote:
> > From: Adam Simonelli <adamsimonelli@gmail.com>
> > 
> > If CONFIG_NULL_TTY_CONSOLE is enabled, and CONFIG_VT is disabled, ttynull
> > will become the default primary console device, based on the link order.
> > 
> > Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
> > if CONFIG_VT is disabled, the default console device falls back to
> > /dev/ttyS0 instead of /dev/tty.
> > 
> > This could cause issues in user space, and hardware problems:
> > 
> > 1. The user space issues include the case where  /dev/ttyS0 is
> > disconnected, and the TCGETS ioctl, which some user space libraries use
> > as a probe to determine if a file is a tty, is called on /dev/console and
> > fails. Programs that call isatty() on /dev/console and get an incorrect
> > false value may skip expected logging to /dev/console
> > 
> > 2. The hardware issues include the case if a user has a science instrument
> > or other device connected to the /dev/ttyS0 port, and they were to upgrade
> > to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
> > sent to the device connected to /dev/ttyS0 unless they edit their kernel
> > command line manually.
> > 
> > The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
> > maintainers an option to avoid this. Disabling CONFIG_VT and enabling
> > CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
> > is not dependant on hardware configuration by default, and avoid
> > unexpected new behavior on devices connected to the /dev/ttyS0 serial
> > port.
> > 
> > Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
> > ---
> >  drivers/tty/Makefile | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> > 
> > diff --git a/drivers/tty/Makefile b/drivers/tty/Makefile
> > index 07aca5184a55..1a1051ecb1af 100644
> > --- a/drivers/tty/Makefile
> > +++ b/drivers/tty/Makefile
> > @@ -11,6 +11,10 @@ obj-$(CONFIG_N_HDLC)		+= n_hdlc.o
> >  obj-$(CONFIG_N_GSM)		+= n_gsm.o
> >  
> >  obj-y				+= vt/
> > +#If ttynull is configured to be a console by default, ensure that it is linked
> > +#earlier before a real one is selected.
> > +obj-$(CONFIG_NULL_TTY_CONSOLE)	+= ttynull.o
> > +
> >  obj-$(CONFIG_HVC_DRIVER)	+= hvc/
> >  obj-y				+= serial/
> >  obj-$(CONFIG_SERIAL_DEV_BUS)	+= serdev/
> > @@ -20,7 +24,13 @@ obj-$(CONFIG_AMIGA_BUILTIN_SERIAL) += amiserial.o
> >  obj-$(CONFIG_MOXA_INTELLIO)	+= moxa.o
> >  obj-$(CONFIG_MOXA_SMARTIO)	+= mxser.o
> >  obj-$(CONFIG_NOZOMI)		+= nozomi.o
> > +
> > +#If ttynull is enabled, but not as a boot console, it is linked and used later
> > +#after the real ones.
> > +ifneq ($(CONFIG_NULL_TTY_CONSOLE),y)
> >  obj-$(CONFIG_NULL_TTY)	        += ttynull.o
> > +endif
> 
> Nit, a " " needs to be after the "#" character, right?
> 
> And ick, this is going to be tricky, changing the link order depending
> on the configuration option setting?  This feels wrong, and messy, and
> very fragile.
> 
Yeah, it did feel kind of odd I will admit, but I was worried if there are any
ramifications that I don't know about if I move it up early for all cases now
or if I should just put it back to how it was in v1...
> thanks,
> 
> greg k-h
>