[PATCH v7 0/3] Optionally allow ttynull to be selected as a default console

[PATCH v7 0/3] Optionally allow ttynull to be selected as a default console

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

When switching to a CONFIG_VT=n world, at least on x86 systems,
/dev/console becomes /dev/ttyS0. This can cause some undesired effects.
/dev/console's behavior is now tied to the physical /dev/ttyS0, which when
disconnected can cause isatty() to fail when /dev/ttyS0 is disconnected,
and users who upgrade to a theoretical vt-less kernel from their
distribution who have a device such as a science instrument connected to
their /dev/ttyS0 port will suddenly see it receive kernel log messages.

When the new CONFIG_NULL_TTY_DEFAULT_CONSOLE option is turned on, this will
allow the ttynull device to be leveraged as the default console. Distributions
that had CONFIG_VT turned on before will be able to leverage this option
to where /dev/console is still backed by a psuedo device, avoiding these
issues, without needing to enable the entire VT subsystem.

v2:
    rebase

v3:
    Clarify commit messages.

    Guard the all the register_console()s in ttynull to prevent it from being
    registered twice.

    Only change the link order if CONFIG_NULL_TTY_CONSOLE is enabled, otherwise
    use the existing order for ttynull if only CONFIG_NULL_TTY is enabled.

    Document why the link order changes in the drivers/tty/Makefile file.

    Replace #ifdefs

v4:
    Remember to actually include the changes to v3 in the cover letter.

v5:
    Correct code formatting in Makefile comment.

v6:
    Change to CONFIG_NULL_TTY_DEFAULT_CONSOLE

    Set the index to -1, and don't set the flags

    Use add_preferred_console() instead of register_console() in ttynull's
    console_initcall when CONFIG_NULL_TTY_DEFAULT_CONSOLE is enabled

v7
    Add a commit message to the first commit, and the Suggested-by

    Correct Kconfig tabs/spaces formatting

    Invert the console_set_on_cmdline check

Adam Simonelli (3):
  ttynull: Always initialize console index to -1
  ttynull: Add an option to allow ttynull to be used as a console device
  tty: Change order of ttynull to be linked sooner if enabled as a
    console.

 drivers/tty/Kconfig   | 15 ++++++++++++++-
 drivers/tty/Makefile  | 12 ++++++++++++
 drivers/tty/ttynull.c | 14 ++++++++++++++
 3 files changed, 40 insertions(+), 1 deletion(-)

-- 
2.45.2

[PATCH v7 1/3] ttynull: Always initialize console index to -1

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

This allows ttynull to be considered in console selection.

Suggested-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
---
 drivers/tty/ttynull.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
index 6b2f7208b564..d8b51edde628 100644
--- a/drivers/tty/ttynull.c
+++ b/drivers/tty/ttynull.c
@@ -57,6 +57,7 @@ static struct tty_driver *ttynull_device(struct console *c, int *index)
 static struct console ttynull_console = {
 	.name = "ttynull",
 	.device = ttynull_device,
+	.index = -1,
 };
 
 static int __init ttynull_init(void)
-- 
2.45.2

[PATCH v7 2/3] ttynull: Add an option to allow ttynull to be used as a console device

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

The new config option, CONFIG_NULL_TTY_DEFAULT_ CONSOLE will allow
ttynull to be initialized by console_initcall() and selected as a
possible console device.

Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
---
 drivers/tty/Kconfig   | 15 ++++++++++++++-
 drivers/tty/ttynull.c | 13 +++++++++++++
 2 files changed, 27 insertions(+), 1 deletion(-)

diff --git a/drivers/tty/Kconfig b/drivers/tty/Kconfig
index 63a494d36a1f..00ec1acb69ac 100644
--- a/drivers/tty/Kconfig
+++ b/drivers/tty/Kconfig
@@ -383,7 +383,20 @@ config NULL_TTY
 	  available or desired.
 
 	  In order to use this driver, you should redirect the console to this
-	  TTY, or boot the kernel with console=ttynull.
+	  TTY, boot the kernel with console=ttynull, or enable
+	  CONFIG_NULL_TTY_DEFAULT_CONSOLE.
+
+	  If unsure, say N.
+
+config NULL_TTY_DEFAULT_CONSOLE
+	bool "Support for console on ttynull"
+	depends on NULL_TTY=y && !VT_CONSOLE
+	help
+	  Say Y here if you want the NULL TTY to be used as a /dev/console
+	  device.
+
+	  This is similar to CONFIG_VT_CONSOLE, but without the dependency on
+	  CONFIG_VT. It uses the ttynull driver as the system console.
 
 	  If unsure, say N.
 
diff --git a/drivers/tty/ttynull.c b/drivers/tty/ttynull.c
index d8b51edde628..67aad3e6f538 100644
--- a/drivers/tty/ttynull.c
+++ b/drivers/tty/ttynull.c
@@ -96,6 +96,19 @@ static int __init ttynull_init(void)
 	return 0;
 }
 
+#ifdef CONFIG_NULL_TTY_DEFAULT_CONSOLE
+static int __init ttynull_register(void)
+{
+	if (console_set_on_cmdline)
+		return 0;
+
+	add_preferred_console("ttynull", 0, NULL);
+
+	return 0;
+}
+console_initcall(ttynull_register);
+#endif
+
 static void __exit ttynull_exit(void)
 {
 	unregister_console(&ttynull_console);
-- 
2.45.2

[PATCH v7 3/3] tty: Change order of ttynull to be linked sooner if enabled as a console.

[adamsimonelli]

From: Adam Simonelli <adamsimonelli@gmail.com>

If CONFIG_NULL_TTY_DEFAULT_CONSOLE is enabled, and CONFIG_VT is disabled,
ttynull will become the default primary console device, based on the link
order.

Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
if CONFIG_VT is disabled, the default console device falls back to
/dev/ttyS0 instead of /dev/tty.

This could cause issues in user space, and hardware problems:

1. The user space issues include the case where  /dev/ttyS0 is
disconnected, and the TCGETS ioctl, which some user space libraries use
as a probe to determine if a file is a tty, is called on /dev/console and
fails. Programs that call isatty() on /dev/console and get an incorrect
false value may skip expected logging to /dev/console.

2. The hardware issues include the case if a user has a science instrument
or other device connected to the /dev/ttyS0 port, and they were to upgrade
to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
sent to the device connected to /dev/ttyS0 unless they edit their kernel
command line manually.

The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
maintainers an option to avoid this. Disabling CONFIG_VT and enabling
CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
is not dependant on hardware configuration by default, and avoid
unexpected new behavior on devices connected to the /dev/ttyS0 serial
port.

Signed-off-by: Adam Simonelli <adamsimonelli@gmail.com>
---
 drivers/tty/Makefile | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/drivers/tty/Makefile b/drivers/tty/Makefile
index 07aca5184a55..7d0414dc31ed 100644
--- a/drivers/tty/Makefile
+++ b/drivers/tty/Makefile
@@ -11,6 +11,12 @@ obj-$(CONFIG_N_HDLC)		+= n_hdlc.o
 obj-$(CONFIG_N_GSM)		+= n_gsm.o
 
 obj-y				+= vt/
+
+# If ttynull is configured to be a console by default, ensure that it is linked
+# earlier before a real one is selected.
+obj-$(CONFIG_NULL_TTY_DEFAULT_CONSOLE) \
+				+= ttynull.o
+
 obj-$(CONFIG_HVC_DRIVER)	+= hvc/
 obj-y				+= serial/
 obj-$(CONFIG_SERIAL_DEV_BUS)	+= serdev/
@@ -20,7 +26,13 @@ obj-$(CONFIG_AMIGA_BUILTIN_SERIAL) += amiserial.o
 obj-$(CONFIG_MOXA_INTELLIO)	+= moxa.o
 obj-$(CONFIG_MOXA_SMARTIO)	+= mxser.o
 obj-$(CONFIG_NOZOMI)		+= nozomi.o
+
+# If ttynull is enabled, but not as a boot console, it is linked and used later
+# after the real ones.
+ifneq ($(CONFIG_NULL_TTY_DEFAULT_CONSOLE),y)
 obj-$(CONFIG_NULL_TTY)	        += ttynull.o
+endif
+
 obj-$(CONFIG_SYNCLINK_GT)	+= synclink_gt.o
 obj-$(CONFIG_PPC_EPAPR_HV_BYTECHAN) += ehv_bytechan.o
 obj-$(CONFIG_GOLDFISH_TTY)	+= goldfish.o
-- 
2.45.2

Re: [PATCH v7 2/3] ttynull: Add an option to allow ttynull to be used as a console device

[Andy Shevchenko]

On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:
>
> From: Adam Simonelli <adamsimonelli@gmail.com>
>
> The new config option, CONFIG_NULL_TTY_DEFAULT_ CONSOLE will allow
> ttynull to be initialized by console_initcall() and selected as a
> possible console device.

...

>           In order to use this driver, you should redirect the console to this
> -         TTY, or boot the kernel with console=ttynull.
> +         TTY, boot the kernel with console=ttynull, or enable
> +         CONFIG_NULL_TTY_DEFAULT_CONSOLE.

I haven't checked what it looks like in menuconfig / nconfig / etc,
but I think that CONFIG_ is redundant here.

> +         If unsure, say N.
> +
> +config NULL_TTY_DEFAULT_CONSOLE
> +       bool "Support for console on ttynull"
> +       depends on NULL_TTY=y && !VT_CONSOLE
> +       help
> +         Say Y here if you want the NULL TTY to be used as a /dev/console
> +         device.
> +
> +         This is similar to CONFIG_VT_CONSOLE, but without the dependency on
> +         CONFIG_VT. It uses the ttynull driver as the system console.

Btw, do those `make nconfig` and friends render the options?

>           If unsure, say N.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v7 3/3] tty: Change order of ttynull to be linked sooner if enabled as a console.

[Andy Shevchenko]

On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:
>
> From: Adam Simonelli <adamsimonelli@gmail.com>
>
> If CONFIG_NULL_TTY_DEFAULT_CONSOLE is enabled, and CONFIG_VT is disabled,
> ttynull will become the default primary console device, based on the link
> order.
>
> Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
> if CONFIG_VT is disabled, the default console device falls back to
> /dev/ttyS0 instead of /dev/tty.
>
> This could cause issues in user space, and hardware problems:
>
> 1. The user space issues include the case where  /dev/ttyS0 is
> disconnected, and the TCGETS ioctl, which some user space libraries use
> as a probe to determine if a file is a tty, is called on /dev/console and
> fails. Programs that call isatty() on /dev/console and get an incorrect
> false value may skip expected logging to /dev/console.
>
> 2. The hardware issues include the case if a user has a science instrument
> or other device connected to the /dev/ttyS0 port, and they were to upgrade
> to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
> sent to the device connected to /dev/ttyS0 unless they edit their kernel
> command line manually.
>
> The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
> maintainers an option to avoid this. Disabling CONFIG_VT and enabling
> CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
> is not dependant on hardware configuration by default, and avoid
> unexpected new behavior on devices connected to the /dev/ttyS0 serial
> port.

...

Since it touches the link order only under drivers/tty the commit
message should mention what the effect will be on the consoles drivers
for which are located elsewhere.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v7 2/3] ttynull: Add an option to allow ttynull to be used as a console device

[Adam Simonelli]

[-- Attachment #1: Type: text/plain, Size: 1721 bytes --]

On Wednesday, March 5, 2025 2:18:39 PM EST Andy Shevchenko wrote:
> On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:
> >
> > From: Adam Simonelli <adamsimonelli@gmail.com>
> >
> > The new config option, CONFIG_NULL_TTY_DEFAULT_ CONSOLE will allow
> > ttynull to be initialized by console_initcall() and selected as a
> > possible console device.
> 
> ...
> 
> >           In order to use this driver, you should redirect the console to this
> > -         TTY, or boot the kernel with console=ttynull.
> > +         TTY, boot the kernel with console=ttynull, or enable
> > +         CONFIG_NULL_TTY_DEFAULT_CONSOLE.
> 
> I haven't checked what it looks like in menuconfig / nconfig / etc,
> but I think that CONFIG_ is redundant here.
> 
OK, I didn't know what one is more typical. Doing a loose grep for
"^<tab><space><space>" and "[A-Z]_[A-Z]" (excluding CONFIG_) seems like there
are more help text lines that mention other config options (~622) than include
the CONFIG (174). I will change it
> > +         If unsure, say N.
> > +
> > +config NULL_TTY_DEFAULT_CONSOLE
> > +       bool "Support for console on ttynull"
> > +       depends on NULL_TTY=y && !VT_CONSOLE
> > +       help
> > +         Say Y here if you want the NULL TTY to be used as a /dev/console
> > +         device.
> > +
> > +         This is similar to CONFIG_VT_CONSOLE, but without the dependency on
> > +         CONFIG_VT. It uses the ttynull driver as the system console.
> 
> Btw, do those `make nconfig` and friends render the options?
Seems like `make nconfig` works, as well as `make xconfig` I will attempt to
attach a png screenshot.
> 
> >           If unsure, say N.
> 
> 


[-- Attachment #2: CONFIG_NULL_TTY_DEFAULT_CONSOLE.png --]
[-- Type: image/png, Size: 115284 bytes --]

Re: [PATCH v7 3/3] tty: Change order of ttynull to be linked sooner if enabled as a console.

[Adam Simonelli]

On Wednesday, March 5, 2025 2:22:06 PM EST Andy Shevchenko wrote:
> On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:
> >
> > From: Adam Simonelli <adamsimonelli@gmail.com>
> >
> > If CONFIG_NULL_TTY_DEFAULT_CONSOLE is enabled, and CONFIG_VT is disabled,
> > ttynull will become the default primary console device, based on the link
> > order.
> >
> > Many distributions ship with CONFIG_VT enabled. On tested desktop hardware
> > if CONFIG_VT is disabled, the default console device falls back to
> > /dev/ttyS0 instead of /dev/tty.
> >
> > This could cause issues in user space, and hardware problems:
> >
> > 1. The user space issues include the case where  /dev/ttyS0 is
> > disconnected, and the TCGETS ioctl, which some user space libraries use
> > as a probe to determine if a file is a tty, is called on /dev/console and
> > fails. Programs that call isatty() on /dev/console and get an incorrect
> > false value may skip expected logging to /dev/console.
> >
> > 2. The hardware issues include the case if a user has a science instrument
> > or other device connected to the /dev/ttyS0 port, and they were to upgrade
> > to a kernel that is disabling the CONFIG_VT option, kernel logs will then be
> > sent to the device connected to /dev/ttyS0 unless they edit their kernel
> > command line manually.
> >
> > The new CONFIG_NULL_TTY_CONSOLE option will give users and distribution
> > maintainers an option to avoid this. Disabling CONFIG_VT and enabling
> > CONFIG_NULL_TTY_CONSOLE will ensure the default kernel console behavior
> > is not dependant on hardware configuration by default, and avoid
> > unexpected new behavior on devices connected to the /dev/ttyS0 serial
> > port.
> 
> ...
> 
> Since it touches the link order only under drivers/tty the commit
> message should mention what the effect will be on the consoles drivers
> for which are located elsewhere.
> 
> 

OK, I will think of what to say. Based on what I tested in throwing a preferred
console in x86 though, it seems like ttynull was still at the top of the list
of /proc/consoles still, I just have to come up with a good way to mention that
in the commit message.

Re: [PATCH v7 2/3] ttynull: Add an option to allow ttynull to be used as a console device

[Andy Shevchenko]

On Thu, Mar 6, 2025 at 3:30 AM Adam Simonelli <adamsimonelli@gmail.com> wrote:
> On Wednesday, March 5, 2025 2:18:39 PM EST Andy Shevchenko wrote:
> > On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:

...

> > >           In order to use this driver, you should redirect the console to this
> > > -         TTY, or boot the kernel with console=ttynull.
> > > +         TTY, boot the kernel with console=ttynull, or enable
> > > +         CONFIG_NULL_TTY_DEFAULT_CONSOLE.
> >
> > I haven't checked what it looks like in menuconfig / nconfig / etc,
> > but I think that CONFIG_ is redundant here.
> >
> OK, I didn't know what one is more typical. Doing a loose grep for
> "^<tab><space><space>" and "[A-Z]_[A-Z]" (excluding CONFIG_) seems like there
> are more help text lines that mention other config options (~622) than include
> the CONFIG (174). I will change it

Thanks, it's better to follow the common practices.

> > > +         If unsure, say N.
> > > +
> > > +config NULL_TTY_DEFAULT_CONSOLE
> > > +       bool "Support for console on ttynull"
> > > +       depends on NULL_TTY=y && !VT_CONSOLE
> > > +       help
> > > +         Say Y here if you want the NULL TTY to be used as a /dev/console
> > > +         device.
> > > +
> > > +         This is similar to CONFIG_VT_CONSOLE, but without the dependency on
> > > +         CONFIG_VT. It uses the ttynull driver as the system console.
> >
> > Btw, do those `make nconfig` and friends render the options?
> Seems like `make nconfig` works, as well as `make xconfig` I will attempt to
> attach a png screenshot.

Thanks. Have you checked the search? I believe it should work with and
without CONFIG_ in the same way.

> > >           If unsure, say N.

-- 
With Best Regards,
Andy Shevchenko

Re: [PATCH v7 2/3] ttynull: Add an option to allow ttynull to be used as a console device

[Adam Simonelli]

On Thursday, March 6, 2025 2:06:14 AM EST Andy Shevchenko wrote:
> On Thu, Mar 6, 2025 at 3:30 AM Adam Simonelli <adamsimonelli@gmail.com> wrote:
> > On Wednesday, March 5, 2025 2:18:39 PM EST Andy Shevchenko wrote:
> > > On Wed, Mar 5, 2025 at 6:30 AM <adamsimonelli@gmail.com> wrote:
> 
> ...
> 
> > > >           In order to use this driver, you should redirect the console to this
> > > > -         TTY, or boot the kernel with console=ttynull.
> > > > +         TTY, boot the kernel with console=ttynull, or enable
> > > > +         CONFIG_NULL_TTY_DEFAULT_CONSOLE.
> > >
> > > I haven't checked what it looks like in menuconfig / nconfig / etc,
> > > but I think that CONFIG_ is redundant here.
> > >
> > OK, I didn't know what one is more typical. Doing a loose grep for
> > "^<tab><space><space>" and "[A-Z]_[A-Z]" (excluding CONFIG_) seems like there
> > are more help text lines that mention other config options (~622) than include
> > the CONFIG (174). I will change it
> 
> Thanks, it's better to follow the common practices.
> 
> > > > +         If unsure, say N.
> > > > +
> > > > +config NULL_TTY_DEFAULT_CONSOLE
> > > > +       bool "Support for console on ttynull"
> > > > +       depends on NULL_TTY=y && !VT_CONSOLE
> > > > +       help
> > > > +         Say Y here if you want the NULL TTY to be used as a /dev/console
> > > > +         device.
> > > > +
> > > > +         This is similar to CONFIG_VT_CONSOLE, but without the dependency on
> > > > +         CONFIG_VT. It uses the ttynull driver as the system console.
> > >
> > > Btw, do those `make nconfig` and friends render the options?
> > Seems like `make nconfig` works, as well as `make xconfig` I will attempt to
> > attach a png screenshot.
> 
> Thanks. Have you checked the search? I believe it should work with and
> without CONFIG_ in the same way.
> 
> > > >           If unsure, say N.
> 
> 
Yeah, it shows up if I search NULL_TTY_DEFAULT_CONSOLE