Support for netconsole as default tty/console?

Support for netconsole as default tty/console?

[Struan Bartlett]

Hi

I help maintain a range of raw metal hosts, VMs, and Amazon EC2
instances that boot Linux within secure private networks. Where
possible, we connect raw metal hosts to a central logging server using
serial lines, and use the console=ttyS0 kernel command line option to
redirect not just kernel boot logs, but all boot script tty output,
via the serial line for central logging. This is particularly useful,
as when critical start-up scripts report issues to the console (as
opposed to logs files) those issues can be monitored centrally, and
diagnosed without the need to log in. In some cases, critical error
messages are only logged to the console, and not to the logs files.

Serial lines do not scale well and, on some of our VMs, and certainly
on EC2 instances, platforms are headless and may be accessed only by
our private network. As mentioned, our systems are running in known
secure private networks accessible only to sysadmins, so
authenticating access to the terminal output is not a particular
concern.

We have had a good experience of using netconsole to provide remote
real-time access to the kernel logs during Linux boot, via our syslog
server. However we have yet to find a reliable way to redirect all
console output (i.e. start-up script not just kernel logs) via the
private network to our logging server, as is possible via a serial
line. We have tried to address the issue in user-space. Patching of
boot scripts to pipe output via tee and socat has been attempted, but
it is hard to make this work comprehensively, and it appears to
interfere with standard shutdown scripts that expect all processes to
be able to be killed (without terminating standard output/error pipes)
before switching run levels.

Adding console=netconsole to the command line does not appear to have
the desired effect. I am not sure if this is because netconsole,
unlike the serial console, does not provide a tty. Can anyone advise
if this understanding is correct? If so, is there an independent
solution to this problem? Failing that, is there general interest in a
tty-capable (maybe output only) extension to the netconsole module?

Kind regards

Re: Support for netconsole as default tty/console?

[Andreas Schwab]

Struan Bartlett <struan.bartlett@gmail.com> writes:

> Adding console=netconsole to the command line does not appear to have
> the desired effect. I am not sure if this is because netconsole,
> unlike the serial console, does not provide a tty. Can anyone advise
> if this understanding is correct? If so, is there an independent
> solution to this problem?

It should not be hard to add a tty driver to netconsole, there are many
examples to borrow from.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Support for netconsole as default tty/console?

[Struan Bartlett]

On 29 March 2014 15:54, Andreas Schwab <schwab@linux-m68k.org> wrote:
> Struan Bartlett <struan.bartlett@gmail.com> writes:
>
>> Adding console=netconsole to the command line does not appear to have
>> the desired effect. I am not sure if this is because netconsole,
>> unlike the serial console, does not provide a tty. Can anyone advise
>> if this understanding is correct? If so, is there an independent
>> solution to this problem?
>
> It should not be hard to add a tty driver to netconsole, there are many
> examples to borrow from.

Thank you. I've begun investigating the sources in drivers/tty. Am I
correct in thinking that this only needs a tty driver, and not also a
serial driver?

Re: Support for netconsole as default tty/console?

[Andreas Schwab]

Struan Bartlett <struan.bartlett@gmail.com> writes:

> Thank you. I've begun investigating the sources in drivers/tty. Am I
> correct in thinking that this only needs a tty driver, and not also a
> serial driver?

See arch/m68k/emu/nfcon.c for a somewhat minimal example (only output).

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Support for netconsole as default tty/console?

[One Thousand Gnomes]

On Sat, 29 Mar 2014 15:54:24 +0100
Andreas Schwab <schwab@linux-m68k.org> wrote:

> Struan Bartlett <struan.bartlett@gmail.com> writes:
> 
> > Adding console=netconsole to the command line does not appear to have
> > the desired effect. I am not sure if this is because netconsole,
> > unlike the serial console, does not provide a tty. Can anyone advise
> > if this understanding is correct? If so, is there an independent
> > solution to this problem?
> 
> It should not be hard to add a tty driver to netconsole, there are many
> examples to borrow from.

Samo Pogacnik added a generic solution a few years ago. The ttyprintk
driver provides you with a tty/console whose 'hardware' is printk and
thus whatever system log device you are using.

Alan

Re: Support for netconsole as default tty/console?

[Andreas Schwab]

One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> writes:

> Samo Pogacnik added a generic solution a few years ago. The ttyprintk
> driver provides you with a tty/console whose 'hardware' is printk and
> thus whatever system log device you are using.

Unfortunately it's an EXPERT setting, thus typically not avaiable.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Support for netconsole as default tty/console?

[One Thousand Gnomes]

On Mon, 31 Mar 2014 12:26:35 +0200
Andreas Schwab <schwab@suse.de> wrote:

> One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> writes:
> 
> > Samo Pogacnik added a generic solution a few years ago. The ttyprintk
> > driver provides you with a tty/console whose 'hardware' is printk and
> > thus whatever system log device you are using.
> 
> Unfortunately it's an EXPERT setting, thus typically not avaiable.

It's however implemented and present. The rest is between you and the
relevant Kconfig maintainer. I'm sure you count as an expert ;-)

(and perhaps moving it into drivers/tty would also be sensible).

Alan

Re: Support for netconsole as default tty/console?

[Andreas Schwab]

One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> writes:

> It's however implemented and present. The rest is between you and the
> relevant Kconfig maintainer. I'm sure you count as an expert ;-)

Indeed.

http://lists.opensuse.org/opensuse-kernel/2014-03/msg00071.html

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Support for netconsole as default tty/console?

[Struan Bartlett]

On 31 March 2014 14:34, Andreas Schwab <schwab@suse.de> wrote:
> One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> writes:
>
>> It's however implemented and present. The rest is between you and the
>> relevant Kconfig maintainer. I'm sure you count as an expert ;-)
>
> Indeed.
>
> http://lists.opensuse.org/opensuse-kernel/2014-03/msg00071.html

I have just completed an implementation of a tty driver for
netconsole, which I was about to submit as a patch. When the kernel
command line is set accordingly (console=netcon0), it has the desired
effect of taking over output to /dev/console when it is loaded. Would
you say this work is wasted, or that it's worth submitting the patch
anyway?

There are differences to the ttyprintk approach. ttyprintk is
currently kernel-only, and not available as a module. ttyprintk adds
special formatting to the output, which netconsole does not. (If
netconsole were to provide special formatting, it might make sense for
it to be syslog-compatible). In defence of netconsole providing a tty
driver, many other output device modules already provide their own tty
(i.e. do not rely on ttyprintk). Also, unlike ttyprintk, netconsole is
already widely used and well documented on the web. Adding a tty
driver to netconsole would be a natural extension of functionality for
those already using it.

Re: Support for netconsole as default tty/console?

[Takashi Iwai]

At Tue, 1 Apr 2014 10:34:27 +0200,
Struan Bartlett wrote:
> 
> On 31 March 2014 14:34, Andreas Schwab <schwab@suse.de> wrote:
> > One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> writes:
> >
> >> It's however implemented and present. The rest is between you and the
> >> relevant Kconfig maintainer. I'm sure you count as an expert ;-)
> >
> > Indeed.
> >
> > http://lists.opensuse.org/opensuse-kernel/2014-03/msg00071.html
> 
> I have just completed an implementation of a tty driver for
> netconsole, which I was about to submit as a patch. When the kernel
> command line is set accordingly (console=netcon0), it has the desired
> effect of taking over output to /dev/console when it is loaded. Would
> you say this work is wasted, or that it's worth submitting the patch
> anyway?
> 
> There are differences to the ttyprintk approach. ttyprintk is
> currently kernel-only, and not available as a module.

Actually, this can be easily fixed.  I'm going to submit a patch (and
another fix patch for the driver's error path).

Note that they don't mean to object against your patching netconsole.
I find also it more convenient to use directly netconsole.


thanks,

Takashi

Re: Support for netconsole as default tty/console?

[Arkadiusz Miskiewicz]

On Monday 31 of March 2014, One Thousand Gnomes wrote:
> On Sat, 29 Mar 2014 15:54:24 +0100
> 
> Andreas Schwab <schwab@linux-m68k.org> wrote:
> > Struan Bartlett <struan.bartlett@gmail.com> writes:
> > > Adding console=netconsole to the command line does not appear to have
> > > the desired effect. I am not sure if this is because netconsole,
> > > unlike the serial console, does not provide a tty. Can anyone advise
> > > if this understanding is correct? If so, is there an independent
> > > solution to this problem?
> > 
> > It should not be hard to add a tty driver to netconsole, there are many
> > examples to borrow from.
> 
> Samo Pogacnik added a generic solution a few years ago. The ttyprintk
> driver provides you with a tty/console whose 'hardware' is printk and
> thus whatever system log device you are using.

Is there a way to get two devices working as console? For example I would like 
to have console on tty0 and on netconsole at the same time. Right now only 
one, last console gets all userspace messages.

> Alan

-- 
Arkadiusz Miśkiewicz, arekm / maven.pl