Re: [PATCH 1/3] printk: Introduce per-console loglevel setting

Re: [PATCH 1/3] printk: Introduce per-console loglevel setting

[Sergey Senozhatsky]

On (09/28/17 17:43), Calvin Owens wrote:
> Not all consoles are created equal: depending on the actual hardware,
> the latency of a printk() call can vary dramatically. The worst examples
> are serial consoles, where it can spin for tens of milliseconds banging
> the UART to emit a message, which can cause application-level problems
> when the kernel spews onto the console.
> 
> At Facebook we use netconsole to monitor our fleet, but we still have
> serial consoles attached on each host for live debugging, and the latter
> has caused problems. An obvious solution is to disable the kernel
> console output to ttyS0, but this makes live debugging frustrating,
> since crashes become silent and opaque to the ttyS0 user. Enabling it on
> the fly when needed isn't feasible, since boxes you need to debug via
> serial are likely to be borked in ways that make this impossible.
> 
> That puts us between a rock and a hard place: we'd love to set
> kernel.printk to KERN_INFO and get all the logs. But while netconsole is
> fast enough to permit that without perturbing userspace, ttyS0 is not,
> and we're forced to limit console logging to KERN_WARNING and higher.
> 
> This patch introduces a new per-console loglevel setting, and changes
> console_unlock() to use max(global_level, per_console_level) when
> deciding whether or not to emit a given log message.
> 
> This lets us have our cake and eat it too: instead of being forced to
> limit all consoles verbosity based on the speed of the slowest one, we
> can "promote" the faster console while still using a conservative system
> loglevel setting to avoid disturbing applications.

Hi Calvin,

Do you have time to address the review feedback and re-spin v2?

	-ss

Re: [PATCH 1/3] printk: Introduce per-console loglevel setting

[Calvin Owens]

On Friday 10/19 at 09:04 +0900, Sergey Senozhatsky wrote:
> On (09/28/17 17:43), Calvin Owens wrote:
> > Not all consoles are created equal: depending on the actual hardware,
> > the latency of a printk() call can vary dramatically. The worst examples
> > are serial consoles, where it can spin for tens of milliseconds banging
> > the UART to emit a message, which can cause application-level problems
> > when the kernel spews onto the console.
> > 
> > At Facebook we use netconsole to monitor our fleet, but we still have
> > serial consoles attached on each host for live debugging, and the latter
> > has caused problems. An obvious solution is to disable the kernel
> > console output to ttyS0, but this makes live debugging frustrating,
> > since crashes become silent and opaque to the ttyS0 user. Enabling it on
> > the fly when needed isn't feasible, since boxes you need to debug via
> > serial are likely to be borked in ways that make this impossible.
> > 
> > That puts us between a rock and a hard place: we'd love to set
> > kernel.printk to KERN_INFO and get all the logs. But while netconsole is
> > fast enough to permit that without perturbing userspace, ttyS0 is not,
> > and we're forced to limit console logging to KERN_WARNING and higher.
> > 
> > This patch introduces a new per-console loglevel setting, and changes
> > console_unlock() to use max(global_level, per_console_level) when
> > deciding whether or not to emit a given log message.
> > 
> > This lets us have our cake and eat it too: instead of being forced to
> > limit all consoles verbosity based on the speed of the slowest one, we
> > can "promote" the faster console while still using a conservative system
> > loglevel setting to avoid disturbing applications.
> 
> Hi Calvin,
> 
> Do you have time to address the review feedback and re-spin v2?

Hi Sergey,

It's in-progress, I'm sorry it hasn't happened sooner.

By embedding the kobject in the console struct, we end up needing to refactor
the console drivers to use the kobject refcount instead of simply calling
kfree(), which is what I'm working on. It ends up being tedious but not
particularly complicated, my goal is to have this up soon :)

Thanks,
Calvin

Re: [PATCH 1/3] printk: Introduce per-console loglevel setting

[Sergey Senozhatsky]

On (10/19/18 22:03), Calvin Owens wrote:
> Hi Sergey,
> 
> It's in-progress, I'm sorry it hasn't happened sooner.

Great!

> By embedding the kobject in the console struct, we end up needing to refactor
> the console drivers to use the kobject refcount instead of simply calling
> kfree(), which is what I'm working on. It ends up being tedious but not
> particularly complicated, my goal is to have this up soon :)

OK :)

	-ss