[PATCH] Increase maximum size of dmesg buffer to 16MB.

[PATCH] Increase maximum size of dmesg buffer to 16MB.

[Vinson Lee]

From: Nate Stahl <stahl@twitter.com>

A full task stack dump of all tasks on a machine can generate more than
4MB of output to dmesg. Dumping this data to the serial console causes
the machine to hang for a number of minutes (an unacceptable impact),
but dumping the same data to memory is feasible if the dmesg buffer is
sized large enough to hold the output. Set to 16MB which will hopefully
be large enough to handle a dump from any of our servers at this time.

Signed-off-by: Nate Stahl <stahl@twitter.com>
Signed-off-by: Vinson Lee <vlee@twitter.com>
---
 init/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/init/Kconfig b/init/Kconfig
index 903505e..266a3e8 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -812,7 +812,7 @@ config IKCONFIG_PROC
 
 config LOG_BUF_SHIFT
 	int "Kernel log buffer size (16 => 64KB, 17 => 128KB)"
-	range 12 21
+	range 12 24
 	default 17
 	depends on PRINTK
 	help
-- 
1.9.1

Re: [PATCH] Increase maximum size of dmesg buffer to 16MB.

[Luis R. Rodriguez]

On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
> From: Nate Stahl <stahl@twitter.com>
> 
> A full task stack dump of all tasks on a machine can generate more than
> 4MB of output to dmesg. Dumping this data to the serial console causes
> the machine to hang for a number of minutes (an unacceptable impact),
> but dumping the same data to memory is feasible if the dmesg buffer is
> sized large enough to hold the output. Set to 16MB which will hopefully
> be large enough to handle a dump from any of our servers at this time.
> 
> Signed-off-by: Nate Stahl <stahl@twitter.com>
> Signed-off-by: Vinson Lee <vlee@twitter.com>

Isn't this the perpetual issue of having large number of CPUs? If so
consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
should scale nicely but you can increase it as well, is it being used?

 Luis

Re: [PATCH] Increase maximum size of dmesg buffer to 16MB.

[Vinson Lee]

On Fri, Dec 12, 2014 at 2:45 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
>> From: Nate Stahl <stahl@twitter.com>
>>
>> A full task stack dump of all tasks on a machine can generate more than
>> 4MB of output to dmesg. Dumping this data to the serial console causes
>> the machine to hang for a number of minutes (an unacceptable impact),
>> but dumping the same data to memory is feasible if the dmesg buffer is
>> sized large enough to hold the output. Set to 16MB which will hopefully
>> be large enough to handle a dump from any of our servers at this time.
>>
>> Signed-off-by: Nate Stahl <stahl@twitter.com>
>> Signed-off-by: Vinson Lee <vlee@twitter.com>
>
> Isn't this the perpetual issue of having large number of CPUs? If so
> consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
> this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
> should scale nicely but you can increase it as well, is it being used?
>
>  Luis


No, it is not being used.

LOG_CPU_MAX_BUF_SHIFT is a 3.17+ config and we do not have any of the
above mentioned production machines running 3.17 or later. This patch
did help us with debugging when running on older stable kernels
though.

Thanks for the quick reply.

Re: [PATCH] Increase maximum size of dmesg buffer to 16MB.

[Luis R. Rodriguez]

On Fri, Dec 12, 2014 at 03:07:23PM -0800, Vinson Lee wrote:
> On Fri, Dec 12, 2014 at 2:45 PM, Luis R. Rodriguez <mcgrof@suse.com> wrote:
> > On Fri, Dec 12, 2014 at 02:38:26PM -0800, Vinson Lee wrote:
> >> From: Nate Stahl <stahl@twitter.com>
> >>
> >> A full task stack dump of all tasks on a machine can generate more than
> >> 4MB of output to dmesg. Dumping this data to the serial console causes
> >> the machine to hang for a number of minutes (an unacceptable impact),
> >> but dumping the same data to memory is feasible if the dmesg buffer is
> >> sized large enough to hold the output. Set to 16MB which will hopefully
> >> be large enough to handle a dump from any of our servers at this time.
> >>
> >> Signed-off-by: Nate Stahl <stahl@twitter.com>
> >> Signed-off-by: Vinson Lee <vlee@twitter.com>
> >
> > Isn't this the perpetual issue of having large number of CPUs? If so
> > consider use of LOG_CPU_MAX_BUF_SHIFT instead, otherwise clarifying how
> > this would be a different issue would be good. LOG_CPU_MAX_BUF_SHIFT
> > should scale nicely but you can increase it as well, is it being used?
> >
> >  Luis
> 
> 
> No, it is not being used.
> 
> LOG_CPU_MAX_BUF_SHIFT is a 3.17+ config and we do not have any of the
> above mentioned production machines running 3.17 or later.

Then consider backporting it.

> This patch did help us with debugging when running on older stable kernels
> though.

Understood, give the other stuff a spin.

  Luis