How to display dom0 kernel printk on hvc0

How to display dom0 kernel printk on hvc0

[manish jaggi]

[-- Attachment #1.1: Type: text/plain, Size: 470 bytes --]

Hi Stefano,

I am debugging a kernel crash when I start xencommons. If I add prints in
kernel the prints are only visible in dmesg and not on dom0 console. This
makes it difficult to view kernel printks at the time of crash.

I tried xen_raw_printks but the print only comes when you press a key, so
while a crash is happening you cannot press a key.

Is there a way to enable printks on hvc0
OR
is there a way to dump __logbuf of linux from xen shell.

-Regards
Manish

[-- Attachment #1.2: Type: text/html, Size: 639 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to display dom0 kernel printk on hvc0

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 2696 bytes --]

On Thu, 7 Aug 2014, manish jaggi wrote:
> Hi Stefano,
> 
> I am debugging a kernel crash when I start xencommons. If I add prints in kernel the prints are only visible in
> dmesg and not on dom0 console. This makes it difficult to view kernel printks at the time of crash.
> 
> I tried xen_raw_printks but the print only comes when you press a key, so while a crash is happening you cannot
> press a key.
> 
> Is there a way to enable printks on hvc0 
> OR
> is there a way to dump __logbuf of linux from xen shell.

You can hack printk and/or early_printk so that everything will go via
the Xen console, see below. Of course this is very hacky, don't tell
anybody that I wrote this patch ;-)


diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c
index 2dc36d0..38bcf29 100644
--- a/arch/arm64/kernel/early_printk.c
+++ b/arch/arm64/kernel/early_printk.c
@@ -31,16 +31,16 @@
 static void __iomem *early_base;
 static void (*printch)(char ch);
 
+void xen_raw_console_write(const char *str);
 /*
  * PL011 single character TX.
  */
 static void pl011_printch(char ch)
 {
-	while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_TXFF)
-		;
-	writeb_relaxed(ch, early_base + UART01x_DR);
-	while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_BUSY)
-		;
+	char buf[2];
+	buf[0] = ch;
+	buf[1] = '\0';
+	xen_raw_console_write(buf);
 }
 
 /*
diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
index 2dc2831..1f586b3 100644
--- a/drivers/tty/hvc/hvc_xen.c
+++ b/drivers/tty/hvc/hvc_xen.c
@@ -630,7 +630,7 @@ void xen_raw_console_write(const char *str)
 	ssize_t len = strlen(str);
 	int rc = 0;
 
-	if (xen_domain()) {
+	if (1||xen_domain()) {
 		rc = dom0_write_console(0, str, len);
 #ifdef CONFIG_X86
 		if (rc == -ENOSYS && xen_hvm_domain())
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 7228258..e035871 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1674,24 +1674,18 @@ EXPORT_SYMBOL(printk_emit);
  *
  * See the vsnprintf() documentation for format string extensions over C99.
  */
+void xen_raw_console_write(const char *str);
 asmlinkage __visible int printk(const char *fmt, ...)
 {
 	va_list args;
-	int r;
+	static char buf[512];
 
-#ifdef CONFIG_KGDB_KDB
-	if (unlikely(kdb_trap_printk)) {
-		va_start(args, fmt);
-		r = vkdb_printf(fmt, args);
-		va_end(args);
-		return r;
-	}
-#endif
 	va_start(args, fmt);
-	r = vprintk_emit(0, -1, NULL, 0, fmt, args);
+	vsnprintf(buf, sizeof(buf), fmt, args);
 	va_end(args);
 
-	return r;
+	xen_raw_console_write(buf);
+	return 1;
 }
 EXPORT_SYMBOL(printk);
 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to display dom0 kernel printk on hvc0

[manish jaggi]

[-- Attachment #1.1: Type: text/plain, Size: 3957 bytes --]

Thanks,
I manged to do something similar in the meantime. I am seeing a crash after
I do /etc/init.d/xencommns start

[<ffffffc00038fee4>] clear_bit+0x14/0x30
[<ffffffc0003d9ca4>] ack_dynirq+0x44/0x58
[<ffffffc0000e6a34>] handle_edge_irq+0x74/0x178
[<ffffffc0003dc0e8>] evtchn_fifo_handle_events+0x280/0x288
[<ffffffc0003d8f50>] __xen_evtchn_do_upcall+0x68/0xd0
[<ffffffc0003d8fc0>] xen_hvm_evtchn_do_upcall+0x8/0x18
[<ffffffc00009271c>] xen_arm_callback+0x4c/0x68
[<ffffffc0000e7560>] handle_percpu_devid_irq+0x88/0x120
[<ffffffc0000e38b4>] generic_handle_irq+0x24/0x40
[<ffffffc000084890>] handle_IRQ+0x40/0xa8
[<ffffffc000081348>] gic_handle_irq+0x50/0xa0

I found that consume_one_event calls handle_irq_for_port which gets IRQ=7
in case of a crash.
What is the use of IRQ 7 ?
IRQ1 is UART which i saw in cat /proc/interrupts

-Regards
Manish


On 7 August 2014 18:14, Stefano Stabellini <stefano.stabellini@eu.citrix.com
> wrote:

> On Thu, 7 Aug 2014, manish jaggi wrote:
> > Hi Stefano,
> >
> > I am debugging a kernel crash when I start xencommons. If I add prints
> in kernel the prints are only visible in
> > dmesg and not on dom0 console. This makes it difficult to view kernel
> printks at the time of crash.
> >
> > I tried xen_raw_printks but the print only comes when you press a key,
> so while a crash is happening you cannot
> > press a key.
> >
> > Is there a way to enable printks on hvc0
> > OR
> > is there a way to dump __logbuf of linux from xen shell.
>
> You can hack printk and/or early_printk so that everything will go via
> the Xen console, see below. Of course this is very hacky, don't tell
> anybody that I wrote this patch ;-)
>
>
> diff --git a/arch/arm64/kernel/early_printk.c
> b/arch/arm64/kernel/early_printk.c
> index 2dc36d0..38bcf29 100644
> --- a/arch/arm64/kernel/early_printk.c
> +++ b/arch/arm64/kernel/early_printk.c
> @@ -31,16 +31,16 @@
>  static void __iomem *early_base;
>  static void (*printch)(char ch);
>
> +void xen_raw_console_write(const char *str);
>  /*
>   * PL011 single character TX.
>   */
>  static void pl011_printch(char ch)
>  {
> -       while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_TXFF)
> -               ;
> -       writeb_relaxed(ch, early_base + UART01x_DR);
> -       while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_BUSY)
> -               ;
> +       char buf[2];
> +       buf[0] = ch;
> +       buf[1] = '\0';
> +       xen_raw_console_write(buf);
>  }
>
>  /*
> diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
> index 2dc2831..1f586b3 100644
> --- a/drivers/tty/hvc/hvc_xen.c
> +++ b/drivers/tty/hvc/hvc_xen.c
> @@ -630,7 +630,7 @@ void xen_raw_console_write(const char *str)
>         ssize_t len = strlen(str);
>         int rc = 0;
>
> -       if (xen_domain()) {
> +       if (1||xen_domain()) {
>                 rc = dom0_write_console(0, str, len);
>  #ifdef CONFIG_X86
>                 if (rc == -ENOSYS && xen_hvm_domain())
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 7228258..e035871 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -1674,24 +1674,18 @@ EXPORT_SYMBOL(printk_emit);
>   *
>   * See the vsnprintf() documentation for format string extensions over
> C99.
>   */
> +void xen_raw_console_write(const char *str);
>  asmlinkage __visible int printk(const char *fmt, ...)
>  {
>         va_list args;
> -       int r;
> +       static char buf[512];
>
> -#ifdef CONFIG_KGDB_KDB
> -       if (unlikely(kdb_trap_printk)) {
> -               va_start(args, fmt);
> -               r = vkdb_printf(fmt, args);
> -               va_end(args);
> -               return r;
> -       }
> -#endif
>         va_start(args, fmt);
> -       r = vprintk_emit(0, -1, NULL, 0, fmt, args);
> +       vsnprintf(buf, sizeof(buf), fmt, args);
>         va_end(args);
>
> -       return r;
> +       xen_raw_console_write(buf);
> +       return 1;
>  }
>  EXPORT_SYMBOL(printk);
>

[-- Attachment #1.2: Type: text/html, Size: 4990 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to display dom0 kernel printk on hvc0

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 5052 bytes --]

On Thu, 7 Aug 2014, manish jaggi wrote:
> Thanks,
> I manged to do something similar in the meantime. I am seeing a crash after I do /etc/init.d/xencommns start
> 
> [<ffffffc00038fee4>] clear_bit+0x14/0x30
> [<ffffffc0003d9ca4>] ack_dynirq+0x44/0x58
> [<ffffffc0000e6a34>] handle_edge_irq+0x74/0x178
> [<ffffffc0003dc0e8>] evtchn_fifo_handle_events+0x280/0x288
> [<ffffffc0003d8f50>] __xen_evtchn_do_upcall+0x68/0xd0
> [<ffffffc0003d8fc0>] xen_hvm_evtchn_do_upcall+0x8/0x18
> [<ffffffc00009271c>] xen_arm_callback+0x4c/0x68
> [<ffffffc0000e7560>] handle_percpu_devid_irq+0x88/0x120
> [<ffffffc0000e38b4>] generic_handle_irq+0x24/0x40
> [<ffffffc000084890>] handle_IRQ+0x40/0xa8
> [<ffffffc000081348>] gic_handle_irq+0x50/0xa0
> 
> I found that consume_one_event calls handle_irq_for_port which gets IRQ=7 in case of a crash.
> What is the use of IRQ 7 ?
> IRQ1 is UART which i saw in cat /proc/interrupts

What kernel version are you using?

It looks like the FIFO event channel is not properly initialized.
You could try switching to the old style 2-level ABI:

diff --git a/drivers/xen/events/events_fifo.c b/drivers/xen/events/events_fifo.c
index 84b4bfb..4a23e08 100644
--- a/drivers/xen/events/events_fifo.c
+++ b/drivers/xen/events/events_fifo.c
@@ -428,6 +428,8 @@ int __init xen_evtchn_fifo_init(void)
 	int cpu = get_cpu();
 	int ret;
 
+	return -1;
+
 	ret = evtchn_fifo_init_control_block(cpu);
 	if (ret < 0)
 		goto out;

You should then see this message at boot:

xen:events: Using 2-level ABI



> -Regards
> Manish
> 
> 
> On 7 August 2014 18:14, Stefano Stabellini <stefano.stabellini@eu.citrix.com> wrote:
>       On Thu, 7 Aug 2014, manish jaggi wrote:
>       > Hi Stefano,
>       >
>       > I am debugging a kernel crash when I start xencommons. If I add prints in kernel the prints are
>       only visible in
>       > dmesg and not on dom0 console. This makes it difficult to view kernel printks at the time of
>       crash.
>       >
>       > I tried xen_raw_printks but the print only comes when you press a key, so while a crash is
>       happening you cannot
>       > press a key.
>       >
>       > Is there a way to enable printks on hvc0 
>       > OR
>       > is there a way to dump __logbuf of linux from xen shell.
> 
> You can hack printk and/or early_printk so that everything will go via
> the Xen console, see below. Of course this is very hacky, don't tell
> anybody that I wrote this patch ;-)
> 
> 
> diff --git a/arch/arm64/kernel/early_printk.c b/arch/arm64/kernel/early_printk.c
> index 2dc36d0..38bcf29 100644
> --- a/arch/arm64/kernel/early_printk.c
> +++ b/arch/arm64/kernel/early_printk.c
> @@ -31,16 +31,16 @@
>  static void __iomem *early_base;
>  static void (*printch)(char ch);
> 
> +void xen_raw_console_write(const char *str);
>  /*
>   * PL011 single character TX.
>   */
>  static void pl011_printch(char ch)
>  {
> -       while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_TXFF)
> -               ;
> -       writeb_relaxed(ch, early_base + UART01x_DR);
> -       while (readl_relaxed(early_base + UART01x_FR) & UART01x_FR_BUSY)
> -               ;
> +       char buf[2];
> +       buf[0] = ch;
> +       buf[1] = '\0';
> +       xen_raw_console_write(buf);
>  }
> 
>  /*
> diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
> index 2dc2831..1f586b3 100644
> --- a/drivers/tty/hvc/hvc_xen.c
> +++ b/drivers/tty/hvc/hvc_xen.c
> @@ -630,7 +630,7 @@ void xen_raw_console_write(const char *str)
>         ssize_t len = strlen(str);
>         int rc = 0;
> 
> -       if (xen_domain()) {
> +       if (1||xen_domain()) {
>                 rc = dom0_write_console(0, str, len);
>  #ifdef CONFIG_X86
>                 if (rc == -ENOSYS && xen_hvm_domain())
> diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
> index 7228258..e035871 100644
> --- a/kernel/printk/printk.c
> +++ b/kernel/printk/printk.c
> @@ -1674,24 +1674,18 @@ EXPORT_SYMBOL(printk_emit);
>   *
>   * See the vsnprintf() documentation for format string extensions over C99.
>   */
> +void xen_raw_console_write(const char *str);
>  asmlinkage __visible int printk(const char *fmt, ...)
>  {
>         va_list args;
> -       int r;
> +       static char buf[512];
> 
> -#ifdef CONFIG_KGDB_KDB
> -       if (unlikely(kdb_trap_printk)) {
> -               va_start(args, fmt);
> -               r = vkdb_printf(fmt, args);
> -               va_end(args);
> -               return r;
> -       }
> -#endif
>         va_start(args, fmt);
> -       r = vprintk_emit(0, -1, NULL, 0, fmt, args);
> +       vsnprintf(buf, sizeof(buf), fmt, args);
>         va_end(args);
> 
> -       return r;
> +       xen_raw_console_write(buf);
> +       return 1;
>  }
>  EXPORT_SYMBOL(printk);
>  
> 
> 
> 
> 

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to display dom0 kernel printk on hvc0

[Julien Grall]

On 08/07/2014 03:36 PM, Stefano Stabellini wrote:
> On Thu, 7 Aug 2014, manish jaggi wrote:
>> Thanks,
>> I manged to do something similar in the meantime. I am seeing a crash after I do /etc/init.d/xencommns start
>>
>> [<ffffffc00038fee4>] clear_bit+0x14/0x30
>> [<ffffffc0003d9ca4>] ack_dynirq+0x44/0x58
>> [<ffffffc0000e6a34>] handle_edge_irq+0x74/0x178
>> [<ffffffc0003dc0e8>] evtchn_fifo_handle_events+0x280/0x288
>> [<ffffffc0003d8f50>] __xen_evtchn_do_upcall+0x68/0xd0
>> [<ffffffc0003d8fc0>] xen_hvm_evtchn_do_upcall+0x8/0x18
>> [<ffffffc00009271c>] xen_arm_callback+0x4c/0x68
>> [<ffffffc0000e7560>] handle_percpu_devid_irq+0x88/0x120
>> [<ffffffc0000e38b4>] generic_handle_irq+0x24/0x40
>> [<ffffffc000084890>] handle_IRQ+0x40/0xa8
>> [<ffffffc000081348>] gic_handle_irq+0x50/0xa0
>>
>> I found that consume_one_event calls handle_irq_for_port which gets IRQ=7 in case of a crash.
>> What is the use of IRQ 7 ?
>> IRQ1 is UART which i saw in cat /proc/interrupts
> 
> What kernel version are you using?
> 
> It looks like the FIFO event channel is not properly initialized.
> You could try switching to the old style 2-level ABI:
> 
> diff --git a/drivers/xen/events/events_fifo.c b/drivers/xen/events/events_fifo.c
> index 84b4bfb..4a23e08 100644
> --- a/drivers/xen/events/events_fifo.c
> +++ b/drivers/xen/events/events_fifo.c
> @@ -428,6 +428,8 @@ int __init xen_evtchn_fifo_init(void)
>  	int cpu = get_cpu();
>  	int ret;
>  
> +	return -1;
> +
>  	ret = evtchn_fifo_init_control_block(cpu);
>  	if (ret < 0)
>  		goto out;

You don't need to recompile the kernel.

xen.fifo_events=0 in the command line will choose the 2-level ABI.

Regards,

-- 
Julien Grall

Re: How to display dom0 kernel printk on hvc0

[David Vrabel]

On 07/08/14 15:36, Stefano Stabellini wrote:
> On Thu, 7 Aug 2014, manish jaggi wrote:
>> Thanks,
>> I manged to do something similar in the meantime. I am seeing a crash after I do /etc/init.d/xencommns start
>>
>> [<ffffffc00038fee4>] clear_bit+0x14/0x30
>> [<ffffffc0003d9ca4>] ack_dynirq+0x44/0x58
>> [<ffffffc0000e6a34>] handle_edge_irq+0x74/0x178
>> [<ffffffc0003dc0e8>] evtchn_fifo_handle_events+0x280/0x288
>> [<ffffffc0003d8f50>] __xen_evtchn_do_upcall+0x68/0xd0
>> [<ffffffc0003d8fc0>] xen_hvm_evtchn_do_upcall+0x8/0x18
>> [<ffffffc00009271c>] xen_arm_callback+0x4c/0x68
>> [<ffffffc0000e7560>] handle_percpu_devid_irq+0x88/0x120
>> [<ffffffc0000e38b4>] generic_handle_irq+0x24/0x40
>> [<ffffffc000084890>] handle_IRQ+0x40/0xa8
>> [<ffffffc000081348>] gic_handle_irq+0x50/0xa0
>>
>> I found that consume_one_event calls handle_irq_for_port which gets IRQ=7 in case of a crash.
>> What is the use of IRQ 7 ?
>> IRQ1 is UART which i saw in cat /proc/interrupts
> 
> What kernel version are you using?

On arm64, 3.14 was buggy.  This was fixed in 3.15-rc6.

> 
> It looks like the FIFO event channel is not properly initialized.
> You could try switching to the old style 2-level ABI:

There's a kernel command line options for this:  xen.fifo_events=0

David

Re: How to display dom0 kernel printk on hvc0

[manish jaggi]

[-- Attachment #1.1: Type: text/plain, Size: 1474 bytes --]

On 7 August 2014 20:31, David Vrabel <david.vrabel@citrix.com> wrote:

> On 07/08/14 15:36, Stefano Stabellini wrote:
> > On Thu, 7 Aug 2014, manish jaggi wrote:
> >> Thanks,
> >> I manged to do something similar in the meantime. I am seeing a crash
> after I do /etc/init.d/xencommns start
> >>
> >> [<ffffffc00038fee4>] clear_bit+0x14/0x30
> >> [<ffffffc0003d9ca4>] ack_dynirq+0x44/0x58
> >> [<ffffffc0000e6a34>] handle_edge_irq+0x74/0x178
> >> [<ffffffc0003dc0e8>] evtchn_fifo_handle_events+0x280/0x288
> >> [<ffffffc0003d8f50>] __xen_evtchn_do_upcall+0x68/0xd0
> >> [<ffffffc0003d8fc0>] xen_hvm_evtchn_do_upcall+0x8/0x18
> >> [<ffffffc00009271c>] xen_arm_callback+0x4c/0x68
> >> [<ffffffc0000e7560>] handle_percpu_devid_irq+0x88/0x120
> >> [<ffffffc0000e38b4>] generic_handle_irq+0x24/0x40
> >> [<ffffffc000084890>] handle_IRQ+0x40/0xa8
> >> [<ffffffc000081348>] gic_handle_irq+0x50/0xa0
> >>
> >> I found that consume_one_event calls handle_irq_for_port which gets
> IRQ=7 in case of a crash.
> >> What is the use of IRQ 7 ?
> >> IRQ1 is UART which i saw in cat /proc/interrupts
> >
> > What kernel version are you using?
>
> On arm64, 3.14 was buggy.  This was fixed in 3.15-rc6.
>
Can you please elaborate your observations, it will help in my case

>
> >
> > It looks like the FIFO event channel is not properly initialized.
> > You could try switching to the old style 2-level ABI:
>
> There's a kernel command line options for this:  xen.fifo_events=0
>
> David
>

[-- Attachment #1.2: Type: text/html, Size: 2322 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: How to display dom0 kernel printk on hvc0

[Julien Grall]

Hello,

On 08/07/2014 09:23 AM, manish jaggi wrote:
> I am debugging a kernel crash when I start xencommons. If I add prints
> in kernel the prints are only visible in dmesg and not on dom0 console.
> This makes it difficult to view kernel printks at the time of crash.
> 
> I tried xen_raw_printks but the print only comes when you press a key,
> so while a crash is happening you cannot press a key.
> 
> Is there a way to enable printks on hvc0 

every printk in the kernel should be printed in hvc if you pass
console=hvc0 on the kernel command line.

You may also have to add ignore_loglevel

Regards,

-- 
Julien Grall