Question on hvc console init

Question on hvc console init

[Iurii Konovalenko]

Hello, all!

I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
Xen revision is 4.4.
I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
In kernel I've found strange behaviour in hvc console init function.
In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:

    if (!xen_domain())
        return 0;

    if (xen_initial_domain())
        ops = &dom0_hvc_ops;
    else {

xen_domain() and xen_initial_domain() are defined to check
xen_domain_type variable. This variable is defined and initialized to
XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
variable is set in same file function xen_guest_init(), that is
early_initcall. But eraly_initcall is called later, than
console_initcall, that's why in time of running xen_cons_init(void)
xen_domain_type is not initialized to correct value and
xen_cons_init() does not initialize console, as returns on first check
"if (!xen_domain())".
It is not critical in normal operation, because we have
device_initcall xen_hvc_init() that is called after xen_guest_init(),
it initialize hvc. But in case of kernel falls before
device_initcall's, we can't see any printouts, that could be useful.

Could you please explain, may be using some configs or arguments in
command line for kernel, how to enable this early console?

Best regards.

Iurii Konovalenko | Senior Software Engineer
GlobalLogic

Re: Question on hvc console init

[Konrad Rzeszutek Wilk]

On Tue, Oct 28, 2014 at 05:53:25PM +0200, Iurii Konovalenko wrote:
> Hello, all!
> 
> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
> Xen revision is 4.4.
> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
> In kernel I've found strange behaviour in hvc console init function.
> In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:
> 
>     if (!xen_domain())
>         return 0;
> 
>     if (xen_initial_domain())
>         ops = &dom0_hvc_ops;
>     else {
> 
> xen_domain() and xen_initial_domain() are defined to check
> xen_domain_type variable. This variable is defined and initialized to
> XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
> variable is set in same file function xen_guest_init(), that is
> early_initcall. But eraly_initcall is called later, than
> console_initcall, that's why in time of running xen_cons_init(void)
> xen_domain_type is not initialized to correct value and
> xen_cons_init() does not initialize console, as returns on first check
> "if (!xen_domain())".
> It is not critical in normal operation, because we have
> device_initcall xen_hvc_init() that is called after xen_guest_init(),
> it initialize hvc. But in case of kernel falls before
> device_initcall's, we can't see any printouts, that could be useful.
> 
> Could you please explain, may be using some configs or arguments in
> command line for kernel, how to enable this early console?


It is explained in the kernel-parameters.txt (https://www.kernel.org/doc/Documentation/kernel-parameters.txt)
:

earlyprintk=	[X86,SH,BLACKFIN,ARM,M68k]
			earlyprintk=vga
			earlyprintk=efi
			earlyprintk=xen
...
> 
> Best regards.
> 
> Iurii Konovalenko | Senior Software Engineer
> GlobalLogic
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: Question on hvc console init

[Julien Grall]

Hi Konrad,

On 10/28/2014 04:53 PM, Konrad Rzeszutek Wilk wrote:
> On Tue, Oct 28, 2014 at 05:53:25PM +0200, Iurii Konovalenko wrote:
>> Hello, all!
>>
>> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
>> Xen revision is 4.4.
>> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
>> In kernel I've found strange behaviour in hvc console init function.
>> In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:
>>
>>     if (!xen_domain())
>>         return 0;
>>
>>     if (xen_initial_domain())
>>         ops = &dom0_hvc_ops;
>>     else {
>>
>> xen_domain() and xen_initial_domain() are defined to check
>> xen_domain_type variable. This variable is defined and initialized to
>> XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
>> variable is set in same file function xen_guest_init(), that is
>> early_initcall. But eraly_initcall is called later, than
>> console_initcall, that's why in time of running xen_cons_init(void)
>> xen_domain_type is not initialized to correct value and
>> xen_cons_init() does not initialize console, as returns on first check
>> "if (!xen_domain())".
>> It is not critical in normal operation, because we have
>> device_initcall xen_hvc_init() that is called after xen_guest_init(),
>> it initialize hvc. But in case of kernel falls before
>> device_initcall's, we can't see any printouts, that could be useful.
>>
>> Could you please explain, may be using some configs or arguments in
>> command line for kernel, how to enable this early console?
> 
> 
> It is explained in the kernel-parameters.txt (https://www.kernel.org/doc/Documentation/kernel-parameters.txt)
> :
> 
> earlyprintk=	[X86,SH,BLACKFIN,ARM,M68k]
> 			earlyprintk=vga
> 			earlyprintk=efi
> 			earlyprintk=xen
> ...

AFAIK, earlyprintk=xen has never work on ARM. This is because we don't
know that we running on Xen until late.

Regards,

-- 
Julien Grall

Re: Question on hvc console init

[Ian Campbell]

On Tue, 2014-10-28 at 17:32 +0000, Julien Grall wrote:
> Hi Konrad,
> 
> On 10/28/2014 04:53 PM, Konrad Rzeszutek Wilk wrote:
> > On Tue, Oct 28, 2014 at 05:53:25PM +0200, Iurii Konovalenko wrote:
> >> Hello, all!
> >>
> >> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
> >> Xen revision is 4.4.
> >> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
> >> In kernel I've found strange behaviour in hvc console init function.
> >> In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:
> >>
> >>     if (!xen_domain())
> >>         return 0;
> >>
> >>     if (xen_initial_domain())
> >>         ops = &dom0_hvc_ops;
> >>     else {
> >>
> >> xen_domain() and xen_initial_domain() are defined to check
> >> xen_domain_type variable. This variable is defined and initialized to
> >> XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
> >> variable is set in same file function xen_guest_init(), that is
> >> early_initcall. But eraly_initcall is called later, than
> >> console_initcall, that's why in time of running xen_cons_init(void)
> >> xen_domain_type is not initialized to correct value and
> >> xen_cons_init() does not initialize console, as returns on first check
> >> "if (!xen_domain())".
> >> It is not critical in normal operation, because we have
> >> device_initcall xen_hvc_init() that is called after xen_guest_init(),
> >> it initialize hvc. But in case of kernel falls before
> >> device_initcall's, we can't see any printouts, that could be useful.
> >>
> >> Could you please explain, may be using some configs or arguments in
> >> command line for kernel, how to enable this early console?
> > 
> > 
> > It is explained in the kernel-parameters.txt (https://www.kernel.org/doc/Documentation/kernel-parameters.txt)
> > :
> > 
> > earlyprintk=	[X86,SH,BLACKFIN,ARM,M68k]
> > 			earlyprintk=vga
> > 			earlyprintk=efi
> > 			earlyprintk=xen
> > ...
> 
> AFAIK, earlyprintk=xen has never work on ARM. This is because we don't
> know that we running on Xen until late.

earlyprintk on arm doesn't parse it's argument at all, it is a boolean
option which just enables printk via the hardcoded (via .config) debug
UART. Because of that it's incompatible with multiplatform kernels as
well.

Some platforms (e.g. arm64) have replace earlyprintk with a more
flexible earlycon, I don't know if arm is one of those (yet?). It would
probably be nice to integrate the xen early print stuff with earlycon at
some point.

Ian.

Re: Question on hvc console init

[Julien Grall]

On 10/29/2014 08:22 AM, Ian Campbell wrote:
> On Tue, 2014-10-28 at 17:32 +0000, Julien Grall wrote:
>> Hi Konrad,
>>
>> On 10/28/2014 04:53 PM, Konrad Rzeszutek Wilk wrote:
>>> On Tue, Oct 28, 2014 at 05:53:25PM +0200, Iurii Konovalenko wrote:
>>>> Hello, all!
>>>>
>>>> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
>>>> Xen revision is 4.4.
>>>> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
>>>> In kernel I've found strange behaviour in hvc console init function.
>>>> In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:
>>>>
>>>>     if (!xen_domain())
>>>>         return 0;
>>>>
>>>>     if (xen_initial_domain())
>>>>         ops = &dom0_hvc_ops;
>>>>     else {
>>>>
>>>> xen_domain() and xen_initial_domain() are defined to check
>>>> xen_domain_type variable. This variable is defined and initialized to
>>>> XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
>>>> variable is set in same file function xen_guest_init(), that is
>>>> early_initcall. But eraly_initcall is called later, than
>>>> console_initcall, that's why in time of running xen_cons_init(void)
>>>> xen_domain_type is not initialized to correct value and
>>>> xen_cons_init() does not initialize console, as returns on first check
>>>> "if (!xen_domain())".
>>>> It is not critical in normal operation, because we have
>>>> device_initcall xen_hvc_init() that is called after xen_guest_init(),
>>>> it initialize hvc. But in case of kernel falls before
>>>> device_initcall's, we can't see any printouts, that could be useful.
>>>>
>>>> Could you please explain, may be using some configs or arguments in
>>>> command line for kernel, how to enable this early console?
>>>
>>>
>>> It is explained in the kernel-parameters.txt (https://www.kernel.org/doc/Documentation/kernel-parameters.txt)
>>> :
>>>
>>> earlyprintk=	[X86,SH,BLACKFIN,ARM,M68k]
>>> 			earlyprintk=vga
>>> 			earlyprintk=efi
>>> 			earlyprintk=xen
>>> ...
>>
>> AFAIK, earlyprintk=xen has never work on ARM. This is because we don't
>> know that we running on Xen until late.
> 
> earlyprintk on arm doesn't parse it's argument at all, it is a boolean
> option which just enables printk via the hardcoded (via .config) debug
> UART. Because of that it's incompatible with multiplatform kernels as
> well.

For now the good way to debug is enabling the earlyprintk for your
platform (Xen is emulating a simple UART for the one it "steal") or
patching Linux with the patch I sent few mails earlier.

> Some platforms (e.g. arm64) have replace earlyprintk with a more
> flexible earlycon, I don't know if arm is one of those (yet?). It would
> probably be nice to integrate the xen early print stuff with earlycon at
> some point.

It seems that earlycon exists also for ARM. I will give a look next week.

Regards,

-- 
Julien Grall

Re: Question on hvc console init

[Julien Grall]

On 10/28/2014 03:53 PM, Iurii Konovalenko wrote:
> Hello, all!

Hello Iurii,
 
> I try to bring up Xen on Renesas Lager board (r8a7790 SoC - R-Car H2).
> Xen revision is 4.4.
> I try to run Linux (kernel 3.14 + LTSI patches) as Dom0.
> In kernel I've found strange behaviour in hvc console init function.
> In file drivers/tty/hvc/hvc_xen.c in function xen_cons_init(void) sources are:
> 
>     if (!xen_domain())
>         return 0;
> 
>     if (xen_initial_domain())
>         ops = &dom0_hvc_ops;
>     else {
> 
> xen_domain() and xen_initial_domain() are defined to check
> xen_domain_type variable. This variable is defined and initialized to
> XEN_NATIVE in arch/arm/xen/enlighten.c. The real value of this
> variable is set in same file function xen_guest_init(), that is
> early_initcall. But eraly_initcall is called later, than
> console_initcall, that's why in time of running xen_cons_init(void)
> xen_domain_type is not initialized to correct value and
> xen_cons_init() does not initialize console, as returns on first check
> "if (!xen_domain())".
> It is not critical in normal operation, because we have
> device_initcall xen_hvc_init() that is called after xen_guest_init(),
> it initialize hvc. But in case of kernel falls before
> device_initcall's, we can't see any printouts, that could be useful.
> 
> Could you please explain, may be using some configs or arguments in
> command line for kernel, how to enable this early console?

AFAIK, this part of the HVC console has never been tested on Xen on ARM.

When the kernel is hanging before the console is effectively setup,
I usually use the hacky patch below.

In any case, you are welcome to send a patch for fixing the early console ;). 
I guess the best solution would be to call xen_guest_init earlier.

Regards,

diff --git a/drivers/tty/hvc/hvc_xen.c b/drivers/tty/hvc/hvc_xen.c
index f1e5742..644fd65 100644
--- a/drivers/tty/hvc/hvc_xen.c
+++ b/drivers/tty/hvc/hvc_xen.c
@@ -306,6 +306,7 @@ void xen_console_resume(void)
                rebind_evtchn_irq(info->evtchn, info->irq);
 }
 
+#ifdef CONFIG_HVC_XEN_FRONTEND
 static void xencons_disconnect_backend(struct xencons_info *info)
 {
        if (info->irq > 0)
@@ -346,7 +347,8 @@ static int xen_console_remove(struct xencons_info *info)
        return 0;
 }
 
-#ifdef CONFIG_HVC_XEN_FRONTEND
+static struct xenbus_driver xencons_driver;
+
 static int xencons_remove(struct xenbus_device *dev)
 {
        return xen_console_remove(dev_get_drvdata(&dev->dev));
@@ -626,7 +628,6 @@ void xen_raw_console_write(const char *str)
        ssize_t len = strlen(str);
        int rc = 0;
 
-       if (xen_domain()) {
                rc = dom0_write_console(0, str, len);
 #ifdef CONFIG_X86
                if (rc == -ENOSYS && xen_hvm_domain())
@@ -638,7 +639,6 @@ outb_print:
                for (i = 0; i < len; i++)
                        outb(str[i], 0xe9);
 #endif
-       }
 }
 
 void xen_raw_printk(const char *fmt, ...)
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index ced2b84..6c7b960 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -1613,6 +1613,8 @@ static size_t cont_print_text(char *text, size_t size)
        return textlen;
 }
 
+#include <xen/hvc-console.h>
+
 asmlinkage int vprintk_emit(int facility, int level,
                            const char *dict, size_t dictlen,
                            const char *fmt, va_list args)
@@ -1680,6 +1682,8 @@ asmlinkage int vprintk_emit(int facility, int level,
         * prefix which might be passed-in as a parameter.
         */
        text_len = vscnprintf(text, sizeof(textbuf), fmt, args);
+    xen_raw_console_write(text);
+
 
        /* mark and strip a trailing newline */
        if (text_len && text[text_len-1] == '\n') {

-- 
Julien Grall