Xen randomly stuck in mdelay() during MP initialization

Xen randomly stuck in mdelay() during MP initialization

[Julian Pidancet]

Hi,

When trying to boot xen 4.1 on new hardware, Xen become stuck in
wakeup_secondary_cpu() in the mdelay function.

    Dprintk("Waiting for send to finish...\n");
    timeout = 0;
    do {
        Dprintk("+");
        udelay(100);
        if ( !x2apic_enabled )
            send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
    } while ( send_status && (timeout++ < 1000) );

    printk("before mdelay\n");
    mdelay(10);
    printk("after mdelay\n");

    Dprintk("Deasserting INIT.\n");

The hang can happen randomly with any of the CPUs to wake up and
sometime doesn't happen at all.
Replacing mdelay(10) with udelay(10) seems to fix the issue.

-- 
Julian

Re: Xen randomly stuck in mdelay() during MP initialization

[Pasi Kärkkäinen]

On Wed, Jan 11, 2012 at 01:06:43PM +0000, Julian Pidancet wrote:
> Hi,
> 

Hello,

> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
> 

Can you define "new hardware" ?


-- Pasi

Re: Xen randomly stuck in mdelay() during MP initialization

[Keir Fraser]

On 11/01/2012 13:06, "Julian Pidancet" <julian.pidancet@gmail.com> wrote:

> Hi,
> 
> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
> 
>     Dprintk("Waiting for send to finish...\n");
>     timeout = 0;
>     do {
>         Dprintk("+");
>         udelay(100);
>         if ( !x2apic_enabled )
>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>     } while ( send_status && (timeout++ < 1000) );
> 
>     printk("before mdelay\n");
>     mdelay(10);
>     printk("after mdelay\n");
> 
>     Dprintk("Deasserting INIT.\n");
> 
> The hang can happen randomly with any of the CPUs to wake up and
> sometime doesn't happen at all.
> Replacing mdelay(10) with udelay(10) seems to fix the issue.

Do you see this in xen-unstable? Hopefully it is working there, and we can
simply backport the fix.

 -- Keir

Re: Xen randomly stuck in mdelay() during MP initialization

[Julian Pidancet]

On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com> wrote:
> On 11/01/2012 13:06, "Julian Pidancet" <julian.pidancet@gmail.com> wrote:
>
>> Hi,
>>
>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>> wakeup_secondary_cpu() in the mdelay function.
>>
>>     Dprintk("Waiting for send to finish...\n");
>>     timeout = 0;
>>     do {
>>         Dprintk("+");
>>         udelay(100);
>>         if ( !x2apic_enabled )
>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>     } while ( send_status && (timeout++ < 1000) );
>>
>>     printk("before mdelay\n");
>>     mdelay(10);
>>     printk("after mdelay\n");
>>
>>     Dprintk("Deasserting INIT.\n");
>>
>> The hang can happen randomly with any of the CPUs to wake up and
>> sometime doesn't happen at all.
>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>
> Do you see this in xen-unstable? Hopefully it is working there, and we can
> simply backport the fix.
>

I checked yesterday and the code of this function is the same in
xen-unstable, but I don't encounter the problem with xen-unstable.

What concerns me is that this is the only place in the function where
mdelay() is used, the rest of the function seems to be using udelay().
What's the difference between the two ?
Does mdelay() relies on some form of timer to execute ?

-- 
Julian

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen randomly stuck in mdelay() during MP initialization

[Jan Beulich]

>>> On 11.01.12 at 15:51, Julian Pidancet <julian.pidancet@gmail.com> wrote:
> On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com> wrote:
>> On 11/01/2012 13:06, "Julian Pidancet" <julian.pidancet@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>>> wakeup_secondary_cpu() in the mdelay function.
>>>
>>>     Dprintk("Waiting for send to finish...\n");
>>>     timeout = 0;
>>>     do {
>>>         Dprintk("+");
>>>         udelay(100);
>>>         if ( !x2apic_enabled )
>>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>>     } while ( send_status && (timeout++ < 1000) );
>>>
>>>     printk("before mdelay\n");
>>>     mdelay(10);
>>>     printk("after mdelay\n");
>>>
>>>     Dprintk("Deasserting INIT.\n");
>>>
>>> The hang can happen randomly with any of the CPUs to wake up and
>>> sometime doesn't happen at all.
>>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>>
>> Do you see this in xen-unstable? Hopefully it is working there, and we can
>> simply backport the fix.
>>
> 
> I checked yesterday and the code of this function is the same in
> xen-unstable,

Hardly.

> but I don't encounter the problem with xen-unstable.

Running in x2apic mode, perhaps? (Despite having been asked
already, you still didn't really provide technical details about your
systems. Nor did you care to attach a log of the successful
attempt with -unstable...)

> What concerns me is that this is the only place in the function where
> mdelay() is used, the rest of the function seems to be using udelay().
> What's the difference between the two ?

Quite obviously the former delays by a number of milliseconds,
while the value passed to the latter is in microseconds.

> Does mdelay() relies on some form of timer to execute ?

No. mdelay() is simply a loop around udelay() - see
xen/include/xen/delay.h.

Jan

Re: Xen randomly stuck in mdelay() during MP initialization

[Julian Pidancet]

On Wed, Jan 11, 2012 at 3:12 PM, Jan Beulich <JBeulich@suse.com> wrote:
>
> Hardly.
>
>> but I don't encounter the problem with xen-unstable.
>
> Running in x2apic mode, perhaps? (Despite having been asked
> already, you still didn't really provide technical details about your
> systems. Nor did you care to attach a log of the successful
> attempt with -unstable...)
>

Yes, disabling x2apic fixes the issue, thank you.
Is x2apic disabled by default on xen-unstable ? It could explain why
it doesn't hang with it.

>> Does mdelay() relies on some form of timer to execute ?
>
> No. mdelay() is simply a loop around udelay() - see
> xen/include/xen/delay.h.
>

Yes, that was quite a stupid question from me, what it does is pretty
obvious by looking at the code.

-- 
Julian

Re: Xen randomly stuck in mdelay() during MP initialization

[Jan Beulich]

>>> On 11.01.12 at 16:28, Julian Pidancet <julian.pidancet@gmail.com> wrote:
> On Wed, Jan 11, 2012 at 3:12 PM, Jan Beulich <JBeulich@suse.com> wrote:
>>
>> Hardly.
>>
>>> but I don't encounter the problem with xen-unstable.
>>
>> Running in x2apic mode, perhaps? (Despite having been asked
>> already, you still didn't really provide technical details about your
>> systems. Nor did you care to attach a log of the successful
>> attempt with -unstable...)
>>
> 
> Yes, disabling x2apic fixes the issue, thank you.
> Is x2apic disabled by default on xen-unstable ? It could explain why
> it doesn't hang with it.

No, it's enabled by default. But AP bringup was changed for the
x2apic case some time ago (which was the reason for my "Hardly"
above).

Jan

Re: Xen randomly stuck in mdelay() during MP initialization

[Keir Fraser]

On 11/01/2012 14:51, "Julian Pidancet" <julian.pidancet@gmail.com> wrote:

> On Wed, Jan 11, 2012 at 1:28 PM, Keir Fraser <keir.xen@gmail.com> wrote:
>> On 11/01/2012 13:06, "Julian Pidancet" <julian.pidancet@gmail.com> wrote:
>> 
>>> Hi,
>>> 
>>> When trying to boot xen 4.1 on new hardware, Xen become stuck in
>>> wakeup_secondary_cpu() in the mdelay function.
>>> 
>>>     Dprintk("Waiting for send to finish...\n");
>>>     timeout = 0;
>>>     do {
>>>         Dprintk("+");
>>>         udelay(100);
>>>         if ( !x2apic_enabled )
>>>             send_status = apic_read(APIC_ICR) & APIC_ICR_BUSY;
>>>     } while ( send_status && (timeout++ < 1000) );
>>> 
>>>     printk("before mdelay\n");
>>>     mdelay(10);
>>>     printk("after mdelay\n");
>>> 
>>>     Dprintk("Deasserting INIT.\n");
>>> 
>>> The hang can happen randomly with any of the CPUs to wake up and
>>> sometime doesn't happen at all.
>>> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>> 
>> Do you see this in xen-unstable? Hopefully it is working there, and we can
>> simply backport the fix.
>> 
> 
> I checked yesterday and the code of this function is the same in
> xen-unstable, but I don't encounter the problem with xen-unstable.

Well that would be because the code of this function is not the same in
xen-4.1 and xen-unstable.

> What concerns me is that this is the only place in the function where
> mdelay() is used, the rest of the function seems to be using udelay().
> What's the difference between the two ?
> Does mdelay() relies on some form of timer to execute ?

mdelay(ms) just calls udelay(1000) ms times.

 -- Keir

Re: Xen randomly stuck in mdelay() during MP initialization

[Igor Mammedov]

On 01/11/2012 02:06 PM, Julian Pidancet wrote:
> Hi,
>
> When trying to boot xen 4.1 on new hardware, Xen become stuck in
> wakeup_secondary_cpu() in the mdelay function.
>
>      Dprintk("Waiting for send to finish...\n");
>      timeout = 0;
>      do {
>          Dprintk("+");
>          udelay(100);
>          if ( !x2apic_enabled )
>              send_status = apic_read(APIC_ICR)&  APIC_ICR_BUSY;
>      } while ( send_status&&  (timeout++<  1000) );
>
>      printk("before mdelay\n");
>      mdelay(10);
>      printk("after mdelay\n");
>
>      Dprintk("Deasserting INIT.\n");
>
> The hang can happen randomly with any of the CPUs to wake up and
> sometime doesn't happen at all.
> Replacing mdelay(10) with udelay(10) seems to fix the issue.
>

Essentially the same issue, check out
   http://article.gmane.org/gmane.comp.emulators.xen.devel/114776/match=ivy+bridge

Problem is that udelay uses 32 bit value from tsc for measuring
elapsed time and at early boot stage something steals
(most likely SMI) boot cpu with a following wrapping of tsc value
in udelay. And in case of mdelay this happens multiple times.
Replacing mdelay with udelay or removing it helps but it may
break boot on other hardware.
I suspect it's a BIOS issue.

-- 
Thanks,
  Igor