iommu=0 leading to panic when system defaults to using x2apic

iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

Why wouldn't iommu=0 imply not using the x2apic, if use of the
iommu indeed is a prerequisite (which by itself seems questionable
as long as APIC IDs aren't wider than 8 bits)?

Looking further at enable_bsp_x2apic(), I also wonder whether
the pre-enabled case, when specifying iommu=0 and x2apic=0,
can actually work if iommu=0 is a prerequisite (intremap_enabled()
would return 1 in that case afaics, so the early check doesn't help).

Thanks, Jan

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

Are you looking at xen-4.0? I think you should look at latest xen-unstable
and we can backpoprt patches if that is more satisfactory. I did a big
cleanup patch yesterday which makes the code smaller and clearer than it
was, yet I don't understand the dependencies between x2apic-enabled-by-bios
and the need for interrupt remapping, and all that stuff.

 -- Keir

On 10/12/2010 09:12, "Jan Beulich" <JBeulich@novell.com> wrote:

> Why wouldn't iommu=0 imply not using the x2apic, if use of the
> iommu indeed is a prerequisite (which by itself seems questionable
> as long as APIC IDs aren't wider than 8 bits)?
> 
> Looking further at enable_bsp_x2apic(), I also wonder whether
> the pre-enabled case, when specifying iommu=0 and x2apic=0,
> can actually work if iommu=0 is a prerequisite (intremap_enabled()
> would return 1 in that case afaics, so the early check doesn't help).
> 
> Thanks, Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote:
> Are you looking at xen-4.0? I think you should look at latest xen-unstable

I looked at 4.0 in parallel with -unstable (non-staging) as of
yesterday.

> and we can backpoprt patches if that is more satisfactory. I did a big
> cleanup patch yesterday which makes the code smaller and clearer than it

I'll have a look at what you did...

> was, yet I don't understand the dependencies between x2apic-enabled-by-bios
> and the need for interrupt remapping, and all that stuff.

... after hopefully those dependencies got clarified by one of the
original authors.

Jan

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote:
>> Are you looking at xen-4.0? I think you should look at latest xen-unstable
> 
> I looked at 4.0 in parallel with -unstable (non-staging) as of
> yesterday.

Even before yesterday, xen-unstable benefits from c/s 22388, which is
intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if
it is fully satisfactory. And it is not backported to 4.0 as yet because,
well, I don't understand this crap enough, basically.

>> and we can backpoprt patches if that is more satisfactory. I did a big
>> cleanup patch yesterday which makes the code smaller and clearer than it
> 
> I'll have a look at what you did...

My change (c/s 22475) is larger, but should have no semantic change. It just
tries to make things smaller and cleaner. For that alone maybe we will
backport it to 4.0 too.

>> was, yet I don't understand the dependencies between x2apic-enabled-by-bios
>> and the need for interrupt remapping, and all that stuff.
> 
> ... after hopefully those dependencies got clarified by one of the
> original authors.

That would be nice. It feels like if BIOS has enabled x2apic we ought to be
able to soldier on with it and not need to panic. Couldn't we ignore the
x2apic-ness? Or turn it off? What about AMD boxes that support x2apic on the
CPUs yet of course do not have VT-d irq remapping? Lots of questions, few
answers. :-)

 -- Keir

> Jan
> 

Re: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 10.12.10 at 12:47, Keir Fraser <keir@xen.org> wrote:
> On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote:
>>> Are you looking at xen-4.0? I think you should look at latest xen-unstable
>> 
>> I looked at 4.0 in parallel with -unstable (non-staging) as of
>> yesterday.
> 
> Even before yesterday, xen-unstable benefits from c/s 22388, which is
> intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if
> it is fully satisfactory. And it is not backported to 4.0 as yet because,
> well, I don't understand this crap enough, basically.

Ah, yes, that makes it better (and I must then have looked at an
older snapshot of -unstable instead).

Still, as you say, a lot of questions remain. On top of what you and
I raised already, I wonder whether the APIC-ID-exceeding-8-bits
case is really being handled correctly now when iommu=0.

Jan

RE: iommu=0 leading to panic when system defaults to using x2apic

[Zhang, Yang Z]

Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will trigger the panic.

best regards
yang

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich
> Sent: Friday, December 10, 2010 8:03 PM
> To: Keir Fraser
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
> >>> On 10.12.10 at 12:47, Keir Fraser <keir@xen.org> wrote:
> > On 10/12/2010 10:58, "Jan Beulich" <JBeulich@novell.com> wrote:
> >
> >>>>> On 10.12.10 at 11:06, Keir Fraser <keir@xen.org> wrote:
> >>> Are you looking at xen-4.0? I think you should look at latest xen-unstable
> >>
> >> I looked at 4.0 in parallel with -unstable (non-staging) as of
> >> yesterday.
> >
> > Even before yesterday, xen-unstable benefits from c/s 22388, which is
> > intended to work around the iommu=0/x2apic=0 dependency a bit. Not sure if
> > it is fully satisfactory. And it is not backported to 4.0 as yet because,
> > well, I don't understand this crap enough, basically.
> 
> Ah, yes, that makes it better (and I must then have looked at an
> older snapshot of -unstable instead).
> 
> Still, as you say, a lot of questions remain. On top of what you and
> I raised already, I wonder whether the APIC-ID-exceeding-8-bits
> case is really being handled correctly now when iommu=0.
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will 
> trigger the panic.

Would mind clarifying where this dependency actually is?

Thanks, Jan

RE: iommu=0 leading to panic when system defaults to using x2apic

[Kay, Allen M]

The architectural requirement is actually between interrupt remapping and x2apic.  Since interrupt remapping is part of the VT-d feature so current software requires all VT-d features enabled in order for x2apic to be enabled.

Strictly speaking DMA remapping is not required for x2apic.  However, queued invalidation is required since interrupt remapping requires queued invalidation.  So x2apic dependency is as follows:

    x2apic->interrupt remapping->queued invalidation

Due to historical reasons, the new VT-d features were built on top of the old ones as they become available.  Is there a requirement to separate this out?  If so, we will need to re-design iommu boot parameter which took a while to get it right so most systems can now boot successfully.

Allen

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich
Sent: Friday, December 10, 2010 7:40 AM
To: Zhang, Yang Z
Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong
Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will 
> trigger the panic.

Would mind clarifying where this dependency actually is?

Thanks, Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
address 32-bit APIC IDs?

 -- Keir

On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:

> The architectural requirement is actually between interrupt remapping and
> x2apic.  Since interrupt remapping is part of the VT-d feature so current
> software requires all VT-d features enabled in order for x2apic to be enabled.
> 
> Strictly speaking DMA remapping is not required for x2apic.  However, queued
> invalidation is required since interrupt remapping requires queued
> invalidation.  So x2apic dependency is as follows:
> 
>     x2apic->interrupt remapping->queued invalidation
> 
> Due to historical reasons, the new VT-d features were built on top of the old
> ones as they become available.  Is there a requirement to separate this out?
> If so, we will need to re-design iommu boot parameter which took a while to
> get it right so most systems can now boot successfully.
> 
> Allen
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich
> Sent: Friday, December 10, 2010 7:40 AM
> To: Zhang, Yang Z
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong
> Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
>>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
>> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will
>> trigger the panic.
> 
> Would mind clarifying where this dependency actually is?
> 
> Thanks, Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: iommu=0 leading to panic when system defaults to using x2apic

[Kay, Allen M]

Yes, interrupt remapping is needed to be the intermediary between legacy IOxAPIC and MSI devices and the new x2APIC in the CPU. 

-----Original Message-----
From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
Sent: Friday, December 10, 2010 10:50 AM
To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
Cc: xen-devel@lists.xensource.com; Han, Weidong
Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
address 32-bit APIC IDs?

 -- Keir

On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:

> The architectural requirement is actually between interrupt remapping and
> x2apic.  Since interrupt remapping is part of the VT-d feature so current
> software requires all VT-d features enabled in order for x2apic to be enabled.
> 
> Strictly speaking DMA remapping is not required for x2apic.  However, queued
> invalidation is required since interrupt remapping requires queued
> invalidation.  So x2apic dependency is as follows:
> 
>     x2apic->interrupt remapping->queued invalidation
> 
> Due to historical reasons, the new VT-d features were built on top of the old
> ones as they become available.  Is there a requirement to separate this out?
> If so, we will need to re-design iommu boot parameter which took a while to
> get it right so most systems can now boot successfully.
> 
> Allen
> 
> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Jan Beulich
> Sent: Friday, December 10, 2010 7:40 AM
> To: Zhang, Yang Z
> Cc: Keir Fraser; xen-devel@lists.xensource.com; Han, Weidong
> Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
>>>> On 10.12.10 at 16:00, "Zhang, Yang Z" <yang.z.zhang@intel.com> wrote:
>> Yes, X2apic is depend on iommu. If disable iommu when using x2apic, it will
>> trigger the panic.
> 
> Would mind clarifying where this dependency actually is?
> 
> Thanks, Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> Yes, interrupt remapping is needed to be the intermediary between legacy 
> IOxAPIC and MSI devices and the new x2APIC in the CPU. 

But isn't this only when there are APIC IDs beyond 255?

Jan

> -----Original Message-----
> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
> Sent: Friday, December 10, 2010 10:50 AM
> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
> Cc: xen-devel@lists.xensource.com; Han, Weidong
> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to 
> using x2apic
> 
> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
> address 32-bit APIC IDs?
> 
>  -- Keir
> 
> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> 
>> The architectural requirement is actually between interrupt remapping and
>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>> software requires all VT-d features enabled in order for x2apic to be 
> enabled.
>> 
>> Strictly speaking DMA remapping is not required for x2apic.  However, queued
>> invalidation is required since interrupt remapping requires queued
>> invalidation.  So x2apic dependency is as follows:
>> 
>>     x2apic->interrupt remapping->queued invalidation
>> 
>> Due to historical reasons, the new VT-d features were built on top of the old
>> ones as they become available.  Is there a requirement to separate this out?
>> If so, we will need to re-design iommu boot parameter which took a while to
>> get it right so most systems can now boot successfully.
>> 
>> Allen

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>> Yes, interrupt remapping is needed to be the intermediary between legacy
>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
> 
> But isn't this only when there are APIC IDs beyond 255?

Apparently not, since even Linux requires irq remapping even when none of
the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
fully understand this particular restriction, mind you.

Actually, my guess is that x2apic mode requires a different format of APIC
message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
support the new message format, and so irq remapping hardware is required to
bridge the two formats, even if no actual irq remapping is occurring.

Is that a canny guess, Allen?

 -- Keir

> Jan
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>> Sent: Friday, December 10, 2010 10:50 AM
>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>> Cc: xen-devel@lists.xensource.com; Han, Weidong
>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>> using x2apic
>> 
>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>> address 32-bit APIC IDs?
>> 
>>  -- Keir
>> 
>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>> 
>>> The architectural requirement is actually between interrupt remapping and
>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>> software requires all VT-d features enabled in order for x2apic to be
>> enabled.
>>> 
>>> Strictly speaking DMA remapping is not required for x2apic.  However, queued
>>> invalidation is required since interrupt remapping requires queued
>>> invalidation.  So x2apic dependency is as follows:
>>> 
>>>     x2apic->interrupt remapping->queued invalidation
>>> 
>>> Due to historical reasons, the new VT-d features were built on top of the
>>> old
>>> ones as they become available.  Is there a requirement to separate this out?
>>> If so, we will need to re-design iommu boot parameter which took a while to
>>> get it right so most systems can now boot successfully.
>>> 
>>> Allen
> 
> 

RE: iommu=0 leading to panic when system defaults to using x2apic

[Kay, Allen M]

Keir/Jan,

My understanding is that cluster mode requires it.  I will get back to you guys after I dig out the details on this - did not get a chance to do this today.

Allen

-----Original Message-----
From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
Sent: Monday, December 13, 2010 1:03 AM
To: Jan Beulich; Kay, Allen M; Zhang, Yang Z
Cc: Han, Weidong; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>> Yes, interrupt remapping is needed to be the intermediary between legacy
>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
> 
> But isn't this only when there are APIC IDs beyond 255?

Apparently not, since even Linux requires irq remapping even when none of
the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
fully understand this particular restriction, mind you.

Actually, my guess is that x2apic mode requires a different format of APIC
message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
support the new message format, and so irq remapping hardware is required to
bridge the two formats, even if no actual irq remapping is occurring.

Is that a canny guess, Allen?

 -- Keir

> Jan
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>> Sent: Friday, December 10, 2010 10:50 AM
>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>> Cc: xen-devel@lists.xensource.com; Han, Weidong
>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>> using x2apic
>> 
>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>> address 32-bit APIC IDs?
>> 
>>  -- Keir
>> 
>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>> 
>>> The architectural requirement is actually between interrupt remapping and
>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>> software requires all VT-d features enabled in order for x2apic to be
>> enabled.
>>> 
>>> Strictly speaking DMA remapping is not required for x2apic.  However, queued
>>> invalidation is required since interrupt remapping requires queued
>>> invalidation.  So x2apic dependency is as follows:
>>> 
>>>     x2apic->interrupt remapping->queued invalidation
>>> 
>>> Due to historical reasons, the new VT-d features were built on top of the
>>> old
>>> ones as they become available.  Is there a requirement to separate this out?
>>> If so, we will need to re-design iommu boot parameter which took a while to
>>> get it right so most systems can now boot successfully.
>>> 
>>> Allen
> 
> 

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

Well, if it is a restriction imposed by cluster mode, you know the next
question is obvious: Why do we bother with cluster mode at all? I don't see
that it yields us any advantage over physical mode, and we could use
physical mode without interrupt remapping, that would seem to be a big bonus
and simplification? Could we just kill our x2apic cluster mode logic?

 -- Keir

On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote:

> Keir/Jan,
> 
> My understanding is that cluster mode requires it.  I will get back to you
> guys after I dig out the details on this - did not get a chance to do this
> today.
> 
> Allen
> 
> -----Original Message-----
> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
> Sent: Monday, December 13, 2010 1:03 AM
> To: Jan Beulich; Kay, Allen M; Zhang, Yang Z
> Cc: Han, Weidong; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
> On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>> Yes, interrupt remapping is needed to be the intermediary between legacy
>>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
>> 
>> But isn't this only when there are APIC IDs beyond 255?
> 
> Apparently not, since even Linux requires irq remapping even when none of
> the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
> fully understand this particular restriction, mind you.
> 
> Actually, my guess is that x2apic mode requires a different format of APIC
> message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
> support the new message format, and so irq remapping hardware is required to
> bridge the two formats, even if no actual irq remapping is occurring.
> 
> Is that a canny guess, Allen?
> 
>  -- Keir
> 
>> Jan
>> 
>>> -----Original Message-----
>>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>>> Sent: Friday, December 10, 2010 10:50 AM
>>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>>> Cc: xen-devel@lists.xensource.com; Han, Weidong
>>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>>> using x2apic
>>> 
>>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>>> address 32-bit APIC IDs?
>>> 
>>>  -- Keir
>>> 
>>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>> 
>>>> The architectural requirement is actually between interrupt remapping and
>>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>>> software requires all VT-d features enabled in order for x2apic to be
>>> enabled.
>>>> 
>>>> Strictly speaking DMA remapping is not required for x2apic.  However,
>>>> queued
>>>> invalidation is required since interrupt remapping requires queued
>>>> invalidation.  So x2apic dependency is as follows:
>>>> 
>>>>     x2apic->interrupt remapping->queued invalidation
>>>> 
>>>> Due to historical reasons, the new VT-d features were built on top of the
>>>> old
>>>> ones as they become available.  Is there a requirement to separate this
>>>> out?
>>>> If so, we will need to re-design iommu boot parameter which took a while to
>>>> get it right so most systems can now boot successfully.
>>>> 
>>>> Allen
>> 
>> 
> 
> 

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

Also, even if we continued to use cluster mode for IPIs (in the hope of
devising a more efficient group IPI algorithm in future) that doesn't stop
us from always exposing physical mode to IOAPICs and MSI devices.

 -- Keir

On 14/12/2010 07:44, "Keir Fraser" <keir@xen.org> wrote:

> Well, if it is a restriction imposed by cluster mode, you know the next
> question is obvious: Why do we bother with cluster mode at all? I don't see
> that it yields us any advantage over physical mode, and we could use
> physical mode without interrupt remapping, that would seem to be a big bonus
> and simplification? Could we just kill our x2apic cluster mode logic?
> 
>  -- Keir
> 
> On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> 
>> Keir/Jan,
>> 
>> My understanding is that cluster mode requires it.  I will get back to you
>> guys after I dig out the details on this - did not get a chance to do this
>> today.
>> 
>> Allen
>> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>> Sent: Monday, December 13, 2010 1:03 AM
>> To: Jan Beulich; Kay, Allen M; Zhang, Yang Z
>> Cc: Han, Weidong; xen-devel@lists.xensource.com
>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>> using x2apic
>> 
>> On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:
>> 
>>>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>>> Yes, interrupt remapping is needed to be the intermediary between legacy
>>>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
>>> 
>>> But isn't this only when there are APIC IDs beyond 255?
>> 
>> Apparently not, since even Linux requires irq remapping even when none of
>> the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
>> fully understand this particular restriction, mind you.
>> 
>> Actually, my guess is that x2apic mode requires a different format of APIC
>> message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
>> support the new message format, and so irq remapping hardware is required to
>> bridge the two formats, even if no actual irq remapping is occurring.
>> 
>> Is that a canny guess, Allen?
>> 
>>  -- Keir
>> 
>>> Jan
>>> 
>>>> -----Original Message-----
>>>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>>>> Sent: Friday, December 10, 2010 10:50 AM
>>>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>>>> Cc: xen-devel@lists.xensource.com; Han, Weidong
>>>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>>>> using x2apic
>>>> 
>>>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>>>> address 32-bit APIC IDs?
>>>> 
>>>>  -- Keir
>>>> 
>>>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>>> 
>>>>> The architectural requirement is actually between interrupt remapping and
>>>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>>>> software requires all VT-d features enabled in order for x2apic to be
>>>> enabled.
>>>>> 
>>>>> Strictly speaking DMA remapping is not required for x2apic.  However,
>>>>> queued
>>>>> invalidation is required since interrupt remapping requires queued
>>>>> invalidation.  So x2apic dependency is as follows:
>>>>> 
>>>>>     x2apic->interrupt remapping->queued invalidation
>>>>> 
>>>>> Due to historical reasons, the new VT-d features were built on top of the
>>>>> old
>>>>> ones as they become available.  Is there a requirement to separate this
>>>>> out?
>>>>> If so, we will need to re-design iommu boot parameter which took a while
>>>>> to
>>>>> get it right so most systems can now boot successfully.
>>>>> 
>>>>> Allen
>>> 
>>> 
>> 
>> 
> 
> 

RE: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> The architectural requirement is actually between interrupt remapping and 
> x2apic.  Since interrupt remapping is part of the VT-d feature so current 
> software requires all VT-d features enabled in order for x2apic to be enabled.
> 
> Strictly speaking DMA remapping is not required for x2apic.  However, queued 
> invalidation is required since interrupt remapping requires queued 
> invalidation.  So x2apic dependency is as follows:
> 
>     x2apic->interrupt remapping->queued invalidation
> 
> Due to historical reasons, the new VT-d features were built on top of the old 
> ones as they become available.  Is there a requirement to separate this out?  
> If so, we will need to re-design iommu boot parameter which took a while to 
> get it right so most systems can now boot successfully.

And why is it that Xen panics when the BIOS pre-enabled x2apic
mode without also enabling interrupt re-mapping, while Linux
(afaict from looking at 2.6.32 code and output from an affected
machine) simply enables interrupt re-mapping in this case?

Thanks, Jan

Re: iommu=0 leading to panic when system defaults to using x2apic

[Weidong Han]

Jan Beulich wrote:
>>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>>>         
>> The architectural requirement is actually between interrupt remapping and 
>> x2apic.  Since interrupt remapping is part of the VT-d feature so current 
>> software requires all VT-d features enabled in order for x2apic to be enabled.
>>
>> Strictly speaking DMA remapping is not required for x2apic.  However, queued 
>> invalidation is required since interrupt remapping requires queued 
>> invalidation.  So x2apic dependency is as follows:
>>
>>     x2apic->interrupt remapping->queued invalidation
>>
>> Due to historical reasons, the new VT-d features were built on top of the old 
>> ones as they become available.  Is there a requirement to separate this out?  
>> If so, we will need to re-design iommu boot parameter which took a while to 
>> get it right so most systems can now boot successfully.
>>     
>
> And why is it that Xen panics when the BIOS pre-enabled x2apic
> mode without also enabling interrupt re-mapping, while Linux
> (afaict from looking at 2.6.32 code and output from an affected
> machine) simply enables interrupt re-mapping in this case?
>
> Thanks, Jan
>
>   
x2apic spec 2.9 section states: The default will be for the BIOS to pass 
the control to the OS with the local x2APICs in xAPIC mode if all x2APIC 
IDs reported by CPUID.0BH:EDX are less than 255, and in x2APIC mode if 
there are any logical processor reporting its x2APIC ID at 255 or greater.

If BIOS transitions to Xen with x2APIC enabled, there must be APIC ID > 
255, which requires interrupt remapping. So we added code to check such 
BIOS issue.

Regards,
Weidong

Re: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 14.12.10 at 09:46, Weidong Han <weidong.han@intel.com> wrote:
> Jan Beulich wrote:
>> And why is it that Xen panics when the BIOS pre-enabled x2apic
>> mode without also enabling interrupt re-mapping, while Linux
>> (afaict from looking at 2.6.32 code and output from an affected
>> machine) simply enables interrupt re-mapping in this case?
>>
>> Thanks, Jan
>>
>>   
> x2apic spec 2.9 section states: The default will be for the BIOS to pass 
> the control to the OS with the local x2APICs in xAPIC mode if all x2APIC 
> IDs reported by CPUID.0BH:EDX are less than 255, and in x2APIC mode if 
> there are any logical processor reporting its x2APIC ID at 255 or greater.
> 
> If BIOS transitions to Xen with x2APIC enabled, there must be APIC ID > 
> 255, which requires interrupt remapping. So we added code to check such 
> BIOS issue.

So what would you tell a customer of yours reporting native Linux
to boot fine on a certain system, but Xen dying early?

Also, from a purely technical perspective, it doesn't seem like
interrupt re-mapping is a requirement for the BIOS: With
native Linux booting fine, there obviously is no issue. And
clearly, as long as the boot CPU has an APIC id fitting into 8
bits, interrupts ought to work fine without re-mapping.

When the boot CPU has a wider APIC ID, obviously the BIOS
won't be able to bring up the system (and hence the question
of booting whatever OS on top of it would be mute anyway).

Jan

RE: iommu=0 leading to panic when system defaults to using x2apic

[Kay, Allen M]

> And why is it that Xen panics when the BIOS pre-enabled x2apic
> mode without also enabling interrupt re-mapping, while Linux
> (afaict from looking at 2.6.32 code and output from an affected
> machine) simply enables interrupt re-mapping in this case?

AFAIK, Linux also requires interrupt remapping for x2apic.  I believe Weidong has fixed the panic issue in Xen when x2apic is enabled without enabling interrupt remapping.  It should now behave the same way as Linux.

Allen

-----Original Message-----
From: Jan Beulich [mailto:JBeulich@novell.com] 
Sent: Tuesday, December 14, 2010 12:22 AM
To: Kay, Allen M; Zhang, Yang Z
Cc: Han, Weidong; xen-devel@lists.xensource.com; Keir Fraser
Subject: RE: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

>>> On 10.12.10 at 19:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
> The architectural requirement is actually between interrupt remapping and 
> x2apic.  Since interrupt remapping is part of the VT-d feature so current 
> software requires all VT-d features enabled in order for x2apic to be enabled.
> 
> Strictly speaking DMA remapping is not required for x2apic.  However, queued 
> invalidation is required since interrupt remapping requires queued 
> invalidation.  So x2apic dependency is as follows:
> 
>     x2apic->interrupt remapping->queued invalidation
> 
> Due to historical reasons, the new VT-d features were built on top of the old 
> ones as they become available.  Is there a requirement to separate this out?  
> If so, we will need to re-design iommu boot parameter which took a while to 
> get it right so most systems can now boot successfully.

And why is it that Xen panics when the BIOS pre-enabled x2apic
mode without also enabling interrupt re-mapping, while Linux
(afaict from looking at 2.6.32 code and output from an affected
machine) simply enables interrupt re-mapping in this case?

Thanks, Jan

RE: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>  And why is it that Xen panics when the BIOS pre-enabled x2apic
>> mode without also enabling interrupt re-mapping, while Linux
>> (afaict from looking at 2.6.32 code and output from an affected
>> machine) simply enables interrupt re-mapping in this case?
> 
> AFAIK, Linux also requires interrupt remapping for x2apic.  I believe 
> Weidong has fixed the panic issue in Xen when x2apic is enabled without 
> enabling interrupt remapping.  It should now behave the same way as Linux.

No - just look at the very first if() in enable_bsp_x2apic().

Jan

RE: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>  And why is it that Xen panics when the BIOS pre-enabled x2apic
>> mode without also enabling interrupt re-mapping, while Linux
>> (afaict from looking at 2.6.32 code and output from an affected
>> machine) simply enables interrupt re-mapping in this case?
> 
> AFAIK, Linux also requires interrupt remapping for x2apic.  I believe 
> Weidong has fixed the panic issue in Xen when x2apic is enabled without 
> enabling interrupt remapping.  It should now behave the same way as Linux.

Oh, and I should have added that I'm currently polishing up
patches from Yinghai Lu to address this as well as other
problems preventing (certain?) large boxes from booting or
functioning correctly.

Jan

Re: iommu=0 leading to panic when system defaults to using x2apic

[Yinghai Lu]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On 12/14/2010 09:08 AM, Jan Beulich wrote:
>>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>>  And why is it that Xen panics when the BIOS pre-enabled x2apic
>>> mode without also enabling interrupt re-mapping, while Linux
>>> (afaict from looking at 2.6.32 code and output from an affected
>>> machine) simply enables interrupt re-mapping in this case?
>>
>> AFAIK, Linux also requires interrupt remapping for x2apic.  I believe 
>> Weidong has fixed the panic issue in Xen when x2apic is enabled without 
>> enabling interrupt remapping.  It should now behave the same way as Linux.
> 
> Oh, and I should have added that I'm currently polishing up
> patches from Yinghai Lu to address this as well as other
> problems preventing (certain?) large boxes from booting or
> functioning correctly.

Yes, just want to make xen aligned to current Linux kernel behavior.
BIOS guys said why Linux non-xen kernel is happy but xen kernel is not.

please check attached patches. Our big box with x2apic-preenabled, AND half cpus' apic id are above 255.

They are on top of xen-4.0.1. and first one is from Weidong in unstable.

Thanks

Yinghai Lu


[-- Attachment #2: patches_x2apic_preenabled_above_255_09010.tar.bz2 --]
[-- Type: application/x-bzip, Size: 7756 bytes --]

[-- Attachment #3: patches_x2apic_preenabled_above_255_dom0_2.6.32.13_0904.tar.bz2 --]
[-- Type: application/x-bzip, Size: 1599 bytes --]

[-- Attachment #4: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: iommu=0 leading to panic when system defaults to using x2apic

[Keir Fraser]

On 14/12/2010 17:52, "Yinghai Lu" <yinghai@kernel.org> wrote:

> On 12/14/2010 09:08 AM, Jan Beulich wrote:
>>>>> On 14.12.10 at 17:53, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>>>  And why is it that Xen panics when the BIOS pre-enabled x2apic
>>>> mode without also enabling interrupt re-mapping, while Linux
>>>> (afaict from looking at 2.6.32 code and output from an affected
>>>> machine) simply enables interrupt re-mapping in this case?
>>> 
>>> AFAIK, Linux also requires interrupt remapping for x2apic.  I believe
>>> Weidong has fixed the panic issue in Xen when x2apic is enabled without
>>> enabling interrupt remapping.  It should now behave the same way as Linux.
>> 
>> Oh, and I should have added that I'm currently polishing up
>> patches from Yinghai Lu to address this as well as other
>> problems preventing (certain?) large boxes from booting or
>> functioning correctly.
> 
> Yes, just want to make xen aligned to current Linux kernel behavior.
> BIOS guys said why Linux non-xen kernel is happy but xen kernel is not.
> 
> please check attached patches. Our big box with x2apic-preenabled, AND half
> cpus' apic id are above 255.
> 
> They are on top of xen-4.0.1. and first one is from Weidong in unstable.

I assume Jan is going to help with posting these as a regular patch series
for inclusion, with separate patches for 4.0 and 4.1 as appropriate.

 -- Keir

> Thanks
> 
> Yinghai Lu
> 

RE: iommu=0 leading to panic when system defaults to using x2apic

[Kay, Allen M]

OK, forget what I said about cluster mode in previous message.  I investigated into this in more detail today and here is what I found:

Keir's interpretation in previous message is basically correct.  Currently, both MSI and IOAPIC address register consists of 32 bits.  Bits 31:20 are occupied by 0xFEE so there aren't enough bits to accommodate full 32-bit APIC ID's.  As a result new format is define for x2apic to accommodate full 32-bit APIC ID's.

The problem is legacy MSI and IOAPIC devices still have 8-bit APIC ID's so interrupt remapping is used to be the intermediary between the legacy MSI devices and ioapics and CPUs in x2APIC mode.

Although it is possible to do turn on x2APIC on systems with less than 256 CPU's without using interrupt remapping, this configuration is not supported by the architecture in order to simplify HW validation.

Instead, BIOS is required to start the system in xAPIC mode for systems with less than 256 CPUs.  It will turn on x2APIC mode for systems with more than 256 CPU's.  On these systems, Xen will need to enable VT-d/interrupt remapping for x2APIC to work properly.

Allen 

-----Original Message-----
From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
Sent: Monday, December 13, 2010 11:45 PM
To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
Cc: Han, Weidong; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to using x2apic

Well, if it is a restriction imposed by cluster mode, you know the next
question is obvious: Why do we bother with cluster mode at all? I don't see
that it yields us any advantage over physical mode, and we could use
physical mode without interrupt remapping, that would seem to be a big bonus
and simplification? Could we just kill our x2apic cluster mode logic?

 -- Keir

On 14/12/2010 02:25, "Kay, Allen M" <allen.m.kay@intel.com> wrote:

> Keir/Jan,
> 
> My understanding is that cluster mode requires it.  I will get back to you
> guys after I dig out the details on this - did not get a chance to do this
> today.
> 
> Allen
> 
> -----Original Message-----
> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
> Sent: Monday, December 13, 2010 1:03 AM
> To: Jan Beulich; Kay, Allen M; Zhang, Yang Z
> Cc: Han, Weidong; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
> using x2apic
> 
> On 13/12/2010 08:15, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>>>>> On 11.12.10 at 01:07, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>> Yes, interrupt remapping is needed to be the intermediary between legacy
>>> IOxAPIC and MSI devices and the new x2APIC in the CPU.
>> 
>> But isn't this only when there are APIC IDs beyond 255?
> 
> Apparently not, since even Linux requires irq remapping even when none of
> the APIC IDs are greater than 255. Unless running on kvm or xen. I don't
> fully understand this particular restriction, mind you.
> 
> Actually, my guess is that x2apic mode requires a different format of APIC
> message with a 32-bit APICID field, legacy IOxAPIC and MSI devices do not
> support the new message format, and so irq remapping hardware is required to
> bridge the two formats, even if no actual irq remapping is occurring.
> 
> Is that a canny guess, Allen?
> 
>  -- Keir
> 
>> Jan
>> 
>>> -----Original Message-----
>>> From: Keir Fraser [mailto:keir.xen@gmail.com] On Behalf Of Keir Fraser
>>> Sent: Friday, December 10, 2010 10:50 AM
>>> To: Kay, Allen M; Jan Beulich; Zhang, Yang Z
>>> Cc: xen-devel@lists.xensource.com; Han, Weidong
>>> Subject: Re: [Xen-devel] iommu=0 leading to panic when system defaults to
>>> using x2apic
>>> 
>>> Ah, and the interrupt remapping dependency is because PCI(e) devices cannot
>>> address 32-bit APIC IDs?
>>> 
>>>  -- Keir
>>> 
>>> On 10/12/2010 18:26, "Kay, Allen M" <allen.m.kay@intel.com> wrote:
>>> 
>>>> The architectural requirement is actually between interrupt remapping and
>>>> x2apic.  Since interrupt remapping is part of the VT-d feature so current
>>>> software requires all VT-d features enabled in order for x2apic to be
>>> enabled.
>>>> 
>>>> Strictly speaking DMA remapping is not required for x2apic.  However,
>>>> queued
>>>> invalidation is required since interrupt remapping requires queued
>>>> invalidation.  So x2apic dependency is as follows:
>>>> 
>>>>     x2apic->interrupt remapping->queued invalidation
>>>> 
>>>> Due to historical reasons, the new VT-d features were built on top of the
>>>> old
>>>> ones as they become available.  Is there a requirement to separate this
>>>> out?
>>>> If so, we will need to re-design iommu boot parameter which took a while to
>>>> get it right so most systems can now boot successfully.
>>>> 
>>>> Allen
>> 
>> 
> 
> 

Re: iommu=0 leading to panic when system defaults to using x2apic

[Jan Beulich]

>>> On 14.12.10 at 19:18, Keir Fraser <keir@xen.org> wrote:
> On 14/12/2010 17:52, "Yinghai Lu" <yinghai@kernel.org> wrote:
>> please check attached patches. Our big box with x2apic-preenabled, AND half
>> cpus' apic id are above 255.
>> 
>> They are on top of xen-4.0.1. and first one is from Weidong in unstable.
> 
> I assume Jan is going to help with posting these as a regular patch series
> for inclusion, with separate patches for 4.0 and 4.1 as appropriate.

Yes indeed.

Jan