PCI Express

PCI Express

[Colin Ngam]

Hi,

Is MSI/MSIX support available for ia64?

Are PCIE devices supported in ia64?

A pointer to any documentations with regards to PCIE and ia64 will be very much
appreciated.

Thanks.

colin

Re: PCI Express

[Grant Grundler]

On Fri, Mar 04, 2005 at 05:42:08PM -0600, Colin Ngam wrote:
> Is MSI/MSIX support available for ia64?

Yes

> Are PCIE devices supported in ia64?

I'm having trouble with the word "supported".
You have a PCI-E Bridge and firmware that can talk to it
hooked up to an ia64 box?

Anyway, linux has some PCI-E support in the generic PCI code.

> A pointer to any documentations with regards to PCIE and ia64 will
> be very much appreciated.

I don't think anything ia64 or linux specific exists.

grant

Re: PCI Express

[Matthew Wilcox]

On Mon, Mar 07, 2005 at 08:48:13AM -0800, Grant Grundler wrote:
> On Fri, Mar 04, 2005 at 05:42:08PM -0600, Colin Ngam wrote:
> > Are PCIE devices supported in ia64?
> 
> I'm having trouble with the word "supported".
> You have a PCI-E Bridge and firmware that can talk to it
> hooked up to an ia64 box?
> 
> Anyway, linux has some PCI-E support in the generic PCI code.

... and I wrote the ia64-specific bits to access extended PCI config space
through the SAL PCI accessors.  See PCI_SAL_EXT_ADDRESS().

Various PCIe features are in the process of being added.  It all depends
what you mean when you say "supported".

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: PCI Express

[Mark Maule]

On Mon, 7 Mar 2005, Matthew Wilcox wrote:

> On Mon, Mar 07, 2005 at 08:48:13AM -0800, Grant Grundler wrote:
> > On Fri, Mar 04, 2005 at 05:42:08PM -0600, Colin Ngam wrote:
> > > Are PCIE devices supported in ia64?
> > 
> > I'm having trouble with the word "supported".
> > You have a PCI-E Bridge and firmware that can talk to it
> > hooked up to an ia64 box?
> > 
> > Anyway, linux has some PCI-E support in the generic PCI code.
> 
> ... and I wrote the ia64-specific bits to access extended PCI config space
> through the SAL PCI accessors.  See PCI_SAL_EXT_ADDRESS().
> 
> Various PCIe features are in the process of being added.  It all depends
> what you mean when you say "supported".
> 

Are there specific lists (other than this one) to be watching for
PCIe discussions?

thanks
Mark

Re: PCI Express

[Matthew Wilcox]

On Mon, Mar 07, 2005 at 11:02:38AM -0600, Mark Maule wrote:
> On Mon, 7 Mar 2005, Matthew Wilcox wrote:
> > Various PCIe features are in the process of being added.  It all depends
> > what you mean when you say "supported".
> 
> Are there specific lists (other than this one) to be watching for
> PCIe discussions?

There's linux-pci@atrey.karlin.mff.cuni.cz and some stuff also gets
discussed on l-k.  There's not a whole lot of ia64-specific content,
so I would only expect it to come up rarely on this list.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: PCI Express

[Colin Ngam]

Grant Grundler wrote:

Hi Grant,

Thanks for your respond.

>On Fri, Mar 04, 2005 at 05:42:08PM -0600, Colin Ngam wrote:
>  
>
>>Is MSI/MSIX support available for ia64?
>>    
>>
>
>Yes
>
I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and 
it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I did not 
find this configured on in any sample ia64 config files, but it does 
exist in defconfig for i386.  Is this just an ia32 config option?

>
>  
>
>>Are PCIE devices supported in ia
>>
>>    
>>
>
>I'm having trouble with the word "supported".
>You have a PCI-E Bridge and firmware that can talk to it
>hooked up to an ia64 box?
>
Pretty soon.

Is there any ia64 box with MSI and PCIE actually "activated and running"?

Thanks.

colin

>
>Anyway, linux has some PCI-E support in the generic PCI code.
>
>  
>
>>A pointer to any documentations with regards to PCIE and ia64 will
>>be very much appreciated.
>>    
>>
>
>I don't think anything ia64 or linux specific exists.
>
>grant
>-
>To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>

RE: PCI Express

[Nguyen, Tom L]

On Monday, March 07, 2005 12:41 PM Colin Ngam wrote:
> I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and

> it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I did
not 
> find this configured on in any sample ia64 config files, but it does 
> exist in defconfig for i386.  Is this just an ia32 config option?

IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar to
IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration
option therefore is always available in IA64.

Thanks,
Long

Re: PCI Express

[Jesse Barnes]

On Monday, March 7, 2005 3:03 pm, Nguyen, Tom L wrote:
> On Monday, March 07, 2005 12:41 PM Colin Ngam wrote:
> > I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and
> >
> > it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I did
>
> not
>
> > find this configured on in any sample ia64 config files, but it does
> > exist in defconfig for i386.  Is this just an ia32 config option?
>
> IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar to
> IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration
> option therefore is always available in IA64.

So the MSIs are programmed to point at the processor SAPIC block?  I think 
that'll work for us on Altix, but if they're pointed at an external Intel (or 
compatible) interrupt controller, we'll have to write new code for Altix.

Jesse

Re: PCI Express

[Grant Grundler]

On Mon, Mar 07, 2005 at 02:40:42PM -0600, Colin Ngam wrote:
> I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11) and 
> it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I did not 
> find this configured on in any sample ia64 config files, but it does 
> exist in defconfig for i386.  Is this just an ia32 config option?

yes

> >I'm having trouble with the word "supported".
> >You have a PCI-E Bridge and firmware that can talk to it
> >hooked up to an ia64 box?
> Pretty soon.
> Is there any ia64 box with MSI and PCIE actually "activated and running"?

I've not seen anything published yet.

*Some* x86-64 boxes have working MSI (and PCI-E).
Recent PCI quirks have been submitted to flag the AMD chipsets that don't
support MSI or MSI-X.

grant

RE: PCI Express

[Nguyen, Tom L]

On Monday, March 07, 2005 3:12 PM Jesse Barnes wrote:

>On Monday, March 7, 2005 3:03 pm, Nguyen, Tom L wrote:
>> On Monday, March 07, 2005 12:41 PM Colin Ngam wrote:
>> > I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11)
and
>> >
>> > it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I
did
>>
>> not
>>
>> > find this configured on in any sample ia64 config files, but it
does
>> > exist in defconfig for i386.  Is this just an ia32 config option?
>>
>> IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar
to
>> IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration
>> option therefore is always available in IA64.

>So the MSIs are programmed to point at the processor SAPIC block?  I
think 
>that'll work for us on Altix, but if they're pointed at an external
Intel >(or 
>compatible) interrupt controller, we'll have to write new code for
Altix.

MSI/MSI-X message address is programmed to point at specific processor,
not an external Intel interrupt controller, as a target.

Thanks,
Long

Re: PCI Express

[Grant Grundler]

On Mon, Mar 07, 2005 at 03:11:46PM -0800, Jesse Barnes wrote:
> So the MSIs are programmed to point at the processor SAPIC block?  I think 
> that'll work for us on Altix, but if they're pointed at an external Intel (or 
> compatible) interrupt controller, we'll have to write new code for Altix.

Assuming your "Processor Interrupt Block" (IIRC) is at 0xfee00000
it should just work.

grant

Re: PCI Express

[Jesse Barnes]

On Monday, March 7, 2005 3:36 pm, Grant Grundler wrote:
> On Mon, Mar 07, 2005 at 03:11:46PM -0800, Jesse Barnes wrote:
> > So the MSIs are programmed to point at the processor SAPIC block?  I
> > think that'll work for us on Altix, but if they're pointed at an external
> > Intel (or compatible) interrupt controller, we'll have to write new code
> > for Altix.
>
> Assuming your "Processor Interrupt Block" (IIRC) is at 0xfee00000
> it should just work.

Cool, I was hoping it just pointed at the same place we use for IPIs.  It 
*should* work on Altix then, assuming the MSI allocation routines don't have 
platform knowledge in them (like ACPI bits or something).

Thanks,
Jesse

Re: PCI Express

[Grant Grundler]

On Mon, Mar 07, 2005 at 03:40:56PM -0800, Jesse Barnes wrote:
> Cool, I was hoping it just pointed at the same place we use for IPIs.  It 
> *should* work on Altix then, assuming the MSI allocation routines don't have 
> platform knowledge in them (like ACPI bits or something).

I submitted a patch to add MSI support to tg3:
	ftp://ftp.parisc-linux.org/patches/diff-2.6.10-tg3_MSI-03

It was rejected because of HW bugs in bcm57xx chips.
But it seems to work fine for me on the HP ZX1 platforms.
Maybe it's worth trying on the altix boxes as well.

grant

Re: PCI Express

[Colin Ngam]

"Nguyen, Tom L" wrote:

> On Monday, March 07, 2005 3:12 PM Jesse Barnes wrote:
>
> >On Monday, March 7, 2005 3:03 pm, Nguyen, Tom L wrote:
> >> On Monday, March 07, 2005 12:41 PM Colin Ngam wrote:
> >> > I looked at Documentation/MSI-HOWTO.txt(linux-ia64-release-2.6.11)
> and
> >> >
> >> > it mentioned that CONFIG_X86_LOCAL_APIC has to be configured.  I
> did
> >>
> >> not
> >>
> >> > find this configured on in any sample ia64 config files, but it
> does
> >> > exist in defconfig for i386.  Is this just an ia32 config option?
> >>
> >> IA64 uses SAPIC instead of IOxAPIC. SAPIC, which is somewhat similar
> to
> >> IOxAPIC, is included in IA64 kernel. MSI/MSI-X support configuration
> >> option therefore is always available in IA64.
>
> >So the MSIs are programmed to point at the processor SAPIC block?  I
> think
> >that'll work for us on Altix, but if they're pointed at an external
> Intel >(or
> >compatible) interrupt controller, we'll have to write new code for
> Altix.
>
> MSI/MSI-X message address is programmed to point at specific processor,
> not an external Intel interrupt controller, as a target.

Hi Long/All,

Thank you (all) very much for the information.

It's been a long time since we tested MSI/MSIX on our Altix boxes.  It has been
longer since I looked at the code.  It works and we are looking forward to
hooking them up with the current Infrastructure available on ia64 -
pci_enable|disable_msi().

On Altix systems, we have a set of "Interrupt Registers" in Memory Address Space
that is initialized to target specific CPUs.  The way we initialize a card's MSI
is:

1.  The Target Address is One of these "Interrupt Registers"
2.  The Data Payload is the IRQ plus some special Altix bits.

This memory write causes the "Interrupt Chipset" to generate a LINTR message to
the configured targeted cpu with the IRQ.  Ofcourse, these registers are Altix
Platform specific.  Moreover, we have chunks of these registers all over the
place.

Is there a more direct mechanism to generate an interrupt(LINTR Message) to a
Processor?  1 of the Special bits that I mentioned in Item 2 above causes our
hardware to flush all posted DMA buffers before allowing the LINTR Message to be
generated to the cpu.

With Altix, it's always interesting, with respect to "just work" with typical
ia64 code base.  Hopefully, it will work cleaner compared to some of our other
efforts.

Are you the maintainer for the MSI code?

Thanks.

colin

>
>
> Thanks,
> Long

Re: PCI Express

[Jesse Barnes]

On Monday, March 7, 2005 8:40 pm, Colin Ngam wrote:
> It's been a long time since we tested MSI/MSIX on our Altix boxes.  It has
> been longer since I looked at the code.  It works and we are looking
> forward to hooking them up with the current Infrastructure available on
> ia64 - pci_enable|disable_msi().
>
> On Altix systems, we have a set of "Interrupt Registers" in Memory Address
> Space that is initialized to target specific CPUs.  The way we initialize a
> card's MSI is:
>
> 1.  The Target Address is One of these "Interrupt Registers"
> 2.  The Data Payload is the IRQ plus some special Altix bits.
>
> This memory write causes the "Interrupt Chipset" to generate a LINTR
> message to the configured targeted cpu with the IRQ.  Ofcourse, these
> registers are Altix Platform specific.  Moreover, we have chunks of these
> registers all over the place.

That's one way to do it, but is obviously very Altix specific.

> Is there a more direct mechanism to generate an interrupt(LINTR Message) to
> a Processor?  1 of the Special bits that I mentioned in Item 2 above causes
> our hardware to flush all posted DMA buffers before allowing the LINTR
> Message to be generated to the cpu.

Yes, see Grant's earlier message about the processor interrupt block.  If we 
target MSIs at that, they'll behave the same way as IPIs, which are already 
platform independent (i.e. they work on Altix, zx1, tiger, etc.) since the 
processor interrupt block is builtin to the CPU.  See Section 5.8.4 of Vol. 2 
of the Itanium Arch. Software Developer's Manual (the blue books).

Jesse

RE: PCI Express

[Nguyen, Tom L]

On Monday, March 07, 2005 8:41 PM Colin wrote:
> On Altix systems, we have a set of "Interrupt Registers" in Memory
Address
> Space that is initialized to target specific CPUs.  The way we 
> initialize a card's MSI is:
>
> 1.  The Target Address is One of these "Interrupt Registers"
> 2.  The Data Payload is the IRQ plus some special Altix bits.
>
> This memory write causes the "Interrupt Chipset" to generate a LINTR
> message to the configured targeted cpu with the IRQ.  Ofcourse, these
> registers are Altix Platform specific.  Moreover, we have chunks of
these
> registers all over the place.

Your solution is Altix chipset specific. MSI affects other architectures
too. General solution would almost certainly be a better one.

> Is there a more direct mechanism to generate an interrupt(LINTR
Message) 
> to a Processor?  

I agree with Jesse's comment.

> With Altix, it's always interesting, with respect to "just work" with 
> typical ia64 code base.  Hopefully, it will work cleaner compared to 
> some of our other efforts.
>
> Are you the maintainer for the MSI code?

I suggest you move any discussion or any proposed changes to LKML
because MSI affects other architectures too. Again, general solution
would almost certainly be a better one.

Thanks,
Long

Re: PCI Express

[Colin Ngam]

"Nguyen, Tom L" wrote:

> On Monday, March 07, 2005 8:41 PM Colin wrote:
> > On Altix systems, we have a set of "Interrupt Registers" in Memory
> Address
> > Space that is initialized to target specific CPUs.  The way we
> > initialize a card's MSI is:
> >
> > 1.  The Target Address is One of these "Interrupt Registers"
> > 2.  The Data Payload is the IRQ plus some special Altix bits.
> >
> > This memory write causes the "Interrupt Chipset" to generate a LINTR
> > message to the configured targeted cpu with the IRQ.  Ofcourse, these
> > registers are Altix Platform specific.  Moreover, we have chunks of
> these
> > registers all over the place.
>
> Your solution is Altix chipset specific. MSI affects other architectures
> too. General solution would almost certainly be a better one.

Hi All,

Yes it is and we are looking for a generic solution.

>
>
> > Is there a more direct mechanism to generate an interrupt(LINTR
> Message)
> > to a Processor?
>
> I agree with Jesse's comment.

Well, unfortunately, we do not send IPI by using the Processor Interrupt Block.
We actually
target a Special Altix Chipset "IPI Interrupt" register that ends up generating
an IPI.  That is why, we
have Platform Specific "send ipi" calls on ia64:

platform_send_ipi()
  File          Line
0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
1 machvec.h     282 #define platform_send_ipi ia64_send_ipi
2 machvec_sn2.h  86 #define platform_send_ipi sn2_send_IPI
3 machvec.h      95 #define platform_send_ipi ia64_mv.send_ipi
4 machvec.h     255 #define platform_send_ipi ia64_send_ipi
5 machvec_sn2.h  83 #define platform_send_ipi sn2_send_IPI

I do not know why we do not use the Processor Interrupt Block, but I will find
out.  But this does
not mean that we cannot use the PIB for MSI.  Ofcourse, it may not be relocated
at
0x0000 0000 FEE0 0000.

>
>
> > With Altix, it's always interesting, with respect to "just work" with
> > typical ia64 code base.  Hopefully, it will work cleaner compared to
> > some of our other efforts.
> >
> > Are you the maintainer for the MSI code?
>
> I suggest you move any discussion or any proposed changes to LKML
> because MSI affects other architectures too. Again, general solution
> would almost certainly be a better one.

We will.  We are looking for a general solution.

We will start looking through the MSI code see how best we can implement
this functionality to support Altix platform.  At that point we will post to
lkml for guidance.

Thanks.

colin

>
>
> Thanks,
> Long
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PCI Express

[Jesse Barnes]

On Tuesday, March 8, 2005 3:48 pm, Colin Ngam wrote:
> Well, unfortunately, we do not send IPI by using the Processor Interrupt
> Block. We actually
> target a Special Altix Chipset "IPI Interrupt" register that ends up
> generating an IPI.  That is why, we
> have Platform Specific "send ipi" calls on ia64:
>
> platform_send_ipi()
>   File          Line
> 0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
> 1 machvec.h     282 #define platform_send_ipi ia64_send_ipi
> 2 machvec_sn2.h  86 #define platform_send_ipi sn2_send_IPI
> 3 machvec.h      95 #define platform_send_ipi ia64_mv.send_ipi
> 4 machvec.h     255 #define platform_send_ipi ia64_send_ipi
> 5 machvec_sn2.h  83 #define platform_send_ipi sn2_send_IPI
>
> I do not know why we do not use the Processor Interrupt Block, but I will
> find out.  But this does
> not mean that we cannot use the PIB for MSI.  Ofcourse, it may not be
> relocated at
> 0x0000 0000 FEE0 0000.

That's beside the point though--MSIs don't use IPIs, they just use the 
processor interrupt block, which is also used by IPIs (on non-sn2 platforms).  
If the processor interrupt block works for us (pretty sure it does) MSIs 
targetted there should also work.  The problem for us is if a processor tries 
to IPI a processor on a different node with the processor interrupt block, 
which I don't think MSIs will try to do.

Jesse

Re: PCI Express

[Colin Ngam]

Jesse Barnes wrote:

> On Tuesday, March 8, 2005 3:48 pm, Colin Ngam wrote:
> > Well, unfortunately, we do not send IPI by using the Processor Interrupt
> > Block. We actually
> > target a Special Altix Chipset "IPI Interrupt" register that ends up
> > generating an IPI.  That is why, we
> > have Platform Specific "send ipi" calls on ia64:
> >
> > platform_send_ipi()
> >   File          Line
> > 0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
> > 1 machvec.h     282 #define platform_send_ipi ia64_send_ipi
> > 2 machvec_sn2.h  86 #define platform_send_ipi sn2_send_IPI
> > 3 machvec.h      95 #define platform_send_ipi ia64_mv.send_ipi
> > 4 machvec.h     255 #define platform_send_ipi ia64_send_ipi
> > 5 machvec_sn2.h  83 #define platform_send_ipi sn2_send_IPI
> >
> > I do not know why we do not use the Processor Interrupt Block, but I will
> > find out.  But this does
> > not mean that we cannot use the PIB for MSI.  Ofcourse, it may not be
> > relocated at
> > 0x0000 0000 FEE0 0000.

Hi Jesse,

>
>
> That's beside the point though--MSIs don't use IPIs, they just use the
> processor interrupt block, which is also used by IPIs (on non-sn2 platforms).
> If the processor interrupt block works for us (pretty sure it does) MSIs

That's what I will find out before deciding to use it.  It is not used for IPIs
on our platform, probably for very good reasons.  If we can and the address is
the same than it is a no brainer.

>
> targetted there should also work.  The problem for us is if a processor tries
> to IPI a processor on a different node with the processor interrupt block,
> which I don't think MSIs will try to do.

Perhaps you can elaborate.  I am not sure what you are trying to say here.

Thanks.

colin

>
>
> Jesse

Re: PCI Express

[Colin Ngam]

Colin Ngam wrote:

> Jesse Barnes wrote:
>
> > On Tuesday, March 8, 2005 3:48 pm, Colin Ngam wrote:
> > > Well, unfortunately, we do not send IPI by using the Processor Interrupt
> > > Block. We actually
> > > target a Special Altix Chipset "IPI Interrupt" register that ends up
> > > generating an IPI.  That is why, we
> > > have Platform Specific "send ipi" calls on ia64:
> > >
> > > platform_send_ipi()
> > >   File          Line
> > > 0 machvec.h     113 #define platform_send_ipi ia64_mv.send_ipi
> > > 1 machvec.h     282 #define platform_send_ipi ia64_send_ipi
> > > 2 machvec_sn2.h  86 #define platform_send_ipi sn2_send_IPI
> > > 3 machvec.h      95 #define platform_send_ipi ia64_mv.send_ipi
> > > 4 machvec.h     255 #define platform_send_ipi ia64_send_ipi
> > > 5 machvec_sn2.h  83 #define platform_send_ipi sn2_send_IPI
> > >
> > > I do not know why we do not use the Processor Interrupt Block, but I will
> > > find out.  But this does
> > > not mean that we cannot use the PIB for MSI.  Ofcourse, it may not be
> > > relocated at
> > > 0x0000 0000 FEE0 0000.
>
> Hi Jesse,
>
> >
> >
> > That's beside the point though--MSIs don't use IPIs, they just use the
> > processor interrupt block, which is also used by IPIs (on non-sn2 platforms).
> > If the processor interrupt block works for us (pretty sure it does) MSIs
>
> That's what I will find out before deciding to use it.  It is not used for IPIs
> on our platform, probably for very good reasons.  If we can and the address is
> the same than it is a no brainer.
>
> >
> > targetted there should also work.  The problem for us is if a processor tries
> > to IPI a processor on a different node with the processor interrupt block,
> > which I don't think MSIs will try to do.
>
> Perhaps you can elaborate.  I am not sure what you are trying to say here.
>
> Thanks.
>
> colin
>
> >
> >
> > Jesse

Hi Jesse,

I beleive the reason why we do not use the Processor Interrupt Block for
generating interrupts is because it does not have the range to reach all the nodes
that we can have in the system.

Thanks.

colin

>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PCI Express

[Jesse Barnes]

On Tuesday, March 08, 2005 4:13 pm, Colin Ngam wrote:
> That's what I will find out before deciding to use it.  It is not used for
> IPIs on our platform, probably for very good reasons.

Yep, because IPIs generated from the local block to a distant processor won't 
work otherwise (afaik).

> If we can and the 
> address is the same than it is a no brainer.
>
> > targetted there should also work.  The problem for us is if a processor
> > tries to IPI a processor on a different node with the processor interrupt
> > block, which I don't think MSIs will try to do.
>
> Perhaps you can elaborate.  I am not sure what you are trying to say here.

The SHub manual is a little vague here, but my understanding is that in order 
to send remote IPIs we need to use the SHub, whereas local IPIs (i.e. 
processor to itself or to the other CPU on the same FSB) will work fine.

An MSI should behave like a processor sending an IPI to itself since its 
address can be targeted at the processor's interrupt block and set to 
generate a local interrupt.  Is that right, Tom & Grant?

If this won't work for us, no biggie, we just have to abstract things a little 
more.  We could make the MSI into a platform specific cookie that we can 
store a SHub/PIC/TIO address in and other platforms can use to target the 
processor interrupt block.  Hopefully we won't have to do that though, since 
it would require a few more changes.

Jesse

Re: PCI Express

[Colin Ngam]

Jesse Barnes wrote:

> On Tuesday, March 08, 2005 4:13 pm, Colin Ngam wrote:
> > That's what I will find out before deciding to use it.  It is not used for
> > IPIs on our platform, probably for very good reasons.
>
> Yep, because IPIs generated from the local block to a distant processor won't
> work otherwise (afaik).
>
> > If we can and the
> > address is the same than it is a no brainer.
> >
> > > targetted there should also work.  The problem for us is if a processor
> > > tries to IPI a processor on a different node with the processor interrupt
> > > block, which I don't think MSIs will try to do.
> >
> > Perhaps you can elaborate.  I am not sure what you are trying to say here.
>
> The SHub manual is a little vague here, but my understanding is that in order
> to send remote IPIs we need to use the SHub, whereas local IPIs (i.e.
> processor to itself or to the other CPU on the same FSB) will work fine.
>
> An MSI should behave like a processor sending an IPI to itself since its
> address can be targeted at the processor's interrupt block and set to
> generate a local interrupt.  Is that right, Tom & Grant?
>
> If this won't work for us, no biggie, we just have to abstract things a little
> more.  We could make the MSI into a platform specific cookie that we can
> store a SHub/PIC/TIO address in and other platforms can use to target the
> processor interrupt block.  Hopefully we won't have to do that though, since
> it would require a few more changes.

May be the way we go.  Needs more perusal of the MSI code.

Thanks.

colin

>
>
> Jesse
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: PCI Express

[Grant Grundler]

On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
...
> An MSI should behave like a processor sending an IPI to itself since its 
> address can be targeted at the processor's interrupt block and set to 
> generate a local interrupt.  Is that right, Tom & Grant?

Yes - that's how I understand it too.

> If this won't work for us, no biggie, we just have to abstract things a
> little more.  We could make the MSI into a platform specific cookie that
> we can store a SHub/PIC/TIO address in and other platforms can use to
> target the processor interrupt block.

Couple more options:
o add hooks for MSI quirk/fixup for *after* the MSI has been assigned
  by generic code.
  
o make sure all MSI only go to "node local" processors. I thought
  platform code has some control over the CPU EID and data portion of
  the MSI. But I haven't looked over that in a few monthes.

grant

Re: PCI Express

[Colin Ngam]

Grant Grundler wrote:

Hi Grant,

Thanks for the info.

Bottomline - Processor Interrupt Block not implemented on Altix.

Thanks.

colin

> On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
> ...
> > An MSI should behave like a processor sending an IPI to itself since its
> > address can be targeted at the processor's interrupt block and set to
> > generate a local interrupt.  Is that right, Tom & Grant?
>
> Yes - that's how I understand it too.
>
> > If this won't work for us, no biggie, we just have to abstract things a
> > little more.  We could make the MSI into a platform specific cookie that
> > we can store a SHub/PIC/TIO address in and other platforms can use to
> > target the processor interrupt block.
>
> Couple more options:
> o add hooks for MSI quirk/fixup for *after* the MSI has been assigned
>   by generic code.
>
> o make sure all MSI only go to "node local" processors. I thought
>   platform code has some control over the CPU EID and data portion of
>   the MSI. But I haven't looked over that in a few monthes.
>
> grant
> -
> To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: PCI Express

[Nguyen, Tom L]

On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
...
> An MSI should behave like a processor sending an IPI to itself since
its 
> address can be targeted at the processor's interrupt block and set to 
> generate a local interrupt.  Is that right, Tom & Grant?

MSI address is programmed at 0xfeex_xxxx (based on PCI Specs) to
generate a memory write to FSB. This address can be configured to target
a platform interrupt controller (like IOAPIC for example), which in turn
generates a memory write to a FSB, or be configured to send a memory
write directly to FSB. Existing MSI support implements a direct
memory-write mechanism (generic solution without platform dependency and
limitation to MSI-X support as an example). With existing direct
memory-write mechanism, a device's MSI address is configured with
current running CPU as a target. MSI support also implements MSI SMP
affinity, which supports IRQ rebalance and allows user to select which
CPU target through /proc/irq/...

I hope it helps.

Thanks,
Long 

Re: PCI Express

[Grant Grundler]

On Wed, Mar 09, 2005 at 08:35:00AM -0800, Nguyen, Tom L wrote:
> Existing MSI support implements a direct
> memory-write mechanism (generic solution without platform dependency and
> limitation to MSI-X support as an example). With existing direct
> memory-write mechanism, a device's MSI address is configured with
> current running CPU as a target.

At some point, PARISC (and Alpha?) will also want to support MSI
using "direct memory write". But like Altix, does not implement
0xfeeXXXXX (1) PIB since PARISC arch predates PCI by a decade or so.

We'll have to figure out which platform specific hooks would enable
parisc/alpha/altix to use existing PCI MSI/MSI-X support.

PA-RISC IPI is as straight forward as Jesse described - just write
to a per CPU address with a data value (vector).

thanks,
grant

(1) the most recent two or three PA-RISC chipsets do implement 0xfeeXXXXX
interrupt block. But firmware hands us the CPU "ID/EID" values to use
when programming the IO SAPIC IRT. See drivers/parisc/iosapic.c
iosapic_set_irt_data() for more details.

Bottomline is, even if we know it's implemented in HW, we don't know if
we are using 0xfeeXXXXX or not for IO SAPIC today. platform hooks
into MSI code would have to sort that out.

Re: PCI Express

[Colin Ngam]

Grant Grundler wrote:

>On Wed, Mar 09, 2005 at 08:35:00AM -0800, Nguyen, Tom L wrote:
>  
>
>>Existing MSI support implements a direct
>>memory-write mechanism (generic solution without platform dependency and
>>limitation to MSI-X support as an example). With existing direct
>>memory-write mechanism, a device's MSI address is configured with
>>current running CPU as a target.
>>    
>>
>
>At some point, PARISC (and Alpha?) will also want to support MSI
>using "direct memory write". But like Altix, does not implement
>0xfeeXXXXX (1) PIB since PARISC arch predates PCI by a decade or so.
>
>We'll have to figure out which platform specific hooks would enable
>parisc/alpha/altix to use existing PCI MSI/MSI-X support.
>
Hi Grant,

Sounds great.

Thanks.

colin

>
>PA-RISC IPI is as straight forward as Jesse described - just write
>to a per CPU address with a data value (vector).
>
>thanks,
>grant
>
>(1) the most recent two or three PA-RISC chipsets do implement 0xfeeXXXXX
>interrupt block. But firmware hands us the CPU "ID/EID" values to use
>when programming the IO SAPIC IRT. See drivers/parisc/iosapic.c
>iosapic_set_irt_data() for more details.
>
>Bottomline is, even if we know it's implemented in HW, we don't know if
>we are using 0xfeeXXXXX or not for IO SAPIC today. platform hooks
>into MSI code would have to sort that out.
>  
>

Re: PCI Express

[Grant Grundler]

On Wed, Mar 09, 2005 at 11:42:39AM -0600, Colin Ngam wrote:
> >We'll have to figure out which platform specific hooks would enable
> >parisc/alpha/altix to use existing PCI MSI/MSI-X support.
>
> Sounds great.

Indeed. :^)

I forgot to mention I've had this on my wish list for ~5 years...
http://lists.parisc-linux.org/pipermail/parisc-linux/1999-December/008066.html

grant

Re: PCI Express

[Colin Ngam]

Nguyen, Tom L wrote:

>On Tue, Mar 08, 2005 at 05:29:13PM -0800, Jesse Barnes wrote:
>...
>  
>
>>An MSI should behave like a processor sending an IPI to itself since
>>    
>>
>its 
>  
>
>>address can be targeted at the processor's interrupt block and set to 
>>generate a local interrupt.  Is that right, Tom & Grant?
>>    
>>
>
>MSI address is programmed at 0xfeex_xxxx (based on PCI Specs) to
>
Hi Tom,

The MSI address portion allows 64bits for the address.  It should be an 
address whereby a specific Platform has configured with sufficient 
information to generate an Interrupt to the targeted "cpu" with the 
payload - IRQ.

I am very much interested in your assertion that the MSI address is 
defined by PCI Spec to be 0xfeex_xxxx.  Can you kindly point me to the 
relevent section?  Perhaps if you have an online copy, can you cut and 
paste the relevent section?

Thanks.

colin

>generate a memory write to FSB. This address can be configured to target
>a platform interrupt controller (like IOAPIC for example), which in turn
>generates a memory write to a FSB, or be configured to send a memory
>write directly to FSB. Existing MSI support implements a direct
>memory-write mechanism (generic solution without platform dependency and
>limitation to MSI-X support as an example). With existing direct
>memory-write mechanism, a device's MSI address is configured with
>current running CPU as a target. MSI support also implements MSI SMP
>affinity, which supports IRQ rebalance and allows user to select which
>CPU target through /proc/irq/...
>
>I hope it helps.
>
>Thanks,
>Long 
>  
>

Re: PCI Express

[Grant Grundler]

On Wed, Mar 09, 2005 at 12:12:37PM -0600, Colin Ngam wrote:
> The MSI address portion allows 64bits for the address.

Yes - *allows*. It also allows devices to implement 32-bit MSI address.
MSI "Control" bits indicate what is implemented.
See PCI2.2 spec or later and search for MSI.

> I am very much interested in your assertion that the MSI address is 
> defined by PCI Spec to be 0xfeex_xxxx.
...
> Can you kindly point me to the relevent section?

It's not and it doesn't belong there.

IA64 specs defines Processor Interrupt Block and where it lives.
See "Intel® Itanium® Software Developer's Manual"
	Volume 2, Part 1, Section 5.8.4
	Volume 2, Part 1, Section 11.9.3


grant

RE: PCI Express

[Nguyen, Tom L]

On Wednesday, March 09, 2005 10:48 AM Grant Grundler wrote:

>> I am very much interested in your assertion that the MSI address is 
>> defined by PCI Spec to be 0xfeex_xxxx.
>...
>> Can you kindly point me to the relevent section?
>
>It's not and it doesn't belong there.
>
>IA64 specs defines Processor Interrupt Block and where it lives.
>See "Intel(r) Itanium(r) Software Developer's Manual"
>	Volume 2, Part 1, Section 5.8.4
>	Volume 2, Part 1, Section 11.9.3

I apologize that it is not in PCI Spec. It is also in Section 8.11 of
"IA-32 Intel Architecture Software Developer's Manual (volume 3)."

Thanks,
Long

Re: PCI Express

[Colin Ngam]

Nguyen, Tom L wrote:

>On Wednesday, March 09, 2005 10:48 AM Grant Grundler wrote:
>
>  
>
>>>I am very much interested in your assertion that the MSI address is 
>>>defined by PCI Spec to be 0xfeex_xxxx.
>>>      
>>>
>>...
>>    
>>
>>>Can you kindly point me to the relevent section?
>>>      
>>>
>>It's not and it doesn't belong there.
>>
>>IA64 specs defines Processor Interrupt Block and where it lives.
>>See "Intel(r) Itanium(r) Software Developer's Manual"
>>	Volume 2, Part 1, Section 5.8.4
>>	Volume 2, Part 1, Section 11.9.3
>>    
>>
>
>I apologize that it is not in PCI Spec. It is also in Section 8.11 of
>"IA-32 Intel Architecture Software Developer's Manual (volume 3)."
>

Hi,

Thanks Tom.

I believe we are all are in agreement that the MSI Address needs to be 
Platform "Tunable" :-)

Thank you.

colin

>
>Thanks,
>Long
>-
>To unsubscribe from this list: send the line "unsubscribe linux-ia64" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  
>