MSI to memory?

MSI to memory?

[Bjorn Helgaas]

The conventional use of MSI is for a PCI adapter to generate processor
interrupts by writing to a local APIC.  But I've seen some things
that lead me to believe it would also allow an adapter to write to
things other than a local APIC, i.e., to memory.

If so, is that a useful capability that should be exposed through
the Linux MSI interface?

Re: MSI to memory?

[Rajesh Shah]

On Thu, Jul 01, 2004 at 12:15:59PM -0600, Bjorn Helgaas wrote:
> The conventional use of MSI is for a PCI adapter to generate processor
> interrupts by writing to a local APIC.  But I've seen some things

On Intel architecture at least, the MSI writes are targeted 
to the chipset (north bridge), not directly to a local APIC.
The chipset knows the special MSI address and data values 
programmed into the PCI device and interprets the data written, 
e.g.  for interrupt redirection hints.

> If so, is that a useful capability that should be exposed through
> the Linux MSI interface?

With MSI, you get a single address/data pair. So MSI interrupts
won't work unless this single entry is programmed to the 
special interrupt specific values that the chipset expects. 
With MSI-X, you get multiple address/data pairs but this is 
presumably because the device thinks it can benefit from 
multiple interrupts.

What type of usage model did you have in mind to have the 
device write to memory instead of using MSI for interrupts?

Rajesh

Re: MSI to memory?

[Bjorn Helgaas]

On Thursday 01 July 2004 12:53 pm, Rajesh Shah wrote:
> On Thu, Jul 01, 2004 at 12:15:59PM -0600, Bjorn Helgaas wrote:
> > If so, is that a useful capability that should be exposed through
> > the Linux MSI interface?
> ...
> What type of usage model did you have in mind to have the 
> device write to memory instead of using MSI for interrupts?

I'm not a driver writer.  I don't have any usage model in mind.  I'm
merely asking whether this is interesting functionality, because I
don't see any generic way to use it, given the current Linux MSI
interfaces.  I guess the answer is "the time to figure out an interface
is when somebody figures out an interesting way to use it."

Re: MSI to memory?

[Davide Rossetti]

Rajesh Shah wrote:

> What type of usage model did you have in mind to have the
>
>device write to memory instead of using MSI for interrupts?
>  
>
for instance for a fast wake-up trick. the driver loops on a memory 
location until the MSI write access changes the memory content...

Re: MSI to memory?

[Helge Hafting]

Davide Rossetti wrote:

> Rajesh Shah wrote:
>
>> What type of usage model did you have in mind to have the
>>
>> device write to memory instead of using MSI for interrupts?
>>  
>>
> for instance for a fast wake-up trick. the driver loops on a memory 
> location until the MSI write access changes the memory content...

Won't that put a bad load on the bus?  Someone else might need it:
* Another cpu in a smp system
* Any device doing bus-master transfers, even in a UP system

That polling loop had better be guaranteed to be _very_ short.

Helge Hafting

Re: MSI to memory?

[Roland Dreier]

    Helge> Won't that put a bad load on the bus?  Someone else might
    Helge> need it: * Another cpu in a smp system * Any device doing
    Helge> bus-master transfers, even in a UP system

Actually with MSI, the PCI device writes directly to a host address.
In the proposed usage in this mail thread, the address is in host
memory, so there's no bus load to poll the memory.  Presumably the
memory will be pulled into cache for the duration of the poll loop, so
there's not even any memory bandwidth consumed.  (Of course this only
works on an architecture where PCI DMA is cache coherent)

 - Roland

Date: Wed, 07 Jul 2004 09:59:04 -0700
In-Reply-To: <40EBF07B.8040003@hist.no> (Helge Hafting's message of "Wed, 07 Jul 2004 14:45:47 +0200")
Message-ID: <52wu1foisn.fsf@topspin.com>
User-Agent: Gnus/5.1006 (Gnus v5.10.6) XEmacs/21.4 (Security Through Obscurity, linux)

Re: MSI to memory?

[Helge Hafting]

Roland Dreier wrote:

>    Helge> Won't that put a bad load on the bus?  Someone else might
>    Helge> need it: * Another cpu in a smp system * Any device doing
>    Helge> bus-master transfers, even in a UP system
>
>Actually with MSI, the PCI device writes directly to a host address.
>In the proposed usage in this mail thread, the address is in host
>memory, so there's no bus load to poll the memory.  Presumably the
>memory will be pulled into cache for the duration of the poll loop, so
>there's not even any memory bandwidth consumed.  (Of course this only
>works on an architecture where PCI DMA is cache coherent)
>  
>
I see.  Still, this should only be used when we expect a
short wait only, similiar to situations when we use a spinlock.
Otherwise, the cpu could be put to better use by scheduling.

Helge Hafting

Re: MSI to memory?

[Zwane Mwaikambo]

On Wed, 7 Jul 2004, Helge Hafting wrote:

> Davide Rossetti wrote:
>
> > Rajesh Shah wrote:
> >
> >> What type of usage model did you have in mind to have the
> >>
> >> device write to memory instead of using MSI for interrupts?
> >>
> >>
> > for instance for a fast wake-up trick. the driver loops on a memory
> > location until the MSI write access changes the memory content...
>
> Won't that put a bad load on the bus?  Someone else might need it:
> * Another cpu in a smp system
> * Any device doing bus-master transfers, even in a UP system
>
> That polling loop had better be guaranteed to be _very_ short.

mwait/monitor is best suited for such things and doesn't result in
pounding on memory. Considering the box has MSI, there is a higher
possibility of it also having the mwait/monitor instructions.

	Zwane

Re: MSI to memory?

[Richard B. Johnson]

On Wed, 7 Jul 2004, Davide Rossetti wrote:

> Rajesh Shah wrote:
>
> > What type of usage model did you have in mind to have the
> >
> >device write to memory instead of using MSI for interrupts?
> >
> >
> for instance for a fast wake-up trick. the driver loops on a memory
> location until the MSI write access changes the memory content...

Who (another cpu??) writes to memory while your CPU is wasting its
time looping? Maybe MSI stands for Major Service Interruption?

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: MSI to memory?

[Rajesh Shah]

On Wed, Jul 07, 2004 at 02:31:47PM +0200, Davide Rossetti wrote:
> Rajesh Shah wrote:
> 
> > What type of usage model did you have in mind to have the
> >
> >device write to memory instead of using MSI for interrupts?
> >  
> >
> for instance for a fast wake-up trick. the driver loops on a memory 
> location until the MSI write access changes the memory content...
> 
But you don't need to use MSI address/data pair for this. Devices 
can do "regular" memory writes to indicate command completion etc.
into driver allocated memory.

Rajesh

Re: MSI to memory?

[Roland Dreier]

    Bjorn> The conventional use of MSI is for a PCI adapter to
    Bjorn> generate processor interrupts by writing to a local APIC.
    Bjorn> But I've seen some things that lead me to believe it would
    Bjorn> also allow an adapter to write to things other than a local
    Bjorn> APIC, i.e., to memory.

    Bjorn> If so, is that a useful capability that should be exposed
    Bjorn> through the Linux MSI interface?

MSI does allow an adapter to write messages to any arbitrary PCI
address (under the control of the host software).  It's not clear to
me that write to anywhere other than a special interrupt generation
address is that useful.

It is true that the current Linux MSI code is quite Intel-specific,
hard-coding Intel addresses and message contents.  At some point, if
I'm able to get documentation on interesting hardware, I would like to
try and move the generation of addresses/messages to arch-specific
code so that Linux can support more general PCI hosts.

Unfortunately, it seems that at least the current HyperTransport PCI-X
tunnels used on Opteron systems do not support MSI (based on my quick
reading of the documentation).  The only hardware I have access to
that supports MSI is the PowerPC 440GP, and I'm not sure how
interesting that is.

I assume that future PCI Express chipsets for Opteron will support
MSI, and hopefully documentation will be available (or manufacturers
will write Linux support).

Does anyone know if there are any ppc64 systems that support MSI?

Thanks,
  Roland