[RFC 0/4] Add SN2 Special Memory driver.

[RFC 0/4] Add SN2 Special Memory driver.

[Robin Holt]

This driver provides three different devices for mmap'ing pages which
are not visible to the kernel.

sgi_fetchops) atomic operations performed by the SN2 memory controller.
	These operations are performed using uncached memory
	references with an offset of the address specifying the
	operation (add, sub) to perform.

sgi_uncached) Provides a device which supports mapping pages which
	will only be referenced uncached.  These use the Intel ia64
	write combining feature.  These need to be in a separate
	granule from regular memory to prevent the FSB from having
	both a cached and an uncached reference to a memory location.

sgi_cached) Provides a device which support cached operations from the
	processor and uncached from processors outside the coherence
	domain.  This provides rapid read access to the 16 words in the
	cache line to data that was written uncached by remote processors.


All of these devices require addresses that have no kernel tlb entry.
The fetchop and uncached driver because the CPU does not allow both cached
and uncached references to the same cache line on the FSB at the same
time and kernel speculation is possible whenever there is a tlb entry.

The cached driver suffers from the other part of that problem.  The kernel
will occasionally have an lfetch.excl with the operation on that address
being predicated.  The processor will prefetch the cache line, never
actually put any data in it, but still consider it dirty and then write
the line back.  This would result in clobbering the data.

All of this is avoided by this driver by handling the pages with physical
addresses only.

Thank you,
Robin Holt

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Christoph Hellwig]

On Thu, Sep 09, 2004 at 11:36:38AM -0500, Robin Holt wrote:
> 
> This driver provides three different devices for mmap'ing pages which
> are not visible to the kernel.
> 
> sgi_fetchops) atomic operations performed by the SN2 memory controller.
> 	These operations are performed using uncached memory
> 	references with an offset of the address specifying the
> 	operation (add, sub) to perform.
> 
> sgi_uncached) Provides a device which supports mapping pages which
> 	will only be referenced uncached.  These use the Intel ia64
> 	write combining feature.  These need to be in a separate
> 	granule from regular memory to prevent the FSB from having
> 	both a cached and an uncached reference to a memory location.
> 
> sgi_cached) Provides a device which support cached operations from the
> 	processor and uncached from processors outside the coherence
> 	domain.  This provides rapid read access to the 16 words in the
> 	cache line to data that was written uncached by remote processors.

From your description only the first two actually use special SGI hardware
features, or did I misread the descruption?  If so they should probably
have a separate driver that works on all ia64 hardware.

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Robin Holt]

On Thu, Sep 09, 2004 at 06:21:56PM +0100, Christoph Hellwig wrote:
> On Thu, Sep 09, 2004 at 11:36:38AM -0500, Robin Holt wrote:
> > 
> > This driver provides three different devices for mmap'ing pages which
> > are not visible to the kernel.
> > 
> > sgi_fetchops) atomic operations performed by the SN2 memory controller.
> > 	These operations are performed using uncached memory
> > 	references with an offset of the address specifying the
> > 	operation (add, sub) to perform.
> > 
> > sgi_uncached) Provides a device which supports mapping pages which
> > 	will only be referenced uncached.  These use the Intel ia64
> > 	write combining feature.  These need to be in a separate
> > 	granule from regular memory to prevent the FSB from having
> > 	both a cached and an uncached reference to a memory location.
> > 
> > sgi_cached) Provides a device which support cached operations from the
> > 	processor and uncached from processors outside the coherence
> > 	domain.  This provides rapid read access to the 16 words in the
> > 	cache line to data that was written uncached by remote processors.
> 
From your description only the first two actually use special SGI hardware
> features, or did I misread the descruption?  If so they should probably
> have a separate driver that works on all ia64 hardware.

I am not sure how to read this.  The fetchop function is a feature specifically
in the SN2 memory controller.  Are you saying we should have a driver that
provides fetchops for all ia64?  That seems wrong since it is a hardware feature.

If, on the other hand, you are saying that the uncached and cached drivers
should be available for all ia64 and the fetchop only if you are on sn2, I
can understand that.  That actually seems rather reasonable.

Would anybody else on ia64 be interested in either the uncached or cached
features?  The cached is really there to support MPI communications across
the coherence domain.  Does anybody else have a NUMA architecture which
mixes cache coherent and non-coherent NUMA?  If not, I don't think that
the cached functionality would ever be preferrable over regular cached
references.

Robin

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Christoph Hellwig]

On Thu, Sep 09, 2004 at 12:59:37PM -0500, Robin Holt wrote:
> I am not sure how to read this.  The fetchop function is a feature specifically
> in the SN2 memory controller.  Are you saying we should have a driver that
> provides fetchops for all ia64?  That seems wrong since it is a hardware feature.
> 
> If, on the other hand, you are saying that the uncached and cached drivers
> should be available for all ia64 and the fetchop only if you are on sn2, I
> can understand that.  That actually seems rather reasonable.

Yes, that's what I meant.  Even if it's mostly useful for SGI's userland
it's implementation shouldn't depend on a SN2 kernel if it doesn't use any
SN2-specific functionality.

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Robin Holt]

On Fri, Sep 10, 2004 at 08:33:57AM +0100, Christoph Hellwig wrote:
> On Thu, Sep 09, 2004 at 12:59:37PM -0500, Robin Holt wrote:
> > I am not sure how to read this.  The fetchop function is a feature specifically
> > in the SN2 memory controller.  Are you saying we should have a driver that
> > provides fetchops for all ia64?  That seems wrong since it is a hardware feature.
> > 
> > If, on the other hand, you are saying that the uncached and cached drivers
> > should be available for all ia64 and the fetchop only if you are on sn2, I
> > can understand that.  That actually seems rather reasonable.
> 
> Yes, that's what I meant.  Even if it's mostly useful for SGI's userland
> it's implementation shouldn't depend on a SN2 kernel if it doesn't use any
> SN2-specific functionality.

Do any other processors have the write combining features of the ia64?  Is this
type of memory useable by any other processor?  If not, does this driver belong
in the arch/ia64/kernel directory?

Thanks,
Robin

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Christoph Hellwig]

On Fri, Sep 10, 2004 at 03:31:13AM -0500, Robin Holt wrote:
> On Fri, Sep 10, 2004 at 08:33:57AM +0100, Christoph Hellwig wrote:
> > On Thu, Sep 09, 2004 at 12:59:37PM -0500, Robin Holt wrote:
> > > I am not sure how to read this.  The fetchop function is a feature specifically
> > > in the SN2 memory controller.  Are you saying we should have a driver that
> > > provides fetchops for all ia64?  That seems wrong since it is a hardware feature.
> > > 
> > > If, on the other hand, you are saying that the uncached and cached drivers
> > > should be available for all ia64 and the fetchop only if you are on sn2, I
> > > can understand that.  That actually seems rather reasonable.
> > 
> > Yes, that's what I meant.  Even if it's mostly useful for SGI's userland
> > it's implementation shouldn't depend on a SN2 kernel if it doesn't use any
> > SN2-specific functionality.
> 
> Do any other processors have the write combining features of the ia64?  Is this
> type of memory useable by any other processor?  If not, does this driver belong
> in the arch/ia64/kernel directory?

Umm, yes I guess it'll be ia64-specific at least as a start,  I just wanted to
avoid having it SN2-specific.

Re: [RFC 0/4] Add SN2 Special Memory driver.

[Jesse Barnes]

On Friday, September 10, 2004 1:32 am, Christoph Hellwig wrote:
> > Do any other processors have the write combining features of the ia64? 
> > Is this type of memory useable by any other processor?  If not, does this
> > driver belong in the arch/ia64/kernel directory?
>
> Umm, yes I guess it'll be ia64-specific at least as a start,  I just wanted
> to avoid having it SN2-specific.

...but if it's a char driver, just put it in drivers/char.  Makes it easier to 
update APIs and such that way.

Jesse