[parisc-linux] Heavy Iron Reference Docs

[parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

Group,

On the this page, in the PA 2.0 reference section:
http://parisc-linux.org/documentation/index.html

Could someone post copies of these two docs?

The PA8200 heavy iron supplement to the architecture ref:
http://docs.hp.com/en/A3725-90004/A3725-90004.pdf

The PA8500 heavy iron supplement to the architecture ref:
http://docs.hp.com/en/A5074-90004/A5074-90004.pdf

Although these are titled V-Class servers they seem to
be the only documentation on the PA8x00 coherent memory
system outside of the HP vaults.

Contain a lot of useful information, like how to make
coherent semaphores (spinlocks) work on the PA8x00
server classes.

(Dave found them)
Mike
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[John David Anglin]

> On the this page, in the PA 2.0 reference section:
> http://parisc-linux.org/documentation/index.html

I also found the following:
http://h21007.www2.hp.com/dspp/files/unprotected/itanium/spinlocks.pdf

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Kyle McMartin]

On Sun, Apr 30, 2006 at 12:36:50AM -0400, John David Anglin wrote:
> 
> I also found the following:
> http://h21007.www2.hp.com/dspp/files/unprotected/itanium/spinlocks.pdf
>

Our in-kernel spinlocks respect this, minus the pretest which probably
isn't necessary for anything but performance. See include/asm-parisc/spinlock.h
__raw_ macros for more details. I suspect mb() is probably a good equivalent
for the notation used as a compiler barrier in that document.

Note, the document also uses LDCW,CO so I suspect this is correct
in our implementation as well. Also note that in PA1.1 LDCW,CO is simply
a hint and will not relax the alignment requirement.

-- --

To address the previous point, I believe when I first discovered
the V-class SAR a year or two ago I tried to hardcode a few of the
ops in them to see if they would work on my C3000. I don't remember
whether the result was a LDW equivalent or an illegal operation trap,
but it didn't work as expected none the less. Until someone from HP
is able to confirm the existance of some of these hversion-dependant
atomic ops, I am going to pretend they don't exist.

However, if someone from HP wants to give details if any magic instructions
exist in any of the PA8x00 processors I have... I'd be more than happy to
try and make use of the information to optimize some of our atomic ops.

Cheers,
	Kyle 
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[John David Anglin]

> Our in-kernel spinlocks respect this, minus the pretest which probably
> isn't necessary for anything but performance. See include/asm-parisc/spinlock.h
> __raw_ macros for more details. I suspect mb() is probably a good equivalent
> for the notation used as a compiler barrier in that document.

They also lack the optimization discussed in section 4.1 paragraph 4
(write zero byte to high order byte of lock word to try to make the
cacheline dirty for the executing cpu).  The mb() should work as a
compiler barrier.

I also suggest using an ordered store for the unlock operation.  This
doesn't cost anything and may help to ensure that the order of memory
accesses as seen by another processor occur in the expected sequence.

> Note, the document also uses LDCW,CO so I suspect this is correct
> in our implementation as well.

I've come to believe that using the ",CO' completer may be a bug.
I think it would be better to drop using the ",CO' completer and use
the trick of storing a zero byte to high-byte of the lock word to
make the cacheline dirty.

It all comes down to this crucial bit of code in the ldcw description:

  (indivisible)
  if (cache line is present and dirty || coherent_system || cc != 0) {
    GR[t] <-- zero_ext(mem_load(space,offset,0,31,NO_HINT),32);
    mem_store(space,offset,0,31,NO_HINT,0);
  } else {
    Dcache_flush(space, offset);
    GR[t] <-- zero_ext(mem_load(space,offset,0,31,NO_HINT),32);
    store_in_memory(space,offset,0,31,NO_HINT,0);
  }

and what happens when we have a MP system that supports cc and
coherent_system == 0.  Obviously, specifying ",CO" on a system
that's fully coherent doesn't make a difference.

This is what the ",CO" completer is supposed to do:

  "The Coherent Operation cache control hint is a recommendation
   to the processor that, if the addressed data is already in the
   cache, it can operate on the addressed data in the cache rather
   than having to update memory."

Note the line doesn't have to be dirty.  Unless extreme care is
used, the line could have been be brought into cache by a load for
data elsewhere on the line.  So for correct operation of 'ldcw,co',
there really must be no inter-processor timing problems in kicking
out cachelines.  Otherwise, we could end up with two dirty cachelines
and a broken spinlock.

It may be this is only reliable on fully coherent systems.  While
the N-class is classified as a UMA machine, it has two system buses
separated by a memory controller.  Each system bus can handle four
processors with L1 and L2 cache.  Thus, it would seem safer to adopt
the prewrite and use ldcw without the cache control completer.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

On Sun April 30 2006 16:25, John David Anglin wrote:
> 
> Note the line doesn't have to be dirty.  Unless extreme care is
> used, the line could have been be brought into cache by a load for
> data elsewhere on the line.  So for correct operation of 'ldcw,co',
> there really must be no inter-processor timing problems in kicking
> out cachelines.  Otherwise, we could end up with two dirty cachelines
> and a broken spinlock.
> 

Somewhere it is written: "No data should be stored on the same cache
line as the lock unless all access is protected by that lock."

I recall the quote, do not recall which document.

That is what you just said above.  It makes sense even if not using
the coherent version of the semaphore instruction.

It makes even more sense if we are directly trying to control the
cache behavior using the coherent version.

One lock per cache line and knowledge of how long the cache line
is on a machine seems to be safe rule for now.

I.E: Fix it first, make it elegant later.

> It may be this is only reliable on fully coherent systems.  While
> the N-class is classified as a UMA machine, it has two system buses
> separated by a memory controller.  Each system bus can handle four
> processors with L1 and L2 cache.  Thus, it would seem safer to adopt
> the prewrite and use ldcw without the cache control completer.
> 

Scratches head...

I wonder where Joel has his processors installed in relation to the
two busses?  Both on same buss or one per buss?  Would his lockups
go away if he picked the other relationship?

I will ask him.  (or perhaps I just did)

Mike
> Dave
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

On Sun April 30 2006 18:01, Michael S. Zick wrote:
> 
> Scratches head...
> 
> I wonder where Joel has his processors installed in relation to the
> two busses?  Both on same buss or one per buss?  Would his lockups
> go away if he picked the other relationship?
> 
> I will ask him.  (or perhaps I just did)
>

Browsing old parisc spinlock code, I find a comment that the N4K
can only have one outstanding PxTLB transaction at a time.

Is it not the PxTLB transactions that implement the inter-processor
coherency?

Which is of course required for inter-processor spinlocks to work.

And, of course, this being Linux, the queue is protected by a spinlock!

Did someone just shoot themselves in the foot?

Mike
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Grant Grundler]

On Sun, Apr 30, 2006 at 06:28:26PM -0500, Michael S. Zick wrote:
> On Sun April 30 2006 18:01, Michael S. Zick wrote:
> > 
> > Scratches head...
> > 
> > I wonder where Joel has his processors installed in relation to the
> > two busses?  Both on same buss or one per buss?  Would his lockups
> > go away if he picked the other relationship?
> > 
> > I will ask him.  (or perhaps I just did)
> >
> 
> Browsing old parisc spinlock code, I find a comment that the N4K
> can only have one outstanding PxTLB transaction at a time.

The original document says:
	The Merced bus is designed to only support a single broadcast
	PxTLB transaction in progress at a time, and this must be guaranteed
	by software. Software must semaphore to guarantee that only one
	processor is trying to issue a broadcast PxTLB at a time.

	Local PxTLBs, however, have no restrictions, since they are executed
	entirely on the issuing processor.

> Is it not the PxTLB transactions that implement the inter-processor
> coherency?

I don't think so.

> 
> Which is of course required for inter-processor spinlocks to work.
> 
> And, of course, this being Linux, the queue is protected by a spinlock!

PA-RISC _only_ has a spinlock.

> Did someone just shoot themselves in the foot?

Do you honestly think the HP HW engineers were _that_ stupid?

They occasionally make mistakes....but designing an SMP machine
that doesn't have at least one working atomic op would be a bit
over the top. ;)

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

On Tue May 2 2006 01:24, Grant Grundler wrote:
> On Sun, Apr 30, 2006 at 06:28:26PM -0500, Michael S. Zick wrote:
>
- - - -
> > Is it not the PxTLB transactions that implement the inter-processor
> > coherency?
> 
> I don't think so.
>

Makes sense, the hardware function transactions, such as coherency
transactions are usually not accessable by even kernel code.

Ignoring diagnostic features of course.
 
> 
> 
> Do you honestly think the HP HW engineers were _that_ stupid?
> 

No.
But programmers of old hardware are not always up to the same
standard.  Typo's, Think-o's, and Brain Farts happen.

Mike
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[John David Anglin]

> Somewhere it is written: "No data should be stored on the same cache
> line as the lock unless all access is protected by that lock."

It's in the arch:

  When using semaphores to synchronize with I/O, care must be taken
  in placing other information in the same cache line as the semaphore.
  Data which is writable, can only be placed in the same cache line as
  a semaphore if access to write the data is controlled by the semaphore.

I think it's easy to misread these two sentences (i.e., to assume
that writeable data can occur on the same line as the semaphore
if the semaphore isn't being used to synchonize with I/O).

I'm almost certain we have more than one semaphore per line in current
kernels and I think that using ldcw,co is dangerous when that's done.
ldcw appears safer because it does a flush if needed.  Still, I worry
that this may not be sufficient because a sync is usually also necessary.
Flushes are weakly ordered.

If we dedicate 128 bytes per semaphore, then possibly ldcw,co will work.
It's also optimal from the contention standpoint.  This is pointed out
in the paper on semaphores.

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

On Sun April 30 2006 18:30, John David Anglin wrote:
> > Somewhere it is written: "No data should be stored on the same cache
> > line as the lock unless all access is protected by that lock."
> 
> It's in the arch:
> 
>   When using semaphores to synchronize with I/O, care must be taken
>   in placing other information in the same cache line as the semaphore.
>   Data which is writable, can only be placed in the same cache line as
>   a semaphore if access to write the data is controlled by the semaphore.
> 
> I think it's easy to misread these two sentences (i.e., to assume
> that writeable data can occur on the same line as the semaphore
> if the semaphore isn't being used to synchonize with I/O).
> 
> I'm almost certain we have more than one semaphore per line in current
> kernels and I think that using ldcw,co is dangerous when that's done.
>

Been giving that some thought, mixed with the prior weeks findings...

Consider;
Two processors;
Two semaphores, unrelated by any program logic, except they share
the same cache line;

Each of the processors grabs an 'unrelated' semaphore -

No matter how I work that problem, there is a failure window in at 
least one of the event order sequences.

I agree: "Don't do that (tm)"

Mike

> ldcw appears safer because it does a flush if needed.  Still, I worry
> that this may not be sufficient because a sync is usually also necessary.
> Flushes are weakly ordered.
> 
> If we dedicate 128 bytes per semaphore, then possibly ldcw,co will work.
> It's also optimal from the contention standpoint.  This is pointed out
> in the paper on semaphores.
> 
> Dave
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Grant Grundler]

On Sun, Apr 30, 2006 at 05:25:13PM -0400, John David Anglin wrote:
...
> I also suggest using an ordered store for the unlock operation.  This
> doesn't cost anything and may help to ensure that the order of memory
> accesses as seen by another processor occur in the expected sequence.

I've had it drilled into my head that all parisc implmentations
have strongly ordered memory subsystems. John Marvin (jsm) has
stated that more than a few times on this list. So "ordered store"
is the same as a regular store.

> It all comes down to this crucial bit of code in the ldcw description:

I'd rather have someone like Jerry Huck or someone in his experience
comment on this before we go down this path.  I'll try to find
someone to consult with this week.

...
> It may be this is only reliable on fully coherent systems.  While
> the N-class is classified as a UMA machine, it has two system buses
> separated by a memory controller.  Each system bus can handle four
> processors with L1 and L2 cache.

AFAIK, N-class has no L2 cache.

But it's worse than you think.
N-class has two _Merced_ busses connected to the memory controller. 
Each Processor is connected via a double pumped Runway Bus to "Dew"
which acts as a "bridge" to one of the Merced Busses.

[ Digression - certain document says:
	In PA-RISC, code fetches are non-coherent, such that PCX-W doesn't
	even supply Vindex bits that would allow the code fetches to be
	coherent.
]

> Thus, it would seem safer to adopt
> the prewrite and use ldcw without the cache control completer.

I'm skeptical but will ask about this.
Maybe I'm thinking kernel space only...is this intended only for
user space or for kernel also?

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[John David Anglin]

> > I also suggest using an ordered store for the unlock operation.  This
> > doesn't cost anything and may help to ensure that the order of memory
> > accesses as seen by another processor occur in the expected sequence.
> 
> I've had it drilled into my head that all parisc implmentations
> have strongly ordered memory subsystems. John Marvin (jsm) has
> stated that more than a few times on this list. So "ordered store"
> is the same as a regular store.

Yes, I know this has been said and I've seen it in documents.
However, HP-UX 11i uses lots of "ordered" stores and one is never
quite sure about the currency of information in documentation.

> > It all comes down to this crucial bit of code in the ldcw description:
> 
> I'd rather have someone like Jerry Huck or someone in his experience
> comment on this before we go down this path.  I'll try to find
> someone to consult with this week.

That would be great and I think help to clear up our questions.

> > It may be this is only reliable on fully coherent systems.  While
> > the N-class is classified as a UMA machine, it has two system buses
> > separated by a memory controller.  Each system bus can handle four
> > processors with L1 and L2 cache.
> 
> AFAIK, N-class has no L2 cache.

However, the rp7400 and rp7420 certainly do.  See
<http://www.ccns.pl/zasoby/13/rp7400techwp5.pdf>.  The rp7420
appears to have 32MB or 64MB of L2 per processor module.

> But it's worse than you think.
> N-class has two _Merced_ busses connected to the memory controller. 
> Each Processor is connected via a double pumped Runway Bus to "Dew"
> which acts as a "bridge" to one of the Merced Busses.
> 
> [ Digression - certain document says:
> 	In PA-RISC, code fetches are non-coherent, such that PCX-W doesn't
> 	even supply Vindex bits that would allow the code fetches to be
> 	coherent.
> ]

Because of this, if you want to execute instructions on the stack,
you have to flush the lines from both the instruction and data cache,
and do a sync before transferring to the code on the stack.

I just think that in machines with multiple data caches, we have to
be careful about the coherence of the data in these caches.  You have
pointed out that I-cache code fetches are non-coherent.  The L2 caches
in the rp7420 are combined.  The logic could provide different behavior
for I and D accesses, then again it might not.

If the hardware on all PA-RISC machines guarantees full coherence
of the D-cache on all machines except the V class, then my concern
is misplaced.  In that case, coherent_system should be 1 and the
",co" completer should have no effect on the semaphore operation,
other than to reduce the alignment requirement on PA 2.0 machines.
This would also imply that the store byte operation shown in the
semaphore paper doesn't improve spinlock performance.

Are there any PA 2.0 machines for which the value of coherent_system
is 0?  If so, which ones?

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Kyle McMartin]

On Tue, May 02, 2006 at 11:10:51AM -0400, John David Anglin wrote:
> Are there any PA 2.0 machines for which the value of coherent_system
> is 0?  If so, which ones?
>

I'll cook a patch which dumps more cache info.

struct pdc_cache_cf {           /* for PDC_CACHE  (I/D-caches) */
	[...]
                cc_cst  : 3,    /* 0 = incoherent D-cache, 1=coherent D-cache */
	[...]
};

Should tell us what we need to know. 
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[John David Anglin]

> On Tue, May 02, 2006 at 11:10:51AM -0400, John David Anglin wrote:
> > Are there any PA 2.0 machines for which the value of coherent_system
> > is 0?  If so, which ones?
> >
> 
> I'll cook a patch which dumps more cache info.
> 
> struct pdc_cache_cf {           /* for PDC_CACHE  (I/D-caches) */
> 	[...]
>                 cc_cst  : 3,    /* 0 = incoherent D-cache, 1=coherent D-cache */
> 	[...]
> };
> 
> Should tell us what we need to know. 

Excellent.  Could this go into show_cache_info() so that
"cat /proc/cpuinfo" will show more cache data?  There's also
some stuff in the cache init routine that doesn't print
because of "#if 0".

Dave
-- 
J. David Anglin                                  dave.anglin@nrc-cnrc.gc.ca
National Research Council of Canada              (613) 990-0752 (FAX: 952-6602)
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Grant Grundler]

On Sat, Apr 29, 2006 at 10:50:49PM -0500, Michael S. Zick wrote:
...
> Although these are titled V-Class servers they seem to
> be the only documentation on the PA8x00 coherent memory
> system outside of the HP vaults.

On page "20 of 248" for V2500 doc:
    "This document does not attempt to duplicate information in that
     manual. Instead, it presents only V2500 server-specific information."

And ISTR V-class has "unique" constraints on coherency.
Sorry - I've forgotten details. But you might start on
page "71" (aka page 89 using xpdf numbering):
|  The instructions which flush based on a memory line address are PDC,
|  FDC and FIC. These instructions are broadcast to other processors
|  within a node that may be sharing the same cache lines. These
|  instructions, therefore, have global effects within a node.
...
|  Cache flush instructions must be followed by a sync instruction to ensure
|  that all flushes have made it to memory.

and Page "75" (aka page 93 of pdf):
|  CTI cache interfaces
|  The PA-RISC architecture (1.0 and 2.0) does not support the concept of a
|  CTI Cache. Thus, there are no PA-RISC architected instructions for
|  issuing the CTI cache operations.

Seems to state V2500 NUMA implementation is NOT supported by the general
PA 2.0 architecture. Be very careful to not generalize statements made in
the V-class specific documents.

I'm currently only certain cc-NUMA V-2500 has cache coherency issues
with DMA.  A seperate exciting topic for discussion once I've retired.
(at least another 20 years or so)  :)


> Contain a lot of useful information, like how to make
> coherent semaphores (spinlocks) work on the PA8x00
> server classes.

How to implement spinlocks on V2500 is not necessarily the
same set of requirements for all other classes of PA20 machines.
Keep in mind parisc-linux currently does NOT support any V-class
machines - ie we have no ability to test theories about locking
other than with HP-UX.


Lastly, if other HP employees get permission to redistribute
the V-class PDF files, I'd be happy to put them on ftp.parisc-linux.org.
Until parisc-linux kernel supports V-class, I have no interest in
pursuing permission to do redistribute those documents.

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] Heavy Iron Reference Docs

[Michael S. Zick]

On Sun April 30 2006 02:03, you wrote:
> 
> Seems to state V2500 NUMA implementation is NOT supported by the general
> PA 2.0 architecture. Be very careful to not generalize statements made in
> the V-class specific documents.
> 

Grant, I wish it was that simple.

What you point out of about the V2500 seems
to be exact.

The V2200 has different memory hardware and
a different means for coherent semaphores.

My biggest mistake was to think that we can
go by the cpu version. There will have to be 
another way found.

HP-UX uses an external library for semaphores
to get it correct, at least sometimes.
The S... is getting deep.

What can be demonstrated at this moment is:

PA8500 - works (?)
PA8600 - broke
PA8700 - works
PA8800 - ???
PA8900 - broke (?)

The lack of documentation that would tell us
why does not mean we have given up though.

Mike
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux