fix swapping on 8xx?

fix swapping on 8xx?

[Marcelo Tosatti]

Hi, 

The following is an attempt to fix swapping on 8xx by not touching
_PAGE_ACCESSED bit if the page is not present.

Dan, what do you think?


diff --git a/arch/ppc/kernel/head_8xx.S b/arch/ppc/kernel/head_8xx.S
index de09787..4451828 100644
--- a/arch/ppc/kernel/head_8xx.S
+++ b/arch/ppc/kernel/head_8xx.S
@@ -41,6 +41,22 @@
 #else
 #define DO_8xx_CPU6(val, reg)
 #endif
+
+/* can't overwrite the _PAGE_ACCESSED bit of a non-present page (might
+   contain swap data). */
+#ifdef CONFIG_SWAP
+#define SET_PAGE_ACCESSED(reg, destreg, tmpreg)		\
+	andi.	tmpreg, reg, _PAGE_PRESENT;		\
+	beq	9f;					\
+	ori	reg, reg, _PAGE_ACCESSED;		\
+	stw	r10, 0(destreg);			\
+9:
+#else
+#define SET_PAGE_ACCESSED(reg, destreg, tmpreg)	\
+	ori	reg, reg, _PAGE_ACCESSED;	\
+	stw	reg, 0(destreg);
+#endif
+
 	.text
 	.globl	_stext
 _stext:
@@ -332,8 +348,7 @@ InstructionTLBMiss:
 	mfspr	r11, SPRN_MD_TWC	/* ....and get the pte address */
 	lwz	r10, 0(r11)	/* Get the pte */
 
-	ori	r10, r10, _PAGE_ACCESSED
-	stw	r10, 0(r11)
+	SET_PAGE_ACCESSED(r10, r11, r3)
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.
@@ -399,8 +414,7 @@ DataStoreTLBMiss:
 	mtspr	SPRN_MD_TWC, r11
 
 	mfspr	r11, SPRN_MD_TWC	/* get the pte address again */
-	ori	r10, r10, _PAGE_ACCESSED
-	stw	r10, 0(r11)
+	SET_PAGE_ACCESSED(r10, r11, r3)	/* and update pte in table */
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.
@@ -507,9 +521,9 @@ DataTLBError:
 
 	/* Update 'changed', among others.
 	*/
-	ori	r10, r10, _PAGE_DIRTY|_PAGE_ACCESSED|_PAGE_HWWRITE
+	ori	r10, r10, _PAGE_DIRTY|_PAGE_HWWRITE
 	mfspr	r11, SPRN_MD_TWC		/* Get pte address again */
-	stw	r10, 0(r11)		/* and update pte in table */
+	SET_PAGE_ACCESSED(r10, r11, r3)		/* and update pte in table */
 
 	/* The Linux PTE won't go exactly into the MMU TLB.
 	 * Software indicator bits 21, 22 and 28 must be clear.

Re: fix swapping on 8xx?

[Dan Malek]

On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:

> The following is an attempt to fix swapping on 8xx by not touching
> _PAGE_ACCESSED bit if the page is not present.

Ugh ....  I suppose.  I hate assembler code macros .......
Somehow, "swapping" and "8xx" just don't belong together.
I'm tempted to add a configuration option that is the complete
opposite of this and assumes are really embedded system.
Mark pages as always accessed, data pages as always dirty,
and you can eliminate lots of TLB faults in systems that are
fairly static.

None of this should really be done in a TLB miss handler,
if there is anything that needs to be updated in a PTE, it should
be pushed out to an error function.  A TLB miss handler
should fit in a cache line ......

	-- Dan

Re: fix swapping on 8xx?

[David Jander]

On Tuesday 08 November 2005 01:55, Dan Malek wrote:
> On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:
> > The following is an attempt to fix swapping on 8xx by not touching
> > _PAGE_ACCESSED bit if the page is not present.
>
> Ugh ....  I suppose.  I hate assembler code macros .......
> Somehow, "swapping" and "8xx" just don't belong together.

Well, at least it sounds ugly together, but it is also at least conceiveable. 
There seem to be people who use PCMCIA for an IDE interface, so swapping may 
become desireable in some cases.

> I'm tempted to add a configuration option that is the complete
> opposite of this and assumes are really embedded system.
> Mark pages as always accessed, data pages as always dirty,
> and you can eliminate lots of TLB faults in systems that are
> fairly static.

It sounds tempting indeed, but should you really notice a performance increase 
out of this?

Greetings,

-- 
David Jander

Re: fix swapping on 8xx?

[Tom Rini]

On Tue, Nov 08, 2005 at 01:59:26PM +0200, David Jander wrote:
> On Tuesday 08 November 2005 01:55, Dan Malek wrote:
> > On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:
> > > The following is an attempt to fix swapping on 8xx by not touching
> > > _PAGE_ACCESSED bit if the page is not present.
> >
> > Ugh ....  I suppose.  I hate assembler code macros .......
> > Somehow, "swapping" and "8xx" just don't belong together.
> 
> Well, at least it sounds ugly together, but it is also at least conceiveable. 
> There seem to be people who use PCMCIA for an IDE interface, so swapping may 
> become desireable in some cases.

I think Dan might be in the camp that says a properly designed embedded
system won't need to swap.  And when I hear about how people do try and
swap on systems like this, I really start agreeing.  Maybe we could make
8xx just select SWAP=n? :)

> > I'm tempted to add a configuration option that is the complete
> > opposite of this and assumes are really embedded system.
> > Mark pages as always accessed, data pages as always dirty,
> > and you can eliminate lots of TLB faults in systems that are
> > fairly static.
> 
> It sounds tempting indeed, but should you really notice a performance increase 
> out of this?

Compared to 8xx in 2.6 today?  Absolutely.

-- 
Tom Rini
http://gate.crashing.org/~trini/

Re: fix swapping on 8xx?

[Dan Malek]

On Nov 8, 2005, at 12:56 PM, Tom Rini wrote:

> I think Dan might be in the camp that says a properly designed embedded
> system won't need to swap.

I'm actually in the camp that knows the majority of systems
running Linux aren't workstations and don't have disk drives.
It would be nice to have selectable features for such applications.

>> It sounds tempting indeed, but should you really notice a performance 
>> increase
>> out of this?

Yes.  Most importantly it adds some predictability which is quite
measurable on this class of processor.  It was a coding mistake way
back in 2.2 that actually exposed this.  I was taking some liberties by
updating more status in the PTE than I should have, and Paulus
made me fix it in 2.4 :-)  It basically eliminates TLB faults associated
with system status tracking, since we aren't paging there isn't any
need to track page aging and such.  If you are paging, it still
functions properly, you may just not pick the most suitable pages
to steal.

Thanks.

	-- Dan

Re: fix swapping on 8xx?

[David Jander]

>> I think Dan might be in the camp that says a properly designed embedded
>> system won't need to swap.
>
> I'm actually in the camp that knows the majority of systems
> running Linux aren't workstations and don't have disk drives.
> It would be nice to have selectable features for such applications.

Well, me too, but there are people out there trying to do strange stuff. I
remember myself even once trying to get swapping to work over nfs, because
I wanted to compile perl5 natively (overnight of course), since
cross-compiling perl is quite a bit of pain. Perl intends to
"include-together" enormous source files, that exhausted the whole 64Mbyte
RAM on my embedded board ;-)
Needless to say, swap over nfs did never work reliably enough to even
start compiling.

>>> It sounds tempting indeed, but should you really notice a performance
>>> increase out of this?
>
> Yes.  Most importantly it adds some predictability which is quite
> measurable on this class of processor. It was a coding mistake way back
> in 2.2 that actually exposed this.  I was taking some liberties by
> updating more status in the PTE than I should have, and Paulus
> made me fix it in 2.4 :-)  It basically eliminates TLB faults associated
> with system status tracking, since we aren't paging there isn't any need
> to track page aging and such.  If you are paging, it still
> functions properly, you may just not pick the most suitable pages to
> steal.

FYI, I have some benchmark results of yesterday's git of kernel-2.6 (with
_tlbie(address); just moved out of the two innermost if()'s in the
function update_mmu_cache()) compared to kernel 2.4.25 from denx, made
with "nbench". The results are quite interesting, and contradicting
(older) benchmark results, Wolfgang made a few months back:

kernel-2.6.14-git-20051107:
---------------------------
TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          32.654  :       0.84  :       0.28
STRING SORT         :          1.7408  :       0.78  :       0.12
BITFIELD            :      8.3466e+06  :       1.43  :       0.30
FP EMULATION        :           3.506  :       1.68  :       0.39
IDEA                :           115.3  :       1.76  :       0.52
HUFFMAN             :          27.855  :       0.77  :       0.25
LU DECOMPOSITION    :         0.35932  :       0.02  :       0.01

kernel-2.4.25-devel-denx:
-------------------------
TEST                : Iterations/sec.  : Old Index   : New Index
                    :                  : Pentium 90* : AMD K6/233*
--------------------:------------------:-------------:------------
NUMERIC SORT        :          30.438  :       0.78  :       0.26
STRING SORT         :          1.5842  :       0.71  :       0.11
BITFIELD            :      7.9506e+06  :       1.36  :       0.28
FP EMULATION        :           3.258  :       1.56  :       0.36
IDEA                :          108.89  :       1.67  :       0.49
HUFFMAN             :          26.281  :       0.73  :       0.23
LU DECOMPOSITION    :         0.32765  :       0.02  :       0.01

This is all on the same hardware: MPC852T + 32Mbyte SDRAM running from
NFS. kernel 2.6 is faster all the way through!!
The results so far are pretty encouraging I'd say. Maybe something like
lmbench would help to shed more light on details important for kernel
testing, like context-switch overhead, mm, etc...
Nbench is quite useless here (a single task doing all sorts of nonsense,
almost no critical syscalls), but that's what makes the enormous
differences even more remarkable IMO. Maybe something is broken in 2.4?

Greetings,

-- 
David Jander

Re: fix swapping on 8xx?

[Wolfgang Denk]

In message <27139.213.84.146.227.1131479070.squirrel@protonic.xs4all.nl> you wrote:
>
> Needless to say, swap over nfs did never work reliably enough to even
> start compiling.

You can work around this. It's painfully slow, but it works. See
http://www.denx.de/wiki/view/DULG/SwappingOverNFS

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Only a fool fights in a burning house.
	-- Kank the Klingon, "Day of the Dove", stardate unknown

Re: fix swapping on 8xx?

[Wolfgang Denk]

In message <20051108175658.GO3839@smtp.west.cox.net> you wrote:
>
> > Well, at least it sounds ugly together, but it is also at least conceiveable. 
> > There seem to be people who use PCMCIA for an IDE interface, so swapping may 
> > become desireable in some cases.
> 
> I think Dan might be in the camp that says a properly designed embedded
> system won't need to swap.  And when I hear about how people do try and
> swap on systems like this, I really start agreeing.  Maybe we could make
> 8xx just select SWAP=n? :)

No! David is right. There are systems out in the field  running  with
IDE  harddisks attached (either through custom hardware interfaces or
through standard PCMCIA adapters), and some of  these  actually  need
and use swap space.

Please do not remove stuff that is needed and used to work (in 2.4).

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Virtual" means never knowing where your next byte is coming from.

Re: fix swapping on 8xx?

[Tom Rini]

On Tue, Nov 08, 2005 at 09:52:59PM +0100, Wolfgang Denk wrote:
> In message <20051108175658.GO3839@smtp.west.cox.net> you wrote:
> >
> > > Well, at least it sounds ugly together, but it is also at least conceiveable. 
> > > There seem to be people who use PCMCIA for an IDE interface, so swapping may 
> > > become desireable in some cases.
> > 
> > I think Dan might be in the camp that says a properly designed embedded
> > system won't need to swap.  And when I hear about how people do try and
> > swap on systems like this, I really start agreeing.  Maybe we could make
> > 8xx just select SWAP=n? :)
> 
> No! David is right. There are systems out in the field  running  with
> IDE  harddisks attached (either through custom hardware interfaces or
> through standard PCMCIA adapters), and some of  these  actually  need
> and use swap space.
> 
> Please do not remove stuff that is needed and used to work (in 2.4).

Note the bug in question exists in 2.4, and I think really even 2.2.

-- 
Tom Rini
http://gate.crashing.org/~trini/

Re: fix swapping on 8xx?

[Wolfgang Denk]

In message <20051108210204.GP3839@smtp.west.cox.net> you wrote:
>
> > Please do not remove stuff that is needed and used to work (in 2.4).
> 
> Note the bug in question exists in 2.4, and I think really even 2.2.

Hm... as far as I can remember (and see in the change logs) the  only
fix  we  implemented  for our 2.4.25 tree to get swapping on a MPC8xx
working was a set of bad offsets in include/asm-ppc/pgtable.h - see
http://www.denx.de/cgi-bin/gitweb.cgi?p=linuxppc_2_4_devel.git;a=commit;h=e6d7a1c83f9fd5820cdb85389d6a8f2207f3641d

With this change swapping works fine for us - some of  our  customers
have been using this code in production for over two years now.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Good morning. This is the telephone company. Due  to  repairs,  we're
giving  you  advance notice that your service will be cut off indefi-
nitely at ten o'clock. That's two minutes from now.

Re: fix swapping on 8xx?

[Dan Malek]

On Nov 8, 2005, at 5:46 PM, Wolfgang Denk wrote:

> Hm... as far as I can remember ....

Yes, your patch looks correct.  I'll just map these changes
on top of the PTEs to verify there aren't some status
bits that we may still incorrectly update.  When I originally
selected the bit positions for the PTEs I didn't properly
consider the usage of the PTE during swap (nor did
I always have the options I wanted :-))

Among everything else I promised, I'll try to get all of these
updates quickly sorted out.

Thanks for all of these pointers to old patches, saves
me time looking them up :-)


	-- Dan

Re: fix swapping on 8xx?

[Dan Malek]

On Nov 8, 2005, at 3:52 PM, Wolfgang Denk wrote:

> Please do not remove stuff that is needed and used to work (in 2.4).

I'm not proposing we remove anything :-)  It would just be nice
to eliminate the overhead of managing the PTEs if we don't
need such a feature.  The complexity of the TLB miss handler
just keeps growing to add/fix this stuff, and it really shouldn't
be in that code path.  All of this support should be pushed
into the error path, and if it was somewhat configurable we
wouldn't be in that path if not necessary.

Thanks.

	-- Dan

Re: fix swapping on 8xx?

[Marcelo Tosatti]

On Tue, Nov 08, 2005 at 10:56:58AM -0700, Tom Rini wrote:
> On Tue, Nov 08, 2005 at 01:59:26PM +0200, David Jander wrote:
> > On Tuesday 08 November 2005 01:55, Dan Malek wrote:
> > > On Nov 7, 2005, at 10:10 AM, Marcelo Tosatti wrote:
> > > > The following is an attempt to fix swapping on 8xx by not touching
> > > > _PAGE_ACCESSED bit if the page is not present.
> > >
> > > Ugh ....  I suppose.  I hate assembler code macros .......
> > > Somehow, "swapping" and "8xx" just don't belong together.
> > 
> > Well, at least it sounds ugly together, but it is also at least conceiveable. 
> > There seem to be people who use PCMCIA for an IDE interface, so swapping may 
> > become desireable in some cases.
> 
> I think Dan might be in the camp that says a properly designed embedded
> system won't need to swap.  And when I hear about how people do try and
> swap on systems like this, I really start agreeing.  Maybe we could make
> 8xx just select SWAP=n? :)

TimeSys shipped their kernel with swapping fix as far as I know (Jason
plyed with it recently).

We'd better not assume what people try to do with their old 8xx's :)

> > > I'm tempted to add a configuration option that is the complete
> > > opposite of this and assumes are really embedded system.
> > > Mark pages as always accessed, data pages as always dirty,
> > > and you can eliminate lots of TLB faults in systems that are
> > > fairly static.
> > 
> > It sounds tempting indeed, but should you really notice a performance increase 
> > out of this?
> 
> Compared to 8xx in 2.6 today?  Absolutely.
> 
> -- 
> Tom Rini
> http://gate.crashing.org/~trini/
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded