LinuxPPC-Dev Archive on lore.kernel.org
 help / color / mirror / Atom feed
* RE: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Joakim Tjernlund @ 2005-04-23 21:55 UTC (permalink / raw)
  To: Dan Malek, Marcelo Tosatti; +Cc: linuxppc-embedded
In-Reply-To: <31d4fa0af2a0d11f029d675ae5bd331c@embeddededge.com>

> On Apr 23, 2005, at 8:42 AM, Marcelo Tosatti wrote:
> 
> >> Does CONFIG_PIN_TLB make a difference?
> >
> > No it does not.
> 
> For some reason this option and code didn't make it from
> 2.4 to 2.6.  It should have some effect on small memory (16Mbyte)
> systems with processor that have more than 16 TLB entries.

Oh, but the CONFIG_PIN_TLB code in head_8xx.S is there. I guess something is missing.

 Jocke

^ permalink raw reply

* RE: 8xx v2.6 TLB problems and suggested workaround
From: Joakim Tjernlund @ 2005-04-23 22:07 UTC (permalink / raw)
  To: Dan Malek; +Cc: linuxppc-embedded
In-Reply-To: <585e588ae8021963207cc6ee9955289e@embeddededge.com>

> On Apr 9, 2005, at 3:03 PM, Joakim Tjernlund wrote:
> 
> > yes, but I think these operates on physical addresses which makes it a 
> > bit harder.
> > I still think this can be resolved in fault.c. Replace
> > 	andis.	r11, r10, 0x0200	/* If set, indicates store op */
> > 	beq	2f
> > in the DTLB Error handler with
> > 	andis.	r11, r10, 0x4800	/* If set, indicates invalid pte or 
> > protection violation */
> > 	bne	2f
> > In fault.c you can check if both store and invalid is set 
> > simultaneously. If it is, clear
> > the store flag and continue as usual.
> 
> The purpose for the code in TLB Error is to create fast path for 
> tracking
> writable pages as dirty.  I think we should stop writing all of this 
> assembler

Do you refer to the assembler code above or some other assembler code such
as my dcbX workaround? I once had a C version of that workaround that lived
in fault.c. Sadly it think it is gone now, not sure if I ever sent it to the list.

> code and if we find anything that isn't simply updating the "dirty" 
> flags, we
> should bail out to the fault.c and do whatever is necessary.  This 
> includes
> simulating any cache instructions that fail.
> 
> As a further performance enhancement, which used to be the standard
> mode for 8xx, we should allow the option to mark all writable pages as
> dirty when the PTE is created.  This eliminates the TLB Error fault in 
> this
> case completely.  Since most 8xx systems don't do page swapping, this
> has no other effect.  If it does page swapping, it may swap more pages
> than necessary, or the option can be disabled to do proper paging.
> 
> Thanks.
> 
> 
> 	-- Dan
> 

^ permalink raw reply

* Re: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Dan Malek @ 2005-04-23 22:09 UTC (permalink / raw)
  To: Joakim.Tjernlund; +Cc: linuxppc-embedded
In-Reply-To: <BCEFJBPJCGFCNMMMIDBHGEIJCLAA.Joakim.Tjernlund@lumentis.se>


On Apr 23, 2005, at 5:51 PM, Joakim Tjernlund wrote:

> Well, every instruction counts. I this case we would have saved
> 2 in ITLB Miss, 3 in DTLB Miss and a cache line write in both.

You have already read the PTE and instructions into the cache,
there are no branches, but not a big deal.

> Would be nice to do away with the kernel space test, but thats a lot 
> harder.

With some clever first level pointer page creation and management
we could do this, but it would be custom 8xx code in generic files.

Thanks.

	-- Dan

^ permalink raw reply

* Re: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Dan Malek @ 2005-04-23 22:12 UTC (permalink / raw)
  To: Joakim.Tjernlund; +Cc: linuxppc-embedded
In-Reply-To: <BCEFJBPJCGFCNMMMIDBHOEIJCLAA.Joakim.Tjernlund@lumentis.se>


On Apr 23, 2005, at 5:55 PM, Joakim Tjernlund wrote:

> Oh, but the CONFIG_PIN_TLB code in head_8xx.S is there. I guess 
> something is missing.

Ooops, I searched for the wrong name, I missed it :-)

Maybe it needs a different set up for something other than the 860.


	-- Dan

^ permalink raw reply

* Re: 8xx v2.6 TLB problems and suggested workaround
From: Dan Malek @ 2005-04-23 22:23 UTC (permalink / raw)
  To: Joakim.Tjernlund; +Cc: linuxppc-embedded
In-Reply-To: <BCEFJBPJCGFCNMMMIDBHIEIKCLAA.Joakim.Tjernlund@lumentis.se>


On Apr 23, 2005, at 6:07 PM, Joakim Tjernlund wrote:

> Do you refer to the assembler code above or some other assembler code 
> such
> as my dcbX workaround? I once had a C version of that workaround that 
> lived
> in fault.c. Sadly it think it is gone now, not sure if I ever sent it 
> to the list.

Yes, the dcbx and anything else we may have to place in there.  At most,
we should emulate whatever is necessary in C code and just invalidate
the TLB entry for reloading, if the PTE has changed.  We shouldn't be
trying to load TLB entries.

Thanks.


	-- Dan

^ permalink raw reply

* Re: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Dan Malek @ 2005-04-23 23:12 UTC (permalink / raw)
  To: Joakim.Tjernlund; +Cc: linuxppc-embedded
In-Reply-To: <BCEFJBPJCGFCNMMMIDBHMEIHCLAA.Joakim.Tjernlund@lumentis.se>


On Apr 23, 2005, at 1:23 PM, Joakim Tjernlund wrote:

> Does CONFIG_PIN_TLB make a difference?

While looking at the code, I noticed this will only work
if you can map all of the memory with wired TLB entries.
If you have more than 24M of memory, make sure you
enable CONFIG_MODULES to eliminate the TLB miss
optimization that will certainly crash a system by failing
to look up kernel PTEs correctly.

Thanks.


	-- Dan

^ permalink raw reply

* RE: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Joakim Tjernlund @ 2005-04-23 23:51 UTC (permalink / raw)
  To: Dan Malek; +Cc: linuxppc-embedded
In-Reply-To: <4f898cca602f86b36579fd02046c1aac@embeddededge.com>

> 
> On Apr 23, 2005, at 1:23 PM, Joakim Tjernlund wrote:
> 
> > Does CONFIG_PIN_TLB make a difference?
> 
> While looking at the code, I noticed this will only work
> if you can map all of the memory with wired TLB entries.
> If you have more than 24M of memory, make sure you
> enable CONFIG_MODULES to eliminate the TLB miss
> optimization that will certainly crash a system by failing
> to look up kernel PTEs correctly.

hmm, I have more than 24MB of memory and I can run CONFIG_PIN_TLB just
fine with modules off in kernel 2.4. Havn't tried 2.6 yet.

 Jocke

^ permalink raw reply

* Re: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Dan Malek @ 2005-04-24  0:00 UTC (permalink / raw)
  To: Joakim.Tjernlund; +Cc: linuxppc-embedded
In-Reply-To: <BCEFJBPJCGFCNMMMIDBHKEILCLAA.Joakim.Tjernlund@lumentis.se>


On Apr 23, 2005, at 7:51 PM, Joakim Tjernlund wrote:

> hmm, I have more than 24MB of memory and I can run CONFIG_PIN_TLB just
> fine with modules off in kernel 2.4. Havn't tried 2.6 yet.

Doh.  Oh, I see.  We only do the optimization for the instruction 
misses.
I'll have to take a closer look at Marcelo's 2.6 tests.


	-- Dan

^ permalink raw reply

* Re: [PATCH 2.6.12-rc3] pmac: save master volume on sleep
From: Colin Leroy @ 2005-04-24 16:24 UTC (permalink / raw)
  To: Benjamin Herrenschmidt
  Cc: Takashi Iwai, linuxppc-dev list, debian-powerpc@lists.debian.org,
	Andrew Morton
In-Reply-To: <1114129034.18680.34.camel@gaston>

On 22 Apr 2005 at 10h04, Benjamin Herrenschmidt wrote:

Hi, 

> > Hi,
> > 
> > Ben's patch that shutdowns master switch and restores it after
> > resume ("pmac: Improve sleep code of tumbler driver") isn't enough
> > here on an iBook (snapper chip).
> > 
> > The master switch is correctly saved and restored, but somehow
> > tumbler_put_master_volume() gets called just after
> > tumbler_set_master_volume() and sets mix->master_vol[*] to 0. So, on
> > resuming, the master switch is reenabled, but the volume is set to
> > 0.
> > 
> > Here's a patch that also saves and restores master_vol.
> > Thanks,
> 
> Looks good.

Takashi: any news or comments on this?

Thanks,
-- 
Colin

^ permalink raw reply

* Re: [PATCH 2.6.12-rc3] pmac: save master volume on sleep
From: Takashi Iwai @ 2005-04-24 16:43 UTC (permalink / raw)
  To: Colin Leroy
  Cc: Takashi Iwai, debian-powerpc@lists.debian.org, Andrew Morton,
	linuxppc-dev list
In-Reply-To: <20050424182456.34e92bd2@jack.colino.net>

Hi Colin,

the patch looks OK to me too.
Could you resend this to Jaroslav to apply it to ALSA tree, since I'm
in vacation...?

Thanks.


Takashi

On 4/24/05, Colin Leroy <colin@colino.net> wrote:
> On 22 Apr 2005 at 10h04, Benjamin Herrenschmidt wrote:
>=20
> Hi,
>=20
> > > Hi,
> > >
> > > Ben's patch that shutdowns master switch and restores it after
> > > resume ("pmac: Improve sleep code of tumbler driver") isn't enough
> > > here on an iBook (snapper chip).
> > >
> > > The master switch is correctly saved and restored, but somehow
> > > tumbler_put_master_volume() gets called just after
> > > tumbler_set_master_volume() and sets mix->master_vol[*] to 0. So, on
> > > resuming, the master switch is reenabled, but the volume is set to
> > > 0.
> > >
> > > Here's a patch that also saves and restores master_vol.
> > > Thanks,
> >
> > Looks good.
>=20
> Takashi: any news or comments on this?
>=20
> Thanks,
> --
> Colin
>=20
>

^ permalink raw reply

* Re: Status of linux-2.5-mpc52xx tree
From: Sylvain Munaut @ 2005-04-24 18:12 UTC (permalink / raw)
  To: Grant Likely; +Cc: linuxppc-embedded
In-Reply-To: <528646bc050422101815562d6c@mail.gmail.com>

Grant Likely wrote:
> Sylvain, I'm wondering what the status is of the linux-2.5-mpc52xx
> tree that you're maintaining.  Specifically, what patches have not yet
> been merged into Linus' tree?  Also, with the whole BK debacle, have
> you given any thought to what you are going to transition to?


Globally I'm not using BK anymore (and don't want to agree with the new 
license that would forbid me anywork on something close to a SCM) so the 
BK tree will be shut down very soon.

As a replacement that will just be patches for the moment, ... I'm 
constructing the set right now. I've received many patches and 
testing/reviewing all of then is taking me more time than I thought. 
Also testing different SCM is time consuming too ...

Globally for now the whole BK thing has considerably slowed my work 
(lots of time to find something else ...). Anyway, when the patches are 
ready and working here, I'll post the link to this list.



	Sylvain

^ permalink raw reply

* Re: v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Wolfgang Denk @ 2005-04-24 20:59 UTC (permalink / raw)
  To: Marcelo Tosatti; +Cc: 26-devel, linux-ppc-embedded
In-Reply-To: <20050421183239.GJ6525@logos.cnet>

Dear Marcelo,

thanks for starting this discussion, and for providing a patch for 8xx.

However, I think we should not only look at the TLB handling problems
on the 8xx processors. This is probably just a part of  the  problem.
In  general  the 2.6 performance on (small) embedded systems is much,
much worse than what we see with a 2.4 kernel.

I put some results (2.4.25 vs. 2.6.11.7 on a MPC860 and on a MPC8240)
at http://www.denx.de/twiki/bin/view/Know/Linux24vs26

Here is the summary:

Using the 2.6 kernel on embedded  systems  implicates  the  following
disadvantages:
* Slow to build: 2.6 takes 30...40% longer to compile
* Big memory footprint in flash: the 2.6 compressed kernel image is
  30...40% bigger
* Big memory footprint in RAM: the 2.6 kernel needs 30...40% more
  RAM; the available RAM size for applications is 700kB smaller
* Slow to boot: 2.6 takes 5...15% longer to boot into multi-user mode
* Slow to run: context switches up to 96% slower, local communication
  latencies up to 80% slower, file system latencies up to 76% slower,
  local communication bandwidth less than 50% in some cases.

It's a disappointing result, indeed.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Another megabytes the dust.

^ permalink raw reply

* [PATCH] ppc trivial iomem annotations: pmac_time.c
From: Al Viro @ 2005-04-24 21:30 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-sent/arch/ppc/platforms/pmac_time.c RC12-rc3-pmac_time/arch/ppc/platforms/pmac_time.c
--- RC12-rc3-sent/arch/ppc/platforms/pmac_time.c	Fri Sep 10 01:46:30 2004
+++ RC12-rc3-pmac_time/arch/ppc/platforms/pmac_time.c	Sun Apr 24 16:20:03 2005
@@ -165,7 +165,7 @@
 via_calibrate_decr(void)
 {
 	struct device_node *vias;
-	volatile unsigned char *via;
+	volatile unsigned char __iomem *via;
 	int count = VIA_TIMER_FREQ_6 / 100;
 	unsigned int dstart, dend;
 
@@ -176,8 +176,7 @@
 		vias = find_devices("via");
 	if (vias == 0 || vias->n_addrs == 0)
 		return 0;
-	via = (volatile unsigned char *)
-		ioremap(vias->addrs[0].address, vias->addrs[0].size);
+	via = ioremap(vias->addrs[0].address, vias->addrs[0].size);
 
 	/* set timer 1 for continuous interrupts */
 	out_8(&via[ACR], (via[ACR] & ~T1MODE) | T1MODE_CONT);
@@ -202,7 +201,7 @@
 	printk(KERN_INFO "via_calibrate_decr: ticks per jiffy = %u (%u ticks)\n",
 	       tb_ticks_per_jiffy, dstart - dend);
 
-	iounmap((void*)via);
+	iounmap(via);
 	
 	return 1;
 }

^ permalink raw reply

* [PATCH] ppc trivial iomem annotations: pmac_smp.c
From: Al Viro @ 2005-04-24 21:31 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-pmac_time/arch/ppc/platforms/pmac_smp.c RC12-rc3-pmac_smp/arch/ppc/platforms/pmac_smp.c
--- RC12-rc3-pmac_time/arch/ppc/platforms/pmac_smp.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-pmac_smp/arch/ppc/platforms/pmac_smp.c	Sun Apr 24 16:20:04 2005
@@ -91,11 +91,11 @@
 #define PSURGE_QUAD_BIC(r, v)	(PSURGE_QUAD_OUT((r), PSURGE_QUAD_IN(r) & ~(v)))
 
 /* virtual addresses for the above */
-static volatile u8 *hhead_base;
-static volatile u8 *quad_base;
-static volatile u32 *psurge_pri_intr;
-static volatile u8 *psurge_sec_intr;
-static volatile u32 *psurge_start;
+static volatile u8 __iomem *hhead_base;
+static volatile u8 __iomem *quad_base;
+static volatile u32 __iomem *psurge_pri_intr;
+static volatile u8 __iomem *psurge_sec_intr;
+static volatile u32 __iomem *psurge_start;
 
 /* values for psurge_type */
 #define PSURGE_NONE		-1
@@ -322,10 +322,10 @@
 		/* All released cards using this HW design have 4 CPUs */
 		ncpus = 4;
 	} else {
-		iounmap((void *) quad_base);
+		iounmap(quad_base);
 		if ((in_8(hhead_base + HHEAD_CONFIG) & 0x02) == 0) {
 			/* not a dual-cpu card */
-			iounmap((void *) hhead_base);
+			iounmap(hhead_base);
 			psurge_type = PSURGE_NONE;
 			return 1;
 		}

^ permalink raw reply

* [PATCH] ppc trivial iomem annotations: chrp
From: Al Viro @ 2005-04-24 21:31 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-pmac_smp/arch/ppc/platforms/chrp_pci.c RC12-rc3-chrp/arch/ppc/platforms/chrp_pci.c
--- RC12-rc3-pmac_smp/arch/ppc/platforms/chrp_pci.c	Fri Mar 11 15:54:45 2005
+++ RC12-rc3-chrp/arch/ppc/platforms/chrp_pci.c	Sun Apr 24 16:20:05 2005
@@ -129,7 +129,7 @@
 	rtas_write_config
 };
 
-volatile struct Hydra *Hydra = NULL;
+volatile struct Hydra __iomem *Hydra = NULL;
 
 int __init
 hydra_init(void)
@@ -175,13 +175,14 @@
 static void __init
 setup_python(struct pci_controller *hose, struct device_node *dev)
 {
-	u32 *reg, val;
+	u32 __iomem *reg;
+	u32 val;
 	unsigned long addr = dev->addrs[0].address;
 
 	setup_indirect_pci(hose, addr + 0xf8000, addr + 0xf8010);
 
 	/* Clear the magic go-slow bit */
-	reg = (u32 *) ioremap(dev->addrs[0].address + 0xf6000, 0x40);
+	reg = ioremap(dev->addrs[0].address + 0xf6000, 0x40);
 	val = in_be32(&reg[12]);
 	if (val & PRG_CL_RESET_VALID) {
 		out_be32(&reg[12], val & ~PRG_CL_RESET_VALID);
diff -urN RC12-rc3-pmac_smp/arch/ppc/platforms/chrp_setup.c RC12-rc3-chrp/arch/ppc/platforms/chrp_setup.c
--- RC12-rc3-pmac_smp/arch/ppc/platforms/chrp_setup.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-chrp/arch/ppc/platforms/chrp_setup.c	Sun Apr 24 16:20:05 2005
@@ -356,7 +356,7 @@
 	struct device_node *np;
 	int len, i;
 	unsigned int *iranges;
-	void *isu;
+	void __iomem *isu;
 
 	np = find_type_devices("open-pic");
 	if (np == NULL || np->n_addrs == 0)
diff -urN RC12-rc3-pmac_smp/include/asm-ppc/hydra.h RC12-rc3-chrp/include/asm-ppc/hydra.h
--- RC12-rc3-pmac_smp/include/asm-ppc/hydra.h	Wed Feb  4 09:34:02 2004
+++ RC12-rc3-chrp/include/asm-ppc/hydra.h	Sun Apr 24 16:20:05 2005
@@ -51,7 +51,7 @@
     char OpenPIC[0x40000];
 };
 
-extern volatile struct Hydra *Hydra;
+extern volatile struct Hydra __iomem *Hydra;
 
 
     /*

^ permalink raw reply

* Re: [26-devel] v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Marcelo Tosatti @ 2005-04-24 16:55 UTC (permalink / raw)
  To: Dan Malek; +Cc: Joakim.Tjernlund, linuxppc-embedded
In-Reply-To: <72ea00c21943f86d32f1a46a62ebc367@embeddededge.com>


Hi Dan, Joakim,

On Sat, Apr 23, 2005 at 08:00:39PM -0400, Dan Malek wrote:
> 
> On Apr 23, 2005, at 7:51 PM, Joakim Tjernlund wrote:
> 
> >hmm, I have more than 24MB of memory and I can run CONFIG_PIN_TLB just
> >fine with modules off in kernel 2.4. Havn't tried 2.6 yet.
> 
> Doh.  Oh, I see.  We only do the optimization for the instruction 
> misses.
> I'll have to take a closer look at Marcelo's 2.6 tests.

The PIN TLB entry option does not make much difference in my tests, 
never did.

Who wrote the code? Are there results which indicate a performance gain
from TLB pinning on 8xx? If so, where are such results? 

One problem that I've noted is that initial_mmu sets {I,D}TLB index
to be 27 (11100). 

MI_RSV4I protects TLB's 27...31.

Given that both {I,D}TLB INDEX's are _decreased_ on each update, it seems
to me that initial_mmu should set {I,D}TLB INDEX to 31, which will then 
decrease down to 27 after 4 TLB's are created.  

Another question that comes to mind is why initial_mmu does create 
additional 8Meg TLB entries for D-cache but not for I-cache: 

#ifdef CONFIG_PIN_TLB
        /* Map two more 8M kernel data pages.
        */
	...
#endif

I'll do some more CONFIG_PIN_TLB tests this week...

--- head_8xx.S.orig2	2005-04-24 17:55:59.000000000 -0300
+++ head_8xx.S	2005-04-24 17:57:44.000000000 -0300
@@ -697,7 +697,7 @@
 	tlbia			/* Invalidate all TLB entries */
 #ifdef CONFIG_PIN_TLB
 	lis	r8, MI_RSV4I@h
-	ori	r8, r8, 0x1c00
+	ori	r8, r8, 0x1f00
 #else
 	li	r8, 0
 #endif
@@ -705,7 +705,7 @@
 
 #ifdef CONFIG_PIN_TLB
 	lis	r10, (MD_RSV4I | MD_RESETVAL)@h
-	ori	r10, r10, 0x1c00
+	ori	r10, r10, 0x1f00
 	mr	r8, r10
 #else
 	lis	r10, MD_RESETVAL@h

^ permalink raw reply

* Re: v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Marcelo Tosatti @ 2005-04-24 17:25 UTC (permalink / raw)
  To: Wolfgang Denk; +Cc: 26-devel, linux-ppc-embedded
In-Reply-To: <20050424205945.372F7C1510@atlas.denx.de>

On Sun, Apr 24, 2005 at 10:59:40PM +0200, Wolfgang Denk wrote:
> Dear Marcelo,

Hi Wolfgang! 

> thanks for starting this discussion, and for providing a patch for 8xx. 

Thanks so much for spending the time to do this, some very interesting
results inside..

> However, I think we should not only look at the TLB handling problems
> on the 8xx processors. This is probably just a part of  the  problem.
> In  general  the 2.6 performance on (small) embedded systems is much,
> much worse than what we see with a 2.4 kernel. 
> 
> I put some results (2.4.25 vs. 2.6.11.7 on a MPC860 and on a MPC8240)
> at http://www.denx.de/twiki/bin/view/Know/Linux24vs26
> 
> Here is the summary:
> 
> Using the 2.6 kernel on embedded  systems  implicates  the  following
> disadvantages:
> * Slow to build: 2.6 takes 30...40% longer to compile
> * Big memory footprint in flash: the 2.6 compressed kernel image is
>   30...40% bigger 
>
> * Big memory footprint in RAM: the 2.6 kernel needs 30...40% more
>   RAM; the available RAM size for applications is 700kB smaller

I've shrank the v2.6 kernel build to a size significantly smaller than our 
v2.4 build, and performance did not increase at all. 

>From that, I could figure that the performance problem, in this case, 
was not related to decreased available free memory. From then on I started
going the TLB direction.

But yes, in general, v2.6 image is bigger and memory consumption is higher 
than v2.4. 

One important project in this area is linux-tiny, which allows one to 
disable unwanted features.

> * Slow to boot: 2.6 takes 5...15% longer to boot into multi-user mode

Others have mentioned, and I agree, that sysfs is likely to be the major 
cause for boot-time slowdown. Have you tried disabling sysfs? 

> * Slow to run: context switches up to 96% slower, local communication
>   latencies up to 80% slower, file system latencies up to 76% slower,
>   local communication bandwidth less than 50% in some cases. 

I've noticed the v2.6 scheduler context switching _more_ than v2.4...

Question: Such huge regressions are seen on MPC8xx only, MPC82xx slowdown 
is not so bad, correct? 

> It's a disappointing result, indeed.

Yes we are in bad shape :( 

^ permalink raw reply

* Re: v2.6 performance slowdown on MPC8xx: Measuring TLB cache misses
From: Wolfgang Denk @ 2005-04-24 22:51 UTC (permalink / raw)
  To: Marcelo Tosatti; +Cc: linux-ppc-embedded
In-Reply-To: <20050424172518.GB22786@logos.cnet>

Dear Marcelo,

in message <20050424172518.GB22786@logos.cnet> you wrote:
> 
> Others have mentioned, and I agree, that sysfs is likely to be the major 
> cause for boot-time slowdown. Have you tried disabling sysfs? 

No, not yet.

> Question: Such huge regressions are seen on MPC8xx only, MPC82xx slowdown 
> is not so bad, correct? 

No. You can find both the LMBENCH summar and the raw data at
http://www.denx.de/twiki/pub/Know/Linux24vs26/lmbench_results resp.
http://www.denx.de/twiki/pub/Know/Linux24vs26/lmbench_results_raw.tar.gz

In most cases the MPC8240 is as bad as the  MPC860;  just  for  local
communication   bandwidth  there  is  a  visible  dependency  on  the
processor: pipes are faster on 8240 but much slower (49%) on the 860,
but the UNIX sockets are 11% slower on 8240 while we  get  about  the
same speed as with 2.4 on the 860, etc.

Here is the context switching part:

Context switching - times in microseconds - smaller is better
-------------------------------------------------------------------------
Host                 OS  2p/0K 2p/16K 2p/64K 8p/16K 8p/64K 16p/16K 16p/64K
                         ctxsw  ctxsw  ctxsw ctxsw  ctxsw   ctxsw   ctxsw
--------- ------------- ------ ------ ------ ------ ------ ------- -------
sp8240     Linux 2.4.25         200.4   69.0  217.1   78.1   219.9    78.5
sp8240     Linux 2.4.25         207.5   63.4  229.9   93.7   230.8    81.1
sp8240     Linux 2.4.25         207.4   72.4  230.6   89.5   233.9    86.3
sp8240    Linux 2.6.11. 8.9400  254.1  143.0  261.3  161.3   259.4   160.6
sp8240    Linux 2.6.11. 8.5100  234.4  127.4  256.4  161.4   251.3   149.0
sp8240    Linux 2.6.11. 8.5400  211.8  128.0  240.2  157.7   243.7   153.9
tqm8xx     Linux 2.4.25   29.4   64.7          78.4           81.6        
tqm8xx     Linux 2.4.25   32.9   56.5          75.8           80.0        
tqm8xx     Linux 2.4.25   29.9   66.7          76.6           80.8        
tqm8xx    Linux 2.6.11.   44.7   90.3         132.1          131.3        
tqm8xx    Linux 2.6.11.   48.8  117.1         132.7          136.6        
tqm8xx    Linux 2.6.11.   47.6   90.7         126.7          133.1        

and the local comm latencies:

*Local* Communication latencies in microseconds - smaller is better
---------------------------------------------------------------------
Host                 OS 2p/0K  Pipe AF     UDP  RPC/   TCP  RPC/ TCP
                        ctxsw       UNIX         UDP         TCP conn
--------- ------------- ----- ----- ---- ----- ----- ----- ----- ----
sp8240     Linux 2.4.25        46.5 120. 522.5 1362. 842.1 1817. 2828
sp8240     Linux 2.4.25        47.1 135. 504.6 1330. 880.2 1838. 2774
sp8240     Linux 2.4.25        47.1 134. 535.2 1369. 855.4 1810. 2929
sp8240    Linux 2.6.11. 8.940  89.4 251. 683.0 1506. 1020. 2021. 3507
sp8240    Linux 2.6.11. 8.510  89.5 251. 701.7 1500. 1075. 2032. 3492
sp8240    Linux 2.6.11. 8.540  88.2 263. 703.1 1550. 1110. 2076. 3588
tqm8xx     Linux 2.4.25  29.4 145.3 309. 682.3 1427. 1000. 1896. 2992
tqm8xx     Linux 2.4.25  32.9 144.3 338. 675.9 1434. 1002. 1933. 2990
tqm8xx     Linux 2.4.25  29.9 150.5 352. 679.4 1429. 1006. 1931. 2983
tqm8xx    Linux 2.6.11.  44.7 238.8 522. 940.4 1629. 1265. 2125. 3792
tqm8xx    Linux 2.6.11.  48.8 255.2 531.             1255.       3750
tqm8xx    Linux 2.6.11.  47.6 258.6 550.             1252.       3783

Actually the 8240 is worse than the 860 in some of the tests...

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
Man did not weave the web of life; he  is  merely  a  strand  in  it.
Whatever he does to the web, he does to himself.     - Seattle [1854]

^ permalink raw reply

* Re: Status of linux-2.5-mpc52xx tree
From: Sam Song @ 2005-04-25  3:46 UTC (permalink / raw)
  To: Sylvain Munaut, Grant Likely; +Cc: linuxppc-embedded

--- Sylvain Munaut <tnt@246tNt.com> wrote:
> Globally I'm not using BK anymore (and don't want to
> agree with the new license that would forbid me 
> anywork on something close to a SCM) so the 
> BK tree will be shut down very soon.

Ummm, I also heard about some news of BK and kernel.
But still haven't had a clear understanding on the new
changes. In addition, I downloaded bk-3.2.3 and
installed
it on my RH9.0 PC yesterday but found a little more 
difficult  to use. What's happening to BK? Anyone
could 
show us a clear picture or reference?

TIA,

Sam 

_________________________________________________________
Do You Yahoo!?
150万曲MP3疯狂搜,带您闯入音乐殿堂
http://music.yisou.com/
美女明星应有尽有,搜遍美图、艳图和酷图
http://image.yisou.com
1G就是1000兆,雅虎电邮自助扩容!
http://cn.rd.yahoo.com/mail_cn/tag/1g/*http://cn.mail.yahoo.com/event/mail_1g/

^ permalink raw reply

* [PATCH] ppc iomem annotations: ->io_base_virt
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

* ->io_base_virt in struct pci_controller is iomem pointer.  Marked as such.
  Most of the places that used it are already annotated to expect iomem.
* places that did gratitious (and wrong) casts a-la
	isa_io_base = (unsigned long)ioremap(...);
	hose->io_base_virt = (void *)isa_io_base;
  turned into
	hose->io_base_virt = ioremap(...);
	isa_io_base = (unsigned long)hose->io_base_virt;
* pci_bus_io_base() annotated as returning iomem pointer.

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/kernel/pci.c RC12-rc3-io_virt_base/arch/ppc/kernel/pci.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/kernel/pci.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-io_virt_base/arch/ppc/kernel/pci.c	Mon Apr 25 00:40:04 2005
@@ -1432,7 +1432,7 @@
 	return NULL;
 }
 
-void*
+void __iomem *
 pci_bus_io_base(unsigned int bus)
 {
 	struct pci_controller *hose;
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/ebony.c RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ebony.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/ebony.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ebony.c	Mon Apr 25 00:40:04 2005
@@ -210,9 +210,8 @@
 	hose->io_space.end = EBONY_PCI_UPPER_IO;
 	hose->mem_space.start = EBONY_PCI_LOWER_MEM;
 	hose->mem_space.end = EBONY_PCI_UPPER_MEM;
-	isa_io_base =
-		(unsigned long)ioremap64(EBONY_PCI_IO_BASE, EBONY_PCI_IO_SIZE);
-	hose->io_base_virt = (void *)isa_io_base;
+	hose->io_base_virt = ioremap64(EBONY_PCI_IO_BASE, EBONY_PCI_IO_SIZE);
+	isa_io_base = (unsigned long)hose->io_base_virt;
 
 	setup_indirect_pci(hose,
 			EBONY_PCI_CFGA_PLB32,
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/luan.c RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/luan.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/luan.c	Fri Mar 11 15:54:45 2005
+++ RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/luan.c	Mon Apr 25 00:40:04 2005
@@ -223,9 +223,8 @@
 	hose->io_space.end = LUAN_PCIX_UPPER_IO;
 	hose->mem_space.start = lower_mem;
 	hose->mem_space.end = upper_mem;
-	isa_io_base =
-		(unsigned long)ioremap64(pcix_io_base, PCIX_IO_SIZE);
-	hose->io_base_virt = (void *)isa_io_base;
+	hose->io_base_virt = ioremap64(pcix_io_base, PCIX_IO_SIZE);
+	isa_io_base = (unsigned long) hose->io_base_virt;
 
 	setup_indirect_pci(hose, cfga, cfgd);
 	hose->set_cfg_type = 1;
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/ocotea.c RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ocotea.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/platforms/4xx/ocotea.c	Fri Mar 11 15:54:45 2005
+++ RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ocotea.c	Mon Apr 25 00:40:04 2005
@@ -227,9 +227,8 @@
 	hose->io_space.end = OCOTEA_PCI_UPPER_IO;
 	hose->mem_space.start = OCOTEA_PCI_LOWER_MEM;
 	hose->mem_space.end = OCOTEA_PCI_UPPER_MEM;
-	isa_io_base =
-		(unsigned long)ioremap64(OCOTEA_PCI_IO_BASE, OCOTEA_PCI_IO_SIZE);
-	hose->io_base_virt = (void *)isa_io_base;
+	hose->io_base_virt = ioremap64(OCOTEA_PCI_IO_BASE, OCOTEA_PCI_IO_SIZE);
+	isa_io_base = (unsigned long) hose->io_base_virt;
 
 	setup_indirect_pci(hose,
 			OCOTEA_PCI_CFGA_PLB32,
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/syslib/m8260_pci.c RC12-rc3-io_virt_base/arch/ppc/syslib/m8260_pci.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/syslib/m8260_pci.c	Tue Jun 29 20:03:48 2004
+++ RC12-rc3-io_virt_base/arch/ppc/syslib/m8260_pci.c	Mon Apr 25 00:40:04 2005
@@ -171,10 +171,9 @@
 	m8260_setup_pci(hose);
         hose->pci_mem_offset = MPC826x_PCI_MEM_OFFSET;
 
-        isa_io_base =
-                (unsigned long) ioremap(MPC826x_PCI_IO_BASE,
+        hose->io_base_virt = ioremap(MPC826x_PCI_IO_BASE,
                                         MPC826x_PCI_IO_SIZE);
-        hose->io_base_virt = (void *) isa_io_base;
+        isa_io_base = (unsigned long) hose->io_base_virt;
  
         /* setup resources */
         pci_init_resource(&hose->mem_resources[0],
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/syslib/mpc52xx_pci.c RC12-rc3-io_virt_base/arch/ppc/syslib/mpc52xx_pci.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/syslib/mpc52xx_pci.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-io_virt_base/arch/ppc/syslib/mpc52xx_pci.c	Mon Apr 25 00:40:04 2005
@@ -205,13 +205,11 @@
 
 	hose->pci_mem_offset = MPC52xx_PCI_MEM_OFFSET;
 
-	isa_io_base =
-		(unsigned long) ioremap(MPC52xx_PCI_IO_BASE,
-					MPC52xx_PCI_IO_SIZE);
-	hose->io_base_virt = (void *) isa_io_base;
+	hose->io_base_virt = ioremap(MPC52xx_PCI_IO_BASE, MPC52xx_PCI_IO_SIZE);
+	isa_io_base = (unsigned long) hose->io_base_virt;
 
 	hose->cfg_addr = &pci_regs->car;
-	hose->cfg_data = (void __iomem *) isa_io_base;
+	hose->cfg_data = hose->io_base_virt;
 
 	/* Setup resources */
 	pci_init_resource(&hose->mem_resources[0],
diff -urN RC12-rc3-ppc-sigcontext/arch/ppc/syslib/ppc85xx_setup.c RC12-rc3-io_virt_base/arch/ppc/syslib/ppc85xx_setup.c
--- RC12-rc3-ppc-sigcontext/arch/ppc/syslib/ppc85xx_setup.c	Fri Mar 11 15:54:45 2005
+++ RC12-rc3-io_virt_base/arch/ppc/syslib/ppc85xx_setup.c	Mon Apr 25 00:40:04 2005
@@ -280,16 +280,14 @@
 	hose_a->io_space.end = MPC85XX_PCI1_UPPER_IO;
 	hose_a->io_base_phys = MPC85XX_PCI1_IO_BASE;
 #ifdef CONFIG_85xx_PCI2
-	isa_io_base =
-		(unsigned long) ioremap(MPC85XX_PCI1_IO_BASE,
+	hose_a->io_base_virt =  ioremap(MPC85XX_PCI1_IO_BASE,
 					MPC85XX_PCI1_IO_SIZE +
 					MPC85XX_PCI2_IO_SIZE);
 #else
-	isa_io_base =
-		(unsigned long) ioremap(MPC85XX_PCI1_IO_BASE,
+	hose_a->io_base_virt =  ioremap(MPC85XX_PCI1_IO_BASE,
 					MPC85XX_PCI1_IO_SIZE);
 #endif
-	hose_a->io_base_virt = (void *) isa_io_base;
+	isa_io_base = (unsigned long)hose_a->io_base_virt;
 
 	/* setup resources */
 	pci_init_resource(&hose_a->mem_resources[0],
@@ -329,8 +327,8 @@
 	hose_b->io_space.start = MPC85XX_PCI2_LOWER_IO;
 	hose_b->io_space.end = MPC85XX_PCI2_UPPER_IO;
 	hose_b->io_base_phys = MPC85XX_PCI2_IO_BASE;
-	hose_b->io_base_virt = (void *) isa_io_base + MPC85XX_PCI1_IO_SIZE;
-
+	hose_b->io_base_virt = hose_a->io_base_virt + MPC85XX_PCI1_IO_SIZE;
+	
 	/* setup resources */
 	pci_init_resource(&hose_b->mem_resources[0],
 			MPC85XX_PCI2_LOWER_MEM,
diff -urN RC12-rc3-ppc-sigcontext/include/asm-ppc/pci-bridge.h RC12-rc3-io_virt_base/include/asm-ppc/pci-bridge.h
--- RC12-rc3-ppc-sigcontext/include/asm-ppc/pci-bridge.h	Fri Mar 11 15:54:57 2005
+++ RC12-rc3-io_virt_base/include/asm-ppc/pci-bridge.h	Mon Apr 25 00:40:04 2005
@@ -12,7 +12,7 @@
  * pci_io_base returns the memory address at which you can access
  * the I/O space for PCI bus number `bus' (or NULL on error).
  */
-extern void *pci_bus_io_base(unsigned int bus);
+extern void __iomem *pci_bus_io_base(unsigned int bus);
 extern unsigned long pci_bus_io_base_phys(unsigned int bus);
 extern unsigned long pci_bus_mem_base_phys(unsigned int bus);
 
@@ -48,7 +48,7 @@
 	int last_busno;
 	int bus_offset;
 
-	void *io_base_virt;
+	void __iomem *io_base_virt;
 	unsigned long io_base_phys;
 
 	/* Some machines (PReP) have a non 1:1 mapping of

^ permalink raw reply

* [PATCH] ppc user annotations: sigcontext
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

	sigcontext.regs is a userland pointer
Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-sent/arch/ppc/kernel/signal.c RC12-rc3-ppc-sigcontext/arch/ppc/kernel/signal.c
--- RC12-rc3-sent/arch/ppc/kernel/signal.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-ppc-sigcontext/arch/ppc/kernel/signal.c	Mon Apr 25 00:40:03 2005
@@ -632,7 +632,7 @@
 	if (__put_user((unsigned long) ka->sa.sa_handler, &sc->handler)
 	    || __put_user(oldset->sig[0], &sc->oldmask)
 	    || __put_user(oldset->sig[1], &sc->_unused[3])
-	    || __put_user((struct pt_regs *)frame, &sc->regs)
+	    || __put_user((struct pt_regs __user *)frame, &sc->regs)
 	    || __put_user(sig, &sc->signal))
 		goto badframe;
 
diff -urN RC12-rc3-sent/include/asm-ppc/sigcontext.h RC12-rc3-ppc-sigcontext/include/asm-ppc/sigcontext.h
--- RC12-rc3-sent/include/asm-ppc/sigcontext.h	Wed Feb  4 10:02:26 2004
+++ RC12-rc3-ppc-sigcontext/include/asm-ppc/sigcontext.h	Mon Apr 25 00:40:03 2005
@@ -9,7 +9,7 @@
 	int		signal;
 	unsigned long	handler;
 	unsigned long	oldmask;
-	struct pt_regs 	*regs;
+	struct pt_regs 	__user *regs;
 };
 
 #endif

^ permalink raw reply

* [PATCH] ppc iomem annotations: pmac_low_i2c
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-ebony/arch/ppc/platforms/pmac_low_i2c.c RC12-rc3-ppc-i2c/arch/ppc/platforms/pmac_low_i2c.c
--- RC12-rc3-ebony/arch/ppc/platforms/pmac_low_i2c.c	Fri Feb  6 22:04:13 2004
+++ RC12-rc3-ppc-i2c/arch/ppc/platforms/pmac_low_i2c.c	Mon Apr 25 01:36:32 2005
@@ -54,7 +54,7 @@
 	int			mode;		/* Current mode */
 	int			channel;	/* Current channel */
 	int			num_channels;	/* Number of channels */
-	unsigned long		base;		/* For keywest-i2c, base address */
+	void __iomem *		base;		/* For keywest-i2c, base address */
 	int			bsteps;		/* And register stepping */
 	int			speed;		/* And speed */
 };
@@ -154,14 +154,12 @@
 
 static inline u8 __kw_read_reg(struct low_i2c_host *host, reg_t reg)
 {
-	return in_8(((volatile u8 *)host->base)
-		+ (((unsigned)reg) << host->bsteps));
+	return in_8(host->base + (((unsigned)reg) << host->bsteps));
 }
 
 static inline void __kw_write_reg(struct low_i2c_host *host, reg_t reg, u8 val)
 {
-	out_8(((volatile u8 *)host->base)
-		+ (((unsigned)reg) << host->bsteps), val);
+	out_8(host->base + (((unsigned)reg) << host->bsteps), val);
 	(void)__kw_read_reg(host, reg_subaddr);
 }
 
@@ -370,7 +368,7 @@
 		break;
 	}	
 	host->mode = pmac_low_i2c_mode_std;
-	host->base = (unsigned long)ioremap(np->addrs[0].address + aoffset,
+	host->base = ioremap(np->addrs[0].address + aoffset,
 						np->addrs[0].size);
 	host->func = keywest_low_i2c_func;
 }

^ permalink raw reply

* [PATCH] ppc iomem annotations: ebony_setup_pcix()
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ebony.c RC12-rc3-ebony/arch/ppc/platforms/4xx/ebony.c
--- RC12-rc3-io_virt_base/arch/ppc/platforms/4xx/ebony.c	Mon Apr 25 00:40:04 2005
+++ RC12-rc3-ebony/arch/ppc/platforms/4xx/ebony.c	Mon Apr 25 00:40:06 2005
@@ -149,7 +149,7 @@
 static void __init
 ebony_setup_pcix(void)
 {
-	void *pcix_reg_base;
+	void __iomem *pcix_reg_base;
 
 	pcix_reg_base = ioremap64(PCIX0_REG_BASE, PCIX_REG_SIZE);
 

^ permalink raw reply

* [PATCH] ppc user annotations: debug_setconetext(2)
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

	3rd argument of sys_debug_setcontext() is also a userland pointer.
Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-ppc-i2c/arch/ppc/kernel/signal.c RC12-rc3-ppc-signal/arch/ppc/kernel/signal.c
--- RC12-rc3-ppc-i2c/arch/ppc/kernel/signal.c	Mon Apr 25 00:40:03 2005
+++ RC12-rc3-ppc-signal/arch/ppc/kernel/signal.c	Mon Apr 25 00:40:09 2005
@@ -511,7 +511,7 @@
 }
 
 int sys_debug_setcontext(struct ucontext __user *ctx,
-			 int ndbg, struct sig_dbg_op *dbg,
+			 int ndbg, struct sig_dbg_op __user *dbg,
 			 int r6, int r7, int r8,
 			 struct pt_regs *regs)
 {

^ permalink raw reply

* [PATCH] ppc sparse annotations: emulate_string_inst()
From: Al Viro @ 2005-04-25  6:04 UTC (permalink / raw)
  To: torvalds; +Cc: linuxppc-dev

	replaced declaration of EA from u32 to unsigned long - this beast
is used only to cast it to (userland) pointer and proper integer type for
that is unsigned long.
Signed-off-by: Al Viro <viro@parcelfarce.linux.theplanet.co.uk>
----
diff -urN RC12-rc3-mveth/arch/ppc/kernel/traps.c RC12-rc3-ppc-traps/arch/ppc/kernel/traps.c
--- RC12-rc3-mveth/arch/ppc/kernel/traps.c	Wed Apr 20 21:25:28 2005
+++ RC12-rc3-ppc-traps/arch/ppc/kernel/traps.c	Mon Apr 25 01:36:38 2005
@@ -403,7 +403,7 @@
 	u8 rA = (instword >> 16) & 0x1f;
 	u8 NB_RB = (instword >> 11) & 0x1f;
 	u32 num_bytes;
-	u32 EA;
+	unsigned long EA;
 	int pos = 0;
 
 	/* Early out if we are an invalid form of lswx */

^ permalink raw reply


This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox