Highmem on PPC?

Highmem on PPC?

[Val Henson]

What's the current state of highmem on the G4?  Does it work, not
work, kinda work, only work on UP... ?

I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which
point I'll be able to work on highmem seriously.

-VAL


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[benh]

>What's the current state of highmem on the G4?  Does it work, not
>work, kinda work, only work on UP... ?
>
>I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which
>point I'll be able to work on highmem seriously.

It's supposed to work. However, I'm having reports of dual 7450 with
1G or 1.5G of RAM locking up, I haven't been able to track those down.

Is this an Apple machine ? The 7450 has a bug that prevents use on
SMP with HW hash table walk. Apple is supposed to implement a HW
workaround for this on their machines, but if it's not an Apple machine,
then you may have to implement software TLB load.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Tue, Feb 05, 2002 at 01:52:56PM +0100, benh@kernel.crashing.org wrote:
>
> >What's the current state of highmem on the G4?  Does it work, not
> >work, kinda work, only work on UP... ?
> >
> >I hope to have a dual 7450 G4 with 1 GB RAM within a week, at which
> >point I'll be able to work on highmem seriously.
>
> It's supposed to work. However, I'm having reports of dual 7450 with
> 1G or 1.5G of RAM locking up, I haven't been able to track those down.
>
> Is this an Apple machine ? The 7450 has a bug that prevents use on
> SMP with HW hash table walk. Apple is supposed to implement a HW
> workaround for this on their machines, but if it's not an Apple machine,
> then you may have to implement software TLB load.

The dual 7450 is an apple machine.  I also have one of our beta
boards, a dual 7410 with 1 GB RAM, and we're planning on having a dual
7450 with 1 GB RAM just as soon as we get the chips.

What's your philosophy about highmem?  We map all lowmem with BATs on
SMP in order to avoid trashing SRR0/SRR1 according to your new comment
in entry.S, but it also mentions "other cpus" without saying which cpu
you're talking about. :) How are you avoiding recursive TLB faults on
the highmem pages themselves?  What cpus did you have in mind when you
wrote the SMP stuff?

I'm currently getting hard crashes on the dual 7410 with highmem.  Not
even the SMI starts xmon.  I'm not sure that it's a software bug,
which is why I'm getting the dual 7450.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Benjamin Herrenschmidt]

>
>The dual 7450 is an apple machine.  I also have one of our beta
>boards, a dual 7410 with 1 GB RAM, and we're planning on having a dual
>7450 with 1 GB RAM just as soon as we get the chips.
>
>What's your philosophy about highmem?  We map all lowmem with BATs on
>SMP in order to avoid trashing SRR0/SRR1 according to your new comment
>in entry.S, but it also mentions "other cpus" without saying which cpu
>you're talking about. :) How are you avoiding recursive TLB faults on
>the highmem pages themselves?  What cpus did you have in mind when you
>wrote the SMP stuff?

By "other CPUs", I'm thinking about CPUs that have no BATs. However,
currently, and except for the Power4 which is handled differently, those
are the embedded (4xx, 8xx) which don't do SMP, at least not now.

I don't think kernel stacks & code can ever be in highmem pages, so there
should be no risk taking a fault on highmem page in the return from
exception path.

>I'm currently getting hard crashes on the dual 7410 with highmem.  Not
>even the SMI starts xmon.  I'm not sure that it's a software bug,
>which is why I'm getting the dual 7450.

Interesting. I have similar reports about dual 800s with 1.5G of RAM.
I don't personally have a machine with that much RAM, so it makes things
a bit difficult to debug. All I noticed so far is that
enabling interrupt distribution will cause occasional lockups on my
dual 7400 at work (not related to highmem). So if you plan to chase
this highmem bug, start by disabling that option to avoid mixing
problems.

Just in case it's a spinlock bug (who knows...) you may want to hack
the spinlock debug output to use the btext engine (basically #define
printf to xmon_printf in the spinlock code and hack xmon/start.c to
force use_screen).

And let me know if you find something ;)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Tue, Feb 05, 2002 at 08:33:16PM +0100, Benjamin Herrenschmidt wrote:
>
> >I'm currently getting hard crashes on the dual 7410 with highmem.  Not
> >even the SMI starts xmon.  I'm not sure that it's a software bug,
> >which is why I'm getting the dual 7450.
>
> Interesting. I have similar reports about dual 800s with 1.5G of RAM.
> I don't personally have a machine with that much RAM, so it makes things
> a bit difficult to debug. All I noticed so far is that
> enabling interrupt distribution will cause occasional lockups on my
> dual 7400 at work (not related to highmem). So if you plan to chase
> this highmem bug, start by disabling that option to avoid mixing
> problems.

I don't use interrupt distribution.

I triggered the absolute hard hang with my usual tests: 5 or 6 of the
following running in the background:

find /mnt/cdrom -exec cat {} \; > /dev/null 2>&1 &
find /usr -exec cat {} \; > /dev/null 2>&1 &

And a make -j8 zImage in the foreground.  I didn't even start the
flood ping before it hung. :)

> Just in case it's a spinlock bug (who knows...) you may want to hack
> the spinlock debug output to use the btext engine (basically #define
> printf to xmon_printf in the spinlock code and hack xmon/start.c to
> force use_screen).

This is actually over serial console, but I'll keep that in mind when
I get the apple.

> And let me know if you find something ;)

:) You bet.  You're sure you're not seeing these hangs?  What are you
using, dual 7400 with 1 GB RAM?

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Benjamin Herrenschmidt]

>> And let me know if you find something ;)
>
>:) You bet.  You're sure you're not seeing these hangs?  What are you
>using, dual 7400 with 1 GB RAM?

I don't have a machine with enough RAM to do highmem. Actually, I could
probably test by hacking max_lowmem to 256Mb on my dual 7400, but it's
my work prod. machine so it's a bit tricky to do such tests on it.
It's not even supposed to run linux most of the time ;)

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Christopher Murtagh]

On Tue, 5 Feb 2002, Benjamin Herrenschmidt wrote:
>Interesting. I have similar reports about dual 800s with 1.5G of RAM. I
>don't personally have a machine with that much RAM, so it makes things a
>bit difficult to debug.

 Hi Ben,

 When I get back from holidays I'm going to try those fixes you
recommended (using 2.4.18pre7 or later and with SUNGEM). I'm going to try
it on my Dual450 first because my Dual800 is currently in production and I
don't have another machine to replace it with at the moment. FWIW, the
Dual800 is very stable running 2.4.18pre1 (1GB) in single CPU mode, only
when I enable SMP does it lock up.

 If there is any particular thing you want me to do to test this thing
(run particular progs, etc.) let me know.

>And let me know if you find something ;)

 Will do.

Cheers,

Chris

--

Christopher Murtagh
Webmaster / Sysadmin
Web Communications Group
McGill University
Montreal, Quebec
Canada

Tel.: (514) 398-3122
Fax:  (514) 398-2017


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Tue, Feb 05, 2002 at 08:33:16PM +0100, Benjamin Herrenschmidt wrote:
>
> And let me know if you find something ;)

I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran
an overnight test with:

NFS mounted root filesystem
while (true) do make clean; make -j8 zImage; done
crashme +2000 666 100 8:00:00 2
Flood ping in and out

On a dual 7410 Gemini with 1 GB RAM.  Used all but 70MB of memory.  No
hangs.

The weird thing is that crashme was supposed to run for 8 hours but
it's been running for over 14 hours now.  Any ideas?

Also, I don't think this was a particularly good test of highmem since
I don't think many bounce buffers were used, or that the kernel had
much reason to map/unmap many highmem pages.  Unfortunately, my SCSI
controller isn't working quite right and I can't test with a hard disk
as a result.  Any ideas for stressing the system harder?

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Benjamin Herrenschmidt]

>I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran
>an overnight test with:
>
>NFS mounted root filesystem
>while (true) do make clean; make -j8 zImage; done
>crashme +2000 666 100 8:00:00 2
>Flood ping in and out
>
>On a dual 7410 Gemini with 1 GB RAM.  Used all but 70MB of memory.  No
>hangs.
>
>The weird thing is that crashme was supposed to run for 8 hours but
>it's been running for over 14 hours now.  Any ideas?
>
>Also, I don't think this was a particularly good test of highmem since
>I don't think many bounce buffers were used, or that the kernel had
>much reason to map/unmap many highmem pages.  Unfortunately, my SCSI
>controller isn't working quite right and I can't test with a hard disk
>as a result.  Any ideas for stressing the system harder?

Well, I don't know what your previous kernel had, but recent _devel
has my fix to limit lowmem to what can be mapped by BATs on SMP.
That may help with the 7410. If the machine ends up beeing really
stable, then I'll have to look into CPU bugs for my 7450 problems.

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Mark A. Greer]

Val Henson wrote:

> Also, I don't think this was a particularly good test of highmem since
> I don't think many bounce buffers were used, or that the kernel had
> much reason to map/unmap many highmem pages.  Unfortunately, my SCSI
> controller isn't working quite right and I can't test with a hard disk
> as a result.  Any ideas for stressing the system harder?

You could try lowering max_low_mem.  You can do this by setting "Code
maturity level option"/"Prompt for advanced kernel configuration options"
and then going under where you selected HIGH_MEM and setting max_low_mem
size to something small.

It will likely take some trial and error to get it stressed enough.
You'll have to start a bunch or processes and look at where they're
getting their memory

I'd be interested in hearing how it goes.

Mark


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Thu, Feb 07, 2002 at 10:51:05PM +0100, Benjamin Herrenschmidt wrote:
>
> >as a result.  Any ideas for stressing the system harder?
>
> Well, I don't know what your previous kernel had, but recent _devel
> has my fix to limit lowmem to what can be mapped by BATs on SMP.
> That may help with the 7410. If the machine ends up beeing really
> stable, then I'll have to look into CPU bugs for my 7450 problems.

The previous kernel was 2.4.12 based, with a hack from me to limit
lowmem to 512MB.  And I was using SCSI, which has hardware bugs on my
board, in addition to exercising bounce buffers much more strongly.
In summary, I don't have the faintest idea what was causing the
hangs.  I'll try booting my old 2.4.12 kernel NFS root and see if it's
just as stable.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Thu, Feb 07, 2002 at 02:53:19PM -0700, Mark A. Greer wrote:
>
> Val Henson wrote:
>
> > Also, I don't think this was a particularly good test of highmem since
> > I don't think many bounce buffers were used, or that the kernel had
> > much reason to map/unmap many highmem pages.  Unfortunately, my SCSI
> > controller isn't working quite right and I can't test with a hard disk
> > as a result.  Any ideas for stressing the system harder?
>
> You could try lowering max_low_mem.  You can do this by setting "Code
> maturity level option"/"Prompt for advanced kernel configuration options"
> and then going under where you selected HIGH_MEM and setting max_low_mem
> size to something small.
>
> It will likely take some trial and error to get it stressed enough.
> You'll have to start a bunch or processes and look at where they're
> getting their memory

See, I don't think that user processes using highmem pages is testing
highmem much.  So we have user pages mapped by PTE's - what's changed?
Not a whole lot.  It's when we use bounce buffers in the kernel or ask
the kernel to map user pages or otherwise trigger the kmap/kunmap code
that I'm interested in.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Benjamin Herrenschmidt]

>The previous kernel was 2.4.12 based, with a hack from me to limit
>lowmem to 512MB.  And I was using SCSI, which has hardware bugs on my
>board, in addition to exercising bounce buffers much more strongly.
>In summary, I don't have the faintest idea what was causing the
>hangs.  I'll try booting my old 2.4.12 kernel NFS root and see if it's
>just as stable.

Could well be a generic kernel bug fixed in the meantime. 2.4.12 wasn't
bug free...

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Mark A. Greer]

Val Henson wrote:

> On Thu, Feb 07, 2002 at 02:53:19PM -0700, Mark A. Greer wrote:
> >
> > Val Henson wrote:
> >
> > > Also, I don't think this was a particularly good test of highmem since
> > > I don't think many bounce buffers were used, or that the kernel had
> > > much reason to map/unmap many highmem pages.  Unfortunately, my SCSI
> > > controller isn't working quite right and I can't test with a hard disk
> > > as a result.  Any ideas for stressing the system harder?
> >
> > You could try lowering max_low_mem.  You can do this by setting "Code
> > maturity level option"/"Prompt for advanced kernel configuration options"
> > and then going under where you selected HIGH_MEM and setting max_low_mem
> > size to something small.
> >
> > It will likely take some trial and error to get it stressed enough.
> > You'll have to start a bunch or processes and look at where they're
> > getting their memory
>
> See, I don't think that user processes using highmem pages is testing
> highmem much.  So we have user pages mapped by PTE's - what's changed?
> Not a whole lot.  It's when we use bounce buffers in the kernel or ask
> the kernel to map user pages or otherwise trigger the kmap/kunmap code
> that I'm interested in.

I agree.  What I had in mind was causing the kernel to kmap pages for some
reason like to copy user data into the buffer cache of a block device (see
generic_file_write).  I would make a pgm to allocate some mem (like the size
of your lowmem), mlock it down, then fork some processes that do short writes
to a block device or something like that.  I think that would do it.

Did you mentioned that your scsi ctlr or drive was bad?  If so, you'll have to
find some other block device, /dev/ram? maybe??

Mark


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Tom Rini]

On Thu, Feb 07, 2002 at 02:43:08PM -0700, Val Henson wrote:
>
> On Tue, Feb 05, 2002 at 08:33:16PM +0100, Benjamin Herrenschmidt wrote:
> >
> > And let me know if you find something ;)
>
> I pulled the latest linuxppc_2_4_devel, ported gemini to it, and ran
> an overnight test with:
>
> NFS mounted root filesystem
> while (true) do make clean; make -j8 zImage; done
> crashme +2000 666 100 8:00:00 2
> Flood ping in and out
>
> On a dual 7410 Gemini with 1 GB RAM.  Used all but 70MB of memory.  No
> hangs.
>
> The weird thing is that crashme was supposed to run for 8 hours but
> it's been running for over 14 hours now.  Any ideas?

Hmm.  Have you tried timing some 'simple' things and seeing if thats
correct?

--
Tom Rini (TR1265)
http://gate.crashing.org/~trini/

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Christopher Murtagh]

On Thu, 7 Feb 2002, Val Henson wrote:
>Also, I don't think this was a particularly good test of highmem since I
>don't think many bounce buffers were used, or that the kernel had much
>reason to map/unmap many highmem pages.  Unfortunately, my SCSI
>controller isn't working quite right and I can't test with a hard disk
>as a result.  Any ideas for stressing the system harder?

 My Dual800 ran fine for well over a week in my office with 1GB of RAM and
with SMP. It also ran with a load average of around 10 for a couple of
hours when I tried to stress test it. It was only when it started getting
a bit of network traffic after moving it into our machine room did it
crash, and then it crashed often. (It was also compiled with GMAC and BMAC
network drivers).

 So, don't know if this info helps or not, but I'd say try giving it a
network stress test as well as a heavy CPU load. Once I turned off SMP, it
seems to be very stable (still with 1 GB of RAM).

Cheers,

Chris

--

Christopher Murtagh
Webmaster / Sysadmin
Web Communications Group
McGill University
Montreal, Quebec
Canada

Tel.: (514) 398-3122
Fax:  (514) 398-2017


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Fri, Feb 08, 2002 at 01:05:23AM -0500, Christopher Murtagh wrote:
>
> On Thu, 7 Feb 2002, Val Henson wrote:
> >Also, I don't think this was a particularly good test of highmem since I
> >don't think many bounce buffers were used, or that the kernel had much
> >reason to map/unmap many highmem pages.  Unfortunately, my SCSI
> >controller isn't working quite right and I can't test with a hard disk
> >as a result.  Any ideas for stressing the system harder?
>
>  My Dual800 ran fine for well over a week in my office with 1GB of RAM and
> with SMP. It also ran with a load average of around 10 for a couple of
> hours when I tried to stress test it. It was only when it started getting
> a bit of network traffic after moving it into our machine room did it
> crash, and then it crashed often. (It was also compiled with GMAC and BMAC
> network drivers).
>
>  So, don't know if this info helps or not, but I'd say try giving it a
> network stress test as well as a heavy CPU load. Once I turned off SMP, it
> seems to be very stable (still with 1 GB of RAM).

Thanks for the suggestion, but I was doing all that work on an NFS
mounted root in addition to the two flood pings. :) I think the
network was being stressed quite well.

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Highmem on PPC?

[Val Henson]

On Tue, Feb 05, 2002 at 08:33:16PM +0100, Benjamin Herrenschmidt wrote:
>
> And let me know if you find something ;)

In my quest for a non-SCSI block device, I tried using loopback over
NFS. :) Okay, loopback doesn't normally work, and it's not surprising
that it doesn't work over NFS with highmem enabled.  If anyone is
interested, this script immediately crashes on the latest 2_4_devel,
only with NFS mounted root and with some memory in highmem:

#!/bin/bash
dd if=/dev/zero of=file bs=1024 count=10000
losetup /dev/loop0 file
mke2fs /dev/loop0

Exception info and backtrace from xmon (I have sysmap included on
Gemini):

1:mon> e
cpu 1: vector: 300 at pc = c001239c (T memcpy+0x24), lr = c00d5b9c (t transfer_none+0x40)
msr = 9032, sp = df9fdf60 [df9fdeb0]
dar = 0, dsisr = 42000000
current = df9fc000, pid = 426, comm = loop0
1:mon> t
backtrace:
c00899e4 (t nfs_prepare_write+0x10)
c00d5e5c (t lo_send+0x13c)
c00d6278 (t do_bh_filebacked+0x98)
c00d6ba0 (t loop_thread+0x194)
c00076b8 (T kernel_thread+0x2c)

I suspect that nfs_prepare_write is not properly kmapping/kunmapping
something.

If anyone else feels like fixing this, feel free... It's not high on
my list of priorities. :)

-VAL

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/