NFS root with cache on

NFS root with cache on

[Quinn Jensen]

Is anyone else having trouble with NFS root on
the 2.4.0 kernel?  It won't come up with the
KSEG0 cache on unless I pepper the network driver
with flush calls.

I'm using the new mips32 mm and cache code with
my own patches to deal with the IDT 334's weird
way bit (it's clear up at bit 12 even though the
cache is only 2k).

Quinn

Re: NFS root with cache on

[Ralf Baechle]

On Thu, Feb 01, 2001 at 01:34:49PM -0700, Quinn Jensen wrote:

> Is anyone else having trouble with NFS root on
> the 2.4.0 kernel?  It won't come up with the
> KSEG0 cache on unless I pepper the network driver
> with flush calls.

That's expected for most old network drivers that don't yet use the
new PCI DMA API documented in Documentation/DMA-mapping.txt.

What driver is this?

  Ralf

Re: NFS root with cache on

[Quinn Jensen]

>> Is anyone else having trouble with NFS root on
>> the 2.4.0 kernel?  It won't come up with the
>> KSEG0 cache on unless I pepper the network driver
>> with flush calls.
> 
> 
> That's expected for most old network drivers that don't yet use the
> new PCI DMA API documented in Documentation/DMA-mapping.txt.
> 
> What driver is this?

Both the stock 2.4.0 tulip and eepro100 drivers.  The
problem doesn't happen when I go back to 2.3.99pre8.

In the tulip driver, both the rx and tx ring descriptors
are adjusted with pci_alloc_consistent() and are only
touched through KSEG1.

But I must be totally clueless because in both kernels
tulip_rx() calls pci_unmap_single() which does nothing.
But the skb data pointers all point to KSEG0.  With
cache on, how in the world will the kernel be able to
see what just got DMA'd into the skb?

Another thing that has been haunting me is that
in 2.3.99pre8, kmalloc() has and #ifdef __mips__ that
flushes the cache and bumps the address up to KSEG1.
This is gone in 2.4.0, but from what I can tell, this
case didn't happen for skb allocations (i.e. dev_alloc_skb)
because they only set GFP_ATOMIC, and not GFP_DMA.

Quinn

Re: NFS root with cache on

[Kevin D. Kissell]

> Another thing that has been haunting me is that
> in 2.3.99pre8, kmalloc() has and #ifdef __mips__ that
> flushes the cache and bumps the address up to KSEG1.

That's really hideous!  When did that happen?
It'll sure help cache coherency problems, but
the performance impact must have been monstrous.

            Kevin K.

Re: NFS root with cache on

[Kevin D. Kissell]

> Another thing that has been haunting me is that
> in 2.3.99pre8, kmalloc() has and #ifdef __mips__ that
> flushes the cache and bumps the address up to KSEG1.

That's really hideous!  When did that happen?
It'll sure help cache coherency problems, but
the performance impact must have been monstrous.

            Kevin K.

Re: NFS root with cache on

[Jun Sun]

Quinn Jensen wrote:
> 
> >> Is anyone else having trouble with NFS root on
> >> the 2.4.0 kernel?  It won't come up with the
> >> KSEG0 cache on unless I pepper the network driver
> >> with flush calls.
> >
> >
> > That's expected for most old network drivers that don't yet use the
> > new PCI DMA API documented in Documentation/DMA-mapping.txt.
> >
> > What driver is this?
> 
> Both the stock 2.4.0 tulip and eepro100 drivers.  The
> problem doesn't happen when I go back to 2.3.99pre8.
> 

Quinn,

Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
authors for MIPS, but I am not sure they are not there.

Jun

Re: NFS root with cache on

[Quinn Jensen]

jsun@hermes.mvista.com wrote:

> Quinn Jensen wrote:
> 
>>>> Is anyone else having trouble with NFS root on
>>>> the 2.4.0 kernel?  It won't come up with the
>>>> KSEG0 cache on unless I pepper the network driver
>>>> with flush calls.
>>> 
>>> 
>>> That's expected for most old network drivers that don't yet use tye
he
>>> new PCI DMA API documented in Documentation/DMA-mapping.txt.
>>> 
>>> What driver is this?
>> 
>> Both the stock 2.4.0 tulip and eepro100 drivers.  The
>> problem doesn't happen when I go back to 2.3.99pre8.
>> 
> 
> Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
> authors for MIPS, but I am not sure they are not there.

Jun,

I have tried that in this case but it didn't help,
because the receive skb data pointers all point to
the KSEG0 view of the data anyway.  But the maddening
thing is, when I force them to KSEG1 I still see the
NFS timeout problem.  So I'm starting to wonder if
the problem is higher up in the stack.  But I do know
that the behavior does not occur on a similarly configured
x86 build of the same kernel.

Quinn

Re: NFS root with cache on

[Jun Sun]

Quinn Jensen wrote:
> 
> jsun@hermes.mvista.com wrote:
> 
> > Quinn Jensen wrote:
> >
> >>>> Is anyone else having trouble with NFS root on
> >>>> the 2.4.0 kernel?  It won't come up with the
> >>>> KSEG0 cache on unless I pepper the network driver
> >>>> with flush calls.
> >>>
> >>>
> >>> That's expected for most old network drivers that don't yet use tye
> he
> >>> new PCI DMA API documented in Documentation/DMA-mapping.txt.
> >>>
> >>> What driver is this?
> >>
> >> Both the stock 2.4.0 tulip and eepro100 drivers.  The
> >> problem doesn't happen when I go back to 2.3.99pre8.
> >>
> >
> > Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
> > authors for MIPS, but I am not sure they are not there.
> 
> Jun,
> 
> I have tried that in this case but it didn't help,
> because the receive skb data pointers all point to
> the KSEG0 view of the data anyway. 

I looked into similar problems a while back.  If I remeber correctly, the data
pointers do point to kseg0.  It is up to the driver to do appropriate
dma_cache_invalidate() (or some functions to that effect) at certain places.

What is the CPU?  It seems logical to suspect about the dma cache routines.

Jun

Re: NFS root with cache on

[Ralf Baechle]

On Mon, Feb 05, 2001 at 10:07:18AM -0800, Jun Sun wrote:

> Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
> authors for MIPS, but I am not sure they are not there.

Copybreak is just an optimization.  So even with this unused or set to a
wrong value the driver should work.

  Ralf

netbooting indy - update, elf2ecoff?

[Kenneth C Barr]

Thank you very much for your help so far!  Here's an update and another
question:

Updates:
1.  Had to attach a console (we have XZ graphics not XL)
2.  An ecoff kernel image downloaded fine.  The ELF ones download halfway
and fail silently (even with the console attached)

Question: As the vmlinux-indy-2.2.1-990226.ecoff kernel begins the boot
sequence, it hangs at "unable to open initial console."  I had unpacked
the filesystem on linux as the FAQ suggests, and my device files look
right.  So, on the NFS server, I symlinked /dev/console to /dev/ttyS0.
Now the message is replaced with a cryptic: "S<<<<<"

Perhaps that kernel doesn't have console support?  Is there any other
explanation for the "S<<<<<" failure where I used to see "...initial
console?"

I'd like to try a different kernel but I can't find an elf2ecoff binary.
Does anyone have this converted or a known-good ecoff kernel?

Ken

netbooting indy - update, elf2ecoff?

[Kenneth C Barr]

Thank you very much for your help so far!  Here's an update and another
question:

Updates:
1.  Had to attach a console (we have XZ graphics not XL)
2.  An ecoff kernel image downloaded fine.  The ELF ones download halfway
and fail silently (even with the console attached)

Question: As the vmlinux-indy-2.2.1-990226.ecoff kernel begins the boot
sequence, it hangs at "unable to open initial console."  I had unpacked
the filesystem on linux as the FAQ suggests, and my device files look
right.  So, on the NFS server, I symlinked /dev/console to /dev/ttyS0.
Now the message is replaced with a cryptic: "S<<<<<"

Perhaps that kernel doesn't have console support?  Is there any other
explanation for the "S<<<<<" failure where I used to see "...initial
console?"

I'd like to try a different kernel but I can't find an elf2ecoff binary.
Does anyone have this converted or a known-good ecoff kernel?

Ken

Re: NFS root with cache on

[Quinn Jensen]

jsun@hermes.mvista.com wrote:

>> 
>> I have tried that in this case but it didn't help,
>> because the receive skb data pointers all point to
>> the KSEG0 view of the data anyway. 
> 
> 
> I looked into similar problems a while back.  If I remeber correctly, the data
> pointers do point to kseg0.  It is up to the driver to do appropriate
> dma_cache_invalidate() (or some functions to that effect) at certain places.

Yes, the tulip driver calls pci_unmap_single() on the receive
buffer, but for mips (in asm-mips/pci.h) this call does
nothing.  And this is what is so confusing.  Only if the
receive buffer was forced to be in KSEG1 would this make
sense.

> 
> What is the CPU?  It seems logical to suspect about the dma cache routines.

Yes, I have scrubbed over my patch to the cache routines
many times, especially since on the IDT 334 the cache-way
selection for indexed cache ops is weird--they left the
way-bit up at bit 12 as if it were an 8KB cache, when
in reality it is only a 2KB cache.

Quinn

Re: NFS root with cache on

[Alan Cox]

> On Mon, Feb 05, 2001 at 10:07:18AM -0800, Jun Sun wrote:
> 
> > Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
> > authors for MIPS, but I am not sure they are not there.
> 
> Copybreak is just an optimization.  So even with this unused or set to a
> wrong value the driver should work.

If you set it wrongly you can get network failures/stalls. The normal cases
are not a problem however. You need to copybreak small frames to avoid filling
the socket queue with empty spaces. Some platforms set it to 1518 to force
a copy thereby aligning the IP header on a 4 byte boundary, which most
PCI bus master drivers wont do if you are not copying

Re: NFS root with cache on

[Alan Cox]

> On Mon, Feb 05, 2001 at 10:07:18AM -0800, Jun Sun wrote:
> 
> > Did you set rx_copybreak to 1518?  I sent patches long time ago to the driver
> > authors for MIPS, but I am not sure they are not there.
> 
> Copybreak is just an optimization.  So even with this unused or set to a
> wrong value the driver should work.

If you set it wrongly you can get network failures/stalls. The normal cases
are not a problem however. You need to copybreak small frames to avoid filling
the socket queue with empty spaces. Some platforms set it to 1518 to force
a copy thereby aligning the IP header on a 4 byte boundary, which most
PCI bus master drivers wont do if you are not copying

Re: netbooting indy - update, elf2ecoff?

[Geert Uytterhoeven]

On Mon, 5 Feb 2001, Kenneth C Barr wrote:
> Question: As the vmlinux-indy-2.2.1-990226.ecoff kernel begins the boot
> sequence, it hangs at "unable to open initial console."  I had unpacked
> the filesystem on linux as the FAQ suggests, and my device files look
> right.  So, on the NFS server, I symlinked /dev/console to /dev/ttyS0.
> Now the message is replaced with a cryptic: "S<<<<<"

However, /dev/console must _not_ be a symlink, but a character special device
with major 5 minor 1 (cfr. Documentation/devices.txt).

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven ------------- Sony Software Development Center Europe (SDCE)
Geert.Uytterhoeven@sonycom.com ------------------- Sint-Stevens-Woluwestraat 55
Voice +32-2-7248626 Fax +32-2-7262686 ---------------- B-1130 Brussels, Belgium

Re: netbooting indy - update, elf2ecoff?

[Klaus Naumann]

On Tue, 6 Feb 2001, Geert Uytterhoeven wrote:

> However, /dev/console must _not_ be a symlink, but a character special device
> with major 5 minor 1 (cfr. Documentation/devices.txt).

Unless you're using Linux/MIPS :)
The console implementation is horrible broken, so
this means that under most circumstances the 5,1 device doesn't
work. So the only option is to make /dev/console a symlink to /dev/ttyS0.

		Cya, Klaus

-- 
Full Name   : Klaus Naumann     | (http://www.mgnet.de/) (Germany)
Nickname    : Spock             | Org.: Mad Guys Network
Phone / FAX : ++49/177/7862964  | E-Mail: (spock@mgnet.de)
PGP Key     : www.mgnet.de/keys/key_spock.txt

Re: netbooting indy - update, elf2ecoff?

[Florian Lohoff]

On Tue, Feb 06, 2001 at 07:07:47PM +0100, Klaus Naumann wrote:
> On Tue, 6 Feb 2001, Geert Uytterhoeven wrote:
> 
> > However, /dev/console must _not_ be a symlink, but a character special device
> > with major 5 minor 1 (cfr. Documentation/devices.txt).
> 
> Unless you're using Linux/MIPS :)
> The console implementation is horrible broken, so
> this means that under most circumstances the 5,1 device doesn't
> work. So the only option is to make /dev/console a symlink to /dev/ttyS0.

This has nothing to do with "Linux/Mips" and is definitly not true
on all subarchs. It might be true on SGI machines but i dont even
there its necessary.

Flo
-- 
Florian Lohoff                  flo@rfc822.org             +49-5201-669912
     Why is it called "common sense" when nobody seems to have any?