Disabling L1 D-cache and side effects

Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]

 
Hi,
  I had to bring up a PPC based SOC with L1 dcache disabled.  I did that
and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see the
file system errors. In case of NFS mount I see error saying failed to
load ld.so library.
 
 Could you guys please share thoughts what are the different side
effects might be causing this.
Thanks,
Marri


________________________________

From: Tirumala Reddy Marri 
Sent: Saturday, September 27, 2008 2:37 PM
To: linuxppc-dev@ozlabs.org
Subject: rootfs mount problem


Hi ,
  I am trying to bring up a new SOC. I am seeing the following error.
Has any one seen this error before. I am pretty sure RAMDISK is not
corrupted.
Thanks,
Marri
------------------- LOG ---------------
NET: Registered protocol family 17
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
prepare_namespace line 361 
prepare_namespace line 385 
RAMDISK: Compressed image found at block 0
invalid compressed format (err=1)
EXT3-fs error (device ram0): ext3_check_descriptors: Block bitmap for
group 0 not in group (block 14728687)!
EXT3-fs: group descriptors corrupted!
EXT2-fs error (device ram0): ext2_check_descriptors: Block bitmap for
group 0 not in group (block 14728687)!
EXT2-fs: group descriptors corrupted!
List of all partitions:
No filesystem could mount root, tried:  ext3 ext2 cramfs
Kernel panic - not syncing: VFS: Unable to mount root fs on
unknown-block(1,0)
Rebooting in 180 seconds..?
 
U-Boot 1.3.2 (Sep 24 2008 - 14:25:32)

----------------------------------------------- back trace -----------
(gdb) bt
#0  __delay (loops=800000) at include/asm/time.h:97
#1  0xc001f07c in panic (fmt=Variable "fmt" is not available.
) at kernel/panic.c:115
#2  0xc0248dc4 in mount_block_root (name=0xc020f9f4 "/dev/root",
flags=32768) at init/do_mounts.c:277
#3  0xc02491b0 in prepare_namespace () at init/do_mounts.c:403
#4  0xc02489a8 in kernel_init (unused=Variable "unused" is not
available.
) at init/main.c:878
#5  0xc000cf08 in kernel_thread ()
Previous frame inner to this frame (corrupt stack?)
(gdb) 
---------------

[-- Attachment #2: Type: text/html, Size: 4558 bytes --]

Re: Disabling L1 D-cache and side effects

[Olof Johansson]

On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote:
>  
> Hi,
>   I had to bring up a PPC based SOC with L1 dcache disabled.  I did that
> and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see the
> file system errors. In case of NFS mount I see error saying failed to
> load ld.so library.
>  
>  Could you guys please share thoughts what are the different side
> effects might be causing this.

There are a number of things you have to be careful about when you
disable caches. Depending on your implementation, you likely can't use
lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that
this is caused only by disabling L1D, and not by any other problems with
your silicon?

If you're doing early bringup of you'll have a whole lot of debug work
in front of you, and this mailing list is probably not the best place
to bring your homework.


-Olof

RE: Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

 Olof,
  Thanks for the response.

  Is there a piece of code in Linux which does the Machine check
recovery and continue normal execution ?
Thanks and Regards,
Marri

-----Original Message-----
From: Olof Johansson [mailto:olof@lixom.net]=20
Sent: Monday, September 29, 2008 11:05 AM
To: Tirumala Reddy Marri
Cc: linuxppc-dev@ozlabs.org
Subject: Re: Disabling L1 D-cache and side effects

On Mon, Sep 29, 2008 at 10:05:41AM -0700, Tirumala Reddy Marri wrote:
> =20
> Hi,
>   I had to bring up a PPC based SOC with L1 dcache disabled.  I did=20
> that and tried to boot Linux using RAMDISK/NFS mount. In RAMDISK I see

> the file system errors. In case of NFS mount I see error saying failed

> to load ld.so library.
> =20
>  Could you guys please share thoughts what are the different side=20
> effects might be causing this.

There are a number of things you have to be careful about when you
disable caches. Depending on your implementation, you likely can't use
lwarx/stwcx, cache ops will not work, etc. Also, are you 100% sure that
this is caused only by disabling L1D, and not by any other problems with
your silicon?

If you're doing early bringup of you'll have a whole lot of debug work
in front of you, and this mailing list is probably not the best place to
bring your homework.


-Olof

Re: Disabling L1 D-cache and side effects

[Olof Johansson]

On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote:
>  Olof,
>   Thanks for the response.
> 
>   Is there a piece of code in Linux which does the Machine check
> recovery and continue normal execution ?

Yes, there is. Several pieces, actually.


-Olof

RE: Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

Hi Olof,
   I see code in arch/powerpc/kernel/traps.c . But only
cbe_machine_check_handler() is the only function assigned to
ppc_md.machine_check_exception. Which is also not doing any recovery. It
just dumps the registers and return 0. Which would cause system Panic.


Could you please point me to the which does the Critical error (Machine
Check) recovery. BTW I am successful booting the Linux until rootfs is
being mounted. It fails to mount the Linux saying that blocks are
corrupted in file system. I had to modify lots of initial bring up code
to disable D-cache and make sure all TLB's are cache inhibited. Ando
also made sure none of the misc_32.S , entry_32.S and head.S makes any
references to d-cache.


Thanks,
Marri

-----Original Message-----
From: Olof Johansson [mailto:olof@lixom.net]=20
Sent: Monday, September 29, 2008 2:14 PM
To: Tirumala Reddy Marri
Cc: linuxppc-dev@ozlabs.org
Subject: Re: Disabling L1 D-cache and side effects

On Mon, Sep 29, 2008 at 02:00:06PM -0700, Tirumala Reddy Marri wrote:
>  Olof,
>   Thanks for the response.
>=20
>   Is there a piece of code in Linux which does the Machine check=20
> recovery and continue normal execution ?

Yes, there is. Several pieces, actually.


-Olof

RE: Disabling L1 D-cache and side effects

[Benjamin Herrenschmidt]

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> Could you please point me to the which does the Critical error (Machine
> Check) recovery. BTW I am successful booting the Linux until rootfs is
> being mounted. It fails to mount the Linux saying that blocks are
> corrupted in file system. I had to modify lots of initial bring up code
> to disable D-cache and make sure all TLB's are cache inhibited. Ando
> also made sure none of the misc_32.S , entry_32.S and head.S makes any
> references to d-cache.

Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that
even if you manage to plaster around all of this in the kernel, whatever
userspace code you'll try to run in userspace will blow up too...

Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

=20
Ben,
  Thanks for the response. I am wondering how user space would get
affected by absence of L1 Dcache.
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 12:16 AM
To: Tirumala Reddy Marri
Cc: Olof Johansson; linuxppc-dev@ozlabs.org
Subject: RE: Disabling L1 D-cache and side effects

On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> Could you please point me to the which does the Critical error=20
> (Machine
> Check) recovery. BTW I am successful booting the Linux until rootfs is

> being mounted. It fails to mount the Linux saying that blocks are=20
> corrupted in file system. I had to modify lots of initial bring up=20
> code to disable D-cache and make sure all TLB's are cache inhibited.=20
> Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
> makes any references to d-cache.

Why the heck are you doing that btw ? AFAIK, as Olof says, things like
atomic operations will not work, dcbz neither etc... it's likely that
even if you manage to plaster around all of this in the kernel, whatever
userspace code you'll try to run in userspace will blow up too...

Cheers,
Ben.

RE: Disabling L1 D-cache and side effects

[Benjamin Herrenschmidt]

On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
> Ben,
>   Thanks for the response. I am wondering how user space would get
> affected by absence of L1 Dcache.

You didn't answer my question :-)

Well, as I said, things like lwarx/stwcx not working, dcbz taking
alignment exceptions, etc...

Ben.

> Thanks,
> Marri
> 
> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org] 
> Sent: Tuesday, September 30, 2008 12:16 AM
> To: Tirumala Reddy Marri
> Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> Subject: RE: Disabling L1 D-cache and side effects
> 
> On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> > Could you please point me to the which does the Critical error 
> > (Machine
> > Check) recovery. BTW I am successful booting the Linux until rootfs is
> 
> > being mounted. It fails to mount the Linux saying that blocks are 
> > corrupted in file system. I had to modify lots of initial bring up 
> > code to disable D-cache and make sure all TLB's are cache inhibited. 
> > Ando also made sure none of the misc_32.S , entry_32.S and head.S 
> > makes any references to d-cache.
> 
> Why the heck are you doing that btw ? AFAIK, as Olof says, things like
> atomic operations will not work, dcbz neither etc... it's likely that
> even if you manage to plaster around all of this in the kernel, whatever
> userspace code you'll try to run in userspace will blow up too...
> 
> Cheers,
> Ben.

RE: Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

Ben,
    I got to bring up Linux on one of the 440 processors with out L1
dcache to do some bench marking  and compare with  L1 d-cache enabled. =20

I am avoiding any references to dcbz ,dcbt and dcbst .   Also the TLB's
are created with cache inhibited. I looked at lwarx/stwcx description,
there seem to be no dependency on L1 cache.

I don't see any critical exceptions or traps. All I  see is /init/bin
failing to execute because data is corrupted.=20

Thanks,
Marri


-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 2:31 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; linuxppc-dev@ozlabs.org
Subject: RE: Disabling L1 D-cache and side effects

On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
> Ben,
>   Thanks for the response. I am wondering how user space would get=20
> affected by absence of L1 Dcache.

You didn't answer my question :-)

Well, as I said, things like lwarx/stwcx not working, dcbz taking
alignment exceptions, etc...

Ben.

> Thanks,
> Marri
>=20
> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> Sent: Tuesday, September 30, 2008 12:16 AM
> To: Tirumala Reddy Marri
> Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> Subject: RE: Disabling L1 D-cache and side effects
>=20
> On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> > Could you please point me to the which does the Critical error=20
> > (Machine
> > Check) recovery. BTW I am successful booting the Linux until rootfs=20
> > is
>=20
> > being mounted. It fails to mount the Linux saying that blocks are=20
> > corrupted in file system. I had to modify lots of initial bring up=20
> > code to disable D-cache and make sure all TLB's are cache inhibited.
> > Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
> > makes any references to d-cache.
>=20
> Why the heck are you doing that btw ? AFAIK, as Olof says, things like

> atomic operations will not work, dcbz neither etc... it's likely that=20
> even if you manage to plaster around all of this in the kernel,=20
> whatever userspace code you'll try to run in userspace will blow up
too...
>=20
> Cheers,
> Ben.

RE: Disabling L1 D-cache and side effects

[Benjamin Herrenschmidt]

On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote:
> Ben,
>     I got to bring up Linux on one of the 440 processors with out L1
> dcache to do some bench marking  and compare with  L1 d-cache enabled.  
> 
> I am avoiding any references to dcbz ,dcbt and dcbst .   Also the TLB's
> are created with cache inhibited. I looked at lwarx/stwcx description,
> there seem to be no dependency on L1 cache.

Ok. Well, they are generally implemented at the L2 level but maybe not
on 440, architecturally, they must be used on cacheable memory but it's
possible that 440 being not SMP coherent, the actual implementation of
those is too dumb to care.

> I don't see any critical exceptions or traps. All I  see is /init/bin
> failing to execute because data is corrupted. 

Have you properly replaced dcbz with multiple stores ? I did some bring
up work internally on some stuff where dcbz wasn't quite there yet and
one pitfall to be careful is that if you force-enable the alternate
CONFIG_8xx implementation in the various copy & memset routines in
arch/powerpc/lib, you also need to fix those implementations to copy
or clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines.

Typically failing to do so causes things like memset to fail to properly
clear things such as page tables and thus random crap occurs.

Cheers,
Ben.

> Thanks,
> Marri

> 
> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org] 
> Sent: Tuesday, September 30, 2008 2:31 PM
> To: Tirumala Reddy Marri
> Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> Subject: RE: Disabling L1 D-cache and side effects
> 
> On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
> > Ben,
> >   Thanks for the response. I am wondering how user space would get 
> > affected by absence of L1 Dcache.
> 
> You didn't answer my question :-)
> 
> Well, as I said, things like lwarx/stwcx not working, dcbz taking
> alignment exceptions, etc...
> 
> Ben.
> 
> > Thanks,
> > Marri
> > 
> > -----Original Message-----
> > From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> > Sent: Tuesday, September 30, 2008 12:16 AM
> > To: Tirumala Reddy Marri
> > Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> > Subject: RE: Disabling L1 D-cache and side effects
> > 
> > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> > > Could you please point me to the which does the Critical error 
> > > (Machine
> > > Check) recovery. BTW I am successful booting the Linux until rootfs 
> > > is
> > 
> > > being mounted. It fails to mount the Linux saying that blocks are 
> > > corrupted in file system. I had to modify lots of initial bring up 
> > > code to disable D-cache and make sure all TLB's are cache inhibited.
> > > Ando also made sure none of the misc_32.S , entry_32.S and head.S 
> > > makes any references to d-cache.
> > 
> > Why the heck are you doing that btw ? AFAIK, as Olof says, things like
> 
> > atomic operations will not work, dcbz neither etc... it's likely that 
> > even if you manage to plaster around all of this in the kernel, 
> > whatever userspace code you'll try to run in userspace will blow up
> too...
> > 
> > Cheers,
> > Ben.

RE: Disabling L1 D-cache and side effects

[Tirumala Reddy Marri]

Ben,
  You are right. After I corrected copy_page and __copy_tofrom_user
functions Linux booted normally. Thanks a lot for the suggestions.
Thanks,
Marri

-----Original Message-----
From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]=20
Sent: Tuesday, September 30, 2008 3:56 PM
To: Tirumala Reddy Marri
Cc: Olof Johansson; linuxppc-dev@ozlabs.org
Subject: RE: Disabling L1 D-cache and side effects

On Tue, 2008-09-30 at 15:26 -0700, Tirumala Reddy Marri wrote:
> Ben,
>     I got to bring up Linux on one of the 440 processors with out L1=20
> dcache to do some bench marking  and compare with  L1 d-cache enabled.
>=20
> I am avoiding any references to dcbz ,dcbt and dcbst .   Also the
TLB's
> are created with cache inhibited. I looked at lwarx/stwcx description,

> there seem to be no dependency on L1 cache.

Ok. Well, they are generally implemented at the L2 level but maybe not
on 440, architecturally, they must be used on cacheable memory but it's
possible that 440 being not SMP coherent, the actual implementation of
those is too dumb to care.

> I don't see any critical exceptions or traps. All I  see is /init/bin=20
> failing to execute because data is corrupted.

Have you properly replaced dcbz with multiple stores ? I did some bring
up work internally on some stuff where dcbz wasn't quite there yet and
one pitfall to be careful is that if you force-enable the alternate
CONFIG_8xx implementation in the various copy & memset routines in
arch/powerpc/lib, you also need to fix those implementations to copy or
clear 32 bytes instead of just 16, as 8xx has 16 byte cache lines.

Typically failing to do so causes things like memset to fail to properly
clear things such as page tables and thus random crap occurs.

Cheers,
Ben.

> Thanks,
> Marri

>=20
> -----Original Message-----
> From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> Sent: Tuesday, September 30, 2008 2:31 PM
> To: Tirumala Reddy Marri
> Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> Subject: RE: Disabling L1 D-cache and side effects
>=20
> On Tue, 2008-09-30 at 09:57 -0700, Tirumala Reddy Marri wrote:
> > Ben,
> >   Thanks for the response. I am wondering how user space would get=20
> > affected by absence of L1 Dcache.
>=20
> You didn't answer my question :-)
>=20
> Well, as I said, things like lwarx/stwcx not working, dcbz taking=20
> alignment exceptions, etc...
>=20
> Ben.
>=20
> > Thanks,
> > Marri
> >=20
> > -----Original Message-----
> > From: Benjamin Herrenschmidt [mailto:benh@kernel.crashing.org]
> > Sent: Tuesday, September 30, 2008 12:16 AM
> > To: Tirumala Reddy Marri
> > Cc: Olof Johansson; linuxppc-dev@ozlabs.org
> > Subject: RE: Disabling L1 D-cache and side effects
> >=20
> > On Mon, 2008-09-29 at 14:38 -0700, Tirumala Reddy Marri wrote:
> > > Could you please point me to the which does the Critical error=20
> > > (Machine
> > > Check) recovery. BTW I am successful booting the Linux until=20
> > > rootfs is
> >=20
> > > being mounted. It fails to mount the Linux saying that blocks are=20
> > > corrupted in file system. I had to modify lots of initial bring up

> > > code to disable D-cache and make sure all TLB's are cache
inhibited.
> > > Ando also made sure none of the misc_32.S , entry_32.S and head.S=20
> > > makes any references to d-cache.
> >=20
> > Why the heck are you doing that btw ? AFAIK, as Olof says, things=20
> > like
>=20
> > atomic operations will not work, dcbz neither etc... it's likely=20
> > that even if you manage to plaster around all of this in the kernel,

> > whatever userspace code you'll try to run in userspace will blow up
> too...
> >=20
> > Cheers,
> > Ben.