Problem booting Linux 2.6 on Virtex-4

Problem booting Linux 2.6 on Virtex-4

[David Baird]

Hi,

I'm having trouble with getting Linux to boot farther than early_init.
 Things start to go obviously wrong after early_init calls memset to
clear the .bss section.  The first symptom I noticed was that the
stack (which currently contains the link register) was getting zeroed
by memset.  This doesn't make any sense because the stack does not
intersect the .bss section;  I confirmed that the do not intersect by
using JTAG+xmd to read the registers and also by inserting print
statements.

So, I experimented further and discovered that different memory
regions seem to be aliased on to each other every 2*32*256 bytes.

e.g.:

    char * ptr = 0;
    unsigned long addr = 0xc0001234; // pick some address
    int n = 1; // pick some integer

    // Now write values to two different addresses:
    ptr[addr] = 42;
    ptr[addr + n*2*32*256] = 24;
    volatile asm ("nop");

    // Now mysteriously, ptr[addr] == 24

The reason I have written 32*256 is because that corresponds to the
number of bytes per cache line times the number of cache lines ...
because I was suspicious this might be related to caching.  However,
with experimentation, I found that the aliasing does not occur every
32*256 bytes, but rather every 32*512 bytes (hence 2*32*256).
Anyways, I looked in embed_config (and confirmed by dumping zImage.elf
with objdump) that the cache is being invalidated with "dccci".
"dccci" is being called, as expected, and therefore everything looks
good in the code.

I am completely perplexed.  The cache appears to be initialized
properly.  I don't have any trouble in real mode.  But as soon the
kernel switches into virtual mode (in head_4xx.S), then this memory
aliasing problem happens.  I wrote a completely standalone assembler
script that initializes the cache, TLBs, and virtual mode in the same
way as embed_config and head_4xx.S and so I am able to reproduce the
problem outside of the kernel.  I can attach that code if someone
thinks it will be useful - just let me know.

Does anyone have an idea about what might be going wrong?  I can send
more information or test anything that you want me to.

-David

Re: Problem booting Linux 2.6 on Virtex-4

[Enno Lübbers]

Hello David,

Am 14.01.2008 um 06:12 schrieb David Baird:

> I'm having trouble with getting Linux to boot farther than early_init.
> [...]
> So, I experimented further and discovered that different memory
> regions seem to be aliased on to each other every 2*32*256 bytes.


This sounds either like a wrong programming of an BRx/ORx memory  
controller register pair (which, AFAIK, the PPC405 does not have), or  
like a messed up address map in EDK. My guess is that an optimized/ 
sloppy implementation of the address decoder for some peripheral in an  
EDK system could produce something like you described; or there's a  
block RAM that's attached to a controller in the wrong way; or the  
bank/address parameters of the DDR controller don't match the physical  
setup... there's a lot that can go wrong obviously on a configurable  
SoC.

Can you be more specific about your hardware platform? Is this a  
reference design? What board are you using?

Regards
- Enno

-- 
Dipl.-Ing. Enno Luebbers
Computer Engineering Group
University of Paderborn	

Warburger Str. 100		
33098 Paderborn			

http://wwwcs.upb.de/cs/ag-platzner
phone:  05251 / 60-5397
fax:    05251 / 60-5377

Re: Problem booting Linux 2.6 on Virtex-4

[David Baird]

On Jan 14, 2008 1:37 AM, Enno L=FCbbers <enno.luebbers@upb.de> wrote:
> Hello David,
>
> Am 14.01.2008 um 06:12 schrieb David Baird:
>
> > I'm having trouble with getting Linux to boot farther than early_init.
> > [...]
> > So, I experimented further and discovered that different memory
> > regions seem to be aliased on to each other every 2*32*256 bytes.
>
>
> This sounds either like a wrong programming of an BRx/ORx memory
> controller register pair (which, AFAIK, the PPC405 does not have), or
> like a messed up address map in EDK. My guess is that an optimized/
> sloppy implementation of the address decoder for some peripheral in an
> EDK system could produce something like you described; or there's a
> block RAM that's attached to a controller in the wrong way; or the
> bank/address parameters of the DDR controller don't match the physical
> setup... there's a lot that can go wrong obviously on a configurable
> SoC.

What has been confusing me is that I am unable to reproduce the
problem in real mode.  I can only reproduce the problem in virtual
mode.  This leads me to believe, perhaps mistakenly, that the hardware
is implemented okay.  OTOH, neither can I see anything wrong with the
software.

> Can you be more specific about your hardware platform? Is this a
> reference design? What board are you using?

I am currently testing code on the ML403 evaluation board.  I used the
Base System Builder in EDK to create the hardware design and DDR SDRAM
is being used as the main RAM starting at address 0x00000000 and also
with OCM BRAM mapped at the very end of the address space (so that
0xfffffffc can contain code to execute on startup).

I am now trying to experiment with the hardware and see if I can find
a hardware reference design.  I will let you know what I figure out.

-David

RE: Problem booting Linux 2.6 on Virtex-4

[Stephen Neuendorffer]

What kernel version are you targeting? Are you using arch/powerpc or =
arch/ppc?  V4 has a data cache errata, which isn't currently in mainline =
arch/powerpc.

        if((mfpvr() & 0xfffff000) =3D=3D 0x20011000) {
            /* PPC errata 213: only for Virtex-4 FX */
            __asm__("mfccr0  0\n\t"
                    "oris    0,0,0x50000000@h\n\t"
                    "mtccr0  0"
                    : : : "0");
        }

Steve

> -----Original Message-----
> From: linuxppc-embedded-bounces+stephen=3Dneuendorffer.name@ozlabs.org =
[mailto:linuxppc-embedded-
> bounces+stephen=3Dneuendorffer.name@ozlabs.org] On Behalf Of David =
Baird
> Sent: Monday, January 14, 2008 12:12 PM
> To: linuxppc-embedded@ozlabs.org
> Subject: Re: Problem booting Linux 2.6 on Virtex-4
>=20
> On Jan 14, 2008 1:37 AM, Enno L=FCbbers <enno.luebbers@upb.de> wrote:
> > Hello David,
> >
> > Am 14.01.2008 um 06:12 schrieb David Baird:
> >
> > > I'm having trouble with getting Linux to boot farther than =
early_init.
> > > [...]
> > > So, I experimented further and discovered that different memory
> > > regions seem to be aliased on to each other every 2*32*256 bytes.
> >
> >
> > This sounds either like a wrong programming of an BRx/ORx memory
> > controller register pair (which, AFAIK, the PPC405 does not have), =
or
> > like a messed up address map in EDK. My guess is that an optimized/
> > sloppy implementation of the address decoder for some peripheral in =
an
> > EDK system could produce something like you described; or there's a
> > block RAM that's attached to a controller in the wrong way; or the
> > bank/address parameters of the DDR controller don't match the =
physical
> > setup... there's a lot that can go wrong obviously on a configurable
> > SoC.
>=20
> What has been confusing me is that I am unable to reproduce the
> problem in real mode.  I can only reproduce the problem in virtual
> mode.  This leads me to believe, perhaps mistakenly, that the hardware
> is implemented okay.  OTOH, neither can I see anything wrong with the
> software.
>=20
> > Can you be more specific about your hardware platform? Is this a
> > reference design? What board are you using?
>=20
> I am currently testing code on the ML403 evaluation board.  I used the
> Base System Builder in EDK to create the hardware design and DDR SDRAM
> is being used as the main RAM starting at address 0x00000000 and also
> with OCM BRAM mapped at the very end of the address space (so that
> 0xfffffffc can contain code to execute on startup).
>=20
> I am now trying to experiment with the hardware and see if I can find
> a hardware reference design.  I will let you know what I figure out.
>=20
> -David
> _______________________________________________
> Linuxppc-embedded mailing list
> Linuxppc-embedded@ozlabs.org
> https://ozlabs.org/mailman/listinfo/linuxppc-embedded

Re: Problem booting Linux 2.6 on Virtex-4

[David Baird]

Hi Steve,

On Jan 14, 2008 2:09 PM, Stephen Neuendorffer
<stephen.neuendorffer@xilinx.com> wrote:
> What kernel version are you targeting? Are you using arch/powerpc or arch/ppc?  V4 has a data cache errata, which isn't currently in mainline arch/powerpc.

I am using arch/ppc/, but to be honest, I don't yet know what the
difference is between those two trees.

Also, I am still learning how to use git, but it appears that I am
using this version:

    Tags: v2.6.24-rc7

I am actually using OpenEmbedded which executes the following commands
to get the Linux tree:

     git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
     cd linux-2.6
     git clone git://git.secretlab.ca/git/linux-2.6-virtex.git  master

>         if((mfpvr() & 0xfffff000) == 0x20011000) {
>             /* PPC errata 213: only for Virtex-4 FX */
>             __asm__("mfccr0  0\n\t"
>                     "oris    0,0,0x50000000@h\n\t"
>                     "mtccr0  0"
>                     : : : "0");
>         }

I tried out this code snippit, but it did not help :-(  I also looked
in head.S and found that this was already in there:

    #if defined(CONFIG_XILINX_VIRTEX_4_FX)
      /* PPC errata 213: only for Virtex-4 FX */
      mfccr0  0
      oris    0,0,0x50000000@h
      mtccr0  0
    #endif

Today, I tried a completely fresh design in EDK 9.2i (whereas I had
been using 9.1i).  I tried the design found in
EDKexamples/Virtex4_PPC_Example_9_2.zip.  This design works :-)  I am
not sure why it works (or rather, why the other one did not work), but
it works.

-David

(P.S. - Sorry I forgot to send to this to the list on the first try)

Re: Problem booting Linux 2.6 on Virtex-4

[Enno Lübbers]

Hi David,

Am 15.01.2008 um 01:14 schrieb David Baird:

> Today, I tried a completely fresh design in EDK 9.2i (whereas I had
> been using 9.1i).  I tried the design found in
> EDKexamples/Virtex4_PPC_Example_9_2.zip.  This design works :-)  I am
> not sure why it works (or rather, why the other one did not work), but
> it works.

I'm using EDK 9.1i  with the reference design from the Xilinx website (www.xilinx.com/ml403) 
. This one is using a PLB BRAM controller for the bootup code section,  
not an OCM BRAM. Maybe there's something wrong with the OCM  
controller? Though OCM is not cached at all...

Anyway, glad it worked out.

Regards
- Enno

-- 
Dipl.-Ing. Enno Luebbers
Computer Engineering Group
University of Paderborn	

Warburger Str. 100		
33098 Paderborn			

http://wwwcs.upb.de/cs/ag-platzner
phone:  05251 / 60-5397
fax:    05251 / 60-5377

Re: Problem booting Linux 2.6 on Virtex-4

[David Baird]

On Jan 14, 2008 1:11 PM, David Baird <dhbaird@gmail.com> wrote:
> I am now trying to experiment with the hardware and see if I can find
> a hardware reference design.  I will let you know what I figure out.

Okay, I got something to work.  I had been using EDK 9.1i.  Today, I
got hold of a copy of EDK 9.2i and I tried out a reference design that
comes with EDK.  Voila!  The aliasing problem seems to have vanished.

So, basically, I still don't know why there was an aliasing problem in
the first place.  But, now something works.  I will still try to
figure out what the original problem was...

-David