[U-Boot] early_malloc() vs. enable_caches()

[U-Boot] early_malloc() vs. enable_caches()

[Tomas Hlavacek]

Hello!

I am working on early_malloc() for U-Boot Driver Model (this malloc is
going to serve for internal DM structures during early init and it has it's
minimalistic heap in global data).

My question is how to correctly switch from early allocator to full-scale
malloc and when to enable caches.

The current state (on ARM) is:
1) gd = id;
3) enable_caches();
3) mem_malloc_init();

Proposed sequence for mallocator (in order no to loose any data from old
and not-relocated part of GD):

1) gd_old = gd;
2) gd = id;
3) mem_malloc_init();
4) relocation of DM structures
5) early_malloc_disab()
6) enable_caches();

Does it make sense? It actually boils down to one fundamental question:
When I have not-rellocated data locked in cache-lines, do I loose them once
enable_caches() is called?

Thanks,
Tomas

-- 
Tom?? Hlav??ek <tmshlvck@gmail.com>

[U-Boot] early_malloc() vs. enable_caches()

[Albert ARIBAUD]

Hi Tomas,

On Sat, 28 Jul 2012 17:46:41 +0200, Tomas Hlavacek <tmshlvck@gmail.com> wrote:
> Hello!
> 
> I am working on early_malloc() for U-Boot Driver Model (this malloc is
> going to serve for internal DM structures during early init and it has it's
> minimalistic heap in global data).
> 
> My question is how to correctly switch from early allocator to full-scale
> malloc and when to enable caches.
> 
> The current state (on ARM) is:
> 1) gd = id;
> 3) enable_caches();
> 3) mem_malloc_init();
> 
> Proposed sequence for mallocator (in order no to loose any data from old
> and not-relocated part of GD):
> 
> 1) gd_old = gd;
> 2) gd = id;
> 3) mem_malloc_init();
> 4) relocation of DM structures
> 5) early_malloc_disab()
> 6) enable_caches();

I suspect the sequence here is atomic because there is only one thread of
execution and no interrupt able to run, but still: would you not rather
disable early malloc *before* relocating DM structures?

> Does it make sense? It actually boils down to one fundamental question:
> When I have not-rellocated data locked in cache-lines, do I loose them once
> enable_caches() is called?

Based on the (possibly wrong) assumption that in the case you describe, caches
are being enabled for the first time, then there is no such thing as "locked
cache lines" because the caches were disabled so far.

If you are enabling the caches for the first time there, then all lines are
(or should be, more exactly) empty and unlocked.

> Thanks,
> Tomas

Amicalement,
-- 
Albert.

[U-Boot] [U-Boot-DM] early_malloc() vs. enable_caches()

[Marek Vasut]

Dear Albert ARIBAUD,

> Hi Tomas,
> 
> On Sat, 28 Jul 2012 17:46:41 +0200, Tomas Hlavacek <tmshlvck@gmail.com> wrote:
> > Hello!
> > 
> > I am working on early_malloc() for U-Boot Driver Model (this malloc is
> > going to serve for internal DM structures during early init and it has
> > it's minimalistic heap in global data).
> > 
> > My question is how to correctly switch from early allocator to full-scale
> > malloc and when to enable caches.
> > 
> > The current state (on ARM) is:
> > 1) gd = id;
> > 3) enable_caches();
> > 3) mem_malloc_init();
> > 
> > Proposed sequence for mallocator (in order no to loose any data from old
> > and not-relocated part of GD):
> > 
> > 1) gd_old = gd;
> > 2) gd = id;
> > 3) mem_malloc_init();
> > 4) relocation of DM structures
> > 5) early_malloc_disab()
> > 6) enable_caches();
> 
> I suspect the sequence here is atomic because there is only one thread of
> execution and no interrupt able to run, but still: would you not rather
> disable early malloc *before* relocating DM structures?

I think this was somehow solved in here ... Tomas can elaborate further I 
believe.

> > Does it make sense? It actually boils down to one fundamental question:
> > When I have not-rellocated data locked in cache-lines, do I loose them
> > once enable_caches() is called?
> 
> Based on the (possibly wrong) assumption that in the case you describe,
> caches are being enabled for the first time, then there is no such thing
> as "locked cache lines" because the caches were disabled so far.

Not so fast, on PXA there was time when we used caches as RAM in the early 
stage. We locked some cachelines and placed stack there (and therefore global 
data etc).

> If you are enabling the caches for the first time there, then all lines are
> (or should be, more exactly) empty and unlocked.
> 
> > Thanks,
> > Tomas
> 
> Amicalement,

Best regards,
Marek Vasut

[U-Boot] [U-Boot-DM] early_malloc() vs. enable_caches()

[Graeme Russ]

Hi Thomas,

P.S. I dropped the DM list...

It took a couple of other emails to get what is going on here...

On 07/29/2012 01:46 AM, Tomas Hlavacek wrote:
> Hello!
> 
> I am working on early_malloc() for U-Boot Driver Model (this malloc is
> going to serve for internal DM structures during early init and it has it's
> minimalistic heap in global data).

Not exactly on-topic, but I really hope that everything is wrapped so a
simple call to malloc() will work pre-relocation. Of course, everything you
malloc pre-relocation will have to be re-malloc'd and relocated after
relocation. Point is, early malloc should not be restricted to the driver
framework

> 
> My question is how to correctly switch from early allocator to full-scale
> malloc and when to enable caches.
> 
> The current state (on ARM) is:
> 1) gd = id;
> 3) enable_caches();
> 3) mem_malloc_init();
> 
> Proposed sequence for mallocator (in order no to loose any data from old
> and not-relocated part of GD):
> 
> 1) gd_old = gd;
> 2) gd = id;
> 3) mem_malloc_init();
> 4) relocation of DM structures
> 5) early_malloc_disab()
> 6) enable_caches();

I'm thinking:

1) Low-level CPU init
2) 'Cache-As-RAM' init
3) Global Data init
4) Pre-console buffer init
5) Early malloc() init
6) Console init
7) ...blah, blah, blah...
8) SDRAM init
9) Relocate Global Data
10) malloc() init
11) 'Disable' early malloc (i.e. malloc() now allocates from SDRAM)
12) Relocate from early_malloc_pool to malloc_pool [1]
13) enable_caches()

[1] I'm thinking possibly compile-time registered hooks...

> Does it make sense? It actually boils down to one fundamental question:
> When I have not-rellocated data locked in cache-lines, do I loose them once
> enable_caches() is called?

I believe that yes, as soon as you enable caching, everything already in
cache (gd, pre-console buffer, early malloc pool etc) is as good as gone

Regards,

Graeme

[U-Boot] [U-Boot-DM] early_malloc() vs. enable_caches()

[Marek Vasut]

Dear Graeme Russ,

> Hi Thomas,
> 
> P.S. I dropped the DM list...
> 
> It took a couple of other emails to get what is going on here...
> 
> On 07/29/2012 01:46 AM, Tomas Hlavacek wrote:
> > Hello!
> > 
> > I am working on early_malloc() for U-Boot Driver Model (this malloc is
> > going to serve for internal DM structures during early init and it has
> > it's minimalistic heap in global data).
> 
> Not exactly on-topic, but I really hope that everything is wrapped so a
> simple call to malloc() will work pre-relocation. Of course, everything you
> malloc pre-relocation will have to be re-malloc'd and relocated after
> relocation. Point is, early malloc should not be restricted to the driver
> framework
> 
> > My question is how to correctly switch from early allocator to full-scale
> > malloc and when to enable caches.
> > 
> > The current state (on ARM) is:
> > 1) gd = id;
> > 3) enable_caches();
> > 3) mem_malloc_init();
> > 
> > Proposed sequence for mallocator (in order no to loose any data from old
> > and not-relocated part of GD):
> > 
> > 1) gd_old = gd;
> > 2) gd = id;
> > 3) mem_malloc_init();
> > 4) relocation of DM structures
> > 5) early_malloc_disab()
> > 6) enable_caches();
> 
> I'm thinking:
> 
> 1) Low-level CPU init
> 2) 'Cache-As-RAM' init
> 3) Global Data init
> 4) Pre-console buffer init

Buffer?

> 5) Early malloc() init
> 6) Console init

You don't need console here ... probably, on some systems.

> 7) ...blah, blah, blah...
> 8) SDRAM init
> 9) Relocate Global Data
> 10) malloc() init
> 11) 'Disable' early malloc (i.e. malloc() now allocates from SDRAM)
> 12) Relocate from early_malloc_pool to malloc_pool [1]
> 13) enable_caches()
> 
> [1] I'm thinking possibly compile-time registered hooks...

Gurr ... you mean like INIT-something framework?

> > Does it make sense? It actually boils down to one fundamental question:
> > When I have not-rellocated data locked in cache-lines, do I loose them
> > once enable_caches() is called?
> 
> I believe that yes, as soon as you enable caching, everything already in
> cache (gd, pre-console buffer, early malloc pool etc) is as good as gone

Right ... that's why now it's copied to a safe location alongside other GD 
(global data)

> Regards,
> 
> Graeme

Best regards,
Marek Vasut

[U-Boot] [U-Boot-DM] early_malloc() vs. enable_caches()

[Tomas Hlavacek]

Hi Graeme,

On Sun, Jul 29, 2012 at 12:53 PM, Graeme Russ <graeme.russ@gmail.com> wrote:

> Not exactly on-topic, but I really hope that everything is wrapped so a
> simple call to malloc() will work pre-relocation. Of course, everything you
> malloc pre-relocation will have to be re-malloc'd and relocated after
> relocation. Point is, early malloc should not be restricted to the driver
> framework

Yes. Actually my intention is to create a wrapper for switching from
early_malloc()
to dlmalloc() based on gd->flags. I am going to do that when the concept of
early_malloc() is finished.

>
> 1) Low-level CPU init
> 2) 'Cache-As-RAM' init
> 3) Global Data init
> 4) Pre-console buffer init
> 5) Early malloc() init
> 6) Console init
> 7) ...blah, blah, blah...
> 8) SDRAM init
> 9) Relocate Global Data
> 10) malloc() init
> 11) 'Disable' early malloc (i.e. malloc() now allocates from SDRAM)
> 12) Relocate from early_malloc_pool to malloc_pool [1]
> 13) enable_caches()
>
> [1] I'm thinking possibly compile-time registered hooks...
>
>> Does it make sense? It actually boils down to one fundamental question:
>> When I have not-rellocated data locked in cache-lines, do I loose them once
>> enable_caches() is called?
>
> I believe that yes, as soon as you enable caching, everything already in
> cache (gd, pre-console buffer, early malloc pool etc) is as good as gone

Thanks for explanation!

We have reconsidered relocation of DM structures. Perhaps we are going to
keep the structures on early_heap (in copied GD) and we are going to convert
pointers inside our structures and copy data after actual relocation when the
dlmalloc and caches are up and running. We think that this might be fastest
approach.

Tomas

-- 
Tom?? Hlav??ek <tmshlvck@gmail.com>

[U-Boot] [U-Boot-DM] early_malloc() vs. enable_caches()

[Graeme Russ]

Hi Thomas,

On Sun, Jul 29, 2012 at 11:19 PM, Tomas Hlavacek <tmshlvck@gmail.com> wrote:
> Hi Graeme,
>
> On Sun, Jul 29, 2012 at 12:53 PM, Graeme Russ <graeme.russ@gmail.com> wrote:
>
>> Not exactly on-topic, but I really hope that everything is wrapped so a
>> simple call to malloc() will work pre-relocation. Of course, everything you
>> malloc pre-relocation will have to be re-malloc'd and relocated after
>> relocation. Point is, early malloc should not be restricted to the driver
>> framework
>
> Yes. Actually my intention is to create a wrapper for switching from
> early_malloc()
> to dlmalloc() based on gd->flags. I am going to do that when the concept of
> early_malloc() is finished.

Yes, I saw that patch.

>> 1) Low-level CPU init
>> 2) 'Cache-As-RAM' init
>> 3) Global Data init
>> 4) Pre-console buffer init
>> 5) Early malloc() init
>> 6) Console init
>> 7) ...blah, blah, blah...
>> 8) SDRAM init
>> 9) Relocate Global Data
>> 10) malloc() init
>> 11) 'Disable' early malloc (i.e. malloc() now allocates from SDRAM)
>> 12) Relocate from early_malloc_pool to malloc_pool [1]
>> 13) enable_caches()
>>
>> [1] I'm thinking possibly compile-time registered hooks...
>>
>>> Does it make sense? It actually boils down to one fundamental question:
>>> When I have not-rellocated data locked in cache-lines, do I loose them once
>>> enable_caches() is called?
>>
>> I believe that yes, as soon as you enable caching, everything already in
>> cache (gd, pre-console buffer, early malloc pool etc) is as good as gone
>
> Thanks for explanation!
>
> We have reconsidered relocation of DM structures. Perhaps we are going to
> keep the structures on early_heap (in copied GD) and we are going to convert

You lost me here - early_heap is in cache like ealry GD but then you talk
about copied GD which is in SDRAM...

> pointers inside our structures and copy data after actual relocation when the
> dlmalloc and caches are up and running. We think that this might be fastest
> approach.

My fear is that early_heap will end up being restricted to be usable only
be the driver framework. Perhaps you could:
1) SDRAM Init
2) Relocate GD
3) Temporarily relocate ealry heap - store offset in GD
4) enable_caches()
5) malloc() init
6) Permanently relocate early heap into malloc heap

Faster, but a lot of mucking around. And drivers are broken between steps
3 and 6 (but that should not be too much of a problem)

Regards,

Graeme