Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Vladimir 'phcoder' Serbinenko]

>> >  - Low memory heap (useful to move code off kern/i386/pc/startup.S).
>> Originally I thought of a path relocator32->relocator users->mm
>> relocator32 is ready for next round of review but is untested. Now I
>> think about it mm patch isn't actually dependent on relocator32, just
>> you won't get some features (as loading big initrds and removal of
>> os_area_size/os_area_addr fields) before relocator32 is used by all
>> loaders. I will adjust mm patch to this and add
>> .(text|data|bss)-lowmem section support.
>
> I don't understand, what is the relation between relocator in loaders and
> low memory heap?
Actually low memory heap is a special case of policy based allocation.
My design is:
I have up to 4 different policies (can be more by modifying defines
but has to be hardcoded for performance reasons and multiple of 4 for
alignment reasons)
Every region knows which allocator it has to use together with which
policy. Current allocators:
-Skip. Don't use this region with given policy
-First. Try to allocate as low as possible
-Last. Try to allocate as high as possible
-Second. Allocate second free chunk from region. It's what is used currently.

The idea behind that design is that often loaders need a big
continuous chunk of memory so if loaders get memory from bottom and
the rest takes memory from top we're likely to have a chunk of
necessary size available.
To take advantage of this design kernel area (first 3/4 of memory
which aren't added to heap) has to be eliminated. For this to happen
all loaders have to use relocator.

However I can make patches without need of relocator (w/o eliminating
kernel area). Just you won't get all the advantages of policy-based
allocations
>
> I'll need to catch up with the lowmem heap discussion.  What's the approach?
>
>> What about savedefault? Which savedefault way you prefer?
>
> I think it would be good to have.  But I haven't followed on the savedefault
> discussion, I just know it would build upon the existing envfile support.
>
>> > Bigger overhauls like the fancy menu
>> I started splitting Collin's patches and actually only quite few need
>> to go to the parts already present in grub2. Perhaps 1.97 can be
>> brought to a state when gfxmenu can be compiled externally?
>
> Depends on how intrusive are those changes :-)
>
I'll present them and I'm ok if they are postponed.
> --
> Robert Millan
>
>  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
>  how) you may access your data; but nobody's threatening your freedom: we
>  still allow you to remove your data and not access it at all."
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Robert Millan]

On Wed, Aug 12, 2009 at 02:45:19PM +0200, Vladimir 'phcoder' Serbinenko wrote:
> >> >  - Low memory heap (useful to move code off kern/i386/pc/startup.S).
> >> Originally I thought of a path relocator32->relocator users->mm
> >> relocator32 is ready for next round of review but is untested. Now I
> >> think about it mm patch isn't actually dependent on relocator32, just
> >> you won't get some features (as loading big initrds and removal of
> >> os_area_size/os_area_addr fields) before relocator32 is used by all
> >> loaders. I will adjust mm patch to this and add
> >> .(text|data|bss)-lowmem section support.
> >
> > I don't understand, what is the relation between relocator in loaders and
> > low memory heap?
> Actually low memory heap is a special case of policy based allocation.
> My design is:
> I have up to 4 different policies (can be more by modifying defines
> but has to be hardcoded for performance reasons and multiple of 4 for
> alignment reasons)
> Every region knows which allocator it has to use together with which
> policy. Current allocators:
> -Skip. Don't use this region with given policy
> -First. Try to allocate as low as possible
> -Last. Try to allocate as high as possible
> -Second. Allocate second free chunk from region. It's what is used currently.
> 
> The idea behind that design is that often loaders need a big
> continuous chunk of memory so if loaders get memory from bottom and
> the rest takes memory from top we're likely to have a chunk of
> necessary size available.

But available memory is several orders of magnitude bigger than the largest
block a loader will need.  So is this really an issue?

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Vladimir 'phcoder' Serbinenko]

> But available memory is several orders of magnitude bigger than the largest
> block a loader will need.  So is this really an issue?
It's not always the case. Two examples
1) Solaris. At least some distributions of solaris use a big (70 MiB
compressed, around 200 MiB compressed) initrd which has to be loaded
as multiboot module in a single chunk. This puts biggest needed chunk
in the same order of magnitude as RAM available on some smaller
systems.
2) XNU hibernating. It requires booter to load hibernating image in a
single chunk. Even though it's compressed it can easily be 50% of
total RAM or more
>
> --
> Robert Millan
>
>  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
>  how) you may access your data; but nobody's threatening your freedom: we
>  still allow you to remove your data and not access it at all."
>
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> http://lists.gnu.org/mailman/listinfo/grub-devel
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

[RFT] Re: Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Vladimir 'phcoder' Serbinenko]

I have this "thingie" available at my git branches mm and mm+move but
it needs more testing. Unless it's tested enough it should be
postponed and not included in 1.97.
Actually I don't understand why it was proposed to include it in 1.97
at all - it changes memory management, bugs in it are likely to be
critical and benefit is only smaller core. If it was up to me I would
just postpone it. But it's possible to put it in 1.97 if someone tests
it enough.
Personally I think nested partition patch should go in 1.97 since it's
more tested and provides useful feature namely: (together with 2
smaller patches) ability to boot solaris.

On Thu, Aug 13, 2009 at 11:01 PM, Vladimir 'phcoder'
Serbinenko<phcoder@gmail.com> wrote:
>> But available memory is several orders of magnitude bigger than the largest
>> block a loader will need.  So is this really an issue?
> It's not always the case. Two examples
> 1) Solaris. At least some distributions of solaris use a big (70 MiB
> compressed, around 200 MiB compressed) initrd which has to be loaded
> as multiboot module in a single chunk. This puts biggest needed chunk
> in the same order of magnitude as RAM available on some smaller
> systems.
> 2) XNU hibernating. It requires booter to load hibernating image in a
> single chunk. Even though it's compressed it can easily be 50% of
> total RAM or more
>>
>> --
>> Robert Millan
>>
>>  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
>>  how) you may access your data; but nobody's threatening your freedom: we
>>  still allow you to remove your data and not access it at all."
>>
>>
>> _______________________________________________
>> Grub-devel mailing list
>> Grub-devel@gnu.org
>> http://lists.gnu.org/mailman/listinfo/grub-devel
>>
>
>
>
> --
> Regards
> Vladimir 'phcoder' Serbinenko
>
> Personal git repository: http://repo.or.cz/w/grub2/phcoder.git
>



-- 
Regards
Vladimir 'phcoder' Serbinenko

Personal git repository: http://repo.or.cz/w/grub2/phcoder.git

Re: [RFT] Re: Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Robert Millan]

On Sun, Aug 16, 2009 at 09:35:37PM +0200, Vladimir 'phcoder' Serbinenko wrote:
> Actually I don't understand why it was proposed to include it in 1.97
> at all - it changes memory management, bugs in it are likely to be
> critical and benefit is only smaller core. If it was up to me I would
> just postpone it. But it's possible to put it in 1.97 if someone tests
> it enough.

Well, I was expecting that this wouldn't have to change anything on how the
"normal" heap works.  The heap we have now doesn't add low memory to its
arena, so in theory it's possible for another mechanism which reuses the
same functions to use a separate region that doesn't disrupt normal heap
operation in any way.

This solves the problem with bloat in startup.S, which is the reason I
proposed lowmem heap as a feature for 1.97.

But I agree that approaches that change internal heap structures shouldn't
be pushed to 1.97.

-- 
Robert Millan

  The DRM opt-in fallacy: "Your data belongs to us. We will decide when (and
  how) you may access your data; but nobody's threatening your freedom: we
  still allow you to remove your data and not access it at all."

Re: Policy-based memory allocations (was Re: RFC: 1.97 roadmap)

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1804 bytes --]

>  =20
>>>>>  - Low memory heap (useful to move code off kern/i386/pc/startup.S)=
=2E
>>>>>          =20
>>>> Originally I thought of a path relocator32->relocator users->mm
>>>> relocator32 is ready for next round of review but is untested. Now I=

>>>> think about it mm patch isn't actually dependent on relocator32, jus=
t
>>>> you won't get some features (as loading big initrds and removal of
>>>> os_area_size/os_area_addr fields) before relocator32 is used by all
>>>> loaders. I will adjust mm patch to this and add
>>>> .(text|data|bss)-lowmem section support.
>>>>        =20
>>> I don't understand, what is the relation between relocator in loaders=
 and
>>> low memory heap?
>>>      =20
>> Actually low memory heap is a special case of policy based allocation.=

>> My design is:
>> I have up to 4 different policies (can be more by modifying defines
>> but has to be hardcoded for performance reasons and multiple of 4 for
>> alignment reasons)
>> Every region knows which allocator it has to use together with which
>> policy. Current allocators:
>> -Skip. Don't use this region with given policy
>> -First. Try to allocate as low as possible
>> -Last. Try to allocate as high as possible
>> -Second. Allocate second free chunk from region. It's what is used cur=
rently.
>>
>> The idea behind that design is that often loaders need a big
>> continuous chunk of memory so if loaders get memory from bottom and
>> the rest takes memory from top we're likely to have a chunk of
>> necessary size available.
>>    =20
>
> But available memory is several orders of magnitude bigger than the lar=
gest
> block a loader will need.  So is this really an issue?
>
>  =20
Resume from hibernation needs a lot of memory in a single chunk


--=20
Regards
Vladimir '=CF=86-coder/phcoder' Serbinenko



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 293 bytes --]