the memory model being used in elks?

the memory model being used in elks?

[u-vpoa]

Hello,

Finally I tried to test the "pres" utility on elks
as opposed to elksemu.

Too bad, it reports lack of memory
and it looks like the maximum available memory is around 30K,
the console shows:

 sys_brk failed: len -32634 > end_seg 30864
 sys_brk failed: len -32632 > end_seg 30864
 sys_brk failed: len 31368 > end_seg 30864
  ...

I assumed somehow a data segment with around 60K available,
is this not the case? 

The same binary works just fine under elksemu.

Rl

Re: the memory model being used in elks?

[One Thousand Gnomes]

On Fri, 8 May 2015 22:14:34 +0200
u-vpoa@aetey.se wrote:

> Hello,
> 
> Finally I tried to test the "pres" utility on elks
> as opposed to elksemu.
> 
> Too bad, it reports lack of memory
> and it looks like the maximum available memory is around 30K,
> the console shows:
> 
>  sys_brk failed: len -32634 > end_seg 30864
>  sys_brk failed: len -32632 > end_seg 30864
>  sys_brk failed: len 31368 > end_seg 30864
>   ...
> 
> I assumed somehow a data segment with around 60K available,
> is this not the case? 

Did you compile it with split I and D ? (-i)


Alan

Re: the memory model being used in elks?

[u-vpoa]

On Sat, May 09, 2015 at 06:03:33PM +0100, One Thousand Gnomes wrote:
> >  sys_brk failed: len 31368 > end_seg 30864
> >   ...
> > 
> > I assumed somehow a data segment with around 60K available,
> > is this not the case? 
> 
> Did you compile it with split I and D ? (-i)

Thanks for the suggestion.

I expected this to be the default, which corresponded to the report
from the elks "file" command (all files and "pres" too were reported as
split i/d). "File" might have been wrong though.

Now I also have rechecked the make logs and bcc documentation.
There was no "-i" flag at the build and the linker command was "bcc".

The man page states that the "-i" flag makes bcc to avoid passing "-i"
to the linker and would in that case lead to creation of an "impure
executable".

IOW, as much as I can see, the binary is built correctly and it is
the memory allocation by the kernel which looks surprizing.

I have tried building kernels both with and without the "advanced
memory manager", the resulting memory limitation looks the same.

Rl

Re: the memory model being used in elks?

[Jody Bruchon]

On 5/9/2015 1:47 PM, u-vpoa@aetey.se wrote:
> IOW, as much as I can see, the binary is built correctly and it is
> the memory allocation by the kernel which looks surprizing.
It is my understanding that bcc assumes DS=ES=SS and both stack and heap 
data use the same code segment. How did you determine the memory usage 
of the program to be 60K? Does that include all usage or only heap usage?

Re: the memory model being used in elks?

[u-vpoa]

On Sat, May 09, 2015 at 01:50:44PM -0400, Jody Bruchon wrote:
> On 5/9/2015 1:47 PM, u-vpoa@aetey.se wrote:
> >IOW, as much as I can see, the binary is built correctly and it is
> >the memory allocation by the kernel which looks surprizing.

> It is my understanding that bcc assumes DS=ES=SS and both stack and heap
> data use the same code segment.

Yes, this is fine.

> How did you determine the memory usage of
> the program to be 60K? Does that include all usage or only heap usage?

I counted the approximate size of the static and allocated data
objects and also tested. This includes all the data ("data+bss"+heap+stack).

This worked fine with separate i/d on Venix/86 on IBM PC XT, where DS=SS
and heap and stack grew from the opposite ends of the segment, heap from
the bottom up and the stack from the top down (the hole in the middle if
big enough was used for other processes' data, not wasted iirc).

Venix also had kernel support for ES as extra data of a process so
f.i. access to the graphic memory was done via ES. The compiler was not
involved in these tricks, it just did not use ES.

Rl

Re: the memory model being used in elks?

[Juan Perez-Sanchez]

On Sat, May 9, 2015 at 1:38 PM,  <u-vpoa@aetey.se> wrote:
> On Sat, May 09, 2015 at 01:50:44PM -0400, Jody Bruchon wrote:
>> On 5/9/2015 1:47 PM, u-vpoa@aetey.se wrote:
>> >IOW, as much as I can see, the binary is built correctly and it is
>> >the memory allocation by the kernel which looks surprizing.
>
>> It is my understanding that bcc assumes DS=ES=SS and both stack and heap
>> data use the same code segment.
>
> Yes, this is fine.
>
>> How did you determine the memory usage of
>> the program to be 60K? Does that include all usage or only heap usage?
>
> I counted the approximate size of the static and allocated data
> objects and also tested. This includes all the data ("data+bss"+heap+stack).
>
> This worked fine with separate i/d on Venix/86 on IBM PC XT, where DS=SS
> and heap and stack grew from the opposite ends of the segment, heap from
> the bottom up and the stack from the top down (the hole in the middle if
> big enough was used for other processes' data, not wasted iirc).
>
> Venix also had kernel support for ES as extra data of a process so
> f.i. access to the graphic memory was done via ES. The compiler was not
> involved in these tricks, it just did not use ES.
>
> Rl
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-8086" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Compile with flags

 -i -H E000

Flag H controls the size of BSS.

Re: the memory model being used in elks?

[u-vpoa]

On Sun, May 10, 2015 at 10:31:38AM -0500, Juan Perez-Sanchez wrote:
> Compile with flags
> 
>  -i -H E000
> 
> Flag H controls the size of BSS.

I see, a minix-like static allocation (I'm guessing this is needed
because ELKS does not yet handle overlapping segments between different
processes so it does not want to allocate a whole segment to each process
per default?).

Yes, indeed, this helped, but I had to both set

 -H0xff00
and
 -DBLON=3584    (reduce the compression block size)

to get rid of "no memory".

The utility still performs better than ELKS compress
but this scaling down should not be needed.

8KB compression block size works under elksemu
(about 9KB extra for the buffers compared to the 3.5KB block),
what causes this difference?

Rl

Re: the memory model being used in elks?

[Alan Cox]

On Sun, 10 May 2015 20:40:27 +0200
u-vpoa@aetey.se wrote:

> On Sun, May 10, 2015 at 10:31:38AM -0500, Juan Perez-Sanchez wrote:
> > Compile with flags
> > 
> >  -i -H E000
> > 
> > Flag H controls the size of BSS.
> 
> I see, a minix-like static allocation (I'm guessing this is needed
> because ELKS does not yet handle overlapping segments between different
> processes so it does not want to allocate a whole segment to each process
> per default?).

Classic Unix defines the memory map as

   <--- Data ---><--- BSS ---> | <--- Space ---> | <--- Stack --- >

Where the upper divide is SP and the lower one is set by brk() or sbrk().
The space in the middle doesn't all really exist if you have an MMU with
any kind of sane paging system.

If you have support built for ELKS pattern binaries then you can do
resizing at runtime as they are laid out to allow the bss to extend

Alan

Re: the memory model being used in elks?

[u-vpoa]

On Sun, May 10, 2015 at 09:27:44PM +0100, Alan Cox wrote:
> On Sun, 10 May 2015 20:40:27 +0200
> u-vpoa@aetey.se wrote:
> > I see, a minix-like static allocation (I'm guessing this is needed
> > because ELKS does not yet handle overlapping segments between different
> > processes so it does not want to allocate a whole segment to each process
> > per default?).
> 
> Classic Unix defines the memory map as
> 
>    <--- Data ---><--- BSS ---> | <--- Space ---> | <--- Stack --- >
> 
> Where the upper divide is SP and the lower one is set by brk() or sbrk().
> The space in the middle doesn't all really exist if you have an MMU with
> any kind of sane paging system.

Even without MMU overlapping 8086 segments allow the use of this <-Space->
for stack or data+bss of other processes. Wonder if/how this is handled
in elks (lacking the time to Use-The-Source...)

> If you have support built for ELKS pattern binaries then you can do
> resizing at runtime as they are laid out to allow the bss to extend

This is what I expected - but rebuilding the kernel with and without
the corresponding option did not seem to change the result.

Indicating the desired data area size with -H improved the outcome but still
the available space looks smaller than it should be. Sigh, I did not plan
to do kernel hacking just to make the pres utility work as it used to. :)

/Rl

Re: the memory model being used in elks?

[Alan Cox]

> Even without MMU overlapping 8086 segments allow the use of this <-Space->
> for stack or data+bss of other processes. Wonder if/how this is handled
> in elks (lacking the time to Use-The-Source...)

Only if you use far pointers which is incredibly expensive, otherwise
taking the address of a stack based variable breaks.

Alan

Re: the memory model being used in elks?

[u-vpoa]

On Sun, May 10, 2015 at 10:51:18PM +0100, Alan Cox wrote:
> > Even without MMU overlapping 8086 segments allow the use of this <-Space->
> > for stack or data+bss of other processes. Wonder if/how this is handled
> > in elks (lacking the time to Use-The-Source...)
> 
> Only if you use far pointers which is incredibly expensive, otherwise
> taking the address of a stack based variable breaks.

Not at all. The process has its full 2^16 address space for data, the
stack always at the top, the data+heap at the bottom, no far pointers,
no problem to access stack based variables.

It is up to the kernel to keep track of the sizes of heaps and stacks
of all processes (which it does in any case) and move processes' data
segments in RAM when necessary so that the segments can and do overlap
but not the data. Then the RAM area inside the unused address space in
the middle of the segment is not wasted.

process1:      |data+heap>                            <stack|
process2:                     |--data+heap-->                    <--stack--|
               |<-------------------- 64K ----------------->|
                              |<-------------------- 64K ----------------->|

As many can overlap as it fits, with some extra margins for the stacks.
The same works of course for combined code+data+heap too.

Rl

Re: the memory model being used in elks?

[Alan Cox]

> but not the data. Then the RAM area inside the unused address space in
> the middle of the segment is not wasted.
> 
> process1:      |data+heap>                            <stack|
> process2:                     |--data+heap-->                    <--stack--|
>                |<-------------------- 64K ----------------->|
>                               |<-------------------- 64K ----------------->|
> 
> As many can overlap as it fits, with some extra margins for the stacks.
> The same works of course for combined code+data+heap too.

Very clever! You stil then need to know the stack sizes and automatically
trap and move which means you need to call a helper on function entry
or alloca, but that's not difficult.

It doesn't work in protected mode, which ELKS supports but for real mode
that would work beautifully.

The ELKS format basically turns the traditional arrangement around with a
predefined stack size rather than heap size, and stack at bottom, so the
heap can then expand upwards without mess.



Alan

Re: the memory model being used in elks?

[u-vpoa]

On Mon, May 11, 2015 at 10:27:08AM +0100, Alan Cox wrote:
> >                |<-------------------- 64K ----------------->|
> >                               |<-------------------- 64K ----------------->|
> > 
> > As many can overlap as it fits, with some extra margins for the stacks.
> > The same works of course for combined code+data+heap too.
> 
> Very clever! You stil then need to know the stack sizes and automatically
> trap and move which means you need to call a helper on function entry
> or alloca, but that's not difficult.

Venix/86 did not use such helpers. I think it was reserving some
reasonable amount for the stack and also was checking the stack
pointer of the running process regularly for entering the "red zone"
or something like that. This worked quite good provided that nobody put
large arrays on the stack - the latter did not work and could kill
the system.

> It doesn't work in protected mode, which ELKS supports but for real mode
> that would work beautifully.

Indeed, why not.

> The ELKS format basically turns the traditional arrangement around with a
> predefined stack size rather than heap size, and stack at bottom, so the
> heap can then expand upwards without mess.

Apparently it preallocates too much for the stack in my case.
Is there any means to tune this per binary?

Rl