Behaviour of the VM on a embedded linux

Behaviour of the VM on a embedded linux

[Wappler Marcel]

Hi Folks,

I'm trying to figure out whats going on an embedded system I have to deal with. It's running a 2.6.24.7 kernel on 32 MBytes of RAM.
There is no swapping. There are some daemons and shells running and - a big monolithic c++ application.

The application runs a lot of pthreads on different real time priority levels. It looks like the application consumes  a huge
ammount of real memory in contrast to the assumption, that large code size is no problem due to paging out pages with unused code.

I'm not so familiar with the VM internas of the Linux kernel - I only have some general ideas about systems using MMUs. So my
assumption was, that even if a application consists of a lot of code, almost all memory pages containing executable code can be
paged out in favour of pages which are needed to store data (stacks, heap, slab). I tried to figuring this out through reading in
different books like "understanding the linux kernel" but those parts I was not able to catch anywhere.

Now there is the idea, that pages which contain code of pthreads running at real time priority cannot be paged out because of the
real time demand.

Is this true?

Thanks a lot,
Marcel

PS: please CC me personaly on replies
PPS: I'm hapy about every plausible idea

Re: Behaviour of the VM on a embedded linux

[Alex Riesen]

2008/8/22 Wappler Marcel <Marcel.Wappler@bridgeco.net>:
> I'm trying to figure out whats going on an embedded system I have to
> deal with. It's running a 2.6.24.7 kernel on 32 MBytes of RAM.
> There is no swapping. There are some daemons and shells running and
> - a big monolithic c++ application.
>
> The application runs a lot of pthreads on different real time priority levels.
> It looks like the application consumes  a huge
> ammount of real memory in contrast to the assumption, that large code
> size is no problem due to paging out pages with unused code.

Maybe the kernel wont page anything if the paging support is compiled out.
IOW, you still need paging code even if there is now swap partitions.

RE: Behaviour of the VM on a embedded linux

[Wappler Marcel]

Alex Riesen wrote:
>> I'm trying to figure out whats going on an embedded system I have to
>> deal with. It's running a 2.6.24.7 kernel on 32 MBytes of RAM.
>> There is no swapping. There are some daemons and shells running and
>> - a big monolithic c++ application.
>> 
>> The application runs a lot of pthreads on different real time
>> priority levels. It looks like the application consumes  a huge
>> ammount of real memory 
>> in contrast to the assumption, that large code size is no problem due
>> to paging out pages with unused code.
> 
> Maybe the kernel wont page anything if the paging support is compiled
> out. IOW, you still need paging code even if there is now swap
> partitions. 

Alex, this is the case - I do observe normal operation of the VM subsytem - it moves memory pages dynamicaly throughout the system.
But: when I create a large file on the tmpfs a kernel OOM occurs and kills the big monolithic application instead of stealing pages
from the application. This is the fact I'm wondering about. In the past every guy told me that code size is no problem on systems
using MMUs because the system can steal pages which contain code of the application in situations of low memory. But in my situation
this is not the case.

Any ideas?

Marcel
PS: please CC me on replies

Re: Behaviour of the VM on a embedded linux

[Christopher Snook]

Wappler Marcel wrote:
> Alex Riesen wrote:
>>> I'm trying to figure out whats going on an embedded system I have to deal
>>> with. It's running a 2.6.24.7 kernel on 32 MBytes of RAM. There is no
>>> swapping. There are some daemons and shells running and - a big
>>> monolithic c++ application.
>>> 
>>> The application runs a lot of pthreads on different real time priority
>>> levels. It looks like the application consumes  a huge ammount of real
>>> memory in contrast to the assumption, that large code size is no problem
>>> due to paging out pages with unused code.
>> Maybe the kernel wont page anything if the paging support is compiled out.
>> IOW, you still need paging code even if there is now swap partitions.
> 
> Alex, this is the case - I do observe normal operation of the VM subsytem -
> it moves memory pages dynamicaly throughout the system. But: when I create a
> large file on the tmpfs a kernel OOM occurs and kills the big monolithic
> application instead of stealing pages from the application. This is the fact
> I'm wondering about. In the past every guy told me that code size is no
> problem on systems using MMUs because the system can steal pages which
> contain code of the application in situations of low memory. But in my
> situation this is not the case.
> 
> Any ideas?
> 
> Marcel PS: please CC me on replies

All these things you're doing in userspace have a memory footprint in 
kernelspace as well, and that memory can't be swapped.  Page tables for your 
tmpfs mappings aren't free.  Kernel stacks and task_structs for your threads 
aren't free.

Also, there are many places in the kernel where a thread may not go to sleep to 
wait for memory to be freed.  The kernel has asynchronous tasks that try to keep 
memory free to avoid this problem, but if you're churning through your big 
monolithic binary, it's getting paged in as fast as the kernel can page it out.

That said, the modern VM is tuned with larger systems in mind, so you may be 
able to improve the situation by tweaking the vm.* sysctls, particularly 
vm.min_free_kbytes.  You can also change oom-killer settings for your process 
via the /proc/$PID/oom_* parameters.  It might help, or it might replace a 
recoverable userspace oom-kill with an unrecoverable kernel oom panic.

Either way, I'd be a little more conservative about code size on very small 
systems with no swap.

-- Chris

Re: Behaviour of the VM on a embedded linux

[Ingo Oeser]

Hi Marcel,

On Friday 22 August 2008, Wappler Marcel wrote:
> Alex, this is the case - I do observe normal operation of the VM subsytem - it moves memory pages dynamicaly throughout the system.
> But: when I create a large file on the tmpfs a kernel OOM occurs and kills the big monolithic application instead of stealing pages
> from the application. This is the fact I'm wondering about. In the past every guy told me that code size is no problem on systems
> using MMUs because the system can steal pages which contain code of the application in situations of low memory. But in my situation
> this is not the case.

Is the application linked statically? If not, code pages might be 
mapped privately and have been written to due to relocation.

Link everything statically to avoid this.


Best Regards

Ingo Oeser

Re: Behaviour of the VM on a embedded linux

[Arjan van de Ven]

On Mon, 25 Aug 2008 01:35:49 +0200
Ingo Oeser <ioe-lkml@rameria.de> wrote:

> Is the application linked statically? If not, code pages might be 
> mapped privately and have been written to due to relocation.
> 
> Link everything statically to avoid this.

... static linking has severe downsides though. You can also just use
prelink to avoid relocations in practice.

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

RE: Behaviour of the VM on a embedded linux

[Wappler Marcel]

Arjan van de Ven wrote:
> On Mon, 25 Aug 2008 01:35:49 +0200
> Ingo Oeser <ioe-lkml@rameria.de> wrote:
>
>> Is the application linked statically? If not, code pages might be
>> mapped privately and have been written to due to relocation.
>>
>> Link everything statically to avoid this.
>
> ... static linking has severe downsides though. You can also just use
> prelink to avoid relocations in practice.

Hm. I did a
cat /proc/`pidof thebigbinary`/smaps | grep Dirty
and got only about 3 MByte of dirty data. So it looks like this is not the (whole) problem (just a part). The Rss part of the /proc/pid/smaps report counts to about 14 MBytes. What is this saying exactly? I thought, those are the pages which aren't stealed by the VM from the app. Am I right?

Marcel