[Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

Hi guys,
Does sendto() going into an XDDP socket involve any memory allocation
that's not automatically getting de-allocated each time it's called?

I have an application running where I am writing a few different types of
data packets to an XDDP socket on a fixed periodic basis, and in a separate
console using "free" I see the memory usage increasing steadily until I
only have about 5MB left, at which point it doesn't grow/shrink anymore. I
don't seem to be losing any data after this happens, though.

I tried temporarily disabling my largest packet type, and saw that the rate
of memory consumption decreases very significantly, which is why I think it
has something to do with this.

I may be able to live with this if it's not causing any real problems but
it is a little unsettling...

Thanks!

Steve

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Philippe Gerum]

On 05/13/2016 01:42 AM, Steve B wrote:
> Hi guys,
> Does sendto() going into an XDDP socket involve any memory allocation
> that's not automatically getting de-allocated each time it's called?
> 
> I have an application running where I am writing a few different types of
> data packets to an XDDP socket on a fixed periodic basis, and in a separate
> console using "free" I see the memory usage increasing steadily until I
> only have about 5MB left, at which point it doesn't grow/shrink anymore. I
> don't seem to be losing any data after this happens, though.
> 
> I tried temporarily disabling my largest packet type, and saw that the rate
> of memory consumption decreases very significantly, which is why I think it
> has something to do with this.
> 
> I may be able to live with this if it's not causing any real problems but
> it is a little unsettling...

The memory is freed upon read() on the other side. Do the datagrams
linger somehow on the (non-rt) read-side?

-- 
Philippe.

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

On Fri, May 13, 2016 at 6:50 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On 05/13/2016 01:42 AM, Steve B wrote:
> > Hi guys,
> > Does sendto() going into an XDDP socket involve any memory allocation
> > that's not automatically getting de-allocated each time it's called?
> >
> > I have an application running where I am writing a few different types of
> > data packets to an XDDP socket on a fixed periodic basis, and in a
> separate
> > console using "free" I see the memory usage increasing steadily until I
> > only have about 5MB left, at which point it doesn't grow/shrink anymore.
> I
> > don't seem to be losing any data after this happens, though.
> >
> > I tried temporarily disabling my largest packet type, and saw that the
> rate
> > of memory consumption decreases very significantly, which is why I think
> it
> > has something to do with this.
> >
> > I may be able to live with this if it's not causing any real problems but
> > it is a little unsettling...
>
> The memory is freed upon read() on the other side. Do the datagrams
> linger somehow on the (non-rt) read-side?
>
> --
> Philippe.
>

The non-rt side just reads the messages and writes them to binary files. I
have a slightly over 1k byte message coming across 100x per second, and log
files are saved off and new files are created every 60 seconds. The files
are pretty consistently coming out at 6.5 MB give or take a few percent (I
think the slop is mostly in the time keeping in the non-RT side). So I
think all of my data is there, though I wonder if whatever buffering there
is between the RT and non-RT side may be overflowing. I would think that I
should be seeing failures in my writes if the buffer is full?

That actually does bring me to the question, do I even need to be buffering
my on board log files to the non-RT side via XDDP at all? Could I just
write the files straight from my RT side?

Thanks,

Steve

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Philippe Gerum]

On 05/13/2016 07:53 PM, Steve B wrote:
> On Fri, May 13, 2016 at 6:50 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
>> On 05/13/2016 01:42 AM, Steve B wrote:
>>> Hi guys,
>>> Does sendto() going into an XDDP socket involve any memory allocation
>>> that's not automatically getting de-allocated each time it's called?
>>>
>>> I have an application running where I am writing a few different types of
>>> data packets to an XDDP socket on a fixed periodic basis, and in a
>> separate
>>> console using "free" I see the memory usage increasing steadily until I
>>> only have about 5MB left, at which point it doesn't grow/shrink anymore.
>> I
>>> don't seem to be losing any data after this happens, though.
>>>
>>> I tried temporarily disabling my largest packet type, and saw that the
>> rate
>>> of memory consumption decreases very significantly, which is why I think
>> it
>>> has something to do with this.
>>>
>>> I may be able to live with this if it's not causing any real problems but
>>> it is a little unsettling...
>>
>> The memory is freed upon read() on the other side. Do the datagrams
>> linger somehow on the (non-rt) read-side?
>>
>> --
>> Philippe.
>>
> 
> The non-rt side just reads the messages and writes them to binary files. I
> have a slightly over 1k byte message coming across 100x per second, and log
> files are saved off and new files are created every 60 seconds. The files
> are pretty consistently coming out at 6.5 MB give or take a few percent (I
> think the slop is mostly in the time keeping in the non-RT side). So I
> think all of my data is there, though I wonder if whatever buffering there
> is between the RT and non-RT side may be overflowing. I would think that I
> should be seeing failures in my writes if the buffer is full?
> 

Correct. If you don't get any error on the write side, then it's fine.

Giving your XDDP connection a private memory pool would allow you to cap
the maximum consumption, and check how much is actually consumed by
looking at /proc/xenomai/heaps/xddp-pool@<port-number>. An XDDP
connection consumes memory from the pool it has been given exclusively.

XDDP_POOLSZ is the setsockopt() option you need to set for that.

> That actually does bring me to the question, do I even need to be buffering
> my on board log files to the non-RT side via XDDP at all? Could I just
> write the files straight from my RT side?
> 

This would entail a switch to Linux mode, which is like demoting a
thread priority-wise, so you would lose RT guarantees doing so from the
RT side. XDDP is there to prevent this, keeping the RT side in primary
mode while sending out the data to the other end.

-- 
Philippe.

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

On Fri, May 13, 2016 at 11:08 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On 05/13/2016 07:53 PM, Steve B wrote:
> > On Fri, May 13, 2016 at 6:50 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> >
> >> On 05/13/2016 01:42 AM, Steve B wrote:
> >>> Hi guys,
> >>> Does sendto() going into an XDDP socket involve any memory allocation
> >>> that's not automatically getting de-allocated each time it's called?
> >>>
> >>> I have an application running where I am writing a few different types
> of
> >>> data packets to an XDDP socket on a fixed periodic basis, and in a
> >> separate
> >>> console using "free" I see the memory usage increasing steadily until I
> >>> only have about 5MB left, at which point it doesn't grow/shrink
> anymore.
> >> I
> >>> don't seem to be losing any data after this happens, though.
> >>>
> >>> I tried temporarily disabling my largest packet type, and saw that the
> >> rate
> >>> of memory consumption decreases very significantly, which is why I
> think
> >> it
> >>> has something to do with this.
> >>>
> >>> I may be able to live with this if it's not causing any real problems
> but
> >>> it is a little unsettling...
> >>
> >> The memory is freed upon read() on the other side. Do the datagrams
> >> linger somehow on the (non-rt) read-side?
> >>
> >> --
> >> Philippe.
> >>
> >
> > The non-rt side just reads the messages and writes them to binary files.
> I
> > have a slightly over 1k byte message coming across 100x per second, and
> log
> > files are saved off and new files are created every 60 seconds. The files
> > are pretty consistently coming out at 6.5 MB give or take a few percent
> (I
> > think the slop is mostly in the time keeping in the non-RT side). So I
> > think all of my data is there, though I wonder if whatever buffering
> there
> > is between the RT and non-RT side may be overflowing. I would think that
> I
> > should be seeing failures in my writes if the buffer is full?
> >
>
> Correct. If you don't get any error on the write side, then it's fine.
>
> Giving your XDDP connection a private memory pool would allow you to cap
> the maximum consumption, and check how much is actually consumed by
> looking at /proc/xenomai/heaps/xddp-pool@<port-number>. An XDDP
> connection consumes memory from the pool it has been given exclusively.
>
> XDDP_POOLSZ is the setsockopt() option you need to set for that.
>
> > That actually does bring me to the question, do I even need to be
> buffering
> > my on board log files to the non-RT side via XDDP at all? Could I just
> > write the files straight from my RT side?
> >
>
> This would entail a switch to Linux mode, which is like demoting a
> thread priority-wise, so you would lose RT guarantees doing so from the
> RT side. XDDP is there to prevent this, keeping the RT side in primary
> mode while sending out the data to the other end.
>
> --
> Philippe.
>

Thanks! I just didn't know for sure if the write() function may have been
wrapped to work in an RT-safe fashion. I originally designed in the scheme
using XDDP for that reason though so glad to know I had the right thought.

I already did set XDDP_POOLSZ to something like 1MB so it's curious that it
would be consuming more than that... I will take a look at
/proc/xenomai/heaps though like you said, to possibly rule out the XDDP as
the offending consumer of memory. I was unaware of that feature!

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

On Fri, May 13, 2016 at 11:19 AM, Steve B <sbattazzo@gmail.com> wrote:

>
>
> On Fri, May 13, 2016 at 11:08 AM, Philippe Gerum <rpm@xenomai.org> wrote:
>
>> On 05/13/2016 07:53 PM, Steve B wrote:
>> > On Fri, May 13, 2016 at 6:50 AM, Philippe Gerum <rpm@xenomai.org>
>> wrote:
>> >
>> >> On 05/13/2016 01:42 AM, Steve B wrote:
>> >>> Hi guys,
>> >>> Does sendto() going into an XDDP socket involve any memory allocation
>> >>> that's not automatically getting de-allocated each time it's called?
>> >>>
>> >>> I have an application running where I am writing a few different
>> types of
>> >>> data packets to an XDDP socket on a fixed periodic basis, and in a
>> >> separate
>> >>> console using "free" I see the memory usage increasing steadily until
>> I
>> >>> only have about 5MB left, at which point it doesn't grow/shrink
>> anymore.
>> >> I
>> >>> don't seem to be losing any data after this happens, though.
>> >>>
>> >>> I tried temporarily disabling my largest packet type, and saw that the
>> >> rate
>> >>> of memory consumption decreases very significantly, which is why I
>> think
>> >> it
>> >>> has something to do with this.
>> >>>
>> >>> I may be able to live with this if it's not causing any real problems
>> but
>> >>> it is a little unsettling...
>> >>
>> >> The memory is freed upon read() on the other side. Do the datagrams
>> >> linger somehow on the (non-rt) read-side?
>> >>
>> >> --
>> >> Philippe.
>> >>
>> >
>> > The non-rt side just reads the messages and writes them to binary
>> files. I
>> > have a slightly over 1k byte message coming across 100x per second, and
>> log
>> > files are saved off and new files are created every 60 seconds. The
>> files
>> > are pretty consistently coming out at 6.5 MB give or take a few percent
>> (I
>> > think the slop is mostly in the time keeping in the non-RT side). So I
>> > think all of my data is there, though I wonder if whatever buffering
>> there
>> > is between the RT and non-RT side may be overflowing. I would think
>> that I
>> > should be seeing failures in my writes if the buffer is full?
>> >
>>
>> Correct. If you don't get any error on the write side, then it's fine.
>>
>> Giving your XDDP connection a private memory pool would allow you to cap
>> the maximum consumption, and check how much is actually consumed by
>> looking at /proc/xenomai/heaps/xddp-pool@<port-number>. An XDDP
>> connection consumes memory from the pool it has been given exclusively.
>>
>> XDDP_POOLSZ is the setsockopt() option you need to set for that.
>>
>> > That actually does bring me to the question, do I even need to be
>> buffering
>> > my on board log files to the non-RT side via XDDP at all? Could I just
>> > write the files straight from my RT side?
>> >
>>
>> This would entail a switch to Linux mode, which is like demoting a
>> thread priority-wise, so you would lose RT guarantees doing so from the
>> RT side. XDDP is there to prevent this, keeping the RT side in primary
>> mode while sending out the data to the other end.
>>
>> --
>> Philippe.
>>
>
> Thanks! I just didn't know for sure if the write() function may have been
> wrapped to work in an RT-safe fashion. I originally designed in the scheme
> using XDDP for that reason though so glad to know I had the right thought.
>
> I already did set XDDP_POOLSZ to something like 1MB so it's curious that
> it would be consuming more than that... I will take a look at
> /proc/xenomai/heaps though like you said, to possibly rule out the XDDP as
> the offending consumer of memory. I was unaware of that feature!
>
>
>
OK, /proc/xenomai/heap shows a constant 1000448 for my xddp: 0 which is
just over what I allocated using POOLSZ.
If I look at "free" or "top" in the OS, I see available memory shrinking by
about 100kB per second, but top does not attribute the memory usage to my
process, which stays at a constant 0.6% memory usage. Does the memory usage
on the RT side not get accounted for by the regular OS? I suppose the other
option is that the memory usage is somewhere in the kernel (like the file
I/O?).

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Philippe Gerum]

On 05/13/2016 08:27 PM, Steve B wrote:
> 
> 
> On Fri, May 13, 2016 at 11:19 AM, Steve B <sbattazzo@gmail.com
> <mailto:sbattazzo@gmail.com>> wrote:
> 
> 
> 
>     On Fri, May 13, 2016 at 11:08 AM, Philippe Gerum <rpm@xenomai.org
>     <mailto:rpm@xenomai.org>> wrote:
> 
>         On 05/13/2016 07:53 PM, Steve B wrote:
>         > On Fri, May 13, 2016 at 6:50 AM, Philippe Gerum
>         <rpm@xenomai.org <mailto:rpm@xenomai.org>> wrote:
>         >
>         >> On 05/13/2016 01:42 AM, Steve B wrote:
>         >>> Hi guys,
>         >>> Does sendto() going into an XDDP socket involve any memory
>         allocation
>         >>> that's not automatically getting de-allocated each time it's
>         called?
>         >>>
>         >>> I have an application running where I am writing a few
>         different types of
>         >>> data packets to an XDDP socket on a fixed periodic basis,
>         and in a
>         >> separate
>         >>> console using "free" I see the memory usage increasing
>         steadily until I
>         >>> only have about 5MB left, at which point it doesn't
>         grow/shrink anymore.
>         >> I
>         >>> don't seem to be losing any data after this happens, though.
>         >>>
>         >>> I tried temporarily disabling my largest packet type, and
>         saw that the
>         >> rate
>         >>> of memory consumption decreases very significantly, which is
>         why I think
>         >> it
>         >>> has something to do with this.
>         >>>
>         >>> I may be able to live with this if it's not causing any real
>         problems but
>         >>> it is a little unsettling...
>         >>
>         >> The memory is freed upon read() on the other side. Do the
>         datagrams
>         >> linger somehow on the (non-rt) read-side?
>         >>
>         >> --
>         >> Philippe.
>         >>
>         >
>         > The non-rt side just reads the messages and writes them to
>         binary files. I
>         > have a slightly over 1k byte message coming across 100x per
>         second, and log
>         > files are saved off and new files are created every 60
>         seconds. The files
>         > are pretty consistently coming out at 6.5 MB give or take a
>         few percent (I
>         > think the slop is mostly in the time keeping in the non-RT
>         side). So I
>         > think all of my data is there, though I wonder if whatever
>         buffering there
>         > is between the RT and non-RT side may be overflowing. I would
>         think that I
>         > should be seeing failures in my writes if the buffer is full?
>         >
> 
>         Correct. If you don't get any error on the write side, then it's
>         fine.
> 
>         Giving your XDDP connection a private memory pool would allow
>         you to cap
>         the maximum consumption, and check how much is actually consumed by
>         looking at /proc/xenomai/heaps/xddp-pool@<port-number>. An XDDP
>         connection consumes memory from the pool it has been given
>         exclusively.
> 
>         XDDP_POOLSZ is the setsockopt() option you need to set for that.
> 
>         > That actually does bring me to the question, do I even need to be buffering
>         > my on board log files to the non-RT side via XDDP at all? Could I just
>         > write the files straight from my RT side?
>         >
> 
>         This would entail a switch to Linux mode, which is like demoting a
>         thread priority-wise, so you would lose RT guarantees doing so
>         from the
>         RT side. XDDP is there to prevent this, keeping the RT side in
>         primary
>         mode while sending out the data to the other end.
> 
>         --
>         Philippe.
> 
> 
>     Thanks! I just didn't know for sure if the write() function may have
>     been wrapped to work in an RT-safe fashion. I originally designed in
>     the scheme using XDDP for that reason though so glad to know I had
>     the right thought.
> 
>     I already did set XDDP_POOLSZ to something like 1MB so it's curious
>     that it would be consuming more than that... I will take a look at
>     /proc/xenomai/heaps though like you said, to possibly rule out the
>     XDDP as the offending consumer of memory. I was unaware of that feature!
> 
> 
> 
> OK, /proc/xenomai/heap shows a constant 1000448 for my xddp: 0 which is
> just over what I allocated using POOLSZ.
> If I look at "free" or "top" in the OS, I see available memory shrinking
> by about 100kB per second, but top does not attribute the memory usage
> to my process, which stays at a constant 0.6% memory usage. Does the
> memory usage on the RT side not get accounted for by the regular OS? I
> suppose the other option is that the memory usage is somewhere in the
> kernel (like the file I/O?).
> 
> 

Maybe. You should have a look at the vmstat output for a while (e.g.
vmstat -a 1).

Cobalt gets all the core memory it needs from __vmalloc() since 3.0.2
(was __get_free_pages() in earlier 3.x releases), then stacks its own
allocation scheme on top of this (cobalt/kernel/heap.c). So the memory
usage on the RT side appears as one shot allocations only for getting
the initial memory underlying the real-time heaps.

-- 
Philippe.

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

On Fri, May 13, 2016 at 11:45 AM, Philippe Gerum <rpm@xenomai.org> wrote:

>
> Maybe. You should have a look at the vmstat output for a while (e.g.
> vmstat -a 1).
>
> Cobalt gets all the core memory it needs from __vmalloc() since 3.0.2
> (was __get_free_pages() in earlier 3.x releases), then stacks its own
> allocation scheme on top of this (cobalt/kernel/heap.c). So the memory
> usage on the RT side appears as one shot allocations only for getting
> the initial memory underlying the real-time heaps.
>
> --
> Philippe.
>

Thanks, this has been informative for me on a few more things to be looking
at during development cycle, if a little off topic for this mailing list.
I think I've actually found what's happening. I went ahead and wiped out
all of the log files even while the process was still running, and found
that the memory is freed up immediately! So the memory consumption is in
the files I'm saving, even though they're supposed to go to the on board
flash filesystem. I understand that they might be stored in RAM initially
before getting synced over to flash, but kind of interesting that the upper
bound on that is almost the entire RAM. There may be an OS setting
somewhere that sets a bound, or an occasional "sync" system call might help
free that memory up if I really need to for some reason.

Now that I've ruled out the XDDP and found what's actually happening, it
becomes more of a Linux question than a Xenomai thing but thanks for
helping me narrow it down!

Steve

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Gilles Chanteperdrix]

On Fri, May 13, 2016 at 12:42:04PM -0700, Steve B wrote:
> On Fri, May 13, 2016 at 11:45 AM, Philippe Gerum <rpm@xenomai.org> wrote:
> 
> >
> > Maybe. You should have a look at the vmstat output for a while (e.g.
> > vmstat -a 1).
> >
> > Cobalt gets all the core memory it needs from __vmalloc() since 3.0.2
> > (was __get_free_pages() in earlier 3.x releases), then stacks its own
> > allocation scheme on top of this (cobalt/kernel/heap.c). So the memory
> > usage on the RT side appears as one shot allocations only for getting
> > the initial memory underlying the real-time heaps.
> >
> > --
> > Philippe.
> >
> 
> Thanks, this has been informative for me on a few more things to be looking
> at during development cycle, if a little off topic for this mailing list.
> I think I've actually found what's happening. I went ahead and wiped out
> all of the log files even while the process was still running, and found
> that the memory is freed up immediately! So the memory consumption is in
> the files I'm saving, even though they're supposed to go to the on board
> flash filesystem. I understand that they might be stored in RAM initially
> before getting synced over to flash, but kind of interesting that the upper
> bound on that is almost the entire RAM. There may be an OS setting
> somewhere that sets a bound, or an occasional "sync" system call might help
> free that memory up if I really need to for some reason.
> 
> Now that I've ruled out the XDDP and found what's actually happening, it
> becomes more of a Linux question than a Xenomai thing but thanks for
> helping me narrow it down!

That is probably Linux filesystem cache: Linux uses all the non
allocated kernel memory to keep copies of what is saved on disk, in
case it would be needed again. If your files are "write and forget",
you may want to use posix_fadvise(POSIX_FADV_DONTNEED) to ask the
kernel to free the memory in the filesystem cache. Note that this
only works for pages that have been written to disk, so a normal
process is to trigger an fsync before posix_fadvise. Also note that
the fsync operation may suspend the caller for some time, so you may
want to do this in a thread separate from the one calling write. I
had to implement this once, and simply used the size of what had
been written as the way of communicating between the two threads,
and had a threshold to trigger the flush + forget operation say,
every 4MiB. You also have to pay attention to force the operation
before you close the file, otherwise the few data that were written
between the last flush and the file closure may still occupy RAM.

Finally, note that for logging, if Xenomai does not wrap write, it
wraps fwrite, so may avoid using the XDDP socket by using Xenomai
stdio wrapping. In that case, of course, you have to use an fflush
before fsync before posix_fadvise. Also, the default buffer sizes of
the stdio wrapping may not suit high volumes logging, and that this
system does not use any synchronization, so may loose some data. You
can know it if you check fwrite return value. You may call
rt_print_init to choose the buffer size for a particular thread, or
use the global variable __cobalt_print_bufsz to override the default
buffer size (which is 16 KiB).

-- 
					    Gilles.
https://click-hack.org

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Philippe Gerum]

On 05/13/2016 09:42 PM, Steve B wrote:
> 
> 
> On Fri, May 13, 2016 at 11:45 AM, Philippe Gerum <rpm@xenomai.org
> <mailto:rpm@xenomai.org>> wrote:
> 
> 
>     Maybe. You should have a look at the vmstat output for a while (e.g.
>     vmstat -a 1).
> 
>     Cobalt gets all the core memory it needs from __vmalloc() since 3.0.2
>     (was __get_free_pages() in earlier 3.x releases), then stacks its own
>     allocation scheme on top of this (cobalt/kernel/heap.c). So the memory
>     usage on the RT side appears as one shot allocations only for getting
>     the initial memory underlying the real-time heaps.
> 
>     --
>     Philippe.
> 
> 
> Thanks, this has been informative for me on a few more things to be
> looking at during development cycle, if a little off topic for this
> mailing list.
> I think I've actually found what's happening. I went ahead and wiped out
> all of the log files even while the process was still running, and found
> that the memory is freed up immediately! So the memory consumption is in
> the files I'm saving, even though they're supposed to go to the on board
> flash filesystem. I understand that they might be stored in RAM
> initially before getting synced over to flash, but kind of interesting
> that the upper bound on that is almost the entire RAM. There may be an
> OS setting somewhere that sets a bound, or an occasional "sync" system
> call might help free that memory up if I really need to for some reason.
> 
> Now that I've ruled out the XDDP and found what's actually happening, it
> becomes more of a Linux question than a Xenomai thing but thanks for
> helping me narrow it down!
> 

As Gilles pointed out, the fs cache is likely eating that memory. You
could try flushing it, before looking at the available free memory again:

# echo 1 > /proc/sys/vm/drop_caches

-- 
Philippe.

Re: [Xenomai] XDDP socket, any memory allocation when sendto() is called?

[Steve B]

On Sat, May 14, 2016 at 6:00 AM, Philippe Gerum <rpm@xenomai.org> wrote:

> On 05/13/2016 09:42 PM, Steve B wrote:
> >
> >
> > On Fri, May 13, 2016 at 11:45 AM, Philippe Gerum <rpm@xenomai.org
> > <mailto:rpm@xenomai.org>> wrote:
> >
> >
> >     Maybe. You should have a look at the vmstat output for a while (e.g.
> >     vmstat -a 1).
> >
> >     Cobalt gets all the core memory it needs from __vmalloc() since 3.0.2
> >     (was __get_free_pages() in earlier 3.x releases), then stacks its own
> >     allocation scheme on top of this (cobalt/kernel/heap.c). So the
> memory
> >     usage on the RT side appears as one shot allocations only for getting
> >     the initial memory underlying the real-time heaps.
> >
> >     --
> >     Philippe.
> >
> >
> > Thanks, this has been informative for me on a few more things to be
> > looking at during development cycle, if a little off topic for this
> > mailing list.
> > I think I've actually found what's happening. I went ahead and wiped out
> > all of the log files even while the process was still running, and found
> > that the memory is freed up immediately! So the memory consumption is in
> > the files I'm saving, even though they're supposed to go to the on board
> > flash filesystem. I understand that they might be stored in RAM
> > initially before getting synced over to flash, but kind of interesting
> > that the upper bound on that is almost the entire RAM. There may be an
> > OS setting somewhere that sets a bound, or an occasional "sync" system
> > call might help free that memory up if I really need to for some reason.
> >
> > Now that I've ruled out the XDDP and found what's actually happening, it
> > becomes more of a Linux question than a Xenomai thing but thanks for
> > helping me narrow it down!
> >
>
> As Gilles pointed out, the fs cache is likely eating that memory. You
> could try flushing it, before looking at the available free memory again:
>
> # echo 1 > /proc/sys/vm/drop_caches
>
> --
> Philippe.
>

Thanks guys! I came to the same conclusion late on Friday that it was the
file system cache when I noticed that the memory was all freed up if I
deleted the saved files. Kind of a silly thing to get worked up about, in
retrospect!

My application is not dynamically allocating anything and I don't need the
extra memory for anything else so I will just leave it to the OS to handle
all of that. I had considered keeping watch over trends of free memory in
my runtime diagnostics but I don't think it's telling me anything useful as
a result of this fact.. memory consumed by my application specifically
might be useful to watch over, but it is now stable so I probably won't
bother to add it in this time.

Sorry to bother you guys on this but it was an interesting little side
track for me anyway!

Steve