[Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[M. Koehrer]

Hi everybody,

I have a strange issue frequently when reading /proc/xenomai/stat.
I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
This happens typcially after I have started and terminated by application that uses a number of tasks.
I am able to reproduce it deterministically...
While the application is running, I can see the stat values perfectly.
Here is the head output of all /proc/xenomai/* files:

==> affinity <==
00000002

==> apc <==
APC          CPU0         CPU1
  0:            0           0    (pipe_wakeup)
  1:           49          54    (lostage_handler)
  2:           12          23    (registry_export)

==> faults <==
TRAP         CPU0        CPU1
  0:            0           0    (Divide error)
  1:            0           0    (Debug)
  3:            0           0    (Int3)
  4:            0           0    (Overflow)
  5:            0           0    (Bounds)
  6:            0           0    (Invalid opcode)
  7:            0           0    (FPU not available)
  8:            0           0    (Double fault)
  9:            0           0    (FPU segment overrun)
 10:            0           0    (Invalid TSS)
 11:            0           0    (Segment not present)
 12:            0           0    (Stack segment)
 13:            0           0    (General protection)
 14:            0           0    (Page fault)
 15:            0           0    (Spurious interrupt)
 16:            0           0    (FPU error)
 17:            0           0    (Alignment check)
 18:            0           0    (Machine check)
 19:            0           0    (SIMD error)

==> hal <==
1.6-03

==> heap <==
size=130560:used=168:pagesz=512

==> interfaces <==
head: error reading `interfaces': Is a directory

==> irq <==
IRQ         CPU0        CPU1
213:           0          13
216:      387539      236720         [timer]
217:           0           0
226:         112         137         [virtual]

==> latency <==
2699

==> registry <==
head: error reading `registry': Is a directory

==> rtdm <==
head: error reading `rtdm': Is a directory

==> sched <==
CPU  PID    PRI      PERIOD   TIMEOUT    STAT       NAME
  0  0       -1      0        0          R          ROOT/0
  1  0       -1      0        0          R          ROOT/1
head: cannot open `stat' for reading: Cannot allocate memory

==> timer <==
status=oneshot:setup=120:tickval=1:jiffies=442215696848

==> version <==
2.3


I am using Xenomai 2.3.0 with some additional COW patches. Kernel 2.6.19.1 SMP.
Xenomai is directly integrated in the kernel,  not a module.

Thanks for any feedback on that

Mathias



-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> Hi everybody,
> 
> I have a strange issue frequently when reading /proc/xenomai/stat.
> I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
> This happens typcially after I have started and terminated by application that uses a number of tasks.
> I am able to reproduce it deterministically...
> While the application is running, I can see the stat values perfectly.
> Here is the head output of all /proc/xenomai/* files:

At first sight looking at the code, there seems to be a memory leak
caused by an unfree kmalloc() block in the sched & stat sequence
routines. Will confirm and fix. Thanks,

-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > Hi everybody,
> > 
> > I have a strange issue frequently when reading /proc/xenomai/stat.
> > I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
> > This happens typcially after I have started and terminated by application that uses a number of tasks.
> > I am able to reproduce it deterministically...
> > While the application is running, I can see the stat values perfectly.
> > Here is the head output of all /proc/xenomai/* files:
> 
> At first sight looking at the code, there seems to be a memory leak
> caused by an unfree kmalloc() block in the sched & stat sequence
> routines. Will confirm and fix. Thanks,
> 

Not confirmed for the leakage part regarding kmalloc(). We do have a
seq_release_private() freeing the allocated block, so back to square #1.
This said, some kernel memory is not released somewhere, as running
vmstat tells me here.

Btw, is the rest of your system functional after the error occurs, or is
a reboot needed afterwise?

-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Gilles Chanteperdrix]

Philippe Gerum wrote:
> On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> 
>>On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
>>
>>>Hi everybody,
>>>
>>>I have a strange issue frequently when reading /proc/xenomai/stat.
>>>I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
>>>This happens typcially after I have started and terminated by application that uses a number of tasks.
>>>I am able to reproduce it deterministically...
>>>While the application is running, I can see the stat values perfectly.
>>>Here is the head output of all /proc/xenomai/* files:
>>
>>At first sight looking at the code, there seems to be a memory leak
>>caused by an unfree kmalloc() block in the sched & stat sequence
>>routines. Will confirm and fix. Thanks,
>>
> 
> 
> Not confirmed for the leakage part regarding kmalloc(). We do have a
> seq_release_private() freeing the allocated block, so back to square #1.
> This said, some kernel memory is not released somewhere, as running
> vmstat tells me here.
> 
> Btw, is the rest of your system functional after the error occurs, or is
> a reboot needed afterwise?

Do you observe the same leakage without the nocow patch ?

-- 
                                                 Gilles Chanteperdrix

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 10:51 +0100, Gilles Chanteperdrix wrote:
> Philippe Gerum wrote:
> > On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > 
> >>On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> >>
> >>>Hi everybody,
> >>>
> >>>I have a strange issue frequently when reading /proc/xenomai/stat.
> >>>I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
> >>>This happens typcially after I have started and terminated by application that uses a number of tasks.
> >>>I am able to reproduce it deterministically...
> >>>While the application is running, I can see the stat values perfectly.
> >>>Here is the head output of all /proc/xenomai/* files:
> >>
> >>At first sight looking at the code, there seems to be a memory leak
> >>caused by an unfree kmalloc() block in the sched & stat sequence
> >>routines. Will confirm and fix. Thanks,
> >>
> > 
> > 
> > Not confirmed for the leakage part regarding kmalloc(). We do have a
> > seq_release_private() freeing the allocated block, so back to square #1.
> > This said, some kernel memory is not released somewhere, as running
> > vmstat tells me here.
> > 
> > Btw, is the rest of your system functional after the error occurs, or is
> > a reboot needed afterwise?
> 
> Do you observe the same leakage without the nocow patch ?
> 

Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
interesting things is that reading /proc/interrupts in loop seems to
silently eat kernel memory by small chunks too. I'm now trying on a
vanilla kernel.

-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 12:23 +0100, Philippe Gerum wrote:
> On Fri, 2007-02-02 at 10:51 +0100, Gilles Chanteperdrix wrote:
> > Philippe Gerum wrote:
> > > On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > > 
> > >>On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > >>
> > >>>Hi everybody,
> > >>>
> > >>>I have a strange issue frequently when reading /proc/xenomai/stat.
> > >>>I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory" error.
> > >>>This happens typcially after I have started and terminated by application that uses a number of tasks.
> > >>>I am able to reproduce it deterministically...
> > >>>While the application is running, I can see the stat values perfectly.
> > >>>Here is the head output of all /proc/xenomai/* files:
> > >>
> > >>At first sight looking at the code, there seems to be a memory leak
> > >>caused by an unfree kmalloc() block in the sched & stat sequence
> > >>routines. Will confirm and fix. Thanks,
> > >>
> > > 
> > > 
> > > Not confirmed for the leakage part regarding kmalloc(). We do have a
> > > seq_release_private() freeing the allocated block, so back to square #1.
> > > This said, some kernel memory is not released somewhere, as running
> > > vmstat tells me here.
> > > 
> > > Btw, is the rest of your system functional after the error occurs, or is
> > > a reboot needed afterwise?
> > 
> > Do you observe the same leakage without the nocow patch ?
> > 
> 
> Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
> interesting things is that reading /proc/interrupts in loop seems to
> silently eat kernel memory by small chunks too. I'm now trying on a
> vanilla kernel.
> 

Confirmed. A vanilla 2.6.19 silently eats memory when
reading /proc/interrupts in loop. Mathias, could you please confirm this
on your box too? TIA,

-- 
Philippe.

Re: Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate

[M. Koehrer]

Hi Philippe, 

I can. Which of the many values in /proc/vmstat is the relevant one to check?

Regards

Mathias

> On Fri, 2007-02-02 at 12:23 +0100, Philippe Gerum wrote:
> > On Fri, 2007-02-02 at 10:51 +0100, Gilles Chanteperdrix wrote:
> > > Philippe Gerum wrote:
> > > > On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > > > 
> > > >>On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > > >>
> > > >>>Hi everybody,
> > > >>>
> > > >>>I have a strange issue frequently when reading /proc/xenomai/stat.
> > > >>>I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory"
> error.
> > > >>>This happens typcially after I have started and terminated by
> application that uses a number of tasks.
> > > >>>I am able to reproduce it deterministically...
> > > >>>While the application is running, I can see the stat values
> perfectly.
> > > >>>Here is the head output of all /proc/xenomai/* files:
> > > >>
> > > >>At first sight looking at the code, there seems to be a memory leak
> > > >>caused by an unfree kmalloc() block in the sched & stat sequence
> > > >>routines. Will confirm and fix. Thanks,
> > > >>
> > > > 
> > > > 
> > > > Not confirmed for the leakage part regarding kmalloc(). We do have a
> > > > seq_release_private() freeing the allocated block, so back to square
> #1.
> > > > This said, some kernel memory is not released somewhere, as running
> > > > vmstat tells me here.
> > > > 
> > > > Btw, is the rest of your system functional after the error occurs, or
> is
> > > > a reboot needed afterwise?
> > > 
> > > Do you observe the same leakage without the nocow patch ?
> > > 
> > 
> > Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
> > interesting things is that reading /proc/interrupts in loop seems to
> > silently eat kernel memory by small chunks too. I'm now trying on a
> > vanilla kernel.
> > 
> 
> Confirmed. A vanilla 2.6.19 silently eats memory when
> reading /proc/interrupts in loop. Mathias, could you please confirm this
> on your box too? TIA,
> 
> -- 
> Philippe.
> 
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate

[Philippe Gerum]

On Fri, 2007-02-02 at 13:58 +0100, M. Koehrer wrote:
> Hi Philippe, 
> 
> I can. Which of the many values in /proc/vmstat is the relevant one to check?
> 

$ vmstat 1
or
$ watch -n 1 free

The value in the free space column should decrease slowly but constantly
while the read loop runs.

> Regards
> 
> Mathias
> 
> > On Fri, 2007-02-02 at 12:23 +0100, Philippe Gerum wrote:
> > > On Fri, 2007-02-02 at 10:51 +0100, Gilles Chanteperdrix wrote:
> > > > Philippe Gerum wrote:
> > > > > On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > > > > 
> > > > >>On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > > > >>
> > > > >>>Hi everybody,
> > > > >>>
> > > > >>>I have a strange issue frequently when reading /proc/xenomai/stat.
> > > > >>>I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory"
> > error.
> > > > >>>This happens typcially after I have started and terminated by
> > application that uses a number of tasks.
> > > > >>>I am able to reproduce it deterministically...
> > > > >>>While the application is running, I can see the stat values
> > perfectly.
> > > > >>>Here is the head output of all /proc/xenomai/* files:
> > > > >>
> > > > >>At first sight looking at the code, there seems to be a memory leak
> > > > >>caused by an unfree kmalloc() block in the sched & stat sequence
> > > > >>routines. Will confirm and fix. Thanks,
> > > > >>
> > > > > 
> > > > > 
> > > > > Not confirmed for the leakage part regarding kmalloc(). We do have a
> > > > > seq_release_private() freeing the allocated block, so back to square
> > #1.
> > > > > This said, some kernel memory is not released somewhere, as running
> > > > > vmstat tells me here.
> > > > > 
> > > > > Btw, is the rest of your system functional after the error occurs, or
> > is
> > > > > a reboot needed afterwise?
> > > > 
> > > > Do you observe the same leakage without the nocow patch ?
> > > > 
> > > 
> > > Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
> > > interesting things is that reading /proc/interrupts in loop seems to
> > > silently eat kernel memory by small chunks too. I'm now trying on a
> > > vanilla kernel.
> > > 
> > 
> > Confirmed. A vanilla 2.6.19 silently eats memory when
> > reading /proc/interrupts in loop. Mathias, could you please confirm this
> > on your box too? TIA,
> > 
> > -- 
> > Philippe.
> > 
> > 
> > 
> 
-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

:> Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
:> interesting things is that reading /proc/interrupts in loop seems to
:> silently eat kernel memory by small chunks too. I'm now trying on a
:> vanilla kernel.
:Confirmed. A vanilla 2.6.19 silently eats memory when
:reading /proc/interrupts in loop. Mathias, could you please confirm this
:on your box too? TIA,

 Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
doesn't.

:
:

		Rus

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 14:59 +0200, Rus V. Brushkoff wrote:
> :> Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
> :> interesting things is that reading /proc/interrupts in loop seems to
> :> silently eat kernel memory by small chunks too. I'm now trying on a
> :> vanilla kernel.
> :Confirmed. A vanilla 2.6.19 silently eats memory when
> :reading /proc/interrupts in loop. Mathias, could you please confirm this
> :on your box too? TIA,
> 
>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
> doesn't.

Thanks. I do have a leak on UP here though; vmstat or any other command
do not eat this memory (at least neither ps or top report any continuous
growth of their vm space while running the test), so I would wildly
guess that's a kernel-based leakage occurring somewhere in the procfs
support.

> 
> :
> :
> 
> 		Rus
> 
> _______________________________________________
> Xenomai-help mailing list
> Xenomai-help@domain.hid
> https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

On Fri, 2 Feb 2007, Philippe Gerum wrote:

:>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
:> doesn't.
:
:Thanks. I do have a leak on UP here though; vmstat or any other command
:do not eat this memory (at least neither ps or top report any continuous
:growth of their vm space while running the test), so I would wildly
:guess that's a kernel-based leakage occurring somewhere in the procfs
:support.

 May be the time to announce this to linux-kernel, last rc7 for 2.6.20 
is out ;)

:
:

		Rus

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

On Fri, 2 Feb 2007, Rus V. Brushkoff wrote:

::>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
::> doesn't.
::Thanks. I do have a leak on UP here though; vmstat or any other command
::do not eat this memory (at least neither ps or top report any continuous
::growth of their vm space while running the test), so I would wildly
::guess that's a kernel-based leakage occurring somewhere in the procfs
::support.
: May be the time to announce this to linux-kernel, last rc7 for 2.6.20 
:is out ;)

Does anybody seeing memleak has nvidia module loaded ?

:
:

		Rus

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Sat, 2007-02-03 at 18:08 +0200, Rus V. Brushkoff wrote:
> On Fri, 2 Feb 2007, Rus V. Brushkoff wrote:
> 
> ::>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
> ::> doesn't.
> ::Thanks. I do have a leak on UP here though; vmstat or any other command
> ::do not eat this memory (at least neither ps or top report any continuous
> ::growth of their vm space while running the test), so I would wildly
> ::guess that's a kernel-based leakage occurring somewhere in the procfs
> ::support.
> : May be the time to announce this to linux-kernel, last rc7 for 2.6.20 
> :is out ;)
> 
> Does anybody seeing memleak has nvidia module loaded ?
> 

Not here.

> :
> :
> 
> 		Rus
-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Dmitry Adamushko]

Hi,

are there any other memory pools increasing while "MemFree" is decreasing?

cat /proc/meminfo

If it's kmalloc() - related, I'd expect "Slab" to be increasing..  [1]

Moreover, if it's left to be running for quite some time, does it
actually causes an "out-of-memory" problem in the end?  [2]

I have run "while true ; do cat /proc/interrupts; done" for some time
and "MemFree" is going down but what's going up is "Buffers" +
"Cached" (mainly, the first one) and not
"Slab".

cat /proc/interrupts returns file content which is different every
time and might be cached (likely - "Buffers").. Actually, do you
observe similar behaviour when reading any other /proc file (or even
any regular) ?

After all, Linux memory manager is quite dynamic in its behaviour, and
"MemFree" does not really answer a question "how much memory is free"
as there are some _reclaimable_ sources of memory ("buffers" and
"cached")..

So what's about [1]? Of course, if you can encounter [2] after some time..

-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Sat, 2007-02-03 at 19:20 +0100, Dmitry Adamushko wrote:
> Hi,
> 
> are there any other memory pools increasing while "MemFree" is decreasing?
> 
> cat /proc/meminfo
> 
> If it's kmalloc() - related, I'd expect "Slab" to be increasing..  [1]
> 
> Moreover, if it's left to be running for quite some time, does it
> actually causes an "out-of-memory" problem in the end?  [2]
> 
> I have run "while true ; do cat /proc/interrupts; done" for some time
> and "MemFree" is going down but what's going up is "Buffers" +
> "Cached" (mainly, the first one) and not
> "Slab".
> 
> cat /proc/interrupts returns file content which is different every
> time and might be cached (likely - "Buffers").. Actually, do you
> observe similar behaviour when reading any other /proc file (or even
> any regular) ?
> 
> After all, Linux memory manager is quite dynamic in its behaviour, and
> "MemFree" does not really answer a question "how much memory is free"
> as there are some _reclaimable_ sources of memory ("buffers" and
> "cached")..
> 
> So what's about [1]? Of course, if you can encounter [2] after some time..
> 

Really, what I see here very much looks like a constant leakage on an
otherwise idle system, not some artefact from the mm; albeit the cached
mem seems stable, the free memory goes down irremediably. Not all /proc
entries seem to cause the issue, and the interesting thing is
that /proc/interrupts and /proc/xenomai/sched|stat all use the seqfile
support to output their data.

Anyway, I did not dig the issue enough code-wise to remove all the
conditionals from my wording, but there seems to be something wrong
somewhere, and that's not Xenomai related. Or, the Linux VM is seasoned
enough to try to make people think it loses memory albeit it doesn't.

-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Dmitry Adamushko]

>
> Really, what I see here very much looks like a constant leakage on an
> otherwise idle system, not some artefact from the mm; albeit the cached
> mem seems stable, the free memory goes down irremediably.

"Slab" counter of /proc/meminfo?

As I can see, seqfile makes use of kmalloc()-ed memory so it should be
accounted by "Slab" if we think of it as a source of the leak.

I suppose, if "MemFree" goes down and non of the other counters (Slab,
Cached, Buffers, Vmalloced,...) get proportionally up, it might mean
something gets memory directly from the buddy allocator - e.g. by
using get_free_pages() for the purpose that's (again) not covered by
any counters..


-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Dmitry Adamushko]

ok, could somebody post 2 snapshots of /proc/meminfo taken :

[1] a few seconds after starting the test;

[2] quite some time later when the memory leak is already noticeable
(but the test is still running)


TIA,

-- 
Best regards,
Dmitry Adamushko

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 287 bytes --]

On Sun, 4 Feb 2007, Dmitry Adamushko wrote:

:ok, could somebody post 2 snapshots of /proc/meminfo taken :
:
:[1] a few seconds after starting the test;
:
:[2] quite some time later when the memory leak is already noticeable
:(but the test is still running)

4 logs attached.

:
:

		Rus

[-- Attachment #2: Type: TEXT/PLAIN, Size: 753 bytes --]

MemTotal:      2057464 kB
MemFree:        334056 kB
Buffers:        106452 kB
Cached:        1241376 kB
SwapCached:          0 kB
Active:        1203696 kB
Inactive:       267024 kB
SwapTotal:     7823612 kB
SwapFree:      7823612 kB
Dirty:             204 kB
Writeback:           0 kB
AnonPages:      122896 kB
Mapped:          39340 kB
Slab:           221060 kB
SReclaimable:   199176 kB
SUnreclaim:      21884 kB
PageTables:       3856 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8852344 kB
Committed_AS:   282672 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    305136 kB
VmallocChunk: 34359432695 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[-- Attachment #3: Type: TEXT/PLAIN, Size: 753 bytes --]

MemTotal:      2057464 kB
MemFree:        342488 kB
Buffers:        105376 kB
Cached:        1240344 kB
SwapCached:          0 kB
Active:        1197592 kB
Inactive:       265448 kB
SwapTotal:     7823612 kB
SwapFree:      7823612 kB
Dirty:             124 kB
Writeback:           0 kB
AnonPages:      117360 kB
Mapped:          38932 kB
Slab:           220572 kB
SReclaimable:   199096 kB
SUnreclaim:      21476 kB
PageTables:       3788 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8852344 kB
Committed_AS:   274604 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    305136 kB
VmallocChunk: 34359432695 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[-- Attachment #4: Type: TEXT/PLAIN, Size: 753 bytes --]

MemTotal:      2057464 kB
MemFree:        343076 kB
Buffers:        104912 kB
Cached:        1240228 kB
SwapCached:          0 kB
Active:        1197136 kB
Inactive:       265180 kB
SwapTotal:     7823612 kB
SwapFree:      7823612 kB
Dirty:              80 kB
Writeback:           0 kB
AnonPages:      117228 kB
Mapped:          38908 kB
Slab:           220384 kB
SReclaimable:   199028 kB
SUnreclaim:      21356 kB
PageTables:       3732 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8852344 kB
Committed_AS:   274316 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    305136 kB
VmallocChunk: 34359432695 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

[-- Attachment #5: Type: TEXT/PLAIN, Size: 753 bytes --]

MemTotal:      2057464 kB
MemFree:        343120 kB
Buffers:        104772 kB
Cached:        1240224 kB
SwapCached:          0 kB
Active:        1197148 kB
Inactive:       265052 kB
SwapTotal:     7823612 kB
SwapFree:      7823612 kB
Dirty:              68 kB
Writeback:           0 kB
AnonPages:      117140 kB
Mapped:          38908 kB
Slab:           220488 kB
SReclaimable:   199032 kB
SUnreclaim:      21456 kB
PageTables:       3772 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   8852344 kB
Committed_AS:   282604 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    305136 kB
VmallocChunk: 34359432695 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Paul]

On Sunday 04 February 2007 14:01, Dmitry Adamushko wrote:
> ok, could somebody post 2 snapshots of /proc/meminfo taken :
>
> [1] a few seconds after starting the test;
>
> [2] quite some time later when the memory leak is already noticeable
> (but the test is still running)

Sixteen minutes of logs at 
http://zathras.tuxcnc.org/vmstats/procmeminfo.log.bz2 - Was running a cat 
of /proc/ipipe/Linux on an x86_64 patched 2.6.19.2. Looks like there may well 
be a problem there..

 Shortly after terminating the test, I experienced a serious OOM kill when 
starting up a memory testing utility - Don't quite know what happened, but 
ended up having to reboot and then running fcsk on a corrupted root 
partition.


Regards, Paul.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

On Sat, 3 Feb 2007, Philippe Gerum wrote:

:> Does anybody seeing memleak has nvidia module loaded ?
:Not here.

 Ok, the last thing I've can see - seems like leak occured at the 
boxes with some uptime - where any interrupt counter is big ehough (nearly 
9 decimal digits). Does anybody can confirm/disprove this suggestion ?

:

		Rus

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Sun, 2007-02-04 at 09:16 +0200, Rus V. Brushkoff wrote:
> On Sat, 3 Feb 2007, Philippe Gerum wrote:
> 
> :> Does anybody seeing memleak has nvidia module loaded ?
> :Not here.
> 
>  Ok, the last thing I've can see - seems like leak occured at the 
> boxes with some uptime - where any interrupt counter is big ehough (nearly 
> 9 decimal digits). Does anybody can confirm/disprove this suggestion ?
> 

I can reproduce the issue systematically even on a freshly booted box
reading /proc/interrupts in loop.

> :
> 
> 		Rus
-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Paul]

[-- Attachment #1: Type: text/plain, Size: 478 bytes --]

On Saturday 03 February 2007 16:08, Rus V. Brushkoff wrote:
> Does anybody seeing memleak has nvidia module loaded ?

No proprietary modules running..

 Attached is a plot of free memory whilst reading /proc/interrupts in a 
continuous loop on an otherwise idle machine.

Computer spec:
 Pentium 166, 64Meg memory.
 2.6.19.2 kernel (with the RTAI patches.)

Plot time is a little over 104 minutes... Also attached is the config of the 
kernel for comparison.


Regards, Paul.



[-- Attachment #2: config.gz --]
[-- Type: application/x-gzip, Size: 18479 bytes --]

[-- Attachment #3: vmstat.png --]
[-- Type: image/png, Size: 9182 bytes --]

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Jan Kiszka]

[-- Attachment #1.1: Type: text/plain, Size: 1029 bytes --]

Philippe Gerum wrote:
> On Fri, 2007-02-02 at 14:59 +0200, Rus V. Brushkoff wrote:
>> :> Yes, 2.6.19-1.6-04 (without nowcow) has the same problem. But the
>> :> interesting things is that reading /proc/interrupts in loop seems to
>> :> silently eat kernel memory by small chunks too. I'm now trying on a
>> :> vanilla kernel.
>> :Confirmed. A vanilla 2.6.19 silently eats memory when
>> :reading /proc/interrupts in loop. Mathias, could you please confirm this
>> :on your box too? TIA,
>>
>>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP 
>> doesn't.
> 
> Thanks. I do have a leak on UP here though; vmstat or any other command
> do not eat this memory (at least neither ps or top report any continuous
> growth of their vm space while running the test), so I would wildly
> guess that's a kernel-based leakage occurring somewhere in the procfs
> support.

Must be .config-dependent. My UP box (.config attached) doesn't expose
any leak on /proc/interrupt loops with 2.6.19.2.

Jan

[-- Attachment #1.2: config.bz2 --]
[-- Type: application/octet-stream, Size: 7462 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Paul]

On Friday 02 February 2007 14:42, Jan Kiszka wrote:
> >>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP
> >> doesn't.

> Must be .config-dependent. My UP box (.config attached) doesn't expose
> any leak on /proc/interrupt loops with 2.6.19.2.

x86_64 SMP on 2.6.19.2 doesn't appear to leak here either. Perhap 
http://lkml.org/lkml/2006/12/26/112 might have been a contributary fix..


Regards, Paul.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Rus V. Brushkoff]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 535 bytes --]

:On Friday 02 February 2007 14:42, Jan Kiszka wrote:
:> >>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP
:> >> doesn't.
:> Must be .config-dependent. My UP box (.config attached) doesn't expose
:> any leak on /proc/interrupt loops with 2.6.19.2.
:
:x86_64 SMP on 2.6.19.2 doesn't appear to leak here either. Perhap 
:http://lkml.org/lkml/2006/12/26/112 might have been a contributary fix..

 With my config (attached) plain 2.6.19.2 x86_64 SMP leaks memory, so this 
is config related.

:
:

		Rus

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 14585 bytes --]

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 17:07 +0200, Rus V. Brushkoff wrote:
> :On Friday 02 February 2007 14:42, Jan Kiszka wrote:
> :> >>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP
> :> >> doesn't.
> :> Must be .config-dependent. My UP box (.config attached) doesn't expose
> :> any leak on /proc/interrupt loops with 2.6.19.2.
> :
> :x86_64 SMP on 2.6.19.2 doesn't appear to leak here either. Perhap 
> :http://lkml.org/lkml/2006/12/26/112 might have been a contributary fix..
> 
>  With my config (attached) plain 2.6.19.2 x86_64 SMP leaks memory, so this 
> is config related.
> 

I don't know for Paul, but unlike Jan, we both run with
CONFIG_DEBUG_KERNEL features on. I'm going to check if this makes a
difference on my setup.

> :
> :
> 
> 		Rus
> _______________________________________________ Xenomai-help mailing list Xenomai-help@domain.hid https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate memory

[Philippe Gerum]

On Fri, 2007-02-02 at 16:16 +0100, Philippe Gerum wrote:
> On Fri, 2007-02-02 at 17:07 +0200, Rus V. Brushkoff wrote:
> > :On Friday 02 February 2007 14:42, Jan Kiszka wrote:
> > :> >>  Tested on K8 x86_64 SMP and K8 x86 UP : SMP system leaks memory, UP
> > :> >> doesn't.
> > :> Must be .config-dependent. My UP box (.config attached) doesn't expose
> > :> any leak on /proc/interrupt loops with 2.6.19.2.
> > :
> > :x86_64 SMP on 2.6.19.2 doesn't appear to leak here either. Perhap 
> > :http://lkml.org/lkml/2006/12/26/112 might have been a contributary fix..
> > 
> >  With my config (attached) plain 2.6.19.2 x86_64 SMP leaks memory, so this 
> > is config related.
> > 
> 
> I don't know for Paul, but unlike Jan, we both run with
> CONFIG_DEBUG_KERNEL features on. I'm going to check if this makes a
> difference on my setup.
> 

Nope, it does not.

> > :
> > :
> > 
> > 		Rus
> > _______________________________________________ Xenomai-help mailing list Xenomai-help@domain.hid https://mail.gna.org/listinfo/xenomai-help
-- 
Philippe.

Re: Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot allocate

[M. Koehrer]

Hi!

I have one kernel module for a special driver running.
If I do not load this kernel module, everything seems to work fine.
That means, this kernel module is probably causing the issue...
I have to look closer to it...
Sorry for any confusion!

Regards

Mathias 


> On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > > Hi everybody,
> > > 
> > > I have a strange issue frequently when reading /proc/xenomai/stat.
> > > I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory"
> error.
> > > This happens typcially after I have started and terminated by
> application that uses a number of tasks.
> > > I am able to reproduce it deterministically...
> > > While the application is running, I can see the stat values perfectly.
> > > Here is the head output of all /proc/xenomai/* files:
> > 
> > At first sight looking at the code, there seems to be a memory leak
> > caused by an unfree kmalloc() block in the sched & stat sequence
> > routines. Will confirm and fix. Thanks,
> > 
> 
> Not confirmed for the leakage part regarding kmalloc(). We do have a
> seq_release_private() freeing the allocated block, so back to square #1.
> This said, some kernel memory is not released somewhere, as running
> vmstat tells me here.
> 
> Btw, is the rest of your system functional after the error occurs, or is
> a reboot needed afterwise?
> 
> -- 
> Philippe.
> 
> 
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2

Re: Re: Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot

[M. Koehrer]

Hi all,

I have now identified the issue and provided a workaround.
I am using a kernel module as a hardware driver.
The kernel module was originally designed for standard linux and I modified it only slightly to 
have Xenomai support.
For configuring the driver, a /dev/xxx character file is provided.

When this file is opened (which lead to the module's open that is registered via register_chrdev)
an interrupt routine is registered.
I do here a rt_intr_create() followed by a rt_intr_enable().
When the /dev/xxx file is closed, the module's release function is called.
This function calls rt_intr_disable() and rt_intr_delete() to stop using the interrupt.

This looks very fine, however it seems to cause problems.
Having this situation, I get the kernel "cannot allocate memory" message.

I did a restructuring of the kernel module to disable the interrupt routine and to
delete the interrupt object not with the release() function but within
cleanup_module(). And this works now perfect.

I do not understand the reason for this but now it is working.

Thanks for all the hints on that issue.

Regards


Mathias

----- Original Nachricht ----
Von:     "M. Koehrer" <mathias_koehrer@domain.hid>
An:      rpm@xenomai.org
Datum:   02.02.2007 11:16
Betreff: Re: Re: [Xenomai-help] Error: cat: /proc/xenomai/stat: Cannot
 allocate

> Hi!
> 
> I have one kernel module for a special driver running.
> If I do not load this kernel module, everything seems to work fine.
> That means, this kernel module is probably causing the issue...
> I have to look closer to it...
> Sorry for any confusion!
> 
> Regards
> 
> Mathias 
> 
> 
> > On Fri, 2007-02-02 at 10:33 +0100, Philippe Gerum wrote:
> > > On Fri, 2007-02-02 at 09:43 +0100, M. Koehrer wrote:
> > > > Hi everybody,
> > > > 
> > > > I have a strange issue frequently when reading /proc/xenomai/stat.
> > > > I do a cat /proc/xenomai/stat and I get an "Cannot allocate memory"
> > error.
> > > > This happens typcially after I have started and terminated by
> > application that uses a number of tasks.
> > > > I am able to reproduce it deterministically...
> > > > While the application is running, I can see the stat values
> perfectly.
> > > > Here is the head output of all /proc/xenomai/* files:
> > > 
> > > At first sight looking at the code, there seems to be a memory leak
> > > caused by an unfree kmalloc() block in the sched & stat sequence
> > > routines. Will confirm and fix. Thanks,
> > > 
> > 
> > Not confirmed for the leakage part regarding kmalloc(). We do have a
> > seq_release_private() freeing the allocated block, so back to square #1.
> > This said, some kernel memory is not released somewhere, as running
> > vmstat tells me here.
> > 
> > Btw, is the rest of your system functional after the error occurs, or is
> > a reboot needed afterwise?
> > 
> > -- 
> > Philippe.
> > 
> > 
> > 
> 
> -- 
> Mathias Koehrer
> mathias_koehrer@domain.hid
> 
> 
> Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
> ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
> und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
> nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
> http://www.arcor.de/rd/emf-dsl-2
> 

-- 
Mathias Koehrer
mathias_koehrer@domain.hid


Viel oder wenig? Schnell oder langsam? Unbegrenzt surfen + telefonieren
ohne Zeit- und Volumenbegrenzung? DAS TOP ANGEBOT JETZT bei Arcor: günstig
und schnell mit DSL - das All-Inclusive-Paket für clevere Doppel-Sparer,
nur  44,85 €  inkl. DSL- und ISDN-Grundgebühr!
http://www.arcor.de/rd/emf-dsl-2