Re: [PROBLEM] hard-lock with kmemtrace, relayfs, and splice

[Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>]

On Sat, Oct 25, 2008 at 08:56:07AM +0800, Peter Teoh wrote:
> ok.   just ignore it.   i think based on "git log" of
> "git://repo.or.cz/linux-2.6/kmemtrace.git":
> 
> commit 0441e5ff6ab71cf7a3e9ec3116f14d0fd7d20d51
> Author: Eduard - Gabriel Munteanu <eduard.munteanu@linux360.ro>
> Date:   Thu Jul 10 20:20:05 2008 +0300
> 
>     kmemtrace: SLOB hooks.
> 
> the repo seemed way out of date.   But a clone of Pekka's slab-2.6's
> topic/kmemtrace branch have no problem.   The steps are as follows:
> 
> 1.   git clone git://git.kernel.org/pub/scm/linux/kernel/git/penberg/slab-2.6.git
> 2.   git checkout -m  origin/topic/kmemtrace
> 3.   git checkout -b kmemtrace

Hi,

Yes, that repo may be outdated, please use Pekka's for the latest
working stuff.

> git log gives:
> 
> commit 51b19be3535c8fbcce6b6f838d89b9a6a4cc5b92
> Author: Tom Zanussi <zanussi@comcast.net>
> Date:   Fri Oct 10 23:58:51 2008 -0500
> 
>     relayfs: fix infinite loop with splice()
> 
> But the userspace tool I still get it from: git://repo.or.cz/kmemtrace-user.git
> 
> Correct?

Yes, I maintain this one and it's up-to-date.

> Now I have some problems:
> 
> a.   I would like to extract out all the commit as diff - may I know
> how to do that? ("git log" only gives the descriptive part).

You can use "git-log -p" to get diffs along with commit history.

> b.   boot-time memory profiling....how can it be done (or extracted
> out)?  (kmemtrace-user does not have that?)

If you enable kmemtrace at boot-time using the "kmemtrace.enabled=yes"
parameter, kmemtrace will start logging just after kmem_cache_init()
finishes. There are plans for making tracing possible even earlier,
however it involves quite a big rework of how relay allocates its
buffers.

> c.   please provide some pointers to documentation:   how do I
> interpret the following:
> 
> Allocation #83468 (CPU0) already exists!
>         by __kmalloc_track_caller+25
>         last touched by __kmalloc_track_caller
> Allocation #83740 (CPU0) already exists!
>         by __kmalloc_track_caller+25
>         last touched by __kmalloc_track_caller

The userspace app writes out those messages when it looks like a memory
region has been allocated twice. In reality, it's most likely a free
hasn't been traced (because a misc free function isn't traced).

> For my purpose, I would like to trace how and where is memory
> allocated - just one one single userspace program which access the
> mmap("/dev/mem") to read the memory.   How can it be done?   "Tracing
> memory" in my context will mean list all the kernel-function +
> memory-allocated-within-the-function + its-PFN-or-PTE-information (to
> pinpoint the exact physical page).   My understanding is that these
> can be done, via checking the value of "current", as each userspace
> program started will have a unique "current" context value, and
> therefore within any kernel function, just check this value, to focus
> tracing on just one particular userspace program.   Correct?

I have made provisions for exporting the whole stack trace, however it
is not yet implemented.

> d.   The following:
> 
> /sys/kernel/debug/kmemtrace>ls -al
> ???? 0
> drwxr-xr-x 2 root root          0 2008-10-25 .
> drwxr-xr-x 9 root root          0 2008-10-25 ..
> -r-------- 1 root root          0 2008-10-25 abi_version
> -r-------- 1 root root 2946936192 2008-10-25 cpu0
> -r-------- 1 root root 1701263952 2008-10-25 cpu1
> -rw------- 1 root root          0 2008-10-25 enabled
> -r-------- 1 root root          0 2008-10-25 total_overruns
> 
> So the information is stored in memory, right?   Is it possible to
> reset it?   I don't want these information to clog up the memory.

Relay uses fixed size buffers, so there is no point in clearing them.

> e.   The balance between cpu0 and cpu1 does not seemed equal, any
> implications on the scheduler/ IO Scheduler?   or memory allocation
> scheduler (if there exists one :-))....sorry about that....my
> knowledge is limited.

Much of the booting sequence is done on a single CPU, then the kernel
switches to SMP. It is natural to see one CPU log more allocations.

> -- 
> Regards,
> Peter Teoh

Thanks for testing.


	Cheers,
	Eduard