Re: [rfc] object collection tracing (was: [PATCH 5/5] proc: export more page flags in /proc/kpageflags)

[Frederic Weisbecker <fweisbec@gmail.com>]

On Tue, Apr 28, 2009 at 09:31:08PM +0800, Wu Fengguang wrote:
> On Tue, Apr 28, 2009 at 08:17:51PM +0800, Ingo Molnar wrote:
> > tent-Transfer-Encoding: quoted-printable
> > Status: RO
> > Content-Length: 5480
> > Lines: 161
> > 
> > 
> > * Wu Fengguang <fengguang.wu@intel.com> wrote:
> > 
> > > > The above 'get object state' interface (which allows passive 
> > > > sampling) - integrated into the tracing framework - would serve 
> > > > that goal, agreed?
> > > 
> > > Agreed. That could in theory a good complement to dynamic 
> > > tracings.
> > > 
> > > Then what will be the canonical form for all the 'get object 
> > > state' interfaces - "object.attr=value", or whatever? [...]
> > 
> > Lemme outline what i'm thinking of.
> > 
> > I'd call the feature "object collection tracing", which would live 
> > in /debug/tracing, accessed via such files:
> > 
> >   /debug/tracing/objects/mm/pages/
> >   /debug/tracing/objects/mm/pages/format
> >   /debug/tracing/objects/mm/pages/filter
> >   /debug/tracing/objects/mm/pages/trace_pipe
> >   /debug/tracing/objects/mm/pages/stats
> >   /debug/tracing/objects/mm/pages/events/
> > 
> > here's the (proposed) semantics of those files:
> > 
> > 1) /debug/tracing/objects/mm/pages/
> > 
> > There's a subsystem / object basic directory structure to make it 
> > easy and intuitive to find our way around there.
> > 
> > 2) /debug/tracing/objects/mm/pages/format
> > 
> > the format file:
> > 
> >   /debug/tracing/objects/mm/pages/format
> > 
> > Would reuse the existing dynamic-tracepoint structured-logging 
> > descriptor format and code (this is upstream already):
> > 
> >  [root@phoenix sched_signal_send]# pwd
> >  /debug/tracing/events/sched/sched_signal_send
> > 
> >  [root@phoenix sched_signal_send]# cat format 
> >  name: sched_signal_send
> >  ID: 24
> >  format:
> > 	field:unsigned short common_type;		offset:0;	size:2;
> > 	field:unsigned char common_flags;		offset:2;	size:1;
> > 	field:unsigned char common_preempt_count;	offset:3;	size:1;
> > 	field:int common_pid;				offset:4;	size:4;
> > 	field:int common_tgid;				offset:8;	size:4;
> > 
> > 	field:int sig;					offset:12;	size:4;
> > 	field:char comm[TASK_COMM_LEN];			offset:16;	size:16;
> > 	field:pid_t pid;				offset:32;	size:4;
> > 
> >  print fmt: "sig: %d  task %s:%d", REC->sig, REC->comm, REC->pid
> > 
> > These format descriptors enumerate fields, types and sizes, in a 
> > structured way that user-space tools can parse easily. (The binary 
> > records that come from the trace_pipe file follow this format 
> > description.)
> > 
> > 3) /debug/tracing/objects/mm/pages/filter
> > 
> > This is the tracing filter that can be set based on the 'format' 
> > descriptor. So with the above (signal-send tracepoint) you can 
> > define such filter expressions:
> > 
> >   echo "(sig == 10 && comm == bash) || sig == 13" > filter
> > 
> > To restrict the 'scope' of the object collection along pretty much 
> > any key or combination of keys. (Or you can leave it as it is and 
> > dump all objects and do keying in user-space.)
> > 
> > [ Using in-kernel filtering is obviously faster that streaming it 
> >   out to user-space - but there might be details and types of 
> >   visualization you want to do in user-space - so we dont want to 
> >   restrict things here. ]
> > 
> > For the mm object collection tracepoint i could imagine such filter 
> > expressions:
> > 
> >   echo "type == shared && file == /sbin/init" > filter
> > 
> > To dump all shared pages that are mapped to /sbin/init.
> > 
> > 4) /debug/tracing/objects/mm/pages/trace_pipe
> > 
> > The 'trace_pipe' file can be used to dump all objects in the 
> > collection, which match the filter ('all objects' by default). The 
> > record format is described in 'format'.
> > 
> > trace_pipe would be a reuse of the existing trace_pipe code: it is a 
> > modern, poll()-able, read()-able, splice()-able pipe abstraction.
> > 
> > 5) /debug/tracing/objects/mm/pages/stats
> > 
> > The 'stats' file would be a reuse of the existing histogram code of 
> > the tracing code. We already make use of it for the branch tracers 
> > and for the workqueue tracer - it could be extended to be applicable 
> > to object collections as well.
> > 
> > The advantage there would be that there's no dumping at all - all 
> > the integration is done straight in the kernel. ( The 'filter' 
> > condition is listened to - increasing flexibility. The filter file 
> > could perhaps also act as a default histogram key. )
> > 
> > 6) /debug/tracing/objects/mm/pages/events/
> > 
> > The 'events' directory offers links back to existing dynamic 
> > tracepoints that are under /debug/tracing/events/. This would serve 
> > as an additional coherent force that keeps dynamic tracepoints 
> > collected by subsystem and by object type as well. (Tools could make 
> > use of this information as well - without being aware of actual 
> > object semantics.)
> > 
> > 
> > There would be a number of other object collections we could 
> > enumerate:
> > 
> >  tasks:
> > 
> >   /debug/tracing/objects/sched/tasks/
> > 
> >  active inodes known to the kernel:
> > 
> >   /debug/tracing/objects/fs/inodes/
> > 
> >  interrupts:
> > 
> >   /debug/tracing/objects/hw/irqs/
> > 
> > etc.
> > 
> > These would use the same 'object collection' framework. Once done we 
> > can use it for many other thing too.
> > 
> > Note how organically integrated it all is with the tracing 
> > framework. You could start from an 'object view' to get an overview 
> > and then go towards a more dynamic view of specific object 
> > attributes (or specific objects), as you drill down on a specific 
> > problem you want to analyze.
> > 
> > How does this all sound to you?
> 
> Great! I saw much opportunity to adapt the not yet submitted
> /proc/filecache interface to the proposed framework.
> 
> Its basic form is:
> 
> #      ino       size   cached cached% refcnt state       age accessed  process         dev             file
> [snip]
>        320          1        4     100      1    D-     50443     1085 udevd           00:11(tmpfs)     /.udev/uevent_seqnum
>     460725        123      124     100     35    --     50444     6795 touch           08:02(sda2)      /lib/libpthread-2.9.so
>     460727         31       32     100     14    --     50444     2007 touch           08:02(sda2)      /lib/librt-2.9.so
>     458865         97       80      82      1    --     50444       49 mount           08:02(sda2)      /lib/libdevmapper.so.1.02.1
>     460090         15       16     100      1    --     50444       48 mount           08:02(sda2)      /lib/libuuid.so.1.2
>     458866         46       48     100      1    --     50444       47 mount           08:02(sda2)      /lib/libblkid.so.1.0
>     460732         43       44     100     69    --     50444     3581 rcS             08:02(sda2)      /lib/libnss_nis-2.9.so
>     460739         87       88     100     73    --     50444     3597 rcS             08:02(sda2)      /lib/libnsl-2.9.so
>     460726         31       32     100     69    --     50444     3581 rcS             08:02(sda2)      /lib/libnss_compat-2.9.so
>     458804        250      252     100     11    --     50445     8175 rcS             08:02(sda2)      /lib/libncurses.so.5.6
>     229540        780      752      96      3    --     50445     7594 init            08:02(sda2)      /bin/bash
>     460735         15       16     100     89    --     50445    17581 init            08:02(sda2)      /lib/libdl-2.9.so
>     460721       1344     1340      99    117    --     50445    48732 init            08:02(sda2)      /lib/libc-2.9.so
>     458801        107      104      97     24    --     50445     3586 init            08:02(sda2)      /lib/libselinux.so.1
>     671870         37       24      65      1    --     50446        1 swapper         08:02(sda2)      /sbin/init
>        175          1    24412     100      1    --     50446        0 swapper         00:01(rootfs)    /dev/root
> 
> The patch basically does a traversal through one or more of the inode
> lists to produce the output:
>         inode_in_use
>         inode_unused
>         sb->s_dirty
>         sb->s_io
>         sb->s_more_io
>         sb->s_inodes
> 
> The filtering feature is a necessity for this interface - or it will
> take considerable time to do a full listing. It supports the following
> filters:
>         { LS_OPT_DIRTY,         "dirty"         },
>         { LS_OPT_CLEAN,         "clean"         },
>         { LS_OPT_INUSE,         "inuse"         },
>         { LS_OPT_EMPTY,         "empty"         },
>         { LS_OPT_ALL,           "all"           },
>         { LS_OPT_DEV,           "dev=%s"        },
> 
> There are two possible challenges for the conversion:
> 
> - One trick it does is to select different lists to traverse on
>   different filter options. Will this be possible in the object
>   tracing framework?



Yeah, I guess.



> - The file name lookup(last field) is the performance killer. Is it
>   possible to skip the file name lookup when the filter failed on the
>   leading fields?


objects collection lays on trace events where filters basically ignore
a whole entry in case of non-matching. Not sure if we can easily only
ignore one field.

But I guess we can do something about the performances...

Could you send us the (sob'ed) patch you made which implements this.
I could try to adapt it to object collection.

Thanks,
Frederic.


> Will the object tracing interface allow such flexibilities?
> (Sorry I'm not yet familiar with the tracing framework.)
> 
> > Can you see any conceptual holes in the scheme, any use-case that 
> > /proc/kpageflags supports but the object collection approach does 
> > not?
> 
> kpageflags is simply a big (perhaps sparse) binary array.
> I'd still prefer to retain its current form - the kernel patches and
> user space tools are all ready made, and I see no benefits in
> converting to the tracing framework.
> 
> > Would you be interested in seeing something like this, if we tried 
> > to implement it in the tracing tree? The majority of the code 
> > already exists, we just need interest from the MM side and we have 
> > to hook it all up. (it is by no means trivial to do - but looks like
> > a very exciting feature.)
> 
> Definitely! /proc/filecache has another 'page view':
> 
>         # head /proc/filecache
>         # file /bin/bash
>         # flags R:referenced A:active M:mmap U:uptodate D:dirty W:writeback X:readahead P:private O:owner b:buffer d:dirty w:writeback
>         # idx   len     state           refcnt
>         0       1       RAMU________    4
>         3       8       RAMU________    4
>         12      1       RAMU________    4
>         14      5       RAMU________    4
>         20      7       RAMU________    4
>         27      2       RAMU________    5
>         29      1       RAMU________    4
> 
> Which is also a good candidate. However I still need to investigate
> whether it offers considerable margins over the mincore() syscall.
> 
> Thanks and Regards,
> Fengguang

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>