[PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrey Vagin]

Here is a preview version. It provides restricted set of functionality.
I would like to collect feedback about this idea.

Currently we use the proc file system, where all information are
presented in text files, what is convenient for humans.  But if we need
to get information about processes from code (e.g. in C), the procfs
doesn't look so cool.

>From code we would prefer to get information in binary format and to be
able to specify which information and for which tasks are required. Here
is a new interface with all these features, which is called task_diag.
In addition it's much faster than procfs.

task_diag is based on netlink sockets and looks like socket-diag, which
is used to get information about sockets.

A request is described by the task_diag_pid structure:

struct task_diag_pid {
       __u64   show_flags;	/* specify which information are required */
       __u64   dump_stratagy;   /* specify a group of processes */

       __u32   pid;
};

A respone is a set of netlink messages. Each message describes one task.
All task properties are divided on groups. A message contains the
TASK_DIAG_MSG group and other groups if they have been requested in
show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
response will contain the TASK_DIAG_CRED group which is described by the
task_diag_creds structure.

struct task_diag_msg {
	__u32	tgid;
	__u32	pid;
	__u32	ppid;
	__u32	tpid;
	__u32	sid;
	__u32	pgid;
	__u8	state;
	char	comm[TASK_DIAG_COMM_LEN];
};

Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL	- get information for all tasks
TASK_DIAG_DUMP_CHILDREN	- get information for children of a specified
			  tasks

The task diag is much faster than the proc file system. We don't need to
create a new file descriptor for each task. We need to send a request
and get a response. It allows to get information for a few task in one
request-response iteration.

I have compared performance of procfs and task-diag for the
"ps ax -o pid,ppid" command.

A test stand contains 10348 processes.
$ ps ax -o pid,ppid | wc -l
10348

$ time ps ax -o pid,ppid > /dev/null

real	0m1.073s
user	0m0.086s
sys	0m0.903s

$ time ./task_diag_all > /dev/null

real	0m0.037s
user	0m0.004s
sys	0m0.020s

And here are statistics about syscalls which were called by each
command.
$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
            20,713      syscalls:sys_exit_open
            20,710      syscalls:sys_exit_close
            20,708      syscalls:sys_exit_read
            10,348      syscalls:sys_exit_newstat
                31      syscalls:sys_exit_write

$ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
               114      syscalls:sys_exit_recvfrom
                49      syscalls:sys_exit_write
                 8      syscalls:sys_exit_mmap
                 4      syscalls:sys_exit_mprotect
                 3      syscalls:sys_exit_newfstat

You can find the test program from this experiment in the last patch.

The idea of this functionality was suggested by Pavel Emelyanov
(xemul@), when he found that operations with /proc forms a significant
part of a checkpointing time.

Ten years ago here was attempt to add a netlink interface to access to /proc
information:
http://lwn.net/Articles/99600/

Signed-off-by: Andrey Vagin <avagin@openvz.org>

git repo: https://github.com/avagin/linux-task-diag

Andrey Vagin (7):
  [RFC] kernel: add a netlink interface to get information about tasks
  kernel: move next_tgid from fs/proc
  task-diag: add ability to get information about all tasks
  task-diag: add a new group to get process credentials
  kernel: add ability to iterate children of a specified task
  task_diag: add ability to dump children
  selftest: check the task_diag functinonality

 fs/proc/array.c                                    |  58 +---
 fs/proc/base.c                                     |  43 ---
 include/linux/proc_fs.h                            |  13 +
 include/uapi/linux/taskdiag.h                      |  89 ++++++
 init/Kconfig                                       |  12 +
 kernel/Makefile                                    |   1 +
 kernel/pid.c                                       |  94 ++++++
 kernel/taskdiag.c                                  | 343 +++++++++++++++++++++
 tools/testing/selftests/task_diag/Makefile         |  16 +
 tools/testing/selftests/task_diag/task_diag.c      |  59 ++++
 tools/testing/selftests/task_diag/task_diag_all.c  |  82 +++++
 tools/testing/selftests/task_diag/task_diag_comm.c | 195 ++++++++++++
 tools/testing/selftests/task_diag/task_diag_comm.h |  47 +++
 tools/testing/selftests/task_diag/taskdiag.h       |   1 +
 14 files changed, 967 insertions(+), 86 deletions(-)
 create mode 100644 include/uapi/linux/taskdiag.h
 create mode 100644 kernel/taskdiag.c
 create mode 100644 tools/testing/selftests/task_diag/Makefile
 create mode 100644 tools/testing/selftests/task_diag/task_diag.c
 create mode 100644 tools/testing/selftests/task_diag/task_diag_all.c
 create mode 100644 tools/testing/selftests/task_diag/task_diag_comm.c
 create mode 100644 tools/testing/selftests/task_diag/task_diag_comm.h
 create mode 120000 tools/testing/selftests/task_diag/taskdiag.h

Cc: Oleg Nesterov <oleg@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Cyrill Gorcunov <gorcunov@openvz.org>
Cc: Pavel Emelyanov <xemul@parallels.com>
Cc: Roger Luethi <rl@hellgate.ch>
-- 
2.1.0

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Arnd Bergmann]

On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> task_diag is based on netlink sockets and looks like socket-diag, which
> is used to get information about sockets.
> 
> A request is described by the task_diag_pid structure:
> 
> struct task_diag_pid {
>        __u64   show_flags;      /* specify which information are required */
>        __u64   dump_stratagy;   /* specify a group of processes */
> 
>        __u32   pid;
> };

Can you explain how the interface relates to the 'taskstats' genetlink
API? Did you consider extending that interface to provide the
information you need instead of basing on the socket-diag?

	Arnd

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[David Ahern]

On 2/17/15 1:20 AM, Andrey Vagin wrote:
> And here are statistics about syscalls which were called by each
> command.
> $ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
>              20,713      syscalls:sys_exit_open
>              20,710      syscalls:sys_exit_close
>              20,708      syscalls:sys_exit_read
>              10,348      syscalls:sys_exit_newstat
>                  31      syscalls:sys_exit_write
>
> $ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
>                 114      syscalls:sys_exit_recvfrom
>                  49      syscalls:sys_exit_write
>                   8      syscalls:sys_exit_mmap
>                   4      syscalls:sys_exit_mprotect
>                   3      syscalls:sys_exit_newfstat

'perf trace -s' gives the summary with stats.
e.g., perf trace -s --  ps ax -o pid,ppid

  ps (23850), 3117 events, 99.3%, 0.000 msec

    syscall            calls      min       avg       max      stddev
                                (msec)    (msec)    (msec)        (%)
    --------------- -------- --------- --------- ---------     ------
    read                 353     0.000     0.010     0.035      3.14%
    write                166     0.006     0.012     0.045      3.03%
    open                 365     0.002     0.005     0.178     11.29%
    close                354     0.001     0.002     0.024      3.57%
    stat                 170     0.002     0.007     0.662     52.99%
    fstat                 19     0.002     0.003     0.003      2.31%
    lseek                  2     0.003     0.003     0.003      6.49%
    mmap                  50     0.004     0.006     0.013      3.40%
...

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andy Lutomirski]

On Feb 17, 2015 12:40 AM, "Andrey Vagin" <avagin@openvz.org> wrote:
>
> Here is a preview version. It provides restricted set of functionality.
> I would like to collect feedback about this idea.
>
> Currently we use the proc file system, where all information are
> presented in text files, what is convenient for humans.  But if we need
> to get information about processes from code (e.g. in C), the procfs
> doesn't look so cool.
>
> From code we would prefer to get information in binary format and to be
> able to specify which information and for which tasks are required. Here
> is a new interface with all these features, which is called task_diag.
> In addition it's much faster than procfs.
>
> task_diag is based on netlink sockets and looks like socket-diag, which
> is used to get information about sockets.
>
> A request is described by the task_diag_pid structure:
>
> struct task_diag_pid {
>        __u64   show_flags;      /* specify which information are required */
>        __u64   dump_stratagy;   /* specify a group of processes */
>
>        __u32   pid;
> };
>
> A respone is a set of netlink messages. Each message describes one task.
> All task properties are divided on groups. A message contains the
> TASK_DIAG_MSG group and other groups if they have been requested in
> show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
> response will contain the TASK_DIAG_CRED group which is described by the
> task_diag_creds structure.
>
> struct task_diag_msg {
>         __u32   tgid;
>         __u32   pid;
>         __u32   ppid;
>         __u32   tpid;
>         __u32   sid;
>         __u32   pgid;
>         __u8    state;
>         char    comm[TASK_DIAG_COMM_LEN];
> };
>
> Another good feature of task_diag is an ability to request information
> for a few processes. Currently here are two stratgies
> TASK_DIAG_DUMP_ALL      - get information for all tasks
> TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
>                           tasks
>
> The task diag is much faster than the proc file system. We don't need to
> create a new file descriptor for each task. We need to send a request
> and get a response. It allows to get information for a few task in one
> request-response iteration.
>
> I have compared performance of procfs and task-diag for the
> "ps ax -o pid,ppid" command.
>
> A test stand contains 10348 processes.
> $ ps ax -o pid,ppid | wc -l
> 10348
>
> $ time ps ax -o pid,ppid > /dev/null
>
> real    0m1.073s
> user    0m0.086s
> sys     0m0.903s
>
> $ time ./task_diag_all > /dev/null
>
> real    0m0.037s
> user    0m0.004s
> sys     0m0.020s
>
> And here are statistics about syscalls which were called by each
> command.
> $ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
>             20,713      syscalls:sys_exit_open
>             20,710      syscalls:sys_exit_close
>             20,708      syscalls:sys_exit_read
>             10,348      syscalls:sys_exit_newstat
>                 31      syscalls:sys_exit_write
>
> $ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
>                114      syscalls:sys_exit_recvfrom
>                 49      syscalls:sys_exit_write
>                  8      syscalls:sys_exit_mmap
>                  4      syscalls:sys_exit_mprotect
>                  3      syscalls:sys_exit_newfstat
>
> You can find the test program from this experiment in the last patch.
>
> The idea of this functionality was suggested by Pavel Emelyanov
> (xemul@), when he found that operations with /proc forms a significant
> part of a checkpointing time.
>
> Ten years ago here was attempt to add a netlink interface to access to /proc
> information:
> http://lwn.net/Articles/99600/

I don't suppose this could use real syscalls instead of netlink.  If
nothing else, netlink seems to conflate pid and net namespaces.

Also, using an asynchronous interface (send, poll?, recv) for
something that's inherently synchronous (as the kernel a local
question) seems awkward to me.

--Andy

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Tue, Feb 17, 2015 at 09:09:47AM -0700, David Ahern wrote:
> On 2/17/15 1:20 AM, Andrey Vagin wrote:
> >And here are statistics about syscalls which were called by each
> >command.
> >$ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
> >             20,713      syscalls:sys_exit_open
> >             20,710      syscalls:sys_exit_close
> >             20,708      syscalls:sys_exit_read
> >             10,348      syscalls:sys_exit_newstat
> >                 31      syscalls:sys_exit_write
> >
> >$ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
> >                114      syscalls:sys_exit_recvfrom
> >                 49      syscalls:sys_exit_write
> >                  8      syscalls:sys_exit_mmap
> >                  4      syscalls:sys_exit_mprotect
> >                  3      syscalls:sys_exit_newfstat
> 
> 'perf trace -s' gives the summary with stats.
> e.g., perf trace -s --  ps ax -o pid,ppid

Thank you for this command, I haven't used it before.

 ps (21301), 145271 events, 100.0%, 0.000 msec

   syscall            calls      min       avg       max      stddev
                               (msec)    (msec)    (msec)        (%)
   --------------- -------- --------- --------- ---------     ------
   read               20717     0.000     0.020     1.631      0.64%
   write                  1     0.019     0.019     0.019      0.00%
   open               20722     0.025     0.035     3.624      0.93%
   close              20719     0.006     0.009     1.059      0.95%
   stat               10352     0.015     0.025     1.748      0.95%
   fstat                 12     0.010     0.012     0.020      6.17%
   lseek                  2     0.011     0.012     0.012      3.08%
   mmap                  30     0.012     0.034     0.094      9.35%
   mprotect              17     0.034     0.045     0.067      4.86%
   munmap                 3     0.028     0.058     0.108     44.12%
   brk                    4     0.011     0.015     0.019     11.24%
   rt_sigaction          25     0.011     0.011     0.014      1.27%
   rt_sigprocmask         1     0.012     0.012     0.012      0.00%
   ioctl                  4     0.010     0.012     0.014      6.94%
   access                 1     0.034     0.034     0.034      0.00%
   execve                 6     0.000     0.496     2.794     92.58%
   uname                  1     0.015     0.015     0.015      0.00%
   getdents              12     0.019     0.691     1.158     13.04%
   getrlimit              1     0.012     0.012     0.012      0.00%
   geteuid                1     0.012     0.012     0.012      0.00%
   arch_prctl             1     0.013     0.013     0.013      0.00%
   futex                  1     0.020     0.020     0.020      0.00%
   set_tid_address        1     0.012     0.012     0.012      0.00%
   openat                 1     0.030     0.030     0.030      0.00%
   set_robust_list        1     0.011     0.011     0.011      0.00%


 task_diag_all (21304), 569 events, 98.6%, 0.000 msec

   syscall            calls      min       avg       max      stddev
                               (msec)    (msec)    (msec)        (%)
   --------------- -------- --------- --------- ---------     ------
   read                   2     0.000     0.045     0.090    100.00%
   write                 77     0.010     0.013     0.083      7.93%
   open                   2     0.031     0.038     0.045     19.64%
   close                  3     0.010     0.014     0.017     13.43%
   fstat                  3     0.011     0.011     0.012      3.79%
   mmap                   8     0.013     0.027     0.049     16.72%
   mprotect               4     0.034     0.043     0.052      8.86%
   munmap                 1     0.031     0.031     0.031      0.00%
   brk                    1     0.014     0.014     0.014      0.00%
   ioctl                  1     0.010     0.010     0.010      0.00%
   access                 1     0.030     0.030     0.030      0.00%
   getpid                 1     0.011     0.011     0.011      0.00%
   socket                 1     0.045     0.045     0.045      0.00%
   sendto                 2     0.091     0.104     0.117     12.63%
   recvfrom             175     0.026     0.093     0.141      1.10%
   bind                   1     0.014     0.014     0.014      0.00%
   execve                 1     0.000     0.000     0.000      0.00%
   arch_prctl             1     0.011     0.011     0.011      0.00%

> 
>  ps (23850), 3117 events, 99.3%, 0.000 msec
> 
>    syscall            calls      min       avg       max      stddev
>                                (msec)    (msec)    (msec)        (%)
>    --------------- -------- --------- --------- ---------     ------
>    read                 353     0.000     0.010     0.035      3.14%
>    write                166     0.006     0.012     0.045      3.03%
>    open                 365     0.002     0.005     0.178     11.29%
>    close                354     0.001     0.002     0.024      3.57%
>    stat                 170     0.002     0.007     0.662     52.99%
>    fstat                 19     0.002     0.003     0.003      2.31%
>    lseek                  2     0.003     0.003     0.003      6.49%
>    mmap                  50     0.004     0.006     0.013      3.40%
> ...

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
> On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> > task_diag is based on netlink sockets and looks like socket-diag, which
> > is used to get information about sockets.
> > 
> > A request is described by the task_diag_pid structure:
> > 
> > struct task_diag_pid {
> >        __u64   show_flags;      /* specify which information are required */
> >        __u64   dump_stratagy;   /* specify a group of processes */
> > 
> >        __u32   pid;
> > };
> 
> Can you explain how the interface relates to the 'taskstats' genetlink
> API? Did you consider extending that interface to provide the
> information you need instead of basing on the socket-diag?

It isn't based on the socket-diag, it looks like socket-diag.

Current task_diag registers a new genl family, but we can use the taskstats
family and add task_diag commands to it.

Thanks,
Andrew

> 
> 	Arnd

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Arnd Bergmann]

On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
> On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
> > On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > is used to get information about sockets.
> > > 
> > > A request is described by the task_diag_pid structure:
> > > 
> > > struct task_diag_pid {
> > >        __u64   show_flags;      /* specify which information are required */
> > >        __u64   dump_stratagy;   /* specify a group of processes */
> > > 
> > >        __u32   pid;
> > > };
> > 
> > Can you explain how the interface relates to the 'taskstats' genetlink
> > API? Did you consider extending that interface to provide the
> > information you need instead of basing on the socket-diag?
> 
> It isn't based on the socket-diag, it looks like socket-diag.
> 
> Current task_diag registers a new genl family, but we can use the taskstats
> family and add task_diag commands to it.

What I meant was more along the lines of making it look like taskstats
by adding new fields to 'struct taskstat' for what you want return.
I don't know if that is possible or a good idea for the information
you want to get out of the kernel, but it seems like a more natural
interface, as it already has some of the same data (comm, gid, pid,
ppid, ...).

	Arnd

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
> On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
> > On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
> > > On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> > > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > > is used to get information about sockets.
> > > > 
> > > > A request is described by the task_diag_pid structure:
> > > > 
> > > > struct task_diag_pid {
> > > >        __u64   show_flags;      /* specify which information are required */
> > > >        __u64   dump_stratagy;   /* specify a group of processes */
> > > > 
> > > >        __u32   pid;
> > > > };
> > > 
> > > Can you explain how the interface relates to the 'taskstats' genetlink
> > > API? Did you consider extending that interface to provide the
> > > information you need instead of basing on the socket-diag?
> > 
> > It isn't based on the socket-diag, it looks like socket-diag.
> > 
> > Current task_diag registers a new genl family, but we can use the taskstats
> > family and add task_diag commands to it.
> 
> What I meant was more along the lines of making it look like taskstats
> by adding new fields to 'struct taskstat' for what you want return.
> I don't know if that is possible or a good idea for the information
> you want to get out of the kernel, but it seems like a more natural
> interface, as it already has some of the same data (comm, gid, pid,
> ppid, ...).

Now I see what you mean. task_diag has more flexible and universal
interface than taskstat. A response of taskstat only contains a
taskstats structure. A response of taskdiag can contains a few types of
properties. Each type is described by its own structure.

Curently here are only two groups of parameters: task_diag_msg and
task_diag_creds.

task_diag_msg contains a few basic parameters.
task_diag_creds contains credentials.

I'm going to add other groups to describe all kind of task properties
which currently are presented in procfs (e.g. /proc/pid/maps,
/proc/pid/fding/*, /proc/pid/status, etc).

One of features of task_diag is an ability to choose which information
are required. This allows to minimize a response size and a time, which
is requred to fill this response.

struct task_diag_msg {
        __u32   tgid;
        __u32   pid;
        __u32   ppid;
        __u32   tpid;
        __u32   sid;
        __u32   pgid;
        __u8    state;
        char    comm[TASK_DIAG_COMM_LEN];
};

struct task_diag_creds {
        struct task_diag_caps cap_inheritable;
        struct task_diag_caps cap_permitted;
        struct task_diag_caps cap_effective;
        struct task_diag_caps cap_bset;

        __u32 uid;
        __u32 euid;
        __u32 suid;
        __u32 fsuid;
        __u32 gid;
        __u32 egid;
        __u32 sgid;
        __u32 fsgid;
};

Thanks,
Andrew
> 
> 	Arnd

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
> On Feb 17, 2015 12:40 AM, "Andrey Vagin" <avagin@openvz.org> wrote:
> >
> > Here is a preview version. It provides restricted set of functionality.
> > I would like to collect feedback about this idea.
> >
> > Currently we use the proc file system, where all information are
> > presented in text files, what is convenient for humans.  But if we need
> > to get information about processes from code (e.g. in C), the procfs
> > doesn't look so cool.
> >
> > From code we would prefer to get information in binary format and to be
> > able to specify which information and for which tasks are required. Here
> > is a new interface with all these features, which is called task_diag.
> > In addition it's much faster than procfs.
> >
> > task_diag is based on netlink sockets and looks like socket-diag, which
> > is used to get information about sockets.
> >
> > A request is described by the task_diag_pid structure:
> >
> > struct task_diag_pid {
> >        __u64   show_flags;      /* specify which information are required */
> >        __u64   dump_stratagy;   /* specify a group of processes */
> >
> >        __u32   pid;
> > };
> >
> > A respone is a set of netlink messages. Each message describes one task.
> > All task properties are divided on groups. A message contains the
> > TASK_DIAG_MSG group and other groups if they have been requested in
> > show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
> > response will contain the TASK_DIAG_CRED group which is described by the
> > task_diag_creds structure.
> >
> > struct task_diag_msg {
> >         __u32   tgid;
> >         __u32   pid;
> >         __u32   ppid;
> >         __u32   tpid;
> >         __u32   sid;
> >         __u32   pgid;
> >         __u8    state;
> >         char    comm[TASK_DIAG_COMM_LEN];
> > };
> >
> > Another good feature of task_diag is an ability to request information
> > for a few processes. Currently here are two stratgies
> > TASK_DIAG_DUMP_ALL      - get information for all tasks
> > TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
> >                           tasks
> >
> > The task diag is much faster than the proc file system. We don't need to
> > create a new file descriptor for each task. We need to send a request
> > and get a response. It allows to get information for a few task in one
> > request-response iteration.
> >
> > I have compared performance of procfs and task-diag for the
> > "ps ax -o pid,ppid" command.
> >
> > A test stand contains 10348 processes.
> > $ ps ax -o pid,ppid | wc -l
> > 10348
> >
> > $ time ps ax -o pid,ppid > /dev/null
> >
> > real    0m1.073s
> > user    0m0.086s
> > sys     0m0.903s
> >
> > $ time ./task_diag_all > /dev/null
> >
> > real    0m0.037s
> > user    0m0.004s
> > sys     0m0.020s
> >
> > And here are statistics about syscalls which were called by each
> > command.
> > $ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
> >             20,713      syscalls:sys_exit_open
> >             20,710      syscalls:sys_exit_close
> >             20,708      syscalls:sys_exit_read
> >             10,348      syscalls:sys_exit_newstat
> >                 31      syscalls:sys_exit_write
> >
> > $ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
> >                114      syscalls:sys_exit_recvfrom
> >                 49      syscalls:sys_exit_write
> >                  8      syscalls:sys_exit_mmap
> >                  4      syscalls:sys_exit_mprotect
> >                  3      syscalls:sys_exit_newfstat
> >
> > You can find the test program from this experiment in the last patch.
> >
> > The idea of this functionality was suggested by Pavel Emelyanov
> > (xemul@), when he found that operations with /proc forms a significant
> > part of a checkpointing time.
> >
> > Ten years ago here was attempt to add a netlink interface to access to /proc
> > information:
> > http://lwn.net/Articles/99600/
> 
> I don't suppose this could use real syscalls instead of netlink.  If
> nothing else, netlink seems to conflate pid and net namespaces.

What do you mean by "conflate pid and net namespaces"?

> 
> Also, using an asynchronous interface (send, poll?, recv) for
> something that's inherently synchronous (as the kernel a local
> question) seems awkward to me.

Actually all requests are handled synchronously. We call sendmsg to send
a request and it is handled in this syscall.
 2)               |  netlink_sendmsg() {
 2)               |    netlink_unicast() {
 2)               |      taskdiag_doit() {
 2)   2.153 us    |        task_diag_fill();
 2)               |        netlink_unicast() {
 2)   0.185 us    |          netlink_attachskb();
 2)   0.291 us    |          __netlink_sendskb();
 2)   2.452 us    |        }
 2) + 33.625 us   |      }
 2) + 54.611 us   |    }
 2) + 76.370 us   |  }
 2)               |  netlink_recvmsg() {
 2)   1.178 us    |    skb_recv_datagram();
 2) + 46.953 us   |  }

If we request information for a group of tasks (NLM_F_DUMP), a first
portion of data is filled from the sendmsg syscall. And then when we read
it, the kernel fills the next portion.

 3)               |  netlink_sendmsg() {
 3)               |    __netlink_dump_start() {
 3)               |      netlink_dump() {
 3)               |        taskdiag_dumpid() {
 3)   0.685 us    |          task_diag_fill();
...
 3)   0.224 us    |          task_diag_fill();
 3) + 74.028 us   |        }
 3) + 88.757 us   |      }
 3) + 89.296 us   |    }
 3) + 98.705 us   |  }
 3)               |  netlink_recvmsg() {
 3)               |    netlink_dump() {
 3)               |      taskdiag_dumpid() {
 3)   0.594 us    |        task_diag_fill();
...
 3)   0.242 us    |        task_diag_fill();
 3) + 60.634 us   |      }
 3) + 72.803 us   |    }
 3) + 88.005 us   |  }
 3)               |  netlink_recvmsg() {
 3)               |    netlink_dump() {
 3)   2.403 us    |      taskdiag_dumpid();
 3) + 26.236 us   |    }
 3) + 40.522 us   |  }
 0) + 20.407 us   |  netlink_recvmsg();


netlink is really good for this type of tasks.  It allows to create an
extendable interface which can be easy customized for different needs.

I don't think that we would want to create another similar interface
just to be independent from network subsystem.

Thanks,
Andrew

> 
> --Andy

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Arnd Bergmann]

On Wednesday 18 February 2015 15:42:11 Andrew Vagin wrote:
> On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
> > On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
> > > On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
> > > > On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> > > > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > > > is used to get information about sockets.
> > > > > 
> > > > > A request is described by the task_diag_pid structure:
> > > > > 
> > > > > struct task_diag_pid {
> > > > >        __u64   show_flags;      /* specify which information are required */
> > > > >        __u64   dump_stratagy;   /* specify a group of processes */
> > > > > 
> > > > >        __u32   pid;
> > > > > };
> > > > 
> > > > Can you explain how the interface relates to the 'taskstats' genetlink
> > > > API? Did you consider extending that interface to provide the
> > > > information you need instead of basing on the socket-diag?
> > > 
> > > It isn't based on the socket-diag, it looks like socket-diag.
> > > 
> > > Current task_diag registers a new genl family, but we can use the taskstats
> > > family and add task_diag commands to it.
> > 
> > What I meant was more along the lines of making it look like taskstats
> > by adding new fields to 'struct taskstat' for what you want return.
> > I don't know if that is possible or a good idea for the information
> > you want to get out of the kernel, but it seems like a more natural
> > interface, as it already has some of the same data (comm, gid, pid,
> > ppid, ...).
> 
> Now I see what you mean. task_diag has more flexible and universal
> interface than taskstat. A response of taskstat only contains a
> taskstats structure. A response of taskdiag can contains a few types of
> properties. Each type is described by its own structure.

Right, so the question is whether that flexibility is actually required
here. Independent of which design you personally prefer, what are the
downsides of extending the existing but less flexible interface?

If it's good enough, that would seem to provide a more consistent
API, which in turn helps users understand the interface and use it
correctly.

> Curently here are only two groups of parameters: task_diag_msg and
> task_diag_creds.
> 
> task_diag_msg contains a few basic parameters.
> task_diag_creds contains credentials.
> 
> I'm going to add other groups to describe all kind of task properties
> which currently are presented in procfs (e.g. /proc/pid/maps,
> /proc/pid/fding/*, /proc/pid/status, etc).
> 
> One of features of task_diag is an ability to choose which information
> are required. This allows to minimize a response size and a time, which
> is requred to fill this response.

I realize that you are trying to optimize for performance, but it
would be nice to quantify this if you want to argue for requiring
a split interface.

> struct task_diag_msg {
>         __u32   tgid;
>         __u32   pid;
>         __u32   ppid;
>         __u32   tpid;
>         __u32   sid;
>         __u32   pgid;
>         __u8    state;
>         char    comm[TASK_DIAG_COMM_LEN];
> };

I guess this part would be a very natural extension to the
existing taskstats structure, and we should only add a new
one here if there are extremely good reasons for it.

> struct task_diag_creds {
>         struct task_diag_caps cap_inheritable;
>         struct task_diag_caps cap_permitted;
>         struct task_diag_caps cap_effective;
>         struct task_diag_caps cap_bset;
> 
>         __u32 uid;
>         __u32 euid;
>         __u32 suid;
>         __u32 fsuid;
>         __u32 gid;
>         __u32 egid;
>         __u32 sgid;
>         __u32 fsgid;
> };

while this part could well be kept separate so you can query
it individually from the rest of taskstats, but through a
related interface.

	Arnd

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andy Lutomirski]

On Feb 18, 2015 6:27 AM, "Andrew Vagin" <avagin@parallels.com> wrote:
>
> On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
> > On Feb 17, 2015 12:40 AM, "Andrey Vagin" <avagin@openvz.org> wrote:
> > >
> > > Here is a preview version. It provides restricted set of functionality.
> > > I would like to collect feedback about this idea.
> > >
> > > Currently we use the proc file system, where all information are
> > > presented in text files, what is convenient for humans.  But if we need
> > > to get information about processes from code (e.g. in C), the procfs
> > > doesn't look so cool.
> > >
> > > From code we would prefer to get information in binary format and to be
> > > able to specify which information and for which tasks are required. Here
> > > is a new interface with all these features, which is called task_diag.
> > > In addition it's much faster than procfs.
> > >
> > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > is used to get information about sockets.
> > >
> > > A request is described by the task_diag_pid structure:
> > >
> > > struct task_diag_pid {
> > >        __u64   show_flags;      /* specify which information are required */
> > >        __u64   dump_stratagy;   /* specify a group of processes */
> > >
> > >        __u32   pid;
> > > };
> > >
> > > A respone is a set of netlink messages. Each message describes one task.
> > > All task properties are divided on groups. A message contains the
> > > TASK_DIAG_MSG group and other groups if they have been requested in
> > > show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
> > > response will contain the TASK_DIAG_CRED group which is described by the
> > > task_diag_creds structure.
> > >
> > > struct task_diag_msg {
> > >         __u32   tgid;
> > >         __u32   pid;
> > >         __u32   ppid;
> > >         __u32   tpid;
> > >         __u32   sid;
> > >         __u32   pgid;
> > >         __u8    state;
> > >         char    comm[TASK_DIAG_COMM_LEN];
> > > };
> > >
> > > Another good feature of task_diag is an ability to request information
> > > for a few processes. Currently here are two stratgies
> > > TASK_DIAG_DUMP_ALL      - get information for all tasks
> > > TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
> > >                           tasks
> > >
> > > The task diag is much faster than the proc file system. We don't need to
> > > create a new file descriptor for each task. We need to send a request
> > > and get a response. It allows to get information for a few task in one
> > > request-response iteration.
> > >
> > > I have compared performance of procfs and task-diag for the
> > > "ps ax -o pid,ppid" command.
> > >
> > > A test stand contains 10348 processes.
> > > $ ps ax -o pid,ppid | wc -l
> > > 10348
> > >
> > > $ time ps ax -o pid,ppid > /dev/null
> > >
> > > real    0m1.073s
> > > user    0m0.086s
> > > sys     0m0.903s
> > >
> > > $ time ./task_diag_all > /dev/null
> > >
> > > real    0m0.037s
> > > user    0m0.004s
> > > sys     0m0.020s
> > >
> > > And here are statistics about syscalls which were called by each
> > > command.
> > > $ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
> > >             20,713      syscalls:sys_exit_open
> > >             20,710      syscalls:sys_exit_close
> > >             20,708      syscalls:sys_exit_read
> > >             10,348      syscalls:sys_exit_newstat
> > >                 31      syscalls:sys_exit_write
> > >
> > > $ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
> > >                114      syscalls:sys_exit_recvfrom
> > >                 49      syscalls:sys_exit_write
> > >                  8      syscalls:sys_exit_mmap
> > >                  4      syscalls:sys_exit_mprotect
> > >                  3      syscalls:sys_exit_newfstat
> > >
> > > You can find the test program from this experiment in the last patch.
> > >
> > > The idea of this functionality was suggested by Pavel Emelyanov
> > > (xemul@), when he found that operations with /proc forms a significant
> > > part of a checkpointing time.
> > >
> > > Ten years ago here was attempt to add a netlink interface to access to /proc
> > > information:
> > > http://lwn.net/Articles/99600/
> >
> > I don't suppose this could use real syscalls instead of netlink.  If
> > nothing else, netlink seems to conflate pid and net namespaces.
>
> What do you mean by "conflate pid and net namespaces"?

A netlink socket is bound to a network namespace, but you should be
returning data specific to a pid namespace.

On a related note, how does this interact with hidepid?  More
generally, what privileges are you requiring to obtain what data?

>
> >
> > Also, using an asynchronous interface (send, poll?, recv) for
> > something that's inherently synchronous (as the kernel a local
> > question) seems awkward to me.
>
> Actually all requests are handled synchronously. We call sendmsg to send
> a request and it is handled in this syscall.
>  2)               |  netlink_sendmsg() {
>  2)               |    netlink_unicast() {
>  2)               |      taskdiag_doit() {
>  2)   2.153 us    |        task_diag_fill();
>  2)               |        netlink_unicast() {
>  2)   0.185 us    |          netlink_attachskb();
>  2)   0.291 us    |          __netlink_sendskb();
>  2)   2.452 us    |        }
>  2) + 33.625 us   |      }
>  2) + 54.611 us   |    }
>  2) + 76.370 us   |  }
>  2)               |  netlink_recvmsg() {
>  2)   1.178 us    |    skb_recv_datagram();
>  2) + 46.953 us   |  }
>
> If we request information for a group of tasks (NLM_F_DUMP), a first
> portion of data is filled from the sendmsg syscall. And then when we read
> it, the kernel fills the next portion.
>
>  3)               |  netlink_sendmsg() {
>  3)               |    __netlink_dump_start() {
>  3)               |      netlink_dump() {
>  3)               |        taskdiag_dumpid() {
>  3)   0.685 us    |          task_diag_fill();
> ...
>  3)   0.224 us    |          task_diag_fill();
>  3) + 74.028 us   |        }
>  3) + 88.757 us   |      }
>  3) + 89.296 us   |    }
>  3) + 98.705 us   |  }
>  3)               |  netlink_recvmsg() {
>  3)               |    netlink_dump() {
>  3)               |      taskdiag_dumpid() {
>  3)   0.594 us    |        task_diag_fill();
> ...
>  3)   0.242 us    |        task_diag_fill();
>  3) + 60.634 us   |      }
>  3) + 72.803 us   |    }
>  3) + 88.005 us   |  }
>  3)               |  netlink_recvmsg() {
>  3)               |    netlink_dump() {
>  3)   2.403 us    |      taskdiag_dumpid();
>  3) + 26.236 us   |    }
>  3) + 40.522 us   |  }
>  0) + 20.407 us   |  netlink_recvmsg();
>
>
> netlink is really good for this type of tasks.  It allows to create an
> extendable interface which can be easy customized for different needs.
>
> I don't think that we would want to create another similar interface
> just to be independent from network subsystem.

I guess this is a bit streamy in that you ask one question and get
multiple answers.

>
> Thanks,
> Andrew
>
> >
> > --Andy

[PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Pavel Odintsov]

Hello, folks!

It's very useful patches and they can do my tasks simpler and faster.

In my day to day work I working with Linux servers with enormous
amount of processes (~25 000 per server). This servers run multiple
hundreds of Linux containers.

If I want analyze processor load, network load or check something else
I use top/atop/htop/netstat. But they work very slow and consume
significant amount of CPU power for parsing multiple thousands text
files in /proc (like /proc/tcp, /proc/udp, /proc/status,
/proc/$pid/status).

Some time ago I worked on malware detection toolkit for Linux -
Antidoto (https://github.com/FastVPSEestiOu/Antidoto) which uses /proc
filesystem very deeply. For detecting malware I need check every
descriptor, every sockets and get complete information about all
processes on system.

But with current text file based architecture of /proc I can't achieve
suitable speed of my toolkit.

For example, there you can look at time of processing all network
connections for server with 20244 processes with
linux_network_activity_tracker.pl
(https://github.com/FastVPSEestiOu/Antidoto/blob/master/linux_network_activity_tracker.pl):

real 1m26.637s
user 0m23.945s
sys 0m43.978s

As you can see this time is very huge but I use latest CPUs from Intel
(Xepn 2697v3).

I have multiple ideas about complete realtime Linux server monitoring
but without ability to pull information from the Linux Kernel faster I
can't realize they.

-- 
Sincerely yours, Pavel Odintsov

[PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Pavel Odintsov]

Hello!

In addition to my post I want to mention another issue related with
slow /proc read in perf toolkit. On my server with 25 000 processes I
need about ~15 minutes for loading perf top toolkit completely.

https://bugzilla.kernel.org/show_bug.cgi?id=86991


-- 
Sincerely yours, Pavel Odintsov

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Wed, Feb 18, 2015 at 03:46:31PM +0100, Arnd Bergmann wrote:
> On Wednesday 18 February 2015 15:42:11 Andrew Vagin wrote:
> > On Wed, Feb 18, 2015 at 12:06:40PM +0100, Arnd Bergmann wrote:
> > > On Wednesday 18 February 2015 00:33:13 Andrew Vagin wrote:
> > > > On Tue, Feb 17, 2015 at 09:53:09AM +0100, Arnd Bergmann wrote:
> > > > > On Tuesday 17 February 2015 11:20:19 Andrey Vagin wrote:
> > > > > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > > > > is used to get information about sockets.
> > > > > > 
> > > > > > A request is described by the task_diag_pid structure:
> > > > > > 
> > > > > > struct task_diag_pid {
> > > > > >        __u64   show_flags;      /* specify which information are required */
> > > > > >        __u64   dump_stratagy;   /* specify a group of processes */
> > > > > > 
> > > > > >        __u32   pid;
> > > > > > };
> > > > > 
> > > > > Can you explain how the interface relates to the 'taskstats' genetlink
> > > > > API? Did you consider extending that interface to provide the
> > > > > information you need instead of basing on the socket-diag?
> > > > 
> > > > It isn't based on the socket-diag, it looks like socket-diag.
> > > > 
> > > > Current task_diag registers a new genl family, but we can use the taskstats
> > > > family and add task_diag commands to it.
> > > 
> > > What I meant was more along the lines of making it look like taskstats
> > > by adding new fields to 'struct taskstat' for what you want return.
> > > I don't know if that is possible or a good idea for the information
> > > you want to get out of the kernel, but it seems like a more natural
> > > interface, as it already has some of the same data (comm, gid, pid,
> > > ppid, ...).
> > 
> > Now I see what you mean. task_diag has more flexible and universal
> > interface than taskstat. A response of taskstat only contains a
> > taskstats structure. A response of taskdiag can contains a few types of
> > properties. Each type is described by its own structure.
> 
> Right, so the question is whether that flexibility is actually required
> here. Independent of which design you personally prefer, what are the
> downsides of extending the existing but less flexible interface?

I have looked at taskstat once again.

The format of response messages for taskstat and taskdiag are the same.
It's a netlink message with a set of nested attributes. New attributes
can be added without breaking backward compatibility.

The request can be expanded to be able to specified which information is
required and for which tasks.

These two features allow to significantly improve performance, because
in this case we don't need to do a system call for each task.

I have done a few experiments to prove these words.

task_proc_all reads /proc/pid/stat for each tast
$ time ./task_proc_all > /dev/null

real	0m1.528s
user	0m0.016s
sys	0m1.341s

task_diag uses task_diag and requests information for each task
separately.
$ time ./task_diag > /dev/null

real	0m1.166s
user	0m0.024s
sys	0m1.127s

task_diag_all uses task_diag and requests information for all tasks in
one request.
$ time ./task_diag_all > /dev/null

real	0m0.077s
user	0m0.018s
sys	0m0.053s

So you can see that the ability to request information for a group of
tasks allows to be more effective.

The summary of this message is that we can use the interface of
taskstats with some extensions.

Arnd, thank you for your opinion and suggestions.

> 
> If it's good enough, that would seem to provide a more consistent
> API, which in turn helps users understand the interface and use it
> correctly.
> 
> > Curently here are only two groups of parameters: task_diag_msg and
> > task_diag_creds.
> > 
> > task_diag_msg contains a few basic parameters.
> > task_diag_creds contains credentials.
> > 
> > I'm going to add other groups to describe all kind of task properties
> > which currently are presented in procfs (e.g. /proc/pid/maps,
> > /proc/pid/fding/*, /proc/pid/status, etc).
> > 
> > One of features of task_diag is an ability to choose which information
> > are required. This allows to minimize a response size and a time, which
> > is requred to fill this response.
> 
> I realize that you are trying to optimize for performance, but it
> would be nice to quantify this if you want to argue for requiring
> a split interface.
> 
> > struct task_diag_msg {
> >         __u32   tgid;
> >         __u32   pid;
> >         __u32   ppid;
> >         __u32   tpid;
> >         __u32   sid;
> >         __u32   pgid;
> >         __u8    state;
> >         char    comm[TASK_DIAG_COMM_LEN];
> > };
> 
> I guess this part would be a very natural extension to the
> existing taskstats structure, and we should only add a new
> one here if there are extremely good reasons for it.

The task_diag_msg structure contains properties which are used more
frequently than statistics from the taststats structure.

The size of the task_diag_msg structure is 44 bytes, the size of the
taststats structure 328.  If we have more data, we need to do more
system calls. So I have done one more experiment to look how it affects
perfomance:

If we use the task_diag_msg structure:
$ time ./task_diag_all > /dev/null

real	0m0.077s
user	0m0.018s
sys	0m0.053s

If we use the taststats structure:
$ time ./task_diag_all > /dev/null

real	0m0.117s
user	0m0.029s
sys	0m0.085s

Thanks,
Andrew

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andrew Vagin]

On Wed, Feb 18, 2015 at 05:18:38PM -0800, Andy Lutomirski wrote:
> On Feb 18, 2015 6:27 AM, "Andrew Vagin" <avagin@parallels.com> wrote:
> >
> > On Tue, Feb 17, 2015 at 11:05:31AM -0800, Andy Lutomirski wrote:
> > > On Feb 17, 2015 12:40 AM, "Andrey Vagin" <avagin@openvz.org> wrote:
> > > >
> > > > Here is a preview version. It provides restricted set of functionality.
> > > > I would like to collect feedback about this idea.
> > > >
> > > > Currently we use the proc file system, where all information are
> > > > presented in text files, what is convenient for humans.  But if we need
> > > > to get information about processes from code (e.g. in C), the procfs
> > > > doesn't look so cool.
> > > >
> > > > From code we would prefer to get information in binary format and to be
> > > > able to specify which information and for which tasks are required. Here
> > > > is a new interface with all these features, which is called task_diag.
> > > > In addition it's much faster than procfs.
> > > >
> > > > task_diag is based on netlink sockets and looks like socket-diag, which
> > > > is used to get information about sockets.
> > > >
> > > > A request is described by the task_diag_pid structure:
> > > >
> > > > struct task_diag_pid {
> > > >        __u64   show_flags;      /* specify which information are required */
> > > >        __u64   dump_stratagy;   /* specify a group of processes */
> > > >
> > > >        __u32   pid;
> > > > };
> > > >
> > > > A respone is a set of netlink messages. Each message describes one task.
> > > > All task properties are divided on groups. A message contains the
> > > > TASK_DIAG_MSG group and other groups if they have been requested in
> > > > show_flags. For example, if show_flags contains TASK_DIAG_SHOW_CRED, a
> > > > response will contain the TASK_DIAG_CRED group which is described by the
> > > > task_diag_creds structure.
> > > >
> > > > struct task_diag_msg {
> > > >         __u32   tgid;
> > > >         __u32   pid;
> > > >         __u32   ppid;
> > > >         __u32   tpid;
> > > >         __u32   sid;
> > > >         __u32   pgid;
> > > >         __u8    state;
> > > >         char    comm[TASK_DIAG_COMM_LEN];
> > > > };
> > > >
> > > > Another good feature of task_diag is an ability to request information
> > > > for a few processes. Currently here are two stratgies
> > > > TASK_DIAG_DUMP_ALL      - get information for all tasks
> > > > TASK_DIAG_DUMP_CHILDREN - get information for children of a specified
> > > >                           tasks
> > > >
> > > > The task diag is much faster than the proc file system. We don't need to
> > > > create a new file descriptor for each task. We need to send a request
> > > > and get a response. It allows to get information for a few task in one
> > > > request-response iteration.
> > > >
> > > > I have compared performance of procfs and task-diag for the
> > > > "ps ax -o pid,ppid" command.
> > > >
> > > > A test stand contains 10348 processes.
> > > > $ ps ax -o pid,ppid | wc -l
> > > > 10348
> > > >
> > > > $ time ps ax -o pid,ppid > /dev/null
> > > >
> > > > real    0m1.073s
> > > > user    0m0.086s
> > > > sys     0m0.903s
> > > >
> > > > $ time ./task_diag_all > /dev/null
> > > >
> > > > real    0m0.037s
> > > > user    0m0.004s
> > > > sys     0m0.020s
> > > >
> > > > And here are statistics about syscalls which were called by each
> > > > command.
> > > > $ perf stat -e syscalls:sys_exit* -- ps ax -o pid,ppid  2>&1 | grep syscalls | sort -n -r | head -n 5
> > > >             20,713      syscalls:sys_exit_open
> > > >             20,710      syscalls:sys_exit_close
> > > >             20,708      syscalls:sys_exit_read
> > > >             10,348      syscalls:sys_exit_newstat
> > > >                 31      syscalls:sys_exit_write
> > > >
> > > > $ perf stat -e syscalls:sys_exit* -- ./task_diag_all  2>&1 | grep syscalls | sort -n -r | head -n 5
> > > >                114      syscalls:sys_exit_recvfrom
> > > >                 49      syscalls:sys_exit_write
> > > >                  8      syscalls:sys_exit_mmap
> > > >                  4      syscalls:sys_exit_mprotect
> > > >                  3      syscalls:sys_exit_newfstat
> > > >
> > > > You can find the test program from this experiment in the last patch.
> > > >
> > > > The idea of this functionality was suggested by Pavel Emelyanov
> > > > (xemul@), when he found that operations with /proc forms a significant
> > > > part of a checkpointing time.
> > > >
> > > > Ten years ago here was attempt to add a netlink interface to access to /proc
> > > > information:
> > > > http://lwn.net/Articles/99600/
> > >
> > > I don't suppose this could use real syscalls instead of netlink.  If
> > > nothing else, netlink seems to conflate pid and net namespaces.
> >
> > What do you mean by "conflate pid and net namespaces"?
> 
> A netlink socket is bound to a network namespace, but you should be
> returning data specific to a pid namespace.

Here is a good question. When we mount a procfs instance, the current
pidns is saved on a superblock. Then if we read data from
this procfs from another pidns, we will see pid-s from the pidns where
this procfs has been mounted.

$ unshare -p -- bash -c '(bash)'
$ cat /proc/self/status | grep ^Pid:
Pid:	15770
$ echo $$
1

A similar situation with socket_diag. A socket_diag socket is bound to a
network namespace. If we open a socket_diag socket and change a network
namespace, it will return infromation about the initial netns.

In this version I always use a current pid namespace.
But to be consistant with other kernel logic, a socket diag has to be
linked with a pidns where it has been created.

> 
> On a related note, how does this interact with hidepid?  More

Currently it always work as procfs with hidepid = 2 (highest level of
security).

> generally, what privileges are you requiring to obtain what data?

It dumps information only if ptrace_may_access(tsk, PTRACE_MODE_READ) returns true

> 
> >
> > >
> > > Also, using an asynchronous interface (send, poll?, recv) for
> > > something that's inherently synchronous (as the kernel a local
> > > question) seems awkward to me.
> >
> > Actually all requests are handled synchronously. We call sendmsg to send
> > a request and it is handled in this syscall.
> >  2)               |  netlink_sendmsg() {
> >  2)               |    netlink_unicast() {
> >  2)               |      taskdiag_doit() {
> >  2)   2.153 us    |        task_diag_fill();
> >  2)               |        netlink_unicast() {
> >  2)   0.185 us    |          netlink_attachskb();
> >  2)   0.291 us    |          __netlink_sendskb();
> >  2)   2.452 us    |        }
> >  2) + 33.625 us   |      }
> >  2) + 54.611 us   |    }
> >  2) + 76.370 us   |  }
> >  2)               |  netlink_recvmsg() {
> >  2)   1.178 us    |    skb_recv_datagram();
> >  2) + 46.953 us   |  }
> >
> > If we request information for a group of tasks (NLM_F_DUMP), a first
> > portion of data is filled from the sendmsg syscall. And then when we read
> > it, the kernel fills the next portion.
> >
> >  3)               |  netlink_sendmsg() {
> >  3)               |    __netlink_dump_start() {
> >  3)               |      netlink_dump() {
> >  3)               |        taskdiag_dumpid() {
> >  3)   0.685 us    |          task_diag_fill();
> > ...
> >  3)   0.224 us    |          task_diag_fill();
> >  3) + 74.028 us   |        }
> >  3) + 88.757 us   |      }
> >  3) + 89.296 us   |    }
> >  3) + 98.705 us   |  }
> >  3)               |  netlink_recvmsg() {
> >  3)               |    netlink_dump() {
> >  3)               |      taskdiag_dumpid() {
> >  3)   0.594 us    |        task_diag_fill();
> > ...
> >  3)   0.242 us    |        task_diag_fill();
> >  3) + 60.634 us   |      }
> >  3) + 72.803 us   |    }
> >  3) + 88.005 us   |  }
> >  3)               |  netlink_recvmsg() {
> >  3)               |    netlink_dump() {
> >  3)   2.403 us    |      taskdiag_dumpid();
> >  3) + 26.236 us   |    }
> >  3) + 40.522 us   |  }
> >  0) + 20.407 us   |  netlink_recvmsg();
> >
> >
> > netlink is really good for this type of tasks.  It allows to create an
> > extendable interface which can be easy customized for different needs.
> >
> > I don't think that we would want to create another similar interface
> > just to be independent from network subsystem.
> 
> I guess this is a bit streamy in that you ask one question and get
> multiple answers.

It's like seq_file in procfs. The kernel allocates a buffer then fills
it, copies it into userspace, fills it again, ... repeats these actions.
And we can read data from file by portions.

Actually here is one more analogy. When we open a file in procfs,
we sends a request to the kernel and a file path is a request body in
this case. But in case of procfs, we can't construct requests, we only
have a set of predefined requests.

> 
> >
> > Thanks,
> > Andrew
> >
> > >
> > > --Andy

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Andy Lutomirski]

On Thu, Feb 19, 2015 at 1:39 PM, Andrew Vagin <avagin@parallels.com> wrote:
> On Wed, Feb 18, 2015 at 05:18:38PM -0800, Andy Lutomirski wrote:
>> > > I don't suppose this could use real syscalls instead of netlink.  If
>> > > nothing else, netlink seems to conflate pid and net namespaces.
>> >
>> > What do you mean by "conflate pid and net namespaces"?
>>
>> A netlink socket is bound to a network namespace, but you should be
>> returning data specific to a pid namespace.
>
> Here is a good question. When we mount a procfs instance, the current
> pidns is saved on a superblock. Then if we read data from
> this procfs from another pidns, we will see pid-s from the pidns where
> this procfs has been mounted.
>
> $ unshare -p -- bash -c '(bash)'
> $ cat /proc/self/status | grep ^Pid:
> Pid:    15770
> $ echo $$
> 1
>
> A similar situation with socket_diag. A socket_diag socket is bound to a
> network namespace. If we open a socket_diag socket and change a network
> namespace, it will return infromation about the initial netns.
>
> In this version I always use a current pid namespace.
> But to be consistant with other kernel logic, a socket diag has to be
> linked with a pidns where it has been created.
>

Attaching a pidns to every freshly created netlink socket seems odd,
but I don't see a better solution that still uses netlink.

>>
>> On a related note, how does this interact with hidepid?  More
>
> Currently it always work as procfs with hidepid = 2 (highest level of
> security).
>
>> generally, what privileges are you requiring to obtain what data?
>
> It dumps information only if ptrace_may_access(tsk, PTRACE_MODE_READ) returns true

Sounds good to me.

>
>>
>> >
>> > >
>> > > Also, using an asynchronous interface (send, poll?, recv) for
>> > > something that's inherently synchronous (as the kernel a local
>> > > question) seems awkward to me.
>> >
>> > Actually all requests are handled synchronously. We call sendmsg to send
>> > a request and it is handled in this syscall.
>> >  2)               |  netlink_sendmsg() {
>> >  2)               |    netlink_unicast() {
>> >  2)               |      taskdiag_doit() {
>> >  2)   2.153 us    |        task_diag_fill();
>> >  2)               |        netlink_unicast() {
>> >  2)   0.185 us    |          netlink_attachskb();
>> >  2)   0.291 us    |          __netlink_sendskb();
>> >  2)   2.452 us    |        }
>> >  2) + 33.625 us   |      }
>> >  2) + 54.611 us   |    }
>> >  2) + 76.370 us   |  }
>> >  2)               |  netlink_recvmsg() {
>> >  2)   1.178 us    |    skb_recv_datagram();
>> >  2) + 46.953 us   |  }
>> >
>> > If we request information for a group of tasks (NLM_F_DUMP), a first
>> > portion of data is filled from the sendmsg syscall. And then when we read
>> > it, the kernel fills the next portion.
>> >
>> >  3)               |  netlink_sendmsg() {
>> >  3)               |    __netlink_dump_start() {
>> >  3)               |      netlink_dump() {
>> >  3)               |        taskdiag_dumpid() {
>> >  3)   0.685 us    |          task_diag_fill();
>> > ...
>> >  3)   0.224 us    |          task_diag_fill();
>> >  3) + 74.028 us   |        }
>> >  3) + 88.757 us   |      }
>> >  3) + 89.296 us   |    }
>> >  3) + 98.705 us   |  }
>> >  3)               |  netlink_recvmsg() {
>> >  3)               |    netlink_dump() {
>> >  3)               |      taskdiag_dumpid() {
>> >  3)   0.594 us    |        task_diag_fill();
>> > ...
>> >  3)   0.242 us    |        task_diag_fill();
>> >  3) + 60.634 us   |      }
>> >  3) + 72.803 us   |    }
>> >  3) + 88.005 us   |  }
>> >  3)               |  netlink_recvmsg() {
>> >  3)               |    netlink_dump() {
>> >  3)   2.403 us    |      taskdiag_dumpid();
>> >  3) + 26.236 us   |    }
>> >  3) + 40.522 us   |  }
>> >  0) + 20.407 us   |  netlink_recvmsg();
>> >
>> >
>> > netlink is really good for this type of tasks.  It allows to create an
>> > extendable interface which can be easy customized for different needs.
>> >
>> > I don't think that we would want to create another similar interface
>> > just to be independent from network subsystem.
>>
>> I guess this is a bit streamy in that you ask one question and get
>> multiple answers.
>
> It's like seq_file in procfs. The kernel allocates a buffer then fills
> it, copies it into userspace, fills it again, ... repeats these actions.
> And we can read data from file by portions.
>
> Actually here is one more analogy. When we open a file in procfs,
> we sends a request to the kernel and a file path is a request body in
> this case. But in case of procfs, we can't construct requests, we only
> have a set of predefined requests.

Fair enough.  Procfs is also a bit absurd and only makes sense because
it's compatible with lots of tools.  In a totally sane world, I would
argue that you should issue one syscall asking questions about a bit
and you should get answers immediately.

--Andy

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Arnaldo Carvalho de Melo]

Em Thu, Feb 19, 2015 at 05:00:12PM +0400, Pavel Odintsov escreveu:
> Hello!
> 
> In addition to my post I want to mention another issue related with
> slow /proc read in perf toolkit. On my server with 25 000 processes I
> need about ~15 minutes for loading perf top toolkit completely.
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=86991

Right, one way would be to, in the 'perf top' case to defer getting
thread information to when we need it, i.e. when we get a sample for a
pid that we have no struct thread associated with.

We would speed up 'perf top' startup but would introduce jitter down the
line, and would be up for races, but hey, we already are, using /proc
:-/

But that would not work for 'perf record', as we need to in advance
generate those records as we don't do any processing of samples...

Yeah, for preexisting threads we do have a problem since day one, what
we use is just what can be done with existing stuff.

I saw that there were some more messages in this thread, its just that I
haven't found them in my mailbox when David Ahern pointed this out this
discussion to me :-\

>From the subject line, there is patchkit, but I couldn't find it... Can
you resend it to me or point me to some url where I can get it?

- Arnaldo

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[David Ahern]

On 2/27/15 1:43 PM, Arnaldo Carvalho de Melo wrote:

>  From the subject line, there is patchkit, but I couldn't find it... Can
> you resend it to me or point me to some url where I can get it?

https://lkml.org/lkml/2015/2/17/64

Re: [PATCH 0/7] [RFC] kernel: add a netlink interface to get information about processes

[Arnaldo Carvalho de Melo]

Em Fri, Feb 27, 2015 at 01:54:03PM -0700, David Ahern escreveu:
> On 2/27/15 1:43 PM, Arnaldo Carvalho de Melo wrote:
> 
> > From the subject line, there is patchkit, but I couldn't find it... Can
> >you resend it to me or point me to some url where I can get it?
> 
> https://lkml.org/lkml/2015/2/17/64

Yeah, I eventually found it, this would be great for perf:

Another good feature of task_diag is an ability to request information
for a few processes. Currently here are two stratgies
TASK_DIAG_DUMP_ALL	- get information for all tasks
TASK_DIAG_DUMP_CHILDREN	- get information for children of a specified
			  tasks


I.e. 'perf record -a' would use that TASK_DIAG_DUMP_ALL to synthesize
PERF_RECORD_{FORK,COMM} events, we would still need some way to generate
the PERF_RECORD_MMAP entries tho.

- Arnaldo