* Re: Container mini-summit notes
[not found] ` <46DEB80B.2040106-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
@ 2007-09-05 14:38 ` Serge E. Hallyn
[not found] ` <20070905143849.GH1403-6s5zFf/epYLPQpwDFJZrxKsjOiXwFzmk@public.gmane.org>
2007-09-05 14:40 ` Container mini-summit notes Daniel Lezcano
` (2 subsequent siblings)
3 siblings, 1 reply; 7+ messages in thread
From: Serge E. Hallyn @ 2007-09-05 14:38 UTC (permalink / raw)
To: Cedric Le Goater; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Quoting Cedric Le Goater (clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org):
...
> Container mini-summit notes v0.01
>
> ======================================================================
>
> Held at Universty Arm's, Cambridge, UK
>
> on 3rd of September, 9h00 to 16h00 :
>
> Kir Kolyshkin <kir-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Masahiko Takahashi <masahiko-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
> Oren Laadan <orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org>
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
> James ??? (Google)
> ??? (NTT)
>
> On the phone (skype with very high noise level)
>
> Srivatsa Vaddagiri <vatsa-xthvdsQ13ZrQT0dZR+AlfA@public.gmane.org>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
>
> on 4th of September, 15h00 to 18h00 :
>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>
Boy I really missed out didn't I...
> = Namespace status
> ======================================================================
>
> * ipc
>
> extend to posix mqueue.
> check that /dev/mqueue can be mounted multiple times
> mqueue sysctls need a fix
> fs.mqueue.queues_max
> fs.mqueue.msg_max
> fs.mqueue.msgsize_max
>
>
> * uname namespace
>
> considered complete but what about setting the kernel version ?
I still say yuck. The kernel won't properly emulate the other kernel
version, so lying seems a horrible thing to do. If you want a vm with
another kernel, use qemu/kvm/xen.
> * user (to complete)
>
> useful today to current container technologies (userns, vserver)
> (uid, userns) checks must be completed to be integrate
> with filesystems
> security needs to be looked at
> so is signal delivery
>
> * pid namespace (in dev)
>
> stable.
>
> Current tasks are :
> . signal handling
> . pid_t cleanups
> remove any reference to task->pid
> keep ->pid in task struct only for performance
> complex ones:
> af_unix credentials
> file locks
> timer stat
> . kthread cleanup
> replace kernel_thread() by the kthread API
> change core kthread API to support signals
> nfs needs extra love. is someone working on it ?
REALLY would like to hear if anyone is.
> do we need hierarchical levels ?
>
>
> * net (in dev)
>
> veth is in dmiller's tree
> sysfs cleanups underway in greg's tree
> eric is working on a mininal patchset acceptable for netdev. will
> ask dmiller advice on the topic
>
> ip isolation could be done with netfilter or security hooks
And has by several people :)
> * device namespace (to do)
>
> we don't want to get rid of mknod() but we also want to limit the
> view of the devices in a container. one way to do this is through a
> device namespace which would only expose a 'white list' of devices
> when unshared. a possible white list is :
>
> /dev/null
> /dev/full
> /dev/zero
> /dev/rtc
> /dev/random
> /dev/pts/*
>
> do we require a extra namespace for /dev/pts/* to handle its
> virtualization or can this be done directly in a device namespace ?
How the hell does a separate device ns protect you from mknod?
> check that /dev/pts can be mounted multiple times.
I previously sent a patch to implement CLONE_NEWPTS, which just
intercepts /dev/pts/ access without multiple mounts. I think that,
plus access to child containers' devpts mounts through the nsproxy
container subsystem, would be a very simple and useful way to
implement this.
Then there is also the array of console devices to turn into a
namespace.
And finally there is the issue of actualy virtual devices. So I want to
start up a virtual server, and I want it's serial port to be connected
to a virtual tty which buffers the last page or few pages of data so I
can connect to the server and see what's going on. I'm not too savvy
here - can we do that now, maybe just through a fake serial device?
> * time (to do)
>
> keep the monotonic timers from expiring when you restart
> required for C/R but will only make sense in "close" environment
>
> * other possible namespace ?
>
> rtc ? which an isolation problem and also a sysctl issue
>
> comment from eric :
> redesign of lsm (a la netfilter) could cover all
> isolation needs.
I'd like to talk to Eric and find out more precisely what he is
thinking.
Note that one issue with using lsm is that it will purely control
access, not resource ids. So container 1 may not have access to
/dev/pts/2, which at least is safe, but it'll see everyone's devices
under devpts.
But yes, this is why I've been saying that userns is useful at the
moment. You should be able to write simple selinux policies to fully
container a userns.
> * namespace management
>
> . entering
> no consensus on how this should be should be done.
>
> probably because the need is related to a container and not just
> namespaces. it should be solved with a container object and
> probably a subsystem.
>
> serge's proposal of sys_hijack() is interesting but will require
> more study because, in UNIX, it's not natural for a child process
> to have 2 parents !
>
> . extending clone to support more flags
>
> new syscall proposal for a clone2(struct clone2_arg_struct* args)
>
> * tests
>
> . ltp for unit
> . keep the integration tests in each container framework.
>
> * Filesystems
>
> . unprivilege mounts (not addressed)
>
> merged
>
> . multiple /sys mounts (in dev)
>
> missing some bits (eric working on it)
>
> . multiple /proc mounts (to complete)
>
> multiple mount done
> to limit access to /proc files, use the user namespace checks ?
> for the contents of each file, use the current context
Just might work...
It kind of looks like having a 'initial' userns which is special-cased,
but I assume the idea is that /proc/ files would be created at boot in
the initial userns, so straightforward checks for userns equivalence
between the writing process and the file would suffice.
> * Console
>
> . a running getty should be covered by tty namespace
I'm not understanding this line.
Is there a fully tty namespace design that was discussed, which this is
summarizing?
> . printk will require some support to be isolated.
>
> = Task Container (from container dev plan)
> ======================================================================
>
> * base features
>
> hierarchical/virtualized containers
> support vserver mgmnt of sub-containers
> locking cleanup
> control file API simplification
> unified container including namespaces
>
> * userpace RBCE to provide controls for
>
> users
> groups
> pgrp
> executable
>
> * specific containers targeted:
>
> split cpusets into
> cpuset
> memset
> network
> connect/bind/accept controller using iptables
>
> controllers :
>
> memory controller (see detail below)
> cpu controller (see detailbelow)
> io controller (see detail below)
> network flow id control
> per-container OOM handler (userspace)
> per-container swap
> per-container disk I/O scheduling
> per container memory reclaim
> per container dirty page (write throttling) limit.
> network rate limiting (outbound) based on container
>
> * misc
>
> User level APIS to identify the resource limits that is allowed to a
> job, for example, how much physical memory a process can use. This
> should seamlessly integrated with non-container environment as well
> (may be with ulimit).
>
> Per container stats, like pages on active list, cpus usage, etc
>
> = Resource Management (from container dev plan)
> ======================================================================
>
> * memory controller
>
> users and requirements:
>
> 1. The containers solution would need resource management
> (including memory control and per container swap files). Paul
> Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have
> all shown interest in the memory controller patches.
>
> 2. The memory controller can account for page cache as well, all
> people interested in limiting page cahce control, can
> theoratically put move all page cache hungry applications under
> the same container.
>
> Planned enhancements to the memory controller
> 1. Improved shared page accounting
> 2. Improved statistics
> 3. Soft-limit memory usage
>
> generic infrastructure work:
> 1. Enhancing containerstats
> a. Working on per controller statistics
> b. Integrating taskstats with containerstats
> 2. CPU accounting framework
> a. Migrate the accounting to be more precis
>
> * cpu controller
>
> users and requirements:
>
> 1. Virtualization solutions like containers and KVM need CPU
> control. KVM for example would like to have both limits and
> guarantees supported by a CPU controller, to control CPU
> allocation to a particular instance.
> 2. Workload management products would like to exploit this for
> providing guaranteed cpu bandwidth and also (hard/soft)
> limiting cpu usage.
>
> work items
> 1. Fine-grained proportional-share fair-group scheduling.
> 2. More accurate SMP fairness
> 3. Hard limit
> 4. SCHED_FIFO type policy for groups
> 5. Improved statistics and debug facility for group scheduler
>
> * io controller
>
> users and requirements:
>
> 1. At a talk presented to the Linux Foundation (OSDL), the
> attendees showed interest in an IO controller to control IO
> bandwidth of various filesystem operations (backup,
> journalling, etc)
>
> work items:
> 1. Proof of concept IO controller and community discussion/feedback
> 2. Development and Integration of the IO controller with containers
>
> open issues
> 1. Automatic tagging/resource classification engine
>
> = Checkpoint/Restart
> ======================================================================
>
> * need to unified the freezer to reach a quiescence point
>
> * overall strategy :
> . checkpoint: in kernel
> . restart : first recreate process tree then let each
> process restart itself
The data transfer method is pretty important to sort out, so that people
can start writing patches to c/r various resources.
I know you've discussed having some sort of 'crfs'. I gather you would
then do a checkpoint by doing something like
cp /mnt/crfs/vserver1 /opt/checkpoints/vserver1/`date +%F`/`date +%R`
I actually like that, so long as it's that, and not actually having the
cr subsystem write the data straight to a regular file.
Another option would be to use relayfs for the checkpoint data dump,
since checkpoint and restart will be assymetrical in nature anyway.
> * possible direction for C/R user api
> . checkpoint/restart syscalls
> . C/R file systems
> solves the set id issue
> elegant but exposes too much the ABI
>
> example :
>
> .
> |-- 0x00003002
> | |-- 0x00003002
> | | |-- attr
> | | |-- signal
> | | |-- signal.altstack
> | | |-- signal.pending
> | | |-- thread
> | | |-- thread.frame
> | | |-- timers
> | | |-- tls
> | | `-- wait.zombies
> | |-- aio
> | |-- attr
> | |-- fds
> | |-- ldt
> | |-- mem.segments
> | |-- numa
> | |-- process
> | |-- signal.action
> | |-- signal.pending
> | |-- sysv.semadj
> | |-- sysv.shmcount
> | `-- thread.list
> |-- af_inet_listening
> |-- af_inet_orphan_count
> |-- af_inet_orphan_data
> |-- af_inet_orphan_info
> |-- files
> | |-- 0
> | |-- 1
> | |-- 10137663680
> | |-- 1014250cdc0
> | |-- 2
> | `-- stdios
> |-- sysv.msq
> |-- sysv.sem
> `-- sysv.shm
>
> * memory C/R
>
> critical for performance
> per-container swapfile ?
Absolutely. per-container swapfile, and memory checkpoint forces write
to swap and writes the pte data out to relayfs or a crfs file ready for
copy.
One tricky issue then is that we really want to do a copy-on-write copy
of the swapfile. We don't want to wait for the swapfile to be fully
copied before restarting the process. Are any of the fancy new
filesystems implementing that? Or are we gonna be using zfs over fuse
for our per-container swapfiles? :)
> * subsystem C/R API.
>
> keep it on the side for the moment <subsys>_cr.c to identify the
> needs of each subsystem before asking the maintainer's comments
It would be great if we could provide a data write/read api so these
subsystems can all be written without worrying about which method we'll
use for dumping checkpoint data and pushing restart data.
> possible cr_ops in some objects (like for network protocols) but
> also ops 'a la' virt_ops to prepare for different C/R strategy :
> brutal, incremental, live migration
Hmm.
> * setting id back to what they where
>
> possible global syscall to set ids of pid,ipc,pts.
> else use the C/R fs
>
> * statefile format
>
> no big issues. let's pick one.
>
> * optimization
>
> parallel C/R
>
>
>
>
>
>
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Container mini-summit notes
[not found] ` <46DEB80B.2040106-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
2007-09-05 14:38 ` Serge E. Hallyn
@ 2007-09-05 14:40 ` Daniel Lezcano
2007-09-05 15:48 ` Balbir Singh
2007-10-01 14:29 ` Cedric Le Goater
3 siblings, 0 replies; 7+ messages in thread
From: Daniel Lezcano @ 2007-09-05 14:40 UTC (permalink / raw)
To: Cedric Le Goater; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Cedric Le Goater wrote:
> Held at De Vere Universty Arms Hotel, Cambridge, UK
>
> * Monday, Sept 3, 9h00 to 16h00 :
>
> Kir Kolyshkin <kir-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Masahiko Takahashi <masahiko-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
> Oren Laadan <orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org>
> James Youngman <youngman-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> ??? (NTT)
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
>
> On the phone (skype with very high noise level)
>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Srivatsa Vaddagiri <vatsa-xthvdsQ13ZrQT0dZR+AlfA@public.gmane.org>
> Dhaval Giani <dhaval-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> Vaidyanathan Srinivasan <svaidy-xthvdsQ13ZrQT0dZR+AlfA@public.gmane.org>
>
> * Tuesday, Sept 4, 15h00 to 18h00 :
>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
>
> = Namespace status
> ======================================================================
>
> * sysv ipc
>
> extend to posix mqueue.
> . check that /dev/mqueue can be mounted multiple times
> . mqueue sysctls will need a fix :
> fs.mqueue.queues_max
> fs.mqueue.msg_max
> fs.mqueue.msgsize_max
>
>
> * uname namespace
>
> considered complete.
>
> what about being able to set the kernel version ?
>
> * user
>
> useful today to current container technologies (openvz, vserver)
>
> uid checks should be replaced by (uid, userns) to complete
> integration with filesystems
> security needs to be looked at
> so is signal delivery
>
> * pid namespace
>
> in dev
>
> signal handling completion underway
> pid_t cleanups
> . the purpose is to remove any explicit reference to
> task->pid
> . keep ->pid in task struct only for performance
> . complex cleanups ones:
> af_unix credentials
> file locks
> timer stat
>
> kthread cleanup
> . replace kernel_thread() by the kthread API
> . change core kthread API to support signals
> . then nfs needs extra love. is someone working on it ?
>
> do we need hierarchical levels ?
>
>
> * net
>
> in dev
>
> veth is in dmiller's tree
> sysfs cleanups underway in greg's tree
> eric is working on a mininal patchset acceptable for netdev. will
> ask dmiller advice on the topic
>
> ip isolation could be done with netfilter or security hooks
>
> * device namespace
>
> to do
>
> we don't want to get rid of mknod() but we also want to limit the
> view of the devices in a container. one way to do this is through a
> device namespace which would only expose a 'white list' of devices
> when unshared. a possible 'white list' is :
>
> /dev/null
> /dev/full
> /dev/zero
> /dev/rtc
> /dev/random
> /dev/pts/*
>
> do we require a extra namespace for /dev/pts/* to handle its
> virtualization or can this be done directly in the device namespace ?
>
> check that /dev/pts can be mounted multiple times.
>
> * time
>
> to do
>
> required for C/R
> will only make sense in a "closed" environment
> the purpose is to keep the monotonic timers from expiring when
> you restart
>
> * other possible namespace ?
>
> rtc ? which is an isolation issue and also a sysctl issue
>
> comment from eric :
> a redesign of lsm, a la netfilter, could cover all
> isolation needs.
>
> * namespace management
>
>
> . entering
>
> no consensus on how this should be done.
>
> probably because the need is related to a container and not just
> namespaces. it should be solved with a container object and
> probably a subsystem.
>
> serge's proposal of sys_hijack() is interesting but will require
> more study because, in UNIX, it's not natural for a child process
> to have 2 parents !
>
> . extending clone to support more flags
>
> new syscall proposal for a clone2(struct clone2_arg_struct* args)
>
> * tests
>
> . ltp for unit
> . keep the integration tests in each container framework.
>
> * Filesystems
>
> . unprivilege mounts (not addressed)
>
> merged
>
> . multiple /sys mounts (in dev)
>
> missing some bits (eric working on it) to decouple sysfs and
> kobjects
>
> . multiple /proc mounts (to complete)
>
> multiple mount done
> to limit access to /proc files, use the user namespace checks ?
> for the contents of each file, use the current context to identify
> namespace
>
> * Console
>
> . a running getty should be covered by tty namespace
> . printk will require some support to be isolated.
>
> = Task Container (from container dev plan)
> ======================================================================
>
> * base features
>
> hierarchical/virtualized containers
> support vserver mgmnt of sub-containers
> locking cleanup
> control file API simplification
> unified container including namespaces
>
> the "container"/"task container" name is ambiguous and it should change to
> "control group"
>
>
> * userpace RBCE to provide controls for
>
> users
> groups
> pgrp
> executable
>
> * specific containers targeted:
>
> split cpusets into
> cpuset
> memset
> network
> connect/bind/accept controller using iptables
Just a comment here. The hooks needed for that are exactly the same as
the security hooks, IMHO, iptables is not right subsystem to use to
catch socket calls.
>
> controllers :
>
> memory controller (see detail below)
>
> cpu controller
>
> Status:
> - Extensions required to CFS core for supporting
> group-scheduling aspects are mostly there (in
> mainline)
>
> Todo:
> - Better SMP group-fairness
> - Hard-limit cpu usage
> - SCHED_FIFO like policy for groups
> - Group priorities (?)
>
> io controller (see detail below)
>
> network flow id control
I'm not sure I get that.
.
>
> per-container OOM handler (userspace)
>
> per-container swap
>
> per-container disk I/O scheduling
>
> per container memory reclaim
>
> per container dirty page (write throttling) limit.
>
> network rate limiting (outbound) based on container
I am not sure I get that too.
As far as I understand, if the container has its own network with a
network namespace, then the rate limiting already exists with tc for
outgoing __and__ incoming traffic. If the container has a simple IP
isolation, the rate limiting can be set with tc again but only for
outgoing traffic.
If the container share the network with other container, I understand
the reason for a network flow id. But in this case, how to handle
incoming traffic ?
IHMO, rate limiting should be handled in conjunction with a network
namespace.
>
> * misc
>
> User level APIS to identify the resource limits that is allowed to a
> job, for example, how much physical memory a process can use. This
> should seamlessly integrated with non-container environment as well
> (may be with ulimit).
>
> Per container stats, like pages on active list, cpus usage, etc
>
> = Resource Management (from container dev plan)
> ======================================================================
>
> * memory controller
>
> users and requirements:
>
> 1. The containers solution would need resource management
> (including memory control and per container swap files). Paul
> Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have
> all shown interest in the memory controller patches.
>
> 2. The memory controller can account for page cache as well, all
> people interested in limiting page cahce control, can
> theoratically put move all page cache hungry applications under
> the same container.
>
> Planned enhancements to the memory controller
> 1. Improved shared page accounting
> 2. Improved statistics
> 3. Soft-limit memory usage
>
> generic infrastructure work:
> 1. Enhancing containerstats
> a. Working on per controller statistics
> b. Integrating taskstats with containerstats
> 2. CPU accounting framework
> a. Migrate the accounting to be more precis
>
> * cpu controller
>
> users and requirements:
>
> 1. Virtualization solutions like containers and KVM need CPU
> control. KVM for example would like to have both limits and
> guarantees supported by a CPU controller, to control CPU
> allocation to a particular instance.
> 2. Workload management products would like to exploit this for
> providing guaranteed cpu bandwidth and also (hard/soft)
> limiting cpu usage.
>
> work items
> 1. Fine-grained proportional-share fair-group scheduling.
> 2. More accurate SMP fairness
> 3. Hard limit
> 4. SCHED_FIFO type policy for groups
> 5. Improved statistics and debug facility for group scheduler
>
> * io controller
>
> users and requirements:
>
> 1. At a talk presented to the Linux Foundation (OSDL), the
> attendees showed interest in an IO controller to control IO
> bandwidth of various filesystem operations (backup,
> journalling, etc)
>
> work items:
> 1. Proof of concept IO controller and community discussion/feedback
> 2. Development and Integration of the IO controller with containers
>
> open issues
> 1. Automatic tagging/resource classification engine
>
> = Checkpoint/Restart
> ======================================================================
>
> * need to unified the freezer to reach a quiescence point
>
> * overall strategy :
> . checkpoint: in kernel
> . restart : first recreate process tree then let each
> process restart itself
>
> * possible direction for C/R user api
> . checkpoint/restart syscalls
> . C/R file systems
> solves the set id issue
> elegant but exposes too much the ABI
>
> example :
>
> .
> |-- 0x00003002
> | |-- 0x00003002
> | | |-- attr
> | | |-- signal
> | | |-- signal.altstack
> | | |-- signal.pending
> | | |-- thread
> | | |-- thread.frame
> | | |-- timers
> | | |-- tls
> | | `-- wait.zombies
> | |-- aio
> | |-- attr
> | |-- fds
> | |-- ldt
> | |-- mem.segments
> | |-- numa
> | |-- process
> | |-- signal.action
> | |-- signal.pending
> | |-- sysv.semadj
> | |-- sysv.shmcount
> | `-- thread.list
> |-- af_inet_listening
> |-- af_inet_orphan_count
> |-- af_inet_orphan_data
> |-- af_inet_orphan_info
> |-- files
> | |-- 0
> | |-- 1
> | |-- 10137663680
> | |-- 1014250cdc0
> | |-- 2
> | `-- stdios
> |-- sysv.msq
> |-- sysv.sem
> `-- sysv.shm
>
> * memory C/R
>
> critical for performance
> per-container swapfile ?
>
> * subsystem C/R API.
>
> keep it on the side for the moment <subsys>_cr.c to identify the
> needs of each subsystem before asking the maintainer's comments
>
> possible cr_ops in some objects (like for network protocols) but
> also ops 'a la' virt_ops to prepare for different C/R strategy :
> brutal, incremental, live migration
>
> * setting id back to what they where
>
> possible global syscall to set ids of pid,ipc,pts.
> else use the C/R fs
>
> * statefile format
>
> no big issues. let's pick one.
>
> * optimization
>
> parallel C/R
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Container mini-summit notes
[not found] ` <46DEB80B.2040106-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
2007-09-05 14:38 ` Serge E. Hallyn
2007-09-05 14:40 ` Container mini-summit notes Daniel Lezcano
@ 2007-09-05 15:48 ` Balbir Singh
2007-10-01 14:29 ` Cedric Le Goater
3 siblings, 0 replies; 7+ messages in thread
From: Balbir Singh @ 2007-09-05 15:48 UTC (permalink / raw)
To: Cedric Le Goater; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Cedric Le Goater wrote:
> Held at De Vere Universty Arms Hotel, Cambridge, UK
>
> * Monday, Sept 3, 9h00 to 16h00 :
>
> Kir Kolyshkin <kir-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Masahiko Takahashi <masahiko-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
> Oren Laadan <orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org>
> James Youngman <youngman-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> ??? (NTT)
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
>
> On the phone (skype with very high noise level)
>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Srivatsa Vaddagiri <vatsa-xthvdsQ13ZrQT0dZR+AlfA@public.gmane.org>
> Dhaval Giani <dhaval-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org>
> Vaidyanathan Srinivasan <svaidy-xthvdsQ13ZrQT0dZR+AlfA@public.gmane.org>
>
> * Tuesday, Sept 4, 15h00 to 18h00 :
>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Paul Menage <menage-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> Eric W. Biederman <ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org>
> Cedric Le Goater <clg-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
>
> = Namespace status
> ======================================================================
>
> * sysv ipc
>
> extend to posix mqueue.
> . check that /dev/mqueue can be mounted multiple times
> . mqueue sysctls will need a fix :
> fs.mqueue.queues_max
> fs.mqueue.msg_max
> fs.mqueue.msgsize_max
>
>
> * uname namespace
>
> considered complete.
>
> what about being able to set the kernel version ?
>
> * user
>
> useful today to current container technologies (openvz, vserver)
>
> uid checks should be replaced by (uid, userns) to complete
> integration with filesystems
> security needs to be looked at
> so is signal delivery
>
> * pid namespace
>
> in dev
>
> signal handling completion underway
> pid_t cleanups
> . the purpose is to remove any explicit reference to
> task->pid
> . keep ->pid in task struct only for performance
> . complex cleanups ones:
> af_unix credentials
> file locks
> timer stat
>
> kthread cleanup
> . replace kernel_thread() by the kthread API
> . change core kthread API to support signals
> . then nfs needs extra love. is someone working on it ?
>
> do we need hierarchical levels ?
>
>
> * net
>
> in dev
>
> veth is in dmiller's tree
> sysfs cleanups underway in greg's tree
> eric is working on a mininal patchset acceptable for netdev. will
> ask dmiller advice on the topic
>
> ip isolation could be done with netfilter or security hooks
>
> * device namespace
>
> to do
>
> we don't want to get rid of mknod() but we also want to limit the
> view of the devices in a container. one way to do this is through a
> device namespace which would only expose a 'white list' of devices
> when unshared. a possible 'white list' is :
>
> /dev/null
> /dev/full
> /dev/zero
> /dev/rtc
> /dev/random
> /dev/pts/*
>
> do we require a extra namespace for /dev/pts/* to handle its
> virtualization or can this be done directly in the device namespace ?
>
> check that /dev/pts can be mounted multiple times.
>
> * time
>
> to do
>
> required for C/R
> will only make sense in a "closed" environment
> the purpose is to keep the monotonic timers from expiring when
> you restart
>
> * other possible namespace ?
>
> rtc ? which is an isolation issue and also a sysctl issue
>
> comment from eric :
> a redesign of lsm, a la netfilter, could cover all
> isolation needs.
>
> * namespace management
>
>
> . entering
>
> no consensus on how this should be done.
>
> probably because the need is related to a container and not just
> namespaces. it should be solved with a container object and
> probably a subsystem.
>
> serge's proposal of sys_hijack() is interesting but will require
> more study because, in UNIX, it's not natural for a child process
> to have 2 parents !
>
> . extending clone to support more flags
>
> new syscall proposal for a clone2(struct clone2_arg_struct* args)
>
> * tests
>
> . ltp for unit
> . keep the integration tests in each container framework.
>
> * Filesystems
>
> . unprivilege mounts (not addressed)
>
> merged
>
> . multiple /sys mounts (in dev)
>
> missing some bits (eric working on it) to decouple sysfs and
> kobjects
>
> . multiple /proc mounts (to complete)
>
> multiple mount done
> to limit access to /proc files, use the user namespace checks ?
> for the contents of each file, use the current context to identify
> namespace
>
> * Console
>
> . a running getty should be covered by tty namespace
> . printk will require some support to be isolated.
>
> = Task Container (from container dev plan)
> ======================================================================
>
> * base features
>
> hierarchical/virtualized containers
> support vserver mgmnt of sub-containers
> locking cleanup
> control file API simplification
> unified container including namespaces
>
> the "container"/"task container" name is ambiguous and it should change to
> "control group"
>
>
> * userpace RBCE to provide controls for
>
> users
> groups
> pgrp
> executable
>
> * specific containers targeted:
>
> split cpusets into
> cpuset
> memset
> network
> connect/bind/accept controller using iptables
>
> controllers :
>
> memory controller (see detail below)
>
The status is that the patches for memory control are in -mm. We need to
stabilize the patches, look at reducing the overhead of double LRU for
root containers. We need to explore the double LRU approach as a longer
term solution.
> cpu controller
>
> Status:
> - Extensions required to CFS core for supporting
> group-scheduling aspects are mostly there (in
> mainline)
>
> Todo:
> - Better SMP group-fairness
> - Hard-limit cpu usage
> - SCHED_FIFO like policy for groups
> - Group priorities (?)
>
> io controller (see detail below)
>
> network flow id control
>
> per-container OOM handler (userspace)
>
> per-container swap
>
> per-container disk I/O scheduling
>
> per container memory reclaim
>
> per container dirty page (write throttling) limit.
>
> network rate limiting (outbound) based on container
>
> * misc
>
> User level APIS to identify the resource limits that is allowed to a
> job, for example, how much physical memory a process can use. This
> should seamlessly integrated with non-container environment as well
> (may be with ulimit).
>
> Per container stats, like pages on active list, cpus usage, etc
>
> = Resource Management (from container dev plan)
> ======================================================================
>
> * memory controller
>
> users and requirements:
>
> 1. The containers solution would need resource management
> (including memory control and per container swap files). Paul
> Menage, YAMOMOTO Takshi, Peter Zijlstra, Pavel Emelianov have
> all shown interest in the memory controller patches.
>
> 2. The memory controller can account for page cache as well, all
> people interested in limiting page cahce control, can
> theoratically put move all page cache hungry applications under
> the same container.
>
> Planned enhancements to the memory controller
> 1. Improved shared page accounting
> 2. Improved statistics
> 3. Soft-limit memory usage
>
> generic infrastructure work:
> 1. Enhancing containerstats
> a. Working on per controller statistics
> b. Integrating taskstats with containerstats
> 2. CPU accounting framework
> a. Migrate the accounting to be more precis
>
> * cpu controller
>
> users and requirements:
>
> 1. Virtualization solutions like containers and KVM need CPU
> control. KVM for example would like to have both limits and
> guarantees supported by a CPU controller, to control CPU
> allocation to a particular instance.
> 2. Workload management products would like to exploit this for
> providing guaranteed cpu bandwidth and also (hard/soft)
> limiting cpu usage.
>
> work items
> 1. Fine-grained proportional-share fair-group scheduling.
> 2. More accurate SMP fairness
> 3. Hard limit
> 4. SCHED_FIFO type policy for groups
> 5. Improved statistics and debug facility for group scheduler
>
> * io controller
>
> users and requirements:
>
> 1. At a talk presented to the Linux Foundation (OSDL), the
> attendees showed interest in an IO controller to control IO
> bandwidth of various filesystem operations (backup,
> journalling, etc)
>
> work items:
> 1. Proof of concept IO controller and community discussion/feedback
> 2. Development and Integration of the IO controller with containers
>
> open issues
> 1. Automatic tagging/resource classification engine
>
> = Checkpoint/Restart
> ======================================================================
>
> * need to unified the freezer to reach a quiescence point
>
> * overall strategy :
> . checkpoint: in kernel
> . restart : first recreate process tree then let each
> process restart itself
>
> * possible direction for C/R user api
> . checkpoint/restart syscalls
> . C/R file systems
> solves the set id issue
> elegant but exposes too much the ABI
>
> example :
>
> .
> |-- 0x00003002
> | |-- 0x00003002
> | | |-- attr
> | | |-- signal
> | | |-- signal.altstack
> | | |-- signal.pending
> | | |-- thread
> | | |-- thread.frame
> | | |-- timers
> | | |-- tls
> | | `-- wait.zombies
> | |-- aio
> | |-- attr
> | |-- fds
> | |-- ldt
> | |-- mem.segments
> | |-- numa
> | |-- process
> | |-- signal.action
> | |-- signal.pending
> | |-- sysv.semadj
> | |-- sysv.shmcount
> | `-- thread.list
> |-- af_inet_listening
> |-- af_inet_orphan_count
> |-- af_inet_orphan_data
> |-- af_inet_orphan_info
> |-- files
> | |-- 0
> | |-- 1
> | |-- 10137663680
> | |-- 1014250cdc0
> | |-- 2
> | `-- stdios
> |-- sysv.msq
> |-- sysv.sem
> `-- sysv.shm
>
> * memory C/R
>
> critical for performance
> per-container swapfile ?
>
> * subsystem C/R API.
>
> keep it on the side for the moment <subsys>_cr.c to identify the
> needs of each subsystem before asking the maintainer's comments
>
> possible cr_ops in some objects (like for network protocols) but
> also ops 'a la' virt_ops to prepare for different C/R strategy :
> brutal, incremental, live migration
>
> * setting id back to what they where
>
> possible global syscall to set ids of pid,ipc,pts.
> else use the C/R fs
>
> * statefile format
>
> no big issues. let's pick one.
>
> * optimization
>
> parallel C/R
>
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linux-foundation.org/mailman/listinfo/containers
--
Warm Regards,
Balbir Singh
Linux Technology Center
IBM, ISTL
^ permalink raw reply [flat|nested] 7+ messages in thread* Re: Container mini-summit notes
[not found] ` <46DEB80B.2040106-NmTC/0ZBporQT0dZR+AlfA@public.gmane.org>
` (2 preceding siblings ...)
2007-09-05 15:48 ` Balbir Singh
@ 2007-10-01 14:29 ` Cedric Le Goater
3 siblings, 0 replies; 7+ messages in thread
From: Cedric Le Goater @ 2007-10-01 14:29 UTC (permalink / raw)
To: Cedric Le Goater; +Cc: containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA
Cedric Le Goater wrote:
> Held at De Vere Universty Arms Hotel, Cambridge, UK
>
> * Monday, Sept 3, 9h00 to 16h00 :
>
> Kir Kolyshkin <kir-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Pavel Emelianov <xemul-GEFAQzZX7r8dnm+yROfE0A@public.gmane.org>
> Masahiko Takahashi <masahiko-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org>
> Oren Laadan <orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org>
> James Youngman <youngman-hpIqsD4AKlfQT0dZR+AlfA@public.gmane.org>
> ??? (NTT)
Fernando Luis Vazquez Cao (NTT)
It took me a while. Sorry !
C.
^ permalink raw reply [flat|nested] 7+ messages in thread