[ANNOUNCE] first stable release of OpenVZ kernel virtualization solution

[ANNOUNCE] first stable release of OpenVZ kernel virtualization solution

[Kirill Korotaev]

Hello,

We are happy to announce the release of a stable version of the OpenVZ 
software, located at http://openvz.org/.

OpenVZ is a kernel virtualization solution which can be considered as a
natural step in the OS kernel evolution: after multiuser and 
multitasking functionality there comes an OpenVZ feature of having 
multiple environments.

Virtualization lets you divide a system into separate isolated
execution environments (called VPSs - Virtual Private Servers). From the
point of view of the VPS owner (root), it looks like a stand-alone 
server. Each VPS has its own filesystem tree, process tree (starting 
from init as in a real system) and so on. The  single-kernel approach 
makes it possible to virtualize with very little overhead, if any.

OpenVZ in-kernel modifications can be divided into several components:

1. Virtualization and isolation.
Many Linux kernel subsystems are virtualized, so each VPS has its own:
- process tree (featuring virtualized pids, so that the init pid is 1);
- filesystems (including virtualized /proc and /sys);
- network (virtual network device, its own ip addresses,
   set of netfilter and routing rules);
- devices (if needed, any VPS can be granted access to real devices
   like network interfaces, serial ports, disk partitions, etc);
- IPC objects.

2. Resource Management.
This subsystem enables multiple VPSs to coexist, providing managed
resource sharing and limiting.
- User Beancounters is a set of per-VPS resource counters, limits,
   and guarantees (kernel memory, network buffers, phys pages, etc.).
- Fair CPU scheduler (SFQ with shares and hard limits).
- Two-level disk quota (first-level: per-VPS quota;
   second-level: ordinary user/group quota inside a VPS)

Resource management is what makes OpenVZ different from other solutions
of this kind (like Linux VServer or FreeBSD jails). There are a few
resources that can be abused from inside a VPS (such as files, IPC
objects, ...) leading to a DoS attack. User Beancounters prevent such
abuses.

As virtualization solution OpenVZ makes it possible to do the same 
things for which people use UML, Xen, QEmu or VMware, but there are 
differences:
(a) there is no ability to run other operating systems
     (although different Linux distros can happily coexist);
(b) performance loss is negligible due to absense of any kind of
     emulation;
(c) resource utilization is much better.

The last point needs to be elaborated on. OpenVZ allows to utilize
system resources such as memory and disk space very efficiently, and
because of that has better performance on memory-critical workloads.
OpenVZ does not run separate kernels in each VPS and saves memory on
kernel internal data. However, even bigger efficiency of OpenVZ comes
from dynamic resource allocation.

With other virtualization solutions, you need to specify in advance the
amount of memory for each virtual machine and create a disk device and
filesystem for it, and the possibilities to change settings later on the
fly are very limited.

The dynamic assignment of resources in OpenVZ can significantly improve 
their utilization. For example, a x86_64 box (2.8 GHz Celeron D, 1GB 
RAM) is capable to run 100 VPSs with a fairly high performance (VPSs 
were serving http requests for 4.2Kb static pages at an overall rate of 
more than 80,000 req/min). Each VPS (running CentOS 4 x86_64) had the 
following set of processes:

[root@ovz-x64 ~]# vzctl exec 1043 ps axf
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 init
11830 ?        Ss     0:00 syslogd -m 0
11897 ?        Ss     0:00 /usr/sbin/sshd
11943 ?        Ss     0:00 xinetd -stayalive -pidfile ...
12218 ?        Ss     0:00 sendmail: accepting connections
12265 ?        Ss     0:00 sendmail: Queue runner@01:00:00
13362 ?        Ss     0:00 /usr/sbin/httpd
13363 ?        S      0:00  \_ /usr/sbin/httpd
13364 ?        S      0:00  \_ /usr/sbin/httpd
13365 ?        S      0:00  \_ /usr/sbin/httpd
13366 ?        S      0:00  \_ /usr/sbin/httpd
13370 ?        S      0:00  \_ /usr/sbin/httpd	
13371 ?        S      0:00  \_ /usr/sbin/httpd
13372 ?        S      0:00  \_ /usr/sbin/httpd
13373 ?        S      0:00  \_ /usr/sbin/httpd
6416 ?        Rs     0:00 ps axf

And the list of running VPSs:

[root@ovz-x64 ~]# vzlist
     VPSID      NPROC STATUS  IP_ADDR         HOSTNAME
      1001         15 running 10.1.1.1        vps1001
      1002         15 running 10.1.1.2        vps1002
      [....skipped....]
      1099         15 running 10.1.1.99       vps1099
      1100         15 running 10.1.1.100      vps1100

On the box with 4Gb of RAM one can expect 400 of such VPSs to run 
without much troubles.

More information is available at http://openvz.org/

Thanks,
OpenVZ team.

Re: [ANNOUNCE] first stable release of OpenVZ kernel virtualization solution

[Andrew Morton]

Kirill Korotaev <dev@sw.ru> wrote:
>
> Hello,
> 
> We are happy to announce the release of a stable version of the OpenVZ 
> software, located at http://openvz.org/.
> 
> OpenVZ is a kernel virtualization solution which can be considered as a
> natural step in the OS kernel evolution: after multiuser and 
> multitasking functionality there comes an OpenVZ feature of having 
> multiple environments.

Are you able to give us a high-level overview of how it actually is
implemented?  IOW: what does the patch do?

> ...
> 
> As virtualization solution OpenVZ makes it possible to do the same 
> things for which people use UML, Xen, QEmu or VMware, but there are 
> differences:
> (a) there is no ability to run other operating systems
>      (although different Linux distros can happily coexist);
> (b) performance loss is negligible due to absense of any kind of
>      emulation;
> (c) resource utilization is much better.

What are OpenVZ's disadvantages wrt the above?

> The dynamic assignment of resources in OpenVZ can significantly improve 
> their utilization. For example, a x86_64 box (2.8 GHz Celeron D, 1GB 
> RAM) is capable to run 100 VPSs with a fairly high performance (VPSs 
> were serving http requests for 4.2Kb static pages at an overall rate of 
> more than 80,000 req/min). Each VPS (running CentOS 4 x86_64) had the 
> following set of processes:
> 
> [root@ovz-x64 ~]# vzctl exec 1043 ps axf
>   PID TTY      STAT   TIME COMMAND
>     1 ?        Ss     0:00 init
> 11830 ?        Ss     0:00 syslogd -m 0
> 11897 ?        Ss     0:00 /usr/sbin/sshd
> 11943 ?        Ss     0:00 xinetd -stayalive -pidfile ...
> 12218 ?        Ss     0:00 sendmail: accepting connections
> 12265 ?        Ss     0:00 sendmail: Queue runner@01:00:00
> 13362 ?        Ss     0:00 /usr/sbin/httpd
> 13363 ?        S      0:00  \_ /usr/sbin/httpd
> 13364 ?        S      0:00  \_ /usr/sbin/httpd
> 13365 ?        S      0:00  \_ /usr/sbin/httpd
> 13366 ?        S      0:00  \_ /usr/sbin/httpd
> 13370 ?        S      0:00  \_ /usr/sbin/httpd	
> 13371 ?        S      0:00  \_ /usr/sbin/httpd
> 13372 ?        S      0:00  \_ /usr/sbin/httpd
> 13373 ?        S      0:00  \_ /usr/sbin/httpd
> 6416 ?        Rs     0:00 ps axf

Do the various kernel instances share httpd text pages?

Re: [ANNOUNCE] first stable release of OpenVZ kernel virtualization solution

[Kirill Korotaev]

Hello Andrew,

>>We are happy to announce the release of a stable version of the OpenVZ 
>>software, located at http://openvz.org/.
>>
>>OpenVZ is a kernel virtualization solution which can be considered as a
>>natural step in the OS kernel evolution: after multiuser and 
>>multitasking functionality there comes an OpenVZ feature of having 
>>multiple environments.
> 
> 
> Are you able to give us a high-level overview of how it actually is
> implemented?  IOW: what does the patch do? 
Will be glad to do so.

On the high-level the system looks like patched Linux Kernel with a 
number of user space tools. The kernel itself boots on a usual Linux 
distribution like RHEL4 and works as usual. But there are extensions 
which allow to create a new VPS context.
User space OpenVZ tools use these extensions to do the following, e.g. 
on VPS start:
- turn on and configure quota on VPS file system subtree.
- chroot to this filesystem tree.
- create a UBC context with configured resource limits/guarantees.
- create a VPS context and exec init in this newly created environment.
- newly spawned init executes VPS initscripts as if it was a usual Linux 
box which has switched power on.

There are two patches that can be found on OpenVZ site.
patch-022stabXXX-combined which is a single consolidated OpenVZ patch 
including driver and mainstream updates. And the more interesting one is 
patch-022stabXXX-core (available from the SRC RPM), which itself 
consists of the following parts:
- mainstream fixes and security fixes
- 4GB split
- User beancounters (kernel/ub/*, include/ub/*). This includes 
accounting and limiting/guarantees of resources.
- Virtualization itself (ve_struct, kernel/vecalls.c - main code for VPS 
start/stop, net/ipv4 - virtualization of TPC/IP and netfilters, 
drivers/net/venet* - virtual network device for VPS, virtual pids, etc.)
- fs/simfs - simple filesystem to fake VPS and return correct values on 
`df` and statfs() output.
- fs/vzdq* - 2-level disk quota.
- kernel/fairsched.c and kernel/sched.c - fair CPU scheduler (SFQ-like 
scheduler with shares and hardlimits)

We strongly believe that Linux kernel can benefit from Virtualization 
and Resource Management very much, so are very interested in your and 
Linus comments on this. Though virtualization and resource management is 
not a full feature set provided by OpenVZ, they can be used for creation 
of more secure environments where untrusted users are involved. For 
example, with virtualization it is possible to isolate set of processes 
on the node, web server or other application, thus vulnerabilities in 
this application won't allow to destroy the whole system (e.g. mail 
server running on the same node), install some backdoors/trojans (in 
kernel modules) etc. Resource control can be used for the same purposes 
when deliberate DoS or bugs in applications should not crash the whole 
system down. Virtualization also allows to do simple 
backups/restore/migration of VPSs between nodes, thus making maintenance 
much easier. So the I would summary up:
- Virtualization helps to isolate service, improves managebility
- Resource Management allows to control resources and prevent DoS from 
utrusted users in multi-user environments. User beancounters can be used 
for usual users instead of VPSs.

Main user space tools for OpenVZ are:
- vzctl, which is used for most of high-level VPS operations like VPS 
creation, start/stop, destroy, configuring, setting UBC and other 
parameters etc.
# vzctl create VPSID
is used to create VPS.

After VPS creation it can be started via issuing:
# vzctl start VPSID

To see set of processes inside VPS one can execute:
# vzctl exec VPSID ps axf

And the most interesting command is 'enter' which allows to get to VPS 
(to 'enter') from host system via changing context to VPS one:
# vzctl enter VPSID
bash#

- vzquota, a tool used for 2level quota support. Allows to turn quota 
on/off, recalculate it etc.

- vzpkg, a set tool of tools allowing to easily manage VPS templates 
(redhat, centos, fedora etc.).

>>As virtualization solution OpenVZ makes it possible to do the same 
>>things for which people use UML, Xen, QEmu or VMware, but there are 
>>differences:
>>(a) there is no ability to run other operating systems
>>     (although different Linux distros can happily coexist);
>>(b) performance loss is negligible due to absense of any kind of
>>     emulation;
>>(c) resource utilization is much better.
> 
> What are OpenVZ's disadvantages wrt the above?

disadvantages:
- unable to run Windows or xBSD on OpenVZ Linux.
- VPS owner can't load/use custom kernel modules
- theoretically stability of such solution has one single point of 
failure - the kernel. This is mitigated by lots of auto (stress) tests 
done by us to be sure the kernel is stable (we have more than >200 
mainstream patches due to this). The stability is the main goal here, 
since servers running multiple VPSs work under much higher load and the 
cost of kernel oops is much higher.
- in some respects dealing with files and processes is harder than with 
VM which is bounded to it's memory, CPU and disk state, e.g. when doing 
backups. However, there are many cases when this is rather an advantage, 
e.g. when you are able to accesses VPS files and processes from the host 
system and do some management actions when VPS itself is stuck or 
performs poorly and no remote access is available.

>>The dynamic assignment of resources in OpenVZ can significantly improve 
>>their utilization. For example, a x86_64 box (2.8 GHz Celeron D, 1GB 
>>RAM) is capable to run 100 VPSs with a fairly high performance (VPSs 
>>were serving http requests for 4.2Kb static pages at an overall rate of 
>>more than 80,000 req/min). Each VPS (running CentOS 4 x86_64) had the 
>>following set of processes:
[skipped]

> Do the various kernel instances share httpd text pages?
Please note, there is a single kernel instance for all VPSs! This means 
commong page cache etc.

But in this particular example each VPS has it's own file tree and it's 
own httpd instance in memory with full set of pages (.text, .data, ...), 
i.e. there were running 100 VPSs without any page sharing involved.

"User Beancounters" accounnting is implemented with page sharing in mind 
and will be correct in this case (i.e. only page fraction will be 
charged). With page sharing this example scales well up to 200+ VPSs. 
And sharing itself can be achieved in multiple ways: common part of file 
system tree (e.g. /lib, /usr, etc.) or via a special filesystem.

Kirill

Re: [ANNOUNCE] first stable release of OpenVZ kernel virtualization solution

[Philippe Pegon]

Hello,

By curiosity, what is the status for IPv6 in OpenVZ (I saw that it was 
in the roadmap on the website, but maybe you have more informations) ?

thanks
--
Philippe Pegon

Kirill Korotaev wrote:
> Hello,
> 
> We are happy to announce the release of a stable version of the OpenVZ 
> software, located at http://openvz.org/.
> 
> OpenVZ is a kernel virtualization solution which can be considered as a
> natural step in the OS kernel evolution: after multiuser and 
> multitasking functionality there comes an OpenVZ feature of having 
> multiple environments.
> 
> Virtualization lets you divide a system into separate isolated
> execution environments (called VPSs - Virtual Private Servers). From the
> point of view of the VPS owner (root), it looks like a stand-alone 
> server. Each VPS has its own filesystem tree, process tree (starting 
> from init as in a real system) and so on. The  single-kernel approach 
> makes it possible to virtualize with very little overhead, if any.
> 
> OpenVZ in-kernel modifications can be divided into several components:
> 
> 1. Virtualization and isolation.
> Many Linux kernel subsystems are virtualized, so each VPS has its own:
> - process tree (featuring virtualized pids, so that the init pid is 1);
> - filesystems (including virtualized /proc and /sys);
> - network (virtual network device, its own ip addresses,
>   set of netfilter and routing rules);
> - devices (if needed, any VPS can be granted access to real devices
>   like network interfaces, serial ports, disk partitions, etc);
> - IPC objects.
> 
> 2. Resource Management.
> This subsystem enables multiple VPSs to coexist, providing managed
> resource sharing and limiting.
> - User Beancounters is a set of per-VPS resource counters, limits,
>   and guarantees (kernel memory, network buffers, phys pages, etc.).
> - Fair CPU scheduler (SFQ with shares and hard limits).
> - Two-level disk quota (first-level: per-VPS quota;
>   second-level: ordinary user/group quota inside a VPS)
> 
> Resource management is what makes OpenVZ different from other solutions
> of this kind (like Linux VServer or FreeBSD jails). There are a few
> resources that can be abused from inside a VPS (such as files, IPC
> objects, ...) leading to a DoS attack. User Beancounters prevent such
> abuses.
> 
> As virtualization solution OpenVZ makes it possible to do the same 
> things for which people use UML, Xen, QEmu or VMware, but there are 
> differences:
> (a) there is no ability to run other operating systems
>     (although different Linux distros can happily coexist);
> (b) performance loss is negligible due to absense of any kind of
>     emulation;
> (c) resource utilization is much better.
> 
> The last point needs to be elaborated on. OpenVZ allows to utilize
> system resources such as memory and disk space very efficiently, and
> because of that has better performance on memory-critical workloads.
> OpenVZ does not run separate kernels in each VPS and saves memory on
> kernel internal data. However, even bigger efficiency of OpenVZ comes
> from dynamic resource allocation.
> 
> With other virtualization solutions, you need to specify in advance the
> amount of memory for each virtual machine and create a disk device and
> filesystem for it, and the possibilities to change settings later on the
> fly are very limited.
> 
> The dynamic assignment of resources in OpenVZ can significantly improve 
> their utilization. For example, a x86_64 box (2.8 GHz Celeron D, 1GB 
> RAM) is capable to run 100 VPSs with a fairly high performance (VPSs 
> were serving http requests for 4.2Kb static pages at an overall rate of 
> more than 80,000 req/min). Each VPS (running CentOS 4 x86_64) had the 
> following set of processes:
> 
> [root@ovz-x64 ~]# vzctl exec 1043 ps axf
>  PID TTY      STAT   TIME COMMAND
>    1 ?        Ss     0:00 init
> 11830 ?        Ss     0:00 syslogd -m 0
> 11897 ?        Ss     0:00 /usr/sbin/sshd
> 11943 ?        Ss     0:00 xinetd -stayalive -pidfile ...
> 12218 ?        Ss     0:00 sendmail: accepting connections
> 12265 ?        Ss     0:00 sendmail: Queue runner@01:00:00
> 13362 ?        Ss     0:00 /usr/sbin/httpd
> 13363 ?        S      0:00  \_ /usr/sbin/httpd
> 13364 ?        S      0:00  \_ /usr/sbin/httpd
> 13365 ?        S      0:00  \_ /usr/sbin/httpd
> 13366 ?        S      0:00  \_ /usr/sbin/httpd
> 13370 ?        S      0:00  \_ /usr/sbin/httpd   
> 13371 ?        S      0:00  \_ /usr/sbin/httpd
> 13372 ?        S      0:00  \_ /usr/sbin/httpd
> 13373 ?        S      0:00  \_ /usr/sbin/httpd
> 6416 ?        Rs     0:00 ps axf
> 
> And the list of running VPSs:
> 
> [root@ovz-x64 ~]# vzlist
>     VPSID      NPROC STATUS  IP_ADDR         HOSTNAME
>      1001         15 running 10.1.1.1        vps1001
>      1002         15 running 10.1.1.2        vps1002
>      [....skipped....]
>      1099         15 running 10.1.1.99       vps1099
>      1100         15 running 10.1.1.100      vps1100
> 
> On the box with 4Gb of RAM one can expect 400 of such VPSs to run 
> without much troubles.
> 
> More information is available at http://openvz.org/
> 
> Thanks,
> OpenVZ team.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/