[Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[M. Mohan Kumar]

Pass-through security model in QEMU 9p server needs root privilege to do few
file operations (like chown, chmod to any mode/uid:gid).  There are two issues
in pass-through security model

1) TOCTTOU vulnerability: Following symbolic links in the server could
provide access to files beyond 9p export path.

2) When libvirt is configured to run qemu as non-root user (for example, if
qemu is configured to run as normal user 'qemu'), running file operations on
pass-through security model would fail because it needs root privileges.

To overcome above issues, following approach is suggested: A new filesytem
type 'proxy' is introduced. Proxy FS uses chroot + socket combination for
securing the vulnerability known with following symbolic links. Intention of 
adding a new filesystem type is to allow qemu to run in non-root mode, but 
doing privileged operations using socket IO.

A new binary (known as proxy helper) will be provided as part of qemu. Proxy
helper will chroot into 9p export path and create a socket pair or a named
socket based on the command line parameter. Qemu and proxy helper will
communicate using this socket.

We need following changes in the libvirt code to accomodate new 'proxy' 
filesystem type:
If qemu 9p server is configured to use 'proxy' FS, libvirt will do
* Create a socket pair
* invoke proxy_helper binary with one of the socket id from the pair as
command line parameters to it with root privilege
* invoke qemu with one of socket id from the pair as paramter to qemu virtfs
after dropping to the configured user privilege.
ie, libvirt will invoke proxy_helper as:
    proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>

and qemu will be invoked with following virtfs parameter:
    -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
	,path=/tmp/,security_model=prox,mount_tag=v_pass

People who want to use proxy_helper without libvirt can use following 
interface:
 $   proxy_helper -s </socket/path> -p <9p-path-to-export>

With following qemu fsdev parameter:
    -virtfs proxy,id=<id>,socket=</socket/path>,path=/tmp/,
	security_model=prox,mount_tag=v_pass


-- 
Regards,
M. Mohan Kumar

Re: [Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[Daniel P. Berrange]

On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
> Pass-through security model in QEMU 9p server needs root privilege to do few
> file operations (like chown, chmod to any mode/uid:gid).  There are two issues
> in pass-through security model
> 
> 1) TOCTTOU vulnerability: Following symbolic links in the server could
> provide access to files beyond 9p export path.
> 
> 2) When libvirt is configured to run qemu as non-root user (for example, if
> qemu is configured to run as normal user 'qemu'), running file operations on
> pass-through security model would fail because it needs root privileges.
> 
> To overcome above issues, following approach is suggested: A new filesytem
> type 'proxy' is introduced. Proxy FS uses chroot + socket combination for
> securing the vulnerability known with following symbolic links. Intention of 
> adding a new filesystem type is to allow qemu to run in non-root mode, but 
> doing privileged operations using socket IO.
> 
> A new binary (known as proxy helper) will be provided as part of qemu. Proxy
> helper will chroot into 9p export path and create a socket pair or a named
> socket based on the command line parameter. Qemu and proxy helper will
> communicate using this socket.
> 
> We need following changes in the libvirt code to accomodate new 'proxy' 
> filesystem type:
> If qemu 9p server is configured to use 'proxy' FS, libvirt will do
> * Create a socket pair
> * invoke proxy_helper binary with one of the socket id from the pair as
> command line parameters to it with root privilege
> * invoke qemu with one of socket id from the pair as paramter to qemu virtfs
> after dropping to the configured user privilege.
> ie, libvirt will invoke proxy_helper as:
>     proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
> 
> and qemu will be invoked with following virtfs parameter:
>     -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
> 	,path=/tmp/,security_model=prox,mount_tag=v_pass

Interesting proposal. Explicitly comparing the security characteristics
of running QEMU as root, vs using the proxy helper

 * QEMU run as root

     - QEMU is root, with full capabilities
     - QEMU has read/write any file/dir, regardless of whether it is
       exported via 9p

 * QEMU run as non-root, with proxy_helper root

     - QEMU is non-root, with no capabilities
     - QEMU has write to only files with matching UID/GID
     - proxy_helper is root, with full capabilities
     - proxy_helper has read/write to any file/dir

Since QEMU can send arbitrary FS calls to the proxy_helper, the overall
security of the system clearly depends on the security of the proxy_helper
process.

If we assume that QEMU gets exploited, and that QEMU can find some flaw
in the proxy_helper that it can exploit, what damage can the proxy_helper
do ?  Clearly we want it to not be able to read/write any files other
than those exported, even when it becomes compromised, ideally also
without requiring SELinux/AppArmour to make it safe. If proxy_helper
is running as root with full capabilities, it can trivially escape
the chroot[1], so this isn't all that nice for overall system security.
The attacker simply needs to find a vulnerability in QEMU and in the
proxy_helper, instead of just in QEMU. We want a stronger guarantee
than that.

In order to be able to chown/chmod files etc, the proxy_helper is going
to require some elevated privileges, however, this does not actually
imply that proxy_helper needs to run as root. You don't want the proxy_helper
to be allowed todo various other things that root would be allowed, even
if it is inside a chroot. eg you don't want it being allowed to reboot,
mlock, change network setting, mount volumes, make device nodes, and
quite alot more.

Looking at the man page 'capabilities(7)', I'd venture to suggest that
the proxy_helper does not in fact need to run as the 'root' UID/GID at
all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE,
CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could
still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user
ID).

So in summary, I think you have a good idea to separate the file handling
code into a separate proxy_helper binary. It should not, however, be run
as root with all capabilities. It can start as root, and then drop to a
non-root user, only keeping the few capabilities it needs. Or in libvirt
case, libvirt can spawn it with the minimal set of capabilities directly.

eg

 * QEMU run as non-root, with proxy_helper partially privileged

     - QEMU is non-root, with no capabilities
     - QEMU has write to only files with matching UID/GID
     - proxy_helper is non-root, with selected capabilities
     - proxy_helper has read/write to any file/dir inside the
       chroot for the 9p export.


Regards,
Daniel

[1] http://www.bpfh.net/simes/computing/chroot-break.html
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[M. Mohan Kumar]

On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote:
> On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
> > Pass-through security model in QEMU 9p server needs root privilege to do
> > few file operations (like chown, chmod to any mode/uid:gid).  There are
> > two issues in pass-through security model
> > 
> > 1) TOCTTOU vulnerability: Following symbolic links in the server could
> > provide access to files beyond 9p export path.
> > 
> > 2) When libvirt is configured to run qemu as non-root user (for example,
> > if qemu is configured to run as normal user 'qemu'), running file
> > operations on pass-through security model would fail because it needs
> > root privileges.
> > 
> > To overcome above issues, following approach is suggested: A new
> > filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket
> > combination for securing the vulnerability known with following symbolic
> > links. Intention of adding a new filesystem type is to allow qemu to run
> > in non-root mode, but doing privileged operations using socket IO.
> > 
> > A new binary (known as proxy helper) will be provided as part of qemu.
> > Proxy helper will chroot into 9p export path and create a socket pair or
> > a named socket based on the command line parameter. Qemu and proxy
> > helper will communicate using this socket.
> > 
> > We need following changes in the libvirt code to accomodate new 'proxy'
> > filesystem type:
> > If qemu 9p server is configured to use 'proxy' FS, libvirt will do
> > * Create a socket pair
> > * invoke proxy_helper binary with one of the socket id from the pair as
> > command line parameters to it with root privilege
> > * invoke qemu with one of socket id from the pair as paramter to qemu
> > virtfs after dropping to the configured user privilege.
> > 
> > ie, libvirt will invoke proxy_helper as:
> >     proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
> > 
> > and qemu will be invoked with following virtfs parameter:
> >     -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
> > 	
> > 	,path=/tmp/,security_model=prox,mount_tag=v_pass
>

Thank you Daniel for the quick response, I was on leave and I could not 
respond to you immediately.
  
> Interesting proposal. Explicitly comparing the security characteristics
> of running QEMU as root, vs using the proxy helper
> 
>  * QEMU run as root
> 
>      - QEMU is root, with full capabilities
>      - QEMU has read/write any file/dir, regardless of whether it is
>        exported via 9p
> 
>  * QEMU run as non-root, with proxy_helper root
> 
>      - QEMU is non-root, with no capabilities
>      - QEMU has write to only files with matching UID/GID
>      - proxy_helper is root, with full capabilities
>      - proxy_helper has read/write to any file/dir
> 
> Since QEMU can send arbitrary FS calls to the proxy_helper, the overall
> security of the system clearly depends on the security of the proxy_helper
> process.
QEMU can't send arbitrary FS calls to the proxy helper. All interfaces are 
defined and if qemu sends arbitrary FS command, proxy helper would return an 
EIO. For example T_OPEN is one of the supported interface between 9p qemu 
server and proxy helper. For more information please look at my chroot 
patchset that I posted to qemu-devel few weeks ago. 
http://lists.gnu.org/archive/html/qemu-devel/2011-09/msg00671.html

> 
> If we assume that QEMU gets exploited, and that QEMU can find some flaw
> in the proxy_helper that it can exploit, what damage can the proxy_helper
> do ?  Clearly we want it to not be able to read/write any files other
> than those exported, even when it becomes compromised, ideally also
> without requiring SELinux/AppArmour to make it safe. If proxy_helper
> is running as root with full capabilities, it can trivially escape
> the chroot[1], so this isn't all that nice for overall system security.
When proxy helper process is forked, its made sure that all open file 
descriptor are closed except the socket descriptor, so that proxy helper can't 
escape chroot jail.

Main aim of providing proxy helper is to
1) Access beyond 9p export path should not be given
2) Remove the limitation that qemu should be run by root user to use pass-
through security model

> The attacker simply needs to find a vulnerability in QEMU and in the
> proxy_helper, instead of just in QEMU. We want a stronger guarantee
> than that.
> 
> In order to be able to chown/chmod files etc, the proxy_helper is going
> to require some elevated privileges, however, this does not actually
> imply that proxy_helper needs to run as root. You don't want the
> proxy_helper to be allowed todo various other things that root would be
> allowed, even if it is inside a chroot. eg you don't want it being allowed
> to reboot, mlock, change network setting, mount volumes, make device
> nodes, and quite alot more.

Proxy helper operations are categorized as:
1) Open/create file and return the descriptor
2) Create directory, link, symbolic link, device nodes, FIFOs etc
3) Remove/rename filesystem object
4) Return stat information
5) Return readlink() information

> 
> Looking at the man page 'capabilities(7)', I'd venture to suggest that
> the proxy_helper does not in fact need to run as the 'root' UID/GID at
> all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE,
> CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could
> still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user
> ID).
IMHO, proxy helper can run as root because it will run only predefined file 
operations.

Re: [Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[Daniel P. Berrange]

On Thu, Sep 29, 2011 at 11:42:47PM +0530, M. Mohan Kumar wrote:
> On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote:
> > On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
> > > Pass-through security model in QEMU 9p server needs root privilege to do
> > > few file operations (like chown, chmod to any mode/uid:gid).  There are
> > > two issues in pass-through security model
> > > 
> > > 1) TOCTTOU vulnerability: Following symbolic links in the server could
> > > provide access to files beyond 9p export path.
> > > 
> > > 2) When libvirt is configured to run qemu as non-root user (for example,
> > > if qemu is configured to run as normal user 'qemu'), running file
> > > operations on pass-through security model would fail because it needs
> > > root privileges.
> > > 
> > > To overcome above issues, following approach is suggested: A new
> > > filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket
> > > combination for securing the vulnerability known with following symbolic
> > > links. Intention of adding a new filesystem type is to allow qemu to run
> > > in non-root mode, but doing privileged operations using socket IO.
> > > 
> > > A new binary (known as proxy helper) will be provided as part of qemu.
> > > Proxy helper will chroot into 9p export path and create a socket pair or
> > > a named socket based on the command line parameter. Qemu and proxy
> > > helper will communicate using this socket.
> > > 
> > > We need following changes in the libvirt code to accomodate new 'proxy'
> > > filesystem type:
> > > If qemu 9p server is configured to use 'proxy' FS, libvirt will do
> > > * Create a socket pair
> > > * invoke proxy_helper binary with one of the socket id from the pair as
> > > command line parameters to it with root privilege
> > > * invoke qemu with one of socket id from the pair as paramter to qemu
> > > virtfs after dropping to the configured user privilege.
> > > 
> > > ie, libvirt will invoke proxy_helper as:
> > >     proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
> > > 
> > > and qemu will be invoked with following virtfs parameter:
> > >     -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
> > > 	
> > > 	,path=/tmp/,security_model=prox,mount_tag=v_pass
> >
> 
> Thank you Daniel for the quick response, I was on leave and I could not 
> respond to you immediately.
>   
> > Interesting proposal. Explicitly comparing the security characteristics
> > of running QEMU as root, vs using the proxy helper
> > 
> >  * QEMU run as root
> > 
> >      - QEMU is root, with full capabilities
> >      - QEMU has read/write any file/dir, regardless of whether it is
> >        exported via 9p
> > 
> >  * QEMU run as non-root, with proxy_helper root
> > 
> >      - QEMU is non-root, with no capabilities
> >      - QEMU has write to only files with matching UID/GID
> >      - proxy_helper is root, with full capabilities
> >      - proxy_helper has read/write to any file/dir
> > 
> > Since QEMU can send arbitrary FS calls to the proxy_helper, the overall
> > security of the system clearly depends on the security of the proxy_helper
> > process.
> QEMU can't send arbitrary FS calls to the proxy helper. All interfaces are 
> defined and if qemu sends arbitrary FS command, proxy helper would return an 
> EIO. For example T_OPEN is one of the supported interface between 9p qemu 
> server and proxy helper. For more information please look at my chroot 
> patchset that I posted to qemu-devel few weeks ago. 
> http://lists.gnu.org/archive/html/qemu-devel/2011-09/msg00671.html

Ok, perhaps I didn't explain what I meant by 'arbitrary FS calls' very
well here.

Lets, say the communication protocol between QEMU and the proxy helper
has a set of FS calls to cover 'read', 'write', 'open', 'unlink', 'link'.
And we have a 9pfs directory to give to the guest '/export/to-the-guest'.

In normal use, the QEMU process can send things like

    open("/export/to-the-guest/somefile").

A malicious QEMU can send 

    open("/etc/shadow")

Or any other path that it cares about, whether or not, the path is within
the location exported to the 9pfs server. The security of the overall
system, thus relies on the proxy helper enforcing that the arbitrary
FS calls from QEMU, are in fact only within '/export/to-the-guest' and
not other directories.

> > If we assume that QEMU gets exploited, and that QEMU can find some flaw
> > in the proxy_helper that it can exploit, what damage can the proxy_helper
> > do ?  Clearly we want it to not be able to read/write any files other
> > than those exported, even when it becomes compromised, ideally also
> > without requiring SELinux/AppArmour to make it safe. If proxy_helper
> > is running as root with full capabilities, it can trivially escape
> > the chroot[1], so this isn't all that nice for overall system security.
>
> When proxy helper process is forked, its made sure that all open file 
> descriptor are closed except the socket descriptor, so that proxy helper can't 
> escape chroot jail.

Unfortunately that is not sufficient. Even if you have closed all
file descriptors except the socket, you can still escape the chroot
jail. There is a nice demo program for escaping from a chroot on
this webpage:

  http://www.bpfh.net/simes/computing/chroot-break.html

If you want to run as root, then at the very least you need to
explicitly drop the CAP_SYS_CHROOT capability to block this
avenue of attack, and also clear it from the boundingset to
prevent it being re-acquired. Even that is likely not sufficient
on its own though, if you retain all other capabilities, since
those can be used to compromise other processes outside the chroot
into doing things.

Basically if the process inside the chroot, is still running as
root, then it is not securely contained.

> Main aim of providing proxy helper is to
> 1) Access beyond 9p export path should not be given
> 2) Remove the limitation that qemu should be run by root user to use pass-
> through security model

Yep, I agree with those aims.

> > The attacker simply needs to find a vulnerability in QEMU and in the
> > proxy_helper, instead of just in QEMU. We want a stronger guarantee
> > than that.
> > 
> > In order to be able to chown/chmod files etc, the proxy_helper is going
> > to require some elevated privileges, however, this does not actually
> > imply that proxy_helper needs to run as root. You don't want the
> > proxy_helper to be allowed todo various other things that root would be
> > allowed, even if it is inside a chroot. eg you don't want it being allowed
> > to reboot, mlock, change network setting, mount volumes, make device
> > nodes, and quite alot more.
> 
> Proxy helper operations are categorized as:
> 1) Open/create file and return the descriptor
> 2) Create directory, link, symbolic link, device nodes, FIFOs etc
> 3) Remove/rename filesystem object
> 4) Return stat information
> 5) Return readlink() information
> 
> > 
> > Looking at the man page 'capabilities(7)', I'd venture to suggest that
> > the proxy_helper does not in fact need to run as the 'root' UID/GID at
> > all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE,
> > CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could
> > still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user
> > ID).
> IMHO, proxy helper can run as root because it will run only predefined file 
> operations.

That is the case if the proxy helper code is perfectly written. I am trying
to think about the scenario where there is a bug (eg heap corruption / stack
overflow) which allows a malicious non-root QEMU process to exploit the
proxy helper to run code that it was *not* intended to run.

If the proxy helper is running root with all capabilities, then a bug in
the proxy helper can easily turn into a full root exploit.

If the proxy helper starts as root, chroots, and then immediately drops to
a non-root user, keeping only the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER
and CAP_DAC_READ_SEARCH capabilities, then a bug in the proxy helper can
only be used to access files within the designated 9pfs export. If the
exported directory does not contain any important host system files, then
it is unlikely it can be used to create a full root exploit.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[M. Mohan Kumar]

> 
> That is the case if the proxy helper code is perfectly written. I am trying
> to think about the scenario where there is a bug (eg heap corruption /
> stack overflow) which allows a malicious non-root QEMU process to exploit
> the proxy helper to run code that it was *not* intended to run.
> 
> If the proxy helper is running root with all capabilities, then a bug in
> the proxy helper can easily turn into a full root exploit.
> 
> If the proxy helper starts as root, chroots, and then immediately drops to
> a non-root user, keeping only the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER
> and CAP_DAC_READ_SEARCH capabilities, then a bug in the proxy helper can
> only be used to access files within the designated 9pfs export. If the
> exported directory does not contain any important host system files, then
> it is unlikely it can be used to create a full root exploit.
> 

Thanks Daniel, I will add 'capabiliies' to proxy helper. CAP_FOWNER capability 
also need.

I am working on the patches. I will post them in few days.

Re: [Qemu-devel] [RFC] Adding new filesystem 'proxy' to 9p

[Markus Armbruster]

"Daniel P. Berrange" <berrange@redhat.com> writes:

> On Thu, Sep 29, 2011 at 11:42:47PM +0530, M. Mohan Kumar wrote:
>> On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote:
[...]
>> > If we assume that QEMU gets exploited, and that QEMU can find some flaw
>> > in the proxy_helper that it can exploit, what damage can the proxy_helper
>> > do ?  Clearly we want it to not be able to read/write any files other
>> > than those exported, even when it becomes compromised, ideally also
>> > without requiring SELinux/AppArmour to make it safe. If proxy_helper
>> > is running as root with full capabilities, it can trivially escape
>> > the chroot[1], so this isn't all that nice for overall system security.
>>
>> When proxy helper process is forked, its made sure that all open file 
>> descriptor are closed except the socket descriptor, so that proxy helper can't 
>> escape chroot jail.
>
> Unfortunately that is not sufficient. Even if you have closed all
> file descriptors except the socket, you can still escape the chroot
> jail. There is a nice demo program for escaping from a chroot on
> this webpage:
>
>   http://www.bpfh.net/simes/computing/chroot-break.html
[...]
> Basically if the process inside the chroot, is still running as
> root, then it is not securely contained.

As Alan Cox observed, "chroot is not and never has been a security
tool."  It's a tool to configure the root of absolute path names.

[...]