UML for arm64

UML for arm64

[Rob Herring]

Hi all,

I'm interested in getting UML working on Arm. Is anyone aware of any 
efforts to do this already? I've not found anything searching around. Or 
perhaps a port to any other arch as I'd imagine any 2nd arch will 
require some refactoring.

Thanks,
Rob

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Johannes Berg]

On Thu, 2023-06-22 at 13:22 -0600, Rob Herring wrote:
> Hi all,
> 
> I'm interested in getting UML working on Arm. Is anyone aware of any 
> efforts to do this already?

Not me ... But yeah, agree that it seems likely to require some
refactoring, even if _some_ degree of attempts were made to keep arch-
specific things there ...

There might be a ton of arch-specific things in the ptrace code too, so
perhaps it'd be easier to start with the seccomp-based model:

https://patchwork.ozlabs.org/project/linux-um/list/?series=329466

But that's not really done yet either.

johannes

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Benjamin Berg]

Hi,

On Thu, 2023-06-22 at 21:41 +0200, Johannes Berg wrote:
> On Thu, 2023-06-22 at 13:22 -0600, Rob Herring wrote:
> > I'm interested in getting UML working on Arm. Is anyone aware of any 
> > efforts to do this already?
> 
> Not me ... But yeah, agree that it seems likely to require some
> refactoring, even if _some_ degree of attempts were made to keep arch-
> specific things there ...
> 
> There might be a ton of arch-specific things in the ptrace code too

At least a lot of the register handling, but that will always be
somewhat architecture specific.

> ... , so perhaps it'd be easier to start with the seccomp-based model:
> 
> https://patchwork.ozlabs.org/project/linux-um/list/?series=329466
> 
> But that's not really done yet either.

Yes, though I have quite some updates locally that solve the security
issues (i.e. prevent host syscalls from userspace). So, it does need
rebasing and I am sure bugfixes, but I think it is getting into a
pretty good shape overall.

Either way, the old patchset will give you a good idea about how it all
works, the changes are mostly in the details. I am happy to push out a
new version sooner rather than later if it might help with any efforts
on your side.

Benjamin

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Richard Weinberger]

[CC'ing Al]

----- Ursprüngliche Mail -----
> Von: "Benjamin Berg" <benjamin@sipsolutions.net>
> An: "Johannes Berg" <johannes@sipsolutions.net>, "robh" <robh@kernel.org>, "richard" <richard@nod.at>, "anton ivanov"
> <anton.ivanov@cambridgegreys.com>
> CC: "linux-um" <linux-um@lists.infradead.org>
> Gesendet: Donnerstag, 22. Juni 2023 22:13:56
> Betreff: Re: UML for arm64

> Hi,
> 
> On Thu, 2023-06-22 at 21:41 +0200, Johannes Berg wrote:
>> On Thu, 2023-06-22 at 13:22 -0600, Rob Herring wrote:
>> > I'm interested in getting UML working on Arm. Is anyone aware of any
>> > efforts to do this already?
>> 
>> Not me ... But yeah, agree that it seems likely to require some
>> refactoring, even if _some_ degree of attempts were made to keep arch-
>> specific things there ...
>> 
>> There might be a ton of arch-specific things in the ptrace code too

I think Al did some work in the past to support archs other than x86.
It was not arm64, though. ppc or sparc IIRC. Maybe he has some input.
 
> At least a lot of the register handling, but that will always be
> somewhat architecture specific.
> 
>> ... , so perhaps it'd be easier to start with the seccomp-based model:
>> 
>> https://patchwork.ozlabs.org/project/linux-um/list/?series=329466
>> 
>> But that's not really done yet either.
> 
> Yes, though I have quite some updates locally that solve the security
> issues (i.e. prevent host syscalls from userspace). So, it does need
> rebasing and I am sure bugfixes, but I think it is getting into a
> pretty good shape overall.
> 
> Either way, the old patchset will give you a good idea about how it all
> works, the changes are mostly in the details. I am happy to push out a
> new version sooner rather than later if it might help with any efforts
> on your side.

I think reviving the seccomp patch set is a good idea in any case.
Maybe it is even a way to allow SMP support.

Thanks,
//richard

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Benjamin Berg]

HI,

On Thu, 2023-06-22 at 22:23 +0200, Richard Weinberger wrote:
> [SNIP]
>  
> > At least a lot of the register handling, but that will always be
> > somewhat architecture specific.
> > 
> > > ... , so perhaps it'd be easier to start with the seccomp-based model:
> > > 
> > > https://patchwork.ozlabs.org/project/linux-um/list/?series=329466
> > > 
> > > But that's not really done yet either.
> > 
> > Yes, though I have quite some updates locally that solve the security
> > issues (i.e. prevent host syscalls from userspace). So, it does need
> > rebasing and I am sure bugfixes, but I think it is getting into a
> > pretty good shape overall.
> > 
> > Either way, the old patchset will give you a good idea about how it all
> > works, the changes are mostly in the details. I am happy to push out a
> > new version sooner rather than later if it might help with any efforts
> > on your side.
> 
> I think reviving the seccomp patch set is a good idea in any case.
> Maybe it is even a way to allow SMP support.

OK, I'll see about resubmitting the SECCOMP patchset.

About SMP support, it is tricky if one wants to do it safely. Doing it
in an unsafe way is probably reasonably straight forward as you can
simply map multiple stub-data areas into the same process. However, the
other threads can read/write memory at the same time, which breaks the
entire memory security model that I had in mind for SECCOMP mode. My
approach there heavily relies on killing the process *after* it mapped
memory without permission, but before this can be exploited.

So, I fear no simple and secure SMP support. Having to do host syscalls
from the userspace process means that you just cannot have another
thread running. And then, you are stuck with multiple processes and the
headache and overhead of keeping their memory in sync.

I think secure SMP we would need some other new and exciting approach
for handling process memory. An example could be a special (memory)
file system on the host that explicitly allows setting up page ranges
to be transparently shared between files.

Benjamin

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Anton Ivanov]

On 22/06/2023 21:13, Benjamin Berg wrote:
> Hi,
> 
> On Thu, 2023-06-22 at 21:41 +0200, Johannes Berg wrote:
>> On Thu, 2023-06-22 at 13:22 -0600, Rob Herring wrote:
>>> I'm interested in getting UML working on Arm. Is anyone aware of any
>>> efforts to do this already?
>>
>> Not me ... But yeah, agree that it seems likely to require some
>> refactoring, even if _some_ degree of attempts were made to keep arch-
>> specific things there ...
>>
>> There might be a ton of arch-specific things in the ptrace code too
> 
> At least a lot of the register handling, but that will always be
> somewhat architecture specific.

In the arm case there may be some paging issues.

Arm pages are not necessarily 4096. I tried to implement non-4096 page support for UML a couple of years ago and gave up.

It will be nice if we manage to fix this. End of the day - UML uses mmap(), so it should be able to use page sizing which differs from the host one.

> 
>> ... , so perhaps it'd be easier to start with the seccomp-based model:
>>
>> https://patchwork.ozlabs.org/project/linux-um/list/?series=329466
>>
>> But that's not really done yet either.
> 
> Yes, though I have quite some updates locally that solve the security
> issues (i.e. prevent host syscalls from userspace). So, it does need
> rebasing and I am sure bugfixes, but I think it is getting into a
> pretty good shape overall.
> 
> Either way, the old patchset will give you a good idea about how it all
> works, the changes are mostly in the details. I am happy to push out a
> new version sooner rather than later if it might help with any efforts
> on your side.
> 
> Benjamin
> 
> _______________________________________________
> linux-um mailing list
> linux-um@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-um
> 

-- 
Anton R. Ivanov
Cambridgegreys Limited. Registered in England. Company Number 10273661
https://www.cambridgegreys.com/

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Benjamin Beichler]

Hi,

> Hi all,
> 
> I'm interested in getting UML working on Arm. Is anyone aware of any
> efforts to do this already? I've not found anything searching around. Or
> perhaps a port to any other arch as I'd imagine any 2nd arch will
> require some refactorin
I am also quite interested in such a port. We may help with some testing 
or even more, if it fits my schedule.

 From my longer digging in the code, I can only confirm, that several 
parts are primarily thought from x86-ISA standpoint and already the 
switch to amd64 is flimsy at some places.


> Thanks,
> Rob
> 
Regards

Benjamin


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Rob Herring]

On Thu, Jun 22, 2023 at 2:14 PM Benjamin Berg <benjamin@sipsolutions.net> wrote:
>
> Hi,
>
> On Thu, 2023-06-22 at 21:41 +0200, Johannes Berg wrote:
> > On Thu, 2023-06-22 at 13:22 -0600, Rob Herring wrote:
> > > I'm interested in getting UML working on Arm. Is anyone aware of any
> > > efforts to do this already?
> >
> > Not me ... But yeah, agree that it seems likely to require some
> > refactoring, even if _some_ degree of attempts were made to keep arch-
> > specific things there ...
> >
> > There might be a ton of arch-specific things in the ptrace code too
>
> At least a lot of the register handling, but that will always be
> somewhat architecture specific.
>
> > ... , so perhaps it'd be easier to start with the seccomp-based model:
> >
> > https://patchwork.ozlabs.org/project/linux-um/list/?series=329466
> >
> > But that's not really done yet either.
>
> Yes, though I have quite some updates locally that solve the security
> issues (i.e. prevent host syscalls from userspace). So, it does need
> rebasing and I am sure bugfixes, but I think it is getting into a
> pretty good shape overall.
>
> Either way, the old patchset will give you a good idea about how it all
> works, the changes are mostly in the details. I am happy to push out a
> new version sooner rather than later if it might help with any efforts
> on your side.

From a quick scan, it looks like there's some cleanups in the series
which would be helpful without seccomp parts. One of the initial
issues I've found is UML using older ptrace interfaces which arm64
doesn't implement. PTRACE_GETREGS for example.

Rob

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Johannes Berg]

On Fri, 2023-06-23 at 16:34 -0600, Rob Herring wrote:
> 
> > 
> > Either way, the old patchset will give you a good idea about how it all
> > works, the changes are mostly in the details. I am happy to push out a
> > new version sooner rather than later if it might help with any efforts
> > on your side.
> 
> From a quick scan, it looks like there's some cleanups in the series
> which would be helpful without seccomp parts. One of the initial
> issues I've found is UML using older ptrace interfaces which arm64
> doesn't implement. PTRACE_GETREGS for example.
> 

I don't think that completely gets rid of PTRACE_GETREGS though, and if
I remember correctly, we really kind of need that there?

Though then again it's all been a while, and I only faulted the seccomp
mode back in in discussions with Benjamin. Looks like we've found a
potentially nicer way to make it secure than his secret-based approach,
and in fact in a way that should even make it SMP-safe, at least in
theory, obviously a lot of infrastructure is missing to make it SMP in
the first place.

Currently, UML has a host process per VMA. Obviously, you need multiple
host processes for SMP (to get SMP), i.e. one per (used) CPU per VMA,
with CLONE_VM.

The problem with the secrets-based approach here for SMP is that the
secret will be readable to the other running in the VMA (1) and then can
be used for circumventing the protection by jumping into the stub area
and calling host syscalls, see
https://patchwork.ozlabs.org/project/linux-um/patch/20221122100759.208290-28-benjamin@sipsolutions.net/
and
https://patchwork.ozlabs.org/project/linux-um/patch/20221122100759.208290-24-benjamin@sipsolutions.net/


Now the new idea we came up with is this: We can make the per-CPU VMA
with CLONE_VM but *not* CLONE_FILES. Then, in the stub, when we need to
execute some real host syscalls on behalf of the child, we

 * send the FD over in a message
 * use the FD for mmap (and also always use mmap instead of mprotect)
 * close the FD

Without CLONE_FILES, another thread cannot "steal" the (real, host) FD,
it's useless in the other thread. Note that I'm talking about host FDs
here, in-UML FDs are just numbers and it works all differently, I'm just
talking about executing host syscalls inside the VMA, to set up the VMA
correctly etc.

The BPF program now allows any the relevant syscalls (2) inside the stub
area, but since you don't have the FD unless you actually executed the
recvmsg() call at the beginning of the stub you can't do anything with
that by jumping into the stub.

There are some other details involved such as having to split the stub
data into a read-only "what to execute" (3) and a writeable "results"
page, but those are reasonably easy to deal with.

At the end of the stub, of course the FD must be closed. This is fine
though since mappings persist after the FD was closed. On the next page
fault we have to do it all again, but yeah, page faults were always
super expensive in UML ...

johannes


(1) unless you extract it directly out of the BPF program into registers
via some magic syscalls that get a BPF return, but that's kind of icky
too

(2) mmap & munmap are the most relevant ones, some others but they're
not that critical for security; notably mprotect is not allowed but must
be done with mmap instead

(3) so another thread can't actually overwrite the instructions of what
the kernel wants to run inside the VMA process from another thread while
it's happening

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Benjamin Berg]

On Sat, 2023-06-24 at 15:15 +0200, Johannes Berg wrote:
> On Fri, 2023-06-23 at 16:34 -0600, Rob Herring wrote:
> > 
> > > 
> > > Either way, the old patchset will give you a good idea about how it all
> > > works, the changes are mostly in the details. I am happy to push out a
> > > new version sooner rather than later if it might help with any efforts
> > > on your side.
> > 
> > From a quick scan, it looks like there's some cleanups in the series
> > which would be helpful without seccomp parts. One of the initial
> > issues I've found is UML using older ptrace interfaces which arm64
> > doesn't implement. PTRACE_GETREGS for example.
> > 
> 
> I don't think that completely gets rid of PTRACE_GETREGS though, and if
> I remember correctly, we really kind of need that there?

The SECCOMP code should not need any ptrace at all. All it does is
read/write the mcontext that is generated by the host. I think there
was just some mangling there to map the basic registers into the format
that UML expects internally (floating point, SSE, etc. are just copied
directly though).

> Though then again it's all been a while, and I only faulted the seccomp
> mode back in in discussions with Benjamin. Looks like we've found a
> potentially nicer way to make it secure than his secret-based approach,
> and in fact in a way that should even make it SMP-safe, at least in
> theory, obviously a lot of infrastructure is missing to make it SMP in
> the first place.

Yeah, the idea with FD passing and CLONE_VM without CLONE_FILES does
indeed seem very promising both for a secure SECCOMP model and SMP
support specifically. It is should be much easier to implement than my
previous secret based syscall authentication idea.

Benjamin

> 
> Currently, UML has a host process per VMA. Obviously, you need multiple
> host processes for SMP (to get SMP), i.e. one per (used) CPU per VMA,
> with CLONE_VM.
> 
> The problem with the secrets-based approach here for SMP is that the
> secret will be readable to the other running in the VMA (1) and then can
> be used for circumventing the protection by jumping into the stub area
> and calling host syscalls, see
> https://patchwork.ozlabs.org/project/linux-um/patch/20221122100759.208290-28-benjamin@sipsolutions.net/
> and
> https://patchwork.ozlabs.org/project/linux-um/patch/20221122100759.208290-24-benjamin@sipsolutions.net/
> 
> 
> Now the new idea we came up with is this: We can make the per-CPU VMA
> with CLONE_VM but *not* CLONE_FILES. Then, in the stub, when we need to
> execute some real host syscalls on behalf of the child, we
> 
>  * send the FD over in a message
>  * use the FD for mmap (and also always use mmap instead of mprotect)
>  * close the FD
> 
> Without CLONE_FILES, another thread cannot "steal" the (real, host) FD,
> it's useless in the other thread. Note that I'm talking about host FDs
> here, in-UML FDs are just numbers and it works all differently, I'm just
> talking about executing host syscalls inside the VMA, to set up the VMA
> correctly etc.
> 
> The BPF program now allows any the relevant syscalls (2) inside the stub
> area, but since you don't have the FD unless you actually executed the
> recvmsg() call at the beginning of the stub you can't do anything with
> that by jumping into the stub.
> 
> There are some other details involved such as having to split the stub
> data into a read-only "what to execute" (3) and a writeable "results"
> page, but those are reasonably easy to deal with.
> 
> At the end of the stub, of course the FD must be closed. This is fine
> though since mappings persist after the FD was closed. On the next page
> fault we have to do it all again, but yeah, page faults were always
> super expensive in UML ...
> 
> johannes
> 
> 
> (1) unless you extract it directly out of the BPF program into registers
> via some magic syscalls that get a BPF return, but that's kind of icky
> too
> 
> (2) mmap & munmap are the most relevant ones, some others but they're
> not that critical for security; notably mprotect is not allowed but must
> be done with mmap instead
> 
> (3) so another thread can't actually overwrite the instructions of what
> the kernel wants to run inside the VMA process from another thread while
> it's happening
> 


_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um

Re: UML for arm64

[Johannes Berg]

On Sat, 2023-06-24 at 22:05 +0200, Benjamin Berg wrote:
> On Sat, 2023-06-24 at 15:15 +0200, Johannes Berg wrote:
> > On Fri, 2023-06-23 at 16:34 -0600, Rob Herring wrote:
> > > 
> > > > 
> > > > Either way, the old patchset will give you a good idea about how it all
> > > > works, the changes are mostly in the details. I am happy to push out a
> > > > new version sooner rather than later if it might help with any efforts
> > > > on your side.
> > > 
> > > From a quick scan, it looks like there's some cleanups in the series
> > > which would be helpful without seccomp parts. One of the initial
> > > issues I've found is UML using older ptrace interfaces which arm64
> > > doesn't implement. PTRACE_GETREGS for example.
> > > 
> > 
> > I don't think that completely gets rid of PTRACE_GETREGS though, and if
> > I remember correctly, we really kind of need that there?
> 
> The SECCOMP code should not need any ptrace at all.
> 

Indeed. For the record, I was referring to "some cleanups in the series"
not getting rid of PTRACE_GETREGS entirely :-)

There are indeed some instances of PTRACE_GETREGS removed in your
patches, however, I believe that they're still needed (in ptrace mode)
after that. If that's true, then it seems like the ptrace mode can't
really be ported to arm64.

Then again, a bunch of the stuff you removed there is for detecting
sysemu support, which is documented to only exist on x86, which I'm not
sure is true because arm64 defines PTRACE_SYSEMU[_SINGLESTEP] and
TIF_SYSCALL_EMU and then the generic ptrace code should actually
implement it?

However, PTRACE_GETREGS is still invoked unconditionally in the
userspace() loop afaict, and thus it can't actually work?


So seems that the best bet for other architectures anyway would be
seccomp mode anyway, since that's pretty well supported across
architectures.

You have some endianness bugs in the BPF code for 64-bit code though, it
only works on little endian ;-)

But there are good reasons to use it anyway, vs. ptrace.

> All it does is
> read/write the mcontext that is generated by the host. I think there
> was just some mangling there to map the basic registers into the format
> that UML expects internally (floating point, SSE, etc. are just copied
> directly though).

Yeah that sounds about right.

> > Though then again it's all been a while, and I only faulted the seccomp
> > mode back in in discussions with Benjamin. Looks like we've found a
> > potentially nicer way to make it secure than his secret-based approach,
> > and in fact in a way that should even make it SMP-safe, at least in
> > theory, obviously a lot of infrastructure is missing to make it SMP in
> > the first place.
> 
> Yeah, the idea with FD passing and CLONE_VM without CLONE_FILES does
> indeed seem very promising both for a secure SECCOMP model and SMP
> support specifically.

I don't believe we'll get SMP support easily one way or the other, but
it's nice to know that seccomp mode won't actively break it :-)

johannes

_______________________________________________
linux-um mailing list
linux-um@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-um