Re: EFI runtime and kexec

Re: EFI runtime and kexec

[H. Peter Anvin]

On 03/01/2013 01:39 PM, Borislav Petkov wrote:
> Hi guys,
> 
> so I was talking with mfleming on IRC and he said I should talk to you
> about it. I actually pestered hpa about it already, sorry :).
> 
> So I've been looking into making EFI runtime services available
> in the kexec'ed kernel. What I've found out so far is that
> efi_enter_virtual_mode() in the first kernel iterates over the EFI
> memmap and ioremaps all those EFI runtime services mappings. On my DELL
> workstation it looks like the dump below.
> 
> Now, once this is done and SetVirtualAddressMap() is called, for the
> duration of this boot, those virtual addresses cannot change as the UEFI
> spec states: "A virtual address map may only be applied one time. Once
> the runtime system is in virtual mode, calls to this function return
> EFI_UNSUPPORTED."
> 
> Now, looking at those mappings, they're spread all over the VA space and
> their size is ~159.887Mb (which is 159MB too many for a goddam BIOS crap
> but whatever, everyone is jumping on this train so I'm gonna have to
> follow, unwillingly.</EndOfRant>)
> 
> AFAICT, in a kexec kernel I'd have to recreate the exact-same mappings,
> i.e. phys_addr -> va for all those regions. And since I'm not an mm guy,
> I'd rather ask the experts before I dive into a catch-22 thing.
> 
> So even if I manage to do the mappings in the kexec kernel correctly for
> all those regions which efi_ioremap() serves only with direct mappings
> through init_memory_mapping(), I probably won't be that lucky with
> regions of type EFI_MEMORY_MAPPED_IO (type 0xb below) for which we
> really do ioremap and those virtual addresses I can't control, AFAICT.
> 
> So what do you guys think, would it be possible
> 
> * to make all EFI runtime services use predefined mappings which are
> globally valid and I can read them out in kexec or
> 
> * make those mappings virtually contiguous so that kexec kernel only
> gets a va_start and a size and after that it knows what to do or
> 
> * an even better idea.
> 
> In general, any suggestion is appreciated.
> 

Adding a few more people.

This has been a big topic, and yes, we have a problem.

We seem to have a few options:

1. We could always map 1:1, with the EFI mappings being in the "user"
part of the virtual address space.  This MAY be what Windows does
already.  Some Apple platforms are known to fail in this configuration,
but perhaps we can blacklist those platforms or do something special.

2. We could always map them into a fixed address that can be relied upon
to be consistent.  The most logical such area is the second quadrant of
the address space (again, in the "user" portion.)  It would be
beneficial if we could define it so that whenever Linux needs to go to
more than 48 virtual address bits at some point in the future this can
be compatible between 48-bit and N-bit kernels, but if that is the only
thing that breaks, then oh well.

3. We could just always map at the kernel virtual address.  The 64-bit
address space is large enough that we could make every ioremap() land at
its identity-mapped address instead of in a unique part of the virtual
address space.

4. We could export a table of mappings to the kexec'd kernel.  In that
case, we have to re-establish those mappings very early in the kernel
boot so that nothing else steps on them.

What is quite interesting in your case is that you have a mishmash of
the identity-mapped and the non-identity-mapped mappings.

	-hpa

Re: EFI runtime and kexec

[Borislav Petkov]

On Fri, Mar 01, 2013 at 02:32:47PM -0800, H. Peter Anvin wrote:
> On 03/01/2013 01:39 PM, Borislav Petkov wrote:
> > Hi guys,
> > 
> > so I was talking with mfleming on IRC and he said I should talk to you
> > about it. I actually pestered hpa about it already, sorry :).
> > 
> > So I've been looking into making EFI runtime services available
> > in the kexec'ed kernel. What I've found out so far is that
> > efi_enter_virtual_mode() in the first kernel iterates over the EFI
> > memmap and ioremaps all those EFI runtime services mappings. On my DELL
> > workstation it looks like the dump below.
> > 
> > Now, once this is done and SetVirtualAddressMap() is called, for the
> > duration of this boot, those virtual addresses cannot change as the UEFI
> > spec states: "A virtual address map may only be applied one time. Once
> > the runtime system is in virtual mode, calls to this function return
> > EFI_UNSUPPORTED."
> > 
> > Now, looking at those mappings, they're spread all over the VA space and
> > their size is ~159.887Mb (which is 159MB too many for a goddam BIOS crap
> > but whatever, everyone is jumping on this train so I'm gonna have to
> > follow, unwillingly.</EndOfRant>)
> > 
> > AFAICT, in a kexec kernel I'd have to recreate the exact-same mappings,
> > i.e. phys_addr -> va for all those regions. And since I'm not an mm guy,
> > I'd rather ask the experts before I dive into a catch-22 thing.
> > 
> > So even if I manage to do the mappings in the kexec kernel correctly for
> > all those regions which efi_ioremap() serves only with direct mappings
> > through init_memory_mapping(), I probably won't be that lucky with
> > regions of type EFI_MEMORY_MAPPED_IO (type 0xb below) for which we
> > really do ioremap and those virtual addresses I can't control, AFAICT.
> > 
> > So what do you guys think, would it be possible
> > 
> > * to make all EFI runtime services use predefined mappings which are
> > globally valid and I can read them out in kexec or
> > 
> > * make those mappings virtually contiguous so that kexec kernel only
> > gets a va_start and a size and after that it knows what to do or
> > 
> > * an even better idea.
> > 
> > In general, any suggestion is appreciated.
> > 
> 
> Adding a few more people.
> 
> This has been a big topic, and yes, we have a problem.
> 
> We seem to have a few options:
> 
> 1. We could always map 1:1, with the EFI mappings being in the "user"
> part of the virtual address space.  This MAY be what Windows does
> already.  Some Apple platforms are known to fail in this configuration,
> but perhaps we can blacklist those platforms or do something special.
> 
> 2. We could always map them into a fixed address that can be relied upon
> to be consistent.  The most logical such area is the second quadrant of
> the address space (again, in the "user" portion.)  It would be
> beneficial if we could define it so that whenever Linux needs to go to
> more than 48 virtual address bits at some point in the future this can
> be compatible between 48-bit and N-bit kernels, but if that is the only
> thing that breaks, then oh well.
> 
> 3. We could just always map at the kernel virtual address.  The 64-bit
> address space is large enough that we could make every ioremap() land at
> its identity-mapped address instead of in a unique part of the virtual
> address space.
> 
> 4. We could export a table of mappings to the kexec'd kernel.  In that
> case, we have to re-establish those mappings very early in the kernel
> boot so that nothing else steps on them.
> 
> What is quite interesting in your case is that you have a mishmash of
> the identity-mapped and the non-identity-mapped mappings.

Yeah, the mishmash comes from regions of type EFI_MEMORY_MAPPED_IO which
are really ioremapped instead of returning the kernel virtual address.

Btw, I always tend to like the simplest approaches so option 3.
is kinda winking at me right now. I don't know whether for those
EFI_MEMORY_MAPPED_IO type regions though, we can simply return the
identity-mapped address.

If we can, the advantage would be great because then the kexec kernel
would simply parse the efi memmap and use __va() on the physical
addresses there and no need for special option passing to it.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: EFI runtime and kexec

[H. Peter Anvin]

On 03/01/2013 02:53 PM, Borislav Petkov wrote:
>>
>> Adding a few more people.
>>
>> This has been a big topic, and yes, we have a problem.
>>
>> We seem to have a few options:
>>
>> 1. We could always map 1:1, with the EFI mappings being in the "user"
>> part of the virtual address space.  This MAY be what Windows does
>> already.  Some Apple platforms are known to fail in this configuration,
>> but perhaps we can blacklist those platforms or do something special.
>>
>> 2. We could always map them into a fixed address that can be relied upon
>> to be consistent.  The most logical such area is the second quadrant of
>> the address space (again, in the "user" portion.)  It would be
>> beneficial if we could define it so that whenever Linux needs to go to
>> more than 48 virtual address bits at some point in the future this can
>> be compatible between 48-bit and N-bit kernels, but if that is the only
>> thing that breaks, then oh well.
>>
>> 3. We could just always map at the kernel virtual address.  The 64-bit
>> address space is large enough that we could make every ioremap() land at
>> its identity-mapped address instead of in a unique part of the virtual
>> address space.
>>
>> 4. We could export a table of mappings to the kexec'd kernel.  In that
>> case, we have to re-establish those mappings very early in the kernel
>> boot so that nothing else steps on them.
>>
>> What is quite interesting in your case is that you have a mishmash of
>> the identity-mapped and the non-identity-mapped mappings.
> 
> Yeah, the mishmash comes from regions of type EFI_MEMORY_MAPPED_IO which
> are really ioremapped instead of returning the kernel virtual address.
> 
> Btw, I always tend to like the simplest approaches so option 3.
> is kinda winking at me right now. I don't know whether for those
> EFI_MEMORY_MAPPED_IO type regions though, we can simply return the
> identity-mapped address.
> 
> If we can, the advantage would be great because then the kexec kernel
> would simply parse the efi memmap and use __va() on the physical
> addresses there and no need for special option passing to it.
> 

We can, and in fact we could do this for *all* ioremap()s in the 64-bit
kernel.  This doesn't help the 32-bit kernel in any way, however.

One thing I *really* don't like about it is that it exposes the kernel
virtual address map as an ABI.

	-hpa

Re: EFI runtime and kexec

[Borislav Petkov]

On Fri, Mar 01, 2013 at 02:58:56PM -0800, H. Peter Anvin wrote:
> >> We seem to have a few options:
> >>
> >> 1. We could always map 1:1, with the EFI mappings being in the "user"
> >> part of the virtual address space.  This MAY be what Windows does
> >> already.  Some Apple platforms are known to fail in this configuration,
> >> but perhaps we can blacklist those platforms or do something special.
> >>
> >> 2. We could always map them into a fixed address that can be relied upon
> >> to be consistent.  The most logical such area is the second quadrant of
> >> the address space (again, in the "user" portion.)  It would be
> >> beneficial if we could define it so that whenever Linux needs to go to
> >> more than 48 virtual address bits at some point in the future this can
> >> be compatible between 48-bit and N-bit kernels, but if that is the only
> >> thing that breaks, then oh well.
> >>
> >> 3. We could just always map at the kernel virtual address.  The 64-bit
> >> address space is large enough that we could make every ioremap() land at
> >> its identity-mapped address instead of in a unique part of the virtual
> >> address space.
> >>
> >> 4. We could export a table of mappings to the kexec'd kernel.  In that
> >> case, we have to re-establish those mappings very early in the kernel
> >> boot so that nothing else steps on them.
> >>
> >> What is quite interesting in your case is that you have a mishmash of
> >> the identity-mapped and the non-identity-mapped mappings.
> > 
> > Yeah, the mishmash comes from regions of type EFI_MEMORY_MAPPED_IO which
> > are really ioremapped instead of returning the kernel virtual address.
> > 
> > Btw, I always tend to like the simplest approaches so option 3.
> > is kinda winking at me right now. I don't know whether for those
> > EFI_MEMORY_MAPPED_IO type regions though, we can simply return the
> > identity-mapped address.
> > 
> > If we can, the advantage would be great because then the kexec kernel
> > would simply parse the efi memmap and use __va() on the physical
> > addresses there and no need for special option passing to it.
> > 
> 
> We can, and in fact we could do this for *all* ioremap()s in the 64-bit
> kernel.  This doesn't help the 32-bit kernel in any way, however.

Right, ok.

> One thing I *really* don't like about it is that it exposes the kernel
> virtual address map as an ABI.

Hmm, yeah, that's nasty. This also means option #2 can go too because
of the fixed addresses. Option #1 is also kinda polluting user address
space so maybe the most elegant one would be #4, AFAICT.

We just need a nice mechanism to tell those mappings to the kexec-d
kernel and when it starts, to establish them right away.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: EFI runtime and kexec

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

On Sat, 2013-03-02 at 00:07 +0100, Borislav Petkov wrote:
> Hmm, yeah, that's nasty. This also means option #2 can go too because
> of the fixed addresses. Option #1 is also kinda polluting user address
> space 

User address space is there to be polluted. Create a "kernel thread" for
invoking EFI, except that this kernel thread actually has userspace page
tables. Set up those page tables however the hell you like, and then
just make sure you always invoke EFI runtime services from that thread.

> so maybe the most elegant one would be #4, AFAICT.
> 
> We just need a nice mechanism to tell those mappings to the kexec-d
> kernel and when it starts, to establish them right away.

Well, there's a *shitload* of things we already need to hand off to the
kexec'd kernel that we don't. We don't pass the EFI system table, which
means that even the ACPI tables aren't found by the child kernel. And
all of the setup_data is missing. And there's probably more.

I was looking at this, but there was *other* breakage with kexec which
was being sorted out at the time, and I kind of got distracted into
doing CSM support in SeaBIOS.

The other option, for the long term, is to fix the damn firmware to
allow SetVirtualAddressMap to be called more than once. It was stupid
for it to be a one-time call anyway. I have a test build of OVMF here
which does just that, but haven't got round to testing it yet. I wish
I'd been given the *patch* not just a binary though...

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: EFI runtime and kexec

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 1054 bytes --]

On Fri, 2013-03-01 at 23:30 +0000, David Woodhouse wrote:
> On Sat, 2013-03-02 at 00:07 +0100, Borislav Petkov wrote:
> > Hmm, yeah, that's nasty. This also means option #2 can go too because
> > of the fixed addresses. Option #1 is also kinda polluting user address
> > space 
> 
> User address space is there to be polluted. Create a "kernel thread" for
> invoking EFI, except that this kernel thread actually has userspace page
> tables. Set up those page tables however the hell you like, and then
> just make sure you always invoke EFI runtime services from that thread.

Oh, I meant to mention: Matthew once told me about a bizarre issue on
MacBook firmware, iirc. Some stuff doesn't work if the virtual address
is above 2GiB. Or below 2GiB. I don't really remember what Matthew told
me at all, to be honest. Only that Here Be Dragons and we don't quite
have free rein about where we place stuff. But I suspect we ought to be
able to find *somewhere* in the user address space that works, even for
32-bit kernels.

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: EFI runtime and kexec

[Matthew Garrett]

On Fri, Mar 01, 2013 at 11:34:13PM +0000, David Woodhouse wrote:

> Oh, I meant to mention: Matthew once told me about a bizarre issue on
> MacBook firmware, iirc. Some stuff doesn't work if the virtual address
> is above 2GiB. Or below 2GiB. I don't really remember what Matthew told
> me at all, to be honest. Only that Here Be Dragons and we don't quite
> have free rein about where we place stuff. But I suspect we ought to be
> able to find *somewhere* in the user address space that works, even for
> 32-bit kernels.

Apple's enough of a special case that I've no objection to 
special-casing them.

-- 
Matthew Garrett | mjg59-1xO5oi07KQx4cg9Nei1l7Q@public.gmane.org

Re: EFI runtime and kexec

[Borislav Petkov]

Just commenting on this one for now, the rest tomorrow cuz I'm half
asleep.

On Fri, Mar 01, 2013 at 11:30:25PM +0000, David Woodhouse wrote:
> The other option, for the long term, is to fix the damn firmware to
> allow SetVirtualAddressMap to be called more than once. It was stupid
> for it to be a one-time call anyway.

Now this would be the cleanest solution. If we can do that, we can then
simply call efi_enter_virtual_mode() in the kexec'd kernel without
the need to pass any options to it. Actually, the kexec'd kernel can
probably run the same efi code as the papa kernel.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: EFI runtime and kexec

[H. Peter Anvin]

On 03/01/2013 03:39 PM, Borislav Petkov wrote:
> Just commenting on this one for now, the rest tomorrow cuz I'm half
> asleep.
> 
> On Fri, Mar 01, 2013 at 11:30:25PM +0000, David Woodhouse wrote:
>> The other option, for the long term, is to fix the damn firmware to
>> allow SetVirtualAddressMap to be called more than once. It was stupid
>> for it to be a one-time call anyway.
> 
> Now this would be the cleanest solution. If we can do that, we can then
> simply call efi_enter_virtual_mode() in the kexec'd kernel without
> the need to pass any options to it. Actually, the kexec'd kernel can
> probably run the same efi code as the papa kernel.
> 

Well, if we can genuinely run with a zero relocation we don't even need
to worry about that.

	-hpa

Re: EFI runtime and kexec

[H. Peter Anvin]

On 03/01/2013 03:07 PM, Borislav Petkov wrote:
> 
> Hmm, yeah, that's nasty. This also means option #2 can go too because
> of the fixed addresses. Option #1 is also kinda polluting user address
> space so maybe the most elegant one would be #4, AFAICT.
> 

No, it doesn't pollute the user address space, because the EFI BIOS is
effectively a process.  We *have* to switch cr3 anyway around EFI calls,
because BIOSes in the field are known to use physical addresses instead
of virtual.

If Windows really does map EFI BIOS 1:1 then that explains why, and is
probably going to be the stable thing going forward.

	-hpa

Re: EFI runtime and kexec

[H. Peter Anvin]

On 03/01/2013 03:30 PM, David Woodhouse wrote:
> On Sat, 2013-03-02 at 00:07 +0100, Borislav Petkov wrote:
>> Hmm, yeah, that's nasty. This also means option #2 can go too because
>> of the fixed addresses. Option #1 is also kinda polluting user address
>> space 
> 
> User address space is there to be polluted. Create a "kernel thread" for
> invoking EFI, except that this kernel thread actually has userspace page
> tables. Set up those page tables however the hell you like, and then
> just make sure you always invoke EFI runtime services from that thread.
> 

Just FYI: as far as I know we already have such a "thread".

	-hpa


-- 
H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Re: EFI runtime and kexec

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Fri, 2013-03-01 at 17:11 -0800, H. Peter Anvin wrote:
> On 03/01/2013 03:30 PM, David Woodhouse wrote:
> > On Sat, 2013-03-02 at 00:07 +0100, Borislav Petkov wrote:
> >> Hmm, yeah, that's nasty. This also means option #2 can go too because
> >> of the fixed addresses. Option #1 is also kinda polluting user address
> >> space 
> > 
> > User address space is there to be polluted. Create a "kernel thread" for
> > invoking EFI, except that this kernel thread actually has userspace page
> > tables. Set up those page tables however the hell you like, and then
> > just make sure you always invoke EFI runtime services from that thread.
> > 
> 
> Just FYI: as far as I know we already have such a "thread".

In that case it seems like this option is fairly much a no-brainer.

Do we make any calls to runtime services during early boot (before we'd
have such a thread set up for use)?

-- 
dwmw2


[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 6171 bytes --]

Re: EFI runtime and kexec

[H. Peter Anvin]

Yes, but way before... not a problem.  It isn't a thread in the scheduler sense.

David Woodhouse <dwmw2-wEGCiKHe2LqWVfeAwA7xHQ@public.gmane.org> wrote:

>On Fri, 2013-03-01 at 17:11 -0800, H. Peter Anvin wrote:
>> On 03/01/2013 03:30 PM, David Woodhouse wrote:
>> > On Sat, 2013-03-02 at 00:07 +0100, Borislav Petkov wrote:
>> >> Hmm, yeah, that's nasty. This also means option #2 can go too
>because
>> >> of the fixed addresses. Option #1 is also kinda polluting user
>address
>> >> space 
>> > 
>> > User address space is there to be polluted. Create a "kernel
>thread" for
>> > invoking EFI, except that this kernel thread actually has userspace
>page
>> > tables. Set up those page tables however the hell you like, and
>then
>> > just make sure you always invoke EFI runtime services from that
>thread.
>> > 
>> 
>> Just FYI: as far as I know we already have such a "thread".
>
>In that case it seems like this option is fairly much a no-brainer.
>
>Do we make any calls to runtime services during early boot (before we'd
>have such a thread set up for use)?

-- 
Sent from my mobile phone. Please excuse brevity and lack of formatting.

Re: EFI runtime and kexec

[Borislav Petkov]

On Fri, Mar 01, 2013 at 06:04:11PM -0800, H. Peter Anvin wrote:
> Yes, but way before... not a problem. It isn't a thread in the
> scheduler sense.

Are you so enigmatically referring to the swapper?

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--