Extending boot protocol & bzImage for paravirt_ops

Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

Well, it seems to be about time to have this conversation again.

A rough overview of the previous thread and requirements is:

   1. bzImage would not be a bare ELF file, but it would contain an ELF
      header+file within it
   2. We need some way to add extra ELF notes into that ELF file
   3. We use a "paravirtualized" bootloader type, with some other field
      to determine which hypervisor we're dealing with
   4. We need to extend the boot info structure to include a pointer to
      some other hypervisor-specific information
   5. When started at the 32-bit entrypoint, bzImage expects:
         1. 32-bit mode
         2. running in any ring (and ring implies nothing about the
            environment)
         3. all segment registers are loaded with flat 4G segments
         4. there may or may not be a valid GDT (segment registers must
            not be reloaded until GDT is explicitly set up)
         5. interrupts disabled
         6. paging may or may not be enabled; if enabled, it has 1:1
            mappings where the ELF file's Phdrs say they should be
   6. When bzImage starts the kernel-proper, it works out how to choose
      the appropriate boot path based on the boot info structure in %esi

How much of this already exists?  Is there enough to start prototyping with?

What was the problem with ELF bzImage?  Is it confirmed to be
problematic, or just suspected?  Could we end up using a plain ELF file
(which would be easiest for me, since I think the existing ELF domain
builder could pretty much deal with that as-is).

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[Eric W. Biederman]

Jeremy Fitzhardinge <jeremy@goop.org> writes:

> Well, it seems to be about time to have this conversation again.
>
> A rough overview of the previous thread and requirements is:
>
>    1. bzImage would not be a bare ELF file, but it would contain an ELF
>       header+file within it
>    2. We need some way to add extra ELF notes into that ELF file

>    3. We use a "paravirtualized" bootloader type, with some other field
>       to determine which hypervisor we're dealing with
>    4. We need to extend the boot info structure to include a pointer to
>       some other hypervisor-specific information
>    5. When started at the 32-bit entrypoint, bzImage expects:
>          1. 32-bit mode
>          2. running in any ring (and ring implies nothing about the
>             environment)
>          3. all segment registers are loaded with flat 4G segments
>          4. there may or may not be a valid GDT (segment registers must
>             not be reloaded until GDT is explicitly set up)
>          5. interrupts disabled
>          6. paging may or may not be enabled; if enabled, it has 1:1
>             mappings where the ELF file's Phdrs say they should be
>    6. When bzImage starts the kernel-proper, it works out how to choose
>       the appropriate boot path based on the boot info structure in %esi

Yes.

> How much of this already exists?  Is there enough to start prototyping with?
>
>
> What was the problem with ELF bzImage?  Is it confirmed to be
> problematic, or just suspected?

There was a problem, Andrews machine would not the kernel with the ELF
header.  It was not root caused.  So we are not certain why.

> Could we end up using a plain ELF file
> (which would be easiest for me, since I think the existing ELF domain
> builder could pretty much deal with that as-is).

I think we should start there and be prepared to munge the ELF
magic number if there are problems.  Placing ELF notes is going
to require a little work but mostly it should be a matter of
copying the from vmlinux.

Most of that though is just packaging.  The meat of the issue
is how do we upgrade the bootloader data.  Do the changes
below sound like everything we need?

 Field name:	loadflags
 Type:		modify (obligatory)
 Offset/size:	0x211/1
 Protocol:	2.00+
 
   This field is a bitmask.
 
   Bit 0 (read):	LOADED_HIGH
 	- If 0, the protected-mode code is loaded at 0x10000.
 	- If 1, the protected-mode code is loaded at 0x100000.

+  Bit 6 (write): KEEP_SEGMENTS
+	Protocol: 2.07+
+	- if 0, reload the segment registers in the 32bit entry point.
+	- if 1, do not reload the segment registers in the 32bit entry point.
+		Assume that %cs %ds %ss %es are all set to flat segments with
+		a base of 0 (or the equivalent for their environment).
 
   Bit 7 (write): CAN_USE_HEAP
 	Set this bit to 1 to indicate that the value entered in the
 	heap_end_ptr is valid.  If this field is clear, some setup code
 	functionality will be disabled.

.....
+Field name:	hardware_subarch
+Type:		write
+Offset/size:	0x23c/4
+Protocol:	2.07+
+
+  In a paravirtualized environment the hardware low level architectural 
+  pieces such as interrupt handling, page table handling, and
+  accessing process control registers needs to be done differently.
+
+  This field allows the bootloader to inform the kernel we are in one
+  one of those environments.
+
+  0x00000000	The default x86/PC environment
+  0x00000001	lguest
+  0x00000002	Xen
+
+Field name:	hardware_subarch_data
+Type:		write
+Offset/size:	0x23c/8
+Protocol:	2.07+
+
+  A pointer to data that is specific to hardware subarch
+

Eric

Re: Extending boot protocol & bzImage for paravirt_ops

[Rusty Russell]

On Fri, 2007-05-25 at 10:46 -0600, Eric W. Biederman wrote:
> Most of that though is just packaging.  The meat of the issue
> is how do we upgrade the bootloader data.  Do the changes
> below sound like everything we need?
> 
>  Field name:	loadflags
>  Type:		modify (obligatory)
>  Offset/size:	0x211/1
>  Protocol:	2.00+
>  
>    This field is a bitmask.
>  
>    Bit 0 (read):	LOADED_HIGH
>  	- If 0, the protected-mode code is loaded at 0x10000.
>  	- If 1, the protected-mode code is loaded at 0x100000.
> 
> +  Bit 6 (write): KEEP_SEGMENTS
> +	Protocol: 2.07+
> +	- if 0, reload the segment registers in the 32bit entry point.
> +	- if 1, do not reload the segment registers in the 32bit entry point.
> +		Assume that %cs %ds %ss %es are all set to flat segments with
> +		a base of 0 (or the equivalent for their environment).

You also want to skip the cli: perhaps a separate flag for this is
appropriate though.

Rest looks fine from an lguest POV.  We don't need the platform data
field either, since we use the first hypercall to get that).

Cheers,
Rusty.

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Rusty Russell wrote:
>>
>> +  Bit 6 (write): KEEP_SEGMENTS
>> +	Protocol: 2.07+
>> +	- if 0, reload the segment registers in the 32bit entry point.
>> +	- if 1, do not reload the segment registers in the 32bit entry point.
>> +		Assume that %cs %ds %ss %es are all set to flat segments with
>> +		a base of 0 (or the equivalent for their environment).
> 
> You also want to skip the cli: perhaps a separate flag for this is
> appropriate though.
> 

Do we have *any* environment that enters the 32-bit code with interrupts
enabled?  I would think that is unsafe no matter how you do it.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

Rusty Russell wrote:
> You also want to skip the cli: perhaps a separate flag for this is
> appropriate though.
>   

Not a problem for Xen; it will be trapped+emulated (ie, ignored).

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[Rusty Russell]

On Sun, 2007-05-27 at 00:47 +0100, Jeremy Fitzhardinge wrote:
> Rusty Russell wrote:
> > You also want to skip the cli: perhaps a separate flag for this is
> > appropriate though.
> >   
> 
> Not a problem for Xen; it will be trapped+emulated (ie, ignored).

Well, I could do that for lguest as a special case, I guess.  Seems icky
tho.

hpa wrote:
> Do we have *any* environment that enters the 32-bit code with interrupts
> enabled?  I would think that is unsafe no matter how you do it.

Yeah, the cli there seems strange: maybe it's safe to simply delete it?

Cheers,
Rusty.

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Rusty Russell wrote:
> 
> hpa wrote:
>> Do we have *any* environment that enters the 32-bit code with interrupts
>> enabled?  I would think that is unsafe no matter how you do it.
> 
> Yeah, the cli there seems strange: maybe it's safe to simply delete it?
> 

I suspect it was simply put there as a "belts and suspenders" kind of
thing, which is perfectly appropriate.  However, I would be highly
surprised if it would have any users.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
>> What was the problem with ELF bzImage?  Is it confirmed to be
>> problematic, or just suspected?
>>     
>
> There was a problem, Andrews machine would not the kernel with the ELF
> header.  It was not root caused.  So we are not certain why.
>   

The Vaio?  Do you think we could use it as kindling when we "deal with"
James's Voyagers?


>> Could we end up using a plain ELF file
>> (which would be easiest for me, since I think the existing ELF domain
>> builder could pretty much deal with that as-is).
>>     
>
> I think we should start there and be prepared to munge the ELF
> magic number if there are problems.  Placing ELF notes is going
> to require a little work but mostly it should be a matter of
> copying the from vmlinux.
>   

I'd really prefer to keep an intact embedded ELF file rather than have a
semi-ELF file.  That way it would be easier to simply load the image and
point the normal ELF machinery at some offset into the file rather than
having to have a special handler for "\x7fBZE" or whatever.

> Most of that though is just packaging.  The meat of the issue
> is how do we upgrade the bootloader data.  Do the changes
> below sound like everything we need?
>   

How does the kernel specify what memory it requires to be mapped (both
for the image itself, and for any memory it needs to use to
decompress/relocate the final kernel image).

If its by using ELF Phdrs, then we need to:

   1. specify the constraints on what mappings are allowed,
   2. specify how the bootloader uses the mappings, and
   3. specify how the ELF file is to be found.

If the bzImage is really a plain ELF file, then 3 is a non-issue. 
Otherwise, should we allocate a field which specifies the offset of the
ELF header within the bzImage, and make all offsets contained in the ELF
header relative to the header itself rather than the overall bzImage?

As far as 1 and 2 go, I propose:

Requirements on the Phdrs in the image:

   1. the Phdrs must map the actual bzImage code and data, with
      appropriate filesz and memsz
   2. the Phdrs must also map all the other memory needed to
      decompress/relocate the kernel, with filesz=0 and memsz set
      appropriately
   3. the Phdr mappings must always be 1:1 (ie, P=V), and RW(X)
   4. the mappings must be as low as possible, to avoid any hole the
      hypervisor requires (if the mappings are too high, the bootloader
      may just fail to start the kernel)

Requirements for the bootloader:

   1. it is always correct for the bootloader to simply leave paging
      disabled; presumably the subarch will tell the kernel that it
      needs to set up its own initial mappings
   2. if paging is enabled, the bootloader creates mappings:
         1. with PAE set to match the kernel's requirements (this
            suggests that loadflags should have a PAE/non-PAE flag)
         2. all mappings are RWX (ie, RW, no NX) [ rationale: this means
            we don't need to deal with pages of different
            types/permissions at this stage of booting, which might be
            especially awkward when recycling pages ]
         3. mappings covering at least the ones specified by the Phdrs,
            but it is acceptible to over-map (for example, map the
            entire address space)
         4. Other non-1:1 mappings may exist outside the image's Phdrs
            (for example, containing the initial pagetables, hypervisor
            address space, and other mappings)


>  Field name:	loadflags
>  Type:		modify (obligatory)
>  Offset/size:	0x211/1
>  Protocol:	2.00+
>  
>    This field is a bitmask.
>  
>    Bit 0 (read):	LOADED_HIGH
>  	- If 0, the protected-mode code is loaded at 0x10000.
>  	- If 1, the protected-mode code is loaded at 0x100000.
>   
+ Bit 5 (read): PAE mode
+    If the bootloader starts kernel with paging enabled, then:
+        - if 0, enable non-PAE mode paging
+        - if 1, enable PAE mode paging

> +  Bit 6 (write): KEEP_SEGMENTS
> +	Protocol: 2.07+
> +	- if 0, reload the segment registers in the 32bit entry point.
>   

So does this mean the bootloader has set up a kernel-compatible GDT?  Or
does it also mean "set up a GDT"?

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> 
> I'd really prefer to keep an intact embedded ELF file rather than have a
> semi-ELF file.  That way it would be easier to simply load the image and
> point the normal ELF machinery at some offset into the file rather than
> having to have a special handler for "\x7fBZE" or whatever.
> 

Pardon me for saying it, but it's hardly a hardship to add support for a
single additional constant in existing code.  I'm starting to think this
argument is approaching not just the silly, but downright ridiculous.

Another option is to have the ELF header at a nonzero offset.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

H. Peter Anvin wrote:
> Pardon me for saying it, but it's hardly a hardship to add support for a
> single additional constant in existing code.  I'm starting to think this
> argument is approaching not just the silly, but downright ridiculous.
>   

Well, if the existing code is a standard libelf/libbfd, it could be tricky.

> Another option is to have the ELF header at a nonzero offset.
>   

Yes, that's exactly what I'm proposing.

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> I think we should start there and be prepared to munge the ELF
> magic number if there are problems.  Placing ELF notes is going
> to require a little work but mostly it should be a matter of
> copying the from vmlinux.
>   

Do you have a semi-current ELF bzImage patch on hand I can start playing
with?

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[Vivek Goyal]

On Wed, May 30, 2007 at 04:16:55PM -0700, Jeremy Fitzhardinge wrote:
> Eric W. Biederman wrote:
> > I think we should start there and be prepared to munge the ELF
> > magic number if there are problems.  Placing ELF notes is going
> > to require a little work but mostly it should be a matter of
> > copying the from vmlinux.
> >   
> 
> Do you have a semi-current ELF bzImage patch on hand I can start playing
> with?
>

Following is the link to the patch which was posted last time and made
into -mm. But andrew had some old Fedora machine (I think FC2) and some
lilo version on that machine which failed to boot, hence this patch was
dropped.

http://marc.info/?l=linux-kernel&m=115989722229057&w=2

Thanks
Vivek

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

Eric W. Biederman wrote:
> ....
> +Field name:	hardware_subarch
> +Type:		write
> +Offset/size:	0x23c/4
> +Protocol:	2.07+
> +
> +  In a paravirtualized environment the hardware low level architectural 
> +  pieces such as interrupt handling, page table handling, and
> +  accessing process control registers needs to be done differently.
> +
> +  This field allows the bootloader to inform the kernel we are in one
> +  one of those environments.
> +
> +  0x00000000	The default x86/PC environment
> +  0x00000001	lguest
> +  0x00000002	Xen
> +
> +Field name:	hardware_subarch_data
> +Type:		write
> +Offset/size:	0x23c/8
>   

offset = 240

> +Protocol:	2.07+
> +
> +  A pointer to data that is specific to hardware subarch
>   

Do we care particularly? If 8 bytes is enough for the subarch, do we
care whether its a pointer or literal? After all, this is just a private
channel between the bootloader and some subarch-specific piece of code
in the kernel.

J

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> 
>> +Protocol:	2.07+
>> +
>> +  A pointer to data that is specific to hardware subarch
>>   
> 
> Do we care particularly? If 8 bytes is enough for the subarch, do we
> care whether its a pointer or literal? After all, this is just a private
> channel between the bootloader and some subarch-specific piece of code
> in the kernel.
> 

I see two options: either we make it a pointer *and a length* so that a
loader can reshuffle it at will (that also implies no absolute pointers
within the data), or it's an opaque cookie anyway.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

H. Peter Anvin wrote:
>> Do we care particularly? If 8 bytes is enough for the subarch, do we
>> care whether its a pointer or literal? After all, this is just a private
>> channel between the bootloader and some subarch-specific piece of code
>> in the kernel.
>>
>>     
>
> I see two options: either we make it a pointer *and a length* so that a
> loader can reshuffle it at will (that also implies no absolute pointers
> within the data), or it's an opaque cookie anyway.
>   

No, it has to be completely opaque.  It might be a pointer to some
special shared memory or something, and not movable.

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> H. Peter Anvin wrote:
>>> Do we care particularly? If 8 bytes is enough for the subarch, do we
>>> care whether its a pointer or literal? After all, this is just a private
>>> channel between the bootloader and some subarch-specific piece of code
>>> in the kernel.
>>>
>>>     
>> I see two options: either we make it a pointer *and a length* so that a
>> loader can reshuffle it at will (that also implies no absolute pointers
>> within the data), or it's an opaque cookie anyway.
> 
> No, it has to be completely opaque.  It might be a pointer to some
> special shared memory or something, and not movable.
> 

Well, if we define is as a movable object then it has to be treated as
such.  It's a protocol definition issue.  If we define it opaque, though
-- of for that matter, if we don't -- we should define what memory it
can live in, though.  Right now, the only "available" memory we have is
end of setup to 0xa0000; the command line is defined to be allocated
from this memory.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

H. Peter Anvin wrote:
> Well, if we define is as a movable object then it has to be treated as
> such.  It's a protocol definition issue.  If we define it opaque, though
> -- of for that matter, if we don't -- we should define what memory it
> can live in, though.  Right now, the only "available" memory we have is
> end of setup to 0xa0000; the command line is defined to be allocated
> from this memory.
>   

Well, I think we can safely say that its something that's only
meaningful in 32/64-bit mode, so we aren't constrained by the real-mode
address space.

One of my goals in this project is to make the boot image, in some way,
completely define which memory it needs it get started.  That means that
the boot loader can either place things knowing they'll avoid the boot
image and/or definitively know that the image is unloadable.

So I don't think its strictly necessary to pre-define what memory this
object can use, since I think it can be safely determined dynamically.

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> 
> Well, I think we can safely say that its something that's only
> meaningful in 32/64-bit mode, so we aren't constrained by the real-mode
> address space.
> 
> One of my goals in this project is to make the boot image, in some way,
> completely define which memory it needs it get started.  That means that
> the boot loader can either place things knowing they'll avoid the boot
> image and/or definitively know that the image is unloadable.
> 
> So I don't think its strictly necessary to pre-define what memory this
> object can use, since I think it can be safely determined dynamically.
> 

That's a method of defining the memory space.

I think the current definition is suitable for entering at the 16-bit
entry point.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

H. Peter Anvin wrote:
> That's a method of defining the memory space.
>
> I think the current definition is suitable for entering at the 16-bit
> entry point.

I agree.  I'm going to assume that if we're booting all the way up from
real mode, we're either on real hardware, or some environment that's
trying hard to be real hardware.  In that case, there won't necessarily
be much need for the subarch-specific data, but even if there is, it can
be way out of the real-mode address space, and therefore be a non-issue
for 16-bit code.

Just to clarify:

In my proposal is that we have bzImage structured something like (where
"|" is concatenation, and "()" is a  blob containing stuff):

    bzImage = 16-bit setup | ELF file (decompressor, compressed kernel)
      

With the intention that 32-bit only bootloader always loads the ELF file
as-is and just runs it.  Aside from the fact that its an ELF file,
there's nothing else about it which really concerns the bootloader,
since once its loaded and running, it does all its own setup.  Its not
clear that code32_start really means much in this case, though I guess
it could point to the same place as the ELF file's entrypoint.

Whereas you're proposing:

    bzImage = 16-bit setup | decompressor | compressed kernel (ELF file)
      

where code32_start points to the decompressor, and some other pointer
points to the compressed kernel data.  And your intent is that an
external bootloader could also interpret the compressed kernel image,
and identify what format its in and handle it appropriately from
outside.  Right?

In both cases, it seems to me that we need an extra boot_param pointer
to point to the offset of the payload blob (ELF file in my case,
compressed kernel in yours).  Yes?

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[H. Peter Anvin]

Jeremy Fitzhardinge wrote:
> 
> Just to clarify:
> 
> In my proposal is that we have bzImage structured something like (where
> "|" is concatenation, and "()" is a  blob containing stuff):
> 
>     bzImage = 16-bit setup | ELF file (decompressor, compressed kernel)
>       
> 
> With the intention that 32-bit only bootloader always loads the ELF file
> as-is and just runs it.  Aside from the fact that its an ELF file,
> there's nothing else about it which really concerns the bootloader,
> since once its loaded and running, it does all its own setup.  Its not
> clear that code32_start really means much in this case, though I guess
> it could point to the same place as the ELF file's entrypoint.
> 

It would have to, because of the way code32_start is defined to work.
We don't get control again after its use as a hook.

> Whereas you're proposing:
> 
>     bzImage = 16-bit setup | decompressor | compressed kernel (ELF file)
>       
> 
> where code32_start points to the decompressor, and some other pointer
> points to the compressed kernel data.  And your intent is that an
> external bootloader could also interpret the compressed kernel image,
> and identify what format its in and handle it appropriately from
> outside.  Right?

Correct.

> In both cases, it seems to me that we need an extra boot_param pointer
> to point to the offset of the payload blob (ELF file in my case,
> compressed kernel in yours).  Yes?

Indeed.

	-hpa

Re: Extending boot protocol & bzImage for paravirt_ops

[Jeremy Fitzhardinge]

H. Peter Anvin wrote:
> It would have to, because of the way code32_start is defined to work.
> We don't get control again after its use as a hook.
>   

Who uses that hook?  The impression I get is that the execution
environment for jumping to that pointer is not very well defined at present.

    J

Re: Extending boot protocol & bzImage for paravirt_ops

[Eric W. Biederman]

Jeremy Fitzhardinge <jeremy@goop.org> writes:

> H. Peter Anvin wrote:
>> It would have to, because of the way code32_start is defined to work.
>> We don't get control again after its use as a hook.
>>   
>
> Who uses that hook?  The impression I get is that the execution
> environment for jumping to that pointer is not very well defined at present.

loadlin.

Eric