[Qemu-devel] [RFC PATCH] docs/booting.rst: start documenting the boot process

[Qemu-devel] [RFC PATCH] docs/booting.rst: start documenting the boot process

[Alex Bennée]

While working on some test cases I realised there was quite a lot of
assumed knowledge about how things boot up. I thought it would be
worth gathering this together in a user facing document where we could
pour in the details and background to the boot process. As it's quite
wordy I thought it should be a separate document to the manual (which
can obviously reference this). So far I've managed almost a thousand
words and haven't actually related anything to QEMU's options yet.

So going forward:

  - is this a useful document to have in docs?
  - are there any other areas to mention?
    - out auto-magic -bios selection seems like something worth covering
  - should we have some worked examples of command lines?
    - I was thinking for example of read-only and pflash examples
  - we should also describe why direct kernel boots exits
    - and also the fact they are not that direct (some code executes
      before a kernel - even if it's less than a full firmware)

Submitted for comment....

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 docs/booting.rst | 127 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 127 insertions(+)
 create mode 100644 docs/booting.rst

diff --git a/docs/booting.rst b/docs/booting.rst
new file mode 100644
index 0000000000..a8a644ff9a
--- /dev/null
+++ b/docs/booting.rst
@@ -0,0 +1,127 @@
+=====================================
+Anatomy of a Boot, a QEMU perspective
+=====================================
+
+This document attempts to give an overview of how machines boot-up and
+how this matters to QEMU. We will discuss firmware and BIOSes and the
+things they do before the OS kernel is loaded and your usable system
+is finally ready.
+
+Firmware
+========
+
+When a CPU is powered up it knows nothing about it's environment. It's
+internal state, including the program counter (PC), will be reset to a
+defined set of values and it will attempt to fetch it's first
+instruction and execute it. It is the job of the firmware to bring a
+CPU up from it's first few instructions to running in a relatively
+sane execution environment. Firmware tends to be specific to the
+hardware in question and is stored on non-volatile memory (memory that
+survives a power off) usually a ROM or flash device on the computers
+main board.
+
+Some examples of what firmware does include:
+
+Early Hardware Setup
+--------------------
+
+Modern hardware often requires configuring before it is usable. For
+example most modern systems won't have working RAM until the memory
+controller has been programmed with the correct timings for whatever
+memory is installed on the system. Processors may boot with a very
+restricted view of the memory map until RAM and other key peripherals
+have been configured to appear in it's address space. Some hardware
+may not even appear until some sort of blob has been loaded into it so
+it can start responding to the CPU.
+
+Fortunately for QEMU we don't have to worry too much about this very
+low level configuration. The device model we present to the CPU at
+start-up will generally respond to IO access from processor straight
+away.
+
+BIOS or Firmware Services
+-------------------------
+
+In the early days of the PC era the BIOS or Basic Input/Output System
+provided an abstraction interface to the operating system which
+allowed them to do basic IO operations without having to directly
+drive the hardware. Since then the scope of these firmware services
+have grown as systems become more and more complex.
+
+Modern firmware often follows the Unified Extensible Firmware
+Interface (UEFI) which provides services like secure boot, persistent
+variables and external time-keeping.
+
+There can often be multiple levels of firmware service functions. For
+example systems which support secure execution enclaves generally have
+a firmware component that executes in this secure mode which the
+operating system can call in a defined secure manner to undertake
+security sensitive tasks on it's behalf.
+
+Hardware Enumeration
+--------------------
+
+It's easy to assume that modern hardware is built to be discover-able
+and all the operating system needs to do is enumerate the various
+buses on the system to find out what hardware exists. While buses like
+PCI and USB do support discovery there is usually much more on a
+modern system than just these two things.
+
+In the embedded world it used to be acceptable to have a custom
+compiled kernel which knew where everything is meant to be. However
+this was a brittle approach and not very flexible - most obviously if
+you try and use a kernel compiled for one piece of hardware on another
+piece of hardware that might nominally have the same processor.
+
+The more modern approach is to have a "generic" kernel that has a
+number of different drivers compiled in which are then enabled based
+on a hardware description provided by the firmware. This allows
+flexibility on both sides. The software distribution is less concerned
+about managing lots of different kernels for different pieces of
+hardware. The hardware manufacturer is also able to make small changes
+to the board over time to fix bugs or change minor components.
+
+The two main methods for this are the Advanced Configuration and Power
+Interface (ACPI) and Device Trees. ACPI originated from the PC world
+although is becoming increasingly common for "enterprise" hardware
+like servers. Device Trees of various forms have existed for a while
+with perhaps the most common being Flattened Device Trees (FDT).
+
+Boot Code
+=========
+
+The line between firmware and boot code is a very blurry one. However
+from a functionality point of view we have moved from ensuring the
+hardware is usable as a computing device to finding and loading a
+kernel which is then going to take over control of the system. Modern
+firmware often has the ability to boot a kernel directly and in some
+systems you might chain through several boot loaders before the final
+kernel takes control.
+
+The boot loader needs to do 3 things:
+
+  - find a kernel and load it into RAM
+  - ensure the CPU is in the correct mode for the kernel to boot
+  - pass any information the kernel may need to boot and can't find itself
+
+Once it has done these things it can jump to the kernel and let it get
+on with things.
+
+Kernel
+======
+
+The Kernel now takes over and will be in charge of the system from now
+on. It will enumerate all the devices on the system (again) and load
+drivers that can control them. It will then locate some sort of
+file-system and eventually start running programs that actually do
+work.
+
+------------------------
+How this relates to QEMU
+------------------------
+
+TODO:
+
+  - -bios and -drive flash
+  - dynamic and fixed hardware definitions
+  - direct kernel boots
-- 
2.20.1

Re: [Qemu-devel] [RFC PATCH] docs/booting.rst: start documenting the boot process

[Peter Maydell]

On Fri, 8 Mar 2019 at 21:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> While working on some test cases I realised there was quite a lot of
> assumed knowledge about how things boot up. I thought it would be
> worth gathering this together in a user facing document where we could
> pour in the details and background to the boot process. As it's quite
> wordy I thought it should be a separate document to the manual (which
> can obviously reference this). So far I've managed almost a thousand
> words and haven't actually related anything to QEMU's options yet.
>
> So going forward:
>
>   - is this a useful document to have in docs?
>   - are there any other areas to mention?
>     - out auto-magic -bios selection seems like something worth covering
>   - should we have some worked examples of command lines?
>     - I was thinking for example of read-only and pflash examples
>   - we should also describe why direct kernel boots exits
>     - and also the fact they are not that direct (some code executes
>       before a kernel - even if it's less than a full firmware)
>
> Submitted for comment....
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  docs/booting.rst | 127 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 127 insertions(+)
>  create mode 100644 docs/booting.rst

We're trying to move towards having no documents in
the root docs/ directory, but instead having them all
in one of the various subdirectories that will each
be one of our five manuals:
https://wiki.qemu.org/Features/Documentation

thanks
-- PMM

Re: [Qemu-devel] [RFC PATCH] docs/booting.rst: start documenting the boot process

[Paolo Bonzini]

On 08/03/19 22:15, Alex Bennée wrote:
> While working on some test cases I realised there was quite a lot of
> assumed knowledge about how things boot up. I thought it would be
> worth gathering this together in a user facing document where we could
> pour in the details and background to the boot process. As it's quite
> wordy I thought it should be a separate document to the manual (which
> can obviously reference this). So far I've managed almost a thousand
> words and haven't actually related anything to QEMU's options yet.

This might be more blog material, but it's certainly good stuff!

Paolo

> So going forward:
> 
>   - is this a useful document to have in docs?
>   - are there any other areas to mention?
>     - out auto-magic -bios selection seems like something worth covering
>   - should we have some worked examples of command lines?
>     - I was thinking for example of read-only and pflash examples
>   - we should also describe why direct kernel boots exits
>     - and also the fact they are not that direct (some code executes
>       before a kernel - even if it's less than a full firmware)
> 
> Submitted for comment....
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> ---
>  docs/booting.rst | 127 +++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 127 insertions(+)
>  create mode 100644 docs/booting.rst
> 
> diff --git a/docs/booting.rst b/docs/booting.rst
> new file mode 100644
> index 0000000000..a8a644ff9a
> --- /dev/null
> +++ b/docs/booting.rst
> @@ -0,0 +1,127 @@
> +=====================================
> +Anatomy of a Boot, a QEMU perspective
> +=====================================
> +
> +This document attempts to give an overview of how machines boot-up and
> +how this matters to QEMU. We will discuss firmware and BIOSes and the
> +things they do before the OS kernel is loaded and your usable system
> +is finally ready.
> +
> +Firmware
> +========
> +
> +When a CPU is powered up it knows nothing about it's environment. It's
> +internal state, including the program counter (PC), will be reset to a
> +defined set of values and it will attempt to fetch it's first
> +instruction and execute it. It is the job of the firmware to bring a
> +CPU up from it's first few instructions to running in a relatively
> +sane execution environment. Firmware tends to be specific to the
> +hardware in question and is stored on non-volatile memory (memory that
> +survives a power off) usually a ROM or flash device on the computers
> +main board.
> +
> +Some examples of what firmware does include:
> +
> +Early Hardware Setup
> +--------------------
> +
> +Modern hardware often requires configuring before it is usable. For
> +example most modern systems won't have working RAM until the memory
> +controller has been programmed with the correct timings for whatever
> +memory is installed on the system. Processors may boot with a very
> +restricted view of the memory map until RAM and other key peripherals
> +have been configured to appear in it's address space. Some hardware
> +may not even appear until some sort of blob has been loaded into it so
> +it can start responding to the CPU.
> +
> +Fortunately for QEMU we don't have to worry too much about this very
> +low level configuration. The device model we present to the CPU at
> +start-up will generally respond to IO access from processor straight
> +away.
> +
> +BIOS or Firmware Services
> +-------------------------
> +
> +In the early days of the PC era the BIOS or Basic Input/Output System
> +provided an abstraction interface to the operating system which
> +allowed them to do basic IO operations without having to directly
> +drive the hardware. Since then the scope of these firmware services
> +have grown as systems become more and more complex.
> +
> +Modern firmware often follows the Unified Extensible Firmware
> +Interface (UEFI) which provides services like secure boot, persistent
> +variables and external time-keeping.
> +
> +There can often be multiple levels of firmware service functions. For
> +example systems which support secure execution enclaves generally have
> +a firmware component that executes in this secure mode which the
> +operating system can call in a defined secure manner to undertake
> +security sensitive tasks on it's behalf.
> +
> +Hardware Enumeration
> +--------------------
> +
> +It's easy to assume that modern hardware is built to be discover-able
> +and all the operating system needs to do is enumerate the various
> +buses on the system to find out what hardware exists. While buses like
> +PCI and USB do support discovery there is usually much more on a
> +modern system than just these two things.
> +
> +In the embedded world it used to be acceptable to have a custom
> +compiled kernel which knew where everything is meant to be. However
> +this was a brittle approach and not very flexible - most obviously if
> +you try and use a kernel compiled for one piece of hardware on another
> +piece of hardware that might nominally have the same processor.
> +
> +The more modern approach is to have a "generic" kernel that has a
> +number of different drivers compiled in which are then enabled based
> +on a hardware description provided by the firmware. This allows
> +flexibility on both sides. The software distribution is less concerned
> +about managing lots of different kernels for different pieces of
> +hardware. The hardware manufacturer is also able to make small changes
> +to the board over time to fix bugs or change minor components.
> +
> +The two main methods for this are the Advanced Configuration and Power
> +Interface (ACPI) and Device Trees. ACPI originated from the PC world
> +although is becoming increasingly common for "enterprise" hardware
> +like servers. Device Trees of various forms have existed for a while
> +with perhaps the most common being Flattened Device Trees (FDT).
> +
> +Boot Code
> +=========
> +
> +The line between firmware and boot code is a very blurry one. However
> +from a functionality point of view we have moved from ensuring the
> +hardware is usable as a computing device to finding and loading a
> +kernel which is then going to take over control of the system. Modern
> +firmware often has the ability to boot a kernel directly and in some
> +systems you might chain through several boot loaders before the final
> +kernel takes control.
> +
> +The boot loader needs to do 3 things:
> +
> +  - find a kernel and load it into RAM
> +  - ensure the CPU is in the correct mode for the kernel to boot
> +  - pass any information the kernel may need to boot and can't find itself
> +
> +Once it has done these things it can jump to the kernel and let it get
> +on with things.
> +
> +Kernel
> +======
> +
> +The Kernel now takes over and will be in charge of the system from now
> +on. It will enumerate all the devices on the system (again) and load
> +drivers that can control them. It will then locate some sort of
> +file-system and eventually start running programs that actually do
> +work.
> +
> +------------------------
> +How this relates to QEMU
> +------------------------
> +
> +TODO:
> +
> +  - -bios and -drive flash
> +  - dynamic and fixed hardware definitions
> +  - direct kernel boots
> 

Re: [Qemu-devel] [RFC PATCH] docs/booting.rst: start documenting the boot process

[Eric Blake]

On 3/8/19 3:15 PM, Alex Bennée wrote:
> While working on some test cases I realised there was quite a lot of
> assumed knowledge about how things boot up. I thought it would be
> worth gathering this together in a user facing document where we could
> pour in the details and background to the boot process. As it's quite
> wordy I thought it should be a separate document to the manual (which
> can obviously reference this). So far I've managed almost a thousand
> words and haven't actually related anything to QEMU's options yet.
> 
> So going forward:
> 
>   - is this a useful document to have in docs?
>   - are there any other areas to mention?
>     - out auto-magic -bios selection seems like something worth covering
>   - should we have some worked examples of command lines?
>     - I was thinking for example of read-only and pflash examples
>   - we should also describe why direct kernel boots exits
>     - and also the fact they are not that direct (some code executes
>       before a kernel - even if it's less than a full firmware)
> 
> Submitted for comment....

I'm not sure where to put it, but it does look useful. Some suggestions:


> +This document attempts to give an overview of how machines boot-up and

s/boot-up/boot/

> +how this matters to QEMU. We will discuss firmware and BIOSes and the
> +things they do before the OS kernel is loaded and your usable system
> +is finally ready.
> +
> +Firmware
> +========
> +
> +When a CPU is powered up it knows nothing about it's environment. It's

s/it's/its/2 ("it's" is only appropriate where "it is" can also be used,
which does not fit either of these uses)

> +internal state, including the program counter (PC), will be reset to a
> +defined set of values and it will attempt to fetch it's first

again

> +instruction and execute it. It is the job of the firmware to bring a
> +CPU up from it's first few instructions to running in a relatively

again

> +sane execution environment. Firmware tends to be specific to the
> +hardware in question and is stored on non-volatile memory (memory that
> +survives a power off) usually a ROM or flash device on the computers
> +main board.
> +
> +Some examples of what firmware does include:
> +
> +Early Hardware Setup
> +--------------------
> +
> +Modern hardware often requires configuring before it is usable. For
> +example most modern systems won't have working RAM until the memory
> +controller has been programmed with the correct timings for whatever
> +memory is installed on the system. Processors may boot with a very
> +restricted view of the memory map until RAM and other key peripherals
> +have been configured to appear in it's address space. Some hardware

again

> +may not even appear until some sort of blob has been loaded into it so
> +it can start responding to the CPU.
> +
> +Fortunately for QEMU we don't have to worry too much about this very
> +low level configuration. The device model we present to the CPU at
> +start-up will generally respond to IO access from processor straight
> +away.
> +
> +BIOS or Firmware Services
> +-------------------------
> +
> +In the early days of the PC era the BIOS or Basic Input/Output System
> +provided an abstraction interface to the operating system which
> +allowed them to do basic IO operations without having to directly
> +drive the hardware. Since then the scope of these firmware services
> +have grown as systems become more and more complex.
> +
> +Modern firmware often follows the Unified Extensible Firmware
> +Interface (UEFI) which provides services like secure boot, persistent
> +variables and external time-keeping.
> +
> +There can often be multiple levels of firmware service functions. For
> +example systems which support secure execution enclaves generally have
> +a firmware component that executes in this secure mode which the
> +operating system can call in a defined secure manner to undertake
> +security sensitive tasks on it's behalf.

again

> +
> +Hardware Enumeration
> +--------------------
> +
> +It's easy to assume that modern hardware is built to be discover-able

Finally, a correct use.

> +and all the operating system needs to do is enumerate the various
> +buses on the system to find out what hardware exists. While buses like
> +PCI and USB do support discovery there is usually much more on a
> +modern system than just these two things.

Is it also worth mentioning that discovery can take time, which is in
contrast to virtual environments that want to boot quickly?

> +
> +In the embedded world it used to be acceptable to have a custom
> +compiled kernel which knew where everything is meant to be. However
> +this was a brittle approach and not very flexible - most obviously if
> +you try and use a kernel compiled for one piece of hardware on another
> +piece of hardware that might nominally have the same processor.
> +
> +The more modern approach is to have a "generic" kernel that has a
> +number of different drivers compiled in which are then enabled based
> +on a hardware description provided by the firmware. This allows
> +flexibility on both sides. The software distribution is less concerned
> +about managing lots of different kernels for different pieces of
> +hardware. The hardware manufacturer is also able to make small changes
> +to the board over time to fix bugs or change minor components.
> +
> +The two main methods for this are the Advanced Configuration and Power
> +Interface (ACPI) and Device Trees. ACPI originated from the PC world
> +although is becoming increasingly common for "enterprise" hardware

s/is/it is/

> +like servers. Device Trees of various forms have existed for a while
> +with perhaps the most common being Flattened Device Trees (FDT).
> +
> +Boot Code
> +=========
> +
> +The line between firmware and boot code is a very blurry one. However
> +from a functionality point of view we have moved from ensuring the
> +hardware is usable as a computing device to finding and loading a
> +kernel which is then going to take over control of the system. Modern
> +firmware often has the ability to boot a kernel directly and in some
> +systems you might chain through several boot loaders before the final
> +kernel takes control.
> +
> +The boot loader needs to do 3 things:
> +
> +  - find a kernel and load it into RAM
> +  - ensure the CPU is in the correct mode for the kernel to boot
> +  - pass any information the kernel may need to boot and can't find itself
> +
> +Once it has done these things it can jump to the kernel and let it get
> +on with things.
> +
> +Kernel
> +======
> +
> +The Kernel now takes over and will be in charge of the system from now
> +on. It will enumerate all the devices on the system (again) and load
> +drivers that can control them. It will then locate some sort of
> +file-system and eventually start running programs that actually do
> +work.
> +
> +------------------------
> +How this relates to QEMU
> +------------------------
> +
> +TODO:
> +
> +  - -bios and -drive flash
> +  - dynamic and fixed hardware definitions
> +  - direct kernel boots
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org