[PATCH v2 0/3] some docs (booting, mttcg, icount)

[PATCH v2 0/3] some docs (booting, mttcg, icount)

[Alex Bennée]

Hi,

This is the current state of documentation patches. They didn't get
much review the first time around so I present them as a series. I
Think we should merge them unless anyone want to argue they actively
muddy our document situation?

The following have no reviews:

 - docs/devel: convert and update MTTCG design document
 - docs/booting.rst: start documenting the boot process

Alex Bennée (3):
  docs/booting.rst: start documenting the boot process
  docs/devel: convert and update MTTCG design document
  docs/devel: add some notes on tcg-icount for developers

 docs/devel/index.rst                          |   2 +
 ...ti-thread-tcg.txt => multi-thread-tcg.rst} |  52 +++---
 docs/devel/tcg-icount.rst                     |  89 ++++++++++
 docs/interop/booting.rst                      | 159 ++++++++++++++++++
 docs/interop/index.rst                        |   1 +
 5 files changed, 284 insertions(+), 19 deletions(-)
 rename docs/devel/{multi-thread-tcg.txt => multi-thread-tcg.rst} (90%)
 create mode 100644 docs/devel/tcg-icount.rst
 create mode 100644 docs/interop/booting.rst

-- 
2.20.1

[PATCH v2 1/3] docs/booting.rst: start documenting the boot process

[Alex Bennée]

While working on some test cases I realised there was quite a lot of
assumed knowledge about how things boot up. I thought it would be
worth gathering this together in a user facing document where we could
pour in the details and background to the boot process. As it's quite
wordy I thought it should be a separate document to the manual (which
can obviously reference this).

The document follows the socratic method and leaves the reader to ask
themselves some questions in an effort to elucidate them about any
problems they may be having.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20190308211557.22589-1-alex.bennee@linaro.org>

---
v2
  - fix a lot of it's/its
  - mention microvm style booting
  - add some questions to the end
---
 docs/interop/booting.rst | 159 +++++++++++++++++++++++++++++++++++++++
 docs/interop/index.rst   |   1 +
 2 files changed, 160 insertions(+)
 create mode 100644 docs/interop/booting.rst

diff --git a/docs/interop/booting.rst b/docs/interop/booting.rst
new file mode 100644
index 00000000000..8579a775d04
--- /dev/null
+++ b/docs/interop/booting.rst
@@ -0,0 +1,159 @@
+..
+  Copyright (c) 2019-2020 Linaro Ltd.
+
+  This work is licensed under the terms of the GNU GPL, version 2 or
+  later. See the COPYING file in the top-level directory.
+
+=====================================
+Anatomy of a Boot, a QEMU perspective
+=====================================
+
+This document attempts to give an overview of how machines boot and
+how this matters to QEMU. We will discuss firmware and BIOSes and the
+things they do before the OS kernel is loaded and your usable system
+is finally ready.
+
+Firmware
+========
+
+When a CPU is powered up it knows nothing about its environment. The
+internal state, including the program counter (PC), will be reset to a
+defined set of values and it will attempt to fetch the first
+instruction and execute it. It is the job of the firmware to bring a
+CPU up from the initial few instructions to running in a relatively
+sane execution environment. Firmware tends to be specific to the
+hardware in question and is stored on non-volatile memory (memory that
+survives a power off) usually a ROM or flash device on the computers
+main board.
+
+Some examples of what firmware does include:
+
+Early Hardware Setup
+--------------------
+
+Modern hardware often requires configuring before it is usable. For
+example most modern systems won't have working RAM until the memory
+controller has been programmed with the correct timings for whatever
+memory is installed on the system. Processors may boot with a very
+restricted view of the memory map until RAM and other key peripherals
+have been configured to appear in its address space. Some hardware
+may not even appear until some sort of blob has been loaded into it so
+it can start responding to the CPU.
+
+Fortunately for QEMU we don't have to worry too much about this very
+low level configuration. The device model we present to the CPU at
+start-up will generally respond to IO access from processor straight
+away.
+
+BIOS or Firmware Services
+-------------------------
+
+In the early days of the PC era the BIOS or Basic Input/Output System
+provided an abstraction interface to the operating system which
+allowed them to do basic IO operations without having to directly
+drive the hardware. Since then the scope of these firmware services
+have grown as systems become more and more complex.
+
+Modern firmware often follows the Unified Extensible Firmware
+Interface (UEFI) which provides services like secure boot, persistent
+variables and external time-keeping.
+
+There can often be multiple levels of firmware service functions. For
+example systems which support secure execution enclaves generally have
+a firmware component that executes in this secure mode which the
+operating system can call in a defined secure manner to undertake
+security sensitive tasks on its behalf.
+
+Hardware Enumeration
+--------------------
+
+It is easy to assume that modern hardware is built to be discover-able
+and all the operating system needs to do is enumerate the various
+buses on the system to find out what hardware exists. While buses like
+PCI and USB do support discovery there is usually much more on a
+modern system than just these two things.
+
+This process of discovery can take some time as devices usually need
+to be probed and some time allowed for the buses to settle and the
+probe complete. For purely virtual machines operating in on-demand
+cloud environments you may operate with stripped down kernels that
+only support a fixed expected environment so they can boot as fast as
+possible.
+
+In the embedded world it used to be acceptable to have a similar
+custom compiled kernel which knew where everything is meant to be.
+However this was a brittle approach and not very flexible. For example
+a general purpose distribution would have to ship a special kernel for
+each variant of hardware you wanted to run on. If you try and use a
+kernel compiled for one platform that nominally uses the same
+processor as another platform the result will rarely work given a
+processor rarely works in isolation.
+
+The more modern approach is to have a "generic" kernel that has a
+number of different drivers compiled in which are then enabled based
+on a hardware description provided by the firmware. This allows
+flexibility on both sides. The software distribution is less concerned
+about managing lots of different kernels for different pieces of
+hardware. The hardware manufacturer is also able to make small changes
+to the board over time to fix bugs or change minor components.
+
+The two main methods for this are the Advanced Configuration and Power
+Interface (ACPI) and Device Trees. ACPI originated from the PC world
+although it is becoming increasingly common for "enterprise" hardware
+like servers. Device Trees of various forms have existed for a while
+with perhaps the most common being Flattened Device Trees (FDT).
+
+Boot Code
+=========
+
+The line between firmware and boot code is a very blurry one. However
+from a functionality point of view we have moved from ensuring the
+hardware is usable as a computing device to finding and loading a
+kernel which is then going to take over control of the system. Modern
+firmware often has the ability to boot a kernel directly and in some
+systems you might chain through several boot loaders before the final
+kernel takes control.
+
+The boot loader needs to do 3 things:
+
+  - find a kernel and load it into RAM
+  - ensure the CPU is in the correct mode for the kernel to boot
+  - pass any information the kernel may need to boot and can't find itself
+
+Once it has done these things it can jump to the kernel and let it get
+on with things.
+
+Kernel
+======
+
+The Kernel now takes over and will be in charge of the system from now
+on. It will enumerate all the devices on the system (again) and load
+drivers that can control them. It will then locate some sort of
+file-system and eventually start running programs that actually do
+work.
+
+Questions to ask yourself
+=========================
+
+Having given this overview of booting here are some questions you
+should ask when diagnosing boot problems.
+
+Hardware
+~~~~~~~~
+
+ - is the platform fixed or dynamic?
+ - is the platform enumeratable (e.g. PCI/USB)?
+
+Firmware
+~~~~~~~~
+
+ - is the firmware built for the platform your are booting?
+ - does the firmware need storage for variables (boot index etc)?
+ - does the firmware provide a service to kernels (e.g. ACPI/EFI)?
+
+Kernel
+~~~~~~
+
+ - is the kernel platform specific or generic?
+ - how will the kernel enumerate the platform?
+ - can the kernel interface talk to the firmware?
diff --git a/docs/interop/index.rst b/docs/interop/index.rst
index 049387ac6de..58d587444b3 100644
--- a/docs/interop/index.rst
+++ b/docs/interop/index.rst
@@ -12,6 +12,7 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
+   booting
    bitmaps
    dbus
    dbus-vmstate
-- 
2.20.1

Re: [PATCH v2 1/3] docs/booting.rst: start documenting the boot process

[Richard Henderson]

On 7/1/20 9:11 AM, Alex Bennée wrote:
> While working on some test cases I realised there was quite a lot of
> assumed knowledge about how things boot up. I thought it would be
> worth gathering this together in a user facing document where we could
> pour in the details and background to the boot process. As it's quite
> wordy I thought it should be a separate document to the manual (which
> can obviously reference this).
> 
> The document follows the socratic method and leaves the reader to ask
> themselves some questions in an effort to elucidate them about any
> problems they may be having.
> 
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20190308211557.22589-1-alex.bennee@linaro.org>

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

Re: [PATCH v2 1/3] docs/booting.rst: start documenting the boot process

[Peter Maydell]

On Wed, 1 Jul 2020 at 17:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> While working on some test cases I realised there was quite a lot of
> assumed knowledge about how things boot up. I thought it would be
> worth gathering this together in a user facing document where we could
> pour in the details and background to the boot process. As it's quite
> wordy I thought it should be a separate document to the manual (which
> can obviously reference this).
>
> The document follows the socratic method and leaves the reader to ask
> themselves some questions in an effort to elucidate them about any
> problems they may be having.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Message-Id: <20190308211557.22589-1-alex.bennee@linaro.org>
>

I have some typo/grammar nits below, but more significantly:
I don't really see why this is a bit of QEMU's documentation
rather than, say, a blog post. It doesn't really say much
about QEMU in particular. It's also rather vague about
what guest architecture it's talking about, and they can
differ significantly both in the real world and in how QEMU
handles them. Also, why put it in interop?

> ---
> v2
>   - fix a lot of it's/its
>   - mention microvm style booting
>   - add some questions to the end
> ---
>  docs/interop/booting.rst | 159 +++++++++++++++++++++++++++++++++++++++
>  docs/interop/index.rst   |   1 +
>  2 files changed, 160 insertions(+)
>  create mode 100644 docs/interop/booting.rst
>
> diff --git a/docs/interop/booting.rst b/docs/interop/booting.rst
> new file mode 100644
> index 00000000000..8579a775d04
> --- /dev/null
> +++ b/docs/interop/booting.rst
> @@ -0,0 +1,159 @@
> +..
> +  Copyright (c) 2019-2020 Linaro Ltd.
> +
> +  This work is licensed under the terms of the GNU GPL, version 2 or
> +  later. See the COPYING file in the top-level directory.
> +
> +=====================================
> +Anatomy of a Boot, a QEMU perspective
> +=====================================
> +
> +This document attempts to give an overview of how machines boot and
> +how this matters to QEMU. We will discuss firmware and BIOSes and the
> +things they do before the OS kernel is loaded and your usable system
> +is finally ready.
> +
> +Firmware
> +========
> +
> +When a CPU is powered up it knows nothing about its environment. The
> +internal state, including the program counter (PC), will be reset to a
> +defined set of values and it will attempt to fetch the first
> +instruction and execute it. It is the job of the firmware to bring a
> +CPU up from the initial few instructions to running in a relatively
> +sane execution environment. Firmware tends to be specific to the
> +hardware in question and is stored on non-volatile memory (memory that
> +survives a power off) usually a ROM or flash device on the computers

", usually"

> +main board.
> +
> +Some examples of what firmware does include:
> +
> +Early Hardware Setup
> +--------------------
> +
> +Modern hardware often requires configuring before it is usable. For
> +example most modern systems won't have working RAM until the memory
> +controller has been programmed with the correct timings for whatever
> +memory is installed on the system. Processors may boot with a very
> +restricted view of the memory map until RAM and other key peripherals
> +have been configured to appear in its address space. Some hardware
> +may not even appear until some sort of blob has been loaded into it so
> +it can start responding to the CPU.
> +
> +Fortunately for QEMU we don't have to worry too much about this very
> +low level configuration. The device model we present to the CPU at
> +start-up will generally respond to IO access from processor straight

"the processor"

> +away.
> +
> +BIOS or Firmware Services
> +-------------------------
> +
> +In the early days of the PC era the BIOS or Basic Input/Output System
> +provided an abstraction interface to the operating system which
> +allowed them to do basic IO operations without having to directly

"allowed the OS"

> +drive the hardware. Since then the scope of these firmware services
> +have grown as systems become more and more complex.

"has grown"

> +
> +Modern firmware often follows the Unified Extensible Firmware
> +Interface (UEFI) which provides services like secure boot, persistent
> +variables and external time-keeping.
> +
> +There can often be multiple levels of firmware service functions. For
> +example systems which support secure execution enclaves generally have
> +a firmware component that executes in this secure mode which the
> +operating system can call in a defined secure manner to undertake
> +security sensitive tasks on its behalf.
> +
> +Hardware Enumeration
> +--------------------
> +
> +It is easy to assume that modern hardware is built to be discover-able

"discoverable"

> +and all the operating system needs to do is enumerate the various
> +buses on the system to find out what hardware exists. While buses like
> +PCI and USB do support discovery there is usually much more on a
> +modern system than just these two things.
> +
> +This process of discovery can take some time as devices usually need
> +to be probed and some time allowed for the buses to settle and the
> +probe complete. For purely virtual machines operating in on-demand

"probe to complete".

> +cloud environments you may operate with stripped down kernels that
> +only support a fixed expected environment so they can boot as fast as
> +possible.
> +
> +In the embedded world it used to be acceptable to have a similar
> +custom compiled kernel which knew where everything is meant to be.
> +However this was a brittle approach and not very flexible. For example
> +a general purpose distribution would have to ship a special kernel for
> +each variant of hardware you wanted to run on. If you try and use a
> +kernel compiled for one platform that nominally uses the same
> +processor as another platform the result will rarely work given a
> +processor rarely works in isolation.

"the result will generally not work."

> +
> +The more modern approach is to have a "generic" kernel that has a
> +number of different drivers compiled in which are then enabled based
> +on a hardware description provided by the firmware. This allows
> +flexibility on both sides. The software distribution is less concerned
> +about managing lots of different kernels for different pieces of
> +hardware. The hardware manufacturer is also able to make small changes
> +to the board over time to fix bugs or change minor components.
> +
> +The two main methods for this are the Advanced Configuration and Power
> +Interface (ACPI) and Device Trees. ACPI originated from the PC world
> +although it is becoming increasingly common for "enterprise" hardware
> +like servers. Device Trees of various forms have existed for a while
> +with perhaps the most common being Flattened Device Trees (FDT).
> +
> +Boot Code
> +=========
> +
> +The line between firmware and boot code is a very blurry one. However
> +from a functionality point of view we have moved from ensuring the
> +hardware is usable as a computing device to finding and loading a
> +kernel which is then going to take over control of the system. Modern
> +firmware often has the ability to boot a kernel directly and in some
> +systems you might chain through several boot loaders before the final
> +kernel takes control.
> +
> +The boot loader needs to do 3 things:
> +
> +  - find a kernel and load it into RAM
> +  - ensure the CPU is in the correct mode for the kernel to boot
> +  - pass any information the kernel may need to boot and can't find itself
> +
> +Once it has done these things it can jump to the kernel and let it get
> +on with things.
> +
> +Kernel
> +======
> +
> +The Kernel now takes over and will be in charge of the system from now
> +on. It will enumerate all the devices on the system (again) and load
> +drivers that can control them. It will then locate some sort of
> +file-system and eventually start running programs that actually do
> +work.
> +
> +Questions to ask yourself
> +=========================
> +
> +Having given this overview of booting here are some questions you
> +should ask when diagnosing boot problems.
> +
> +Hardware
> +~~~~~~~~

Previously you used '-----' as the title format for "next
level below "====="; Sphinx will complain that you're not
being consistent (see the patchew emails).

> +
> + - is the platform fixed or dynamic?
> + - is the platform enumeratable (e.g. PCI/USB)?
> +
> +Firmware
> +~~~~~~~~
> +
> + - is the firmware built for the platform your are booting?

"you are"

> + - does the firmware need storage for variables (boot index etc)?
> + - does the firmware provide a service to kernels (e.g. ACPI/EFI)?
> +
> +Kernel
> +~~~~~~
> +
> + - is the kernel platform specific or generic?
> + - how will the kernel enumerate the platform?
> + - can the kernel interface talk to the firmware?
> diff --git a/docs/interop/index.rst b/docs/interop/index.rst
> index 049387ac6de..58d587444b3 100644
> --- a/docs/interop/index.rst
> +++ b/docs/interop/index.rst
> @@ -12,6 +12,7 @@ Contents:
>  .. toctree::
>     :maxdepth: 2
>
> +   booting
>     bitmaps
>     dbus
>     dbus-vmstate
> --

thanks
-- PMM

Re: [PATCH v2 1/3] docs/booting.rst: start documenting the boot process

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Wed, 1 Jul 2020 at 17:16, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> While working on some test cases I realised there was quite a lot of
>> assumed knowledge about how things boot up. I thought it would be
>> worth gathering this together in a user facing document where we could
>> pour in the details and background to the boot process. As it's quite
>> wordy I thought it should be a separate document to the manual (which
>> can obviously reference this).
>>
>> The document follows the socratic method and leaves the reader to ask
>> themselves some questions in an effort to elucidate them about any
>> problems they may be having.
>>
>> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
>> Message-Id: <20190308211557.22589-1-alex.bennee@linaro.org>
>>
>
> I have some typo/grammar nits below, but more significantly:
> I don't really see why this is a bit of QEMU's documentation
> rather than, say, a blog post.

The aim was to make it an initial overview so when people ask about why
they can't boot their systems they have considered the questions at the
end. It does seem to be a repeating topic on the IRC channel.

> It doesn't really say much
> about QEMU in particular. It's also rather vague about
> what guest architecture it's talking about, and they can
> differ significantly both in the real world and in how QEMU
> handles them.

Perhaps what we really need is a shorter punchier section offering
guidance in the manual itself? It just seemed quite a big topic which is
why I ended up with a fairly long explanation without even getting to
specifics.

I could certainly see the document being expanded in the future with
more concrete examples for various architectures and scenarios.

> Also, why put it in interop?

interop being where QEMU interfaces with other things such as kernels
and firmwares? It seems too user focused for devel and too long for the
manual itself.

-- 
Alex Bennée

[PATCH v2 2/3] docs/devel: convert and update MTTCG design document

[Alex Bennée]

Do a light conversion to .rst and clean-up some of the language at the
start now MTTCG has been merged for a while.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
---
 docs/devel/index.rst                          |  1 +
 ...ti-thread-tcg.txt => multi-thread-tcg.rst} | 52 ++++++++++++-------
 2 files changed, 34 insertions(+), 19 deletions(-)
 rename docs/devel/{multi-thread-tcg.txt => multi-thread-tcg.rst} (90%)

diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index bb8238c5d6d..4ecaea3643f 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -23,6 +23,7 @@ Contents:
    decodetree
    secure-coding-practices
    tcg
+   multi-thread-tcg
    tcg-plugins
    bitops
    reset
diff --git a/docs/devel/multi-thread-tcg.txt b/docs/devel/multi-thread-tcg.rst
similarity index 90%
rename from docs/devel/multi-thread-tcg.txt
rename to docs/devel/multi-thread-tcg.rst
index 3c85ac0eab9..42158b77c70 100644
--- a/docs/devel/multi-thread-tcg.txt
+++ b/docs/devel/multi-thread-tcg.rst
@@ -1,15 +1,17 @@
-Copyright (c) 2015-2016 Linaro Ltd.
+..
+  Copyright (c) 2015-2020 Linaro Ltd.
 
-This work is licensed under the terms of the GNU GPL, version 2 or
-later. See the COPYING file in the top-level directory.
+  This work is licensed under the terms of the GNU GPL, version 2 or
+  later. See the COPYING file in the top-level directory.
 
 Introduction
 ============
 
-This document outlines the design for multi-threaded TCG system-mode
-emulation. The current user-mode emulation mirrors the thread
-structure of the translated executable. Some of the work will be
-applicable to both system and linux-user emulation.
+This document outlines the design for multi-threaded TCG (a.k.a MTTCG)
+system-mode emulation. user-mode emulation has always mirrored the
+thread structure of the translated executable although some of the
+changes done for MTTCG system emulation have improved the stability of
+linux-user emulation.
 
 The original system-mode TCG implementation was single threaded and
 dealt with multiple CPUs with simple round-robin scheduling. This
@@ -21,9 +23,18 @@ vCPU Scheduling
 ===============
 
 We introduce a new running mode where each vCPU will run on its own
-user-space thread. This will be enabled by default for all FE/BE
-combinations that have had the required work done to support this
-safely.
+user-space thread. This is enabled by default for all FE/BE
+combinations where the host memory model is able to accommodate the
+guest (TCG_GUEST_DEFAULT_MO & ~TCG_TARGET_DEFAULT_MO is zero) and the
+guest has had the required work done to support this safely
+(TARGET_SUPPORTS_MTTCG).
+
+System emulation will fall back to the original round robin approach
+if:
+
+* forced by --accel tcg,thread=single
+* enabling --icount mode
+* 64 bit guests on 32 bit hosts (TCG_OVERSIZED_GUEST)
 
 In the general case of running translated code there should be no
 inter-vCPU dependencies and all vCPUs should be able to run at full
@@ -61,7 +72,9 @@ have their block-to-block jumps patched.
 Global TCG State
 ----------------
 
-### User-mode emulation
+User-mode emulation
+~~~~~~~~~~~~~~~~~~~
+
 We need to protect the entire code generation cycle including any post
 generation patching of the translated code. This also implies a shared
 translation buffer which contains code running on all cores. Any
@@ -78,9 +91,11 @@ patching.
 
 Code generation is serialised with mmap_lock().
 
-### !User-mode emulation
+!User-mode emulation
+~~~~~~~~~~~~~~~~~~~~
+
 Each vCPU has its own TCG context and associated TCG region, thereby
-requiring no locking.
+requiring no locking during translation.
 
 Translation Blocks
 ------------------
@@ -92,6 +107,7 @@ including:
 
   - debugging operations (breakpoint insertion/removal)
   - some CPU helper functions
+  - linux-user spawning it's first thread
 
 This is done with the async_safe_run_on_cpu() mechanism to ensure all
 vCPUs are quiescent when changes are being made to shared global
@@ -250,8 +266,10 @@ to enforce a particular ordering of memory operations from the point
 of view of external observers (e.g. another processor core). They can
 apply to any memory operations as well as just loads or stores.
 
-The Linux kernel has an excellent write-up on the various forms of
-memory barrier and the guarantees they can provide [1].
+The Linux kernel has an excellent `write-up
+<https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt>`
+on the various forms of memory barrier and the guarantees they can
+provide.
 
 Barriers are often wrapped around synchronisation primitives to
 provide explicit memory ordering semantics. However they can be used
@@ -352,7 +370,3 @@ an exclusive lock which ensures all emulation is serialised.
 While the atomic helpers look good enough for now there may be a need
 to look at solutions that can more closely model the guest
 architectures semantics.
-
-==========
-
-[1] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/plain/Documentation/memory-barriers.txt
-- 
2.20.1

Re: [PATCH v2 2/3] docs/devel: convert and update MTTCG design document

[Richard Henderson]

On 7/1/20 9:11 AM, Alex Bennée wrote:
> -This document outlines the design for multi-threaded TCG system-mode
> -emulation. The current user-mode emulation mirrors the thread
> -structure of the translated executable. Some of the work will be
> -applicable to both system and linux-user emulation.
> +This document outlines the design for multi-threaded TCG (a.k.a MTTCG)
> +system-mode emulation. user-mode emulation has always mirrored the

Full stop wants capitalization for the next sentence.  Though perhaps em-dash
or semicolon would work as well to connect these clauses.


> +guest (TCG_GUEST_DEFAULT_MO & ~TCG_TARGET_DEFAULT_MO is zero) and the

Isn't there some markup for code?  `...` perhaps?


Otherwise,
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

r~

[PATCH v2 3/3] docs/devel: add some notes on tcg-icount for developers

[Alex Bennée]

This attempts to bring together my understanding of the requirements
for icount behaviour into one reference document for our developer
notes.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Pavel Dovgalyuk <dovgaluk@ispras.ru>
Cc: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200619135844.23307-1-alex.bennee@linaro.org>

---
v2
  - fix copyright date
  - it's -> its
  - drop mentioned of gen_io_end()
  - remove and correct original conjecture
v3
  - include link in index
---
 docs/devel/index.rst      |  1 +
 docs/devel/tcg-icount.rst | 89 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+)
 create mode 100644 docs/devel/tcg-icount.rst

diff --git a/docs/devel/index.rst b/docs/devel/index.rst
index 4ecaea3643f..ae6eac7c9c6 100644
--- a/docs/devel/index.rst
+++ b/docs/devel/index.rst
@@ -23,6 +23,7 @@ Contents:
    decodetree
    secure-coding-practices
    tcg
+   tcg-icount
    multi-thread-tcg
    tcg-plugins
    bitops
diff --git a/docs/devel/tcg-icount.rst b/docs/devel/tcg-icount.rst
new file mode 100644
index 00000000000..cb51cb34dde
--- /dev/null
+++ b/docs/devel/tcg-icount.rst
@@ -0,0 +1,89 @@
+..
+   Copyright (c) 2020, Linaro Limited
+   Written by Alex Bennée
+
+
+========================
+TCG Instruction Counting
+========================
+
+TCG has long supported a feature known as icount which allows for
+instruction counting during execution. This should be confused with
+cycle accurate emulation - QEMU does not attempt to emulate how long
+an instruction would take on real hardware. That is a job for other
+more detailed (and slower) tools that simulate the rest of a
+micro-architecture.
+
+This feature is only available for system emulation and is
+incompatible with multi-threaded TCG. It can be used to better align
+execution time with wall-clock time so a "slow" device doesn't run too
+fast on modern hardware. It can also provides for a degree of
+deterministic execution and is an essential part of the record/replay
+support in QEMU.
+
+Core Concepts
+=============
+
+At its heart icount is simply a count of executed instructions which
+is stored in the TimersState of QEMU's timer sub-system. The number of
+executed instructions can then be used to calculate QEMU_CLOCK_VIRTUAL
+which represents the amount of elapsed time in the system since
+execution started. Depending on the icount mode this may either be a
+fixed number of ns per instructions or adjusted as execution continues
+to keep wall clock time and virtual time in sync.
+
+To be able to calculate the number of executed instructions the
+translator starts by allocating a budget of instructions to be
+executed. The budget of instructions is limited by how long it will be
+until the next timer will expire. We store this budget as part of a
+vCPU icount_decr field which shared with the machinery for handling
+cpu_exit(). The whole field is checked at the start of every
+translated block and will cause a return to the outer loop to deal
+with whatever caused the exit.
+
+In the case of icount before the flag is checked we subtract the
+number of instructions the translation block would execute. If this
+would cause the instruction budget to got negative we exit the main
+loop and regenerate a new translation block with exactly the right
+number of instructions to take the budget to 0 meaning whatever timer
+was due to expire will expire exactly when we exit the main run loop.
+
+Dealing with MMIO
+-----------------
+
+While we can adjust the instruction budget for known events like timer
+expiry we can not do the same for MMIO. Every load/store we execute
+might potentially trigger an I/O event at which point we will need an
+up to date and accurate reading of the icount number.
+
+To deal with this case when an I/O access is made we:
+
+  - restore un-executed instructions to the icount budget
+  - re-compile a single [1]_ instruction block for the current PC
+  - exit the cpu loop and execute the re-compiled block
+
+The new block is created with the CF_LAST_IO compile flag which
+ensures the final instruction translation starts with a call to
+gen_io_start() so we don't enter a perpetual loop constantly
+recompiling a single instruction block. For translators using the
+common translator_loop this is done automatically.
+  
+.. [1] sometimes two instructions if dealing with delay slots  
+
+Other I/O operations
+--------------------
+
+MMIO isn't the only type of operation for which we might need a
+correct and accurate clock. IO port instructions and accesses to
+system registers are the common examples here. These instructions have
+to be handled by the individual translators which have the knowledge
+of which operations are I/O operations.
+
+.. warning:: Any instruction that eventually causes an access to
+             QEMU_CLOCK_VIRTUAL needs to be preceded by a
+             gen_io_start() and must also be the last instruction
+             translated in the block.
+   
+
+
+
-- 
2.20.1

Re: [PATCH v2 3/3] docs/devel: add some notes on tcg-icount for developers

[Peter Maydell]

On Wed, 1 Jul 2020 at 17:11, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> This attempts to bring together my understanding of the requirements
> for icount behaviour into one reference document for our developer
> notes.
>
> Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
> Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Pavel Dovgalyuk <dovgaluk@ispras.ru>
> Cc: Peter Maydell <peter.maydell@linaro.org>
> Message-Id: <20200619135844.23307-1-alex.bennee@linaro.org>
>
> ---
> v2
>   - fix copyright date
>   - it's -> its
>   - drop mentioned of gen_io_end()
>   - remove and correct original conjecture
> v3
>   - include link in index
> ---
>  docs/devel/index.rst      |  1 +
>  docs/devel/tcg-icount.rst | 89 +++++++++++++++++++++++++++++++++++++++
>  2 files changed, 90 insertions(+)
>  create mode 100644 docs/devel/tcg-icount.rst
>
> diff --git a/docs/devel/index.rst b/docs/devel/index.rst
> index 4ecaea3643f..ae6eac7c9c6 100644
> --- a/docs/devel/index.rst
> +++ b/docs/devel/index.rst
> @@ -23,6 +23,7 @@ Contents:
>     decodetree
>     secure-coding-practices
>     tcg
> +   tcg-icount
>     multi-thread-tcg
>     tcg-plugins
>     bitops
> diff --git a/docs/devel/tcg-icount.rst b/docs/devel/tcg-icount.rst
> new file mode 100644
> index 00000000000..cb51cb34dde
> --- /dev/null
> +++ b/docs/devel/tcg-icount.rst
> @@ -0,0 +1,89 @@
> +..
> +   Copyright (c) 2020, Linaro Limited
> +   Written by Alex Bennée
> +
> +
> +========================
> +TCG Instruction Counting
> +========================
> +
> +TCG has long supported a feature known as icount which allows for
> +instruction counting during execution. This should be confused with

Shurely "should not be confused" :-)

> +cycle accurate emulation - QEMU does not attempt to emulate how long
> +an instruction would take on real hardware. That is a job for other
> +more detailed (and slower) tools that simulate the rest of a
> +micro-architecture.
> +
> +This feature is only available for system emulation and is
> +incompatible with multi-threaded TCG. It can be used to better align
> +execution time with wall-clock time so a "slow" device doesn't run too
> +fast on modern hardware. It can also provides for a degree of
> +deterministic execution and is an essential part of the record/replay
> +support in QEMU.
> +
> +Core Concepts
> +=============
> +
> +At its heart icount is simply a count of executed instructions which
> +is stored in the TimersState of QEMU's timer sub-system. The number of
> +executed instructions can then be used to calculate QEMU_CLOCK_VIRTUAL
> +which represents the amount of elapsed time in the system since
> +execution started. Depending on the icount mode this may either be a
> +fixed number of ns per instructions or adjusted as execution continues

"per instruction"

> +to keep wall clock time and virtual time in sync.
> +
> +To be able to calculate the number of executed instructions the
> +translator starts by allocating a budget of instructions to be
> +executed. The budget of instructions is limited by how long it will be
> +until the next timer will expire. We store this budget as part of a
> +vCPU icount_decr field which shared with the machinery for handling
> +cpu_exit(). The whole field is checked at the start of every
> +translated block and will cause a return to the outer loop to deal
> +with whatever caused the exit.
> +
> +In the case of icount before the flag is checked we subtract the

"of icount, "

> +number of instructions the translation block would execute. If this
> +would cause the instruction budget to got negative we exit the main

"to go negative"

> +loop and regenerate a new translation block with exactly the right
> +number of instructions to take the budget to 0 meaning whatever timer

"to 0. This means that whatever timer"

> +was due to expire will expire exactly when we exit the main run loop.
> +
> +Dealing with MMIO
> +-----------------
> +
> +While we can adjust the instruction budget for known events like timer
> +expiry we can not do the same for MMIO. Every load/store we execute

"cannot"

> +might potentially trigger an I/O event at which point we will need an

"event, at which point"

> +up to date and accurate reading of the icount number.
> +
> +To deal with this case when an I/O access is made we:

"this case, when"

> +
> +  - restore un-executed instructions to the icount budget
> +  - re-compile a single [1]_ instruction block for the current PC
> +  - exit the cpu loop and execute the re-compiled block
> +
> +The new block is created with the CF_LAST_IO compile flag which
> +ensures the final instruction translation starts with a call to
> +gen_io_start() so we don't enter a perpetual loop constantly
> +recompiling a single instruction block. For translators using the
> +common translator_loop this is done automatically.
> +
> +.. [1] sometimes two instructions if dealing with delay slots
> +
> +Other I/O operations
> +--------------------
> +
> +MMIO isn't the only type of operation for which we might need a
> +correct and accurate clock. IO port instructions and accesses to
> +system registers are the common examples here. These instructions have
> +to be handled by the individual translators which have the knowledge
> +of which operations are I/O operations.
> +
> +.. warning:: Any instruction that eventually causes an access to
> +             QEMU_CLOCK_VIRTUAL needs to be preceded by a
> +             gen_io_start() and must also be the last instruction
> +             translated in the block.

I think I would prefer some text phrased in a way that more
explicitly states what the frontend code has to do, like:

======
When the translator is handling an instruction of this kind:
 * it must call gen_io_start() if icount is enabled, at some
   point before the generation of the code which actually does
   the I/O, using a code fragment similar to:
        if (tb_cflags(s->base.tb) & CF_USE_ICOUNT) {
            gen_io_start();
        }
 * it must end the TB immediately after this instruction

Note that some older front-ends call a "gen_io_end()" function:
this is obsolete and should not be used.
======

thanks
-- PMM

Re: [PATCH v2 0/3] some docs (booting, mttcg, icount)

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200701161153.30988-1-alex.bennee@linaro.org/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      util/cutils.o
  CC      util/unicode.o

Warning, treated as error:
/tmp/qemu-test/src/docs/interop/booting.rst:141:Title level inconsistent:

Hardware
---
  CC      util/qemu-thread-win32.o
  CC      util/envlist.o

Warning, treated as error:
/tmp/qemu-test/src/docs/interop/booting.rst:141:Title level inconsistent:

Hardware
~~~~~~~~
  CC      util/path.o
make: *** [Makefile:1108: .docs_interop_qemu-ga.8.sentinel.] Error 2
make: *** Deleting file '.docs_interop_qemu-ga.8.sentinel.'
make: *** Waiting for unfinished jobs....
make: *** [Makefile:1093: docs/interop/index.html] Error 2
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
    sys.exit(main())
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=8aa5ca1bf60d4a9a844021819abc67e4', '-u', '1001', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-chc922my/src/docker-src.2020-07-01-12.52.42.12836:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=8aa5ca1bf60d4a9a844021819abc67e4
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-chc922my/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    3m5.559s
user    0m8.570s


The full log is available at
http://patchew.org/logs/20200701161153.30988-1-alex.bennee@linaro.org/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [PATCH v2 0/3] some docs (booting, mttcg, icount)

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200701161153.30988-1-alex.bennee@linaro.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PATCH  v2 0/3] some docs (booting, mttcg, icount)
Type: series
Message-id: 20200701161153.30988-1-alex.bennee@linaro.org

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
f0c6933 docs/devel: add some notes on tcg-icount for developers
b4ed66b docs/devel: convert and update MTTCG design document
2c9c289 docs/booting.rst: start documenting the boot process

=== OUTPUT BEGIN ===
1/3 Checking commit 2c9c289aa77c (docs/booting.rst: start documenting the boot process)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#24: 
new file mode 100644

total: 0 errors, 1 warnings, 166 lines checked

Patch 1/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
2/3 Checking commit b4ed66b650ef (docs/devel: convert and update MTTCG design document)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#27: 
rename from docs/devel/multi-thread-tcg.txt

total: 0 errors, 1 warnings, 97 lines checked

Patch 2/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
3/3 Checking commit f0c6933d34a5 (docs/devel: add some notes on tcg-icount for developers)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

ERROR: trailing whitespace
#107: FILE: docs/devel/tcg-icount.rst:70:
+  $

ERROR: trailing whitespace
#108: FILE: docs/devel/tcg-icount.rst:71:
+.. [1] sometimes two instructions if dealing with delay slots  $

ERROR: trailing whitespace
#123: FILE: docs/devel/tcg-icount.rst:86:
+   $

total: 3 errors, 1 warnings, 96 lines checked

Patch 3/3 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200701161153.30988-1-alex.bennee@linaro.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com