[RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[Alex Bennée]

This attempts to bring together my understanding of the requirements
for icount behaviour into one reference document for our developer
notes. It currently make one piece of conjecture which I think is true
that we don't need gen_io_start/end statements for non-MMIO related
I/O operations.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Pavel Dovgalyuk <dovgaluk@ispras.ru>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Peter Maydell <peter.maydell@linaro.org>
---
 docs/devel/tcg-icount.rst | 86 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 86 insertions(+)
 create mode 100644 docs/devel/tcg-icount.rst

diff --git a/docs/devel/tcg-icount.rst b/docs/devel/tcg-icount.rst
new file mode 100644
index 00000000000..53d08ce9282
--- /dev/null
+++ b/docs/devel/tcg-icount.rst
@@ -0,0 +1,86 @@
+..
+   Copyright (c) 2019, Linaro Limited
+   Written by Alex Bennée
+
+
+========================
+TCG Instruction Counting
+========================
+
+TCG has long supported a feature known as icount which allows for
+instruction counting during execution. This should be confused with
+cycle accurate emulation - QEMU does not attempt to emulate how long
+an instruction would take on real hardware. That is a job for other
+more detailed (and slower) tools that simulate the rest of a
+micro-architecture.
+
+This feature is only available for system emulation and is
+incompatible with multi-threaded TCG. It can be used to better align
+execution time with wall-clock time so a "slow" device doesn't run too
+fast on modern hardware. It can also provides for a degree of
+deterministic execution and is an essential part of the record/replay
+support in QEMU.
+
+Core Concepts
+=============
+
+At it's heart icount is simply a count of executed instructions which
+is stored in the TimersState of QEMU's timer sub-system. The number of
+executed instructions can then be used to calculate QEMU_CLOCK_VIRTUAL
+which represents the amount of elapsed time in the system since
+execution started. Depending on the icount mode this may either be a
+fixed number of ns per instructions or adjusted as execution continues
+to keep real time and virtual time in sync.
+
+To be able to calculate the number of executed instructions the
+translator starts by allocating a budget of instructions to be
+executed. The budget of instructions is limited by how long it will be
+until the next timer will expire. We store this budget as part of a
+CPUs icount_decr field which shared with the machinery for handling
+cpu_exit(). The whole field is checked at the start of every
+translated block and will cause us to return to the outer loop to deal
+with whatever caused the exit.
+
+In the case of icount before the flag is checked we subtract the
+number of instructions the translation block would execute. If this
+would cause the instruction budget to got negative we exit the main
+loop and regenerate a new translation block with exactly the right
+number of instructions to take the budget to 0 meaning whatever timer
+was due to expire will expire exactly when we exit the main run loop.
+
+Dealing with MMIO
+-----------------
+
+While we can adjust the instruction budget for known events like timer
+expiry we can not do the same for MMIO. Every load/store we execute
+might potentially trigger an I/O event at which point we will need an
+up to date and accurate reading of the icount number.
+
+To deal with this case when an I/O access is made we:
+
+  - restore un-executed instructions to the icount budget
+  - re-compile a single [1]_ instruction block for the current PC
+  - exit the cpu loop and execute the re-compiled block
+
+The new block is created with the CF_LAST_IO compile flag which
+ensures the final instruction is wrapped with a
+gen_io_start()/gen_io_end() pair so we don't enter a perpetual loop
+constantly recompiling a single instruction block. For translators
+using the common translator_loop this is done automatically.
+  
+.. [1] sometimes two instructions if dealing with delay slots  
+
+Other I/O operations
+--------------------
+
+MMIO isn't the only type of operation for which we might need a
+correct and accurate clock. IO port instructions and accesses to
+system registers are the common examples here. For the clock to be
+accurate you end a translation block on these instructions.
+
+.. warning:: (CONJECTURE) instructions that won't get trapped in the
+             io_read/writex shouldn't need gen_io_start/end blocks
+             around them.
+
+
+
-- 
2.20.1

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200619135844.23307-1-alex.bennee@linaro.org/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      crypto/tlscredsx509.o
  CC      crypto/tlssession.o

Warning, treated as error:
/tmp/qemu-test/src/docs/devel/tcg-icount.rst:document isn't included in any toctree
  CC      crypto/secret_common.o
  CC      crypto/secret.o
---
  CC      qom/qom-qobject.o
  CC      qom/object_interfaces.o
  CC      qemu-io.o
make: *** [Makefile:1088: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=22d5a51c49734f0fbfca837a9ba20368', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-r_d8dc9j/src/docker-src.2020-06-19-10.47.26.31220:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=22d5a51c49734f0fbfca837a9ba20368
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-r_d8dc9j/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    2m44.378s
user    0m8.422s


The full log is available at
http://patchew.org/logs/20200619135844.23307-1-alex.bennee@linaro.org/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200619135844.23307-1-alex.bennee@linaro.org/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers
Type: series
Message-id: 20200619135844.23307-1-alex.bennee@linaro.org

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
2cb15db docs/devel: add some notes on tcg-icount for developers

=== OUTPUT BEGIN ===
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#21: 
new file mode 100644

ERROR: trailing whitespace
#95: FILE: docs/devel/tcg-icount.rst:70:
+  $

ERROR: trailing whitespace
#96: FILE: docs/devel/tcg-icount.rst:71:
+.. [1] sometimes two instructions if dealing with delay slots  $

total: 2 errors, 1 warnings, 86 lines checked

Commit 2cb15db2a94c (docs/devel: add some notes on tcg-icount for developers) has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200619135844.23307-1-alex.bennee@linaro.org/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[Peter Maydell]

On Fri, 19 Jun 2020 at 14:58, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> This attempts to bring together my understanding of the requirements
> for icount behaviour into one reference document for our developer
> notes. It currently make one piece of conjecture which I think is true
> that we don't need gen_io_start/end statements for non-MMIO related
> I/O operations.

> +Other I/O operations
> +--------------------
> +
> +MMIO isn't the only type of operation for which we might need a
> +correct and accurate clock. IO port instructions and accesses to
> +system registers are the common examples here. For the clock to be
> +accurate you end a translation block on these instructions.
> +
> +.. warning:: (CONJECTURE) instructions that won't get trapped in the
> +             io_read/writex shouldn't need gen_io_start/end blocks
> +             around them.

I think this is backwards -- instructions where icount is handled
by io_readx/io_writex are the ones that don't need to be marked
with gen_io_start. It's the i/o instructions that don't go through
io_readx/io_writex that need gen_io_start.

thanks
-- PMM

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200619135844.23307-1-alex.bennee@linaro.org/



Hi,

This series failed the asan build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-debug@fedora TARGET_LIST=x86_64-softmmu J=14 NETWORK=1
=== TEST SCRIPT END ===

  CC      ui/trace.o
  CC      hw/core/trace.o

Warning, treated as error:
/tmp/qemu-test/src/docs/devel/tcg-icount.rst:document isn't included in any toctree
  CC      hw/display/trace.o
  CC      qapi/trace.o
---
  CC      stubs/runstate-check.o
  CC      stubs/semihost.o
  CC      stubs/set-fd-handler.o
make: *** [Makefile:1088: docs/devel/index.html] Error 2
make: *** Waiting for unfinished jobs....
Traceback (most recent call last):
  File "./tests/docker/docker.py", line 669, in <module>
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--label', 'com.qemu.instance.uuid=4b7116456d8441268d522e8434872604', '-u', '1003', '--security-opt', 'seccomp=unconfined', '--rm', '-e', 'TARGET_LIST=x86_64-softmmu', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew2/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-wgrif8gc/src/docker-src.2020-06-19-10.51.34.8511:/var/tmp/qemu:z,ro', 'qemu:fedora', '/var/tmp/qemu/run', 'test-debug']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=4b7116456d8441268d522e8434872604
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-wgrif8gc/src'
make: *** [docker-run-test-debug@fedora] Error 2

real    4m14.587s
user    0m8.118s


The full log is available at
http://patchew.org/logs/20200619135844.23307-1-alex.bennee@linaro.org/testing.asan/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[Paolo Bonzini]

On 19/06/20 16:54, Peter Maydell wrote:
>> +
>> +MMIO isn't the only type of operation for which we might need a
>> +correct and accurate clock. IO port instructions and accesses to
>> +system registers are the common examples here. For the clock to be
>> +accurate you end a translation block on these instructions.
>> +
>> +.. warning:: (CONJECTURE) instructions that won't get trapped in the
>> +             io_read/writex shouldn't need gen_io_start/end blocks
>> +             around them.
> I think this is backwards -- instructions where icount is handled
> by io_readx/io_writex are the ones that don't need to be marked
> with gen_io_start. It's the i/o instructions that don't go through
> io_readx/io_writex that need gen_io_start.

Yes, and likewise instructions where icount is handled by
io_readx/io_writex need not terminate the TB.

Paolo

Re: [RFC PATCH] docs/devel: add some notes on tcg-icount for developers

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Fri, 19 Jun 2020 at 14:58, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> This attempts to bring together my understanding of the requirements
>> for icount behaviour into one reference document for our developer
>> notes. It currently make one piece of conjecture which I think is true
>> that we don't need gen_io_start/end statements for non-MMIO related
>> I/O operations.
>
>> +Other I/O operations
>> +--------------------
>> +
>> +MMIO isn't the only type of operation for which we might need a
>> +correct and accurate clock. IO port instructions and accesses to
>> +system registers are the common examples here. For the clock to be
>> +accurate you end a translation block on these instructions.
>> +
>> +.. warning:: (CONJECTURE) instructions that won't get trapped in the
>> +             io_read/writex shouldn't need gen_io_start/end blocks
>> +             around them.
>
> I think this is backwards -- instructions where icount is handled
> by io_readx/io_writex are the ones that don't need to be marked
> with gen_io_start. It's the i/o instructions that don't go through
> io_readx/io_writex that need gen_io_start.

There are two types of MMIO accesses we generate:

 - normal loads/stores which if they end up accessing I/O ports
   eventually trap in io_read/writex where as they haven't been marked
   with can_do_io (via gen_io_start()) will trigger a recompile and
   exit.

 - loads and stores emitted while CF_LAST_IO && icount is in effect
   (from the above recompile) where they should be the last instruction
   in the block so the icount is correct when the do whatever they do.

What I've missed is the one other place where cpu->can_do_io is checked
which is when we read cpu_get_icount_raw_locked. In this case it is
effectively an assert that we have marked the instruction as potentially
accessing icount - although not that the gen_io_start/end was actually on
the last instruction of the block.

-- 
Alex Bennée