Re: [PATCH 0/6] tracetool: add mypy --strict checking [AI discussion ahead!]

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 7156 bytes --]

On Wed, Oct 08, 2025 at 08:35:39AM +0200, Paolo Bonzini wrote:
> [People in Cc are a mix of Python people, tracing people, and people
>  who followed the recent AI discussions. - Paolo]
> 
> This series adds type annotations to tracetool. While useful on its own, 
> it also served as an experiment in whether AI tools could be useful and
> appropriate for mechanical code transformations that may not involve
> copyrightable expression.
> 
> In this version, the types were added mostly with the RightTyper tool
> (https://github.com/RightTyper/RightTyper), which uses profiling to detect
> the types of arguments and return types at run time.  However, because
> adding type annotations is such a narrow and verifiable task, I also developed
> a parallel version using an LLM, to provide some data on topics such as:
> 
> - how much choice/creativity is there in writing type annotations?
>   Is it closer to writing functional code or to refactoring?
> 
> - how does AI output for such mechanical transformations compare to other
>   automated tools, whose output is known not to be copyrightable? 
> 
> - what is the kind of time saving that the tool can provide?
> 
> - how effective is an LLM for this task?  Is the required human help a
>   pain to provide, or can the human keep the fun part for itself?
> 
> While the version submitted here does not use any LLM-generated content
> in compliance with QEMU's policy, the first version was developed using
> an LLM tool (Claude Code) with a very simple prompt:
>     
>       Add type annotations to tracetool.py and all .py files in
>       tracetool/*.py.  Use new-style annotations, for example list[str]
>       instead of typing.List[str].  When done, use mypy to find any
>       issues.
> 
> The results were surprisingly good for such a simple-minded test, and
> while there were some issues they turned out to be mostly due to dead
> code that confused the LLM.  Thus, I removed the dead code (this is
> already in qemu.git) and then did the actual experiment, running
> both the RightTyper script (which is detailed in patch 4) and the LLM
> in separate branches.  The remaining errors from the LLM were also
> reasonably easy to fix, so I did that manually:
> 
>       tracetool/__init__.py:239: error: Need type annotation for "_args" (hint: "_args: list[<type>] = ...")  [var-annotated]
>       tracetool/__init__.py:272: error: Argument 1 to "Arguments" has incompatible type "list[tuple[str, str]]"; expected "list[tuple[str, str] | Arguments]"  [arg-type]
>       tracetool/__init__.py:579: error: Incompatible types in assignment (expression has type "str | None", variable has type "None")  [assignment]
>       tracetool/__init__.py:580: error: Incompatible types in assignment (expression has type "str | None", variable has type "None")  [assignment]
> 
> After reviewing the changes, I followed up with another prompt for mechanical
> changes:
> 
>       Change "backend: Any" to "backend: Wrapper" in all tracetool/format/ files
> 
> Honestly, I could have done this part in less time without any help; I
> was just curious to see if the LLM would also remove the unused import.
> Not only it didn't; this prompt didn't even touch most of the files so
> I did the change, ran isort and called it a day.
> 
> Comparing the results from RightTyper and AI, both tools benefit from human
> help: the dead code removal which I mentioned, the small cleanups in patch 1,
> the final manual fixes for the remaining (mostly trivial) errors.  But at
> least in this case, AI is a winner:
> 
> - it left basically no unannotated code: fixing the above errors was enough
>   to pass "mypy --strict", unlike RightTyper which needed a little more work
>   due to its profiling-based nature and a few other limitations (see patch 5).
> 
> - most importantly, trying various other tools that didn't work, as well as
>   figuring out how to use RightTyper, took a couple hours more.  Surprising
>   as it was, I could not find any static type inferencing tool for Python;
>   neither pytype nor pyre worked for me.  This is also why I think this
>   is not apples to oranges, but a fair comparison between AI-based and
>   regular tooling.
> 
> I want to highlight "in this case".  I had other experiments where the
> LLM's output was all but awful, and I wasted time debugging code that
> I should have thrown away immediately!
> 
> After the diffstat, you can find a diff from this series to the version
> based on Claude Code.  It's impossible to be 100% objective but,
> besides being functionally equivalent, I don't think either would be
> identifiable as written by an LLM, by a person, by a tool+human combo,
> or even by a good type inferencing tool (if it existed).
> 
> Based on this experience, my answer to the copyrightability question is
> that, for this kind of narrow request, the output of AI can be treated as
> the output of an imperfect tool, rather than as creative content potentially
> tainted by the training material.  Of course this is one data point and
> is intended as an experiment rather than a policy recommendation.
> 
> Paolo
> 
> 
> 
> Paolo Bonzini (6):
>   tracetool: rename variable with conflicting types
>   tracetool: apply isort and add check
>   tracetool: "import annotations"
>   tracetool: add type annotations
>   tracetool: complete typing annotations
>   tracetool: add typing checks to "make -C python check"
> 
>  python/tests/tracetool-isort.sh              |  4 +
>  python/tests/tracetool-mypy.sh               |  5 ++
>  scripts/tracetool.py                         | 12 +--
>  scripts/tracetool/__init__.py                | 84 ++++++++++----------
>  scripts/tracetool/backend/__init__.py        | 21 ++---
>  scripts/tracetool/backend/dtrace.py          | 19 ++---
>  scripts/tracetool/backend/ftrace.py          | 13 +--
>  scripts/tracetool/backend/log.py             | 13 +--
>  scripts/tracetool/backend/simple.py          | 19 ++---
>  scripts/tracetool/backend/syslog.py          | 13 +--
>  scripts/tracetool/backend/ust.py             | 11 +--
>  scripts/tracetool/format/__init__.py         |  9 ++-
>  scripts/tracetool/format/c.py                |  7 +-
>  scripts/tracetool/format/d.py                |  7 +-
>  scripts/tracetool/format/h.py                |  7 +-
>  scripts/tracetool/format/log_stap.py         | 12 +--
>  scripts/tracetool/format/rs.py               |  7 +-
>  scripts/tracetool/format/simpletrace_stap.py |  7 +-
>  scripts/tracetool/format/stap.py             | 10 ++-
>  scripts/tracetool/format/ust_events_c.py     |  7 +-
>  scripts/tracetool/format/ust_events_h.py     |  7 +-
>  21 files changed, 173 insertions(+), 121 deletions(-)
>  create mode 100755 python/tests/tracetool-isort.sh
>  create mode 100755 python/tests/tracetool-mypy.sh
> -- 
> 2.51.0
> 

Discussion on AI policy (does not affect these patches) and future
cleanups aside, this looks good to go.

Thanks, applied to my tracing tree:
https://gitlab.com/stefanha/qemu/commits/tracing

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]