[PATCH 0/7] Enhance documentation for new developers

[PATCH 0/7] Enhance documentation for new developers

[Pierrick Bouvier]

This series extends our documentation with new pages to help developers
onboarding on QEMU. It focuses on providing a big picture of QEMU (to a
modest extend).

As such, it was written to be simple, short, easy to understand, and pointing to
more details. It provides another way to dive into details instead of simply
hitting the "search" box.

The first patches enhance the existing developer section. They provide
information about b4 and git-publish, two important tools that I learnt from my
coworkers, and were not presented anywhere, and were really missing IMHO.

Then, we introduce a new Codebase page, presenting (succintly) the various parts
of QEMU, and what every folder of the codebase contains.
We then add a glossary with the most recurrent acronyms we hear in our daily
conversations on the mailing list.
Finally, we add an "How-to" page which present how to build and test qemu, and
how to contribute a patch. It's definitely a repetition of existing information,
but the goal was to have a self contained page with all the commands I run
daily personally, and that someone would be interested to have.

When reviewing, please keep in mind this is targeting someone who discovers
QEMU, and not someone who contributed to the project for several years. What is
obvious for you will not be obvious for a random young developer.

That said, please free to point if something is "false", or "really incomplete".
It can be hard to summarize in one or two sentences complex parts, but that's
the intent here.

Your feedback on content or organization is very welcome!

Thanks,
Pierrick

Pierrick Bouvier (7):
  docs/devel: remove dead video link for sourcehut submit process
  docs/devel: add git-publish for patch submitting
  docs/devel: add b4 for patch retrieval
  docs/devel: add information on how to setup build environments
  docs: add a codebase section
  docs: add a glossary
  docs: add a how to section

 docs/about/build-platforms.rst         |   4 +-
 docs/about/emulation.rst               |   2 +
 docs/codebase/index.rst                | 211 ++++++++++++++++++++++
 docs/devel/build-environment.rst       | 114 ++++++++++++
 docs/devel/build-system.rst            |   2 +
 docs/devel/control-flow-integrity.rst  |   2 +
 docs/devel/decodetree.rst              |   2 +
 docs/devel/ebpf_rss.rst                |   2 +
 docs/devel/index-build.rst             |   1 +
 docs/devel/index-internals.rst         |   2 +
 docs/devel/migration/main.rst          |   2 +
 docs/devel/multi-thread-tcg.rst        |   2 +
 docs/devel/qapi-code-gen.rst           |   1 +
 docs/devel/submitting-a-patch.rst      |  29 ++-
 docs/devel/testing/main.rst            |   9 +-
 docs/devel/testing/qtest.rst           |   2 +
 docs/glossary/index.rst                | 238 +++++++++++++++++++++++++
 docs/how-to/index.rst                  | 146 +++++++++++++++
 docs/index.rst                         |   5 +
 docs/interop/qemu-ga.rst               |   2 +
 docs/system/arm/virt.rst               |   2 +
 docs/system/images.rst                 |   2 +
 docs/system/qemu-block-drivers.rst.inc |   2 +
 docs/tools/qemu-nbd.rst                |   2 +
 docs/tools/qemu-storage-daemon.rst     |   2 +
 docs/user/main.rst                     |   6 +
 26 files changed, 788 insertions(+), 6 deletions(-)
 create mode 100644 docs/codebase/index.rst
 create mode 100644 docs/devel/build-environment.rst
 create mode 100644 docs/glossary/index.rst
 create mode 100644 docs/how-to/index.rst

-- 
2.39.5

[PATCH 1/7] docs/devel: remove dead video link for sourcehut submit process

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/devel/submitting-a-patch.rst | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
index 83e9092b8c0..349c32ee3a9 100644
--- a/docs/devel/submitting-a-patch.rst
+++ b/docs/devel/submitting-a-patch.rst
@@ -252,10 +252,7 @@ patches to the QEMU mailing list by following these steps:
 #. Send your patches to the QEMU mailing list using the web-based
    ``git-send-email`` UI at https://git.sr.ht/~USERNAME/qemu/send-email
 
-`This video
-<https://spacepub.space/videos/watch/ad258d23-0ac6-488c-83fc-2bacf578de3a>`__
-shows the web-based ``git-send-email`` workflow. Documentation is
-available `here
+Documentation for sourcehut is available `here
 <https://man.sr.ht/git.sr.ht/#sending-patches-upstream>`__.
 
 .. _cc_the_relevant_maintainer:
-- 
2.39.5

Re: [PATCH 1/7] docs/devel: remove dead video link for sourcehut submit process

[Thomas Huth]

On 18/11/2024 18.23, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>   docs/devel/submitting-a-patch.rst | 5 +----
>   1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
> index 83e9092b8c0..349c32ee3a9 100644
> --- a/docs/devel/submitting-a-patch.rst
> +++ b/docs/devel/submitting-a-patch.rst
> @@ -252,10 +252,7 @@ patches to the QEMU mailing list by following these steps:
>   #. Send your patches to the QEMU mailing list using the web-based
>      ``git-send-email`` UI at https://git.sr.ht/~USERNAME/qemu/send-email
>   
> -`This video
> -<https://spacepub.space/videos/watch/ad258d23-0ac6-488c-83fc-2bacf578de3a>`__
> -shows the web-based ``git-send-email`` workflow. Documentation is
> -available `here
> +Documentation for sourcehut is available `here
>   <https://man.sr.ht/git.sr.ht/#sending-patches-upstream>`__.
>   
>   .. _cc_the_relevant_maintainer:

Reviewed-by: Thomas Huth <thuth@redhat.com>

[PATCH 2/7] docs/devel: add git-publish for patch submitting

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
index 349c32ee3a9..953682f20cb 100644
--- a/docs/devel/submitting-a-patch.rst
+++ b/docs/devel/submitting-a-patch.rst
@@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
 
 .. _if_you_cannot_send_patch_emails:
 
+Use git-publish
+~~~~~~~~~~~~~~~
+
+If you already configured git send-email, you can simply use `git-publish
+<https://github.com/stefanha/git-publish>`__ to send series.
+
+::
+
+    $ git checkout master -b my-feature
+    $ # work on new commits, add your 'Signed-off-by' lines to each
+    $ git publish
+    $ ... more work, rebase on master, ...
+    $ git publish # will send a v2
+
 If you cannot send patch emails
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-- 
2.39.5

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Marcin Juszkiewicz]

W dniu 18.11.2024 o 18:23, Pierrick Bouvier pisze:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>   docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>   1 file changed, 14 insertions(+)
> 
> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
> index 349c32ee3a9..953682f20cb 100644
> --- a/docs/devel/submitting-a-patch.rst
> +++ b/docs/devel/submitting-a-patch.rst
> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>   
>   .. _if_you_cannot_send_patch_emails:
>   
> +Use git-publish
> +~~~~~~~~~~~~~~~
> +
> +If you already configured git send-email, you can simply use `git-publish
> +<https://github.com/stefanha/git-publish>`__ to send series.
> +
> +::
> +
> +    $ git checkout master -b my-feature
> +    $ # work on new commits, add your 'Signed-off-by' lines to each
> +    $ git publish
> +    $ ... more work, rebase on master, ...
> +    $ git publish # will send a v2

You recommend 'b4 shazam' in 3/7 patch so why not here? Both 'b4' and 
'git-publish' seem to do same stuff - handle patch series and send them 
upstream.

b4 allows to keep To/Cc emails inside of cover letter which makes it 
easy to not miss anyone needed.

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Daniel P. Berrangé]

On Tue, Nov 19, 2024 at 09:41:40AM +0100, Marcin Juszkiewicz wrote:
> W dniu 18.11.2024 o 18:23, Pierrick Bouvier pisze:
> > Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> > ---
> >   docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
> >   1 file changed, 14 insertions(+)
> > 
> > diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
> > index 349c32ee3a9..953682f20cb 100644
> > --- a/docs/devel/submitting-a-patch.rst
> > +++ b/docs/devel/submitting-a-patch.rst
> > @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
> >   .. _if_you_cannot_send_patch_emails:
> > +Use git-publish
> > +~~~~~~~~~~~~~~~
> > +
> > +If you already configured git send-email, you can simply use `git-publish
> > +<https://github.com/stefanha/git-publish>`__ to send series.
> > +
> > +::
> > +
> > +    $ git checkout master -b my-feature
> > +    $ # work on new commits, add your 'Signed-off-by' lines to each
> > +    $ git publish
> > +    $ ... more work, rebase on master, ...
> > +    $ git publish # will send a v2
> 
> You recommend 'b4 shazam' in 3/7 patch so why not here? Both 'b4' and
> 'git-publish' seem to do same stuff - handle patch series and send them
> upstream.

git-publish is what we already recommended to people both elsewhere in
this file, and more prominently in the README.rst file, so maintaining
that practice is correct.

> b4 allows to keep To/Cc emails inside of cover letter which makes it easy to
> not miss anyone needed.

git-publish automatically CC's people by correlating the files touched in
the commits against our MAINTAINERS file, so it "does the right thing"
in the majority of cases. We recommend it because it makes it much harder
for novice users to mess up patch series submission.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Pierrick Bouvier]

On 11/19/24 01:12, Daniel P. Berrangé wrote:
> On Tue, Nov 19, 2024 at 09:41:40AM +0100, Marcin Juszkiewicz wrote:
>> W dniu 18.11.2024 o 18:23, Pierrick Bouvier pisze:
>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> ---
>>>    docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>>>    1 file changed, 14 insertions(+)
>>>
>>> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
>>> index 349c32ee3a9..953682f20cb 100644
>>> --- a/docs/devel/submitting-a-patch.rst
>>> +++ b/docs/devel/submitting-a-patch.rst
>>> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>>>    .. _if_you_cannot_send_patch_emails:
>>> +Use git-publish
>>> +~~~~~~~~~~~~~~~
>>> +
>>> +If you already configured git send-email, you can simply use `git-publish
>>> +<https://github.com/stefanha/git-publish>`__ to send series.
>>> +
>>> +::
>>> +
>>> +    $ git checkout master -b my-feature
>>> +    $ # work on new commits, add your 'Signed-off-by' lines to each
>>> +    $ git publish
>>> +    $ ... more work, rebase on master, ...
>>> +    $ git publish # will send a v2
>>
>> You recommend 'b4 shazam' in 3/7 patch so why not here? Both 'b4' and
>> 'git-publish' seem to do same stuff - handle patch series and send them
>> upstream.
> 
> git-publish is what we already recommended to people both elsewhere in
> this file, and more prominently in the README.rst file, so maintaining
> that practice is correct.
> 

At the time I started working on QEMU, I missed that information, 
because I focused on the manual (missing the content of the README), so 
that's why I add it here as well.

>> b4 allows to keep To/Cc emails inside of cover letter which makes it easy to
>> not miss anyone needed.
> 
> git-publish automatically CC's people by correlating the files touched in
> the commits against our MAINTAINERS file, so it "does the right thing"
> in the majority of cases. We recommend it because it makes it much harder
> for novice users to mess up patch series submission.
>

I agree with that. From my personal experience, the hardest part when 
onboarding with email workflow is more to understand it (which steps you 
go through until the contribution is accepted), more than learning a 
specific tool in particular.
So having a "simple" tool like git-publish is good.

> With regards,
> Daniel

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Pierrick Bouvier]

On 11/19/24 00:41, Marcin Juszkiewicz wrote:
> W dniu 18.11.2024 o 18:23, Pierrick Bouvier pisze:
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>    docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>>    1 file changed, 14 insertions(+)
>>
>> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
>> index 349c32ee3a9..953682f20cb 100644
>> --- a/docs/devel/submitting-a-patch.rst
>> +++ b/docs/devel/submitting-a-patch.rst
>> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>>    
>>    .. _if_you_cannot_send_patch_emails:
>>    
>> +Use git-publish
>> +~~~~~~~~~~~~~~~
>> +
>> +If you already configured git send-email, you can simply use `git-publish
>> +<https://github.com/stefanha/git-publish>`__ to send series.
>> +
>> +::
>> +
>> +    $ git checkout master -b my-feature
>> +    $ # work on new commits, add your 'Signed-off-by' lines to each
>> +    $ git publish
>> +    $ ... more work, rebase on master, ...
>> +    $ git publish # will send a v2
> 
> You recommend 'b4 shazam' in 3/7 patch so why not here? Both 'b4' and
> 'git-publish' seem to do same stuff - handle patch series and send them
> upstream.
>

Are you using b4 for your QEMU contributions?

When I started working on QEMU, I tried it before git-publish and had an 
error when trying to send my series. I don't remember exactly what it 
was, but something related to finding the reviewers for a given commit. 
When I asked, team members pointed me towards git-publish.

> b4 allows to keep To/Cc emails inside of cover letter which makes it
> easy to not miss anyone needed.

git-publish works pretty well for me, and does what you expect (keep 
your cover letter/cc and track your versions).

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Alex Bennée]

Pierrick Bouvier <pierrick.bouvier@linaro.org> writes:

> On 11/19/24 00:41, Marcin Juszkiewicz wrote:
>> W dniu 18.11.2024 o 18:23, Pierrick Bouvier pisze:
>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> ---
>>>    docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>>>    1 file changed, 14 insertions(+)
>>>
>>> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
>>> index 349c32ee3a9..953682f20cb 100644
>>> --- a/docs/devel/submitting-a-patch.rst
>>> +++ b/docs/devel/submitting-a-patch.rst
>>> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>>>       .. _if_you_cannot_send_patch_emails:
>>>    +Use git-publish
>>> +~~~~~~~~~~~~~~~
>>> +
>>> +If you already configured git send-email, you can simply use `git-publish
>>> +<https://github.com/stefanha/git-publish>`__ to send series.
>>> +
>>> +::
>>> +
>>> +    $ git checkout master -b my-feature
>>> +    $ # work on new commits, add your 'Signed-off-by' lines to each
>>> +    $ git publish
>>> +    $ ... more work, rebase on master, ...
>>> +    $ git publish # will send a v2
>> You recommend 'b4 shazam' in 3/7 patch so why not here? Both 'b4'
>> and
>> 'git-publish' seem to do same stuff - handle patch series and send them
>> upstream.
>>
>
> Are you using b4 for your QEMU contributions?
>
> When I started working on QEMU, I tried it before git-publish and had
> an error when trying to send my series. I don't remember exactly what
> it was, but something related to finding the reviewers for a given
> commit. When I asked, team members pointed me towards git-publish.

I tend to use b4 to apply patches because it works well with the
archives and collects together the tags for you. I have not used it to
publish patches to the list.

My sending workflow used done manually (rather via my carefully
customised Emacs org workflow) but now I just use git publish to send
series to the list. I still occasionally use git send-email directly for
sending individual patches with something like:

  git send-email --subject-prefix="RFC PATCH" --to qemu HEAD^..

I still send pull requests by hand:

  git send-email --confirm=never --quiet ${mailto} ${series}.pull/*

but thats probably because I haven't figured out the steps for git
publish to do that.

Anyway tldr; b4 to apply *from* list, git publish to send *to* list.

>
>> b4 allows to keep To/Cc emails inside of cover letter which makes it
>> easy to not miss anyone needed.
>
> git-publish works pretty well for me, and does what you expect (keep
> your cover letter/cc and track your versions).

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Daniel P. Berrangé]

On Wed, Nov 20, 2024 at 09:51:27AM +0000, Alex Bennée wrote:
> 
> I still send pull requests by hand:
> 
>   git send-email --confirm=never --quiet ${mailto} ${series}.pull/*
> 
> but thats probably because I haven't figured out the steps for git
> publish to do that.

A one-time setup task is to tell it what you preferred git remote is.
I use "gitlab" as the name of my personal gitlab fork of QEMU, so 

  $ git config remote.pushDefault gitlab

With that one-time setup task, sending a pull rquest needs nothing more
than adding  '--pull' to the git-publish cli. It'll trigger signing of
the tag, so be sure you have your gpg key available, and then push both
the branch and signed tag to your git remote, and send emails in its
normal way.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Daniel P. Berrangé]

On Mon, Nov 18, 2024 at 09:23:52AM -0800, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
> 
> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
> index 349c32ee3a9..953682f20cb 100644
> --- a/docs/devel/submitting-a-patch.rst
> +++ b/docs/devel/submitting-a-patch.rst
> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>  
>  .. _if_you_cannot_send_patch_emails:

This needs to remain associated with the heading that is *after*
your new block of text, while here you need to add a new anchor

   .. _use_git_publish:

>  
> +Use git-publish
> +~~~~~~~~~~~~~~~
> +
> +If you already configured git send-email, you can simply use `git-publish
> +<https://github.com/stefanha/git-publish>`__ to send series.
> +
> +::
> +
> +    $ git checkout master -b my-feature
> +    $ # work on new commits, add your 'Signed-off-by' lines to each
> +    $ git publish
> +    $ ... more work, rebase on master, ...
> +    $ git publish # will send a v2

Also add a note

 "Each time you post a series, git-publish will create a local tag
  with the format ``<branchname>-v<version>`` to record the patch
  series."

> +
>  If you cannot send patch emails
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  
> -- 
> 2.39.5
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 2/7] docs/devel: add git-publish for patch submitting

[Pierrick Bouvier]

On 11/19/24 01:04, Daniel P. Berrangé wrote:
> On Mon, Nov 18, 2024 at 09:23:52AM -0800, Pierrick Bouvier wrote:
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   docs/devel/submitting-a-patch.rst | 14 ++++++++++++++
>>   1 file changed, 14 insertions(+)
>>
>> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
>> index 349c32ee3a9..953682f20cb 100644
>> --- a/docs/devel/submitting-a-patch.rst
>> +++ b/docs/devel/submitting-a-patch.rst
>> @@ -237,6 +237,20 @@ attachments can be used as a last resort on a first-time submission.
>>   
>>   .. _if_you_cannot_send_patch_emails:
> 
> This needs to remain associated with the heading that is *after*
> your new block of text, while here you need to add a new anchor
> 

Oops, that was not intentional, thanks for pointing this.
Fixed and added the new anchor in case someone needs it.

>     .. _use_git_publish:
> 
>>   
>> +Use git-publish
>> +~~~~~~~~~~~~~~~
>> +
>> +If you already configured git send-email, you can simply use `git-publish
>> +<https://github.com/stefanha/git-publish>`__ to send series.
>> +
>> +::
>> +
>> +    $ git checkout master -b my-feature
>> +    $ # work on new commits, add your 'Signed-off-by' lines to each
>> +    $ git publish
>> +    $ ... more work, rebase on master, ...
>> +    $ git publish # will send a v2
> 
> Also add a note
> 
>   "Each time you post a series, git-publish will create a local tag
>    with the format ``<branchname>-v<version>`` to record the patch
>    series."
> 

Added this 👍.

>> +
>>   If you cannot send patch emails
>>   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>   
>> -- 
>> 2.39.5
>>
> 
> With regards,
> Daniel

[PATCH 3/7] docs/devel: add b4 for patch retrieval

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/devel/submitting-a-patch.rst | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
index 953682f20cb..d25c677d996 100644
--- a/docs/devel/submitting-a-patch.rst
+++ b/docs/devel/submitting-a-patch.rst
@@ -417,6 +417,16 @@ For more details on how QEMU's stable process works, refer to the
 
 .. _participating_in_code_review:
 
+Retrieve an existing series
+---------------------------
+
+If you want to apply an existing series on top of your tree, you can simply use
+`b4 <https://github.com/mricon/b4>`__.
+
+::
+
+    b4 shazam $msg-id
+
 Participating in Code Review
 ----------------------------
 
-- 
2.39.5

Re: [PATCH 3/7] docs/devel: add b4 for patch retrieval

[Daniel P. Berrangé]

On Mon, Nov 18, 2024 at 09:23:53AM -0800, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/devel/submitting-a-patch.rst | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/docs/devel/submitting-a-patch.rst b/docs/devel/submitting-a-patch.rst
> index 953682f20cb..d25c677d996 100644
> --- a/docs/devel/submitting-a-patch.rst
> +++ b/docs/devel/submitting-a-patch.rst
> @@ -417,6 +417,16 @@ For more details on how QEMU's stable process works, refer to the
>  
>  .. _participating_in_code_review:
>  
> +Retrieve an existing series
> +---------------------------
> +
> +If you want to apply an existing series on top of your tree, you can simply use
> +`b4 <https://github.com/mricon/b4>`__.
> +
> +::
> +
> +    b4 shazam $msg-id

I'm wondering how b4 knows where to find the mails for $msg-id  for QEMU ?


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/7] docs/devel: add b4 for patch retrieval

[Marcin Juszkiewicz]

W dniu 19.11.2024 o 10:14, Daniel P. Berrangé pisze:

>> +    b4 shazam $msg-id
> 
> I'm wondering how b4 knows where to find the mails for $msg-id  for QEMU ?

It goes for https://lore.kernel.org/qemu-devel/$msg-id by default or 
checks b4.midmask in git config.

Re: [PATCH 3/7] docs/devel: add b4 for patch retrieval

[Daniel P. Berrangé]

On Tue, Nov 19, 2024 at 10:56:36AM +0100, Marcin Juszkiewicz wrote:
> W dniu 19.11.2024 o 10:14, Daniel P. Berrangé pisze:
> 
> > > +    b4 shazam $msg-id
> > 
> > I'm wondering how b4 knows where to find the mails for $msg-id  for QEMU ?
> 
> It goes for https://lore.kernel.org/qemu-devel/$msg-id by default or checks
> b4.midmask in git config.

How does it figure out to look at  '/qemu-devel/' by default  ?

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 3/7] docs/devel: add b4 for patch retrieval

[Marcin Juszkiewicz]

W dniu 19.11.2024 o 11:07, Daniel P. Berrangé pisze:
> On Tue, Nov 19, 2024 at 10:56:36AM +0100, Marcin Juszkiewicz wrote:
>> W dniu 19.11.2024 o 10:14, Daniel P. Berrangé pisze:
>>
>>>> +    b4 shazam $msg-id
>>>
>>> I'm wondering how b4 knows where to find the mails for $msg-id  for QEMU ?
>>
>> It goes for https://lore.kernel.org/qemu-devel/$msg-id by default or checks
>> b4.midmask in git config.
> 
> How does it figure out to look at  '/qemu-devel/' by default  ?

Sorry, was wrong. It goes to https://lore.kernel.org/$msg-id as this is 
enough to get mail.

[PATCH 4/7] docs/devel: add information on how to setup build environments

[Pierrick Bouvier]

MacOS and Linux are straightforward, but Windows needs a bit more
details.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/about/build-platforms.rst   |   4 +-
 docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
 docs/devel/index-build.rst       |   1 +
 3 files changed, 118 insertions(+), 1 deletion(-)
 create mode 100644 docs/devel/build-environment.rst

diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
index 6102f00aec0..c1ea53db834 100644
--- a/docs/about/build-platforms.rst
+++ b/docs/about/build-platforms.rst
@@ -29,6 +29,9 @@ The `Repology`_ site is a useful resource to identify
 currently shipped versions of software in various operating systems,
 though it does not cover all distros listed below.
 
+You can find how to install build dependencies for different systems on the
+:ref:`setup-build-env` page.
+
 Supported host architectures
 ----------------------------
 
@@ -130,7 +133,6 @@ Optional build dependencies
   cross compilation using ``docker`` or ``podman``, or to use pre-built
   binaries distributed with QEMU.
 
-
 Windows
 -------
 
diff --git a/docs/devel/build-environment.rst b/docs/devel/build-environment.rst
new file mode 100644
index 00000000000..d9a66f5fcc6
--- /dev/null
+++ b/docs/devel/build-environment.rst
@@ -0,0 +1,114 @@
+
+.. _setup-build-env:
+
+Setup build environment
+=======================
+
+QEMU uses a lot of dependencies on the host system. glib2 is used everywhere in
+the code base, and most of the other dependencies are optional.
+
+We present here simple instructions to enable native builds on most popular
+systems.
+
+You can find additional instructions on `QEMU wiki <https://wiki.qemu.org/>`_:
+
+- `Linux <https://wiki.qemu.org/Hosts/Linux>`_
+- `MacOS <https://wiki.qemu.org/Hosts/Mac>`_
+- `Windows <https://wiki.qemu.org/Hosts/W32>`_
+- `BSD <https://wiki.qemu.org/Hosts/BSD>`_
+
+Linux
+-----
+
+Fedora
+++++++
+
+::
+
+    sudo dnf update && sudo dnf builddep qemu
+
+Debian/Ubuntu
++++++++++++++
+
+You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
+Then, use apt to install dependencies:
+
+::
+
+    sudo apt update && sudo apt build-dep qemu
+
+MacOS
+-----
+
+You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
+install dependencies:
+
+::
+
+    brew update && brew install $(brew deps --include-build qemu)
+
+Windows
+-------
+
+You first need to install `MSYS2 <https://www.msys2.org/>`_.
+MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
+x86_64 environments are based on GCC, while aarch64 is based on Clang.
+
+We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
+(only available on windows-aarch64 hosts).
+
+Then, you can open a windows shell, and enter msys2 env using:
+
+::
+
+    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
+    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
+
+MSYS2 package manager does not offer a built-in way to install build
+dependencies. You can start with this list of packages using pacman:
+
+Note: Dependencies need to be installed again if you use a different MSYS2
+environment.
+
+::
+
+    # update MSYS2 itself, you need to reopen your shell at the end.
+    pacman -Syu
+    pacman -S \
+        base-devel binutils bison diffutils flex git grep make sed \
+        ${MINGW_PACKAGE_PREFIX}-toolchain \
+        ${MINGW_PACKAGE_PREFIX}-glib2 \
+        ${MINGW_PACKAGE_PREFIX}-gtk3 \
+        ${MINGW_PACKAGE_PREFIX}-libnfs \
+        ${MINGW_PACKAGE_PREFIX}-libssh \
+        ${MINGW_PACKAGE_PREFIX}-ninja \
+        ${MINGW_PACKAGE_PREFIX}-pixman \
+        ${MINGW_PACKAGE_PREFIX}-pkgconf \
+        ${MINGW_PACKAGE_PREFIX}-python \
+        ${MINGW_PACKAGE_PREFIX}-SDL2 \
+        ${MINGW_PACKAGE_PREFIX}-zstd
+
+If you want to install all dependencies, it's possible to use recipe used to
+build QEMU in MSYS2 itself.
+
+::
+
+    pacman -S wget
+    wget https://raw.githubusercontent.com/msys2/MINGW-packages/refs/heads/master/mingw-w64-qemu/PKGBUILD
+    # Some packages may be missing for your environment, installation will still
+    # be done though.
+    makepkg -s PKGBUILD || true
+
+Build on windows-aarch64
+++++++++++++++++++++++++
+
+When trying to cross compile meson for x86_64 using UCRT64 or MINGW64 env,
+configure will run into an error because the cpu detected is not correct.
+
+Meson detects x86_64 processes emulated, so you need to manually set the cpu,
+and force a cross compilation (with empty prefix).
+
+::
+
+    ./configure --cpu=x86_64 --cross-prefix=
+
diff --git a/docs/devel/index-build.rst b/docs/devel/index-build.rst
index 0023953be36..0745c81a264 100644
--- a/docs/devel/index-build.rst
+++ b/docs/devel/index-build.rst
@@ -8,6 +8,7 @@ some of the basics if you are adding new files and targets to the build.
    :maxdepth: 3
 
    build-system
+   build-environment
    kconfig
    docs
    qapi-code-gen
-- 
2.39.5

Re: [PATCH 4/7] docs/devel: add information on how to setup build environments

[Daniel P. Berrangé]

On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
> MacOS and Linux are straightforward, but Windows needs a bit more
> details.
> 
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/about/build-platforms.rst   |   4 +-
>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>  docs/devel/index-build.rst       |   1 +
>  3 files changed, 118 insertions(+), 1 deletion(-)
>  create mode 100644 docs/devel/build-environment.rst
> 
> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
> index 6102f00aec0..c1ea53db834 100644
> --- a/docs/about/build-platforms.rst
> +++ b/docs/about/build-platforms.rst
> @@ -29,6 +29,9 @@ The `Repology`_ site is a useful resource to identify
>  currently shipped versions of software in various operating systems,
>  though it does not cover all distros listed below.
>  
> +You can find how to install build dependencies for different systems on the
> +:ref:`setup-build-env` page.
> +
>  Supported host architectures
>  ----------------------------
>  
> @@ -130,7 +133,6 @@ Optional build dependencies
>    cross compilation using ``docker`` or ``podman``, or to use pre-built
>    binaries distributed with QEMU.
>  
> -
>  Windows
>  -------
>  
> diff --git a/docs/devel/build-environment.rst b/docs/devel/build-environment.rst
> new file mode 100644
> index 00000000000..d9a66f5fcc6
> --- /dev/null
> +++ b/docs/devel/build-environment.rst
> @@ -0,0 +1,114 @@
> +
> +.. _setup-build-env:
> +
> +Setup build environment
> +=======================
> +
> +QEMU uses a lot of dependencies on the host system. glib2 is used everywhere in
> +the code base, and most of the other dependencies are optional.
> +
> +We present here simple instructions to enable native builds on most popular
> +systems.
> +
> +You can find additional instructions on `QEMU wiki <https://wiki.qemu.org/>`_:
> +
> +- `Linux <https://wiki.qemu.org/Hosts/Linux>`_
> +- `MacOS <https://wiki.qemu.org/Hosts/Mac>`_
> +- `Windows <https://wiki.qemu.org/Hosts/W32>`_
> +- `BSD <https://wiki.qemu.org/Hosts/BSD>`_


We generally suffer from having information spread over multiple sources,
giving us the burden of keeping the different places consistent, which we
pretty consistently fail at.

It is a good think to add build env docs to qemu.git where we actually
have oversight / review to catch mistakes. If we do this though, IMHO,
we should be deleting the wiki pages and making them into 302 redirects
to our new in-tree docs.

> +
> +Linux
> +-----
> +
> +Fedora
> +++++++
> +
> +::
> +
> +    sudo dnf update && sudo dnf builddep qemu
> +
> +Debian/Ubuntu
> ++++++++++++++
> +
> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
> +Then, use apt to install dependencies:
> +
> +::
> +
> +    sudo apt update && sudo apt build-dep qemu
> +
> +MacOS
> +-----
> +
> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
> +install dependencies:
> +
> +::
> +
> +    brew update && brew install $(brew deps --include-build qemu)


The downside in recommending the 'build dep' approach is that it misses
out on deps that have been newly introduced in qemu.git, since whatever
old version of QEMU the distros are shipping.  It also misses deps for
any features the distro vendor has decided to exclude.

Can we put a caveat describing this limitation at the top, so that users
have pointer if things don't quite go the way we expected.

> +
> +Windows
> +-------
> +
> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
> +
> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
> +(only available on windows-aarch64 hosts).

Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
block the use of CLang for Windows because of its lack of support for
'gcc_struct' attributes, so I would have expected it to fail

> +
> +Then, you can open a windows shell, and enter msys2 env using:
> +
> +::
> +
> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
> +
> +MSYS2 package manager does not offer a built-in way to install build
> +dependencies. You can start with this list of packages using pacman:
> +

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 4/7] docs/devel: add information on how to setup build environments

[Alex Bennée]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
>> MacOS and Linux are straightforward, but Windows needs a bit more
>> details.
>> 
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>  docs/about/build-platforms.rst   |   4 +-
>>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>>  docs/devel/index-build.rst       |   1 +
>>  3 files changed, 118 insertions(+), 1 deletion(-)
>>  create mode 100644 docs/devel/build-environment.rst
>> 
<snip>
>> +Fedora
>> +++++++
>> +
>> +::
>> +
>> +    sudo dnf update && sudo dnf builddep qemu
>> +
>> +Debian/Ubuntu
>> ++++++++++++++
>> +
>> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
>> +Then, use apt to install dependencies:
>> +
>> +::
>> +
>> +    sudo apt update && sudo apt build-dep qemu
>> +
>> +MacOS
>> +-----
>> +
>> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
>> +install dependencies:
>> +
>> +::
>> +
>> +    brew update && brew install $(brew deps --include-build qemu)
>
>
> The downside in recommending the 'build dep' approach is that it misses
> out on deps that have been newly introduced in qemu.git, since whatever
> old version of QEMU the distros are shipping.  It also misses deps for
> any features the distro vendor has decided to exclude.
>
> Can we put a caveat describing this limitation at the top, so that users
> have pointer if things don't quite go the way we expected.

At a recent QEMU workshop I presented I pointed to lcitool as a way to
list the minimal dependencies QEMU needs:

  ./tests/lcitool/libvirt-ci/bin/lcitool variables -f yaml debian-12 ./tests/lcitool/projects/qemu-minimal.yml

It wouldn't be the hardest thing to extend lcitool to generate a pkg
install line instead of a list. Although I notice it fails with the full
set of dependancies:

  ✗  ./tests/lcitool/libvirt-ci/bin/lcitool -d tests/lcitool variables -f shell debian-12 ./tests/lcitool/projects/qemu.yml
  Package generic name resolution error: Package libcbor not present in mappings

I note there is already an "install" action but I'm wary of what it does.

>
>> +
>> +Windows
>> +-------
>> +
>> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
>> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
>> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
>> +
>> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
>> +(only available on windows-aarch64 hosts).
>
> Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
> block the use of CLang for Windows because of its lack of support for
> 'gcc_struct' attributes, so I would have expected it to fail
>
>> +
>> +Then, you can open a windows shell, and enter msys2 env using:
>> +
>> +::
>> +
>> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
>> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
>> +
>> +MSYS2 package manager does not offer a built-in way to install build
>> +dependencies. You can start with this list of packages using pacman:
>> +
>
> With regards,
> Daniel

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 4/7] docs/devel: add information on how to setup build environments

[Daniel P. Berrangé]

On Tue, Nov 19, 2024 at 11:08:12AM +0000, Alex Bennée wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
> >> MacOS and Linux are straightforward, but Windows needs a bit more
> >> details.
> >> 
> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> >> ---
> >>  docs/about/build-platforms.rst   |   4 +-
> >>  docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
> >>  docs/devel/index-build.rst       |   1 +
> >>  3 files changed, 118 insertions(+), 1 deletion(-)
> >>  create mode 100644 docs/devel/build-environment.rst
> >> 
> <snip>
> >> +Fedora
> >> +++++++
> >> +
> >> +::
> >> +
> >> +    sudo dnf update && sudo dnf builddep qemu
> >> +
> >> +Debian/Ubuntu
> >> ++++++++++++++
> >> +
> >> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
> >> +Then, use apt to install dependencies:
> >> +
> >> +::
> >> +
> >> +    sudo apt update && sudo apt build-dep qemu
> >> +
> >> +MacOS
> >> +-----
> >> +
> >> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
> >> +install dependencies:
> >> +
> >> +::
> >> +
> >> +    brew update && brew install $(brew deps --include-build qemu)
> >
> >
> > The downside in recommending the 'build dep' approach is that it misses
> > out on deps that have been newly introduced in qemu.git, since whatever
> > old version of QEMU the distros are shipping.  It also misses deps for
> > any features the distro vendor has decided to exclude.
> >
> > Can we put a caveat describing this limitation at the top, so that users
> > have pointer if things don't quite go the way we expected.
> 
> At a recent QEMU workshop I presented I pointed to lcitool as a way to
> list the minimal dependencies QEMU needs:
> 
>   ./tests/lcitool/libvirt-ci/bin/lcitool variables -f yaml debian-12 ./tests/lcitool/projects/qemu-minimal.yml
> 
> It wouldn't be the hardest thing to extend lcitool to generate a pkg
> install line instead of a list. Although I notice it fails with the full
> set of dependancies:
> 
>   ✗  ./tests/lcitool/libvirt-ci/bin/lcitool -d tests/lcitool variables -f shell debian-12 ./tests/lcitool/projects/qemu.yml
>   Package generic name resolution error: Package libcbor not present in mappings

I think your libvirt-ci git submodule might be out of date. The current
generated Dockerfiles include libcbor and its in the mappings.

> I note there is already an "install" action but I'm wary of what it does.

That's related to lcitool functionality for provisioning VMs - kinda like
QEMUs tests/vm/ stuff.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 4/7] docs/devel: add information on how to setup build environments

[Pierrick Bouvier]

On 11/19/24 01:24, Daniel P. Berrangé wrote:
> On Mon, Nov 18, 2024 at 09:23:54AM -0800, Pierrick Bouvier wrote:
>> MacOS and Linux are straightforward, but Windows needs a bit more
>> details.
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   docs/about/build-platforms.rst   |   4 +-
>>   docs/devel/build-environment.rst | 114 +++++++++++++++++++++++++++++++
>>   docs/devel/index-build.rst       |   1 +
>>   3 files changed, 118 insertions(+), 1 deletion(-)
>>   create mode 100644 docs/devel/build-environment.rst
>>
>> diff --git a/docs/about/build-platforms.rst b/docs/about/build-platforms.rst
>> index 6102f00aec0..c1ea53db834 100644
>> --- a/docs/about/build-platforms.rst
>> +++ b/docs/about/build-platforms.rst
>> @@ -29,6 +29,9 @@ The `Repology`_ site is a useful resource to identify
>>   currently shipped versions of software in various operating systems,
>>   though it does not cover all distros listed below.
>>   
>> +You can find how to install build dependencies for different systems on the
>> +:ref:`setup-build-env` page.
>> +
>>   Supported host architectures
>>   ----------------------------
>>   
>> @@ -130,7 +133,6 @@ Optional build dependencies
>>     cross compilation using ``docker`` or ``podman``, or to use pre-built
>>     binaries distributed with QEMU.
>>   
>> -
>>   Windows
>>   -------
>>   
>> diff --git a/docs/devel/build-environment.rst b/docs/devel/build-environment.rst
>> new file mode 100644
>> index 00000000000..d9a66f5fcc6
>> --- /dev/null
>> +++ b/docs/devel/build-environment.rst
>> @@ -0,0 +1,114 @@
>> +
>> +.. _setup-build-env:
>> +
>> +Setup build environment
>> +=======================
>> +
>> +QEMU uses a lot of dependencies on the host system. glib2 is used everywhere in
>> +the code base, and most of the other dependencies are optional.
>> +
>> +We present here simple instructions to enable native builds on most popular
>> +systems.
>> +
>> +You can find additional instructions on `QEMU wiki <https://wiki.qemu.org/>`_:
>> +
>> +- `Linux <https://wiki.qemu.org/Hosts/Linux>`_
>> +- `MacOS <https://wiki.qemu.org/Hosts/Mac>`_
>> +- `Windows <https://wiki.qemu.org/Hosts/W32>`_
>> +- `BSD <https://wiki.qemu.org/Hosts/BSD>`_
> 
> 
> We generally suffer from having information spread over multiple sources,
> giving us the burden of keeping the different places consistent, which we
> pretty consistently fail at.
> 

Agree on that.

> It is a good think to add build env docs to qemu.git where we actually
> have oversight / review to catch mistakes. If we do this though, IMHO,
> we should be deleting the wiki pages and making them into 302 redirects
> to our new in-tree docs.
> 

Sure.
However, there is still some useful information there, that are not 
reflected in this patch.

We could migrate the information later, but for now, I can't spend many 
cycles to try to build all configurations described on the wiki to make 
sure it's up to date.
So I just included information I'm sure it works, and that cover the 
basic needs to build on all major systems.

Hope this is a starting point we can maybe accept, and we can enhance 
the documentation later. But asking to do all this at once is a big 
task, and I feel it's a lot to ask for it as part of the current series.

Would you accept that, or prefer to not include anything before we have 
purged the wikis completely?

>> +
>> +Linux
>> +-----
>> +
>> +Fedora
>> +++++++
>> +
>> +::
>> +
>> +    sudo dnf update && sudo dnf builddep qemu
>> +
>> +Debian/Ubuntu
>> ++++++++++++++
>> +
>> +You first need to enable `Sources List <https://wiki.debian.org/SourcesList>`_.
>> +Then, use apt to install dependencies:
>> +
>> +::
>> +
>> +    sudo apt update && sudo apt build-dep qemu
>> +
>> +MacOS
>> +-----
>> +
>> +You first need to install `Homebrew <https://brew.sh/>`_. Then, use it to
>> +install dependencies:
>> +
>> +::
>> +
>> +    brew update && brew install $(brew deps --include-build qemu)
> 
> 
> The downside in recommending the 'build dep' approach is that it misses
> out on deps that have been newly introduced in qemu.git, since whatever
> old version of QEMU the distros are shipping.  It also misses deps for
> any features the distro vendor has decided to exclude.
> 
> Can we put a caveat describing this limitation at the top, so that users
> have pointer if things don't quite go the way we expected.
>

I can add a note about it. It's especially true for Linux, as on Windows 
(MSYS2) and MacOS (homebrew), they usually track latest release of QEMU, 
so the build list should match, given a reduced time frame after the 
release.

>> +
>> +Windows
>> +-------
>> +
>> +You first need to install `MSYS2 <https://www.msys2.org/>`_.
>> +MSYS2 offers `different environments <https://www.msys2.org/docs/environments/>`_.
>> +x86_64 environments are based on GCC, while aarch64 is based on Clang.
>> +
>> +We recommend to use MINGW64 for windows-x86_64 and CLANGARM64 for windows-aarch64
>> +(only available on windows-aarch64 hosts).
> 
> Does CLANGARM64 really work with QEMU ?   We go out of our way to actively
> block the use of CLang for Windows because of its lack of support for
> 'gcc_struct' attributes, so I would have expected it to fail
> 

I can drop the windows-arm64 part.
For the support, it's a discussion we add previously on the concerned 
series. In current master, it won't even go past the configure step 
because of intentional restriction we have in meson.build.
My initial hope was to remove it, but I understand that we prefer to 
wait for gcc_struct support in clang. So be it, I can live with a local 
fix for that meanwhile :).

>> +
>> +Then, you can open a windows shell, and enter msys2 env using:
>> +
>> +::
>> +
>> +    c:/msys64/msys2_shell.cmd -defterm -here -no-start -mingw64
>> +    # Replace -ucrt64 by -clangarm64 or -ucrt64 for other environments.
>> +
>> +MSYS2 package manager does not offer a built-in way to install build
>> +dependencies. You can start with this list of packages using pacman:
>> +
> 
> With regards,
> Daniel

[PATCH 5/7] docs: add a codebase section

[Pierrick Bouvier]

Present the various parts of QEMU and organization of codebase.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/about/emulation.rst               |   2 +
 docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
 docs/devel/decodetree.rst              |   2 +
 docs/devel/ebpf_rss.rst                |   2 +
 docs/devel/index-internals.rst         |   2 +
 docs/devel/migration/main.rst          |   2 +
 docs/devel/qapi-code-gen.rst           |   1 +
 docs/devel/testing/main.rst            |   9 +-
 docs/devel/testing/qtest.rst           |   2 +
 docs/index.rst                         |   3 +
 docs/interop/qemu-ga.rst               |   2 +
 docs/system/qemu-block-drivers.rst.inc |   2 +
 docs/tools/qemu-storage-daemon.rst     |   2 +
 docs/user/main.rst                     |   6 +
 14 files changed, 247 insertions(+), 1 deletion(-)
 create mode 100644 docs/codebase/index.rst

diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
index 3028d5fff7a..3bc3579434f 100644
--- a/docs/about/emulation.rst
+++ b/docs/about/emulation.rst
@@ -176,6 +176,8 @@ for that architecture.
     - System
     - Tensilica ISS SIMCALL
 
+.. _tcg-plugins:
+
 TCG Plugins
 -----------
 
diff --git a/docs/codebase/index.rst b/docs/codebase/index.rst
new file mode 100644
index 00000000000..353830ef175
--- /dev/null
+++ b/docs/codebase/index.rst
@@ -0,0 +1,211 @@
+========
+Codebase
+========
+
+This section present the various parts of QEMU and how codebase is organized.
+
+Beyond giving succint descriptions, the goal is to offer links to various
+parts of the documentation/codebase.
+
+Subsystems
+----------
+
+An exhaustive list of subsystems and files associated can be found in
+`MAINTAINERS <https://gitlab.com/qemu-project/qemu/-/blob/master/MAINTAINERS>`_
+file.
+
+Some of the main QEMU subsystems are:
+
+- `Accelerators<Accelerators>`
+- Block devices and `disk images<disk images>` support
+- `CI<ci>` and `Tests<testing>`
+- `Devices<device-emulation>` & Board models
+- `Documentation <documentation-root>`
+- `GDB support<GDB usage>`
+- `Migration<migration>`
+- `Monitor<QEMU monitor>`
+- `QOM (QEMU Object Model)<qom>`
+- `System mode<System emulation>`
+- `TCG (Tiny Code Generator)<tcg>`
+- `User mode<user-mode>` (`Linux<linux-user-mode>` & `BSD<bsd-user-mode>`)
+- User Interfaces
+
+More documentation on QEMU subsystems can be found on :ref:`internal-subsystem`
+page.
+
+The Grand tour
+--------------
+
+We present briefly here what every folder of the codebase contains. Hop on!
+
+* `accel <https://gitlab.com/qemu-project/qemu/-/tree/master/accel>`_:
+  Infrastructure and architecture agnostic code related to the various
+  `accelerators <Accelerators>` supported by QEMU
+  (TCG, KVM, hvf, whpx, xen, nvmm).
+  Contains interfaces for operations that will be implemented per
+  `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_.
+* `audio <https://gitlab.com/qemu-project/qemu/-/tree/master/audio>`_:
+  Audio (host) support.
+* `authz <https://gitlab.com/qemu-project/qemu/-/tree/master/authz>`_:
+  `QEMU Authorization framework<client authorization>`.
+* `backends <https://gitlab.com/qemu-project/qemu/-/tree/master/backends>`_:
+  Various backends used for device emulation.
+* `block <https://gitlab.com/qemu-project/qemu/-/tree/master/block>`_:
+  Block devices and `image formats<disk images>` implementation.
+* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
+  `BSD User mode<bsd-user-mode>`.
+* build: Where the code built goes!
+* `chardev <https://gitlab.com/qemu-project/qemu/-/tree/master/chardev>`_:
+  Various backends used by char devices.
+* `common-user <https://gitlab.com/qemu-project/qemu/-/tree/master/common-user>`_:
+  User-mode assembly code for dealing with signals occuring during syscalls.
+* `configs <https://gitlab.com/qemu-project/qemu/-/tree/master/configs>`_:
+  Makefiles defining configurations to build QEMU.
+* `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib>`_:
+  Community contributed devices/plugins/tools.
+* `crypto <https://gitlab.com/qemu-project/qemu/-/tree/master/crypto>`_:
+  Cryptographic algorithms used in QEMU.
+* `disas <https://gitlab.com/qemu-project/qemu/-/tree/master/disas>`_:
+  Disassembly functions used by QEMU target code.
+* `docs <https://gitlab.com/qemu-project/qemu/-/tree/master/docs>`_:
+  QEMU Documentation.
+* `dump <https://gitlab.com/qemu-project/qemu/-/tree/master/dump>`_:
+  Code to dump memory of a running VM.
+* `ebpf <https://gitlab.com/qemu-project/qemu/-/tree/master/ebpf>`_:
+  eBPF program support in QEMU. `virtio-net RSS<ebpf-rss>` uses it.
+* `fpu <https://gitlab.com/qemu-project/qemu/-/tree/master/fpu>`_:
+  Floating-point software emulation.
+* `fsdev <https://gitlab.com/qemu-project/qemu/-/tree/master/fsdev>`_:
+  `VirtFS <https://www.linux-kvm.org/page/VirtFS>`_ support.
+* `gdbstub <https://gitlab.com/qemu-project/qemu/-/tree/master/gdbstub>`_:
+  `GDB <GDB usage>` support.
+* `gdb-xml <https://gitlab.com/qemu-project/qemu/-/tree/master/gdb-xml>`_:
+  Set of XML files describing architectures and used by `gdbstub <GDB usage>`.
+* `host <https://gitlab.com/qemu-project/qemu/-/tree/master/host>`_:
+  Various architecture specific header files (crypto, atomic, memory
+  operations).
+* `linux-headers <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-headers>`_:
+  A subset of headers imported from Linux kernel and used for implementing
+  KVM support and user-mode.
+* `linux-user <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-user>`_:
+  `User mode <user-mode>` implementation. Contains one folder per target
+  architecture.
+* `.gitlab-ci.d <https://gitlab.com/qemu-project/qemu/-/tree/master/.gitlab-ci.d>`_:
+  `CI <ci>` yaml and scripts.
+* `include <https://gitlab.com/qemu-project/qemu/-/tree/master/include>`_:
+  All headers associated to different subsystems in QEMU. Hierachy used mirrors
+  source code organization and naming.
+* `hw <https://gitlab.com/qemu-project/qemu/-/tree/master/hw>`_:
+  `Devices <device-emulation>` and boards emulation. Devices are categorized by
+  type/protocol/architecture and located in associated subfolder.
+* `io <https://gitlab.com/qemu-project/qemu/-/tree/master/io>`_:
+  QEMU `I/O channels <https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04208.html>`_.
+* `libdecnumber <https://gitlab.com/qemu-project/qemu/-/tree/master/libdecnumber>`_:
+  Import of gcc library, used to implement decimal number arithmetic.
+* `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/migration>`__:
+  `Migration framework <migration>`.
+* `monitor <https://gitlab.com/qemu-project/qemu/-/tree/master/monitor>`_:
+  `Monitor <QEMU monitor>` implementation (HMP & QMP).
+* `nbd <https://gitlab.com/qemu-project/qemu/-/tree/master/nbd>`_:
+  QEMU `NBD (Network Block Device) <nbd>` server.
+* `net <https://gitlab.com/qemu-project/qemu/-/tree/master/net>`_:
+  Network (host) support.
+* `pc-bios <https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios>`_:
+  Contains pre-built firmware binaries and boot images, ready to use in
+  QEMU without compilation.
+* `plugins <https://gitlab.com/qemu-project/qemu/-/tree/master/plugins>`_:
+  `TCG plugins <tcg-plugins>` core implementation. Plugins can be found in
+  `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg/plugins>`__
+  and `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/plugins>`__
+  folders.
+* `po <https://gitlab.com/qemu-project/qemu/-/tree/master/po>`_:
+  Translation files.
+* `python <https://gitlab.com/qemu-project/qemu/-/tree/master/python>`_:
+  Python part of our build/test system.
+* `qapi <https://gitlab.com/qemu-project/qemu/-/tree/master/qapi>`_:
+  `QAPI <qapi>` implementation.
+* `qobject <https://gitlab.com/qemu-project/qemu/-/tree/master/qobject>`_:
+  QEMU Object implementation.
+* `qga <https://gitlab.com/qemu-project/qemu/-/tree/master/qga>`_:
+  QEMU `Guest agent <qemu-ga>` implementation.
+* `qom <https://gitlab.com/qemu-project/qemu/-/tree/master/qom>`_:
+  QEMU `Object model <qom>` implementation, with monitor associated commands.
+* `replay <https://gitlab.com/qemu-project/qemu/-/tree/master/replay>`_:
+  QEMU `record/replay <replay>` implementation.
+* `roms <https://gitlab.com/qemu-project/qemu/-/tree/master/roms>`_:
+  Contains source code for various firmware and ROMs, which can be compiled if
+  custom or updated versions are needed.
+* `rust <https://gitlab.com/qemu-project/qemu/-/tree/master/rust>`_:
+  Rust integration in QEMU. It contains the new interfaces defined and
+  associated devices using it.
+* `scripts <https://gitlab.com/qemu-project/qemu/-/tree/master/scripts>`_:
+  Collection of scripts used in build and test systems, and various
+  tools for QEMU codebase and execution traces.
+* `scsi <https://gitlab.com/qemu-project/qemu/-/tree/master/scsi>`_:
+  Code related to SCSI support, used by SCSI devices.
+* `semihosting <https://gitlab.com/qemu-project/qemu/-/tree/master/semihosting>`_:
+  QEMU `Semihosting <Semihosting>` implementation.
+* `stats <https://gitlab.com/qemu-project/qemu/-/tree/master/stats>`_:
+  `Monitor <QEMU monitor>` stats commands implementation.
+* `storage-daemon <https://gitlab.com/qemu-project/qemu/-/tree/master/storage-daemon>`_:
+  QEMU `Storage daemon <storage-daemon>` implementation.
+* `stubs <https://gitlab.com/qemu-project/qemu/-/tree/master/stubs>`_:
+  Various stubs (empty functions) used to compile QEMU with specific
+  configurations.
+* `subprojects <https://gitlab.com/qemu-project/qemu/-/tree/master/subprojects>`_:
+  QEMU submodules used by QEMU build system.
+* `system <https://gitlab.com/qemu-project/qemu/-/tree/master/system>`_:
+  QEMU `system mode <System emulation>` implementation (cpu, mmu, boot support).
+* `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_:
+  Contains code for all target architectures supported (one subfolder
+  per arch). For every architecture, you can find accelerator specific
+  implementations.
+* `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tcg>`_:
+  `TCG <tcg>` related code.
+  Contains one subfolder per host supported architecture.
+* `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_:
+  QEMU `test <testing>` suite
+
+  - `avocado <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/avocado>`_:
+    Functional tests booting full VM using `Avocado framework <checkavocado-ref>`.
+    Those tests will be transformed and moved into
+    `tests/functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_
+    in the future.
+  - `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_:
+    Data for various tests.
+  - `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_:
+    Testsuite for `decodetree <decodetree>` implementation.
+  - `docker <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/docker>`_:
+    Code and scripts to create `containers <container-ref>` used in `CI <ci>`.
+  - `fp <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/fp>`_:
+    QEMU testsuite for soft float implementation.
+  - `functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_:
+    `Functional tests <checkfunctional-ref>` (full VM boot).
+  - `lcitool <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/lcitool>`_:
+    Generate dockerfiles for CI containers.
+  - `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/migration>`_:
+    Test scripts and data for `Migration framework <migration>`.
+  - `multiboot <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/multiboot>`_:
+    Test multiboot functionality for x86_64/i386.
+  - `qapi-schema <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qapi-schema>`_:
+    Test scripts and data for `QAPI <qapi-tests>`.
+  - `qemu-iotests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qemu-iotests>`_:
+    `Disk image and block tests <qemu-iotests>`.
+  - `qtest <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qtest>`_:
+    `Device emulation testing <qtest>`.
+  - `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg>`__:
+    `TCG related tests <checktcg-ref>`. Contains code per architecture
+    (subfolder) and multiarch tests as well.
+  - `tsan <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tsan>`_:
+    `Suppressions <tsan-suppressions>` for thread sanitizer.
+  - `uefi-test-tools <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/uefi-test-tools>`_:
+    Test tool for UEFI support.
+  - `unit <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/unit>`_:
+    QEMU `Unit tests <unit-tests>`.
+* `trace <https://gitlab.com/qemu-project/qemu/-/tree/master/trace>`_:
+  `Tracing framework <tracing>`. Used to print information associated to various
+  events during execution.
+* `ui <https://gitlab.com/qemu-project/qemu/-/tree/master/ui>`_:
+  QEMU User interfaces.
+* `util <https://gitlab.com/qemu-project/qemu/-/tree/master/util>`_:
+  Utility code used by other parts of QEMU.
diff --git a/docs/devel/decodetree.rst b/docs/devel/decodetree.rst
index e3392aa7057..98ad33a4870 100644
--- a/docs/devel/decodetree.rst
+++ b/docs/devel/decodetree.rst
@@ -1,3 +1,5 @@
+.. _decodetree:
+
 ========================
 Decodetree Specification
 ========================
diff --git a/docs/devel/ebpf_rss.rst b/docs/devel/ebpf_rss.rst
index 4a68682b31a..ed5d33767bd 100644
--- a/docs/devel/ebpf_rss.rst
+++ b/docs/devel/ebpf_rss.rst
@@ -1,3 +1,5 @@
+.. _ebpf-rss:
+
 ===========================
 eBPF RSS virtio-net support
 ===========================
diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst
index ab9fbc44826..bca597c6589 100644
--- a/docs/devel/index-internals.rst
+++ b/docs/devel/index-internals.rst
@@ -1,3 +1,5 @@
+.. _internal-subsystem:
+
 Internal Subsystem Information
 ------------------------------
 
diff --git a/docs/devel/migration/main.rst b/docs/devel/migration/main.rst
index c2857fc2446..cdd4f4a6d7e 100644
--- a/docs/devel/migration/main.rst
+++ b/docs/devel/migration/main.rst
@@ -1,3 +1,5 @@
+.. _migration:
+
 ===================
 Migration framework
 ===================
diff --git a/docs/devel/qapi-code-gen.rst b/docs/devel/qapi-code-gen.rst
index 583207a8ec2..3e26d2d1047 100644
--- a/docs/devel/qapi-code-gen.rst
+++ b/docs/devel/qapi-code-gen.rst
@@ -9,6 +9,7 @@ How to use the QAPI code generator
    This work is licensed under the terms of the GNU GPL, version 2 or
    later.  See the COPYING file in the top-level directory.
 
+.. _qapi:
 
 Introduction
 ============
diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
index 91f4dc61fb5..9869bcf0341 100644
--- a/docs/devel/testing/main.rst
+++ b/docs/devel/testing/main.rst
@@ -39,6 +39,8 @@ Before running tests, it is best to build QEMU programs first. Some tests
 expect the executables to exist and will fail with obscure messages if they
 cannot find them.
 
+.. _unit-tests:
+
 Unit tests
 ~~~~~~~~~~
 
@@ -126,6 +128,8 @@ successfully on various hosts. The following list shows some best practices:
   #ifdef in the codes. If the whole test suite cannot run on Windows, disable
   the build in the meson.build file.
 
+.. _qapi-tests:
+
 QAPI schema tests
 ~~~~~~~~~~~~~~~~~
 
@@ -160,6 +164,8 @@ check-block
 are in the "auto" group).
 See the "QEMU iotests" section below for more information.
 
+.. _qemu-iotests:
+
 QEMU iotests
 ------------
 
@@ -679,6 +685,8 @@ The above exitcode=0 has TSan continue without error if any warnings are found.
 This allows for running the test and then checking the warnings afterwards.
 If you want TSan to stop and exit with error on warnings, use exitcode=66.
 
+.. _tsan-suppressions:
+
 TSan Suppressions
 ~~~~~~~~~~~~~~~~~
 Keep in mind that for any data race warning, although there might be a data race
@@ -901,7 +909,6 @@ You can run the avocado tests simply by executing:
 
 See :ref:`checkavocado-ref` for more details.
 
-
 .. _checktcg-ref:
 
 Testing with "make check-tcg"
diff --git a/docs/devel/testing/qtest.rst b/docs/devel/testing/qtest.rst
index c5b8546b3eb..73ef7702b7b 100644
--- a/docs/devel/testing/qtest.rst
+++ b/docs/devel/testing/qtest.rst
@@ -1,3 +1,5 @@
+.. _qtest:
+
 ========================================
 QTest Device Emulation Testing Framework
 ========================================
diff --git a/docs/index.rst b/docs/index.rst
index 0b9ee9901d9..cb5e5098b65 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -3,6 +3,8 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
+.. _documentation-root:
+
 ================================
 Welcome to QEMU's documentation!
 ================================
@@ -18,3 +20,4 @@ Welcome to QEMU's documentation!
    interop/index
    specs/index
    devel/index
+   codebase/index
diff --git a/docs/interop/qemu-ga.rst b/docs/interop/qemu-ga.rst
index 11f7bae4600..d16cc1b9f07 100644
--- a/docs/interop/qemu-ga.rst
+++ b/docs/interop/qemu-ga.rst
@@ -1,3 +1,5 @@
+.. _qemu-ga:
+
 QEMU Guest Agent
 ================
 
diff --git a/docs/system/qemu-block-drivers.rst.inc b/docs/system/qemu-block-drivers.rst.inc
index 384e95ba765..cfe1acb78ae 100644
--- a/docs/system/qemu-block-drivers.rst.inc
+++ b/docs/system/qemu-block-drivers.rst.inc
@@ -500,6 +500,8 @@ What you should *never* do:
 - expect it to work when loadvm'ing
 - write to the FAT directory on the host system while accessing it with the guest system
 
+.. _nbd:
+
 NBD access
 ~~~~~~~~~~
 
diff --git a/docs/tools/qemu-storage-daemon.rst b/docs/tools/qemu-storage-daemon.rst
index ea00149a63a..35ab2d78074 100644
--- a/docs/tools/qemu-storage-daemon.rst
+++ b/docs/tools/qemu-storage-daemon.rst
@@ -1,3 +1,5 @@
+.. _storage-daemon:
+
 ===================
 QEMU Storage Daemon
 ===================
diff --git a/docs/user/main.rst b/docs/user/main.rst
index 7a126ee8093..80a77f0a0c9 100644
--- a/docs/user/main.rst
+++ b/docs/user/main.rst
@@ -1,3 +1,5 @@
+.. _user-mode:
+
 QEMU User space emulator
 ========================
 
@@ -42,6 +44,8 @@ QEMU was conceived so that ultimately it can emulate itself. Although it
 is not very useful, it is an important test to show the power of the
 emulator.
 
+.. _linux-user-mode:
+
 Linux User space emulator
 -------------------------
 
@@ -175,6 +179,8 @@ Other binaries
    * ``qemu-sparc64`` can execute some Sparc64 (Sparc64 CPU, 64 bit ABI) and
      SPARC32PLUS binaries (Sparc64 CPU, 32 bit ABI).
 
+.. _bsd-user-mode:
+
 BSD User space emulator
 -----------------------
 
-- 
2.39.5

Re: [PATCH 5/7] docs: add a codebase section

[Peter Maydell]

On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
<pierrick.bouvier@linaro.org> wrote:
>
> Present the various parts of QEMU and organization of codebase.
>
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>

I like this; it's something I've thought for a while would
be good to have, but which I never got round to trying to
put together. Thanks for doing this!

Mostly my comments below are spelling/typo nits and
other minor stuff.

> ---
>  docs/about/emulation.rst               |   2 +
>  docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
>  docs/devel/decodetree.rst              |   2 +
>  docs/devel/ebpf_rss.rst                |   2 +
>  docs/devel/index-internals.rst         |   2 +
>  docs/devel/migration/main.rst          |   2 +
>  docs/devel/qapi-code-gen.rst           |   1 +
>  docs/devel/testing/main.rst            |   9 +-
>  docs/devel/testing/qtest.rst           |   2 +
>  docs/index.rst                         |   3 +
>  docs/interop/qemu-ga.rst               |   2 +
>  docs/system/qemu-block-drivers.rst.inc |   2 +
>  docs/tools/qemu-storage-daemon.rst     |   2 +
>  docs/user/main.rst                     |   6 +
>  14 files changed, 247 insertions(+), 1 deletion(-)
>  create mode 100644 docs/codebase/index.rst
>
> diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
> index 3028d5fff7a..3bc3579434f 100644
> --- a/docs/about/emulation.rst
> +++ b/docs/about/emulation.rst
> @@ -176,6 +176,8 @@ for that architecture.
>      - System
>      - Tensilica ISS SIMCALL
>
> +.. _tcg-plugins:
> +
>  TCG Plugins
>  -----------
>
> diff --git a/docs/codebase/index.rst b/docs/codebase/index.rst
> new file mode 100644
> index 00000000000..353830ef175
> --- /dev/null
> +++ b/docs/codebase/index.rst

This is developer documentation, so I think it should
be in docs/devel/.

> @@ -0,0 +1,211 @@
> +========
> +Codebase
> +========
> +
> +This section present the various parts of QEMU and how codebase is organized.

"presents"; "how the codebase"

> +
> +Beyond giving succint descriptions, the goal is to offer links to various

"succinct"

> +parts of the documentation/codebase.
> +
> +Subsystems
> +----------
> +
> +An exhaustive list of subsystems and files associated can be found in

"associated files"

"the MAINTAINERS file"

> +`MAINTAINERS <https://gitlab.com/qemu-project/qemu/-/blob/master/MAINTAINERS>`_
> +file.
> +
> +Some of the main QEMU subsystems are:
> +
> +- `Accelerators<Accelerators>`
> +- Block devices and `disk images<disk images>` support
> +- `CI<ci>` and `Tests<testing>`
> +- `Devices<device-emulation>` & Board models
> +- `Documentation <documentation-root>`
> +- `GDB support<GDB usage>`
> +- `Migration<migration>`
> +- `Monitor<QEMU monitor>`
> +- `QOM (QEMU Object Model)<qom>`
> +- `System mode<System emulation>`
> +- `TCG (Tiny Code Generator)<tcg>`
> +- `User mode<user-mode>` (`Linux<linux-user-mode>` & `BSD<bsd-user-mode>`)
> +- User Interfaces
> +
> +More documentation on QEMU subsystems can be found on :ref:`internal-subsystem`
> +page.
> +
> +The Grand tour
> +--------------
> +
> +We present briefly here what every folder of the codebase contains. Hop on!


I think it would be helpful to mention at the top of this list something
like:

 The folder name links here will take you to that folder in our
 gitlab repository; other links will take you to more detailed
 documentation for that subsystem, where we have it. Unfortunately
 not every subsystem has documentation yet, so sometimes the
 source code is all you have.

just so readers know that if e.g. they have a local source tree
already there's no point in clicking on the 'crypto' link etc,
but that other links are going to go somewhere with more
human-written detail.

> +
> +* `accel <https://gitlab.com/qemu-project/qemu/-/tree/master/accel>`_:
> +  Infrastructure and architecture agnostic code related to the various
> +  `accelerators <Accelerators>` supported by QEMU
> +  (TCG, KVM, hvf, whpx, xen, nvmm).
> +  Contains interfaces for operations that will be implemented per
> +  `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_.
> +* `audio <https://gitlab.com/qemu-project/qemu/-/tree/master/audio>`_:
> +  Audio (host) support.
> +* `authz <https://gitlab.com/qemu-project/qemu/-/tree/master/authz>`_:
> +  `QEMU Authorization framework<client authorization>`.
> +* `backends <https://gitlab.com/qemu-project/qemu/-/tree/master/backends>`_:
> +  Various backends used for device emulation.
> +* `block <https://gitlab.com/qemu-project/qemu/-/tree/master/block>`_:
> +  Block devices and `image formats<disk images>` implementation.
> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
> +  `BSD User mode<bsd-user-mode>`.
> +* build: Where the code built goes!

The built code doesn't have to be in 'build'. We could say:

 * build: You can tell the QEMU build system to put the built code
   anywhere you like. By default it will go into a directory named
   ``build``. Sometimes documentation will assume this default
   for convenience when describing command lines; you can always
   replace it with the path to your build tree.

?

> +* `chardev <https://gitlab.com/qemu-project/qemu/-/tree/master/chardev>`_:
> +  Various backends used by char devices.
> +* `common-user <https://gitlab.com/qemu-project/qemu/-/tree/master/common-user>`_:
> +  User-mode assembly code for dealing with signals occuring during syscalls.
> +* `configs <https://gitlab.com/qemu-project/qemu/-/tree/master/configs>`_:
> +  Makefiles defining configurations to build QEMU.
> +* `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib>`_:
> +  Community contributed devices/plugins/tools.
> +* `crypto <https://gitlab.com/qemu-project/qemu/-/tree/master/crypto>`_:
> +  Cryptographic algorithms used in QEMU.
> +* `disas <https://gitlab.com/qemu-project/qemu/-/tree/master/disas>`_:
> +  Disassembly functions used by QEMU target code.
> +* `docs <https://gitlab.com/qemu-project/qemu/-/tree/master/docs>`_:
> +  QEMU Documentation.
> +* `dump <https://gitlab.com/qemu-project/qemu/-/tree/master/dump>`_:
> +  Code to dump memory of a running VM.
> +* `ebpf <https://gitlab.com/qemu-project/qemu/-/tree/master/ebpf>`_:
> +  eBPF program support in QEMU. `virtio-net RSS<ebpf-rss>` uses it.
> +* `fpu <https://gitlab.com/qemu-project/qemu/-/tree/master/fpu>`_:
> +  Floating-point software emulation.
> +* `fsdev <https://gitlab.com/qemu-project/qemu/-/tree/master/fsdev>`_:
> +  `VirtFS <https://www.linux-kvm.org/page/VirtFS>`_ support.
> +* `gdbstub <https://gitlab.com/qemu-project/qemu/-/tree/master/gdbstub>`_:
> +  `GDB <GDB usage>` support.
> +* `gdb-xml <https://gitlab.com/qemu-project/qemu/-/tree/master/gdb-xml>`_:
> +  Set of XML files describing architectures and used by `gdbstub <GDB usage>`.
> +* `host <https://gitlab.com/qemu-project/qemu/-/tree/master/host>`_:
> +  Various architecture specific header files (crypto, atomic, memory
> +  operations).
> +* `linux-headers <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-headers>`_:
> +  A subset of headers imported from Linux kernel and used for implementing
> +  KVM support and user-mode.

We could add here

 Don't change the files in here by hand; instead you can use the
 ``scripts/update-linux-headers.sh`` script to update them from a
 kernel source tree.

But maybe that's starting to be a bit much info for an entry in this list.

> +* `linux-user <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-user>`_:
> +  `User mode <user-mode>` implementation. Contains one folder per target
> +  architecture.
> +* `.gitlab-ci.d <https://gitlab.com/qemu-project/qemu/-/tree/master/.gitlab-ci.d>`_:
> +  `CI <ci>` yaml and scripts.
> +* `include <https://gitlab.com/qemu-project/qemu/-/tree/master/include>`_:
> +  All headers associated to different subsystems in QEMU. Hierachy used mirrors

"The hierarchy"

> +  source code organization and naming.
> +* `hw <https://gitlab.com/qemu-project/qemu/-/tree/master/hw>`_:
> +  `Devices <device-emulation>` and boards emulation. Devices are categorized by
> +  type/protocol/architecture and located in associated subfolder.
> +* `io <https://gitlab.com/qemu-project/qemu/-/tree/master/io>`_:
> +  QEMU `I/O channels <https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04208.html>`_.
> +* `libdecnumber <https://gitlab.com/qemu-project/qemu/-/tree/master/libdecnumber>`_:
> +  Import of gcc library, used to implement decimal number arithmetic.
> +* `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/migration>`__:
> +  `Migration framework <migration>`.
> +* `monitor <https://gitlab.com/qemu-project/qemu/-/tree/master/monitor>`_:
> +  `Monitor <QEMU monitor>` implementation (HMP & QMP).
> +* `nbd <https://gitlab.com/qemu-project/qemu/-/tree/master/nbd>`_:
> +  QEMU `NBD (Network Block Device) <nbd>` server.
> +* `net <https://gitlab.com/qemu-project/qemu/-/tree/master/net>`_:
> +  Network (host) support.
> +* `pc-bios <https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios>`_:
> +  Contains pre-built firmware binaries and boot images, ready to use in
> +  QEMU without compilation.
> +* `plugins <https://gitlab.com/qemu-project/qemu/-/tree/master/plugins>`_:
> +  `TCG plugins <tcg-plugins>` core implementation. Plugins can be found in
> +  `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg/plugins>`__
> +  and `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/plugins>`__
> +  folders.
> +* `po <https://gitlab.com/qemu-project/qemu/-/tree/master/po>`_:
> +  Translation files.
> +* `python <https://gitlab.com/qemu-project/qemu/-/tree/master/python>`_:
> +  Python part of our build/test system.
> +* `qapi <https://gitlab.com/qemu-project/qemu/-/tree/master/qapi>`_:
> +  `QAPI <qapi>` implementation.
> +* `qobject <https://gitlab.com/qemu-project/qemu/-/tree/master/qobject>`_:
> +  QEMU Object implementation.
> +* `qga <https://gitlab.com/qemu-project/qemu/-/tree/master/qga>`_:
> +  QEMU `Guest agent <qemu-ga>` implementation.
> +* `qom <https://gitlab.com/qemu-project/qemu/-/tree/master/qom>`_:
> +  QEMU `Object model <qom>` implementation, with monitor associated commands.
> +* `replay <https://gitlab.com/qemu-project/qemu/-/tree/master/replay>`_:
> +  QEMU `record/replay <replay>` implementation.
> +* `roms <https://gitlab.com/qemu-project/qemu/-/tree/master/roms>`_:
> +  Contains source code for various firmware and ROMs, which can be compiled if
> +  custom or updated versions are needed.
> +* `rust <https://gitlab.com/qemu-project/qemu/-/tree/master/rust>`_:
> +  Rust integration in QEMU. It contains the new interfaces defined and
> +  associated devices using it.
> +* `scripts <https://gitlab.com/qemu-project/qemu/-/tree/master/scripts>`_:
> +  Collection of scripts used in build and test systems, and various
> +  tools for QEMU codebase and execution traces.
> +* `scsi <https://gitlab.com/qemu-project/qemu/-/tree/master/scsi>`_:
> +  Code related to SCSI support, used by SCSI devices.
> +* `semihosting <https://gitlab.com/qemu-project/qemu/-/tree/master/semihosting>`_:
> +  QEMU `Semihosting <Semihosting>` implementation.
> +* `stats <https://gitlab.com/qemu-project/qemu/-/tree/master/stats>`_:
> +  `Monitor <QEMU monitor>` stats commands implementation.
> +* `storage-daemon <https://gitlab.com/qemu-project/qemu/-/tree/master/storage-daemon>`_:
> +  QEMU `Storage daemon <storage-daemon>` implementation.
> +* `stubs <https://gitlab.com/qemu-project/qemu/-/tree/master/stubs>`_:
> +  Various stubs (empty functions) used to compile QEMU with specific
> +  configurations.
> +* `subprojects <https://gitlab.com/qemu-project/qemu/-/tree/master/subprojects>`_:
> +  QEMU submodules used by QEMU build system.
> +* `system <https://gitlab.com/qemu-project/qemu/-/tree/master/system>`_:
> +  QEMU `system mode <System emulation>` implementation (cpu, mmu, boot support).
> +* `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_:
> +  Contains code for all target architectures supported (one subfolder
> +  per arch). For every architecture, you can find accelerator specific
> +  implementations.
> +* `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tcg>`_:
> +  `TCG <tcg>` related code.
> +  Contains one subfolder per host supported architecture.
> +* `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_:
> +  QEMU `test <testing>` suite
> +
> +  - `avocado <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/avocado>`_:
> +    Functional tests booting full VM using `Avocado framework <checkavocado-ref>`.
> +    Those tests will be transformed and moved into
> +    `tests/functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_
> +    in the future.
> +  - `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_:
> +    Data for various tests.
> +  - `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_:
> +    Testsuite for `decodetree <decodetree>` implementation.
> +  - `docker <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/docker>`_:
> +    Code and scripts to create `containers <container-ref>` used in `CI <ci>`.
> +  - `fp <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/fp>`_:
> +    QEMU testsuite for soft float implementation.
> +  - `functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_:
> +    `Functional tests <checkfunctional-ref>` (full VM boot).
> +  - `lcitool <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/lcitool>`_:
> +    Generate dockerfiles for CI containers.
> +  - `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/migration>`_:
> +    Test scripts and data for `Migration framework <migration>`.
> +  - `multiboot <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/multiboot>`_:
> +    Test multiboot functionality for x86_64/i386.
> +  - `qapi-schema <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qapi-schema>`_:
> +    Test scripts and data for `QAPI <qapi-tests>`.
> +  - `qemu-iotests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qemu-iotests>`_:
> +    `Disk image and block tests <qemu-iotests>`.
> +  - `qtest <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qtest>`_:
> +    `Device emulation testing <qtest>`.
> +  - `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg>`__:
> +    `TCG related tests <checktcg-ref>`. Contains code per architecture
> +    (subfolder) and multiarch tests as well.
> +  - `tsan <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tsan>`_:
> +    `Suppressions <tsan-suppressions>` for thread sanitizer.
> +  - `uefi-test-tools <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/uefi-test-tools>`_:
> +    Test tool for UEFI support.
> +  - `unit <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/unit>`_:
> +    QEMU `Unit tests <unit-tests>`.
> +* `trace <https://gitlab.com/qemu-project/qemu/-/tree/master/trace>`_:
> +  `Tracing framework <tracing>`. Used to print information associated to various
> +  events during execution.
> +* `ui <https://gitlab.com/qemu-project/qemu/-/tree/master/ui>`_:
> +  QEMU User interfaces.
> +* `util <https://gitlab.com/qemu-project/qemu/-/tree/master/util>`_:
> +  Utility code used by other parts of QEMU.

thanks
-- PMM

Re: [PATCH 5/7] docs: add a codebase section

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> Present the various parts of QEMU and organization of codebase.
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>
> I like this; it's something I've thought for a while would
> be good to have, but which I never got round to trying to
> put together. Thanks for doing this!
>
> Mostly my comments below are spelling/typo nits and
> other minor stuff.
>
>> ---
>>  docs/about/emulation.rst               |   2 +
>>  docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
>>  docs/devel/decodetree.rst              |   2 +
>>  docs/devel/ebpf_rss.rst                |   2 +
>>  docs/devel/index-internals.rst         |   2 +
>>  docs/devel/migration/main.rst          |   2 +
>>  docs/devel/qapi-code-gen.rst           |   1 +
>>  docs/devel/testing/main.rst            |   9 +-
>>  docs/devel/testing/qtest.rst           |   2 +
>>  docs/index.rst                         |   3 +
>>  docs/interop/qemu-ga.rst               |   2 +
>>  docs/system/qemu-block-drivers.rst.inc |   2 +
>>  docs/tools/qemu-storage-daemon.rst     |   2 +
>>  docs/user/main.rst                     |   6 +
>>  14 files changed, 247 insertions(+), 1 deletion(-)
>>  create mode 100644 docs/codebase/index.rst
>>
<snip>
>> +  Block devices and `image formats<disk images>` implementation.
>> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
>> +  `BSD User mode<bsd-user-mode>`.
>> +* build: Where the code built goes!
>
> The built code doesn't have to be in 'build'. We could say:
>
>  * build: You can tell the QEMU build system to put the built code
>    anywhere you like. By default it will go into a directory named
>    ``build``. Sometimes documentation will assume this default
>    for convenience when describing command lines; you can always
>    replace it with the path to your build tree.
>
> ?

I always recommend creating a builds directory and having multiple build
trees under it:

  🕙17:22:02 alex@draig:qemu.git  on  testing/next [$!?] 
  ➜  tree -L 1 builds
  builds
  ├── all
  ├── all.clang
  ├── all.disabled
  ├── arm64-system.crossbuild
  ├── arm.afl
  ├── arm.all
  ├── arm.debug
  ├── arm.gcovr
  ├── arm.user.32bit
  ├── bisect
  ├── debug
  ├── debug.clang
  ├── deprecated
  ├── disable-misc
  ├── disable.plugins
  ├── disable-tcg
  ├── disable-user
  ├── extra.libs
  ├── gcov
  ├── gcov2
  ├── lto
  ├── profiling
  ├── qemu-system-arm
  ├── rust
  ├── sanitisers
  ├── sanitizer.clang
  ├── system
  ├── system.i386
  ├── system.nodefaults
  ├── system.threadsan
  ├── tcg
  ├── tci
  ├── tools-and-docs
  ├── trs.debug
  ├── tsan
  ├── user
  ├── user.static
  ├── virtio-gpu
  └── xen


-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 5/7] docs: add a codebase section

[Peter Maydell]

On Tue, 3 Dec 2024 at 17:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>
> Peter Maydell <peter.maydell@linaro.org> writes:
>
> > On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> > <pierrick.bouvier@linaro.org> wrote:
> >>
> >> Present the various parts of QEMU and organization of codebase.
> >>
> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> >
> > I like this; it's something I've thought for a while would
> > be good to have, but which I never got round to trying to
> > put together. Thanks for doing this!
> >
> > Mostly my comments below are spelling/typo nits and
> > other minor stuff.
> >
> >> ---
> >>  docs/about/emulation.rst               |   2 +
> >>  docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
> >>  docs/devel/decodetree.rst              |   2 +
> >>  docs/devel/ebpf_rss.rst                |   2 +
> >>  docs/devel/index-internals.rst         |   2 +
> >>  docs/devel/migration/main.rst          |   2 +
> >>  docs/devel/qapi-code-gen.rst           |   1 +
> >>  docs/devel/testing/main.rst            |   9 +-
> >>  docs/devel/testing/qtest.rst           |   2 +
> >>  docs/index.rst                         |   3 +
> >>  docs/interop/qemu-ga.rst               |   2 +
> >>  docs/system/qemu-block-drivers.rst.inc |   2 +
> >>  docs/tools/qemu-storage-daemon.rst     |   2 +
> >>  docs/user/main.rst                     |   6 +
> >>  14 files changed, 247 insertions(+), 1 deletion(-)
> >>  create mode 100644 docs/codebase/index.rst
> >>
> <snip>
> >> +  Block devices and `image formats<disk images>` implementation.
> >> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
> >> +  `BSD User mode<bsd-user-mode>`.
> >> +* build: Where the code built goes!
> >
> > The built code doesn't have to be in 'build'. We could say:
> >
> >  * build: You can tell the QEMU build system to put the built code
> >    anywhere you like. By default it will go into a directory named
> >    ``build``. Sometimes documentation will assume this default
> >    for convenience when describing command lines; you can always
> >    replace it with the path to your build tree.
> >
> > ?
>
> I always recommend creating a builds directory and having multiple build
> trees under it:

Indeed, that's what I like to do too, but I don't think this
document is the right place to make that kind of recommendation.

-- PMM

Re: [PATCH 5/7] docs: add a codebase section

[Pierrick Bouvier]

On 12/3/24 09:46, Peter Maydell wrote:
> On Tue, 3 Dec 2024 at 17:22, Alex Bennée <alex.bennee@linaro.org> wrote:
>>
>> Peter Maydell <peter.maydell@linaro.org> writes:
>>
>>> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
>>> <pierrick.bouvier@linaro.org> wrote:
>>>>
>>>> Present the various parts of QEMU and organization of codebase.
>>>>
>>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>>
>>> I like this; it's something I've thought for a while would
>>> be good to have, but which I never got round to trying to
>>> put together. Thanks for doing this!
>>>
>>> Mostly my comments below are spelling/typo nits and
>>> other minor stuff.
>>>
>>>> ---
>>>>   docs/about/emulation.rst               |   2 +
>>>>   docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
>>>>   docs/devel/decodetree.rst              |   2 +
>>>>   docs/devel/ebpf_rss.rst                |   2 +
>>>>   docs/devel/index-internals.rst         |   2 +
>>>>   docs/devel/migration/main.rst          |   2 +
>>>>   docs/devel/qapi-code-gen.rst           |   1 +
>>>>   docs/devel/testing/main.rst            |   9 +-
>>>>   docs/devel/testing/qtest.rst           |   2 +
>>>>   docs/index.rst                         |   3 +
>>>>   docs/interop/qemu-ga.rst               |   2 +
>>>>   docs/system/qemu-block-drivers.rst.inc |   2 +
>>>>   docs/tools/qemu-storage-daemon.rst     |   2 +
>>>>   docs/user/main.rst                     |   6 +
>>>>   14 files changed, 247 insertions(+), 1 deletion(-)
>>>>   create mode 100644 docs/codebase/index.rst
>>>>
>> <snip>
>>>> +  Block devices and `image formats<disk images>` implementation.
>>>> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
>>>> +  `BSD User mode<bsd-user-mode>`.
>>>> +* build: Where the code built goes!
>>>
>>> The built code doesn't have to be in 'build'. We could say:
>>>
>>>   * build: You can tell the QEMU build system to put the built code
>>>     anywhere you like. By default it will go into a directory named
>>>     ``build``. Sometimes documentation will assume this default
>>>     for convenience when describing command lines; you can always
>>>     replace it with the path to your build tree.
>>>
>>> ?
>>
>> I always recommend creating a builds directory and having multiple build
>> trees under it:
> 
> Indeed, that's what I like to do too, but I don't think this
> document is the right place to make that kind of recommendation.
>

I agree with Peter.
People who understand this need already know how to do it, and it's not 
the best place to mention that.

As well, using several folders can be error prone (using "old" binaries 
by mistake happens quickly), and it's something I usually don't advise 
for beginners. Using a symlink that points to the right folder is less 
error prone, but it opens the path to another (custom and personal) 
layer to the build command.

Another (more simple) solution is to use a single folder, and simply 
rely on ccache for quick rebuilds.

There are probably as many solutions and opinions as there are 
developers on this thread, so it's better to simply mention one build 
folder, and nothing else.

> -- PMM

Re: [PATCH 5/7] docs: add a codebase section

[Daniel P. Berrangé]

On Tue, Dec 03, 2024 at 05:22:50PM +0000, Alex Bennée wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
> > On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> > <pierrick.bouvier@linaro.org> wrote:
> >>
> >> Present the various parts of QEMU and organization of codebase.
> >>
> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> >
> > I like this; it's something I've thought for a while would
> > be good to have, but which I never got round to trying to
> > put together. Thanks for doing this!
> >
> > Mostly my comments below are spelling/typo nits and
> > other minor stuff.
> >
> >> ---
> >>  docs/about/emulation.rst               |   2 +
> >>  docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
> >>  docs/devel/decodetree.rst              |   2 +
> >>  docs/devel/ebpf_rss.rst                |   2 +
> >>  docs/devel/index-internals.rst         |   2 +
> >>  docs/devel/migration/main.rst          |   2 +
> >>  docs/devel/qapi-code-gen.rst           |   1 +
> >>  docs/devel/testing/main.rst            |   9 +-
> >>  docs/devel/testing/qtest.rst           |   2 +
> >>  docs/index.rst                         |   3 +
> >>  docs/interop/qemu-ga.rst               |   2 +
> >>  docs/system/qemu-block-drivers.rst.inc |   2 +
> >>  docs/tools/qemu-storage-daemon.rst     |   2 +
> >>  docs/user/main.rst                     |   6 +
> >>  14 files changed, 247 insertions(+), 1 deletion(-)
> >>  create mode 100644 docs/codebase/index.rst
> >>
> <snip>
> >> +  Block devices and `image formats<disk images>` implementation.
> >> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
> >> +  `BSD User mode<bsd-user-mode>`.
> >> +* build: Where the code built goes!
> >
> > The built code doesn't have to be in 'build'. We could say:
> >
> >  * build: You can tell the QEMU build system to put the built code
> >    anywhere you like. By default it will go into a directory named
> >    ``build``. Sometimes documentation will assume this default
> >    for convenience when describing command lines; you can always
> >    replace it with the path to your build tree.
> >
> > ?
> 
> I always recommend creating a builds directory and having multiple build
> trees under it:

I can understand why you do that, but I'm doubtful the need to have
many parallel build directories is the common case. IOW, I expect
that for the majority of contributors the default single 'build'
directory is sufficient.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 5/7] docs: add a codebase section

[Pierrick Bouvier]

On 12/3/24 07:53, Peter Maydell wrote:
> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> Present the various parts of QEMU and organization of codebase.
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> 
> I like this; it's something I've thought for a while would
> be good to have, but which I never got round to trying to
> put together. Thanks for doing this!
> 
> Mostly my comments below are spelling/typo nits and
> other minor stuff.
> 
>> ---
>>   docs/about/emulation.rst               |   2 +
>>   docs/codebase/index.rst                | 211 +++++++++++++++++++++++++
>>   docs/devel/decodetree.rst              |   2 +
>>   docs/devel/ebpf_rss.rst                |   2 +
>>   docs/devel/index-internals.rst         |   2 +
>>   docs/devel/migration/main.rst          |   2 +
>>   docs/devel/qapi-code-gen.rst           |   1 +
>>   docs/devel/testing/main.rst            |   9 +-
>>   docs/devel/testing/qtest.rst           |   2 +
>>   docs/index.rst                         |   3 +
>>   docs/interop/qemu-ga.rst               |   2 +
>>   docs/system/qemu-block-drivers.rst.inc |   2 +
>>   docs/tools/qemu-storage-daemon.rst     |   2 +
>>   docs/user/main.rst                     |   6 +
>>   14 files changed, 247 insertions(+), 1 deletion(-)
>>   create mode 100644 docs/codebase/index.rst
>>
>> diff --git a/docs/about/emulation.rst b/docs/about/emulation.rst
>> index 3028d5fff7a..3bc3579434f 100644
>> --- a/docs/about/emulation.rst
>> +++ b/docs/about/emulation.rst
>> @@ -176,6 +176,8 @@ for that architecture.
>>       - System
>>       - Tensilica ISS SIMCALL
>>
>> +.. _tcg-plugins:
>> +
>>   TCG Plugins
>>   -----------
>>
>> diff --git a/docs/codebase/index.rst b/docs/codebase/index.rst
>> new file mode 100644
>> index 00000000000..353830ef175
>> --- /dev/null
>> +++ b/docs/codebase/index.rst
> 
> This is developer documentation, so I think it should
> be in docs/devel/.
> 
>> @@ -0,0 +1,211 @@
>> +========
>> +Codebase
>> +========
>> +
>> +This section present the various parts of QEMU and how codebase is organized.
> 
> "presents"; "how the codebase"
> 
>> +
>> +Beyond giving succint descriptions, the goal is to offer links to various
> 
> "succinct"
> 
>> +parts of the documentation/codebase.
>> +
>> +Subsystems
>> +----------
>> +
>> +An exhaustive list of subsystems and files associated can be found in
> 
> "associated files"
> 
> "the MAINTAINERS file"
> 
>> +`MAINTAINERS <https://gitlab.com/qemu-project/qemu/-/blob/master/MAINTAINERS>`_
>> +file.
>> +
>> +Some of the main QEMU subsystems are:
>> +
>> +- `Accelerators<Accelerators>`
>> +- Block devices and `disk images<disk images>` support
>> +- `CI<ci>` and `Tests<testing>`
>> +- `Devices<device-emulation>` & Board models
>> +- `Documentation <documentation-root>`
>> +- `GDB support<GDB usage>`
>> +- `Migration<migration>`
>> +- `Monitor<QEMU monitor>`
>> +- `QOM (QEMU Object Model)<qom>`
>> +- `System mode<System emulation>`
>> +- `TCG (Tiny Code Generator)<tcg>`
>> +- `User mode<user-mode>` (`Linux<linux-user-mode>` & `BSD<bsd-user-mode>`)
>> +- User Interfaces
>> +
>> +More documentation on QEMU subsystems can be found on :ref:`internal-subsystem`
>> +page.
>> +
>> +The Grand tour
>> +--------------
>> +
>> +We present briefly here what every folder of the codebase contains. Hop on!
> 
> 
> I think it would be helpful to mention at the top of this list something
> like:
> 
>   The folder name links here will take you to that folder in our
>   gitlab repository; other links will take you to more detailed
>   documentation for that subsystem, where we have it. Unfortunately
>   not every subsystem has documentation yet, so sometimes the
>   source code is all you have.
> 
> just so readers know that if e.g. they have a local source tree
> already there's no point in clicking on the 'crypto' link etc,
> but that other links are going to go somewhere with more
> human-written detail.
> 
>> +
>> +* `accel <https://gitlab.com/qemu-project/qemu/-/tree/master/accel>`_:
>> +  Infrastructure and architecture agnostic code related to the various
>> +  `accelerators <Accelerators>` supported by QEMU
>> +  (TCG, KVM, hvf, whpx, xen, nvmm).
>> +  Contains interfaces for operations that will be implemented per
>> +  `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_.
>> +* `audio <https://gitlab.com/qemu-project/qemu/-/tree/master/audio>`_:
>> +  Audio (host) support.
>> +* `authz <https://gitlab.com/qemu-project/qemu/-/tree/master/authz>`_:
>> +  `QEMU Authorization framework<client authorization>`.
>> +* `backends <https://gitlab.com/qemu-project/qemu/-/tree/master/backends>`_:
>> +  Various backends used for device emulation.
>> +* `block <https://gitlab.com/qemu-project/qemu/-/tree/master/block>`_:
>> +  Block devices and `image formats<disk images>` implementation.
>> +* `bsd-user <https://gitlab.com/qemu-project/qemu/-/tree/master/bsd-user>`_:
>> +  `BSD User mode<bsd-user-mode>`.
>> +* build: Where the code built goes!
> 
> The built code doesn't have to be in 'build'. We could say:
> 
>   * build: You can tell the QEMU build system to put the built code
>     anywhere you like. By default it will go into a directory named
>     ``build``. Sometimes documentation will assume this default
>     for convenience when describing command lines; you can always
>     replace it with the path to your build tree.
> 
> ?

I'm ok with the first two sentences but the third one (Sometimes.*) adds 
little value, and much more verbosity.

> 
>> +* `chardev <https://gitlab.com/qemu-project/qemu/-/tree/master/chardev>`_:
>> +  Various backends used by char devices.
>> +* `common-user <https://gitlab.com/qemu-project/qemu/-/tree/master/common-user>`_:
>> +  User-mode assembly code for dealing with signals occuring during syscalls.
>> +* `configs <https://gitlab.com/qemu-project/qemu/-/tree/master/configs>`_:
>> +  Makefiles defining configurations to build QEMU.
>> +* `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib>`_:
>> +  Community contributed devices/plugins/tools.
>> +* `crypto <https://gitlab.com/qemu-project/qemu/-/tree/master/crypto>`_:
>> +  Cryptographic algorithms used in QEMU.
>> +* `disas <https://gitlab.com/qemu-project/qemu/-/tree/master/disas>`_:
>> +  Disassembly functions used by QEMU target code.
>> +* `docs <https://gitlab.com/qemu-project/qemu/-/tree/master/docs>`_:
>> +  QEMU Documentation.
>> +* `dump <https://gitlab.com/qemu-project/qemu/-/tree/master/dump>`_:
>> +  Code to dump memory of a running VM.
>> +* `ebpf <https://gitlab.com/qemu-project/qemu/-/tree/master/ebpf>`_:
>> +  eBPF program support in QEMU. `virtio-net RSS<ebpf-rss>` uses it.
>> +* `fpu <https://gitlab.com/qemu-project/qemu/-/tree/master/fpu>`_:
>> +  Floating-point software emulation.
>> +* `fsdev <https://gitlab.com/qemu-project/qemu/-/tree/master/fsdev>`_:
>> +  `VirtFS <https://www.linux-kvm.org/page/VirtFS>`_ support.
>> +* `gdbstub <https://gitlab.com/qemu-project/qemu/-/tree/master/gdbstub>`_:
>> +  `GDB <GDB usage>` support.
>> +* `gdb-xml <https://gitlab.com/qemu-project/qemu/-/tree/master/gdb-xml>`_:
>> +  Set of XML files describing architectures and used by `gdbstub <GDB usage>`.
>> +* `host <https://gitlab.com/qemu-project/qemu/-/tree/master/host>`_:
>> +  Various architecture specific header files (crypto, atomic, memory
>> +  operations).
>> +* `linux-headers <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-headers>`_:
>> +  A subset of headers imported from Linux kernel and used for implementing
>> +  KVM support and user-mode.
> 
> We could add here
> 
>   Don't change the files in here by hand; instead you can use the
>   ``scripts/update-linux-headers.sh`` script to update them from a
>   kernel source tree.
> 
> But maybe that's starting to be a bit much info for an entry in this list.
> 

Yes, it's a very specific detail and I doubt someone will try to update 
this after reading this doc page. They can still try and see what 
happens on the mailing list :)

>> +* `linux-user <https://gitlab.com/qemu-project/qemu/-/tree/master/linux-user>`_:
>> +  `User mode <user-mode>` implementation. Contains one folder per target
>> +  architecture.
>> +* `.gitlab-ci.d <https://gitlab.com/qemu-project/qemu/-/tree/master/.gitlab-ci.d>`_:
>> +  `CI <ci>` yaml and scripts.
>> +* `include <https://gitlab.com/qemu-project/qemu/-/tree/master/include>`_:
>> +  All headers associated to different subsystems in QEMU. Hierachy used mirrors
> 
> "The hierarchy"
> 
>> +  source code organization and naming.
>> +* `hw <https://gitlab.com/qemu-project/qemu/-/tree/master/hw>`_:
>> +  `Devices <device-emulation>` and boards emulation. Devices are categorized by
>> +  type/protocol/architecture and located in associated subfolder.
>> +* `io <https://gitlab.com/qemu-project/qemu/-/tree/master/io>`_:
>> +  QEMU `I/O channels <https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg04208.html>`_.
>> +* `libdecnumber <https://gitlab.com/qemu-project/qemu/-/tree/master/libdecnumber>`_:
>> +  Import of gcc library, used to implement decimal number arithmetic.
>> +* `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/migration>`__:
>> +  `Migration framework <migration>`.
>> +* `monitor <https://gitlab.com/qemu-project/qemu/-/tree/master/monitor>`_:
>> +  `Monitor <QEMU monitor>` implementation (HMP & QMP).
>> +* `nbd <https://gitlab.com/qemu-project/qemu/-/tree/master/nbd>`_:
>> +  QEMU `NBD (Network Block Device) <nbd>` server.
>> +* `net <https://gitlab.com/qemu-project/qemu/-/tree/master/net>`_:
>> +  Network (host) support.
>> +* `pc-bios <https://gitlab.com/qemu-project/qemu/-/tree/master/pc-bios>`_:
>> +  Contains pre-built firmware binaries and boot images, ready to use in
>> +  QEMU without compilation.
>> +* `plugins <https://gitlab.com/qemu-project/qemu/-/tree/master/plugins>`_:
>> +  `TCG plugins <tcg-plugins>` core implementation. Plugins can be found in
>> +  `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg/plugins>`__
>> +  and `contrib <https://gitlab.com/qemu-project/qemu/-/tree/master/contrib/plugins>`__
>> +  folders.
>> +* `po <https://gitlab.com/qemu-project/qemu/-/tree/master/po>`_:
>> +  Translation files.
>> +* `python <https://gitlab.com/qemu-project/qemu/-/tree/master/python>`_:
>> +  Python part of our build/test system.
>> +* `qapi <https://gitlab.com/qemu-project/qemu/-/tree/master/qapi>`_:
>> +  `QAPI <qapi>` implementation.
>> +* `qobject <https://gitlab.com/qemu-project/qemu/-/tree/master/qobject>`_:
>> +  QEMU Object implementation.
>> +* `qga <https://gitlab.com/qemu-project/qemu/-/tree/master/qga>`_:
>> +  QEMU `Guest agent <qemu-ga>` implementation.
>> +* `qom <https://gitlab.com/qemu-project/qemu/-/tree/master/qom>`_:
>> +  QEMU `Object model <qom>` implementation, with monitor associated commands.
>> +* `replay <https://gitlab.com/qemu-project/qemu/-/tree/master/replay>`_:
>> +  QEMU `record/replay <replay>` implementation.
>> +* `roms <https://gitlab.com/qemu-project/qemu/-/tree/master/roms>`_:
>> +  Contains source code for various firmware and ROMs, which can be compiled if
>> +  custom or updated versions are needed.
>> +* `rust <https://gitlab.com/qemu-project/qemu/-/tree/master/rust>`_:
>> +  Rust integration in QEMU. It contains the new interfaces defined and
>> +  associated devices using it.
>> +* `scripts <https://gitlab.com/qemu-project/qemu/-/tree/master/scripts>`_:
>> +  Collection of scripts used in build and test systems, and various
>> +  tools for QEMU codebase and execution traces.
>> +* `scsi <https://gitlab.com/qemu-project/qemu/-/tree/master/scsi>`_:
>> +  Code related to SCSI support, used by SCSI devices.
>> +* `semihosting <https://gitlab.com/qemu-project/qemu/-/tree/master/semihosting>`_:
>> +  QEMU `Semihosting <Semihosting>` implementation.
>> +* `stats <https://gitlab.com/qemu-project/qemu/-/tree/master/stats>`_:
>> +  `Monitor <QEMU monitor>` stats commands implementation.
>> +* `storage-daemon <https://gitlab.com/qemu-project/qemu/-/tree/master/storage-daemon>`_:
>> +  QEMU `Storage daemon <storage-daemon>` implementation.
>> +* `stubs <https://gitlab.com/qemu-project/qemu/-/tree/master/stubs>`_:
>> +  Various stubs (empty functions) used to compile QEMU with specific
>> +  configurations.
>> +* `subprojects <https://gitlab.com/qemu-project/qemu/-/tree/master/subprojects>`_:
>> +  QEMU submodules used by QEMU build system.
>> +* `system <https://gitlab.com/qemu-project/qemu/-/tree/master/system>`_:
>> +  QEMU `system mode <System emulation>` implementation (cpu, mmu, boot support).
>> +* `target <https://gitlab.com/qemu-project/qemu/-/tree/master/target>`_:
>> +  Contains code for all target architectures supported (one subfolder
>> +  per arch). For every architecture, you can find accelerator specific
>> +  implementations.
>> +* `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tcg>`_:
>> +  `TCG <tcg>` related code.
>> +  Contains one subfolder per host supported architecture.
>> +* `tests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests>`_:
>> +  QEMU `test <testing>` suite
>> +
>> +  - `avocado <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/avocado>`_:
>> +    Functional tests booting full VM using `Avocado framework <checkavocado-ref>`.
>> +    Those tests will be transformed and moved into
>> +    `tests/functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_
>> +    in the future.
>> +  - `data <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/data>`_:
>> +    Data for various tests.
>> +  - `decode <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/decode>`_:
>> +    Testsuite for `decodetree <decodetree>` implementation.
>> +  - `docker <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/docker>`_:
>> +    Code and scripts to create `containers <container-ref>` used in `CI <ci>`.
>> +  - `fp <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/fp>`_:
>> +    QEMU testsuite for soft float implementation.
>> +  - `functional <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/functional>`_:
>> +    `Functional tests <checkfunctional-ref>` (full VM boot).
>> +  - `lcitool <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/lcitool>`_:
>> +    Generate dockerfiles for CI containers.
>> +  - `migration <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/migration>`_:
>> +    Test scripts and data for `Migration framework <migration>`.
>> +  - `multiboot <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/multiboot>`_:
>> +    Test multiboot functionality for x86_64/i386.
>> +  - `qapi-schema <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qapi-schema>`_:
>> +    Test scripts and data for `QAPI <qapi-tests>`.
>> +  - `qemu-iotests <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qemu-iotests>`_:
>> +    `Disk image and block tests <qemu-iotests>`.
>> +  - `qtest <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/qtest>`_:
>> +    `Device emulation testing <qtest>`.
>> +  - `tcg <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tcg>`__:
>> +    `TCG related tests <checktcg-ref>`. Contains code per architecture
>> +    (subfolder) and multiarch tests as well.
>> +  - `tsan <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/tsan>`_:
>> +    `Suppressions <tsan-suppressions>` for thread sanitizer.
>> +  - `uefi-test-tools <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/uefi-test-tools>`_:
>> +    Test tool for UEFI support.
>> +  - `unit <https://gitlab.com/qemu-project/qemu/-/tree/master/tests/unit>`_:
>> +    QEMU `Unit tests <unit-tests>`.
>> +* `trace <https://gitlab.com/qemu-project/qemu/-/tree/master/trace>`_:
>> +  `Tracing framework <tracing>`. Used to print information associated to various
>> +  events during execution.
>> +* `ui <https://gitlab.com/qemu-project/qemu/-/tree/master/ui>`_:
>> +  QEMU User interfaces.
>> +* `util <https://gitlab.com/qemu-project/qemu/-/tree/master/util>`_:
>> +  Utility code used by other parts of QEMU.
> 
> thanks
> -- PMM

Thanks for all the suggestions, I'll integrate those.

Pierrick

[PATCH 6/7] docs: add a glossary

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/devel/control-flow-integrity.rst |   2 +
 docs/devel/multi-thread-tcg.rst       |   2 +
 docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
 docs/index.rst                        |   1 +
 docs/system/arm/virt.rst              |   2 +
 docs/system/images.rst                |   2 +
 docs/tools/qemu-nbd.rst               |   2 +
 7 files changed, 249 insertions(+)
 create mode 100644 docs/glossary/index.rst

diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
index e6b73a4fe1a..3d5702fa4cc 100644
--- a/docs/devel/control-flow-integrity.rst
+++ b/docs/devel/control-flow-integrity.rst
@@ -1,3 +1,5 @@
+.. _cfi:
+
 ============================
 Control-Flow Integrity (CFI)
 ============================
diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
index d706c27ea74..7fd0a07633d 100644
--- a/docs/devel/multi-thread-tcg.rst
+++ b/docs/devel/multi-thread-tcg.rst
@@ -4,6 +4,8 @@
   This work is licensed under the terms of the GNU GPL, version 2 or
   later. See the COPYING file in the top-level directory.
 
+.. _mttcg:
+
 ==================
 Multi-threaded TCG
 ==================
diff --git a/docs/glossary/index.rst b/docs/glossary/index.rst
new file mode 100644
index 00000000000..a2d4f3eae16
--- /dev/null
+++ b/docs/glossary/index.rst
@@ -0,0 +1,238 @@
+.. _Glossary:
+
+--------
+Glossary
+--------
+
+This section of the manual presents *simply* acronyms and terms QEMU developers
+use.
+
+Accelerator
+-----------
+
+A specific API used to accelerate execution of guest instructions. It can be
+hardware-based, through a virtualization API provided by the host OS (kvm, hvf,
+whpx, ...) or software-based (tcg). See this description of `supported
+accelerators<Accelerators>`.
+
+Board
+-----
+
+QEMU system defines board models for various architectures. It's a description
+of a SoC (system-on-chip) with various devices pre-configured, and can be
+selected with the option ``-machine`` of qemu-system.
+For virtual machines, you'll use ``virt`` board model, designed for this use
+case. As an example, for Arm architecture, you can find the `model code
+<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/arm/virt.c>`_ and
+associated `documentation <arm-virt>`.
+
+Block
+-----
+
+Block drivers are the available `disk formats <block-drivers>` available, and
+block devices `(see Block device section on options page)<sec_005finvocation>`
+are using them to implement disks for a virtual machine.
+
+CFI
+---
+
+Control Flow Integrity is a hardening technique used to prevent exploits
+targeting QEMU by detecting unexpected branches during execution. QEMU `actively
+supports<cfi>` being compiled with CFI enabled.
+
+Device
+------
+
+QEMU is able to emulate a CPU, and all the hardware interacting with it,
+including many devices. When QEMU runs a virtual machine using a hardware-based
+accelerator, it is responsible for emulating, using software, all devices.
+
+EDK2
+----
+
+EDK2, as known as `TianoCore <https://www.tianocore.org/>`_, is an open source
+implementation of UEFI standard. It's ran by QEMU to support UEFI for virtual
+machines.
+
+gdbstub
+-------
+
+QEMU implements a `gdb server <GDB usage>`, allowing gdb to attach to it and
+debug a running virtual machine, or a program in user-mode. This allows to debug
+a given architecture without having access to hardware.
+
+glib2
+-----
+
+`GLib2 <https://docs.gtk.org/glib/>`_ is one of the most important library we
+are using through the codebase. It provides many data structures, macros, string
+and thread utilities and portable functions across different OS. It's required
+to build QEMU.
+
+Guest agent
+-----------
+
+`QEMU Guest agent <qemu-ga>` is a daemon intended to be executed by guest
+virtual machines and providing various services to help QEMU to interact with
+it.
+
+Guest/Host
+----------
+
+Guest is the architecture of the virtual machine, which is emulated.
+Host is the architecture on which QEMU is running on, which is native.
+
+Hypervisor
+----------
+
+The formal definition of an hypervisor is a program than can be used to manage a
+virtual machine. QEMU itself is an hypervisor.
+
+In the context of QEMU, an hypervisor is an API, provided by the Host OS,
+allowing to execute virtual machines. Linux implementation is KVM (and supports
+Xen as well). For MacOS, it's HVF. Windows defines WHPX. And NetBSD provides
+NVMM.
+
+Migration
+---------
+
+QEMU can save and restore the execution of a virtual machine, including across
+different machines. This is provided by the `Migration framework<migration>`.
+
+NBD
+---
+
+`QEMU Network Block Device server <qemu-nbd>` is a tool that can be used to
+mount and access QEMU images, providing functionality similar to a loop device.
+
+Mailing List
+------------
+
+This is `where <https://wiki.qemu.org/Contribute/MailingLists>`_ all the
+development happens! Changes are posted as series, that all developers can
+review and share feedback for.
+
+For reporting issues, our `GitLab
+<https://gitlab.com/qemu-project/qemu/-/issues>`_ tracker is the best place.
+
+MMU / softmmu
+-------------
+
+The Memory Management Unit is responsible for translating virtual addresses to
+physical addresses and managing memory protection. QEMU system mode is named
+"softmmu" precisely because it implements this in software, including a TLB
+(Translation lookaside buffer), for the guest virtual machine.
+
+QEMU user-mode does not implement a full software MMU, but "simply" translates
+virtual addresses by adding a specific offset, and relying on host MMU/OS
+instead.
+
+Monitor / QMP / HMP
+-------------------
+
+`QEMU Monitor <QEMU monitor>` is a text interface which can be used to interact
+with a running virtual machine.
+
+QMP stands for QEMU Monitor Protocol and is a json based interface.
+HMP stands for Human Monitor Protocol and is a set of text commands available
+for users who prefer natural language to json.
+
+MTTCG
+-----
+
+Multiple cpus support was first implemented using a round-robin algorithm
+running on a single thread. Later on, `Multi-threaded TCG <mttcg>` was developed
+to benefit from multiple cores to speed up execution.
+
+Plugins
+-------
+
+`TCG Plugins <TCG Plugins>` is an API used to instrument guest code, in system
+and user mode. The end goal is to have a similar set of functionality compared
+to `DynamoRIO <https://dynamorio.org/>`_ or `valgrind <https://valgrind.org/>`_.
+
+One key advantage of QEMU plugins is that they can be used to perform
+architecture agnostic instrumentation.
+
+Patchwork
+---------
+
+`Patchwork <https://patchew.org/QEMU/>`_ is a website that tracks
+patches on the Mailing List.
+
+PR
+--
+
+Once a series is reviewed and accepted by a subsystem maintainer, it will be
+included in a PR (Pull Request) that the project maintainer will merge into QEMU
+main branch, after running tests.
+
+QCOW
+----
+
+QEMU Copy On Write is a disk format developed by QEMU. It provides transparent
+compression, automatic extension, and many other advantages over a raw image.
+
+QEMU
+----
+
+`QEMU (Quick Emulator) <https://www.qemu.org/>`_ is a generic and open source
+machine emulator and virtualizer.
+
+QOM
+---
+
+`QEMU Object Model <qom>` is an object oriented API used to define various
+devices and hardware in the QEMU codebase.
+
+Record/replay
+-------------
+
+`Record/replay <replay>` is a feature of QEMU allowing to have a deterministic
+and reproducible execution of a virtual machine.
+
+Rust
+----
+
+`A new programming language <https://www.rust-lang.org/>`_, memory safe by
+default. We didn't see a more efficient way to create debates and tensions in
+a community of C programmers since the birth of C++.
+
+System mode
+-----------
+
+QEMU System mode emulates a full machine, including its cpu, memory and devices.
+It can be accelerated to hardware speed by using one of the hypervisors QEMU
+supports. It is referenced as softmmu as well.
+
+TCG
+---
+
+`Tiny Code Generator <tcg>` is an intermediate representation (IR) used to run
+guest instructions on host cpu, with both architectures possibly being
+different.
+
+It is one of the accelerator supported by QEMU, and supports a lot of
+guest/host architectures.
+
+User mode
+---------
+
+QEMU User mode allows to run programs for a guest architecture, on a host
+architecture, by translating system calls and using TCG. It is available for
+Linux and BSD.
+
+VirtIO
+------
+
+VirtIO is an open standard used to define and implement virtual devices with a
+minimal overhead, defining a set of data structures and hypercalls (similar to
+system calls, but targeting an hypervisor, which happens to be QEMU in our
+case). It's designed to be more efficient than emulating a real device, by
+minimizing the amount of interactions between a guest VM and its hypervisor.
+
+vhost-user
+----------
+
+`Vhost-user <vhost_user>` is an interface used to implement VirtIO devices
+outside of QEMU itself.
diff --git a/docs/index.rst b/docs/index.rst
index cb5e5098b65..2cad84cd77c 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -21,3 +21,4 @@ Welcome to QEMU's documentation!
    specs/index
    devel/index
    codebase/index
+   glossary/index
diff --git a/docs/system/arm/virt.rst b/docs/system/arm/virt.rst
index e67e7f0f7c5..11ceb898264 100644
--- a/docs/system/arm/virt.rst
+++ b/docs/system/arm/virt.rst
@@ -1,3 +1,5 @@
+.. _arm-virt:
+
 'virt' generic virtual platform (``virt``)
 ==========================================
 
diff --git a/docs/system/images.rst b/docs/system/images.rst
index d000bd6b6f1..a5551173c97 100644
--- a/docs/system/images.rst
+++ b/docs/system/images.rst
@@ -82,4 +82,6 @@ VM snapshots currently have the following known limitations:
 -  A few device drivers still have incomplete snapshot support so their
    state is not saved or restored properly (in particular USB).
 
+.. _block-drivers:
+
 .. include:: qemu-block-drivers.rst.inc
diff --git a/docs/tools/qemu-nbd.rst b/docs/tools/qemu-nbd.rst
index 329f44d9895..4f21b7904ac 100644
--- a/docs/tools/qemu-nbd.rst
+++ b/docs/tools/qemu-nbd.rst
@@ -1,3 +1,5 @@
+.. _qemu-nbd:
+
 =====================================
 QEMU Disk Network Block Device Server
 =====================================
-- 
2.39.5

Re: [PATCH 6/7] docs: add a glossary

[Peter Maydell]

On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
<pierrick.bouvier@linaro.org> wrote:
>
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/devel/control-flow-integrity.rst |   2 +
>  docs/devel/multi-thread-tcg.rst       |   2 +
>  docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
>  docs/index.rst                        |   1 +
>  docs/system/arm/virt.rst              |   2 +
>  docs/system/images.rst                |   2 +
>  docs/tools/qemu-nbd.rst               |   2 +
>  7 files changed, 249 insertions(+)
>  create mode 100644 docs/glossary/index.rst

I think this is a good idea; we've had at least one bug
report from a user pointing out that we had a term in
our docs which we didn't define ("block driver"):
https://gitlab.com/qemu-project/qemu/-/issues/2611
I have some comments on specific entries below.

> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
> index e6b73a4fe1a..3d5702fa4cc 100644
> --- a/docs/devel/control-flow-integrity.rst
> +++ b/docs/devel/control-flow-integrity.rst
> @@ -1,3 +1,5 @@
> +.. _cfi:
> +
>  ============================
>  Control-Flow Integrity (CFI)
>  ============================
> diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
> index d706c27ea74..7fd0a07633d 100644
> --- a/docs/devel/multi-thread-tcg.rst
> +++ b/docs/devel/multi-thread-tcg.rst
> @@ -4,6 +4,8 @@
>    This work is licensed under the terms of the GNU GPL, version 2 or
>    later. See the COPYING file in the top-level directory.
>
> +.. _mttcg:
> +
>  ==================
>  Multi-threaded TCG
>  ==================
> diff --git a/docs/glossary/index.rst b/docs/glossary/index.rst
> new file mode 100644
> index 00000000000..a2d4f3eae16
> --- /dev/null
> +++ b/docs/glossary/index.rst

I guess it makes sense to give this its own subdir, since we want
it to come at the end of the manual. The other option would be
to put it directly into docs/.

> @@ -0,0 +1,238 @@
> +.. _Glossary:
> +
> +--------
> +Glossary
> +--------
> +
> +This section of the manual presents *simply* acronyms and terms QEMU developers
> +use.

What's "simply" intended to mean here?

> +
> +Accelerator
> +-----------
> +
> +A specific API used to accelerate execution of guest instructions. It can be
> +hardware-based, through a virtualization API provided by the host OS (kvm, hvf,
> +whpx, ...) or software-based (tcg). See this description of `supported

Comma after ')'.

> +accelerators<Accelerators>`.
> +
> +Board
> +-----

I think the correct term here is "machine" -- that's what the
command line option is named, it's what the QOM class is, etc.
So the major glossary entry should be "Machine". Some people
(including me!) and some of the documentation uses "board" as a
synonym for "machine", so we should have a glossary entry for
"board", but it should just say "Another name for 'machine'" and
xref to the "machine" entry.

> +
> +QEMU system defines board models for various architectures. It's a description
> +of a SoC (system-on-chip) with various devices pre-configured, and can be
> +selected with the option ``-machine`` of qemu-system.

SoCs are not the same as boards.

We could say something like:

QEMU's system emulation models many different types of hardware.
A machine model (sometimes called a board model) is the model
of a complete virtual system with RAM, one or more CPUs, and
various devices.

We could also put in a link to
https://www.qemu.org/docs/master/system/targets.html
which is where we document what our machine types are.

> +For virtual machines, you'll use ``virt`` board model, designed for this use
> +case. As an example, for Arm architecture, you can find the `model code
> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/arm/virt.c>`_ and
> +associated `documentation <arm-virt>`.

I think I would delete this paragraph. 'virt' is only the
board type for virtual machines for some architectures; on
x86 it doesn't exist for, example. Our user facing
docs (that link above) are where we should suggest what
the best machine type to use is. And the codebase-guide
page is where we would say where machine type source code is.

> +
> +Block
> +-----
> +
> +Block drivers are the available `disk formats <block-drivers>` available, and
> +block devices `(see Block device section on options page)<sec_005finvocation>`
> +are using them to implement disks for a virtual machine.

Block drivers aren't just disk formats; there are some filter
drivers too. Somebody on the block side could probably
provide a better definition here.

> +
> +CFI
> +---
> +
> +Control Flow Integrity is a hardening technique used to prevent exploits
> +targeting QEMU by detecting unexpected branches during execution. QEMU `actively
> +supports<cfi>` being compiled with CFI enabled.
> +
> +Device
> +------
> +
> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
> +including many devices. When QEMU runs a virtual machine using a hardware-based
> +accelerator, it is responsible for emulating, using software, all devices.

This definition doesn't actually define what a device is :-)

> +
> +EDK2
> +----
> +
> +EDK2, as known as `TianoCore <https://www.tianocore.org/>`_, is an open source
> +implementation of UEFI standard. It's ran by QEMU to support UEFI for virtual
> +machines.

Replace last sentence with
"QEMU virtual machines that boot a UEFI BIOS usually use EDK2."
?

> +
> +gdbstub
> +-------
> +
> +QEMU implements a `gdb server <GDB usage>`, allowing gdb to attach to it and
> +debug a running virtual machine, or a program in user-mode. This allows to debug
> +a given architecture without having access to hardware.

"allows debugging the guest code that is running inside QEMU."

> +
> +glib2
> +-----
> +
> +`GLib2 <https://docs.gtk.org/glib/>`_ is one of the most important library we

"libraries"

> +are using through the codebase. It provides many data structures, macros, string
> +and thread utilities and portable functions across different OS. It's required
> +to build QEMU.
> +
> +Guest agent
> +-----------
> +
> +`QEMU Guest agent <qemu-ga>` is a daemon intended to be executed by guest

"The QEMU Guest Agent"

"intended to be run within virtual machines. It provides various services"

> +virtual machines and providing various services to help QEMU to interact with
> +it.
> +
> +Guest/Host
> +----------

Make these two separate glossary entries, which cross reference each other.

> +
> +Guest is the architecture of the virtual machine, which is emulated.

"Sometimes this is called the 'target' architecture, but that term
can be ambiguous."

> +Host is the architecture on which QEMU is running on, which is native.


We could also have an entry for Target

 The term "target" can be ambiguous. In most places in QEMU it is used
 as a synonym for "guest"; for example the code for emulating Arm CPUs
 is in ``target/arm/``. However in the TCG subsystem "target" refers
 to the architecture which QEMU is running on, i.e. the "host".


> +
> +Hypervisor
> +----------
> +
> +The formal definition of an hypervisor is a program than can be used to manage a
> +virtual machine. QEMU itself is an hypervisor.

"a hypervisor". QEMU isn't really a hypervisor, though...


> +
> +In the context of QEMU, an hypervisor is an API, provided by the Host OS,
> +allowing to execute virtual machines. Linux implementation is KVM (and supports
> +Xen as well). For MacOS, it's HVF. Windows defines WHPX. And NetBSD provides
> +NVMM.
> +
> +Migration
> +---------
> +
> +QEMU can save and restore the execution of a virtual machine, including across
> +different machines. This is provided by the `Migration framework<migration>`.

"between different host systems".

> +
> +NBD
> +---
> +
> +`QEMU Network Block Device server <qemu-nbd>` is a tool that can be used to

"The QEMU ..."

> +mount and access QEMU images, providing functionality similar to a loop device.
> +
> +Mailing List
> +------------
> +
> +This is `where <https://wiki.qemu.org/Contribute/MailingLists>`_ all the
> +development happens! Changes are posted as series, that all developers can
> +review and share feedback for.
> +
> +For reporting issues, our `GitLab
> +<https://gitlab.com/qemu-project/qemu/-/issues>`_ tracker is the best place.
> +
> +MMU / softmmu
> +-------------
> +
> +The Memory Management Unit is responsible for translating virtual addresses to
> +physical addresses and managing memory protection. QEMU system mode is named
> +"softmmu" precisely because it implements this in software, including a TLB
> +(Translation lookaside buffer), for the guest virtual machine.
> +
> +QEMU user-mode does not implement a full software MMU, but "simply" translates
> +virtual addresses by adding a specific offset, and relying on host MMU/OS
> +instead.
> +
> +Monitor / QMP / HMP
> +-------------------
> +
> +`QEMU Monitor <QEMU monitor>` is a text interface which can be used to interact

"The QEMU Monitor"

> +with a running virtual machine.
> +
> +QMP stands for QEMU Monitor Protocol and is a json based interface.
> +HMP stands for Human Monitor Protocol and is a set of text commands available
> +for users who prefer natural language to json.
> +
> +MTTCG
> +-----
> +
> +Multiple cpus support was first implemented using a round-robin algorithm

"Multiple CPU support"

> +running on a single thread. Later on, `Multi-threaded TCG <mttcg>` was developed
> +to benefit from multiple cores to speed up execution.
> +
> +Plugins
> +-------
> +
> +`TCG Plugins <TCG Plugins>` is an API used to instrument guest code, in system
> +and user mode. The end goal is to have a similar set of functionality compared
> +to `DynamoRIO <https://dynamorio.org/>`_ or `valgrind <https://valgrind.org/>`_.
> +
> +One key advantage of QEMU plugins is that they can be used to perform
> +architecture agnostic instrumentation.
> +
> +Patchwork
> +---------
> +
> +`Patchwork <https://patchew.org/QEMU/>`_ is a website that tracks
> +patches on the Mailing List.

Patchwork and patchew are different systems. Patchew's URL is
https://patchew.org/QEMU/

(There is a patchwork instance that tracks qemu-devel patches,
at https://patchwork.kernel.org/project/qemu-devel/list/ , but
I'm not aware of any developers that are actively using it, so
I don't think it merits being mentioned in the glossary.)

> +
> +PR
> +--
> +
> +Once a series is reviewed and accepted by a subsystem maintainer, it will be
> +included in a PR (Pull Request) that the project maintainer will merge into QEMU
> +main branch, after running tests.

I think we could probably also usefully say

"The QEMU project doesn't currently expect most developers to
directly submit pull requests."

just to flag up that our development model isn't like the
currently-popular github/gitlab one where a PR is how you
send contributions.

> +
> +QCOW
> +----
> +
> +QEMU Copy On Write is a disk format developed by QEMU. It provides transparent
> +compression, automatic extension, and many other advantages over a raw image.

We want to be a bit careful here, because the "qcow" format
is not something we recommend for new use -- "qcow2" is what
you actually want.

https://www.qemu.org/docs/master/system/qemu-block-drivers.html#cmdoption-image-formats-arg-qcow2

> +
> +QEMU
> +----
> +
> +`QEMU (Quick Emulator) <https://www.qemu.org/>`_ is a generic and open source
> +machine emulator and virtualizer.
> +
> +QOM
> +---
> +
> +`QEMU Object Model <qom>` is an object oriented API used to define various
> +devices and hardware in the QEMU codebase.
> +
> +Record/replay
> +-------------
> +
> +`Record/replay <replay>` is a feature of QEMU allowing to have a deterministic
> +and reproducible execution of a virtual machine.
> +
> +Rust
> +----
> +
> +`A new programming language <https://www.rust-lang.org/>`_, memory safe by
> +default. We didn't see a more efficient way to create debates and tensions in
> +a community of C programmers since the birth of C++.

:-)  but I think we should probably avoid the joke in our docs.

> +
> +System mode
> +-----------
> +
> +QEMU System mode emulates a full machine, including its cpu, memory and devices.
> +It can be accelerated to hardware speed by using one of the hypervisors QEMU
> +supports. It is referenced as softmmu as well.

https://www.qemu.org/docs/master/about/index.html already has
text defining system emulation and user emulation, so we don't
really need to re-invent new phrasing for those here.

> +
> +TCG
> +---
> +
> +`Tiny Code Generator <tcg>` is an intermediate representation (IR) used to run
> +guest instructions on host cpu, with both architectures possibly being
> +different.

I would say

  TCG is the QEMU Tiny Code Generator; it is the JIT system we use
  to emulate a guest CPU in software.

That's enough for users to understand what it means (I hope); if
they want to know more specifics like about the intermediate
representation they can follow the link.

> +
> +It is one of the accelerator supported by QEMU, and supports a lot of

"accelerators"

> +guest/host architectures.
> +
> +User mode
> +---------
> +
> +QEMU User mode allows to run programs for a guest architecture, on a host
> +architecture, by translating system calls and using TCG. It is available for
> +Linux and BSD.
> +
> +VirtIO
> +------
> +
> +VirtIO is an open standard used to define and implement virtual devices with a
> +minimal overhead, defining a set of data structures and hypercalls (similar to
> +system calls, but targeting an hypervisor, which happens to be QEMU in our
> +case). It's designed to be more efficient than emulating a real device, by
> +minimizing the amount of interactions between a guest VM and its hypervisor.
> +
> +vhost-user
> +----------
> +
> +`Vhost-user <vhost_user>` is an interface used to implement VirtIO devices
> +outside of QEMU itself.

thanks
-- PMM

Re: [PATCH 6/7] docs: add a glossary

[Alex Bennée]

Peter Maydell <peter.maydell@linaro.org> writes:

> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>  docs/devel/control-flow-integrity.rst |   2 +
>>  docs/devel/multi-thread-tcg.rst       |   2 +
>>  docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
>>  docs/index.rst                        |   1 +
>>  docs/system/arm/virt.rst              |   2 +
>>  docs/system/images.rst                |   2 +
>>  docs/tools/qemu-nbd.rst               |   2 +
>>  7 files changed, 249 insertions(+)
>>  create mode 100644 docs/glossary/index.rst
>
> I think this is a good idea; we've had at least one bug
> report from a user pointing out that we had a term in
> our docs which we didn't define ("block driver"):
> https://gitlab.com/qemu-project/qemu/-/issues/2611
> I have some comments on specific entries below.
>
>> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
>> index e6b73a4fe1a..3d5702fa4cc 100644
>> --- a/docs/devel/control-flow-integrity.rst
<snip>
>> +
>> +Device
>> +------
>> +
>> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
>> +including many devices. When QEMU runs a virtual machine using a hardware-based
>> +accelerator, it is responsible for emulating, using software, all devices.
>
> This definition doesn't actually define what a device is :-)

Also we can xref to:

  https://qemu.readthedocs.io/en/v9.1.0/system/device-emulation.html

where we go into a bit more detail about what a device, bus, frontend
and backend are.

>
>> +
>> +EDK2
>> +----
>> +
>> +EDK2, as known as `TianoCore <https://www.tianocore.org/>`_, is an open source
>> +implementation of UEFI standard. It's ran by QEMU to support UEFI for virtual
>> +machines.
>
> Replace last sentence with
> "QEMU virtual machines that boot a UEFI BIOS usually use EDK2."
> ?
>
<snip>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH 6/7] docs: add a glossary

[Pierrick Bouvier]

On 12/3/24 10:10, Alex Bennée wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> 
>> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
>> <pierrick.bouvier@linaro.org> wrote:
>>>
>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>> ---
>>>   docs/devel/control-flow-integrity.rst |   2 +
>>>   docs/devel/multi-thread-tcg.rst       |   2 +
>>>   docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
>>>   docs/index.rst                        |   1 +
>>>   docs/system/arm/virt.rst              |   2 +
>>>   docs/system/images.rst                |   2 +
>>>   docs/tools/qemu-nbd.rst               |   2 +
>>>   7 files changed, 249 insertions(+)
>>>   create mode 100644 docs/glossary/index.rst
>>
>> I think this is a good idea; we've had at least one bug
>> report from a user pointing out that we had a term in
>> our docs which we didn't define ("block driver"):
>> https://gitlab.com/qemu-project/qemu/-/issues/2611
>> I have some comments on specific entries below.
>>
>>> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
>>> index e6b73a4fe1a..3d5702fa4cc 100644
>>> --- a/docs/devel/control-flow-integrity.rst
> <snip>
>>> +
>>> +Device
>>> +------
>>> +
>>> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
>>> +including many devices. When QEMU runs a virtual machine using a hardware-based
>>> +accelerator, it is responsible for emulating, using software, all devices.
>>
>> This definition doesn't actually define what a device is :-)
> 
> Also we can xref to:
> 
>    https://qemu.readthedocs.io/en/v9.1.0/system/device-emulation.html
> 
> where we go into a bit more detail about what a device, bus, frontend
> and backend are.
> 

Good point, I'll add it.

>>
>>> +
>>> +EDK2
>>> +----
>>> +
>>> +EDK2, as known as `TianoCore <https://www.tianocore.org/>`_, is an open source
>>> +implementation of UEFI standard. It's ran by QEMU to support UEFI for virtual
>>> +machines.
>>
>> Replace last sentence with
>> "QEMU virtual machines that boot a UEFI BIOS usually use EDK2."
>> ?
>>
> <snip>
> 

Re: [PATCH 6/7] docs: add a glossary

[Pierrick Bouvier]

On 12/3/24 09:37, Peter Maydell wrote:
> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   docs/devel/control-flow-integrity.rst |   2 +
>>   docs/devel/multi-thread-tcg.rst       |   2 +
>>   docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
>>   docs/index.rst                        |   1 +
>>   docs/system/arm/virt.rst              |   2 +
>>   docs/system/images.rst                |   2 +
>>   docs/tools/qemu-nbd.rst               |   2 +
>>   7 files changed, 249 insertions(+)
>>   create mode 100644 docs/glossary/index.rst
> 
> I think this is a good idea; we've had at least one bug
> report from a user pointing out that we had a term in
> our docs which we didn't define ("block driver"):
> https://gitlab.com/qemu-project/qemu/-/issues/2611
> I have some comments on specific entries below.
> 

And people can be free to add new entries later. However, we should 
resist the temptation to add too many details. It should stay simple and 
understandable, even if not all technical nuances are not represented.

>> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
>> index e6b73a4fe1a..3d5702fa4cc 100644
>> --- a/docs/devel/control-flow-integrity.rst
>> +++ b/docs/devel/control-flow-integrity.rst
>> @@ -1,3 +1,5 @@
>> +.. _cfi:
>> +
>>   ============================
>>   Control-Flow Integrity (CFI)
>>   ============================
>> diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
>> index d706c27ea74..7fd0a07633d 100644
>> --- a/docs/devel/multi-thread-tcg.rst
>> +++ b/docs/devel/multi-thread-tcg.rst
>> @@ -4,6 +4,8 @@
>>     This work is licensed under the terms of the GNU GPL, version 2 or
>>     later. See the COPYING file in the top-level directory.
>>
>> +.. _mttcg:
>> +
>>   ==================
>>   Multi-threaded TCG
>>   ==================
>> diff --git a/docs/glossary/index.rst b/docs/glossary/index.rst
>> new file mode 100644
>> index 00000000000..a2d4f3eae16
>> --- /dev/null
>> +++ b/docs/glossary/index.rst
> 
> I guess it makes sense to give this its own subdir, since we want
> it to come at the end of the manual. The other option would be
> to put it directly into docs/.
> 

 From your comment, it's not clear for me if it's ok as it is, or if you 
want a change.
Can you elaborate on that?

>> @@ -0,0 +1,238 @@
>> +.. _Glossary:
>> +
>> +--------
>> +Glossary
>> +--------
>> +
>> +This section of the manual presents *simply* acronyms and terms QEMU developers
>> +use.
> 
> What's "simply" intended to mean here?
> 

"in a straightforward or plain manner".
I can remove this word if you think it does not serve any purpose.

>> +
>> +Accelerator
>> +-----------
>> +
>> +A specific API used to accelerate execution of guest instructions. It can be
>> +hardware-based, through a virtualization API provided by the host OS (kvm, hvf,
>> +whpx, ...) or software-based (tcg). See this description of `supported
> 
> Comma after ')'.
> 

Thanks.

>> +accelerators<Accelerators>`.
>> +
>> +Board
>> +-----
> 
> I think the correct term here is "machine" -- that's what the
> command line option is named, it's what the QOM class is, etc.
> So the major glossary entry should be "Machine". Some people
> (including me!) and some of the documentation uses "board" as a
> synonym for "machine", so we should have a glossary entry for
> "board", but it should just say "Another name for 'machine'" and
> xref to the "machine" entry.
> 

It's a good point. I thought the same when I wrote it (and finally chose 
Board). I'll rename it to machine and add the board entry to point to it.

>> +
>> +QEMU system defines board models for various architectures. It's a description
>> +of a SoC (system-on-chip) with various devices pre-configured, and can be
>> +selected with the option ``-machine`` of qemu-system.
> 
> SoCs are not the same as boards.
> 
> We could say something like:
> 
> QEMU's system emulation models many different types of hardware.
> A machine model (sometimes called a board model) is the model
> of a complete virtual system with RAM, one or more CPUs, and
> various devices.
> 
> We could also put in a link to
> https://www.qemu.org/docs/master/system/targets.html
> which is where we document what our machine types are.
> 

How do you distinguish a SoC and cpu? Is a SoC cpu + devices?
Isn't a board/machine a set of SoC + devices attached?

The definition does not say a board is a SoC, but maybe the wording is 
confusing.

>> +For virtual machines, you'll use ``virt`` board model, designed for this use
>> +case. As an example, for Arm architecture, you can find the `model code
>> +<https://gitlab.com/qemu-project/qemu/-/blob/master/hw/arm/virt.c>`_ and
>> +associated `documentation <arm-virt>`.
> 
> I think I would delete this paragraph. 'virt' is only the
> board type for virtual machines for some architectures; on
> x86 it doesn't exist for, example. Our user facing
> docs (that link above) are where we should suggest what
> the best machine type to use is. And the codebase-guide
> page is where we would say where machine type source code is.
> 

Ok.

>> +
>> +Block
>> +-----
>> +
>> +Block drivers are the available `disk formats <block-drivers>` available, and
>> +block devices `(see Block device section on options page)<sec_005finvocation>`
>> +are using them to implement disks for a virtual machine.
> 
> Block drivers aren't just disk formats; there are some filter
> drivers too. Somebody on the block side could probably
> provide a better definition here.
> 

I'm open to a more exact definition. The two terms (drivers and devices) 
seem to overlap on some parts, so I came up with this trivial definition.

>> +
>> +CFI
>> +---
>> +
>> +Control Flow Integrity is a hardening technique used to prevent exploits
>> +targeting QEMU by detecting unexpected branches during execution. QEMU `actively
>> +supports<cfi>` being compiled with CFI enabled.
>> +
>> +Device
>> +------
>> +
>> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
>> +including many devices. When QEMU runs a virtual machine using a hardware-based
>> +accelerator, it is responsible for emulating, using software, all devices.
> 
> This definition doesn't actually define what a device is :-)
> 

Indeed :)
Should we explain what is a computer device?
The goal was just to say that QEMU can emulate hardware interacting with 
the cpu, which could be a possible definition. So people can associate 
that QEMU devices are nothing else than a "normal" computer device.

>> +
>> +EDK2
>> +----
>> +
>> +EDK2, as known as `TianoCore <https://www.tianocore.org/>`_, is an open source
>> +implementation of UEFI standard. It's ran by QEMU to support UEFI for virtual
>> +machines.
> 
> Replace last sentence with
> "QEMU virtual machines that boot a UEFI BIOS usually use EDK2."
> ?
> 
>> +
>> +gdbstub
>> +-------
>> +
>> +QEMU implements a `gdb server <GDB usage>`, allowing gdb to attach to it and
>> +debug a running virtual machine, or a program in user-mode. This allows to debug
>> +a given architecture without having access to hardware.
> 
> "allows debugging the guest code that is running inside QEMU."
> 
>> +
>> +glib2
>> +-----
>> +
>> +`GLib2 <https://docs.gtk.org/glib/>`_ is one of the most important library we
> 
> "libraries"
> 
>> +are using through the codebase. It provides many data structures, macros, string
>> +and thread utilities and portable functions across different OS. It's required
>> +to build QEMU.
>> +
>> +Guest agent
>> +-----------
>> +
>> +`QEMU Guest agent <qemu-ga>` is a daemon intended to be executed by guest
> 
> "The QEMU Guest Agent"
> 
> "intended to be run within virtual machines. It provides various services"
> 
>> +virtual machines and providing various services to help QEMU to interact with
>> +it.
>> +
>> +Guest/Host
>> +----------
> 
> Make these two separate glossary entries, which cross reference each other.
> 
>> +
>> +Guest is the architecture of the virtual machine, which is emulated.
> 
> "Sometimes this is called the 'target' architecture, but that term
> can be ambiguous."
> 
>> +Host is the architecture on which QEMU is running on, which is native.
> 
> 
> We could also have an entry for Target
> 
>   The term "target" can be ambiguous. In most places in QEMU it is used
>   as a synonym for "guest"; for example the code for emulating Arm CPUs
>   is in ``target/arm/``. However in the TCG subsystem "target" refers
>   to the architecture which QEMU is running on, i.e. the "host".
> 

It's a good point, and a very confusing one.
I'll add it and a link to docs/devel/tcg-ops.rst, that clarifies this 
for TCG.

> 
>> +
>> +Hypervisor
>> +----------
>> +
>> +The formal definition of an hypervisor is a program than can be used to manage a
>> +virtual machine. QEMU itself is an hypervisor.
> 
> "a hypervisor". QEMU isn't really a hypervisor, though...
> 

It's a shortcut, and I'm open to change it. It brings an interesting 
question though.

Technically, QEMU interacts with hypervisor APIs built in various OSes. 
On the other hand, when we use TCG, it's an emulator instead.

But as you can't use KVM/hvf/whpx by itself, how do you name the program 
interacting with it, and emulating the rest of the VM?

The correct word is probably "virtualizer", but from searching on 
Internet, it seems that "vmm" and "virtualizer" are considered the same 
as an "hypervisor". The difference is subtle, and maybe we have an 
opportunity here to clarify it.

> 
>> +
>> +In the context of QEMU, an hypervisor is an API, provided by the Host OS,
>> +allowing to execute virtual machines. Linux implementation is KVM (and supports
>> +Xen as well). For MacOS, it's HVF. Windows defines WHPX. And NetBSD provides
>> +NVMM.
>> +
>> +Migration
>> +---------
>> +
>> +QEMU can save and restore the execution of a virtual machine, including across
>> +different machines. This is provided by the `Migration framework<migration>`.
> 
> "between different host systems".
> 
>> +
>> +NBD
>> +---
>> +
>> +`QEMU Network Block Device server <qemu-nbd>` is a tool that can be used to
> 
> "The QEMU ..."
> 
>> +mount and access QEMU images, providing functionality similar to a loop device.
>> +
>> +Mailing List
>> +------------
>> +
>> +This is `where <https://wiki.qemu.org/Contribute/MailingLists>`_ all the
>> +development happens! Changes are posted as series, that all developers can
>> +review and share feedback for.
>> +
>> +For reporting issues, our `GitLab
>> +<https://gitlab.com/qemu-project/qemu/-/issues>`_ tracker is the best place.
>> +
>> +MMU / softmmu
>> +-------------
>> +
>> +The Memory Management Unit is responsible for translating virtual addresses to
>> +physical addresses and managing memory protection. QEMU system mode is named
>> +"softmmu" precisely because it implements this in software, including a TLB
>> +(Translation lookaside buffer), for the guest virtual machine.
>> +
>> +QEMU user-mode does not implement a full software MMU, but "simply" translates
>> +virtual addresses by adding a specific offset, and relying on host MMU/OS
>> +instead.
>> +
>> +Monitor / QMP / HMP
>> +-------------------
>> +
>> +`QEMU Monitor <QEMU monitor>` is a text interface which can be used to interact
> 
> "The QEMU Monitor"
> 
>> +with a running virtual machine.
>> +
>> +QMP stands for QEMU Monitor Protocol and is a json based interface.
>> +HMP stands for Human Monitor Protocol and is a set of text commands available
>> +for users who prefer natural language to json.
>> +
>> +MTTCG
>> +-----
>> +
>> +Multiple cpus support was first implemented using a round-robin algorithm
> 
> "Multiple CPU support"
> 
>> +running on a single thread. Later on, `Multi-threaded TCG <mttcg>` was developed
>> +to benefit from multiple cores to speed up execution.
>> +
>> +Plugins
>> +-------
>> +
>> +`TCG Plugins <TCG Plugins>` is an API used to instrument guest code, in system
>> +and user mode. The end goal is to have a similar set of functionality compared
>> +to `DynamoRIO <https://dynamorio.org/>`_ or `valgrind <https://valgrind.org/>`_.
>> +
>> +One key advantage of QEMU plugins is that they can be used to perform
>> +architecture agnostic instrumentation.
>> +
>> +Patchwork
>> +---------
>> +
>> +`Patchwork <https://patchew.org/QEMU/>`_ is a website that tracks
>> +patches on the Mailing List.
> 
> Patchwork and patchew are different systems. Patchew's URL is
> https://patchew.org/QEMU/
> 
> (There is a patchwork instance that tracks qemu-devel patches,
> at https://patchwork.kernel.org/project/qemu-devel/list/ , but
> I'm not aware of any developers that are actively using it, so
> I don't think it merits being mentioned in the glossary.)
> 

I've been confused by that, and just thought it was two different 
instances (fork me if you can) of the "same" thing.
How would you define patchew?
When we say patchwork, do we implicitely mean patchew?

if I understand currently, patchew is what we want to mention in our 
doc? (and mention it's not associated to patchwork).

>> +
>> +PR
>> +--
>> +
>> +Once a series is reviewed and accepted by a subsystem maintainer, it will be
>> +included in a PR (Pull Request) that the project maintainer will merge into QEMU
>> +main branch, after running tests.
> 
> I think we could probably also usefully say
> 
> "The QEMU project doesn't currently expect most developers to
> directly submit pull requests."
> 
> just to flag up that our development model isn't like the
> currently-popular github/gitlab one where a PR is how you
> send contributions.
> 

This is interesting.

For the majority of developers nowadays, a PR is a GitHub/GitLab PR.
Despite the fact we use the original PR meaning (in git terms), it's 
probably confusing when new comers hear pull request.

>> +
>> +QCOW
>> +----
>> +
>> +QEMU Copy On Write is a disk format developed by QEMU. It provides transparent
>> +compression, automatic extension, and many other advantages over a raw image.
> 
> We want to be a bit careful here, because the "qcow" format
> is not something we recommend for new use -- "qcow2" is what
> you actually want.
> 
> https://www.qemu.org/docs/master/system/qemu-block-drivers.html#cmdoption-image-formats-arg-qcow2
> 

Sounds good.

For my personal knowledge: during this work I discovered that we had 
qcow3. From what I understood, it seems to be included in what we called 
qcow2 today. Is that correct?

>> +
>> +QEMU
>> +----
>> +
>> +`QEMU (Quick Emulator) <https://www.qemu.org/>`_ is a generic and open source
>> +machine emulator and virtualizer.
>> +
>> +QOM
>> +---
>> +
>> +`QEMU Object Model <qom>` is an object oriented API used to define various
>> +devices and hardware in the QEMU codebase.
>> +
>> +Record/replay
>> +-------------
>> +
>> +`Record/replay <replay>` is a feature of QEMU allowing to have a deterministic
>> +and reproducible execution of a virtual machine.
>> +
>> +Rust
>> +----
>> +
>> +`A new programming language <https://www.rust-lang.org/>`_, memory safe by
>> +default. We didn't see a more efficient way to create debates and tensions in
>> +a community of C programmers since the birth of C++.
> 
> :-)  but I think we should probably avoid the joke in our docs.
> 

I had one smile, I'm happy to remove it now :).

More seriously, I can complete after "memory safe by default", "The QEMU 
community is currently working to integrate Rust in the codebase for 
various subsystems".

>> +
>> +System mode
>> +-----------
>> +
>> +QEMU System mode emulates a full machine, including its cpu, memory and devices.
>> +It can be accelerated to hardware speed by using one of the hypervisors QEMU
>> +supports. It is referenced as softmmu as well.
> 
> https://www.qemu.org/docs/master/about/index.html already has
> text defining system emulation and user emulation, so we don't
> really need to re-invent new phrasing for those here.
> 

I can repeat the definition we have there:
"System Emulation provides a virtual model of an entire machine (CPU, 
memory and emulated devices) to run a guest OS. In this mode the CPU may 
be fully emulated, or it may work with a hypervisor such as KVM, Xen or 
Hypervisor.Framework to allow the guest to run directly on the host CPU."

However, I think mentioning softmmu is important, as it's a common 
confusing name (to the new comer) coming from target list.

>> +
>> +TCG
>> +---
>> +
>> +`Tiny Code Generator <tcg>` is an intermediate representation (IR) used to run
>> +guest instructions on host cpu, with both architectures possibly being
>> +different.
> 
> I would say
> 
>    TCG is the QEMU Tiny Code Generator; it is the JIT system we use
>    to emulate a guest CPU in software.
> 
> That's enough for users to understand what it means (I hope); if
> they want to know more specifics like about the intermediate
> representation they can follow the link.
> 

I'm ok with your definition, TCG is wider than the IR we use.

>> +
>> +It is one of the accelerator supported by QEMU, and supports a lot of
> 
> "accelerators"
> 
>> +guest/host architectures.
>> +
>> +User mode
>> +---------
>> +
>> +QEMU User mode allows to run programs for a guest architecture, on a host
>> +architecture, by translating system calls and using TCG. It is available for
>> +Linux and BSD.
>> +
>> +VirtIO
>> +------
>> +
>> +VirtIO is an open standard used to define and implement virtual devices with a
>> +minimal overhead, defining a set of data structures and hypercalls (similar to
>> +system calls, but targeting an hypervisor, which happens to be QEMU in our
>> +case). It's designed to be more efficient than emulating a real device, by
>> +minimizing the amount of interactions between a guest VM and its hypervisor.
>> +
>> +vhost-user
>> +----------
>> +
>> +`Vhost-user <vhost_user>` is an interface used to implement VirtIO devices
>> +outside of QEMU itself.
> 
> thanks
> -- PMM

Thanks for your review Peter!

Pierrick

Re: [PATCH 6/7] docs: add a glossary

[Peter Maydell]

On Tue, 3 Dec 2024 at 20:32, Pierrick Bouvier
<pierrick.bouvier@linaro.org> wrote:
>
> On 12/3/24 09:37, Peter Maydell wrote:
> > On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
> > <pierrick.bouvier@linaro.org> wrote:
> >>
> >> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> >> ---
> >>   docs/devel/control-flow-integrity.rst |   2 +
> >>   docs/devel/multi-thread-tcg.rst       |   2 +
> >>   docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
> >>   docs/index.rst                        |   1 +
> >>   docs/system/arm/virt.rst              |   2 +
> >>   docs/system/images.rst                |   2 +
> >>   docs/tools/qemu-nbd.rst               |   2 +
> >>   7 files changed, 249 insertions(+)
> >>   create mode 100644 docs/glossary/index.rst
> >
> > I think this is a good idea; we've had at least one bug
> > report from a user pointing out that we had a term in
> > our docs which we didn't define ("block driver"):
> > https://gitlab.com/qemu-project/qemu/-/issues/2611
> > I have some comments on specific entries below.
> >
>
> And people can be free to add new entries later. However, we should
> resist the temptation to add too many details. It should stay simple and
> understandable, even if not all technical nuances are not represented.
>
> >> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
> >> index e6b73a4fe1a..3d5702fa4cc 100644
> >> --- a/docs/devel/control-flow-integrity.rst
> >> +++ b/docs/devel/control-flow-integrity.rst
> >> @@ -1,3 +1,5 @@
> >> +.. _cfi:
> >> +
> >>   ============================
> >>   Control-Flow Integrity (CFI)
> >>   ============================
> >> diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
> >> index d706c27ea74..7fd0a07633d 100644
> >> --- a/docs/devel/multi-thread-tcg.rst
> >> +++ b/docs/devel/multi-thread-tcg.rst
> >> @@ -4,6 +4,8 @@
> >>     This work is licensed under the terms of the GNU GPL, version 2 or
> >>     later. See the COPYING file in the top-level directory.
> >>
> >> +.. _mttcg:
> >> +
> >>   ==================
> >>   Multi-threaded TCG
> >>   ==================
> >> diff --git a/docs/glossary/index.rst b/docs/glossary/index.rst
> >> new file mode 100644
> >> index 00000000000..a2d4f3eae16
> >> --- /dev/null
> >> +++ b/docs/glossary/index.rst
> >
> > I guess it makes sense to give this its own subdir, since we want
> > it to come at the end of the manual. The other option would be
> > to put it directly into docs/.
> >
>
>  From your comment, it's not clear for me if it's ok as it is, or if you
> want a change.
> Can you elaborate on that?

It means I'm not sure. We end up with a subdirectory with only
one file in it and where there's no expectation we'd ever want
to add any more files to it. On the other hand it does keep it
out of the docs/ top level directory, which currently has a
fair amount of cruft awaiting cleanup.

I guess on balance I would make this docs/glossary.rst,
unless you anticipate wanting to split this into multiple
files or have something else in docs/glossary/ later.

> >> @@ -0,0 +1,238 @@
> >> +.. _Glossary:
> >> +
> >> +--------
> >> +Glossary
> >> +--------
> >> +
> >> +This section of the manual presents *simply* acronyms and terms QEMU developers
> >> +use.
> >
> > What's "simply" intended to mean here?
> >
>
> "in a straightforward or plain manner".
> I can remove this word if you think it does not serve any purpose.

You could phrase it as "presents brief definitions of acronyms
and terms", I think.

> >> +QEMU system defines board models for various architectures. It's a description
> >> +of a SoC (system-on-chip) with various devices pre-configured, and can be
> >> +selected with the option ``-machine`` of qemu-system.
> >
> > SoCs are not the same as boards.
> >
> > We could say something like:
> >
> > QEMU's system emulation models many different types of hardware.
> > A machine model (sometimes called a board model) is the model
> > of a complete virtual system with RAM, one or more CPUs, and
> > various devices.
> >
> > We could also put in a link to
> > https://www.qemu.org/docs/master/system/targets.html
> > which is where we document what our machine types are.
> >
>
> How do you distinguish a SoC and cpu? Is a SoC cpu + devices?
> Isn't a board/machine a set of SoC + devices attached?

An SoC is a "system on chip". To quote wikipedia's definition:

"A system on a chip or system-on-chip is an integrated circuit that
integrates most or all components of a computer or electronic system.
These components usually include an on-chip central processing unit
(CPU), memory interfaces, input/output devices and interfaces, and
secondary storage interfaces, often alongside other components such
as radio modems and a graphics processing unit (GPU) – all on a
single substrate or microchip."

An SoC always contains a CPU, but it will have a lot more
than that built into it too. And the SoC only has "most"
of the system components, so the whole machine will be
an SoC plus some other things.

Generally a board/machine that uses an SoC will have on it:
 * an SoC
 * the actual memory
 * perhaps one or two extra devices external to the SoC
 * connectors for things like serial ports, SD cards, etc
   (which generally wire up to SoC pins)
 * a crystal or similar to act as the main clock source

So if you look at a photo of a development board that uses
an SoC, there will be one large chip which is the SoC,
some RAM chips, a bunch of connectors and one or two smaller
chips. Not every device will be inside the SoC, but
generally almost all of them are.

QEMU's machine models for this kind of board match the
organization of the hardware; looking at hw/arm/sabrelite.c
which is a machine model you can see that it has:
 * an instance of the fsl-imx6 SoC device object
 * the main memory
 * some NOR flash memory
 * some configuration and wiring up of things

And the SoC itself is in hw/arm/fsl-imx6.c, and is a
QOM device object that creates and wires up the CPUs,
UARTs, USB controller, and various other devices that
this particular SoC includes. In this case we only have
one board model using this SoC, but for some SoCs we
have several board models that all use the same SoC
but wire up different external devices to it.

Some of our machine models are models of systems that
don't use an SoC at all. This is rare in the Arm world,
but for instance the SPARC machines like the SS-5 are
like that -- the real hardware had a discrete CPU chip
and a bunch of devices in their own chips on the
motherboard, and QEMU's model of that hardware has
a machine model which directly creates the CPU and
the various devices. (And some of our older machine
models are models of hardware that *does* have an SoC
but where we didn't model that level of abstraction,
so they directly create devices in the machine model
that really ought to be inside an SoC object.
hw/arm/stellaris.c is one example of that.)

> >> +
> >> +Block
> >> +-----
> >> +
> >> +Block drivers are the available `disk formats <block-drivers>` available, and
> >> +block devices `(see Block device section on options page)<sec_005finvocation>`
> >> +are using them to implement disks for a virtual machine.
> >
> > Block drivers aren't just disk formats; there are some filter
> > drivers too. Somebody on the block side could probably
> > provide a better definition here.
> >
>
> I'm open to a more exact definition. The two terms (drivers and devices)
> seem to overlap on some parts, so I came up with this trivial definition.

Yeah, the driver vs device split is a good one (basically
the device is the front-end visible to the guest, and the
driver is the back-end that provides it with storage
via an abstracted API). The nit I'm picking here is that
not every block driver is there to provide support for
an on-host disk format.

> >> +
> >> +CFI
> >> +---
> >> +
> >> +Control Flow Integrity is a hardening technique used to prevent exploits
> >> +targeting QEMU by detecting unexpected branches during execution. QEMU `actively
> >> +supports<cfi>` being compiled with CFI enabled.
> >> +
> >> +Device
> >> +------
> >> +
> >> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
> >> +including many devices. When QEMU runs a virtual machine using a hardware-based
> >> +accelerator, it is responsible for emulating, using software, all devices.
> >
> > This definition doesn't actually define what a device is :-)
> >
>
> Indeed :)
> Should we explain what is a computer device?
> The goal was just to say that QEMU can emulate hardware interacting with
> the cpu, which could be a possible definition. So people can associate
> that QEMU devices are nothing else than a "normal" computer device.

We could say, perhaps:

In QEMU, a device is a piece of hardware visible to the guest.
Examples include UARTs, PCI controllers, PCI cards, VGA controllers,
and many more.



> >> +
> >> +Hypervisor
> >> +----------
> >> +
> >> +The formal definition of an hypervisor is a program than can be used to manage a
> >> +virtual machine. QEMU itself is an hypervisor.
> >
> > "a hypervisor". QEMU isn't really a hypervisor, though...
> >
>
> It's a shortcut, and I'm open to change it. It brings an interesting
> question though.
>
> Technically, QEMU interacts with hypervisor APIs built in various OSes.
> On the other hand, when we use TCG, it's an emulator instead.
>
> But as you can't use KVM/hvf/whpx by itself, how do you name the program
> interacting with it, and emulating the rest of the VM?
>
> The correct word is probably "virtualizer", but from searching on
> Internet, it seems that "vmm" and "virtualizer" are considered the same
> as an "hypervisor". The difference is subtle, and maybe we have an
> opportunity here to clarify it.


> >> +Patchwork
> >> +---------
> >> +
> >> +`Patchwork <https://patchew.org/QEMU/>`_ is a website that tracks
> >> +patches on the Mailing List.
> >
> > Patchwork and patchew are different systems. Patchew's URL is
> > https://patchew.org/QEMU/
> >
> > (There is a patchwork instance that tracks qemu-devel patches,
> > at https://patchwork.kernel.org/project/qemu-devel/list/ , but
> > I'm not aware of any developers that are actively using it, so
> > I don't think it merits being mentioned in the glossary.)
> >
>
> I've been confused by that, and just thought it was two different
> instances (fork me if you can) of the "same" thing.
> How would you define patchew?
> When we say patchwork, do we implicitely mean patchew?

No. patchwork is patchwork, and patchew is patchew -- these
are entirely different pieces of software that happen to do
similar jobs.

> if I understand currently, patchew is what we want to mention in our
> doc? (and mention it's not associated to patchwork).

We don't use patchwork, so we don't need to mention it anywhere.

> >> +Once a series is reviewed and accepted by a subsystem maintainer, it will be
> >> +included in a PR (Pull Request) that the project maintainer will merge into QEMU
> >> +main branch, after running tests.
> >
> > I think we could probably also usefully say
> >
> > "The QEMU project doesn't currently expect most developers to
> > directly submit pull requests."
> >
> > just to flag up that our development model isn't like the
> > currently-popular github/gitlab one where a PR is how you
> > send contributions.
> >
>
> This is interesting.
>
> For the majority of developers nowadays, a PR is a GitHub/GitLab PR.
> Despite the fact we use the original PR meaning (in git terms), it's
> probably confusing when new comers hear pull request.
>
> >> +
> >> +QCOW
> >> +----
> >> +
> >> +QEMU Copy On Write is a disk format developed by QEMU. It provides transparent
> >> +compression, automatic extension, and many other advantages over a raw image.
> >
> > We want to be a bit careful here, because the "qcow" format
> > is not something we recommend for new use -- "qcow2" is what
> > you actually want.
> >
> > https://www.qemu.org/docs/master/system/qemu-block-drivers.html#cmdoption-image-formats-arg-qcow2
> >
>
> Sounds good.
>
> For my personal knowledge: during this work I discovered that we had
> qcow3. From what I understood, it seems to be included in what we called
> qcow2 today. Is that correct?

I have no idea -- you'd need to ask somebody who works on the
block layer

thanks
-- PMM

Re: [PATCH 6/7] docs: add a glossary

[Pierrick Bouvier]

On 12/5/24 07:23, Peter Maydell wrote:
> On Tue, 3 Dec 2024 at 20:32, Pierrick Bouvier
> <pierrick.bouvier@linaro.org> wrote:
>>
>> On 12/3/24 09:37, Peter Maydell wrote:
>>> On Mon, 18 Nov 2024 at 17:24, Pierrick Bouvier
>>> <pierrick.bouvier@linaro.org> wrote:
>>>>
>>>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>>>> ---
>>>>    docs/devel/control-flow-integrity.rst |   2 +
>>>>    docs/devel/multi-thread-tcg.rst       |   2 +
>>>>    docs/glossary/index.rst               | 238 ++++++++++++++++++++++++++
>>>>    docs/index.rst                        |   1 +
>>>>    docs/system/arm/virt.rst              |   2 +
>>>>    docs/system/images.rst                |   2 +
>>>>    docs/tools/qemu-nbd.rst               |   2 +
>>>>    7 files changed, 249 insertions(+)
>>>>    create mode 100644 docs/glossary/index.rst
>>>
>>> I think this is a good idea; we've had at least one bug
>>> report from a user pointing out that we had a term in
>>> our docs which we didn't define ("block driver"):
>>> https://gitlab.com/qemu-project/qemu/-/issues/2611
>>> I have some comments on specific entries below.
>>>
>>
>> And people can be free to add new entries later. However, we should
>> resist the temptation to add too many details. It should stay simple and
>> understandable, even if not all technical nuances are not represented.
>>
>>>> diff --git a/docs/devel/control-flow-integrity.rst b/docs/devel/control-flow-integrity.rst
>>>> index e6b73a4fe1a..3d5702fa4cc 100644
>>>> --- a/docs/devel/control-flow-integrity.rst
>>>> +++ b/docs/devel/control-flow-integrity.rst
>>>> @@ -1,3 +1,5 @@
>>>> +.. _cfi:
>>>> +
>>>>    ============================
>>>>    Control-Flow Integrity (CFI)
>>>>    ============================
>>>> diff --git a/docs/devel/multi-thread-tcg.rst b/docs/devel/multi-thread-tcg.rst
>>>> index d706c27ea74..7fd0a07633d 100644
>>>> --- a/docs/devel/multi-thread-tcg.rst
>>>> +++ b/docs/devel/multi-thread-tcg.rst
>>>> @@ -4,6 +4,8 @@
>>>>      This work is licensed under the terms of the GNU GPL, version 2 or
>>>>      later. See the COPYING file in the top-level directory.
>>>>
>>>> +.. _mttcg:
>>>> +
>>>>    ==================
>>>>    Multi-threaded TCG
>>>>    ==================
>>>> diff --git a/docs/glossary/index.rst b/docs/glossary/index.rst
>>>> new file mode 100644
>>>> index 00000000000..a2d4f3eae16
>>>> --- /dev/null
>>>> +++ b/docs/glossary/index.rst
>>>
>>> I guess it makes sense to give this its own subdir, since we want
>>> it to come at the end of the manual. The other option would be
>>> to put it directly into docs/.
>>>
>>
>>   From your comment, it's not clear for me if it's ok as it is, or if you
>> want a change.
>> Can you elaborate on that?
> 
> It means I'm not sure. We end up with a subdirectory with only
> one file in it and where there's no expectation we'd ever want
> to add any more files to it. On the other hand it does keep it
> out of the docs/ top level directory, which currently has a
> fair amount of cruft awaiting cleanup.
> 
> I guess on balance I would make this docs/glossary.rst,
> unless you anticipate wanting to split this into multiple
> files or have something else in docs/glossary/ later.
> 
>>>> @@ -0,0 +1,238 @@
>>>> +.. _Glossary:
>>>> +
>>>> +--------
>>>> +Glossary
>>>> +--------
>>>> +
>>>> +This section of the manual presents *simply* acronyms and terms QEMU developers
>>>> +use.
>>>
>>> What's "simply" intended to mean here?
>>>
>>
>> "in a straightforward or plain manner".
>> I can remove this word if you think it does not serve any purpose.
> 
> You could phrase it as "presents brief definitions of acronyms
> and terms", I think.
> 
>>>> +QEMU system defines board models for various architectures. It's a description
>>>> +of a SoC (system-on-chip) with various devices pre-configured, and can be
>>>> +selected with the option ``-machine`` of qemu-system.
>>>
>>> SoCs are not the same as boards.
>>>
>>> We could say something like:
>>>
>>> QEMU's system emulation models many different types of hardware.
>>> A machine model (sometimes called a board model) is the model
>>> of a complete virtual system with RAM, one or more CPUs, and
>>> various devices.
>>>
>>> We could also put in a link to
>>> https://www.qemu.org/docs/master/system/targets.html
>>> which is where we document what our machine types are.
>>>
>>
>> How do you distinguish a SoC and cpu? Is a SoC cpu + devices?
>> Isn't a board/machine a set of SoC + devices attached?
> 
> An SoC is a "system on chip". To quote wikipedia's definition:
> 
> "A system on a chip or system-on-chip is an integrated circuit that
> integrates most or all components of a computer or electronic system.
> These components usually include an on-chip central processing unit
> (CPU), memory interfaces, input/output devices and interfaces, and
> secondary storage interfaces, often alongside other components such
> as radio modems and a graphics processing unit (GPU) – all on a
> single substrate or microchip."
> 
> An SoC always contains a CPU, but it will have a lot more
> than that built into it too. And the SoC only has "most"
> of the system components, so the whole machine will be
> an SoC plus some other things.
> 
> Generally a board/machine that uses an SoC will have on it:
>   * an SoC
>   * the actual memory
>   * perhaps one or two extra devices external to the SoC
>   * connectors for things like serial ports, SD cards, etc
>     (which generally wire up to SoC pins)
>   * a crystal or similar to act as the main clock source
> 
> So if you look at a photo of a development board that uses
> an SoC, there will be one large chip which is the SoC,
> some RAM chips, a bunch of connectors and one or two smaller
> chips. Not every device will be inside the SoC, but
> generally almost all of them are.
> 
> QEMU's machine models for this kind of board match the
> organization of the hardware; looking at hw/arm/sabrelite.c
> which is a machine model you can see that it has:
>   * an instance of the fsl-imx6 SoC device object
>   * the main memory
>   * some NOR flash memory
>   * some configuration and wiring up of things
> 
> And the SoC itself is in hw/arm/fsl-imx6.c, and is a
> QOM device object that creates and wires up the CPUs,
> UARTs, USB controller, and various other devices that
> this particular SoC includes. In this case we only have
> one board model using this SoC, but for some SoCs we
> have several board models that all use the same SoC
> but wire up different external devices to it.
> 
> Some of our machine models are models of systems that
> don't use an SoC at all. This is rare in the Arm world,
> but for instance the SPARC machines like the SS-5 are
> like that -- the real hardware had a discrete CPU chip
> and a bunch of devices in their own chips on the
> motherboard, and QEMU's model of that hardware has
> a machine model which directly creates the CPU and
> the various devices. (And some of our older machine
> models are models of hardware that *does* have an SoC
> but where we didn't model that level of abstraction,
> so they directly create devices in the machine model
> that really ought to be inside an SoC object.
> hw/arm/stellaris.c is one example of that.)
> 

Thanks for the answer on this.
My specific implicit question was: do we have boards with cpu only (and 
not a SoC), which you answered.

>>>> +
>>>> +Block
>>>> +-----
>>>> +
>>>> +Block drivers are the available `disk formats <block-drivers>` available, and
>>>> +block devices `(see Block device section on options page)<sec_005finvocation>`
>>>> +are using them to implement disks for a virtual machine.
>>>
>>> Block drivers aren't just disk formats; there are some filter
>>> drivers too. Somebody on the block side could probably
>>> provide a better definition here.
>>>
>>
>> I'm open to a more exact definition. The two terms (drivers and devices)
>> seem to overlap on some parts, so I came up with this trivial definition.
> 
> Yeah, the driver vs device split is a good one (basically
> the device is the front-end visible to the guest, and the
> driver is the back-end that provides it with storage
> via an abstracted API). The nit I'm picking here is that
> not every block driver is there to provide support for
> an on-host disk format.
> 
>>>> +
>>>> +CFI
>>>> +---
>>>> +
>>>> +Control Flow Integrity is a hardening technique used to prevent exploits
>>>> +targeting QEMU by detecting unexpected branches during execution. QEMU `actively
>>>> +supports<cfi>` being compiled with CFI enabled.
>>>> +
>>>> +Device
>>>> +------
>>>> +
>>>> +QEMU is able to emulate a CPU, and all the hardware interacting with it,
>>>> +including many devices. When QEMU runs a virtual machine using a hardware-based
>>>> +accelerator, it is responsible for emulating, using software, all devices.
>>>
>>> This definition doesn't actually define what a device is :-)
>>>
>>
>> Indeed :)
>> Should we explain what is a computer device?
>> The goal was just to say that QEMU can emulate hardware interacting with
>> the cpu, which could be a possible definition. So people can associate
>> that QEMU devices are nothing else than a "normal" computer device.
> 
> We could say, perhaps:
> 
> In QEMU, a device is a piece of hardware visible to the guest.
> Examples include UARTs, PCI controllers, PCI cards, VGA controllers,
> and many more.
> 
> 
> 
>>>> +
>>>> +Hypervisor
>>>> +----------
>>>> +
>>>> +The formal definition of an hypervisor is a program than can be used to manage a
>>>> +virtual machine. QEMU itself is an hypervisor.
>>>
>>> "a hypervisor". QEMU isn't really a hypervisor, though...
>>>
>>
>> It's a shortcut, and I'm open to change it. It brings an interesting
>> question though.
>>
>> Technically, QEMU interacts with hypervisor APIs built in various OSes.
>> On the other hand, when we use TCG, it's an emulator instead.
>>
>> But as you can't use KVM/hvf/whpx by itself, how do you name the program
>> interacting with it, and emulating the rest of the VM?
>>
>> The correct word is probably "virtualizer", but from searching on
>> Internet, it seems that "vmm" and "virtualizer" are considered the same
>> as an "hypervisor". The difference is subtle, and maybe we have an
>> opportunity here to clarify it.
> 
> 
>>>> +Patchwork
>>>> +---------
>>>> +
>>>> +`Patchwork <https://patchew.org/QEMU/>`_ is a website that tracks
>>>> +patches on the Mailing List.
>>>
>>> Patchwork and patchew are different systems. Patchew's URL is
>>> https://patchew.org/QEMU/
>>>
>>> (There is a patchwork instance that tracks qemu-devel patches,
>>> at https://patchwork.kernel.org/project/qemu-devel/list/ , but
>>> I'm not aware of any developers that are actively using it, so
>>> I don't think it merits being mentioned in the glossary.)
>>>
>>
>> I've been confused by that, and just thought it was two different
>> instances (fork me if you can) of the "same" thing.
>> How would you define patchew?
>> When we say patchwork, do we implicitely mean patchew?
> 
> No. patchwork is patchwork, and patchew is patchew -- these
> are entirely different pieces of software that happen to do
> similar jobs.
> 
>> if I understand currently, patchew is what we want to mention in our
>> doc? (and mention it's not associated to patchwork).
> 
> We don't use patchwork, so we don't need to mention it anywhere.
> 
>>>> +Once a series is reviewed and accepted by a subsystem maintainer, it will be
>>>> +included in a PR (Pull Request) that the project maintainer will merge into QEMU
>>>> +main branch, after running tests.
>>>
>>> I think we could probably also usefully say
>>>
>>> "The QEMU project doesn't currently expect most developers to
>>> directly submit pull requests."
>>>
>>> just to flag up that our development model isn't like the
>>> currently-popular github/gitlab one where a PR is how you
>>> send contributions.
>>>
>>
>> This is interesting.
>>
>> For the majority of developers nowadays, a PR is a GitHub/GitLab PR.
>> Despite the fact we use the original PR meaning (in git terms), it's
>> probably confusing when new comers hear pull request.
>>
>>>> +
>>>> +QCOW
>>>> +----
>>>> +
>>>> +QEMU Copy On Write is a disk format developed by QEMU. It provides transparent
>>>> +compression, automatic extension, and many other advantages over a raw image.
>>>
>>> We want to be a bit careful here, because the "qcow" format
>>> is not something we recommend for new use -- "qcow2" is what
>>> you actually want.
>>>
>>> https://www.qemu.org/docs/master/system/qemu-block-drivers.html#cmdoption-image-formats-arg-qcow2
>>>
>>
>> Sounds good.
>>
>> For my personal knowledge: during this work I discovered that we had
>> qcow3. From what I understood, it seems to be included in what we called
>> qcow2 today. Is that correct?
> 
> I have no idea -- you'd need to ask somebody who works on the
> block layer
> 
> thanks
> -- PMM

Thanks for all the suggestions, I'll integrate those in next version.

[PATCH 7/7] docs: add a how to section

[Pierrick Bouvier]

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
---
 docs/devel/build-system.rst |   2 +
 docs/how-to/index.rst       | 146 ++++++++++++++++++++++++++++++++++++
 docs/index.rst              |   1 +
 3 files changed, 149 insertions(+)
 create mode 100644 docs/how-to/index.rst

diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
index d42045a2325..db444787e37 100644
--- a/docs/devel/build-system.rst
+++ b/docs/devel/build-system.rst
@@ -1,3 +1,5 @@
+.. _build:
+
 ==================================
 The QEMU build system architecture
 ==================================
diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst
new file mode 100644
index 00000000000..3a9d4d777df
--- /dev/null
+++ b/docs/how-to/index.rst
@@ -0,0 +1,146 @@
+.. _how-to:
+
+------
+How to
+------
+
+This section of the manual will give you some commands to do various tasks with
+QEMU. It does not intend to be complete, but to be simple.
+
+Build
+-----
+
+First you need setup your `build environment <setup-build-env>`.
+
+Then, you can build QEMU using:
+
+::
+
+    git clone https://gitlab.com/qemu-project/qemu
+    cd qemu
+    ./configure
+    ninja -C build
+    # all binaries are in ./build
+
+By default, QEMU build is optimized. You may want to switch to debug builds
+instead (non optimized, and with more runtime checks enabled):
+
+::
+
+    ./configure --enable-debug
+
+It's recommended to use sanitizers to catch issues when developing your change.
+
+::
+
+    ./configure --enable-asan --enable-ubsan
+    # Of course, you can combine debug and sanitizers if needed
+
+You can find more information on `build page <build>`.
+
+Test
+----
+
+QEMU has a lot of tests, mainly in 4 categories:
+
+::
+
+    # run tests related to TCG. They are based on Makefiles.
+    make check-tcg
+    # run system tests, running a full VM, with avocado framework
+    make check-avocado
+    # run functional tests, running a full VM, integrated with Meson
+    make check-functional
+    # run all other tests, integrated with Meson
+    make check
+
+You can find more information on `testing page<testing>`.
+
+Use QEMU
+--------
+
+To create a 20 gigabytes disk image usable with qemu-system:
+
+::
+
+    qemu-img create system.img 20g
+
+To run an x86_64 system emulated, with 4 cpus, 8G of memory and an install iso:
+
+::
+
+    qemu-system-x86_64 -smp 4 -m 8G system.img -cdrom install.iso
+
+To boot directly a Linux Kernel:
+
+::
+
+    qemu-system-x86_64 -kernel bzImage -hda system.img -append "root=/dev/hda"
+
+To boot an aarch64 system emulated, you need to specify a UEFI and associated
+pflash. Once started, you can switch to Serial output by clicking on View ->
+Serial0.
+
+::
+
+    # UEFI can be obtained from debian package qemu-efi-aarch64.
+    # First, we need to copy a file to save UEFI variables:
+    # cp /usr/share/AAVMF/AAVMF_VARS.fd .
+    qemu-system-aarch64 \
+        -m 8G \
+        -smp 4 \
+        -M virt \
+        -cpu max \
+        -device virtio-blk-pci,drive=root \
+        -drive if=none,id=root,file=system.img \
+        -drive if=pflash,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd \
+        -drive if=pflash,file=AAVMF_VARS.fd \
+        -cdrom install.iso
+
+To run git using QEMU user-mode:
+
+::
+
+    qemu-x86_64 /usr/bin/git --version
+
+Contribute
+----------
+
+We recommend using `git-publish <https://github.com/stefanha/git-publish>`_ for
+contributing. You need to configure `git send-email
+<https://git-send-email.io/>`_ first.
+
+::
+
+    git checkout -b my_feature
+    ... # edit, build, test
+    # When ready to send the series...
+
+    # Add upstream QEMU repo as a remote.
+    git remote add upstream https://gitlab.com/qemu-project/qemu
+    # Fetch all new content.
+    git fetch -a upstream
+
+    # Rebase your branch on top of upstream master, and include a signoff.
+    git rebase -i upstream/master --signoff
+    # Check your patches are correct.
+    ./scripts/checkpatch.pl $(git merge-base upstream/master HEAD)..HEAD
+
+    # Send your series, you'll be given a chance to edit cover letter for it.
+    git-publish
+
+    # After review, and other changes, you can send a v2 simply by using:
+    git-publish
+
+If you need to apply locally an existing series, you can use `b4
+<https://github.com/mricon/b4>`_ (installable via pip) to retrieve it:
+
+::
+
+    b4 shazam <series_msg_id>
+    # message id is an identifier present in email sent.
+    # when using patchwork, it is the last part of a series url (2024...):
+    # https://patchew.org/QEMU/20241118021820.4928-1-joel@jms.id.au/
+
+More complete information is available on our `Submit a patch page
+<submitting-a-patch>`.
diff --git a/docs/index.rst b/docs/index.rst
index 2cad84cd77c..e275a9223dd 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -20,5 +20,6 @@ Welcome to QEMU's documentation!
    interop/index
    specs/index
    devel/index
+   how-to/index
    codebase/index
    glossary/index
-- 
2.39.5

Re: [PATCH 7/7] docs: add a how to section

[Daniel P. Berrangé]

On Mon, Nov 18, 2024 at 09:23:57AM -0800, Pierrick Bouvier wrote:
> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
> ---
>  docs/devel/build-system.rst |   2 +
>  docs/how-to/index.rst       | 146 ++++++++++++++++++++++++++++++++++++
>  docs/index.rst              |   1 +
>  3 files changed, 149 insertions(+)
>  create mode 100644 docs/how-to/index.rst
> 
> diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
> index d42045a2325..db444787e37 100644
> --- a/docs/devel/build-system.rst
> +++ b/docs/devel/build-system.rst
> @@ -1,3 +1,5 @@
> +.. _build:
> +
>  ==================================
>  The QEMU build system architecture
>  ==================================
> diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst
> new file mode 100644
> index 00000000000..3a9d4d777df
> --- /dev/null
> +++ b/docs/how-to/index.rst
> @@ -0,0 +1,146 @@
> +.. _how-to:
> +
> +------
> +How to
> +------
> +
> +This section of the manual will give you some commands to do various tasks with
> +QEMU. It does not intend to be complete, but to be simple.
> +
> +Build
> +-----
> +
> +First you need setup your `build environment <setup-build-env>`.
> +
> +Then, you can build QEMU using:
> +
> +::
> +
> +    git clone https://gitlab.com/qemu-project/qemu
> +    cd qemu
> +    ./configure
> +    ninja -C build
> +    # all binaries are in ./build
> +
> +By default, QEMU build is optimized. You may want to switch to debug builds
> +instead (non optimized, and with more runtime checks enabled):
> +
> +::
> +
> +    ./configure --enable-debug
> +
> +It's recommended to use sanitizers to catch issues when developing your change.
> +
> +::
> +
> +    ./configure --enable-asan --enable-ubsan
> +    # Of course, you can combine debug and sanitizers if needed
> +
> +You can find more information on `build page <build>`.
> +
> +Test
> +----
> +
> +QEMU has a lot of tests, mainly in 4 categories:
> +
> +::
> +
> +    # run tests related to TCG. They are based on Makefiles.
> +    make check-tcg
> +    # run system tests, running a full VM, with avocado framework
> +    make check-avocado
> +    # run functional tests, running a full VM, integrated with Meson
> +    make check-functional
> +    # run all other tests, integrated with Meson
> +    make check
> +
> +You can find more information on `testing page<testing>`.
> +
> +Use QEMU
> +--------
> +
> +To create a 20 gigabytes disk image usable with qemu-system:
> +
> +::
> +
> +    qemu-img create system.img 20g
> +
> +To run an x86_64 system emulated, with 4 cpus, 8G of memory and an install iso:
> +
> +::
> +
> +    qemu-system-x86_64 -smp 4 -m 8G system.img -cdrom install.iso
> +
> +To boot directly a Linux Kernel:
> +
> +::
> +
> +    qemu-system-x86_64 -kernel bzImage -hda system.img -append "root=/dev/hda"
> +
> +To boot an aarch64 system emulated, you need to specify a UEFI and associated
> +pflash. Once started, you can switch to Serial output by clicking on View ->
> +Serial0.
> +
> +::
> +
> +    # UEFI can be obtained from debian package qemu-efi-aarch64.
> +    # First, we need to copy a file to save UEFI variables:
> +    # cp /usr/share/AAVMF/AAVMF_VARS.fd .
> +    qemu-system-aarch64 \
> +        -m 8G \
> +        -smp 4 \
> +        -M virt \
> +        -cpu max \
> +        -device virtio-blk-pci,drive=root \
> +        -drive if=none,id=root,file=system.img \
> +        -drive if=pflash,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd \
> +        -drive if=pflash,file=AAVMF_VARS.fd \
> +        -cdrom install.iso
> +
> +To run git using QEMU user-mode:
> +
> +::
> +
> +    qemu-x86_64 /usr/bin/git --version
> +
> +Contribute
> +----------
> +
> +We recommend using `git-publish <https://github.com/stefanha/git-publish>`_ for
> +contributing. You need to configure `git send-email
> +<https://git-send-email.io/>`_ first.
> +
> +::
> +
> +    git checkout -b my_feature
> +    ... # edit, build, test
> +    # When ready to send the series...
> +
> +    # Add upstream QEMU repo as a remote.
> +    git remote add upstream https://gitlab.com/qemu-project/qemu
> +    # Fetch all new content.
> +    git fetch -a upstream
> +
> +    # Rebase your branch on top of upstream master, and include a signoff.
> +    git rebase -i upstream/master --signoff
> +    # Check your patches are correct.
> +    ./scripts/checkpatch.pl $(git merge-base upstream/master HEAD)..HEAD
> +
> +    # Send your series, you'll be given a chance to edit cover letter for it.
> +    git-publish
> +
> +    # After review, and other changes, you can send a v2 simply by using:
> +    git-publish
> +
> +If you need to apply locally an existing series, you can use `b4
> +<https://github.com/mricon/b4>`_ (installable via pip) to retrieve it:
> +
> +::
> +
> +    b4 shazam <series_msg_id>
> +    # message id is an identifier present in email sent.
> +    # when using patchwork, it is the last part of a series url (2024...):
> +    # https://patchew.org/QEMU/20241118021820.4928-1-joel@jms.id.au/
> +
> +More complete information is available on our `Submit a patch page
> +<submitting-a-patch>`.

I'm far from convinced any of the above content is a good idea, given
it is duplicating stuff we've already got elsewhere in our manual.
This gives us the extra burden of ensuring different parts of the
manual are consistent in what they're recommending which is something
we are historically bad at doing.

I think there is scope for having a "how-tos" section, but this particular
choices of examples are not the best place to kick it off with IMHO. Also,
for the sake of scalability, I'd suggest that each "how to" is a standalone
file in the how-tos/ sub-directory, otherwise the index.rst will quickly
become enourmous.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 7/7] docs: add a how to section

[Pierrick Bouvier]

On 11/19/24 01:29, Daniel P. Berrangé wrote:
> On Mon, Nov 18, 2024 at 09:23:57AM -0800, Pierrick Bouvier wrote:
>> Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
>> ---
>>   docs/devel/build-system.rst |   2 +
>>   docs/how-to/index.rst       | 146 ++++++++++++++++++++++++++++++++++++
>>   docs/index.rst              |   1 +
>>   3 files changed, 149 insertions(+)
>>   create mode 100644 docs/how-to/index.rst
>>
>> diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
>> index d42045a2325..db444787e37 100644
>> --- a/docs/devel/build-system.rst
>> +++ b/docs/devel/build-system.rst
>> @@ -1,3 +1,5 @@
>> +.. _build:
>> +
>>   ==================================
>>   The QEMU build system architecture
>>   ==================================
>> diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst
>> new file mode 100644
>> index 00000000000..3a9d4d777df
>> --- /dev/null
>> +++ b/docs/how-to/index.rst
>> @@ -0,0 +1,146 @@
>> +.. _how-to:
>> +
>> +------
>> +How to
>> +------
>> +
>> +This section of the manual will give you some commands to do various tasks with
>> +QEMU. It does not intend to be complete, but to be simple.
>> +
>> +Build
>> +-----
>> +
>> +First you need setup your `build environment <setup-build-env>`.
>> +
>> +Then, you can build QEMU using:
>> +
>> +::
>> +
>> +    git clone https://gitlab.com/qemu-project/qemu
>> +    cd qemu
>> +    ./configure
>> +    ninja -C build
>> +    # all binaries are in ./build
>> +
>> +By default, QEMU build is optimized. You may want to switch to debug builds
>> +instead (non optimized, and with more runtime checks enabled):
>> +
>> +::
>> +
>> +    ./configure --enable-debug
>> +
>> +It's recommended to use sanitizers to catch issues when developing your change.
>> +
>> +::
>> +
>> +    ./configure --enable-asan --enable-ubsan
>> +    # Of course, you can combine debug and sanitizers if needed
>> +
>> +You can find more information on `build page <build>`.
>> +
>> +Test
>> +----
>> +
>> +QEMU has a lot of tests, mainly in 4 categories:
>> +
>> +::
>> +
>> +    # run tests related to TCG. They are based on Makefiles.
>> +    make check-tcg
>> +    # run system tests, running a full VM, with avocado framework
>> +    make check-avocado
>> +    # run functional tests, running a full VM, integrated with Meson
>> +    make check-functional
>> +    # run all other tests, integrated with Meson
>> +    make check
>> +
>> +You can find more information on `testing page<testing>`.
>> +
>> +Use QEMU
>> +--------
>> +
>> +To create a 20 gigabytes disk image usable with qemu-system:
>> +
>> +::
>> +
>> +    qemu-img create system.img 20g
>> +
>> +To run an x86_64 system emulated, with 4 cpus, 8G of memory and an install iso:
>> +
>> +::
>> +
>> +    qemu-system-x86_64 -smp 4 -m 8G system.img -cdrom install.iso
>> +
>> +To boot directly a Linux Kernel:
>> +
>> +::
>> +
>> +    qemu-system-x86_64 -kernel bzImage -hda system.img -append "root=/dev/hda"
>> +
>> +To boot an aarch64 system emulated, you need to specify a UEFI and associated
>> +pflash. Once started, you can switch to Serial output by clicking on View ->
>> +Serial0.
>> +
>> +::
>> +
>> +    # UEFI can be obtained from debian package qemu-efi-aarch64.
>> +    # First, we need to copy a file to save UEFI variables:
>> +    # cp /usr/share/AAVMF/AAVMF_VARS.fd .
>> +    qemu-system-aarch64 \
>> +        -m 8G \
>> +        -smp 4 \
>> +        -M virt \
>> +        -cpu max \
>> +        -device virtio-blk-pci,drive=root \
>> +        -drive if=none,id=root,file=system.img \
>> +        -drive if=pflash,readonly=on,file=/usr/share/AAVMF/AAVMF_CODE.fd \
>> +        -drive if=pflash,file=AAVMF_VARS.fd \
>> +        -cdrom install.iso
>> +
>> +To run git using QEMU user-mode:
>> +
>> +::
>> +
>> +    qemu-x86_64 /usr/bin/git --version
>> +
>> +Contribute
>> +----------
>> +
>> +We recommend using `git-publish <https://github.com/stefanha/git-publish>`_ for
>> +contributing. You need to configure `git send-email
>> +<https://git-send-email.io/>`_ first.
>> +
>> +::
>> +
>> +    git checkout -b my_feature
>> +    ... # edit, build, test
>> +    # When ready to send the series...
>> +
>> +    # Add upstream QEMU repo as a remote.
>> +    git remote add upstream https://gitlab.com/qemu-project/qemu
>> +    # Fetch all new content.
>> +    git fetch -a upstream
>> +
>> +    # Rebase your branch on top of upstream master, and include a signoff.
>> +    git rebase -i upstream/master --signoff
>> +    # Check your patches are correct.
>> +    ./scripts/checkpatch.pl $(git merge-base upstream/master HEAD)..HEAD
>> +
>> +    # Send your series, you'll be given a chance to edit cover letter for it.
>> +    git-publish
>> +
>> +    # After review, and other changes, you can send a v2 simply by using:
>> +    git-publish
>> +
>> +If you need to apply locally an existing series, you can use `b4
>> +<https://github.com/mricon/b4>`_ (installable via pip) to retrieve it:
>> +
>> +::
>> +
>> +    b4 shazam <series_msg_id>
>> +    # message id is an identifier present in email sent.
>> +    # when using patchwork, it is the last part of a series url (2024...):
>> +    # https://patchew.org/QEMU/20241118021820.4928-1-joel@jms.id.au/
>> +
>> +More complete information is available on our `Submit a patch page
>> +<submitting-a-patch>`.
> 
> I'm far from convinced any of the above content is a good idea, given
> it is duplicating stuff we've already got elsewhere in our manual.
> This gives us the extra burden of ensuring different parts of the
> manual are consistent in what they're recommending which is something
> we are historically bad at doing.
> 

My idea was to keep this whole together instead of scattering those 
"basic" use cases across all the documentation, which makes them hard to 
find by design.

> I think there is scope for having a "how-tos" section, but this particular
> choices of examples are not the best place to kick it off with IMHO. Also,
> for the sake of scalability, I'd suggest that each "how to" is a standalone
> file in the how-tos/ sub-directory, otherwise the index.rst will quickly
> become enourmous.
> 

I will leave that page out of this series, if you feel it's not a good 
approach for now.
In case we revisit the idea later, I think the how-to section should 
*not* become a huge wiki page, where all the niche use cases we imagine 
are documented, but simply a way for new developers to do the basic 
operations (build/execute/test/contribute) they need. That's the 
compromise I tried to create here, based on my personal experience as a 
new developer on QEMU.

> With regards,
> Daniel