Re: [REPORT] [GSoC - TCG Continuous Benchmarking] [#2] Dissecting QEMU Into Three Main Parts

["Lukáš Doktor" <ldoktor@redhat.com>]

[-- Attachment #1.1: Type: text/plain, Size: 4573 bytes --]

Dne 30. 06. 20 v 11:41 Aleksandar Markovic napsal(a):
> уто, 30. јун 2020. у 06:34 Lukáš Doktor <ldoktor@redhat.com> је написао/ла:
>>
>> Dne 29. 06. 20 v 12:25 Ahmed Karaman napsal(a):
>>> Hi,
>>>
>>> The second report of the TCG Continuous Benchmarking series builds
>>> upon the QEMU performance metrics calculated in the previous report.
>>> This report presents a method to dissect the number of instructions
>>> executed by a QEMU invocation into three main phases:
>>> - Code Generation
>>> - JIT Execution
>>> - Helpers Execution
>>> It devises a Python script that automates this process.
>>>
>>> After that, the report presents an experiment for comparing the
>>> output of running the script on 17 different targets. Many conclusions
>>> can be drawn from the results and two of them are discussed in the
>>> analysis section.
>>>
>>> Report link:
>>> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/Dissecting-QEMU-Into-Three-Main-Parts/
>>>
>>> Previous reports:
>>> Report 1 - Measuring Basic Performance Metrics of QEMU:
>>> https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06692.html
>>>
>>> Best regards,
>>> Ahmed Karaman
>>
>> Hello Ahmed,
>>
>> very nice reading, both reports so far. One thing that could be better displayed is the system you used this to generate. This would come handy especially later when you move from examples to actual reports. I think it'd make sense to add a section with a clear definition of the machine as well as the operation system, qemu version and eventually other deps (like compiler, flags, ...). For this report something like:
>>
>> architecture: x86_64
>> cpu_codename: Kaby Lake
>> cpu: i7-8650U
>> ram: 32GB DDR4
>> os: Fedora 32
>> qemu: 470dd165d152ff7ceac61c7b71c2b89220b3aad7
>> compiler: gcc-10.1.1-1.fc32.x86_64
>> flags: --target-list="x86_64-softmmu,ppc64-softmmu,aarch64-softmmu,s390x-softmmu,riscv64-softmmu" --disable-werror --disable-sparse --enable-sdl --enable-kvm  --enable-vhost-net --enable-vhost-net --enable-attr  --enable-kvm  --enable-fdt   --enable-vnc --enable-seccomp --block-drv-rw-whitelist="vmdk,null-aio,quorum,null-co,blkverify,file,nbd,raw,blkdebug,host_device,qed,nbd,iscsi,gluster,rbd,qcow2,throttle,copy-on-read" --python=/usr/bin/python3 --enable-linux-io-uring
>>
>> would do. Maybe it'd be even a good idea to create a script to report this basic set of information and add it after each of the perf scripts so people don't forget to double-check the conditions, but others might disagree so take this only as a suggestion.
>>
> 
> I just want to follow up on this observation here, and not related to
> Ahmed's report at all.
> 
> We often receive bug reports of the following style: "I have Debian
> 10.2 system and mips emulation misbehaves". As you may imagine, I
> assign the bug to myself, install Debian 10.2 system on my
> experimental box, and mips emulation works like charm.
> <banging-head-against-the-wall-emoji> Obviously, I need more info on
> the submitter's system.
> 
> After all these years, we don't have (or at least I don't know about
> it) a script that we could give the submitter, that picks up various
> aspects of his system. This script, since it is not "for presentation"
> could be even far more aggressive in picking ups system information
> that what Lukas mentioned above. It could collect the output of
> various relevant commands, and yip it in a single file. We should have
> "get_system_info.py" in our scripts directory!
> 
> Sincerely,
> Aleksandar
> 

Well this itself is a very complicated matter that could deserve a GSoC project. It's hard to balance the utils required to obtain the knowledge. I'm fond of sosreport, that is heavily used by RH but the result is quite big. Slightly smaller set can be generated via ansible, which itself gathers a lot of useful information. If we are to speak only about minimal approach especially tailored to qemu, than I'd suggest taking a look at `avocado.utils` especially `avocado.utils.cpu` as Avocado is already used for qemu testing.

Anyway don't consider this as a complete list, I just wanted to demonstrate how difficult and complex this subject is.

Regards,
Lukáš

> 
>> Regards,
>> Lukáš
>>
>> PS: Automated cpu codenames, hosts OSes and such could be tricky, but one can use other libraries or just best-effort-approach with fallback to "unknown" to let people filling it manually or adding their branch to your script.
>>
>> Regards,
>> Lukáš
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]