[GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

[-- Attachment #1: Type: text/plain, Size: 2876 bytes --]


Hi, everybody.

I am going to propose several ideas for QEMU participation in GSoC/Outreachy in next few days. This is the first one. Please feel free to give an honest feedback.

Yours,
Aleksandar



Measure and Analyze Performance of
QEMU User and System Mode Emulation


PLANNED ACTIVITIES

PART I: (user mode)

   a) select around a dozen test programs (resembling components of SPEC benchmark, but must be open source, and preferably license compatible with QEMU); test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
   b) measure execution time and other performance data in user mode across all platforms for ToT:
       - try to improve performance if there is an obvious bottleneck (but this is unlikely);
       - develop tests that will be protection against performance regressions in future.
   c) measure execution time in user-mode for selected platforms for all QEMU versions in last 5 years:
       - confirm performance improvements and/or detect performance degradations.
   d) summarize all results in a comprehensive form, using also graphics/data visualization.

PART II: (system mode)

   a) measure execution time and other performance data for boot/shutdown cycle for selected machines for ToT:
       - try to improve performance if there is an obvious bottleneck;
       - develop tests that will be protection against performance regressions in future.
   b) summarize all results in a comprehensive form.


DELIVERABLES

1) Each maintainer for target will be given a list of top 25 functions in terms of spent host time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.

2) Each maintainer for machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations will be also provided, if the judgment is they are useful and relevant.

3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.

(parts 1) and 2) will be, of course, published to everybody, maintainers are simply singled out as main recipients and decision-makers on possible next action items)

Deliverable will be distributed over wide time interval (in other words, they will not be presented just at the end of project, but gradually during project execution).


Mentor: Aleksandar Markovic (myself) (but, I am perfectly fine if somebody else wants to mentor the project, if interested)

Student: open


That would be all, feel free to ask for additional info and/or clarification.
 

[-- Attachment #2: Type: text/html, Size: 3482 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 572 bytes --]

On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.

Tracking performance is a good idea and something that has not been done
upstream yet.  A few questions:

 * Will benchmarks be run automatically (e.g. nightly or weekly) on
   someone's hardware or does every TCG architecture maintainer need to
   run them manually for themselves?
 * Where will the benchmark result history be stored?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
>
> Tracking performance is a good idea and something that has not been done
> upstream yet.

Thanks for the interest, Stefan!

>  A few questions:
>
>  * Will benchmarks be run automatically (e.g. nightly or weekly) on
>    someone's hardware or does every TCG architecture maintainer need to
>    run them manually for themselves?

If the community wants it, definitely yes. Once the methodology is
developed, it should be straightforward to setup nightly and/or weekly
benchmarks - that could definitely include sending mails with reports
to the entire list or just individuals or subgroups. The recipient
choice is just a matter or having decent criteria about
appropriateness of information within the message (e.g. not to flood
the list with the data most people are not really interested).

For linux-user tests, they are typically very quick, and nightly tests
are quite feasible to run. On someone hardware, of course, and
consistently always on the same hardware, if possible. If it makes
sense, one could setup multiple test beds with a variety of hardware
setups.

For system mode tests, I knoe they are much more difficult to
automate, and, on top of that, there could be greater risk of
hangs/crashes Also, considering the number of machines we support,
those tests could consume much more time - perhaps even one day would
not be sufficient, if we have many machines and boot/shutdown
variants. For these reason, perhaps weekly executions would be more
appropriate for them, and, in general, given greater complexity, the
expectation from system-mode performance tests should be better kept
quite low for now.

>  * Where will the benchmark result history be stored?
>

If emailing is set up, the results could be reconstructed from emails.
But, yes, it would be better if the result history is kept somewhere
on an internet-connected file server

Yours,
Aleksandar

> Stefan

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2551 bytes --]

On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote:
> On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > > 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
> >
> > Tracking performance is a good idea and something that has not been done
> > upstream yet.
> 
> Thanks for the interest, Stefan!
> 
> >  A few questions:
> >
> >  * Will benchmarks be run automatically (e.g. nightly or weekly) on
> >    someone's hardware or does every TCG architecture maintainer need to
> >    run them manually for themselves?
> 
> If the community wants it, definitely yes. Once the methodology is
> developed, it should be straightforward to setup nightly and/or weekly
> benchmarks - that could definitely include sending mails with reports
> to the entire list or just individuals or subgroups. The recipient
> choice is just a matter or having decent criteria about
> appropriateness of information within the message (e.g. not to flood
> the list with the data most people are not really interested).
> 
> For linux-user tests, they are typically very quick, and nightly tests
> are quite feasible to run. On someone hardware, of course, and
> consistently always on the same hardware, if possible. If it makes
> sense, one could setup multiple test beds with a variety of hardware
> setups.
> 
> For system mode tests, I knoe they are much more difficult to
> automate, and, on top of that, there could be greater risk of
> hangs/crashes Also, considering the number of machines we support,
> those tests could consume much more time - perhaps even one day would
> not be sufficient, if we have many machines and boot/shutdown
> variants. For these reason, perhaps weekly executions would be more
> appropriate for them, and, in general, given greater complexity, the
> expectation from system-mode performance tests should be better kept
> quite low for now.
> 
> >  * Where will the benchmark result history be stored?
> >
> 
> If emailing is set up, the results could be reconstructed from emails.
> But, yes, it would be better if the result history is kept somewhere
> on an internet-connected file server

Thanks.  I don't want to overcomplicate this project.  The main thing is
to identify the stakeholders (TCG target maintainers?) and make sure
they are happy.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

On Wed, Jan 22, 2020 at 12:28 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>
> On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote:
> > On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > >
> > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > > > 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
> > >
> > > Tracking performance is a good idea and something that has not been done
> > > upstream yet.
> >
> > Thanks for the interest, Stefan!
> >
> > >  A few questions:
> > >
> > >  * Will benchmarks be run automatically (e.g. nightly or weekly) on
> > >    someone's hardware or does every TCG architecture maintainer need to
> > >    run them manually for themselves?
> >
> > If the community wants it, definitely yes. Once the methodology is
> > developed, it should be straightforward to setup nightly and/or weekly
> > benchmarks - that could definitely include sending mails with reports
> > to the entire list or just individuals or subgroups. The recipient
> > choice is just a matter or having decent criteria about
> > appropriateness of information within the message (e.g. not to flood
> > the list with the data most people are not really interested).
> >
> > For linux-user tests, they are typically very quick, and nightly tests
> > are quite feasible to run. On someone hardware, of course, and
> > consistently always on the same hardware, if possible. If it makes
> > sense, one could setup multiple test beds with a variety of hardware
> > setups.
> >
> > For system mode tests, I knoe they are much more difficult to
> > automate, and, on top of that, there could be greater risk of
> > hangs/crashes Also, considering the number of machines we support,
> > those tests could consume much more time - perhaps even one day would
> > not be sufficient, if we have many machines and boot/shutdown
> > variants. For these reason, perhaps weekly executions would be more
> > appropriate for them, and, in general, given greater complexity, the
> > expectation from system-mode performance tests should be better kept
> > quite low for now.
> >
> > >  * Where will the benchmark result history be stored?
> > >
> >
> > If emailing is set up, the results could be reconstructed from emails.
> > But, yes, it would be better if the result history is kept somewhere
> > on an internet-connected file server
>
> Thanks.  I don't want to overcomplicate this project.  The main thing is
> to identify the stakeholders (TCG target maintainers?) and make sure
> they are happy.
>

Yes, Stefan, TCG target maintainers would be the main stakeholders. To
some extent, various Machine maintainers would also be stakeholders,
but they will most likely come back to TCG target maintainers looking
for solution. In a literal sense, a number of maintainers were
initially going to be very unhappy seeing the results (for example,
seeing that the machine or entire target performs poorly compared to
similar machines/targets), but after a while they should and will
become happy realizing the problem was identified, and the culprit is
at least approximately determined.

I intentionally wanted to keep the project description simple in order
to be realistic and not develop high expectation among any of us. And
if the student proved to be capable, it will be very easy to add some
more useful tasks for him in this area, to be included in his/hers
GSoC/Outreachy activities.

He had just today one case of performance degradation identified manually:

https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg06326.html

This project aims to do these kind of things easier, and possibly in
an automated way. Howard did this by manual measurements for one
particular setup, but this project will cover much much more.

Thanks, Stefan, again for your interest - and everything else!

Aleksandar



> Stefan

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4413 bytes --]

On Sun, Jan 26, 2020 at 05:50:24PM +0100, Aleksandar Markovic wrote:
> On Wed, Jan 22, 2020 at 12:28 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> >
> > On Tue, Jan 21, 2020 at 03:07:53PM +0100, Aleksandar Markovic wrote:
> > > On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
> > > >
> > > > On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
> > > > > 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
> > > >
> > > > Tracking performance is a good idea and something that has not been done
> > > > upstream yet.
> > >
> > > Thanks for the interest, Stefan!
> > >
> > > >  A few questions:
> > > >
> > > >  * Will benchmarks be run automatically (e.g. nightly or weekly) on
> > > >    someone's hardware or does every TCG architecture maintainer need to
> > > >    run them manually for themselves?
> > >
> > > If the community wants it, definitely yes. Once the methodology is
> > > developed, it should be straightforward to setup nightly and/or weekly
> > > benchmarks - that could definitely include sending mails with reports
> > > to the entire list or just individuals or subgroups. The recipient
> > > choice is just a matter or having decent criteria about
> > > appropriateness of information within the message (e.g. not to flood
> > > the list with the data most people are not really interested).
> > >
> > > For linux-user tests, they are typically very quick, and nightly tests
> > > are quite feasible to run. On someone hardware, of course, and
> > > consistently always on the same hardware, if possible. If it makes
> > > sense, one could setup multiple test beds with a variety of hardware
> > > setups.
> > >
> > > For system mode tests, I knoe they are much more difficult to
> > > automate, and, on top of that, there could be greater risk of
> > > hangs/crashes Also, considering the number of machines we support,
> > > those tests could consume much more time - perhaps even one day would
> > > not be sufficient, if we have many machines and boot/shutdown
> > > variants. For these reason, perhaps weekly executions would be more
> > > appropriate for them, and, in general, given greater complexity, the
> > > expectation from system-mode performance tests should be better kept
> > > quite low for now.
> > >
> > > >  * Where will the benchmark result history be stored?
> > > >
> > >
> > > If emailing is set up, the results could be reconstructed from emails.
> > > But, yes, it would be better if the result history is kept somewhere
> > > on an internet-connected file server
> >
> > Thanks.  I don't want to overcomplicate this project.  The main thing is
> > to identify the stakeholders (TCG target maintainers?) and make sure
> > they are happy.
> >
> 
> Yes, Stefan, TCG target maintainers would be the main stakeholders. To
> some extent, various Machine maintainers would also be stakeholders,
> but they will most likely come back to TCG target maintainers looking
> for solution. In a literal sense, a number of maintainers were
> initially going to be very unhappy seeing the results (for example,
> seeing that the machine or entire target performs poorly compared to
> similar machines/targets), but after a while they should and will
> become happy realizing the problem was identified, and the culprit is
> at least approximately determined.
> 
> I intentionally wanted to keep the project description simple in order
> to be realistic and not develop high expectation among any of us. And
> if the student proved to be capable, it will be very easy to add some
> more useful tasks for him in this area, to be included in his/hers
> GSoC/Outreachy activities.
> 
> He had just today one case of performance degradation identified manually:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg06326.html
> 
> This project aims to do these kind of things easier, and possibly in
> an automated way. Howard did this by manual measurements for one
> particular setup, but this project will cover much much more.
> 
> Thanks, Stefan, again for your interest - and everything else!

Please go ahead and add this project idea to the wiki:
https://wiki.qemu.org/Google_Summer_of_Code_2020#How_to_add_a_project_idea

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

[-- Attachment #1: Type: text/plain, Size: 445 bytes --]

>
> Please go ahead and add this project idea to the wiki:
> https://wiki.qemu.org/Google_Summer_of_Code_2020#How_to_add_a_project_idea
>

Hi, Stefan,

I set up the proposal wiki page:

https://wiki.qemu.org/Google_Summer_of_Code_2020#Performance_Measurement.2C_Analysis.2C_and_Presentation

Anything else I need to do?

I see Feb 5, 20h European is the "organization deadline":
https://summerofcode.withgoogle.com/

Yours,
Aleksandar

> Stefan

[-- Attachment #2: Type: text/html, Size: 871 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 498 bytes --]

On Tue, Feb 04, 2020 at 05:52:09PM +0100, Aleksandar Markovic wrote:
> >
> > Please go ahead and add this project idea to the wiki:
> > https://wiki.qemu.org/Google_Summer_of_Code_2020#How_to_add_a_project_idea
> >
> 
> Hi, Stefan,
> 
> I set up the proposal wiki page:
> 
> https://wiki.qemu.org/Google_Summer_of_Code_2020#Performance_Measurement.2C_Analysis.2C_and_Presentation
> 
> Anything else I need to do?

Thanks!  Your idea is included in QEMU's GSoC 2020 effort.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Wainer dos Santos Moschetta]

On 1/21/20 12:07 PM, Aleksandar Markovic wrote:
> On Mon, Jan 20, 2020 at 3:51 PM Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Sat, Jan 18, 2020 at 03:08:37PM +0100, Aleksandar Markovic wrote:
>>> 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
>> Tracking performance is a good idea and something that has not been done
>> upstream yet.
> Thanks for the interest, Stefan!
>
>>   A few questions:
>>
>>   * Will benchmarks be run automatically (e.g. nightly or weekly) on
>>     someone's hardware or does every TCG architecture maintainer need to
>>     run them manually for themselves?
> If the community wants it, definitely yes. Once the methodology is
> developed, it should be straightforward to setup nightly and/or weekly
> benchmarks - that could definitely include sending mails with reports
> to the entire list or just individuals or subgroups. The recipient
> choice is just a matter or having decent criteria about
> appropriateness of information within the message (e.g. not to flood
> the list with the data most people are not really interested).
>
> For linux-user tests, they are typically very quick, and nightly tests
> are quite feasible to run. On someone hardware, of course, and
> consistently always on the same hardware, if possible. If it makes
> sense, one could setup multiple test beds with a variety of hardware
> setups.
>
> For system mode tests, I knoe they are much more difficult to
> automate, and, on top of that, there could be greater risk of
> hangs/crashes Also, considering the number of machines we support,
> those tests could consume much more time - perhaps even one day would
> not be sufficient, if we have many machines and boot/shutdown
> variants. For these reason, perhaps weekly executions would be more
> appropriate for them, and, in general, given greater complexity, the
> expectation from system-mode performance tests should be better kept
> quite low for now.
>
>>   * Where will the benchmark result history be stored?
>>
> If emailing is set up, the results could be reconstructed from emails.
> But, yes, it would be better if the result history is kept somewhere
> on an internet-connected file server


If you eventually choose Gitlab CI for weekly/nightly executions then 
results can be simply archived [1].

Also it can be attached machines in Gitlab CI then running the 
system-mode experiment always on same environment.

[1] https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html

IMHO, it is a very good GSoC proposal.

- Wainer

>
> Yours,
> Aleksandar
>
>> Stefan

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Lukáš Doktor]

[-- Attachment #1.1: Type: text/plain, Size: 4573 bytes --]

Dne 18. 01. 20 v 15:08 Aleksandar Markovic napsal(a):
> Hi, everybody.
> 
> I am going to propose several ideas for QEMU participation in GSoC/Outreachy in next few days. This is the first one. Please feel free to give an honest feedback.
> 
> Yours,
> Aleksandar
> 

Hello Aleksandr,

sounds like a good plan, I'd like to be involved as well.

Why? At Rad Hat I'm exploring a way to monitor qemu performance. At this point it's x86_64 whole system only, but it should be flexible enough to work on various setups. Good news is we're in a process of upstreamizing our setup so it might actually serve for the part II of your proposal. It's not ready yet as it contains many ugly and downstream parts, but I'm replacing the custom modules with Ansible and cleaning things from internal parts as having it upstream is a high priority at this point. Our motivation is to allow public upstream testing (again, starting with x86, but more will hopefully come).

Your proposal is fairly generic, I'm wondering which way it will turn. I like the part I, it might catch low-level changes and should lower the variability of results. In part II I'm a bit scared of how the scope will grow (based on what I saw in my experiment). You have host, host kernel, host system, qemu, guest kernel, guest system and than the tested app, which might result in a great jitter. Additionally qemu contains many features that need to be utilized, you have various disk formats, block devices, various passthrough options, ... as well as host/guest tune settings. It's gonna be hard to not to get lost in the depth and to deliver something useful while extendable for the future...

Anyway, please keep me in the loop and good luck with leading this into the right direction...

Regards,
Lukáš

> 
> 
> *Measure and Analyze Performance of
> QEMU User and System Mode Emulation*
> 
> 
> _/PLANNED ACTIVITIES/_
> 
> PART I: (user mode)
> 
>    a) select around a dozen test programs (resembling components of SPEC benchmark, but must be open source, and preferably license compatible with QEMU); test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
>    b) measure execution time and other performance data in user mode across all platforms for ToT:
>        - try to improve performance if there is an obvious bottleneck (but this is unlikely);
>        - develop tests that will be protection against performance regressions in future.
>    c) measure execution time in user-mode for selected platforms for all QEMU versions in last 5 years:
>        - confirm performance improvements and/or detect performance degradations.
>    d) summarize all results in a comprehensive form, using also graphics/data visualization.
> 
> PART II: (system mode)
> 
>    a) measure execution time and other performance data for boot/shutdown cycle for selected machines for ToT:
>        - try to improve performance if there is an obvious bottleneck;
>        - develop tests that will be protection against performance regressions in future.
>    b) summarize all results in a comprehensive form.
> 
> 
> /_DELIVERABLES_/
> 
> 1) Each maintainer for target will be given a list of top 25 functions in terms of spent host time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
> 
> 2) Each maintainer for machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
> 
> 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
> 
> (parts 1) and 2) will be, of course, published to everybody, maintainers are simply singled out as main recipients and decision-makers on possible next action items)
> 
> Deliverable will be distributed over wide time interval (in other words, they will not be presented just at the end of project, but gradually during project execution).
> 
> 
> /Mentor:/ Aleksandar Markovic (myself) (but, I am perfectly fine if somebody else wants to mentor the project, if interested)
> 
> /Student:/ open
> 
> 
> That would be all, feel free to ask for additional info and/or clarification.
>  



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [GSoC/Outreachy QEMU project proposal] Measure and Analyze QEMU Performance

[Aleksandar Markovic]

On Tue, Jan 21, 2020 at 9:08 AM Lukáš Doktor <ldoktor@redhat.com> wrote:
>
> Dne 18. 01. 20 v 15:08 Aleksandar Markovic napsal(a):
> > Hi, everybody.
> >
> > I am going to propose several ideas for QEMU participation in GSoC/Outreachy in next few days. This is the first one. Please feel free to give an honest feedback.
> >
> > Yours,
> > Aleksandar
> >
>
> Hello Aleksandr,
>
> sounds like a good plan, I'd like to be involved as well.
>

Sure, I am glad to heard this.

> Why? At Rad Hat I'm exploring a way to monitor qemu performance. At this point it's x86_64 whole system only, but it should be flexible enough to work on various setups. Good news is we're in a process of upstreamizing our setup so it might actually serve for the part II of your proposal. It's not ready yet as it contains many ugly and downstream parts, but I'm replacing the custom modules with Ansible and cleaning things from internal parts as having it upstream is a high priority at this point. Our motivation is to allow public upstream testing (again, starting with x86, but more will hopefully come).
>
> Your proposal is fairly generic, I'm wondering which way it will turn. I like the part I, it might catch low-level changes and should lower the variability of results. In part II I'm a bit scared of how the scope will grow (based on what I saw in my experiment). You have host, host kernel, host system, qemu, guest kernel, guest system and than the tested app, which might result in a great jitter. Additionally qemu contains many features that need to be utilized, you have various disk formats, block devices, various passthrough options, ... as well as host/guest tune settings. It's gonna be hard to not to get lost in the depth and to deliver something useful while extendable for the future...
>

My first impression is that your work and this proposal could be
viewed much more as complementary, rather than largely overlapping.

Yes, I am quite aware of the problem of data explosion, and I already
explore different possibilities of dealing with it.

Also, a student realistically can't do aweful lot of difficult work
for 3 or 4 months, so I plan to focus on simplicity, and the community
could further develop something more complex, if needed.

> Anyway, please keep me in the loop and good luck with leading this into the right direction...
>

Definitely, and thanks!

Best regards,
Aleksandar

> Regards,
> Lukáš
>
> >
> >
> > *Measure and Analyze Performance of
> > QEMU User and System Mode Emulation*
> >
> >
> > _/PLANNED ACTIVITIES/_
> >
> > PART I: (user mode)
> >
> >    a) select around a dozen test programs (resembling components of SPEC benchmark, but must be open source, and preferably license compatible with QEMU); test programs should be distributed like this: 4-5 FPU CPU-intensive, 4-5 non-FPU CPU intensive, 1-2 I/O intensive;
> >    b) measure execution time and other performance data in user mode across all platforms for ToT:
> >        - try to improve performance if there is an obvious bottleneck (but this is unlikely);
> >        - develop tests that will be protection against performance regressions in future.
> >    c) measure execution time in user-mode for selected platforms for all QEMU versions in last 5 years:
> >        - confirm performance improvements and/or detect performance degradations.
> >    d) summarize all results in a comprehensive form, using also graphics/data visualization.
> >
> > PART II: (system mode)
> >
> >    a) measure execution time and other performance data for boot/shutdown cycle for selected machines for ToT:
> >        - try to improve performance if there is an obvious bottleneck;
> >        - develop tests that will be protection against performance regressions in future.
> >    b) summarize all results in a comprehensive form.
> >
> >
> > /_DELIVERABLES_/
> >
> > 1) Each maintainer for target will be given a list of top 25 functions in terms of spent host time for each benchmark described in the previous section. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
> >
> > 2) Each maintainer for machine (that has successful boot/shutdown cycle) will be given a list of top 25 functions in terms of spent host time during boot/shutdown cycle. Additional information and observations will be also provided, if the judgment is they are useful and relevant.
> >
> > 3) The community will be given all devised performance measurement methods in the form of easily reproducible step-by-step setup and execution procedures.
> >
> > (parts 1) and 2) will be, of course, published to everybody, maintainers are simply singled out as main recipients and decision-makers on possible next action items)
> >
> > Deliverable will be distributed over wide time interval (in other words, they will not be presented just at the end of project, but gradually during project execution).
> >
> >
> > /Mentor:/ Aleksandar Markovic (myself) (but, I am perfectly fine if somebody else wants to mentor the project, if interested)
> >
> > /Student:/ open
> >
> >
> > That would be all, feel free to ask for additional info and/or clarification.
> >
>
>