[REPORT] [GSoC - TCG Continuous Benchmarking] [#5] Finding Commits Affecting QEMU Performance

[REPORT] [GSoC - TCG Continuous Benchmarking] [#5] Finding Commits Affecting QEMU Performance

[Ahmed Karaman]

Hi,

The fifth report of the TCG Continuous Benchmarking project concludes
a mini-series of three reports that dealt with the performance
comparison and analysis of QEMU 5.0 and 5.1-pre-soft-freeze.

The report presents a new Python script that utilizes "git bisect" for
running a binary search within a specified range of commits to
automatically detect the commit causing a performance improvement or
degradation.

The new script is then used to find the commit introducing the PowerPC
performance degradation as well as that introducing the performance
improvement in MIPS. The results obtained for both commits proves the
correctness of the conclusions and analyses presented in the two
previous reports.

Report link:
https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/Finding-Commits-Affecting-QEMU-Performance/

Previous reports:
Report 1 - Measuring Basic Performance Metrics of QEMU:
https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06692.html
Report 2 - Dissecting QEMU Into Three Main Parts:
https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg09441.html
Report 3 - QEMU 5.0 and 5.1-pre-soft-freeze Dissect Comparison:
https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg01978.html
Report 4 - Listing QEMU Helpers and Function Callees:
https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg04227.html

Best regards,
Ahmed Karaman

Re: [REPORT] [GSoC - TCG Continuous Benchmarking] [#5] Finding Commits Affecting QEMU Performance

[Alex Bennée]

Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:

> Hi,
>
> The fifth report of the TCG Continuous Benchmarking project concludes
> a mini-series of three reports that dealt with the performance
> comparison and analysis of QEMU 5.0 and 5.1-pre-soft-freeze.
>
> The report presents a new Python script that utilizes "git bisect" for
> running a binary search within a specified range of commits to
> automatically detect the commit causing a performance improvement or
> degradation.

Excellent stuff.

> The new script is then used to find the commit introducing the PowerPC
> performance degradation as well as that introducing the performance
> improvement in MIPS. The results obtained for both commits proves the
> correctness of the conclusions and analyses presented in the two
> previous reports.

I can certainly envision a mechanism where 0673ec slows things down. I
wonder if it would come back if instead of inline function calls we
ended up making concrete flattend versions, e.g.:

bool QEMU_FLATTEN float64_eq(float64 a, float64 b, float_status *s)
{
    return float64_compare(a, b, s) == float_relation_equal;
}

PPC is of course more affected by these changes than others because
HARDFLOAT never gets a chance to kick in. Looking at the objdump of
f64_compare there should surely be an opportunity to loose some of the
branches when looking for a certain test result?

>
> Report link:
> https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/Finding-Commits-Affecting-QEMU-Performance/
>
> Previous reports:
> Report 1 - Measuring Basic Performance Metrics of QEMU:
> https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06692.html
> Report 2 - Dissecting QEMU Into Three Main Parts:
> https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg09441.html
> Report 3 - QEMU 5.0 and 5.1-pre-soft-freeze Dissect Comparison:
> https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg01978.html
> Report 4 - Listing QEMU Helpers and Function Callees:
> https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg04227.html
>
> Best regards,
> Ahmed Karaman


-- 
Alex Bennée

Re: [REPORT] [GSoC - TCG Continuous Benchmarking] [#5] Finding Commits Affecting QEMU Performance

[Ahmed Karaman]

On Tue, Jul 21, 2020 at 1:54 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Ahmed Karaman <ahmedkhaledkaraman@gmail.com> writes:
>
> > Hi,
> >
> > The fifth report of the TCG Continuous Benchmarking project concludes
> > a mini-series of three reports that dealt with the performance
> > comparison and analysis of QEMU 5.0 and 5.1-pre-soft-freeze.
> >
> > The report presents a new Python script that utilizes "git bisect" for
> > running a binary search within a specified range of commits to
> > automatically detect the commit causing a performance improvement or
> > degradation.
>
> Excellent stuff.

Thanks for your continued support!

>
> > The new script is then used to find the commit introducing the PowerPC
> > performance degradation as well as that introducing the performance
> > improvement in MIPS. The results obtained for both commits proves the
> > correctness of the conclusions and analyses presented in the two
> > previous reports.
>
> I can certainly envision a mechanism where 0673ec slows things down. I
> wonder if it would come back if instead of inline function calls we
> ended up making concrete flattend versions, e.g.:
>
> bool QEMU_FLATTEN float64_eq(float64 a, float64 b, float_status *s)
> {
>     return float64_compare(a, b, s) == float_relation_equal;
> }
>
> PPC is of course more affected by these changes than others because
> HARDFLOAT never gets a chance to kick in. Looking at the objdump of
> f64_compare there should surely be an opportunity to loose some of the
> branches when looking for a certain test result?

Interesting, I will try to tinker a little bit with the float64
functions and will let you know if I find anything interesting.

>
> >
> > Report link:
> > https://ahmedkrmn.github.io/TCG-Continuous-Benchmarking/Finding-Commits-Affecting-QEMU-Performance/
> >
> > Previous reports:
> > Report 1 - Measuring Basic Performance Metrics of QEMU:
> > https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg06692.html
> > Report 2 - Dissecting QEMU Into Three Main Parts:
> > https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg09441.html
> > Report 3 - QEMU 5.0 and 5.1-pre-soft-freeze Dissect Comparison:
> > https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg01978.html
> > Report 4 - Listing QEMU Helpers and Function Callees:
> > https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg04227.html
> >
> > Best regards,
> > Ahmed Karaman
>
>
> --
> Alex Bennée

Best regards,
Ahmed Karaman