Re: [OE-core] [PATCH v2 0/5] Improvements for performance test report view

Ninette Adhikari
2024-05-13T15:34:31Z

[-- Attachment #1.1: Type: text/plain, Size: 13310 bytes --]

Thanks for your response Randy:)
For the Stdev error in the chart I have opened a ticket here:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15482
Hope this helps!

Thanks,
Ninette

On Wed, May 8, 2024 at 5:54 PM Randy MacLeod <randy.macleod@windriver.com>
wrote:

> On 2024-05-07 10:17 a.m., Ninette Adhikari wrote:
>
> Hi Randy,
>
> Thank you so much for the detailed feedback. This is very helpful:) Here
> are my comments:
>
> - whitespace warning: Thanks for the catch. I'll make sure to check
> the patches next time to make sure there are no warnings.
> - local setup: You are right this can be updated. Do I need to update
> this in the cover letter and resend it? Let me know, then I can also
> include the update on --commit argument.
>
> Since this is only in the cover letter and not in a commit and, since I
> was able to make use of the work and document how I did that
> on the email thread, so I won't insist on a v3.
>
>
> - stdev as nan: I was able to reproduce the issue and I think this is
> where the nan error is being generated.
> https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331
> <https://urldefense.com/v3/__https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py*n331__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4fhQcEM2g$>.
> This error persists in the current master branch. Unfortunately I'm
> not exactly sure on the logic of how this is generated and was not able to
> resolve this issue. If you have any suggestions on how to proceed do let me
> know.
>
> I'm not sure off-hand. Would you open a defect in
> bugzilla.yoctoproject.org to record the problem and what you know about
> it?
>
>
> - vertical dashed line (VDL) behavior: I tried to troubleshoot this
> but it seems like Apache echarts tooltip
> <https://urldefense.com/v3/__https://echarts.apache.org/en/option.html*tooltip__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4d-aXgexg$>
> is only triggered when there is a data point in the x axis. And in your
> example the data points are on the edges of the bar. Some of the charts do
> have continuous data points and hence seems smoother. Hope this is not too
> much of a holdup.
>
> Well, now that I've seen it, I can't unsee it and my OCD is triggered! ;-)
>
> I don't think it's a big problem for anyone else so let's not worry about
> it unless you'd like to report it in the YP bugzilla as an enhancement.
>
> Thanks again,
>
> ../Randy
>
>
>
> - [image: tooltip.gif]
>
>
> Let me know if you have any other questions. Thanks again!
>
> Ninette
>
> On Fri, May 3, 2024 at 8:10 PM Randy MacLeod <randy.macleod@windriver.com>
> wrote:
>
>> On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org
>> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>
>> wrote:
>>
>> On 2024-05-03 10:43 a.m., Ninette Adhikari wrote:
>>
>> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
>> The current report can be accessed here:
>> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
>> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
>> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes.
>> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.
>>
>> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:
>>
>> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
>> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts.
>> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
>> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
>>
>> The tooltips are very nice and useful so
>>
>> - The charts are created as step charts instead of plain line charts.
>> - Add start time, size, and commit number to tooltip.
>> - Add dark mode view
>>
>> Updated report screenshots: https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0ahttps://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208
>>
>>
>> Very nice! Thanks for the work and the v2 improvements.
>>
>>
>> I applied the patches by saving them from email and there was a minor
>> whitespace warning which
>> is really nothing to worry about but if you want to avoid such noise, you
>> could send yourself
>> patches by email and then apply them in a different branch if you want to
>> check for such problems.
>>
>>
>> ❯ git am /tmp/ninette-v2/*
>> Applying: oe-build-perf-report: Add apache echarts to make report
>> interactive
>> Applying: oe-build-perf-report: Display more than 300 commits and date
>> instead of commit number
>> Applying: oe-build-perf-report: Improve report styling and add
>> descriptions
>> .git/rebase-apply/patch:240: trailing whitespace.
>> start_time = time
>> warning: 1 line adds whitespace errors.
>> Applying: oe-build-perf-report: Update chart tooltip and chart type
>> Applying: oe-build-perf-report: Add dark mode
>>
>> poky.git on ninette-v2 [$?]
>> ❯ ls /tmp/ninette-v2/*
>> '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report: Add
>> apache echarts to make report interactive - "Ninette Adhikari via
>> lists.openembedded.org
>> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>"
>> <ninette=thehoodiefirm.com@lists.openembedded.org>
>> <ninette=thehoodiefirm.com@lists.openembedded.org> - 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display more than
>> 300 commits and date instead of commit number - Ninette Adhikari
>> <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03
>> 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve report
>> styling and add descriptions - Ninette Adhikari
>> <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com> - 2024-05-03
>> 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update chart
>> tooltip and chart type - Ninette Adhikari <ninette@thehoodiefirm.com>
>> <ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark mode -
>> Ninette Adhikari <ninette@thehoodiefirm.com> <ninette@thehoodiefirm.com>
>> - 2024-05-03 1043.eml'
>>
>> For local setup, you can do the following:
>>
>> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)
>>
>> 2. In the poky repository run the following to build the report HTML:
>> ```bash
>> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
>>
>> These are the same setup step as last time I think and again, they aren't
>> quite right but
>> the script tells you do do:
>>
>> ❯ git fetch origin
>> refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86
>>
>>
>> and that works.
>>
>>
>> ```
>> Note:
>> - Add your local path to the yocto-buildstats repo
>> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
>> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.
>>
>> Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to be a
>> commit ID in the yocto-buildstats repo.
>>
>>
>> One thing that you can think about fixing or just ignore is that the
>> --commit arg needs to be the
>> full commit hash not the shortened version. i.e. this doesn't work:
>>
>> ❯ ./scripts/oe-build-perf-report -r
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>> --branch "master" --commit "663f180574" --branch2 "master" --html >
>> /tmp/ninette-v2-test-663f180574.html
>> Traceback (most recent call last):
>> File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>> line 617, in <module>
>> sys.exit(main())
>> File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>> line 532, in main
>> index1 = gitarchive.rev_find(revs, 'commit', args.commit)
>> File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py",
>> line 282, in rev_find
>> raise ValueError("Unable to find '{}' value '{}'".format(attr, val))
>> ValueError: Unable to find 'commit' value '663f180574'
>>
>> Since the script is going to be used in other scripts run on the
>> autobuilder, this is a low priority enhancement.
>>
>>
>> I was happy to see that a newer poky commit, worked fine:
>>
>> ❯ ./scripts/oe-build-perf-report -r
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>> --branch "master" --commit "632e3170595bb32717c0471a55a619b4b33fe787"
>> --branch2 "master" --html >
>> /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html
>> INFO: Parsing buildstats from
>> 'refs/notes/buildstats/perf-debian11/master/qemux86'
>>
>>
>> but I did notice: Stdev: nan
>>
>>
>> whereas in your linked charts, and in my first run with commit:
>> 663f1805742ff6fb6955719d0ab7846a425debcf
>>
>> the correct numerical standard deviation is displayed.
>>
>> I can debug this is you can't reproduce it yourself.
>>
>> * if you can't ...
>>
>>
>> Also, I wanted to draw your attention to how the vertical dashed line
>> (VDL) behaves
>> to see if it's something that can (or should) be changed.
>>
>> If you zoom in on a graph the VDL is placed at the left edge of bar
>> until the pointer passes the mid-point of the bar:
>>
>>
>> at which time the VDL moves to the next bar/data point:
>>
>>
>>
>> It's a small thing but it would be nice if:
>> 1) the VDL was placed in the centre of the bar and
>> 2) the commit number /tooltip data only changed when the pointer move
>> to a different bar
>> so the same data would be displayed as one swept across a given bar.
>>
>> What do you think?
>>
>>
>> Picky, eh! ;-)
>>
>>
>> That's all for now.
>>
>> ../Randy
>>
>>
>>
>> Again. these reports and beautiful interactive charts are great to see.
>>
>> Thanks!
>>
>> ../Randy
>>
>>
>> Ninette Adhikari (5):
>> oe-build-perf-report: Add apache echarts to make report interactive
>> oe-build-perf-report: Display more than 300 commits and date instead
>> of commit number
>> oe-build-perf-report: Improve report styling and add descriptions
>> oe-build-perf-report: Update chart tooltip and chart type
>> oe-build-perf-report: Add dark mode
>>
>> .../build_perf/html/measurement_chart.html | 140 ++++++++++++------
>> scripts/lib/build_perf/html/report.html | 124 +++++++++++-----
>> scripts/lib/build_perf/report.py | 5 +-
>> scripts/oe-build-perf-report | 6 +-
>> 4 files changed, 193 insertions(+), 82 deletions(-)
>>
>>
>>
>> --
>> # Randy MacLeod
>> # Wind River Linux
>>
>>
>>
>>
>>
>>
>> --
>> # Randy MacLeod
>> # Wind River Linux
>>
>>
>
> --
> Ninette Adhikari
> Software developer
> The Neighbourhoodie Software GmbH
> Harzer Straße 39, 12059 Berlin
> neighbourhood.ie
> <https://urldefense.com/v3/__http://neighbourhood.ie__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4ey5Kw-Rg$>
>
> Handelsregister HRB 157851 B Amtsgericht Charlottenburg
> Geschäftsführung: Jan Lehnardt, Simone Haas
>
>
> --
> # Randy MacLeod
> # Wind River Linux
>
>

[-- Attachment #1.2: Type: text/html, Size: 21788 bytes --]

[-- Attachment #2: tooltip.gif --]
[-- Type: image/gif, Size: 248998 bytes --]

[-- Attachment #3: ipZPodcPGxdg4E4H.png --]
[-- Type: image/png, Size: 65076 bytes --]

[-- Attachment #4: EVQkTyvIsYDJlcL7.png --]
[-- Type: image/png, Size: 2280301 bytes --]

[-- Attachment #5: ceOAY8Zii7KShL9e.png --]
[-- Type: image/png, Size: 2068675 bytes --]