Re: [OE-core] [PATCH v2 0/5] Improvements for performance test report view

Randy MacLeod
2024-05-08T15:54:19Z

[-- Attachment #1: Type: text/plain, Size: 14195 bytes --]

On 2024-05-07 10:17 a.m., Ninette Adhikari wrote:
> Hi Randy,
>
> Thank you so much for the detailed feedback. This is very helpful:)
> Here are my comments:
>
> * whitespace warning: Thanks for the catch. I'll make sure to check
> the patches next time to make sure there are no warnings.
> * local setup: You are right this can be updated. Do I need to
> update this in the cover letter and resend it? Let me know, then I
> can also include the update on --commit argument.
>
Since this is only in the cover letter and not in a commit and, since I
was able to make use of the work and document how I did that
on the email thread, so I won't insist on a v3.
>
> * stdev as nan: I was able to reproduce the issue and I think this
> is where the nan error is being generated.
> https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331
> <https://urldefense.com/v3/__https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py*n331__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4fhQcEM2g$>.
> This error persists in the current master branch. Unfortunately
> I'm not exactly sure on the logic of how this is generated and was
> not able to resolve this issue. If you have any suggestions on how
> to proceed do let me know.
>
I'm not sure off-hand. Would you open a defect in
bugzilla.yoctoproject.org to record the problem and what you know about it?
>
> * vertical dashed line (VDL) behavior: I tried to troubleshoot this
> but it seems like Apache echarts tooltip
> <https://urldefense.com/v3/__https://echarts.apache.org/en/option.html*tooltip__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4d-aXgexg$>
> is only triggered when there is a data point in the x axis. And in
> your example the data points are on the edges of the bar. Some of
> the charts do have continuous data points and hence seems
> smoother. Hope this is not too much of a holdup.
>
Well, now that I've seen it, I can't unsee it and my OCD is triggered! ;-)

I don't think it's a big problem for anyone else so let's not worry
about it unless you'd like to report it in the YP bugzilla as an
enhancement.

Thanks again,

../Randy

> * tooltip.gif
>
> Let me know if you have any other questions. Thanks again!
>
> Ninette
>
> On Fri, May 3, 2024 at 8:10 PM Randy MacLeod
> <randy.macleod@windriver.com> wrote:
>
> On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org
> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>
> wrote:
>> On 2024-05-03 10:43 a.m., Ninette Adhikari wrote:
>>> This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
>>> The current report can be accessed here:
>>> Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
>>> The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
>>> The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes.
>>> The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.
>>>
>>> The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:
>>>
>>> - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
>>> - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts.
>>> - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
>>> - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
>> The tooltips are very nice and useful so
>>> - The charts are created as step charts instead of plain line charts.
>>> - Add start time, size, and commit number to tooltip.
>>> - Add dark mode view
>>>
>>> Updated report screenshots:
>>> https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
>>> https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003
>>> https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a
>>> https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208
>>
>>
>> Very nice! Thanks for the work and the v2 improvements.
>>
>>
>> I applied the patches by saving them from email and there was a
>> minor whitespace warning which
>> is really nothing to worry about but if you want to avoid such
>> noise, you could send yourself
>> patches by email and then apply them in a different branch if you
>> want to check for such problems.
>>
>>
>> ❯ git am /tmp/ninette-v2/*
>> Applying: oe-build-perf-report: Add apache echarts to make report
>> interactive
>> Applying: oe-build-perf-report: Display more than 300 commits and
>> date instead of commit number
>> Applying: oe-build-perf-report: Improve report styling and add
>> descriptions
>> .git/rebase-apply/patch:240: trailing whitespace.
>>     start_time = time
>> warning: 1 line adds whitespace errors.
>> Applying: oe-build-perf-report: Update chart tooltip and chart type
>> Applying: oe-build-perf-report: Add dark mode
>>
>> poky.git on ninette-v2 [$?]
>> ❯ ls /tmp/ninette-v2/*
>> '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report:
>> Add apache echarts to make report interactive - "Ninette Adhikari
>> via lists.openembedded.org
>> <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>"
>> <ninette=thehoodiefirm.com@lists.openembedded.org>
>> <mailto:ninette=thehoodiefirm.com@lists.openembedded.org> -
>> 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display
>> more than 300 commits and date instead of commit number - Ninette
>> Adhikari <ninette@thehoodiefirm.com>
>> <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve
>> report styling and add descriptions - Ninette Adhikari
>> <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> -
>> 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update
>> chart tooltip and chart type - Ninette Adhikari
>> <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> -
>> 2024-05-03 1043.eml'
>> '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark
>> mode - Ninette Adhikari <ninette@thehoodiefirm.com>
>> <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
>>
>>> For local setup, you can do the following:
>>>
>>> 1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)
>>>
>>> 2. In the poky repository run the following to build the report HTML:
>>> ```bash
>>> ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
>>
>> These are the same setup step as last time I think and again,
>> they aren't quite right but
>> the script tells you do do:
>>
>> ❯ git fetch origin
>> refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86
>>
>>
>> and that works.
>>
>>
>>> ```
>>> Note:
>>> - Add your local path to the yocto-buildstats repo
>>> - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
>>> - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.
>>
>> Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to
>> be a commit ID in the yocto-buildstats repo.
>>
>>
>> One thing that you can think about fixing or just ignore is that
>> the --commit arg needs to be the
>> full commit hash not the shortened version. i.e. this doesn't work:
>>
>> ❯ ./scripts/oe-build-perf-report -r
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>> --branch "master" --commit "663f180574" --branch2 "master" --html
>> > /tmp/ninette-v2-test-663f180574.html
>> Traceback (most recent call last):
>>   File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>> line 617, in <module>
>>     sys.exit(main())
>>   File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>> line 532, in main
>>     index1 = gitarchive.rev_find(revs, 'commit', args.commit)
>>   File
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py",
>> line 282, in rev_find
>>     raise ValueError("Unable to find '{}' value
>> '{}'".format(attr, val))
>> ValueError: Unable to find 'commit' value '663f180574'
>>
>> Since the script is going to be used in other scripts run on the
>> autobuilder, this is a low priority enhancement.
>>
>>
>> I was happy to see that a newer poky commit, worked fine:
>>
>> ❯ ./scripts/oe-build-perf-report -r
>> "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>> --branch "master" --commit
>> "632e3170595bb32717c0471a55a619b4b33fe787" --branch2 "master"
>> --html >
>> /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html
>> INFO: Parsing buildstats from
>> 'refs/notes/buildstats/perf-debian11/master/qemux86'
>>
>>
>> but I did notice:  Stdev: nan
>>
>>
>> whereas in your linked charts, and in my first run with commit:
>> 663f1805742ff6fb6955719d0ab7846a425debcf
>>
>> the correct numerical standard deviation is displayed.
>>
>> I can debug this is you can't reproduce it yourself.
>>
> * if you can't  ...
>
>
> Also, I wanted to draw your attention to how the vertical dashed
> line (VDL) behaves
> to see if it's something that can (or should) be changed.
>
> If you zoom in on a graph the VDL is placed at the left edge of
> bar until the pointer passes the mid-point of the bar:
>
>
> at which time the VDL moves to the next bar/data point:
>
>
>
> It's a small thing but it would be nice if:
>   1) the VDL was placed in the centre of the bar and
>   2) the commit number /tooltip data only changed when the pointer
> move to a different bar
>       so the same data would be displayed as one swept across a
> given bar.
>
> What do you think?
>
>
> Picky, eh! ;-)
>
>
> That's all for now.
>
> ../Randy
>
>
>>
>> Again. these reports and beautiful interactive charts are great
>> to see.
>>
>> Thanks!
>>
>> ../Randy
>>
>>
>>> Ninette Adhikari (5):
>>> oe-build-perf-report: Add apache echarts to make report interactive
>>> oe-build-perf-report: Display more than 300 commits and date instead
>>> of commit number
>>> oe-build-perf-report: Improve report styling and add descriptions
>>> oe-build-perf-report: Update chart tooltip and chart type
>>> oe-build-perf-report: Add dark mode
>>>
>>> .../build_perf/html/measurement_chart.html | 140 ++++++++++++------
>>> scripts/lib/build_perf/html/report.html | 124 +++++++++++-----
>>> scripts/lib/build_perf/report.py | 5 +-
>>> scripts/oe-build-perf-report | 6 +-
>>> 4 files changed, 193 insertions(+), 82 deletions(-)
>>>
>>
>> --
>> # Randy MacLeod
>> # Wind River Linux
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#199007):https://lists.openembedded.org/g/openembedded-core/message/199007
>> Mute This Topic:https://lists.openembedded.org/mt/105889450/3616765
>> Group Owner:openembedded-core+owner@lists.openembedded.org
>> Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub [randy.macleod@windriver.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>
> --
> # Randy MacLeod
> # Wind River Linux
>
>
>
> --
> Ninette Adhikari
> Software developer
> The Neighbourhoodie Software GmbH
> Harzer Straße 39, 12059 Berlin
> neighbourhood.ie
> <https://urldefense.com/v3/__http://neighbourhood.ie__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4ey5Kw-Rg$>
>
> Handelsregister HRB 157851 B Amtsgericht Charlottenburg
> Geschäftsführung: Jan Lehnardt, Simone Haas

--
# Randy MacLeod
# Wind River Linux

[-- Attachment #2.1: Type: text/html, Size: 22747 bytes --]

[-- Attachment #2.2: tooltip.gif --]
[-- Type: image/gif, Size: 248998 bytes --]

[-- Attachment #2.3: ipZPodcPGxdg4E4H.png --]
[-- Type: image/png, Size: 65076 bytes --]

[-- Attachment #2.4: EVQkTyvIsYDJlcL7.png --]
[-- Type: image/png, Size: 2280301 bytes --]

[-- Attachment #2.5: ceOAY8Zii7KShL9e.png --]
[-- Type: image/png, Size: 2068675 bytes --]