Re: [OE-core] [PATCH v2 0/5] Improvements for performance test report view

[Randy MacLeod <randy.macleod@windriver.com>]

[-- Attachment #1: Type: text/plain, Size: 14195 bytes --]

On 2024-05-07 10:17 a.m., Ninette Adhikari wrote:
> Hi Randy,
>
> Thank you so much for the detailed feedback. This is very helpful:) 
> Here are my comments:
>
>   * whitespace warning: Thanks for the catch. I'll make sure to check
>     the patches next time to make sure there are no warnings.
>   * local setup: You are right this can be updated. Do I need to
>     update this in the cover letter and resend it? Let me know, then I
>     can also include the update on --commit argument.
>
Since this is only in the cover letter and not in a commit and, since I 
was able to make use of the work and document how I did that
on the email thread, so I won't insist on a v3.
>
>   * stdev as nan: I was able to reproduce the issue and I think this
>     is where the nan error is being generated.
>     https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py#n331
>     <https://urldefense.com/v3/__https://git.yoctoproject.org/poky/tree/scripts/lib/build_perf/report.py*n331__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4fhQcEM2g$>.
>     This error persists in the current master branch. Unfortunately
>     I'm not exactly sure on the logic of how this is generated and was
>     not able to resolve this issue. If you have any suggestions on how
>     to proceed do let me know.
>
I'm not sure off-hand. Would you open a defect in 
bugzilla.yoctoproject.org to record the problem and what you know about it?
>
>   * vertical dashed line (VDL) behavior: I tried to troubleshoot this
>     but it seems like Apache echarts tooltip
>     <https://urldefense.com/v3/__https://echarts.apache.org/en/option.html*tooltip__;Iw!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4d-aXgexg$>
>     is only triggered when there is a data point in the x axis. And in
>     your example the data points are on the edges of the bar. Some of
>     the charts do have continuous data points and hence seems
>     smoother. Hope this is not too much of a holdup.
>
Well, now that I've seen it, I can't unsee it and my OCD is triggered! ;-)

I don't think it's a big problem for anyone else so let's not worry 
about it unless you'd like to report it in the YP bugzilla as an 
enhancement.

Thanks again,

../Randy


>   * tooltip.gif
>
> Let me know if you have any other questions. Thanks again!
>
> Ninette
>
> On Fri, May 3, 2024 at 8:10 PM Randy MacLeod 
> <randy.macleod@windriver.com> wrote:
>
>     On 2024-05-03 1:22 p.m., Randy MacLeod via lists.openembedded.org
>     <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>
>     wrote:
>>     On 2024-05-03 10:43 a.m., Ninette Adhikari wrote:
>>>     This work is done according to "Milestone 9: Build performance test report view" as stated in the Scope of Work with Sovereign Tech Fund (STF) (https://www.sovereigntechfund.de/).
>>>     The current report can be accessed here:
>>>     Performance test report HTML (https://autobuilder.yocto.io/pub/non-release/20240117-15/testresults/buildperf-alma8/perf-alma8_master_20240117090048_663f180574.html)
>>>     The report is created using the `oe-build-perf-report` script in the poky repository. This script generates a performance test report in HTML format using the data from the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) repository.
>>>     The report displays the performance test results in line chart format. The chart x-axis represents the commit numbers, and the y-axis represents the test duration in minutes.
>>>     The report also includes a table that displays the measurement statistics data for each test. The report is interactive and allows users to zoom in on specific sections of the line chart.
>>>
>>>     The current report format required some updates to make it more interactive and user-friendly. And this patch addresses such improvements:
>>>
>>>     - Add [Apache echart](https://echarts.apache.org/en/index.html) library to create oe build performance report charts and make them interactive.
>>>     - Restructure data to time and value array format to be used by echarts. It also converts test duration to minutes and adds zoom to the charts.
>>>     - Update measurement statistics data to include `start_time` so that time can be displayed instead of commit numbers on the chart. It also updates default commit history length to 300.
>>>     - Add styling updates including page margin, labels for x and y axis, tooltip, and section descriptions.
>>     The tooltips are very nice and useful so
>>>     - The charts are created as step charts instead of plain line charts.
>>>     - Add start time, size, and commit number to tooltip.
>>>     - Add dark mode view
>>>
>>>     Updated report screenshots:
>>>     https://github.com/neighbourhoodie/poky/assets/13760198/65a1890c-fd2a-40d4-ac90-f13055735e53
>>>     https://github.com/neighbourhoodie/poky/assets/13760198/b40c326b-17d2-42e2-8e41-72e52ed2c003
>>>     https://github.com/neighbourhoodie/poky/assets/13760198/cc7ec996-9dab-435a-8fdc-82a2a4193c0a
>>>     https://github.com/neighbourhoodie/poky/assets/13760198/6e0fe09d-50e5-4b0b-b70b-6943f71b5208
>>
>>
>>     Very nice! Thanks for the work and the v2 improvements.
>>
>>
>>     I applied the patches by saving them from email and there was a
>>     minor whitespace warning which
>>     is really nothing to worry about but if you want to avoid such
>>     noise, you could send yourself
>>     patches by email and then apply them in a different branch if you
>>     want to check for such problems.
>>
>>
>>     ❯ git am /tmp/ninette-v2/*
>>     Applying: oe-build-perf-report: Add apache echarts to make report
>>     interactive
>>     Applying: oe-build-perf-report: Display more than 300 commits and
>>     date instead of commit number
>>     Applying: oe-build-perf-report: Improve report styling and add
>>     descriptions
>>     .git/rebase-apply/patch:240: trailing whitespace.
>>         start_time = time
>>     warning: 1 line adds whitespace errors.
>>     Applying: oe-build-perf-report: Update chart tooltip and chart type
>>     Applying: oe-build-perf-report: Add dark mode
>>
>>     poky.git on ninette-v2 [$?]
>>     ❯ ls /tmp/ninette-v2/*
>>     '/tmp/ninette-v2/[OE-core] [PATCH v2 1_5] oe-build-perf-report:
>>     Add apache echarts to make report interactive - "Ninette Adhikari
>>     via lists.openembedded.org
>>     <https://urldefense.com/v3/__http://lists.openembedded.org__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4cx6ub3iw$>"
>>     <ninette=thehoodiefirm.com@lists.openembedded.org>
>>     <mailto:ninette=thehoodiefirm.com@lists.openembedded.org> -
>>     2024-05-03 1043.eml'
>>     '/tmp/ninette-v2/[PATCH v2 2_5] oe-build-perf-report: Display
>>     more than 300 commits and date instead of commit number - Ninette
>>     Adhikari <ninette@thehoodiefirm.com>
>>     <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
>>     '/tmp/ninette-v2/[PATCH v2 3_5] oe-build-perf-report: Improve
>>     report styling and add descriptions - Ninette Adhikari
>>     <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> -
>>     2024-05-03 1043.eml'
>>     '/tmp/ninette-v2/[PATCH v2 4_5] oe-build-perf-report: Update
>>     chart tooltip and chart type - Ninette Adhikari
>>     <ninette@thehoodiefirm.com> <mailto:ninette@thehoodiefirm.com> -
>>     2024-05-03 1043.eml'
>>     '/tmp/ninette-v2/[PATCH v2 5_5] oe-build-perf-report: Add dark
>>     mode - Ninette Adhikari <ninette@thehoodiefirm.com>
>>     <mailto:ninette@thehoodiefirm.com> - 2024-05-03 1043.eml'
>>
>>>     For local setup, you can do the following:
>>>
>>>     1. Clone the yocto-buildstats (https://git.yoctoproject.org/yocto-buildstats/) and the poky repository (https://git.yoctoproject.org/poky/)
>>>
>>>     2. In the poky repository run the following to build the report HTML:
>>>     ```bash
>>>     ./scripts/oe-build-perf-report -r "LOCAL_PATH_TO_YOCTO_BUILDSTATS" --branch "master" --commit "663f1805742ff6fb6955719d0ab7846a425debcf" --branch2 "master" --html > test.html
>>
>>     These are the same setup step as last time I think and again,
>>     they aren't quite right but
>>     the script tells you do do:
>>
>>     ❯ git fetch origin
>>     refs/notes/buildstats/perf-debian11/master/qemux86:refs/notes/buildstats/perf-debian11/master/qemux86
>>
>>
>>     and that works.
>>
>>
>>>     ```
>>>     Note:
>>>     - Add your local path to the yocto-buildstats repo
>>>     - The above command builds the report in a file called `test.html`. You can access it in the root directory in poky.
>>>     - This exmaple report uses the commit `663f1805742ff6fb6955719d0ab7846a425debcf` from `master` branch.
>>
>>     Hmmm, ah, that's a poky commit id. I (stupidly!) expected it to
>>     be a commit ID in the yocto-buildstats repo.
>>
>>
>>     One thing that you can think about fixing or just ignore is that
>>     the --commit arg needs to be the
>>     full commit hash not the shortened version. i.e. this doesn't work:
>>
>>     ❯ ./scripts/oe-build-perf-report -r
>>     "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>>     --branch "master" --commit "663f180574" --branch2 "master" --html
>>     > /tmp/ninette-v2-test-663f180574.html
>>     Traceback (most recent call last):
>>       File
>>     "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>>     line 617, in <module>
>>         sys.exit(main())
>>       File
>>     "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/./scripts/oe-build-perf-report",
>>     line 532, in main
>>         index1 = gitarchive.rev_find(revs, 'commit', args.commit)
>>       File
>>     "/media/rmacleod/gitter/rmacleod/src/distro/yocto/poky.git/meta/lib/oeqa/utils/gitarchive.py",
>>     line 282, in rev_find
>>         raise ValueError("Unable to find '{}' value
>>     '{}'".format(attr, val))
>>     ValueError: Unable to find 'commit' value '663f180574'
>>
>>     Since the script is going to be used in other scripts run on the
>>     autobuilder, this is a low priority enhancement.
>>
>>
>>     I was happy to see that a newer poky commit, worked fine:
>>
>>     ❯ ./scripts/oe-build-perf-report -r
>>     "/media/rmacleod/gitter/rmacleod/src/distro/yocto/yocto-buildstats.git"
>>     --branch "master" --commit
>>     "632e3170595bb32717c0471a55a619b4b33fe787" --branch2 "master"
>>     --html >
>>     /tmp/ninette-v2-test-632e3170595bb32717c0471a55a619b4b33fe787.html
>>     INFO: Parsing buildstats from
>>     'refs/notes/buildstats/perf-debian11/master/qemux86'
>>
>>
>>     but I did notice:  Stdev: nan
>>
>>
>>     whereas in your linked charts, and in my first run with commit:
>>     663f1805742ff6fb6955719d0ab7846a425debcf
>>
>>     the correct numerical standard deviation is displayed.
>>
>>     I can debug this is you can't reproduce it yourself.
>>
>     * if you can't  ...
>
>
>     Also, I wanted to draw your attention to how the vertical dashed
>     line (VDL) behaves
>     to see if it's something that can (or should) be changed.
>
>     If you zoom in on a graph the VDL is placed at the left edge of 
>     bar until the pointer passes the mid-point of the bar:
>
>
>     at which time the VDL moves to the next bar/data point:
>
>
>
>     It's a small thing but it would be nice if:
>       1) the VDL was placed in the centre of the bar and
>       2) the commit number /tooltip data only changed when the pointer
>     move to a different bar
>           so the same data would be displayed as one swept across a
>     given bar.
>
>     What do you think?
>
>
>     Picky, eh! ;-)
>
>
>     That's all for now.
>
>     ../Randy
>
>
>>
>>     Again. these reports and beautiful interactive charts are great
>>     to see.
>>
>>     Thanks!
>>
>>     ../Randy
>>
>>
>>>     Ninette Adhikari (5):
>>>        oe-build-perf-report: Add apache echarts to make report interactive
>>>        oe-build-perf-report: Display more than 300 commits and date instead
>>>          of commit number
>>>        oe-build-perf-report: Improve report styling and add descriptions
>>>        oe-build-perf-report: Update chart tooltip and chart type
>>>        oe-build-perf-report: Add dark mode
>>>
>>>       .../build_perf/html/measurement_chart.html    | 140 ++++++++++++------
>>>       scripts/lib/build_perf/html/report.html       | 124 +++++++++++-----
>>>       scripts/lib/build_perf/report.py              |   5 +-
>>>       scripts/oe-build-perf-report                  |   6 +-
>>>       4 files changed, 193 insertions(+), 82 deletions(-)
>>>
>>
>>     -- 
>>     # Randy MacLeod
>>     # Wind River Linux
>>
>>     -=-=-=-=-=-=-=-=-=-=-=-
>>     Links: You receive all messages sent to this group.
>>     View/Reply Online (#199007):https://lists.openembedded.org/g/openembedded-core/message/199007
>>     Mute This Topic:https://lists.openembedded.org/mt/105889450/3616765
>>     Group Owner:openembedded-core+owner@lists.openembedded.org
>>     Unsubscribe:https://lists.openembedded.org/g/openembedded-core/unsub  [randy.macleod@windriver.com]
>>     -=-=-=-=-=-=-=-=-=-=-=-
>>
>
>     -- 
>     # Randy MacLeod
>     # Wind River Linux
>
>
>
> -- 
> Ninette Adhikari
> Software developer
> The Neighbourhoodie Software GmbH
> Harzer Straße 39, 12059 Berlin
> neighbourhood.ie 
> <https://urldefense.com/v3/__http://neighbourhood.ie__;!!AjveYdw8EvQ!ZWtNp8Bad2XL45Cx1mXbHemIY3wl2IPMBdPCgq8oL8mHVd9EqQHuHxTrflNrFkuwbwokVjGdK9UtctqWrK5jy4ey5Kw-Rg$>
>
> Handelsregister HRB 157851 B Amtsgericht Charlottenburg
> Geschäftsführung: Jan Lehnardt, Simone Haas


-- 
# Randy MacLeod
# Wind River Linux

[-- Attachment #2.1: Type: text/html, Size: 22747 bytes --]

[-- Attachment #2.2: tooltip.gif --]
[-- Type: image/gif, Size: 248998 bytes --]

[-- Attachment #2.3: ipZPodcPGxdg4E4H.png --]
[-- Type: image/png, Size: 65076 bytes --]

[-- Attachment #2.4: EVQkTyvIsYDJlcL7.png --]
[-- Type: image/png, Size: 2280301 bytes --]

[-- Attachment #2.5: ceOAY8Zii7KShL9e.png --]
[-- Type: image/png, Size: 2068675 bytes --]