Re: [PATCH v6 14/15] scripts/simplebench: improve ascii table: add difference line

[Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>]

25.09.2020 13:24, Max Reitz wrote:
> On 18.09.20 20:19, Vladimir Sementsov-Ogievskiy wrote:
>> Performance improvements / degradations are usually discussed in
>> percentage. Let's make the script calculate it for us.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   scripts/simplebench/simplebench.py | 46 +++++++++++++++++++++++++++---
>>   1 file changed, 42 insertions(+), 4 deletions(-)
>>
>> diff --git a/scripts/simplebench/simplebench.py b/scripts/simplebench/simplebench.py
>> index 56d3a91ea2..0ff05a38b8 100644
>> --- a/scripts/simplebench/simplebench.py
>> +++ b/scripts/simplebench/simplebench.py
> 
> [...]
> 
>> +            for j in range(0, i):
>> +                env_j = results['envs'][j]
>> +                res_j = case_results[env_j['id']]
>> +
>> +                if 'average' not in res_j:
>> +                    # Failed result
>> +                    cell += ' --'
>> +                    continue
>> +
>> +                col_j = chr(ord('A') + j)
>> +                avg_j = res_j['average']
>> +                delta = (res['average'] - avg_j) / avg_j * 100
> 
> I was wondering why you’d subtract, when percentage differences usually
> mean a quotient.  Then I realized that this would usually be written as:
> 
> (res['average'] / avg_j - 1) * 100
> 
>> +                delta_delta = (res['delta'] + res_j['delta']) / avg_j * 100
> 
> Why not use the new format_percent for both cases?

because I want less precision here

> 
>> +                cell += f' {col_j}{round(delta):+}±{round(delta_delta)}%'
> 
> I don’t know what I should think about ±delta_delta.  If I saw “Compared
> to run A, this is +42.1%±2.0%”, I would think that you calculated the
> difference between each run result, and then based on that array
> calculated average and standard deviation.
> 
> Furthermore, I don’t even know what the delta_delta is supposed to tell
> you.  It isn’t even a delta_delta, it’s an average_delta.

not avarage, but sum of errors. And it shows the error for the delta

> 
> The delta_delta would be (res['delta'] / res_j['delta'] - 1) * 100.0.

and this shows nothing.

Assume we have = A = 10+-2 and B = 15+-2

The difference is (15-10)+-(2+2) = 5+-4.
And your formula will give (2/2 - 1) *100 = 0, which is wrong.

Anyway, my code is mess)

> And that might be presented perhaps like “+42.1% Δ± +2.0%” (if delta
> were the SD, “Δx̅=+42.1% Δσ=+2.0%” would also work; although, again, I do
> interpret ± as the SD anyway).
> 

I feel that I'm bad in statistics :( I'll learn a little and make a new version.

-- 
Best regards,
Vladimir