[PATCH for-5.0 v2 0/3] benchmark util

[PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

Hi all!

Here is simple benchmarking utility, to generate performance
comparison tables, like the following:

----------  -------------  -------------  -------------
            backup-1       backup-2       mirror
ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
----------  -------------  -------------  -------------

This is a v2, as v1 was inside
 "[RFC 00/24] backup performance: block_status + async"

I'll use this benchmark in other series, hope someone
will like it.

Vladimir Sementsov-Ogievskiy (3):
  python: add simplebench.py
  python: add qemu/bench_block_job.py
  python: add example usage of simplebench

 python/bench-example.py        |  80 +++++++++++++++++++++
 python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
 python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
 3 files changed, 323 insertions(+)
 create mode 100644 python/bench-example.py
 create mode 100755 python/qemu/bench_block_job.py
 create mode 100644 python/simplebench.py

-- 
2.18.0

[PATCH for-5.0 v2 1/3] python: add simplebench.py

[Vladimir Sementsov-Ogievskiy]

Add simple benchmark table creator.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/simplebench.py | 128 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)
 create mode 100644 python/simplebench.py

diff --git a/python/simplebench.py b/python/simplebench.py
new file mode 100644
index 0000000000..59e7314ff6
--- /dev/null
+++ b/python/simplebench.py
@@ -0,0 +1,128 @@
+#!/usr/bin/env python
+#
+# Simple benchmarking framework
+#
+# Copyright (c) 2019 Virtuozzo International GmbH.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+def bench_one(test_func, test_env, test_case, count=5, initial_run=True):
+    """Benchmark one test-case
+
+    test_func   -- benchmarking function with prototype
+                   test_func(env, case), which takes test_env and test_case
+                   arguments and returns {'seconds': int} (which is benchmark
+                   result) on success and {'error': str} on error. Returned
+                   dict may contain any other additional fields.
+    test_env    -- test environment - opaque first argument for test_func
+    test_case   -- test case - opaque second argument for test_func
+    count       -- how many times to call test_func, to calculate average
+    initial_run -- do initial run of test_func, which don't get into result
+
+    Returns dict with the following fields:
+        'runs':     list of test_func results
+        'average':  average seconds per run (exists only if at least one run
+                    succeeded)
+        'delta':    maximum delta between test_func result and the average
+                    (exists only if at least one run succeeded)
+        'n-failed': number of failed runs (exists only if at least one run
+                    failed)
+    """
+    if initial_run:
+        print('  #initial run:')
+        print('   ', test_func(test_env, test_case))
+
+    runs = []
+    for i in range(count):
+        print('  #run {}'.format(i+1))
+        res = test_func(test_env, test_case)
+        print('   ', res)
+        runs.append(res)
+
+    result = {'runs': runs}
+
+    successed = [r for r in runs if ('seconds' in r)]
+    if successed:
+        avg = sum(r['seconds'] for r in successed) / len(successed)
+        result['average'] = avg
+        result['delta'] = max(abs(r['seconds'] - avg) for r in successed)
+
+    if len(successed) < count:
+        result['n-failed'] = count - len(successed)
+
+    return result
+
+
+def ascii_one(result):
+    """Return ASCII representation of bench_one() returned dict."""
+    if 'average' in result:
+        s = '{:.2f} +- {:.2f}'.format(result['average'], result['delta'])
+        if 'n-failed' in result:
+            s += '\n({} failed)'.format(result['n-failed'])
+        return s
+    else:
+        return 'FAILED'
+
+
+def bench(test_func, test_envs, test_cases, *args, **vargs):
+    """Fill benchmark table
+
+    test_func -- benchmarking function, see bench_one for description
+    test_envs -- list of test environments, see bench_one
+    test_cases -- list of test cases, see bench_one
+    args, vargs -- additional arguments for bench_one
+
+    Returns dict with the following fields:
+        'envs':  test_envs
+        'cases': test_cases
+        'tab':   filled 2D array, where cell [i][j] is bench_one result for
+                 test_cases[i] for test_envs[j] (i.e., rows are test cases and
+                 columns are test environments)
+    """
+    tab = {}
+    results = {
+        'envs': test_envs,
+        'cases': test_cases,
+        'tab': tab
+    }
+    n = 1
+    n_tests = len(test_envs) * len(test_cases)
+    for env in test_envs:
+        for case in test_cases:
+            print('Testing {}/{}: {} :: {}'.format(n, n_tests,
+                                                   env['id'], case['id']))
+            if case['id'] not in tab:
+                tab[case['id']] = {}
+            tab[case['id']][env['id']] = bench_one(test_func, env, case,
+                                                   *args, **vargs)
+            n += 1
+
+    print('Done')
+    return results
+
+
+def ascii(results):
+    """Return ASCII representation of bench() returned dict."""
+    from tabulate import tabulate
+
+    tab = [[""] + [c['id'] for c in results['envs']]]
+    for case in results['cases']:
+        row = [case['id']]
+        for env in results['envs']:
+            row.append(ascii_one(results['tab'][case['id']][env['id']]))
+        tab.append(row)
+
+    return tabulate(tab)
-- 
2.18.0

[PATCH for-5.0 v2 2/3] python: add qemu/bench_block_job.py

[Vladimir Sementsov-Ogievskiy]

Add block-job benchmarking helper functions.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++++++
 1 file changed, 115 insertions(+)
 create mode 100755 python/qemu/bench_block_job.py

diff --git a/python/qemu/bench_block_job.py b/python/qemu/bench_block_job.py
new file mode 100755
index 0000000000..93f3956158
--- /dev/null
+++ b/python/qemu/bench_block_job.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python
+#
+# Benchmark block jobs
+#
+# Copyright (c) 2019 Virtuozzo International GmbH.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+
+import socket
+import json
+from .machine import QEMUMachine
+from .qmp import QMPConnectError
+
+
+def bench_block_job(cmd, cmd_args, qemu_args):
+    """Benchmark block-job
+
+    cmd       -- qmp command to run block-job (like blockdev-backup)
+    cmd_args  -- dict of qmp command arguments
+    qemu_args -- list of Qemu command line arguments, including path to Qemu
+                 binary
+
+    Returns {'seconds': int} on success and {'error': str} on failure, dict may
+    contain addional 'vm-log' field. Return value is compatible with
+    simplebench lib.
+    """
+
+    vm = QEMUMachine(qemu_args[0], args=qemu_args[1:])
+
+    try:
+        vm.launch()
+    except OSError as e:
+        return {'error': 'popen failed: ' + str(e)}
+    except (QMPConnectError, socket.timeout):
+        return {'error': 'qemu failed: ' + str(vm.get_log())}
+
+    try:
+        res = vm.qmp(cmd, **cmd_args)
+        if res != {'return': {}}:
+            vm.shutdown()
+            return {'error': '"{}" command failed: {}'.format(cmd, str(res))}
+
+        e = vm.event_wait('JOB_STATUS_CHANGE')
+        assert e['data']['status'] == 'created'
+        start_ms = e['timestamp']['seconds'] * 1000000 + \
+            e['timestamp']['microseconds']
+
+        e = vm.events_wait((('BLOCK_JOB_READY', None),
+                            ('BLOCK_JOB_COMPLETED', None),
+                            ('BLOCK_JOB_FAILED', None)), timeout=True)
+        if e['event'] not in ('BLOCK_JOB_READY', 'BLOCK_JOB_COMPLETED'):
+            vm.shutdown()
+            return {'error': 'block-job failed: ' + str(e),
+                    'vm-log': vm.get_log()}
+        end_ms = e['timestamp']['seconds'] * 1000000 + \
+            e['timestamp']['microseconds']
+    finally:
+        vm.shutdown()
+
+    return {'seconds': (end_ms - start_ms) / 1000000.0}
+
+
+# Bench backup or mirror
+def bench_block_copy(qemu_binary, cmd, source, target):
+    """Helper to run bench_block_job() for mirror or backup"""
+    assert cmd in ('blockdev-backup', 'blockdev-mirror')
+
+    source['node-name'] = 'source'
+    target['node-name'] = 'target'
+
+    return bench_block_job(cmd,
+                           {'job-id': 'job0', 'device': 'source',
+                            'target': 'target', 'sync': 'full'},
+                           [qemu_binary,
+                            '-blockdev', json.dumps(source),
+                            '-blockdev', json.dumps(target)])
+
+
+def drv_file(filename):
+    return {'driver': 'file', 'filename': filename,
+            'cache': {'direct': True}, 'aio': 'native'}
+
+
+def drv_nbd(host, port):
+    return {'driver': 'nbd',
+            'server': {'type': 'inet', 'host': host, 'port': port}}
+
+
+if __name__ == '__main__':
+    import sys
+
+    if len(sys.argv) < 4:
+        print('USAGE: {} <qmp block-job command name> '
+              '<json string of arguments for the command> '
+              '<qemu binary path and arguments>'.format(sys.argv[0]))
+        exit(1)
+
+    res = bench_block_job(sys.argv[1], json.loads(sys.argv[2]), sys.argv[3:])
+    if 'seconds' in res:
+        print('{:.2f}'.format(res['seconds']))
+    else:
+        print(res)
-- 
2.18.0

[PATCH for-5.0 v2 3/3] python: add example usage of simplebench

[Vladimir Sementsov-Ogievskiy]

This example may be used as a template for custom benchmark.
It illustrates three things to prepare:
 - define bench_func
 - define test environments (columns)
 - define test cases (rows)
And final call of simplebench API.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/bench-example.py | 80 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 80 insertions(+)
 create mode 100644 python/bench-example.py

diff --git a/python/bench-example.py b/python/bench-example.py
new file mode 100644
index 0000000000..4ccd4fc3dd
--- /dev/null
+++ b/python/bench-example.py
@@ -0,0 +1,80 @@
+#!/usr/bin/env python3
+#
+# Benchmark example
+#
+# Copyright (c) 2019 Virtuozzo International GmbH.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import simplebench
+from qemu.bench_block_job import bench_block_copy, drv_file, drv_nbd
+
+
+def bench_func(env, case):
+    """ Handle one "cell" of benchmarking table. """
+    return bench_block_copy(env['qemu_binary'], env['cmd'],
+                            case['source'], case['target'])
+
+
+# You may set the following five variables to correct values, to turn this
+# example to real benchmark.
+ssd_source = '/path-to-raw-source-image-at-ssd'
+ssd_target = '/path-to-raw-target-image-at-ssd'
+hdd_target = '/path-to-raw-source-image-at-hdd'
+nbd_ip = 'nbd-ip-addr'
+nbd_port = 'nbd-port-number'
+
+# Test-cases are "rows" in benchmark resulting table, 'id' is a caption for
+# the row, other fields are handled by bench_func.
+test_cases = [
+    {
+        'id': 'ssd -> ssd',
+        'source': drv_file(ssd_source),
+        'target': drv_file(ssd_target)
+    },
+    {
+        'id': 'ssd -> hdd',
+        'source': drv_file(ssd_source),
+        'target': drv_file(hdd_target)
+    },
+    {
+        'id': 'ssd -> nbd',
+        'source': drv_file(ssd_source),
+        'target': drv_nbd(nbd_ip, nbd_port)
+    },
+]
+
+# Test-envs are "columns" in benchmark resulting table, 'id is a caption for
+# the column, other fields are handled by bench_func.
+test_envs = [
+    {
+        'id': 'backup-1',
+        'cmd': 'blockdev-backup',
+        'qemu_binary': '/path-to-qemu-binary-1'
+    },
+    {
+        'id': 'backup-2',
+        'cmd': 'blockdev-backup',
+        'qemu_binary': '/path-to-qemu-binary-2'
+    },
+    {
+        'id': 'mirror',
+        'cmd': 'blockdev-mirror',
+        'qemu_binary': '/path-to-qemu-binary-1'
+    }
+]
+
+result = simplebench.bench(bench_func, test_envs, test_cases, count=3)
+print(simplebench.ascii(result))
-- 
2.18.0

Re: [PATCH for-5.0 v2 0/3] benchmark util

[Aleksandar Markovic]

On Tue, Nov 26, 2019 at 4:49 PM Vladimir Sementsov-Ogievskiy
<vsementsov@virtuozzo.com> wrote:
>
> Hi all!
>
> Here is simple benchmarking utility, to generate performance
> comparison tables, like the following:
>
> ----------  -------------  -------------  -------------
>             backup-1       backup-2       mirror
> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
> ----------  -------------  -------------  -------------
>
> This is a v2, as v1 was inside
>  "[RFC 00/24] backup performance: block_status + async"
>
> I'll use this benchmark in other series, hope someone
> will like it.
>

Vladimir,

I really like this idea, even though I am interested in benchmarks of
different nature. Certainly a beautiful and handy tool for doing
detection of performance regressions (and also confirmation of
performance optimizations).

Did you run the tool on the previous QEMU versions, to detect change
in numbers between QEMU versions? Do you have the results of the same
benchmark for QEMU 2.12, 3.0, 4.0,... ?

What are units used in the table? Seconds, minutes? Hopefully, not hours?

Yours,
Aleksandar

> Vladimir Sementsov-Ogievskiy (3):
>   python: add simplebench.py
>   python: add qemu/bench_block_job.py
>   python: add example usage of simplebench
>
>  python/bench-example.py        |  80 +++++++++++++++++++++
>  python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>  python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>  3 files changed, 323 insertions(+)
>  create mode 100644 python/bench-example.py
>  create mode 100755 python/qemu/bench_block_job.py
>  create mode 100644 python/simplebench.py
>
> --
> 2.18.0
>
>

Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

27.11.2019 22:58, Aleksandar Markovic wrote:
> On Tue, Nov 26, 2019 at 4:49 PM Vladimir Sementsov-Ogievskiy
> <vsementsov@virtuozzo.com> wrote:
>>
>> Hi all!
>>
>> Here is simple benchmarking utility, to generate performance
>> comparison tables, like the following:
>>
>> ----------  -------------  -------------  -------------
>>              backup-1       backup-2       mirror
>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>> ----------  -------------  -------------  -------------
>>
>> This is a v2, as v1 was inside
>>   "[RFC 00/24] backup performance: block_status + async"
>>
>> I'll use this benchmark in other series, hope someone
>> will like it.
>>
> 
> Vladimir,
> 
> I really like this idea, even though I am interested in benchmarks of
> different nature. Certainly a beautiful and handy tool for doing
> detection of performance regressions (and also confirmation of
> performance optimizations).

Hi Aleksandar!

Glad you are interested!

> 
> Did you run the tool on the previous QEMU versions, to detect change
> in numbers between QEMU versions? Do you have the results of the same
> benchmark for QEMU 2.12, 3.0, 4.0,... ?

No, I don't really interested in such comparisons, my goal is to measure
backup performance in context of my backup/block-copy series.

But such statistics may be simply generated, just make several git work-trees
per release you interested in, fill test_envs variable (together with other
variables) in python/bench-example.py (patch 03), and run it.

> 
> What are units used in the table? Seconds, minutes? Hopefully, not hours?

Seconds of course, I'm not so patient :)
Also, if you are interested, 1000M raw images filled with ones used to make
this benchmark table. However, it's in cover-letter only as example.

backup-1 and mirror are upstream. backup-2 is our downstream. nbd gain of
backup-2 is related to async requests. ssd gain for backup-1 is related to
copy offloading (wow, it works!).

> 
> Yours,
> Aleksandar
> 
>> Vladimir Sementsov-Ogievskiy (3):
>>    python: add simplebench.py
>>    python: add qemu/bench_block_job.py
>>    python: add example usage of simplebench
>>
>>   python/bench-example.py        |  80 +++++++++++++++++++++
>>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>   3 files changed, 323 insertions(+)
>>   create mode 100644 python/bench-example.py
>>   create mode 100755 python/qemu/bench_block_job.py
>>   create mode 100644 python/simplebench.py
>>
>> --
>> 2.18.0
>>
>>


-- 
Best regards,
Vladimir

Re: [PATCH for-5.0 v2 0/3] benchmark util

[Cleber Rosa]

[-- Attachment #1: Type: text/plain, Size: 2056 bytes --]

On Tue, Nov 26, 2019 at 06:48:45PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Here is simple benchmarking utility, to generate performance
> comparison tables, like the following:
> 
> ----------  -------------  -------------  -------------
>             backup-1       backup-2       mirror
> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
> ----------  -------------  -------------  -------------
> 
> This is a v2, as v1 was inside
>  "[RFC 00/24] backup performance: block_status + async"
> 
> I'll use this benchmark in other series, hope someone
> will like it.
> 
> Vladimir Sementsov-Ogievskiy (3):
>   python: add simplebench.py
>   python: add qemu/bench_block_job.py
>   python: add example usage of simplebench
> 
>  python/bench-example.py        |  80 +++++++++++++++++++++
>  python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>  python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>  3 files changed, 323 insertions(+)
>  create mode 100644 python/bench-example.py
>  create mode 100755 python/qemu/bench_block_job.py
>  create mode 100644 python/simplebench.py
> 
> -- 
> 2.18.0
> 

Hi Vladimir,

This looks interesting.

Do you think the execution of "test cases" in an "environment" are a
generic enough concept that could be reused (or reuse other system)?
My point is that it'd be nice to do the same thing say for the
acceptance tests, or any tests for that matter.  For instance, for
known parameters, we could record what's the time difference between
booting a guest with q35 or pc machine types and virtio-block or
virtio-scsi devices.

BTW, This reminded me of a IOzone[1] test runner / results analyzer:

  https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/io/disk/iozone.py

I'm also cc'ing Lukáš Doktor, who has actively worked in something
similar.

Cheers,
- Cleber.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

06.12.2019 22:43, Cleber Rosa wrote:
> On Tue, Nov 26, 2019 at 06:48:45PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Here is simple benchmarking utility, to generate performance
>> comparison tables, like the following:
>>
>> ----------  -------------  -------------  -------------
>>              backup-1       backup-2       mirror
>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>> ----------  -------------  -------------  -------------
>>
>> This is a v2, as v1 was inside
>>   "[RFC 00/24] backup performance: block_status + async"
>>
>> I'll use this benchmark in other series, hope someone
>> will like it.
>>
>> Vladimir Sementsov-Ogievskiy (3):
>>    python: add simplebench.py
>>    python: add qemu/bench_block_job.py
>>    python: add example usage of simplebench
>>
>>   python/bench-example.py        |  80 +++++++++++++++++++++
>>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>   3 files changed, 323 insertions(+)
>>   create mode 100644 python/bench-example.py
>>   create mode 100755 python/qemu/bench_block_job.py
>>   create mode 100644 python/simplebench.py
>>
>> -- 
>> 2.18.0
>>
> 
> Hi Vladimir,
> 
> This looks interesting.
> 
> Do you think the execution of "test cases" in an "environment" are a
> generic enough concept that could be reused (or reuse other system)?

that was my goal: to make something generic enough, but simple. Yes,
it can be reused for anything

> My point is that it'd be nice to do the same thing say for the
> acceptance tests, or any tests for that matter.  For instance, for
> known parameters, we could record what's the time difference between
> booting a guest with q35 or pc machine types and virtio-block or
> virtio-scsi devices.
> 
> BTW, This reminded me of a IOzone[1] test runner / results analyzer:
> 
>    https://github.com/avocado-framework-tests/avocado-misc-tests/blob/master/io/disk/iozone.py
> 
> I'm also cc'ing Lukáš Doktor, who has actively worked in something
> similar.
> 
> Cheers,
> - Cleber.
> 


-- 
Best regards,
Vladimir

Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

ping

26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Here is simple benchmarking utility, to generate performance
> comparison tables, like the following:
> 
> ----------  -------------  -------------  -------------
>              backup-1       backup-2       mirror
> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
> ----------  -------------  -------------  -------------
> 
> This is a v2, as v1 was inside
>   "[RFC 00/24] backup performance: block_status + async"
> 
> I'll use this benchmark in other series, hope someone
> will like it.
> 
> Vladimir Sementsov-Ogievskiy (3):
>    python: add simplebench.py
>    python: add qemu/bench_block_job.py
>    python: add example usage of simplebench
> 
>   python/bench-example.py        |  80 +++++++++++++++++++++
>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>   3 files changed, 323 insertions(+)
>   create mode 100644 python/bench-example.py
>   create mode 100755 python/qemu/bench_block_job.py
>   create mode 100644 python/simplebench.py
> 


-- 
Best regards,
Vladimir

ping Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

ping

26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
> 
> Here is simple benchmarking utility, to generate performance
> comparison tables, like the following:
> 
> ----------  -------------  -------------  -------------
>              backup-1       backup-2       mirror
> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
> ----------  -------------  -------------  -------------
> 
> This is a v2, as v1 was inside
>   "[RFC 00/24] backup performance: block_status + async"
> 
> I'll use this benchmark in other series, hope someone
> will like it.
> 
> Vladimir Sementsov-Ogievskiy (3):
>    python: add simplebench.py
>    python: add qemu/bench_block_job.py
>    python: add example usage of simplebench
> 
>   python/bench-example.py        |  80 +++++++++++++++++++++
>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>   3 files changed, 323 insertions(+)
>   create mode 100644 python/bench-example.py
>   create mode 100755 python/qemu/bench_block_job.py
>   create mode 100644 python/simplebench.py
> 


-- 
Best regards,
Vladimir

Re: ping Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

pingg..

Hi! Could it be merged at all?

20.01.2020 12:10, Vladimir Sementsov-Ogievskiy wrote:
> ping
> 
> 26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all!
>>
>> Here is simple benchmarking utility, to generate performance
>> comparison tables, like the following:
>>
>> ----------  -------------  -------------  -------------
>>              backup-1       backup-2       mirror
>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>> ----------  -------------  -------------  -------------
>>
>> This is a v2, as v1 was inside
>>   "[RFC 00/24] backup performance: block_status + async"
>>
>> I'll use this benchmark in other series, hope someone
>> will like it.
>>
>> Vladimir Sementsov-Ogievskiy (3):
>>    python: add simplebench.py
>>    python: add qemu/bench_block_job.py
>>    python: add example usage of simplebench
>>
>>   python/bench-example.py        |  80 +++++++++++++++++++++
>>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>   3 files changed, 323 insertions(+)
>>   create mode 100644 python/bench-example.py
>>   create mode 100755 python/qemu/bench_block_job.py
>>   create mode 100644 python/simplebench.py
>>
> 
> 


-- 
Best regards,
Vladimir

Re: ping Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

Hi!

Is problem in "S: Odd fixes" in Python section of MAINTAINERS?

Will it be correct, if I send a patch to MAINTAINERS, proposing
myself as maintainer of Python scripts and s/Odd fixes/Maintained/ ?

And then just send pull request with this series, as "nobody minds"?

08.02.2020 13:36, Vladimir Sementsov-Ogievskiy wrote:
> pingg..
> 
> Hi! Could it be merged at all?
> 
> 20.01.2020 12:10, Vladimir Sementsov-Ogievskiy wrote:
>> ping
>>
>> 26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi all!
>>>
>>> Here is simple benchmarking utility, to generate performance
>>> comparison tables, like the following:
>>>
>>> ----------  -------------  -------------  -------------
>>>              backup-1       backup-2       mirror
>>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>>> ----------  -------------  -------------  -------------
>>>
>>> This is a v2, as v1 was inside
>>>   "[RFC 00/24] backup performance: block_status + async"
>>>
>>> I'll use this benchmark in other series, hope someone
>>> will like it.
>>>
>>> Vladimir Sementsov-Ogievskiy (3):
>>>    python: add simplebench.py
>>>    python: add qemu/bench_block_job.py
>>>    python: add example usage of simplebench
>>>
>>>   python/bench-example.py        |  80 +++++++++++++++++++++
>>>   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>>   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>>   3 files changed, 323 insertions(+)
>>>   create mode 100644 python/bench-example.py
>>>   create mode 100755 python/qemu/bench_block_job.py
>>>   create mode 100644 python/simplebench.py
>>>
>>
>>
> 
> 


-- 
Best regards,
Vladimir

Re: ping Re: [PATCH for-5.0 v2 0/3] benchmark util

[Eduardo Habkost]

Sorry, this is due to lack of bandwidth of maintainers who can
review those patches.

I have one suggestion: if you make your script self-contained
inside a scripts/ subdirectory, it would be simpler to merge it
without detailed reviews from others.

The python/ subdirectory is supposed to appear on sys.path, so
maybe simplebench.py and qemu/bench_block_job.py can stay there,
but bench-example.py is not a loadable Python module and
shouldn't be there.

I see two possible options:

a) Moving everything to a scripts/simplebench subdirectory.
b) Moving only bench-example.py to scripts/, and do the sys.path
   hacking the other scripts do.

On either case, please add your name to MAINTAINERS as the
maintainer of those new files.


On Thu, Feb 27, 2020 at 04:18:00PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Hi!
> 
> Is problem in "S: Odd fixes" in Python section of MAINTAINERS?
> 
> Will it be correct, if I send a patch to MAINTAINERS, proposing
> myself as maintainer of Python scripts and s/Odd fixes/Maintained/ ?
> 
> And then just send pull request with this series, as "nobody minds"?
> 
> 08.02.2020 13:36, Vladimir Sementsov-Ogievskiy wrote:
> > pingg..
> > 
> > Hi! Could it be merged at all?
> > 
> > 20.01.2020 12:10, Vladimir Sementsov-Ogievskiy wrote:
> > > ping
> > > 
> > > 26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
> > > > Hi all!
> > > > 
> > > > Here is simple benchmarking utility, to generate performance
> > > > comparison tables, like the following:
> > > > 
> > > > ----------  -------------  -------------  -------------
> > > >              backup-1       backup-2       mirror
> > > > ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
> > > > ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
> > > > ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
> > > > ----------  -------------  -------------  -------------
> > > > 
> > > > This is a v2, as v1 was inside
> > > >   "[RFC 00/24] backup performance: block_status + async"
> > > > 
> > > > I'll use this benchmark in other series, hope someone
> > > > will like it.
> > > > 
> > > > Vladimir Sementsov-Ogievskiy (3):
> > > >    python: add simplebench.py
> > > >    python: add qemu/bench_block_job.py
> > > >    python: add example usage of simplebench
> > > > 
> > > >   python/bench-example.py        |  80 +++++++++++++++++++++
> > > >   python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
> > > >   python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
> > > >   3 files changed, 323 insertions(+)
> > > >   create mode 100644 python/bench-example.py
> > > >   create mode 100755 python/qemu/bench_block_job.py
> > > >   create mode 100644 python/simplebench.py
> > > > 
> > > 
> > > 
> > 
> > 
> 
> 
> -- 
> Best regards,
> Vladimir
> 

-- 
Eduardo

Re: ping Re: [PATCH for-5.0 v2 0/3] benchmark util

[Vladimir Sementsov-Ogievskiy]

27.02.2020 23:09, Eduardo Habkost wrote:
> Sorry, this is due to lack of bandwidth of maintainers who can
> review those patches.
> 
> I have one suggestion: if you make your script self-contained
> inside a scripts/ subdirectory, it would be simpler to merge it
> without detailed reviews from others.

That works for me

> 
> The python/ subdirectory is supposed to appear on sys.path, so

Hmm. Ok, I think, we'll always be able to move shareable parts of simplebench
into python/ if needed. So it's OK to keep it in scripts. I just
thought that python/ is a new home for python scripts :) So, it's OK
to keep the whole thing at scripts/ for now.

> maybe simplebench.py and qemu/bench_block_job.py can stay there,
> but bench-example.py is not a loadable Python module and
> shouldn't be there.
> 
> I see two possible options:
> 
> a) Moving everything to a scripts/simplebench subdirectory.
> b) Moving only bench-example.py to scripts/, and do the sys.path
>     hacking the other scripts do.
> 
> On either case, please add your name to MAINTAINERS as the
> maintainer of those new files.
> 

OK, thanks!

> 
> On Thu, Feb 27, 2020 at 04:18:00PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> Hi!
>>
>> Is problem in "S: Odd fixes" in Python section of MAINTAINERS?
>>
>> Will it be correct, if I send a patch to MAINTAINERS, proposing
>> myself as maintainer of Python scripts and s/Odd fixes/Maintained/ ?
>>
>> And then just send pull request with this series, as "nobody minds"?
>>
>> 08.02.2020 13:36, Vladimir Sementsov-Ogievskiy wrote:
>>> pingg..
>>>
>>> Hi! Could it be merged at all?
>>>
>>> 20.01.2020 12:10, Vladimir Sementsov-Ogievskiy wrote:
>>>> ping
>>>>
>>>> 26.11.2019 18:48, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Hi all!
>>>>>
>>>>> Here is simple benchmarking utility, to generate performance
>>>>> comparison tables, like the following:
>>>>>
>>>>> ----------  -------------  -------------  -------------
>>>>>               backup-1       backup-2       mirror
>>>>> ssd -> ssd  0.43 +- 0.00   4.48 +- 0.06   4.38 +- 0.02
>>>>> ssd -> hdd  10.60 +- 0.08  10.69 +- 0.18  10.57 +- 0.05
>>>>> ssd -> nbd  33.81 +- 0.37  10.67 +- 0.17  10.07 +- 0.07
>>>>> ----------  -------------  -------------  -------------
>>>>>
>>>>> This is a v2, as v1 was inside
>>>>>    "[RFC 00/24] backup performance: block_status + async"
>>>>>
>>>>> I'll use this benchmark in other series, hope someone
>>>>> will like it.
>>>>>
>>>>> Vladimir Sementsov-Ogievskiy (3):
>>>>>     python: add simplebench.py
>>>>>     python: add qemu/bench_block_job.py
>>>>>     python: add example usage of simplebench
>>>>>
>>>>>    python/bench-example.py        |  80 +++++++++++++++++++++
>>>>>    python/qemu/bench_block_job.py | 115 +++++++++++++++++++++++++++++
>>>>>    python/simplebench.py          | 128 +++++++++++++++++++++++++++++++++
>>>>>    3 files changed, 323 insertions(+)
>>>>>    create mode 100644 python/bench-example.py
>>>>>    create mode 100755 python/qemu/bench_block_job.py
>>>>>    create mode 100644 python/simplebench.py
>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>> -- 
>> Best regards,
>> Vladimir
>>
> 


-- 
Best regards,
Vladimir