* [PATCH 1/3] fsperf: add a few helper functions
2021-11-12 8:04 [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Anand Jain
@ 2021-11-12 8:04 ` Anand Jain
2021-11-12 8:04 ` [PATCH 3/3] fsperf: add a new test case diorandread Anand Jain
2021-11-12 14:34 ` [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Josef Bacik
2 siblings, 0 replies; 5+ messages in thread
From: Anand Jain @ 2021-11-12 8:04 UTC (permalink / raw)
To: josef; +Cc: dsterba, linux-btrfs
This is in preparation to add read policy benchmarking add the following
helpes
get_fstype
get_fsid
get_readpolicies
get_active_readpolicy
set_readpolicy
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
src/utils.py | 45 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 45 insertions(+)
diff --git a/src/utils.py b/src/utils.py
index 5629894ce1fb..a7f0f1baf448 100644
--- a/src/utils.py
+++ b/src/utils.py
@@ -8,6 +8,8 @@ import itertools
import numbers
import datetime
import statistics
+import subprocess
+import re
LOWER_IS_BETTER = 0
HIGHER_IS_BETTER = 1
@@ -230,3 +232,46 @@ def print_comparison_table(baseline, results):
table_rows.append(cur)
table.add_rows(table_rows)
print(table.draw())
+
+def get_fstype(device):
+ fstype = subprocess.check_output("blkid -s TYPE -o value "+device, shell=True)
+ # strip the output b'btrfs\n'
+ return (str(fstype).removesuffix("\\n'")).removeprefix("b'")
+
+def get_fsid(device):
+ fsid = subprocess.check_output("blkid -s UUID -o value "+device, shell=True)
+ # Raw output is something like this
+ # b'abcf123f-7e95-40cd-8322-0d32773cb4ec\n'
+ # strip off extra characters.
+ return str(fsid)[2:38]
+
+def get_readpolicies(device):
+ fsid = get_fsid(device)
+ sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "r")
+ # Strip '[ ]' around the active policy
+ policies = (((sysfs.read()).strip()).strip("[")).strip("]")
+ sysfs.close()
+ return policies
+
+def get_active_readpolicy(device):
+ fsid = get_fsid(device)
+ sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "r")
+ policies = (sysfs.read()).strip()
+ # Output is as below, pick the policy within '[ ]'
+ # device [pid] latency
+ active = re.search(r"\[([A-Za-z0-9_]+)\]", policies)
+ sysfs.close()
+ return active.group(1)
+
+def set_readpolicy(device, policy="pid"):
+ if not policy in get_readpolicies(device):
+ print("Policy '{}' is invalid".format(policy))
+ sys.exit(1)
+ return
+ fsid = get_fsid(device)
+ # Ran out of ideas why run_command fails.
+ # command = "echo "+policy+" > /sys/fs/btrfs/"+fsid+"/read_policy"
+ # run_command(command)
+ sysfs = open("/sys/fs/btrfs/"+fsid+"/read_policy", "w")
+ ret = sysfs.write(policy)
+ sysfs.close()
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 3/3] fsperf: add a new test case diorandread
2021-11-12 8:04 [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Anand Jain
2021-11-12 8:04 ` [PATCH 1/3] fsperf: add a few helper functions Anand Jain
@ 2021-11-12 8:04 ` Anand Jain
2021-11-12 14:34 ` [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Josef Bacik
2 siblings, 0 replies; 5+ messages in thread
From: Anand Jain @ 2021-11-12 8:04 UTC (permalink / raw)
To: josef; +Cc: dsterba, linux-btrfs
The test case 'diorandread' is a readonly direct io test case using fio.
The primary aim of this test case is to benchmark read policies.
An optional read policy parameter can be specified in the local.cfg file
as shown below
readpolicy=pid
When the readpolicy specified in the local.cfg file, the setup() in the
test case using it shall validate if the fstype is btrfs.
To compare the readpolicies the developer can run './fsperf diorandread'
with the supported read policies under a section. As shown below.
For example:
$ cat local.fg
::
[btrfs-pid]
device=/dev/vg/scratch0
mkfs=mkfs.btrfs -f -draid1 -mraid1 /dev/vg/scratch1
mount=mount -o noatime
readpolicy=pid
[btrfs-latency]
device=/dev/vg/scratch0
mkfs=mkfs.btrfs -f -draid1 -mraid1 /dev/vg/scratch1
mount=mount -o noatime
readpolicy=latency
$ ./fsperf -c btrfs-pid diorandread
$ ./fsperf -c btrfs-latency -t -C btrfs-pid diorandread
Signed-off-by: Anand Jain <anand.jain@oracle.com>
---
local-cfg-example | 6 ++++++
tests/dio-randread.py | 22 ++++++++++++++++++++++
2 files changed, 28 insertions(+)
create mode 100644 tests/dio-randread.py
diff --git a/local-cfg-example b/local-cfg-example
index a9034f98fbee..4b7fd0703feb 100644
--- a/local-cfg-example
+++ b/local-cfg-example
@@ -6,6 +6,12 @@ device=/dev/nvme0n1
mkfs=mkfs.btrfs -f
mount=mount -o noatime
+[btrfs-raid1]
+device=/dev/vg/scratch0
+mkfs=mkfs.btrfs -f -draid1 -mraid1 /dev/vg/scratch1
+mount=mount -o noatime
+readpolicy=pid
+
[xfs]
device=/dev/nvme0n1
iosched=none
diff --git a/tests/dio-randread.py b/tests/dio-randread.py
new file mode 100644
index 000000000000..05ced875bb6d
--- /dev/null
+++ b/tests/dio-randread.py
@@ -0,0 +1,22 @@
+from PerfTest import FioTest
+from utils import get_fstype
+from utils import set_readpolicy
+from utils import get_active_readpolicy
+
+class DioRandread(FioTest):
+ name = "diorandread"
+ command = ("--name diorandread --direct=1 --size=1g --rw=randread "
+ "--runtime=60 --iodepth=1024 --nrfiles=16 "
+ "--numjobs=16 --group_reporting")
+
+ def setup(self, config, section):
+ fstype = get_fstype(config.get(section, 'device'))
+
+ if not fstype == "btrfs":
+ return
+
+ if config.has_option(section, 'readpolicy'):
+ policy = config.get(section, 'readpolicy')
+ set_readpolicy(config.get(section, 'device'), policy)
+ policy = get_active_readpolicy(config.get(section, 'device'))
+ print("\tReadpolicy is set to '{}'".format(policy))
--
2.33.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark
2021-11-12 8:04 [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Anand Jain
2021-11-12 8:04 ` [PATCH 1/3] fsperf: add a few helper functions Anand Jain
2021-11-12 8:04 ` [PATCH 3/3] fsperf: add a new test case diorandread Anand Jain
@ 2021-11-12 14:34 ` Josef Bacik
2021-11-15 11:57 ` Anand Jain
2 siblings, 1 reply; 5+ messages in thread
From: Josef Bacik @ 2021-11-12 14:34 UTC (permalink / raw)
To: Anand Jain; +Cc: dsterba, linux-btrfs
On Fri, Nov 12, 2021 at 04:04:36PM +0800, Anand Jain wrote:
> I am sending this early to seek feedback on the best way to benchmark
> more than one read policy.
>
> Patch 1,2 adds helpers to support a new setup.
> Patch 3 adds a generic readonly dio fio benchmark script.
>
This looks reasonable to me, the only thing is I'd want a way for the setup()
part of the test to see that we don't have readpolicy support and simply skip.
Right now there's no mechanism for skipping a test, so add that and it'll be
good.
Also you can just do PR's through github if you want. Thanks,
Josef
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark
2021-11-12 14:34 ` [PATCH 0/3 RFC] fsperf: enhancements supporting read policy benchmark Josef Bacik
@ 2021-11-15 11:57 ` Anand Jain
0 siblings, 0 replies; 5+ messages in thread
From: Anand Jain @ 2021-11-15 11:57 UTC (permalink / raw)
To: Josef Bacik; +Cc: dsterba, linux-btrfs
On 12/11/2021 22:34, Josef Bacik wrote:
> On Fri, Nov 12, 2021 at 04:04:36PM +0800, Anand Jain wrote:
>> I am sending this early to seek feedback on the best way to benchmark
>> more than one read policy.
>>
>> Patch 1,2 adds helpers to support a new setup.
>> Patch 3 adds a generic readonly dio fio benchmark script.
>>
>
> This looks reasonable to me, the only thing is I'd want a way for the setup()
> part of the test to see that we don't have readpolicy support and simply skip.
> Right now there's no mechanism for skipping a test, so add that and it'll be
> good.
Right. I have added it.
> Also you can just do PR's through github if you want. Thanks,
I included the above changes and sent a PR for this.
Thanks, Anand
>
> Josef
>
^ permalink raw reply [flat|nested] 5+ messages in thread