Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[kernel test robot]

[-- Attachment #1: Type: text/plain, Size: 4043 bytes --]

CC: kbuild-all(a)lists.01.org
In-Reply-To: <20210209203041.21493-5-mrostecki@suse.de>
References: <20210209203041.21493-5-mrostecki@suse.de>
TO: Michal Rostecki <mrostecki@suse.de>

Hi Michal,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on v5.11-rc7]
[also build test WARNING on next-20210125]
[cannot apply to kdave/for-next]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Michal-Rostecki/Add-roundrobin-raid1-read-policy/20210210-053511
base:    92bf22614b21a2706f4993b278017e437f7785b3
:::::: branch date: 10 hours ago
:::::: commit date: 10 hours ago
config: i386-randconfig-m021-20210209 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>

New smatch warnings:
fs/btrfs/volumes.c:635 btrfs_check_mixed() error: uninitialized symbol 'prev_device'.

Old smatch warnings:
fs/btrfs/volumes.c:1264 open_fs_devices() error: we previously assumed 'latest_dev' could be null (see line 1252)
fs/btrfs/volumes.h:218 btrfs_device_set_total_bytes() warn: statement has no effect 31
fs/btrfs/volumes.h:219 btrfs_device_set_disk_total_bytes() warn: statement has no effect 31
fs/btrfs/volumes.h:220 btrfs_device_set_bytes_used() warn: statement has no effect 31

vim +/prev_device +635 fs/btrfs/volumes.c

4fde46f0cc71c7 Anand Jain      2015-06-17  619  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  620  /*
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  621   * Checks if after adding the new device the filesystem is going to have mixed
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  622   * types of devices (non-rotational and rotational).
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  623   *
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  624   * @fs_devices:          list of devices
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  625   * @new_device_rotating: if the new device is rotational
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  626   *
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  627   * Returns true if there are mixed types of devices, otherwise returns false.
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  628   */
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  629  static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  630  			      bool new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  631  {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  632  	struct btrfs_device *device, *prev_device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  633  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  634  	list_for_each_entry(device, &fs_devices->devices, dev_list) {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09 @635  		if (prev_device == NULL &&
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  636  		    device->rotating != new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  637  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  638  		if (prev_device != NULL &&
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  639  		    (device->rotating != prev_device->rotating ||
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  640  		     device->rotating != new_device_rotating))
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  641  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  642  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  643  		prev_device = device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  644  	}
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  645  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  646  	return false;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  647  }
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  648  

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 37174 bytes --]

[PATCH RFC 0/6] Add roundrobin raid1 read policy

[Michal Rostecki]

From: Michal Rostecki <mrostecki@suse.com>

This patch series adds a new raid1 read policy - roundrobin. For each
request, it selects the mirror which has lower load than queue depth.
Load is defined  as the number of inflight requests + a penalty value
(if the scheduled request is not local to the last processed request for
a rotational disk).

The series consists of preparational changes which add necessary
information to the btrfs_device struct and the change with the policy.

This policy was tested with fio and compared with the default `pid`
policy.

The singlethreaded test has the following parameters:

  [global]
  name=btrfs-raid1-seqread
  filename=btrfs-raid1-seqread
  rw=read
  bs=64k
  direct=0
  numjobs=1
  time_based=0

  [file1]
  size=10G
  ioengine=libaio

and shows the following results:

- raid1c3 with 3 HDDs:
  3 x Segate Barracuda ST2000DM008 (2TB)
  * pid policy
    READ: bw=217MiB/s (228MB/s), 217MiB/s-217MiB/s (228MB/s-228MB/s),
    io=10.0GiB (10.7GB), run=47082-47082msec
  * roundrobin policy
    READ: bw=409MiB/s (429MB/s), 409MiB/s-409MiB/s (429MB/s-429MB/s),
    io=10.0GiB (10.7GB), run=25028-25028mse
- raid1c3 with 2 HDDs and 1 SSD:
  2 x Segate Barracuda ST2000DM008 (2TB)
  1 x Crucial CT256M550SSD1 (256GB)
  * pid policy (the worst case when only HDDs were chosen)
    READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s),
    io=10.0GiB (10.7GB), run=46577-46577mse
  * pid policy (the best case when SSD was used as well)
    READ: bw=513MiB/s (538MB/s), 513MiB/s-513MiB/s (538MB/s-538MB/s),
    io=10.0GiB (10.7GB), run=19954-19954msec
  * roundrobin (there are no noticeable differences when testing multiple
    times)
    READ: bw=541MiB/s (567MB/s), 541MiB/s-541MiB/s (567MB/s-567MB/s),
    io=10.0GiB (10.7GB), run=18933-18933msec

The multithreaded test has the following parameters:

  [global]
  name=btrfs-raid1-seqread
  filename=btrfs-raid1-seqread
  rw=read
  bs=64k
  direct=0
  numjobs=8
  time_based=0

  [file1]
  size=10G
  ioengine=libaio

and shows the following results:

- raid1c3 with 3 HDDs: 3 x Segate Barracuda ST2000DM008 (2TB)
  3 x Segate Barracuda ST2000DM008 (2TB)
  * pid policy
    READ: bw=1569MiB/s (1645MB/s), 196MiB/s-196MiB/s (206MB/s-206MB/s),
    io=80.0GiB (85.9GB), run=52210-52211msec
  * roundrobin
    READ: bw=1733MiB/s (1817MB/s), 217MiB/s-217MiB/s (227MB/s-227MB/s),
    io=80.0GiB (85.9GB), run=47269-47271msec
- raid1c3 with 2 HDDs and 1 SSD:
  2 x Segate Barracuda ST2000DM008 (2TB)
  1 x Crucial CT256M550SSD1 (256GB)
  * pid policy
    READ: bw=1843MiB/s (1932MB/s), 230MiB/s-230MiB/s (242MB/s-242MB/s),
    io=80.0GiB (85.9GB), run=44449-44450msec
  * roundrobin
    READ: bw=2485MiB/s (2605MB/s), 311MiB/s-311MiB/s (326MB/s-326MB/s),
    io=80.0GiB (85.9GB), run=32969-32970msec

To measure the performance of each policy and find optimal penalty
values, I created scripts which are available here:

https://gitlab.com/vadorovsky/btrfs-perf
https://github.com/mrostecki/btrfs-perf

Michal Rostecki (6):
  btrfs: Add inflight BIO request counter
  btrfs: Store the last device I/O offset
  btrfs: Add stripe_physical function
  btrfs: Check if the filesystem is has mixed type of devices
  btrfs: sysfs: Add directory for read policies
  btrfs: Add roundrobin raid1 read policy

 fs/btrfs/ctree.h   |   3 +
 fs/btrfs/disk-io.c |   3 +
 fs/btrfs/sysfs.c   | 144 ++++++++++++++++++++++++++----
 fs/btrfs/volumes.c | 218 +++++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/volumes.h |  22 +++++
 5 files changed, 366 insertions(+), 24 deletions(-)

-- 
2.30.0

[PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michal Rostecki]

From: Michal Rostecki <mrostecki@suse.com>

Add the btrfs_check_mixed() function which checks if the filesystem has
the mixed type of devices (non-rotational and rotational). This
information is going to be used in roundrobin raid1 read policy.

Signed-off-by: Michal Rostecki <mrostecki@suse.com>
---
 fs/btrfs/volumes.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
 fs/btrfs/volumes.h |  7 +++++++
 2 files changed, 49 insertions(+), 2 deletions(-)

diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 1ac364a2f105..1ad30a595722 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -617,6 +617,35 @@ static int btrfs_free_stale_devices(const char *path,
 	return ret;
 }
 
+/*
+ * Checks if after adding the new device the filesystem is going to have mixed
+ * types of devices (non-rotational and rotational).
+ *
+ * @fs_devices:          list of devices
+ * @new_device_rotating: if the new device is rotational
+ *
+ * Returns true if there are mixed types of devices, otherwise returns false.
+ */
+static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
+			      bool new_device_rotating)
+{
+	struct btrfs_device *device, *prev_device;
+
+	list_for_each_entry(device, &fs_devices->devices, dev_list) {
+		if (prev_device == NULL &&
+		    device->rotating != new_device_rotating)
+			return true;
+		if (prev_device != NULL &&
+		    (device->rotating != prev_device->rotating ||
+		     device->rotating != new_device_rotating))
+			return true;
+
+		prev_device = device;
+	}
+
+	return false;
+}
+
 /*
  * This is only used on mount, and we are protected from competing things
  * messing with our fs_devices by the uuid_mutex, thus we do not need the
@@ -629,6 +658,7 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 	struct request_queue *q;
 	struct block_device *bdev;
 	struct btrfs_super_block *disk_super;
+	bool rotating;
 	u64 devid;
 	int ret;
 
@@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
 	}
 
 	q = bdev_get_queue(bdev);
-	if (!blk_queue_nonrot(q))
+	rotating = !blk_queue_nonrot(q);
+	device->rotating = rotating;
+	if (rotating)
 		fs_devices->rotating = true;
+	if (!fs_devices->mixed)
+		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
 
 	device->bdev = bdev;
 	clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
@@ -2418,6 +2452,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
 	fs_devices->open_devices = 0;
 	fs_devices->missing_devices = 0;
 	fs_devices->rotating = false;
+	fs_devices->mixed = false;
 	list_add(&seed_devices->seed_list, &fs_devices->seed_list);
 
 	generate_random_uuid(fs_devices->fsid);
@@ -2522,6 +2557,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 	int seeding_dev = 0;
 	int ret = 0;
 	bool locked = false;
+	bool rotating;
 
 	if (sb_rdonly(sb) && !fs_devices->seeding)
 		return -EROFS;
@@ -2621,8 +2657,12 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
 
 	atomic64_add(device->total_bytes, &fs_info->free_chunk_space);
 
-	if (!blk_queue_nonrot(q))
+	rotating = !blk_queue_nonrot(q);
+	device->rotating = rotating;
+	if (rotating)
 		fs_devices->rotating = true;
+	if (!fs_devices->mixed)
+		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
 
 	orig_super_total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
 	btrfs_set_super_total_bytes(fs_info->super_copy,
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 6e544317a377..594f1207281c 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -147,6 +147,9 @@ struct btrfs_device {
 	/* I/O stats for raid1 mirror selection */
 	struct percpu_counter inflight;
 	atomic_t last_offset;
+
+	/* If the device is rotational */
+	bool rotating;
 };
 
 /*
@@ -274,6 +277,10 @@ struct btrfs_fs_devices {
 	 * nonrot flag set
 	 */
 	bool rotating;
+	/* Set when we find or add both nonrot and rot disks in the
+	 * filesystem
+	 */
+	bool mixed;
 
 	struct btrfs_fs_info *fs_info;
 	/* sysfs kobjects */
-- 
2.30.0

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michał Mirosław]

On Tue, Feb 09, 2021 at 09:30:38PM +0100, Michal Rostecki wrote:
> From: Michal Rostecki <mrostecki@suse.com>
> 
> Add the btrfs_check_mixed() function which checks if the filesystem has
> the mixed type of devices (non-rotational and rotational). This
> information is going to be used in roundrobin raid1 read policy.a
[...]
> @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
>  	}
>  
>  	q = bdev_get_queue(bdev);
> -	if (!blk_queue_nonrot(q))
> +	rotating = !blk_queue_nonrot(q);
> +	device->rotating = rotating;
> +	if (rotating)
>  		fs_devices->rotating = true;
> +	if (!fs_devices->mixed)
> +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
[...]

Since this is adding to a set, a faster way is:

if (fs_devices->rotating != rotating)
	fs_devices->mixed = true;

The scan might be necessary on device removal, though.

> -	if (!blk_queue_nonrot(q))
> +	rotating = !blk_queue_nonrot(q);
> +	device->rotating = rotating;
> +	if (rotating)
>  		fs_devices->rotating = true;
> +	if (!fs_devices->mixed)
> +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);

Duplication. Maybe pull all this into a function?

Best Regards,
Michał Mirosław

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michal Rostecki]

On Wed, Feb 10, 2021 at 05:08:05AM +0100, Michał Mirosław wrote:
> On Tue, Feb 09, 2021 at 09:30:38PM +0100, Michal Rostecki wrote:
> > From: Michal Rostecki <mrostecki@suse.com>
> > 
> > Add the btrfs_check_mixed() function which checks if the filesystem has
> > the mixed type of devices (non-rotational and rotational). This
> > information is going to be used in roundrobin raid1 read policy.a
> [...]
> > @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> >  	}
> >  
> >  	q = bdev_get_queue(bdev);
> > -	if (!blk_queue_nonrot(q))
> > +	rotating = !blk_queue_nonrot(q);
> > +	device->rotating = rotating;
> > +	if (rotating)
> >  		fs_devices->rotating = true;
> > +	if (!fs_devices->mixed)
> > +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> [...]
> 
> Since this is adding to a set, a faster way is:
> 
> if (fs_devices->rotating != rotating)
> 	fs_devices->mixed = true;
> 

Good idea.

> The scan might be necessary on device removal, though.
> 

And good point. I didn't address the case of device removal at all.

> > -	if (!blk_queue_nonrot(q))
> > +	rotating = !blk_queue_nonrot(q);
> > +	device->rotating = rotating;
> > +	if (rotating)
> >  		fs_devices->rotating = true;
> > +	if (!fs_devices->mixed)
> > +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> 
> Duplication. Maybe pull all this into a function?
> 

Will do that as well.

> Best Regards,
> Michał Mirosław

Regards,
Michal

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michal Rostecki]

On Wed, Feb 10, 2021 at 05:08:05AM +0100, Michał Mirosław wrote:
> On Tue, Feb 09, 2021 at 09:30:38PM +0100, Michal Rostecki wrote:
> > From: Michal Rostecki <mrostecki@suse.com>
> > 
> > Add the btrfs_check_mixed() function which checks if the filesystem has
> > the mixed type of devices (non-rotational and rotational). This
> > information is going to be used in roundrobin raid1 read policy.a
> [...]
> > @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> >  	}
> >  
> >  	q = bdev_get_queue(bdev);
> > -	if (!blk_queue_nonrot(q))
> > +	rotating = !blk_queue_nonrot(q);
> > +	device->rotating = rotating;
> > +	if (rotating)
> >  		fs_devices->rotating = true;
> > +	if (!fs_devices->mixed)
> > +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> [...]
> 
> Since this is adding to a set, a faster way is:
> 
> if (fs_devices->rotating != rotating)
> 	fs_devices->mixed = true;
> 
> The scan might be necessary on device removal, though.
> 

Actually, that's not going to work in case of appenging a rotational
device when all previous devices are non-rotational.

  if (rotating)
        fs_devices->rotating = true;
  if (fs_devices->rotating != rotating)
        fs_devices->mixed = true;

If all devices are non-rotational, we start with the following
attributes:

fs_devices->rotating: false
fs_devices->mixed: false

Then, while appending a rotational disk, we have:

  rotating = true;
  if (rotating)                         // if (true)
        fs_devices->rotating = true;    // overriding with `true`
  if (fs_devices->rotating != rotating) // if (true != true), which is false
        fs_devices->mixed = true;       // NOT EXECUTED

So we end up fs_devices->mixed being `false`, despite having a mixed
array.

Inverting the order of those `if` checks would break the other
permuitations which start with rotational disks.

Therefore, to cover all cases, I think we need a full check, always.

Regards,
Michal

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michał Mirosław]

On Fri, Feb 12, 2021 at 06:26:41PM +0000, Michal Rostecki wrote:
> On Wed, Feb 10, 2021 at 05:08:05AM +0100, Michał Mirosław wrote:
> > On Tue, Feb 09, 2021 at 09:30:38PM +0100, Michal Rostecki wrote:
> > > From: Michal Rostecki <mrostecki@suse.com>
> > > 
> > > Add the btrfs_check_mixed() function which checks if the filesystem has
> > > the mixed type of devices (non-rotational and rotational). This
> > > information is going to be used in roundrobin raid1 read policy.a
> > [...]
> > > @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> > >  	}
> > >  
> > >  	q = bdev_get_queue(bdev);
> > > -	if (!blk_queue_nonrot(q))
> > > +	rotating = !blk_queue_nonrot(q);
> > > +	device->rotating = rotating;
> > > +	if (rotating)
> > >  		fs_devices->rotating = true;
> > > +	if (!fs_devices->mixed)
> > > +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> > [...]
> > 
> > Since this is adding to a set, a faster way is:
> > 
> > if (fs_devices->rotating != rotating)
> > 	fs_devices->mixed = true;
> > 
> > The scan might be necessary on device removal, though.
> Actually, that's not going to work in case of appenging a rotational
> device when all previous devices are non-rotational.
[...]
> Inverting the order of those `if` checks would break the other
> permuitations which start with rotational disks.

But not if you would add:

if (adding first device)
	fs_devices->rotating = rotating;

before the checks.

But them, there is a simpler way: count how many rotating vs non-rotating
devices there are while adding them. Like:

rotating ? ++n_rotating : ++n_fixed;

And then on remove you'd have it covered.

Best Regards
Michał Mirosław

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michal Rostecki]

On Sat, Feb 13, 2021 at 12:36:02AM +0100, Michał Mirosław wrote:
> On Fri, Feb 12, 2021 at 06:26:41PM +0000, Michal Rostecki wrote:
> > On Wed, Feb 10, 2021 at 05:08:05AM +0100, Michał Mirosław wrote:
> > > On Tue, Feb 09, 2021 at 09:30:38PM +0100, Michal Rostecki wrote:
> > > > From: Michal Rostecki <mrostecki@suse.com>
> > > > 
> > > > Add the btrfs_check_mixed() function which checks if the filesystem has
> > > > the mixed type of devices (non-rotational and rotational). This
> > > > information is going to be used in roundrobin raid1 read policy.a
> > > [...]
> > > > @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> > > >  	}
> > > >  
> > > >  	q = bdev_get_queue(bdev);
> > > > -	if (!blk_queue_nonrot(q))
> > > > +	rotating = !blk_queue_nonrot(q);
> > > > +	device->rotating = rotating;
> > > > +	if (rotating)
> > > >  		fs_devices->rotating = true;
> > > > +	if (!fs_devices->mixed)
> > > > +		fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> > > [...]
> > > 
> > > Since this is adding to a set, a faster way is:
> > > 
> > > if (fs_devices->rotating != rotating)
> > > 	fs_devices->mixed = true;
> > > 
> > > The scan might be necessary on device removal, though.
> > Actually, that's not going to work in case of appenging a rotational
> > device when all previous devices are non-rotational.
> [...]
> > Inverting the order of those `if` checks would break the other
> > permuitations which start with rotational disks.
> 
> But not if you would add:
> 
> if (adding first device)
> 	fs_devices->rotating = rotating;
> 
> before the checks.
> 
> But them, there is a simpler way: count how many rotating vs non-rotating
> devices there are while adding them. Like:
> 
> rotating ? ++n_rotating : ++n_fixed;
> 
> And then on remove you'd have it covered.

I like the idea of storing numbers and simply checking them. I use it in
v2 - though probably in a different form, and I will most likely move
the whole logic around checking device types to separate functions, to
not bloat btrfs_open_one_device() and the others too much.

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Dan Carpenter]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

Hi Michal,

url:    https://github.com/0day-ci/linux/commits/Michal-Rostecki/Add-roundrobin-raid1-read-policy/20210210-053511
base:    92bf22614b21a2706f4993b278017e437f7785b3
config: i386-randconfig-m021-20210209 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>

New smatch warnings:
fs/btrfs/volumes.c:635 btrfs_check_mixed() error: uninitialized symbol 'prev_device'.

Old smatch warnings:
fs/btrfs/volumes.c:1264 open_fs_devices() error: we previously assumed 'latest_dev' could be null (see line 1252)

vim +/prev_device +635 fs/btrfs/volumes.c

bd7608b9a9bbf6 Michal Rostecki 2021-02-09  629  static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  630  			      bool new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  631  {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  632  	struct btrfs_device *device, *prev_device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  633  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  634  	list_for_each_entry(device, &fs_devices->devices, dev_list) {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09 @635  		if (prev_device == NULL &&
                                                                    ^^^^^^^^^^^^^^^^^^^
Uninitialized.

bd7608b9a9bbf6 Michal Rostecki 2021-02-09  636  		    device->rotating != new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  637  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  638  		if (prev_device != NULL &&
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  639  		    (device->rotating != prev_device->rotating ||
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  640  		     device->rotating != new_device_rotating))
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  641  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  642  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  643  		prev_device = device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  644  	}
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  645  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  646  	return false;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  647  }

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 37174 bytes --]

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Dan Carpenter]

[-- Attachment #1: Type: text/plain, Size: 2401 bytes --]

Hi Michal,

url:    https://github.com/0day-ci/linux/commits/Michal-Rostecki/Add-roundrobin-raid1-read-policy/20210210-053511
base:    92bf22614b21a2706f4993b278017e437f7785b3
config: i386-randconfig-m021-20210209 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0

If you fix the issue, kindly add following tag as appropriate
Reported-by: kernel test robot <lkp@intel.com>
Reported-by: Dan Carpenter <dan.carpenter@oracle.com>

New smatch warnings:
fs/btrfs/volumes.c:635 btrfs_check_mixed() error: uninitialized symbol 'prev_device'.

Old smatch warnings:
fs/btrfs/volumes.c:1264 open_fs_devices() error: we previously assumed 'latest_dev' could be null (see line 1252)

vim +/prev_device +635 fs/btrfs/volumes.c

bd7608b9a9bbf6 Michal Rostecki 2021-02-09  629  static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  630  			      bool new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  631  {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  632  	struct btrfs_device *device, *prev_device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  633  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  634  	list_for_each_entry(device, &fs_devices->devices, dev_list) {
bd7608b9a9bbf6 Michal Rostecki 2021-02-09 @635  		if (prev_device == NULL &&
                                                                    ^^^^^^^^^^^^^^^^^^^
Uninitialized.

bd7608b9a9bbf6 Michal Rostecki 2021-02-09  636  		    device->rotating != new_device_rotating)
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  637  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  638  		if (prev_device != NULL &&
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  639  		    (device->rotating != prev_device->rotating ||
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  640  		     device->rotating != new_device_rotating))
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  641  			return true;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  642  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  643  		prev_device = device;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  644  	}
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  645  
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  646  	return false;
bd7608b9a9bbf6 Michal Rostecki 2021-02-09  647  }

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

[-- Attachment #2: config.gz --]
[-- Type: application/gzip, Size: 37174 bytes --]

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Filipe Manana]

On Tue, Feb 9, 2021 at 9:32 PM Michal Rostecki <mrostecki@suse.de> wrote:
>
> From: Michal Rostecki <mrostecki@suse.com>
>
> Add the btrfs_check_mixed() function which checks if the filesystem has
> the mixed type of devices (non-rotational and rotational). This
> information is going to be used in roundrobin raid1 read policy.
>
> Signed-off-by: Michal Rostecki <mrostecki@suse.com>
> ---
>  fs/btrfs/volumes.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
>  fs/btrfs/volumes.h |  7 +++++++
>  2 files changed, 49 insertions(+), 2 deletions(-)
>
> diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> index 1ac364a2f105..1ad30a595722 100644
> --- a/fs/btrfs/volumes.c
> +++ b/fs/btrfs/volumes.c
> @@ -617,6 +617,35 @@ static int btrfs_free_stale_devices(const char *path,
>         return ret;
>  }
>
> +/*
> + * Checks if after adding the new device the filesystem is going to have mixed
> + * types of devices (non-rotational and rotational).
> + *
> + * @fs_devices:          list of devices
> + * @new_device_rotating: if the new device is rotational
> + *
> + * Returns true if there are mixed types of devices, otherwise returns false.
> + */
> +static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
> +                             bool new_device_rotating)
> +{
> +       struct btrfs_device *device, *prev_device;
> +
> +       list_for_each_entry(device, &fs_devices->devices, dev_list) {
> +               if (prev_device == NULL &&

Hum, prev_device is not initialized when we enter the first iteration
of the loop.

> +                   device->rotating != new_device_rotating)
> +                       return true;
> +               if (prev_device != NULL &&
> +                   (device->rotating != prev_device->rotating ||

Here it's more dangerous, dereferencing an uninitialized pointer can
result in a crash.

With this fixed, it would be better to redo the benchmarks when using
mixed device types.

Thanks.

> +                    device->rotating != new_device_rotating))
> +                       return true;
> +
> +               prev_device = device;
> +       }
> +
> +       return false;
> +}
> +
>  /*
>   * This is only used on mount, and we are protected from competing things
>   * messing with our fs_devices by the uuid_mutex, thus we do not need the
> @@ -629,6 +658,7 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
>         struct request_queue *q;
>         struct block_device *bdev;
>         struct btrfs_super_block *disk_super;
> +       bool rotating;
>         u64 devid;
>         int ret;
>
> @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
>         }
>
>         q = bdev_get_queue(bdev);
> -       if (!blk_queue_nonrot(q))
> +       rotating = !blk_queue_nonrot(q);
> +       device->rotating = rotating;
> +       if (rotating)
>                 fs_devices->rotating = true;
> +       if (!fs_devices->mixed)
> +               fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
>
>         device->bdev = bdev;
>         clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
> @@ -2418,6 +2452,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
>         fs_devices->open_devices = 0;
>         fs_devices->missing_devices = 0;
>         fs_devices->rotating = false;
> +       fs_devices->mixed = false;
>         list_add(&seed_devices->seed_list, &fs_devices->seed_list);
>
>         generate_random_uuid(fs_devices->fsid);
> @@ -2522,6 +2557,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
>         int seeding_dev = 0;
>         int ret = 0;
>         bool locked = false;
> +       bool rotating;
>
>         if (sb_rdonly(sb) && !fs_devices->seeding)
>                 return -EROFS;
> @@ -2621,8 +2657,12 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
>
>         atomic64_add(device->total_bytes, &fs_info->free_chunk_space);
>
> -       if (!blk_queue_nonrot(q))
> +       rotating = !blk_queue_nonrot(q);
> +       device->rotating = rotating;
> +       if (rotating)
>                 fs_devices->rotating = true;
> +       if (!fs_devices->mixed)
> +               fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
>
>         orig_super_total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
>         btrfs_set_super_total_bytes(fs_info->super_copy,
> diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
> index 6e544317a377..594f1207281c 100644
> --- a/fs/btrfs/volumes.h
> +++ b/fs/btrfs/volumes.h
> @@ -147,6 +147,9 @@ struct btrfs_device {
>         /* I/O stats for raid1 mirror selection */
>         struct percpu_counter inflight;
>         atomic_t last_offset;
> +
> +       /* If the device is rotational */
> +       bool rotating;
>  };
>
>  /*
> @@ -274,6 +277,10 @@ struct btrfs_fs_devices {
>          * nonrot flag set
>          */
>         bool rotating;
> +       /* Set when we find or add both nonrot and rot disks in the
> +        * filesystem
> +        */
> +       bool mixed;
>
>         struct btrfs_fs_info *fs_info;
>         /* sysfs kobjects */
> --
> 2.30.0
>


-- 
Filipe David Manana,

“Whether you think you can, or you think you can't — you're right.”

Re: [PATCH RFC 4/6] btrfs: Check if the filesystem is has mixed type of devices

[Michal Rostecki]

On Wed, Feb 10, 2021 at 10:09:10AM +0000, Filipe Manana wrote:
> On Tue, Feb 9, 2021 at 9:32 PM Michal Rostecki <mrostecki@suse.de> wrote:
> >
> > From: Michal Rostecki <mrostecki@suse.com>
> >
> > Add the btrfs_check_mixed() function which checks if the filesystem has
> > the mixed type of devices (non-rotational and rotational). This
> > information is going to be used in roundrobin raid1 read policy.
> >
> > Signed-off-by: Michal Rostecki <mrostecki@suse.com>
> > ---
> >  fs/btrfs/volumes.c | 44 ++++++++++++++++++++++++++++++++++++++++++--
> >  fs/btrfs/volumes.h |  7 +++++++
> >  2 files changed, 49 insertions(+), 2 deletions(-)
> >
> > diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
> > index 1ac364a2f105..1ad30a595722 100644
> > --- a/fs/btrfs/volumes.c
> > +++ b/fs/btrfs/volumes.c
> > @@ -617,6 +617,35 @@ static int btrfs_free_stale_devices(const char *path,
> >         return ret;
> >  }
> >
> > +/*
> > + * Checks if after adding the new device the filesystem is going to have mixed
> > + * types of devices (non-rotational and rotational).
> > + *
> > + * @fs_devices:          list of devices
> > + * @new_device_rotating: if the new device is rotational
> > + *
> > + * Returns true if there are mixed types of devices, otherwise returns false.
> > + */
> > +static bool btrfs_check_mixed(struct btrfs_fs_devices *fs_devices,
> > +                             bool new_device_rotating)
> > +{
> > +       struct btrfs_device *device, *prev_device;
> > +
> > +       list_for_each_entry(device, &fs_devices->devices, dev_list) {
> > +               if (prev_device == NULL &&
> 
> Hum, prev_device is not initialized when we enter the first iteration
> of the loop.
> 
> > +                   device->rotating != new_device_rotating)
> > +                       return true;
> > +               if (prev_device != NULL &&
> > +                   (device->rotating != prev_device->rotating ||
> 
> Here it's more dangerous, dereferencing an uninitialized pointer can
> result in a crash.
> 
> With this fixed, it would be better to redo the benchmarks when using
> mixed device types.
> 
> Thanks.
> 

Thanks for pointing that out. Will fix and redo benchmarks for v2.

> > +                    device->rotating != new_device_rotating))
> > +                       return true;
> > +
> > +               prev_device = device;
> > +       }
> > +
> > +       return false;
> > +}
> > +
> >  /*
> >   * This is only used on mount, and we are protected from competing things
> >   * messing with our fs_devices by the uuid_mutex, thus we do not need the
> > @@ -629,6 +658,7 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> >         struct request_queue *q;
> >         struct block_device *bdev;
> >         struct btrfs_super_block *disk_super;
> > +       bool rotating;
> >         u64 devid;
> >         int ret;
> >
> > @@ -669,8 +699,12 @@ static int btrfs_open_one_device(struct btrfs_fs_devices *fs_devices,
> >         }
> >
> >         q = bdev_get_queue(bdev);
> > -       if (!blk_queue_nonrot(q))
> > +       rotating = !blk_queue_nonrot(q);
> > +       device->rotating = rotating;
> > +       if (rotating)
> >                 fs_devices->rotating = true;
> > +       if (!fs_devices->mixed)
> > +               fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> >
> >         device->bdev = bdev;
> >         clear_bit(BTRFS_DEV_STATE_IN_FS_METADATA, &device->dev_state);
> > @@ -2418,6 +2452,7 @@ static int btrfs_prepare_sprout(struct btrfs_fs_info *fs_info)
> >         fs_devices->open_devices = 0;
> >         fs_devices->missing_devices = 0;
> >         fs_devices->rotating = false;
> > +       fs_devices->mixed = false;
> >         list_add(&seed_devices->seed_list, &fs_devices->seed_list);
> >
> >         generate_random_uuid(fs_devices->fsid);
> > @@ -2522,6 +2557,7 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
> >         int seeding_dev = 0;
> >         int ret = 0;
> >         bool locked = false;
> > +       bool rotating;
> >
> >         if (sb_rdonly(sb) && !fs_devices->seeding)
> >                 return -EROFS;
> > @@ -2621,8 +2657,12 @@ int btrfs_init_new_device(struct btrfs_fs_info *fs_info, const char *device_path
> >
> >         atomic64_add(device->total_bytes, &fs_info->free_chunk_space);
> >
> > -       if (!blk_queue_nonrot(q))
> > +       rotating = !blk_queue_nonrot(q);
> > +       device->rotating = rotating;
> > +       if (rotating)
> >                 fs_devices->rotating = true;
> > +       if (!fs_devices->mixed)
> > +               fs_devices->mixed = btrfs_check_mixed(fs_devices, rotating);
> >
> >         orig_super_total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
> >         btrfs_set_super_total_bytes(fs_info->super_copy,
> > diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
> > index 6e544317a377..594f1207281c 100644
> > --- a/fs/btrfs/volumes.h
> > +++ b/fs/btrfs/volumes.h
> > @@ -147,6 +147,9 @@ struct btrfs_device {
> >         /* I/O stats for raid1 mirror selection */
> >         struct percpu_counter inflight;
> >         atomic_t last_offset;
> > +
> > +       /* If the device is rotational */
> > +       bool rotating;
> >  };
> >
> >  /*
> > @@ -274,6 +277,10 @@ struct btrfs_fs_devices {
> >          * nonrot flag set
> >          */
> >         bool rotating;
> > +       /* Set when we find or add both nonrot and rot disks in the
> > +        * filesystem
> > +        */
> > +       bool mixed;
> >
> >         struct btrfs_fs_info *fs_info;
> >         /* sysfs kobjects */
> > --
> > 2.30.0
> >
> 
> 
> -- 
> Filipe David Manana,
> 
> “Whether you think you can, or you think you can't — you're right.”