[RFC PATCH 3/5] md/raid1: do not send random reads to a rot disk

["Chen Cheng" <chencheng@fnnas.com>]

From: Chen Cheng <chencheng@fnnas.com>

The previous change keeps sequential nonrot I/O off a rot disk.

Current:
1. Sequential reads stay on nonrot when a nonrot disk is readable.
2. Random reads still pick the disk with the lowest nr_pending.
3. Rot disks also join that compare.

Problem:
1. A nonrot disk is fast. It can have more pending I/O.
2. A rot disk is slow. It can have fewer pending I/O.
3. Then the next random read goes to the rot disk.
4. Why send a random 4k read to the slow disk?

Improve:
1. If a nonrot disk is readable, do not use rot disks for
   min_pending.
2. Random reads stay on nonrot disks.
3. If no nonrot disk is readable, still pick a rot disk by head
   position.

Tested with fio libaio direct=1 (NVMe scheduler none, SATA
scheduler mq-deadline):

- RAID1 of Predator GM9000 + SATA HDD:
  4k randread QD1 jobs=1: 0.097 GB/s, 100/0 NVMe -> 0.084 GB/s,
    100/0 NVMe. Both disks have pending 0, so stock already stayed
    on NVMe.
  4k randread QD8 jobs=1: 0.363 GB/s, 99.6/0.4 NVMe/HDD,
    clat 87 us -> 0.671 GB/s, 100/0 NVMe, clat 46 us (-47%)
  4k randread QD16 jobs=1: 0.668 GB/s, 99.7/0.3 NVMe/HDD,
    clat 95 us -> 1.215 GB/s, 100/0 NVMe, clat 52 us (-45%)

- RAID1 of Fanxiang S103Pro + SATA HDD:
  4k randread QD1 jobs=1: 0.077 GB/s, 100/0 Fanxiang -> 0.073 GB/s,
    100/0 Fanxiang
  4k randread QD8 jobs=1: 0.278 GB/s, 99.5/0.5 Fanxiang/HDD,
    clat 114 us -> 0.389 GB/s, 100/0 Fanxiang, clat 78 us (-31%)
  4k randread QD16 jobs=1: 0.395 GB/s, 99.6/0.4 Fanxiang/HDD,
    clat 162 us -> 0.398 GB/s, 100/0 Fanxiang, clat 159 us
    (already at the Fanxiang limit)

Signed-off-by: Chen Cheng <chencheng@fnnas.com>
---
 drivers/md/raid1.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 36520e48826f..523b55d42779 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -832,10 +832,11 @@ static int choose_best_rdev(struct r1conf *conf, struct r1bio *r1_bio)
 	for (disk = 0 ; disk < conf->raid_disks * 2 ; disk++) {
 		struct md_rdev *rdev;
 		sector_t dist;
 		unsigned int pending;
 		bool nonrot;
+		bool can_pick;
 
 		if (r1_bio->bios[disk] == IO_BLOCKED)
 			continue;
 
 		rdev = conf->mirrors[disk].rdev;
@@ -848,15 +849,16 @@ static int choose_best_rdev(struct r1conf *conf, struct r1bio *r1_bio)
 
 		pending = atomic_read(&rdev->nr_pending);
 		dist = abs(r1_bio->sector -
 			   READ_ONCE(conf->mirrors[disk].head_position));
 		nonrot = test_bit(Nonrot, &rdev->flags);
+		can_pick = nonrot || !has_nonrot;
 
 		/* Don't change to another disk for sequential reads */
 		if (is_sequential(conf, disk, r1_bio)) {
 			if (!should_choose_next(conf, disk) && !pending &&
-			    (nonrot || !has_nonrot))
+			    can_pick)
 				return disk;
 
 			/*
 			 * Add 'pending' to avoid choosing this disk if
 			 * there is other idle disk.
@@ -871,11 +873,12 @@ static int choose_best_rdev(struct r1conf *conf, struct r1bio *r1_bio)
 				ctl.sequential_disk = disk;
 				ctl.sequential_nonrot = nonrot;
 			}
 		}
 
-		if (is_better_disk(pending, disk, nonrot, &ctl,
+		if (can_pick &&
+		    is_better_disk(pending, disk, nonrot, &ctl,
 				   rr_start, conf->raid_disks)) {
 			ctl.min_pending = pending;
 			ctl.min_pending_disk = disk;
 			ctl.min_pending_nonrot = nonrot;
 		}
-- 
2.55.0