Linux Software RAID /sysfs repair issue

Linux Software RAID /sysfs repair issue

[Fairbanks, David]

Hello Linux SW RAID maintainers;

I am a software engineer at Stratus Technologies in Maynard, MA.
I am running into an issue using the /sysfs "repair" functionality.
kernel version: 2.6.18-87.el5 (RHEL5, update 2)

I have a 2 member RAID level 1 set consisting of 2 SAS drives on an
Adaptec aic94xx HBA. I built the raid set using the following command:

mdadm -C /dev/md7 -size=5000000 -b internal -n2 -l1 /dev/sdb /dev/sde

I inject a medium error onto one of the disks using sg_write_long:

sg_write_long -lba=2000 -xfer_len=580 /dev/sdb

I then execute:

echo repair > /sys/block/md7/md/sync_action

I have done some testing and I have found that if the lba is on a 4K
byte aligned boundary (e.g. -lba=2000), the repair succeeds as expected.
However, if the lba is "not" on a 4K byte aligned boundary (e.g.
-lba=2001), the medium error is detected, but the disk gets removed from
the raid set. The medium error is not repaired. The following messages
appear in /var/log/messages:

May 12 13:46:24 leeloo kernel: md: syncing RAID array md7
May 12 13:46:24 leeloo kernel: md: minimum _guaranteed_ reconstruction
speed: 1000 KB/sec/disc.
May 12 13:46:24 leeloo kernel: md: using maximum available idle IO
bandwidth (but not more than 200000 KB/sec) for reconstruction.
May 12 13:46:24 leeloo kernel: md: using 128k window, over a total of
5000000 blocks.
May 12 13:46:33 leeloo kernel: sd 0:0:1:0: SCSI error: return code =
x08000002
May 12 13:46:33 leeloo kernel: sdh: Current: sense key: Medium Error
May 12 13:46:33 leeloo kernel:     Add. Sense: Unrecovered read error
May 12 13:46:33 leeloo kernel:
May 12 13:46:33 leeloo kernel: Info fld=0x7d1
May 12 13:46:33 leeloo kernel: end_request: I/O error, dev sdh, sector
2001
May 12 13:46:39 leeloo kernel: sd 0:0:1:0: SCSI error: return code =
0x00050000
May 12 13:46:39 leeloo kernel: end_request: I/O error, dev sdh, sector
1920
May 12 13:46:39 leeloo kernel: raid1: Disk failure on sdh, disabling
device.
May 12 13:46:39 leeloo kernel:  Operation continuing on 1 devices
May 12 13:46:39 leeloo kernel: md: md7: sync done

I originally thought this was a low level driver issue. However, this is
also reproducible on parallel scsi and Fibre Channel configurations. I
have also tried this on the latest kernel with the same results. I can
provide any other info necessary.

Because this appears to be a 4K byte alignment (cache block size) issue,
I am not sure if this is an issue in the block layer (?).

Thanks;

Dave

Re: Linux Software RAID /sysfs repair issue

[Neil Brown]

On Tuesday May 13, David.Fairbanks@stratus.com wrote:
> Hello Linux SW RAID maintainers;
> 
> I am a software engineer at Stratus Technologies in Maynard, MA.
> I am running into an issue using the /sysfs "repair" functionality.
> kernel version: 2.6.18-87.el5 (RHEL5, update 2)

Hi David.  Thanks for reporting this problem.

Unfortunately I cannot reproduce it as my test machine has SATA drives
which don't seem to support WRITE_LONG.

What should be happening during the 'repair' is that we should be
doing 64K reads from all devices and comparing the data.
If we hit a read error in one of these reads, the data read from the
other device should be written over the area with the read error.

This suggests that 64K offsets might be significant, but other 4K
offsets shouldn't be... yet obviously they are (2000 isn't a multiple
of 128)......

4K is significant as that is the page size.  The 64K is 16 pages, and
there is a 'vector' of addresses that is passed down with the request.
If something is getting confused there, we could get a 4K artifact.

Yes, I think I see it.  Could you please try with this patch, against
a recent kernel?

If that is the correct fix, I'll need to check elsewhere in the code
to make sure I haven't made that mistake multiple times.

Thanks,
NeilBrown


Signed-off-by: Neil Brown <neilb@suse.de>

### Diffstat output
 ./drivers/md/raid1.c |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff .prev/drivers/md/raid1.c ./drivers/md/raid1.c
--- .prev/drivers/md/raid1.c	2008-04-29 12:27:58.000000000 +1000
+++ ./drivers/md/raid1.c	2008-05-15 10:15:54.000000000 +1000
@@ -1295,10 +1295,13 @@ static void sync_request_write(mddev_t *
 					sbio->bi_sector = r1_bio->sector +
 						conf->mirrors[i].rdev->data_offset;
 					sbio->bi_bdev = conf->mirrors[i].rdev->bdev;
-					for (j = 0; j < vcnt ; j++)
+					for (j = 0; j < vcnt ; j++) {
+						sbio->bi_io_vec[j].bv_len += sbio->bi_io_vec[j].bv_offset;
+						sbio->bi_io_vec[j].bv_offset = 0;						
 						memcpy(page_address(sbio->bi_io_vec[j].bv_page),
 						       page_address(pbio->bi_io_vec[j].bv_page),
 						       PAGE_SIZE);
+					}
 
 				}
 			}