Simulating faulty disk

Simulating faulty disk

[Yathindra]

Hi,

I'm trying to simulate a faulty disk behavior on linux. Basically, I 
want to inject various disk failure patterns
such as medium errors, unresponsive disk etc.

What is the best way to go about it. I heard about scsi_debug driver but 
it can only simulate medium
errors on fixed sector 0x1234. Also, scsi fault injection using 
Systemtap seems to be a user space tool
limited to files.

Any suggestions is greatly appreciated.

Thanks,
Yathi

Re: Simulating faulty disk

[Yathindra]

Hi Bryan,

I saw this too but it does not have features to delay the commands. And 
it works only at the file level.

I thought about scsi_debug driver but it seems to able to inject medium 
errors on sector 0x1234 only.
I'm not sure why that limitation exists though.

Thanks,
Yathi
On 10/19/2011 10:47 PM, Bryan Mesich wrote:
> On Wed, Oct 19, 2011 at 09:18:57PM -0600, Yathindra wrote:
>> Hi,
>>
>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>> want to inject various disk failure patterns
>> such as medium errors, unresponsive disk etc.
>>
>> What is the best way to go about it. I heard about scsi_debug driver but
>> it can only simulate medium
>> errors on fixed sector 0x1234. Also, scsi fault injection using
>> Systemtap seems to be a user space tool
>> limited to files.
>>
>> Any suggestions is greatly appreciated.
> I saw this go by the linux-scsi list a while back thinking it
> might be useful sometime down the road:
>
> http://lwn.net/Articles/265187/
>
> I think the project is hosted on sourceforge.net at the following
> URL:
>
> http://scsifaultinjtst.sourceforge.net/
>
> I had also read through part of a paper that was co-authored by
> the same person (from linux symposium 2008):
>
> http://www.linuxsymposium.org/archives/OLS/Reprints-2008/tanaka-reprint.pdf
>
> Doesn't look like there has been much activity since 2009, but it
> looks like it has the functionality you need.
>
> Bryan
>
>> Thanks,
>> Yathi

Re: Simulating faulty disk

[Yathindra]

Hi,

Can anybody with experience in using scsi_debug driver tell me if it can
be used to inject medium errors on various file/disk blocks.

And also, can it delay commands so we can simulate an unresponsive disk.

Thanks,
Yathi
On 10/20/2011 1:30 AM, Yathindra wrote:
> Hi Bryan,
>
> I saw this too but it does not have features to delay the commands. 
> And it works only at the file level.
>
> I thought about scsi_debug driver but it seems to able to inject 
> medium errors on sector 0x1234 only.
> I'm not sure why that limitation exists though.
>
> Thanks,
> Yathi
> On 10/19/2011 10:47 PM, Bryan Mesich wrote:
>> On Wed, Oct 19, 2011 at 09:18:57PM -0600, Yathindra wrote:
>>> Hi,
>>>
>>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>>> want to inject various disk failure patterns
>>> such as medium errors, unresponsive disk etc.
>>>
>>> What is the best way to go about it. I heard about scsi_debug driver 
>>> but
>>> it can only simulate medium
>>> errors on fixed sector 0x1234. Also, scsi fault injection using
>>> Systemtap seems to be a user space tool
>>> limited to files.
>>>
>>> Any suggestions is greatly appreciated.
>> I saw this go by the linux-scsi list a while back thinking it
>> might be useful sometime down the road:
>>
>> http://lwn.net/Articles/265187/
>>
>> I think the project is hosted on sourceforge.net at the following
>> URL:
>>
>> http://scsifaultinjtst.sourceforge.net/
>>
>> I had also read through part of a paper that was co-authored by
>> the same person (from linux symposium 2008):
>>
>> http://www.linuxsymposium.org/archives/OLS/Reprints-2008/tanaka-reprint.pdf 
>>
>>
>> Doesn't look like there has been much activity since 2009, but it
>> looks like it has the functionality you need.
>>
>> Bryan
>>
>>> Thanks,
>>> Yathi
>

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 04:18 AM, Yathindra wrote:
> I'm trying to simulate a faulty disk behavior on linux. Basically, I
> want to inject various disk failure patterns
> such as medium errors, unresponsive disk etc.

If using a device-mapper device is acceptable for your purpose there are 
the dm-flakey and dm-delay targets that can add delays, drop and corrupt 
writes and introduce intermittent I/O errors.

The targets can be stacked to give devices having any combination of 
these behaviours.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

Thanks for that info. Could you please tell me where I can find more 
details about using it.

Thanks again,
Yathi

On 10/20/2011 3:27 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:18 AM, Yathindra wrote:
>> I'm trying to simulate a faulty disk behavior on linux. Basically, I
>> want to inject various disk failure patterns
>> such as medium errors, unresponsive disk etc.
>
> If using a device-mapper device is acceptable for your purpose there 
> are the dm-flakey and dm-delay targets that can add delays, drop and 
> corrupt writes and introduce intermittent I/O errors.
>
> The targets can be stacked to give devices having any combination of 
> these behaviours.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 04:45 PM, Yathindra wrote:
> Hi Bryn,
>
> Thanks for that info. Could you please tell me where I can find more
> details about using it.
>
> Thanks again,
> Yathi

There's a couple of brief files describing the targets in the 
Documentation/device-mapper directory:

http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
http://www.kernel.org/doc/Documentation/device-mapper/delay.txt

They both work like the linear target (except that they introduce the 
specified faults according to their parameters).

To set up a device with both you'd need to use a pair of stacked devices 
so something like:

dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"

This will create a device /dev/mapper/f0 that is drops out for 1s in 
every 10 and has a 500ms delay on reads and writes. SECTORS is the size 
of the underlying device in sectors (use e.g. $(blockdev --getsize)).

You can optionally specify separate delays/device for writes to the 
delay layer and write loss / write corruption for the flakey layer - see 
the doc files and also the comments above flakey_ctr and delay_ctr for 
more details.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Thanks so much Bryn. Appreciate it.

On 10/20/2011 9:56 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:45 PM, Yathindra wrote:
>> Hi Bryn,
>>
>> Thanks for that info. Could you please tell me where I can find more
>> details about using it.
>>
>> Thanks again,
>> Yathi
>
> There's a couple of brief files describing the targets in the 
> Documentation/device-mapper directory:
>
> http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
> http://www.kernel.org/doc/Documentation/device-mapper/delay.txt
>
> They both work like the linear target (except that they introduce the 
> specified faults according to their parameters).
>
> To set up a device with both you'd need to use a pair of stacked 
> devices so something like:
>
> dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
> dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"
>
> This will create a device /dev/mapper/f0 that is drops out for 1s in 
> every 10 and has a 500ms delay on reads and writes. SECTORS is the 
> size of the underlying device in sectors (use e.g. $(blockdev 
> --getsize)).
>
> You can optionally specify separate delays/device for writes to the 
> delay layer and write loss / write corruption for the flakey layer - 
> see the doc files and also the comments above flakey_ctr and delay_ctr 
> for more details.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

Sorry to disturb you again. I tried to create a delay device below but 
it failed.

 >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
device-mapper: reload ioctl failed: Invalid argument
Command failed

In /var/log/messages I see this,
  device-mapper: ioctl: error adding target to table

 >uname -r
2.6.32-24-generic-pae

 >dmsetup --version
Library version:   1.02.39 (2009-10-26)
Driver version:    4.15.0

Am I doing something wrong.

Thanks,
Yathi

On 10/20/2011 9:56 AM, Bryn M. Reeves wrote:
> On 10/20/2011 04:45 PM, Yathindra wrote:
>> Hi Bryn,
>>
>> Thanks for that info. Could you please tell me where I can find more
>> details about using it.
>>
>> Thanks again,
>> Yathi
>
> There's a couple of brief files describing the targets in the 
> Documentation/device-mapper directory:
>
> http://www.kernel.org/doc/Documentation/device-mapper/dm-flakey.txt
> http://www.kernel.org/doc/Documentation/device-mapper/delay.txt
>
> They both work like the linear target (except that they introduce the 
> specified faults according to their parameters).
>
> To set up a device with both you'd need to use a pair of stacked 
> devices so something like:
>
> dmsetup create d0 --table="0 $SECTORS delay $DEVICE 0 500"
> dmsetup create f0 --table="0 $SECTORS flakey /dev/mapper/d0 0 9 1"
>
> This will create a device /dev/mapper/f0 that is drops out for 1s in 
> every 10 and has a 500ms delay on reads and writes. SECTORS is the 
> size of the underlying device in sectors (use e.g. $(blockdev 
> --getsize)).
>
> You can optionally specify separate delays/device for writes to the 
> delay layer and write loss / write corruption for the flakey layer - 
> see the doc files and also the comments above flakey_ctr and delay_ctr 
> for more details.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Bryn M. Reeves]

On 10/20/2011 05:46 PM, Yathindra wrote:
>  >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
> /dev/sdb 0 500"
> device-mapper: reload ioctl failed: Invalid argument
> Command failed
>
> In /var/log/messages I see this,
> device-mapper: ioctl: error adding target to table

I don't see this here:

# dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
# dmsetup info -c d0
Name             Maj Min Stat Open Targ Event  UUID 

d0               253   3 L--w    1    1      0
# dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
flakey /dev/mapper/d0 0 9 1"
# dmsetup info -c f0
Name             Maj Min Stat Open Targ Event  UUID 

f0               253   4 L--w    0    1      0

This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally compiled 
dm-delay since it's not enabled in the Fedora kernels.

Might want to move this thread over to dm-devel if you're having trouble 
with it.

Regards,
Bryn.

Re: Simulating faulty disk

[Yathindra]

Oh I see. Maybe dm-delay is not enabled in ubuntu either.

Could you please tell me how I could enable it.

Thanks,
Yathi

On 10/20/2011 11:40 AM, Bryn M. Reeves wrote:
> On 10/20/2011 05:46 PM, Yathindra wrote:
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>>
>> In /var/log/messages I see this,
>> device-mapper: ioctl: error adding target to table
>
> I don't see this here:
>
> # dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
> /dev/sdb 0 500"
> # dmsetup info -c d0
> Name             Maj Min Stat Open Targ Event  UUID
> d0               253   3 L--w    1    1      0
> # dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
> flakey /dev/mapper/d0 0 9 1"
> # dmsetup info -c f0
> Name             Maj Min Stat Open Targ Event  UUID
> f0               253   4 L--w    0    1      0
>
> This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally 
> compiled dm-delay since it's not enabled in the Fedora kernels.
>
> Might want to move this thread over to dm-devel if you're having 
> trouble with it.
>
> Regards,
> Bryn.

Re: Simulating faulty disk

[Yathindra]

Hi Bryn,

I compiled a fresh kernel and enabled flakey and delay modules. Now am 
able to create disks but
it fails to mount.

dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
/dev/sdb 0 500"
dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` flakey 
/dev/mapper/d0 0 9 1"
mkfs -t ext3 /dev/mapper/f0
...
...
ext2fs_mkdir: Attempt to read block from filesystem resulted in short 
read while creating root dir

Can we even mount such a device ?

Thanks,
Yathi



On 10/20/2011 11:40 AM, Bryn M. Reeves wrote:
> On 10/20/2011 05:46 PM, Yathindra wrote:
>> >dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay
>> /dev/sdb 0 500"
>> device-mapper: reload ioctl failed: Invalid argument
>> Command failed
>>
>> In /var/log/messages I see this,
>> device-mapper: ioctl: error adding target to table
>
> I don't see this here:
>
> # dmsetup create d0 --table="0 `blockdev --getsize /dev/sdb` delay 
> /dev/sdb 0 500"
> # dmsetup info -c d0
> Name             Maj Min Stat Open Targ Event  UUID
> d0               253   3 L--w    1    1      0
> # dmsetup create f0 --table="0 `blockdev --getsize /dev/mapper/d0` 
> flakey /dev/mapper/d0 0 9 1"
> # dmsetup info -c f0
> Name             Maj Min Stat Open Targ Event  UUID
> f0               253   4 L--w    0    1      0
>
> This is using the f15 2.6.40.3 (aka kernel-3.0) with a locally 
> compiled dm-delay since it's not enabled in the Fedora kernels.
>
> Might want to move this thread over to dm-devel if you're having 
> trouble with it.
>
> Regards,
> Bryn.