Simulating Drive Failure on Mirrored OS drive

Simulating Drive Failure on Mirrored OS drive

[andy liebman]

Various documents on Mirrored OS Drives suggest you can simulate the 
failure of one of your mirrored pairs by marking one or more partitions 
as "failed" with mdadm.

Indeed when I mark a drive as "failed", if I have the mdadm monitoring 
daemon running, I get an email warning me that my RAIDS are degraded.

Also, I can reboot with only one drive plugged in and the OS comes up 
fine (albeit degraded).

This gives me a REASONABLE degree of confidence that the Mirrored 
Partitions will continue working okay if one drive should fail. However, 
I would like to run a more definitive test.

I tried simply unplugging one drive from its power and from its SATA 
connector. The OS didn't like that at all. My KDE session kept running, 
but I could no longer open any new terminals. I couldn't become root in 
an existing terminal that was already running. And I couldn't SSH into 
the machine.

It was like I had an OS running on only a 1/4th of a cylinder. I 
couldn't even make the OS cleanly shutdown or reboot.

I know that simply unplugging a drive is not the same as a drive failing 
or timing out. But is there a more realistic way to simulate a failure 
so that I can know that the mirror will work when it's needed?

Andy Liebman

Re: Simulating Drive Failure on Mirrored OS drive

[David Greaves]

andy liebman wrote:
> I tried simply unplugging one drive from its power and from its SATA
> connector. The OS didn't like that at all. My KDE session kept running,
> but I could no longer open any new terminals. I couldn't become root in
> an existing terminal that was already running. And I couldn't SSH into
> the machine.
That's likely to be because sata hotswap isn't supported (yet).
dmesg should give you more info.

 > I know that simply unplugging a drive is not the same as a drive failing
> or timing out. But is there a more realistic way to simulate a failure
> so that I can know that the mirror will work when it's needed?

Read up on the md-faulty device.

Also, FWIW, md works just fine :)
(Lots of other things can go wrong so testing your setup is a food idea though)

David

Re: Simulating Drive Failure on Mirrored OS drive

[andy liebman]

> 
> Read up on the md-faulty device.

Got any links to this? As I said, we know how to set the device as 
faulty, but I'm not convinced this is a good simulation of a drive that 
fails (times out, becomes unresponsive, etc.)

Andy

Re: Simulating Drive Failure on Mirrored OS drive

[Michael Tokarev]

andy liebman wrote:
> 
>>
>> Read up on the md-faulty device.
> 
> Got any links to this? As I said, we know how to set the device as
> faulty, but I'm not convinced this is a good simulation of a drive that
> fails (times out, becomes unresponsive, etc.)

Note that 'set device as faulty' is NOT the same as `md-faulty device'.
Read mdamd(8) manpage, and see options `-l' (level) and `-p' (parity)
for create mode.

/mjt