Out of sync mirrors

Out of sync mirrors

[Ana Paula]

Hi,
I am using linux 2.6.8 for my research on raid systems.

I need to have the disks of an array out of sync for a while.
I want to be able to stop a disk request queue (blk_stop_queue) and
start it sometime later. Meanwhile its mirrors should be serving r/w
as if this mirror was up.
Whenever the stopped mirror comes back it will serve all the requests
already sent to it.

I am queueing up the requests sent to all others while my mirror queue
is stopped.

The problem is that the other mirrors stop serving as well. Even if I
don't stop the queue, only delay its request a little bit, still the
others also slow down.

Question: how do I get raid to allow mirrors to be out of sync?

Obviously, I am missing something out in the code.

Any clues?

Thanks,
Ana Paula

Re: Out of sync mirrors

[Bill Davidsen]

Ana Paula wrote:
> Hi,
> I am using linux 2.6.8 for my research on raid systems.
>
> I need to have the disks of an array out of sync for a while.
> I want to be able to stop a disk request queue (blk_stop_queue) and
> start it sometime later. Meanwhile its mirrors should be serving r/w
> as if this mirror was up.
> Whenever the stopped mirror comes back it will serve all the requests
> already sent to it.
>
> I am queueing up the requests sent to all others while my mirror queue
> is stopped.
>
> The problem is that the other mirrors stop serving as well. Even if I
> don't stop the queue, only delay its request a little bit, still the
> others also slow down.
>
> Question: how do I get raid to allow mirrors to be out of sync?
>
> Obviously, I am missing something out in the code.
>
> Any clues?
>   

Take a look at "write-mostly" and see if creative use of it will assist. 
I suspect you're doing something which needs a bitmap, as well.

-- 
Bill Davidsen <davidsen@tmr.com>
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

Re: Out of sync mirrors

[Ana Paula]

I got an idea today, of not letting the sync thread start while the
disks are out of sync.

I am not sure if it will suffice to fool the raid.

-AP

On Thu, Jul 24, 2008 at 6:37 PM, Bill Davidsen <davidsen@tmr.com> wrote:
>
> Ana Paula wrote:
>>
>> Hi,
>> I am using linux 2.6.8 for my research on raid systems.
>>
>> I need to have the disks of an array out of sync for a while.
>> I want to be able to stop a disk request queue (blk_stop_queue) and
>> start it sometime later. Meanwhile its mirrors should be serving r/w
>> as if this mirror was up.
>> Whenever the stopped mirror comes back it will serve all the requests
>> already sent to it.
>>
>> I am queueing up the requests sent to all others while my mirror queue
>> is stopped.
>>
>> The problem is that the other mirrors stop serving as well. Even if I
>> don't stop the queue, only delay its request a little bit, still the
>> others also slow down.
>>
>> Question: how do I get raid to allow mirrors to be out of sync?
>>
>> Obviously, I am missing something out in the code.
>>
>> Any clues?
>>
>
> Take a look at "write-mostly" and see if creative use of it will assist. I suspect you're doing something which needs a bitmap, as well.
>
> --
> Bill Davidsen <davidsen@tmr.com>
>  "Woe unto the statesman who makes war without a reason that will still
>  be valid when the war is over..." Otto von Bismark
>

RE: Out of sync mirrors

[David Lethe]

>Hi,
>I am using linux 2.6.8 for my research on raid systems.
>
>I need to have the disks of an array out of sync for a while.
>I want to be able to stop a disk request queue (blk_stop_queue) and
>start it sometime later. Meanwhile its mirrors should be serving r/w
>as if this mirror was up.
>Whenever the stopped mirror comes back it will serve all the requests
>already sent to it.
>
>I am queueing up the requests sent to all others while my mirror queue
>is stopped.
>
>The problem is that the other mirrors stop serving as well. Even if I
>don't stop the queue, only delay its request a little bit, still the
>others also slow down.
>
>Question: how do I get raid to allow mirrors to be out of sync?
>
>Obviously, I am missing something out in the code.
>
>Any clues?
>
>Thanks,
>Ana Paula

Ana:

You are making this way too hard for yourself
If you want to stop the queue but let the RAID system continue
processing the queue, then just sending I/O from whatever program you
are using to generate the load.  This won't let you start up with a
partially unflushed queue, but if for whatever reason you need to
measure effects of pending I/Os and timing, then you can obtain a
reasonably good simulation by increasing the queue depth and repeating
the test.

As for A way to get RAID (I assume software RAID, a la the md driver),
to get out of sync is to "remove" a disk drive.  If you are afraid to
crack the case, then use a cheap USB enclosure for one of the disks and
then turn it off or unplug the USB connector.  You can also "remove" a
disk by sending it a command to go into a self-test, perform a low-level
format, send a mode select to change the block size, there are lots of
ways if you think outside of the box.

All of these "failure" scenarios will interact differently, just turning
a disk drive off creates a real failure, not a quasi-emulated more
graceful failure that may be easier for you to time and measure the
effects.

Good luck.  

David @ santools.com

Re: Out of sync mirrors

[Ana Paula]

> Ana:
>
> You are making this way too hard for yourself
> If you want to stop the queue but let the RAID system continue
> processing the queue, then just sending I/O from whatever program you
> are using to generate the load.  This won't let you start up with a
> partially unflushed queue, but if for whatever reason you need to
> measure effects of pending I/Os and timing, then you can obtain a
> reasonably good simulation by increasing the queue depth and repeating
> the test.

I didn't quite understood what you are saying. I need to stop the
queue in order to see its growth.

>
> As for A way to get RAID (I assume software RAID, a la the md driver),
> to get out of sync is to "remove" a disk drive.  If you are afraid to
> crack the case, then use a cheap USB enclosure for one of the disks and
> then turn it off or unplug the USB connector.  You can also "remove" a
> disk by sending it a command to go into a self-test, perform a low-level
> format, send a mode select to change the block size, there are lots of
> ways if you think outside of the box.

By doing this I wouldn't see its queue building up, would I?

Basically I need to simulate a raid with disks of different speeds.
Where a faster mirror would process its requests before the slower
ones.
For reads it processes more requests to keep the raid load balanced.
But for writes I can't load balance, all have to write. The problem is
that the faster is slowing down to meet the slower disk throughput.

See the queue build up is key.

Thanks,
AP

>
> All of these "failure" scenarios will interact differently, just turning
> a disk drive off creates a real failure, not a quasi-emulated more
> graceful failure that may be easier for you to time and measure the
> effects.
>
> Good luck.
>
> David @ santools.com
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>