Linux md and iscsi problems

Linux md and iscsi problems

[aristizb]

Hello.

Currently I am working on a RAID-1 created using mdadm (version 2.6.2)  
with a local disc and a remote disc connected via iSCSI (open iscsi  
for the initiator and iscsi enterprise target).

I am testing the performance of the RAID writing data to the array  
disc having my target disc on a simulated link of 1 Mbps, I also  
defined my slow device as write-mostly and I am using the write-behind  
option so I can write with a decent speed without having to receive  
the ACK from the slow device.

My test constantly writes a file on the raid for around 20 minutes,  
but then the slow device is marked as faulty; I can recover the raid  
removing the device and adding it again, but the resynchronization  
process takes a long time (obviously because of the slow device).

Here is the message from the kernel when the slow disc fails:
sd 6:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT,SUGGEST_OK


Summarizing, I have two questions about the behavior of Linux md with  
slow devices:

1. Is it possible to modify some kind of time-out parameter on the  
mdadm tool so the slow device wouldn't be marked as faulty because of  
its slow performance.

2. Is it possible to control the "buffer" size of the RAID?, in other  
words, can I control the amount of data I can write to the local disc  
before I receive an acknowledgment from the slow device when I am  
using the write-behind option.


Any help is highly appreciated.

Thanks.


Juan Aristizabal.

Re: Linux md and iscsi problems

[Neil Brown]

On Friday February 1, aristizb@ualberta.ca wrote:
> 
> 
> Summarizing, I have two questions about the behavior of Linux md with  
> slow devices:
> 
> 1. Is it possible to modify some kind of time-out parameter on the  
> mdadm tool so the slow device wouldn't be marked as faulty because of  
> its slow performance.

No.  md doesn't do timeouts at all.  The underlying device does.
So if you are getting time out errors from the iscsi initiator, then
you need to change the timeout value used by the iscsi initiator.  md
has no part to play in this.  It just sends a request and eventually
gets either 'success' or 'fail'.

> 
> 2. Is it possible to control the "buffer" size of the RAID?, in other  
> words, can I control the amount of data I can write to the local disc  
> before I receive an acknowledgment from the slow device when I am  
> using the write-behind option.

No.  md/raid1 simply calls 'kmalloc' to get space to buffer each write
as the write arrives.  If the allocation succeeds, it is used to
perform the write lazily.  If the allocation fails, the write is
performs synchronously.

What did you hope to achieve by such tuning?  It can probably be
added if it is generally useful.

NeilBrown

Re: Linux md and iscsi problems

[aristizb]

Good morning.


Quoting Neil Brown <neilb@suse.de>:

> On Friday February 1, aristizb@ualberta.ca wrote:
>>
>>
>> Summarizing, I have two questions about the behavior of Linux md with
>> slow devices:
>>
>> 1. Is it possible to modify some kind of time-out parameter on the
>> mdadm tool so the slow device wouldn't be marked as faulty because of
>> its slow performance.
>
> No.  md doesn't do timeouts at all.  The underlying device does.
> So if you are getting time out errors from the iscsi initiator, then
> you need to change the timeout value used by the iscsi initiator.  md
> has no part to play in this.  It just sends a request and eventually
> gets either 'success' or 'fail'.
>

1. What seems strange to me is that under the same conditions running  
the test only with iscsi, the initiator never fails. I had some  
problems with iscsi on the past and they  were solved changing the  
time-out parameters on  the initiator side, but now that I added the  
md layer I am getting errors with the slow device.



>>
>> 2. Is it possible to control the "buffer" size of the RAID?, in other
>> words, can I control the amount of data I can write to the local disc
>> before I receive an acknowledgment from the slow device when I am
>> using the write-behind option.
>
> No.  md/raid1 simply calls 'kmalloc' to get space to buffer each write
> as the write arrives.  If the allocation succeeds, it is used to
> perform the write lazily.  If the allocation fails, the write is
> performs synchronously.
>
> What did you hope to achieve by such tuning?  It can probably be
> added if it is generally useful.
>
> NeilBrown
>
>

2. The idea here is to implement remote replication, by having a  
RAID-1 I can create a mirror of my local disk and use it for backup or  
for any other purposes on a centralized location.

Changing the buffer will allow me to improve performance on the  
writing, so a user will always experience local writing speed, while I  
am still sending data across the network to the mirror device.


Thanks a lot for your time.


Juan Aristizabal.