raid io requests not parallel?

raid io requests not parallel?

[Jonathan Baccash]

I'm using kernel linux-2.6.15-gentoo-r1, and I noticed performance of
the software RAID-1 is not as good as I would have expected on my two
SATA drives, and I was wondering if anyone has an idea what may be
happening. The test I run is 1024 16k direct-IO reads/writes from
random locations within a 1GB file (on a RAID-1 partition), with my
disk caches set to
write-through mode. In the MT (multi-threaded) case, I issue them from
8 threads (so it's 128 requests per thread):

Random read: 10.295 sec
Random write: 19.142 sec
MT Random read: 5.276 sec
MT Random write: 19.839 sec

As expected, the multi-threaded reads are 2x as fast as single-threaded
reads. But I would have expected (assuming the write to both disks can
occur in parallel) that the random writes are about the same speed (10
seconds) as the single-threaded random reads, for both the
single-threaded and multi-threaded write cases. The fact that the
multi-threaded reads were
twice as fast indicates to me that read requests can occur in parallel.

So.... why doesn't the raid issue the writes in parallel? Thanks in
advance for any help.

Jon.

Re: raid io requests not parallel?

[Arjan van de Ven]

> So.... why doesn't the raid issue the writes in parallel? 

because you use O_DIRECT, which means your application waits for the
write to hit the disk before doing the next write ;)

Re: raid io requests not parallel?

[Robert Hancock]

Jonathan Baccash wrote:
> I'm using kernel linux-2.6.15-gentoo-r1, and I noticed performance of
> the software RAID-1 is not as good as I would have expected on my two
> SATA drives, and I was wondering if anyone has an idea what may be
> happening. The test I run is 1024 16k direct-IO reads/writes from
> random locations within a 1GB file (on a RAID-1 partition), with my
> disk caches set to
> write-through mode. In the MT (multi-threaded) case, I issue them from
> 8 threads (so it's 128 requests per thread):
> 
> Random read: 10.295 sec
> Random write: 19.142 sec
> MT Random read: 5.276 sec
> MT Random write: 19.839 sec
> 
> As expected, the multi-threaded reads are 2x as fast as single-threaded
> reads. But I would have expected (assuming the write to both disks can
> occur in parallel) that the random writes are about the same speed (10
> seconds) as the single-threaded random reads, for both the
> single-threaded and multi-threaded write cases. The fact that the
> multi-threaded reads were
> twice as fast indicates to me that read requests can occur in parallel.
> 
> So.... why doesn't the raid issue the writes in parallel? Thanks in
> advance for any help.

Because it's RAID 1, I'm assuming. The data has to be written to both 
drives, unlike reads which can be satisfied from either one.

-- 
Robert Hancock      Saskatoon, SK, Canada
To email, remove "nospam" from hancockr@nospamshaw.ca
Home Page: http://www.roberthancock.com/

Re: raid io requests not parallel?

[Chuck Ebbert]

In-Reply-To: <e0e4cb3e0607151704o479371afpc9332a08fb84ba09@mail.gmail.com>

On Sat, 15 Jul 2006 17:04:57 -0700, Jonathan Baccash wrote:

> As expected, the multi-threaded reads are 2x as fast as single-threaded
> reads. But I would have expected (assuming the write to both disks can
> occur in parallel) that the random writes are about the same speed (10
> seconds) as the single-threaded random reads, for both the
> single-threaded and multi-threaded write cases. The fact that the
> multi-threaded reads were
> twice as fast indicates to me that read requests can occur in parallel.
> 
> So.... why doesn't the raid issue the writes in parallel? Thanks in
> advance for any help.

But it does issue writes in parallel.  The problem is in the way RAID1
works.  When you do a read from the RAID1, it issues one read request
to one of the underlying disks.  When you do a write, it issues one write
request to _each_ of the underlying disks.  So with two disks in the
mirror set, every write issued by your program causes two disk writes.

-- 
Chuck
 "You can't read a newspaper if you can't read."  --George W. Bush

Re: raid io requests not parallel?

[Avi Kivity]

Jonathan Baccash wrote:
>
> I'm using kernel linux-2.6.15-gentoo-r1, and I noticed performance of
> the software RAID-1 is not as good as I would have expected on my two
> SATA drives, and I was wondering if anyone has an idea what may be
> happening. The test I run is 1024 16k direct-IO reads/writes from
> random locations within a 1GB file (on a RAID-1 partition), with my
> disk caches set to
> write-through mode. In the MT (multi-threaded) case, I issue them from
> 8 threads (so it's 128 requests per thread):
>
> Random read: 10.295 sec
> Random write: 19.142 sec
> MT Random read: 5.276 sec
> MT Random write: 19.839 sec
>
> As expected, the multi-threaded reads are 2x as fast as single-threaded
> reads. But I would have expected (assuming the write to both disks can
> occur in parallel) that the random writes are about the same speed (10
> seconds) as the single-threaded random reads, for both the
> single-threaded and multi-threaded write cases. The fact that the
> multi-threaded reads were
> twice as fast indicates to me that read requests can occur in parallel.
>
> So.... why doesn't the raid issue the writes in parallel? Thanks in
> advance for any help.
>
The writes are issued in parallel, but both disks have to be written.  
Each head has to service 1024 write requests (compared to just 512 read 
requests).


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: raid io requests not parallel?

[Jonathan Baccash]

> Each head has to service 1024 write requests (compared to just 512 read
> requests).

By that logic, it would take twice as long for my writes to finish.
Why is it taking 4x as long in my parallel test? I don't think any of
the repliers understood my question. What I want to know is, during a
single direct IO write request, I would expect the raid software to
issue the write request to both disks simultaneously. But the evidence
indicates that isn't what is happening. I would expect a raid-1 write
to take about as long to write a single block as a single write to a
single disk (assuming no other disk activity), because I would expect
two writes to happen concurrently.

Re: raid io requests not parallel?

[Chuck Ebbert]

In-Reply-To: <e0e4cb3e0607160938k70819e40g4172f5917045ebf8@mail.gmail.com>

On Sun, 16 Jul 2006 09:38:25 -0700, Jonathan Baccash wrote:

> > Each head has to service 1024 write requests (compared to just 512 read
> > requests).
>
> By that logic, it would take twice as long for my writes to finish.
> Why is it taking 4x as long in my parallel test?

Because a single read not only goes to just one disk, it is sent to
the disk with the lowest expected seek time for that request.  This
cuts average read time in half, on average.

(See drivers/md/raid1.c::read_balance().)

> I would expect a raid-1 write
> to take about as long to write a single block as a single write to a
> single disk (assuming no other disk activity), because I would expect
> two writes to happen concurrently.

You didn't post any benchmarks showing results for single write to a
single disk.

-- 
Chuck
 Think. Or you will be replaced with a small shell script.  --dwmw2

Re: raid io requests not parallel?

[Jonathan Baccash]

> Because a single read not only goes to just one disk, it is sent to
> the disk with the lowest expected seek time for that request.  This
> cuts average read time in half, on average.
>
> (See drivers/md/raid1.c::read_balance().)

Thanks Chuck! That is making some sense, although I think I'll have to
read and think about it some more before I quite understand.

> You didn't post any benchmarks showing results for single write to a
> single disk.

I didn't think I could get any repeatable results for a single write,
so I was trying to understand what is happening based on runs of
longer writes.

Re: raid io requests not parallel?

[Neil Brown]

On Saturday July 15, jbaccash@gmail.com wrote:
> I'm using kernel linux-2.6.15-gentoo-r1, and I noticed performance of
> the software RAID-1 is not as good as I would have expected on my two
> SATA drives, and I was wondering if anyone has an idea what may be
> happening. The test I run is 1024 16k direct-IO reads/writes from
> random locations within a 1GB file (on a RAID-1 partition), with my
> disk caches set to
> write-through mode. In the MT (multi-threaded) case, I issue them from
> 8 threads (so it's 128 requests per thread):
> 
> Random read: 10.295 sec
> Random write: 19.142 sec

Odd.  I would expect these two numbers to be a lot closer together.

Try changing the IO scheduler on the drives and see if it makes a
difference.
e.g.
  cat /sys/block/XXX/queue/scheduler
  echo cfq > /sys/block/XXX/queue/scheduler
  echo deadline > /sys/block/XXX/queue/scheduler

See what works best.

NeilBrown



> MT Random read: 5.276 sec
> MT Random write: 19.839 sec
> 
> As expected, the multi-threaded reads are 2x as fast as single-threaded
> reads. But I would have expected (assuming the write to both disks can
> occur in parallel) that the random writes are about the same speed (10
> seconds) as the single-threaded random reads, for both the
> single-threaded and multi-threaded write cases. The fact that the
> multi-threaded reads were
> twice as fast indicates to me that read requests can occur in parallel.
> 
> So.... why doesn't the raid issue the writes in parallel? Thanks in
> advance for any help.
> 
> Jon.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: raid io requests not parallel?

[Jens Axboe]

On Mon, Jul 17 2006, Neil Brown wrote:
> On Saturday July 15, jbaccash@gmail.com wrote:
> > I'm using kernel linux-2.6.15-gentoo-r1, and I noticed performance of
> > the software RAID-1 is not as good as I would have expected on my two
> > SATA drives, and I was wondering if anyone has an idea what may be
> > happening. The test I run is 1024 16k direct-IO reads/writes from
> > random locations within a 1GB file (on a RAID-1 partition), with my
> > disk caches set to
> > write-through mode. In the MT (multi-threaded) case, I issue them from
> > 8 threads (so it's 128 requests per thread):
> > 
> > Random read: 10.295 sec
> > Random write: 19.142 sec
> 
> Odd.  I would expect these two numbers to be a lot closer together.
> 
> Try changing the IO scheduler on the drives and see if it makes a
> difference.
> e.g.
>   cat /sys/block/XXX/queue/scheduler
>   echo cfq > /sys/block/XXX/queue/scheduler
>   echo deadline > /sys/block/XXX/queue/scheduler
> 
> See what works best.

His cache is set to write through, 16kb direct writes in that case will
be a lot slower than the equivalent reads. 10 vs 20 seconds does not
sounds out of the question.

-- 
Jens Axboe