raid10 centos5 vs. centos6 300% worse random write performance

raid10 centos5 vs. centos6 300% worse random write performance

[Wes]

Why raid10 driver from Centos 6 has a 300% slower random write performance
(random read stays the same) than Centos 5?

I run tests on centos 5. 

./seekmark -f /dev/md3 -t 8 -s 1000 -w destroy-data
WRITE benchmarking against /dev/md3 1838736 MB
threads to spawn: 8
seeks per thread: 1000
io size in bytes: 512
write data is randomly generated
Spawning worker 0 to do 1000 seeks
Spawning worker 1 to do 1000 seeks
Spawning worker 2 to do 1000 seeks
Spawning worker 3 to do 1000 seeks
Spawning worker 4 to do 1000 seeks
Spawning worker 5 to do 1000 seeks
Spawning worker 6 to do 1000 seeks
Spawning worker 7 to do 1000 seeks
thread 5 completed, time: 39.75, 25.16 seeks/sec, 39.8ms per request
thread 1 completed, time: 40.99, 24.39 seeks/sec, 41.0ms per request
thread 7 completed, time: 41.35, 24.18 seeks/sec, 41.4ms per request
thread 4 completed, time: 41.59, 24.04 seeks/sec, 41.6ms per request
thread 2 completed, time: 41.69, 23.99 seeks/sec, 41.7ms per request
thread 3 completed, time: 41.90, 23.87 seeks/sec, 41.9ms per request
thread 0 completed, time: 42.23, 23.68 seeks/sec, 42.2ms per request
thread 6 completed, time: 42.24, 23.67 seeks/sec, 42.2ms per request

total time: 42.26, time per WRITE request(ms): 5.282
189.31 total seeks per sec, 23.66 WRITE seeks per sec per thread

Then installed centos 6 (the same kickstart just ISO changed) preserving
partitons.
I created raid10 with the same command as on centos 5 (mdadm -C /dev/md3
-e0.9 -n4 -l10 -pf2 -c2048 /dev/sda4 /dev/sdb4 /dev/sdc4 /dev/sdd4)
When resync completed I run the same test command and got:

WRITE benchmarking against /dev/md3 1838736 MB
threads to spawn: 8
seeks per thread: 1000
io size in bytes: 512
write data is randomly generated
Spawning worker 0 to do 1000 seeks
Spawning worker 1 to do 1000 seeks
Spawning worker 2 to do 1000 seeks
Spawning worker 3 to do 1000 seeks
Spawning worker 4 to do 1000 seeks
Spawning worker 5 to do 1000 seeks
Spawning worker 6 to do 1000 seeks
Spawning worker 7 to do 1000 seeks
thread 5 completed, time: 118.53, 8.44 seeks/sec, 118.5ms per request
thread 7 completed, time: 122.78, 8.14 seeks/sec, 122.8ms per request
thread 3 completed, time: 124.16, 8.05 seeks/sec, 124.2ms per request
thread 0 completed, time: 125.71, 7.95 seeks/sec, 125.7ms per request
thread 6 completed, time: 125.75, 7.95 seeks/sec, 125.7ms per request
thread 4 completed, time: 125.78, 7.95 seeks/sec, 125.8ms per request
thread 2 completed, time: 126.58, 7.90 seeks/sec, 126.6ms per request
thread 1 completed, time: 126.80, 7.89 seeks/sec, 126.8ms per request

total time: 126.81, time per WRITE request(ms): 15.851
63.09 total seeks per sec, 7.89 WRITE seeks per sec per thread

I recreated the array with all mdadm metas (0.9 1.2 1.1) - still the same
poor random write performance.

Please share your ideas.

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Mikael Abrahamsson]

On Thu, 25 Jul 2013, Wes wrote:

> Why raid10 driver from Centos 6 has a 300% slower random write performance
> (random read stays the same) than Centos 5?
>
> Please share your ideas.

If my research is correct, centos5 is 2.6.18 kernel and centos6 is 2.6.32 
kernel?

So just guessing, although my experience with this was from using a 
filesystem which it seems you are not doing?

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/writebarr.html
http://monolight.cc/2011/06/barriers-caches-filesystems/

Does seekmark use barriers to assure that data has actually been written? 
In that case it could be that 2.6.18 has different behaviour from 2.6.32 
when it comes to barriers and that explains the speed difference.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Wes]

Mikael Abrahamsson <swmike <at> swm.pp.se> writes:

> 
> On Thu, 25 Jul 2013, Wes wrote:
> 
> > Why raid10 driver from Centos 6 has a 300% slower random write performance
> > (random read stays the same) than Centos 5?
> >
> > Please share your ideas.
> 
> ...
> Does seekmark use barriers to assure that data has actually been written? 
> In that case it could be that 2.6.18 has different behaviour from 2.6.32 
> when it comes to barriers and that explains the speed difference.
> 


Yes, the kernels are as you stated.
I used seekmark from http://learnitwithme.com/?page_id=267 
and I am not so savvy to say if it uses barriers or not.
Linear write speed (dd) is the same on centos5 and centos6. 
Just the random write (seekmark) got horrible.

I found a way to disable barriers but for filesystems only. 
As I am testing block device not sure if barriers have something to it.

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Wes]

Mikael Abrahamsson <swmike <at> swm.pp.se> writes:

> 
> On Thu, 25 Jul 2013, Wes wrote:
> 
> > Why raid10 driver from Centos 6 has a 300% slower random write performance
> > (random read stays the same) than Centos 5?
> >
> > Please share your ideas.
> 
> ... 
> Does seekmark use barriers to assure that data has actually been written? 
> In that case it could be that 2.6.18 has different behaviour from 2.6.32 
> when it comes to barriers and that explains the speed difference.
> 


I also made random write tests with raid1 and these return the same results
on centos 5 and 6 (+/- 1%)
thread 1 completed, time: 85.34, 11.72 seeks/sec, 85.3ms per request
...
93.28 total seeks per sec, 11.66 WRITE seeks per sec per thread

And raid10 random write speed on centos 5 is twice as much as above - this
is expected as there are 4 spindles.
But centos6's raid10 (4 spindles) random write speed (~120ms/req) is even
slower than 2 spindle raid1 (~80ms/req).
This experiment seems to narrow the problem down to raid10.

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Wes]

Mikael Abrahamsson <swmike <at> swm.pp.se> writes:

> Does seekmark use barriers to assure that data has actually been written? 
> In that case it could be that 2.6.18 has different behaviour from 2.6.32 
> when it comes to barriers and that explains the speed difference.
>



Mikael, looks like you were right.

Aside from seekmark I was also testing with random dd to not relay on single
measurment tool.
 
I run found out it is not only related to raid but to block devices in
general. I run 'hdparm -W0 /dev/sda' on cetos5 and got the same poor
behavior of centos6.

Anyway I cannot still find a way to enable drive write cache on centos6.
hdparm reports it is enabled but results are the same (poor) no matter
if after 'hdparm -W0 /dev/sda' or 'hdparm -W1 /dev/sda' so now I am guessing
write cache must be blocked somewhere in the kernel.

I still cannot find a way to enable write cache in centos 6.
Booting with 'barriers=off' kernel parameter and 'barrier=0' in fstab does
not help.
 

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Marcus Sorensen]

Seekmark is very simple. It opens a block device with O_RDWR and
O_SYNC, divides the disk into block_size chunks, spawns a bunch of
threads, and each one chooses a random block, seeks there, writes,
then chooses another, seeks there, writes, etc.  There shouldn't be
any write barrier issue, since there's no filesystem involved. You can
also point it at a file on a filesystem and it will do the same with
that file, the O_SYNC *should* flush on every write.

There could be IO scheduler differences between the kernels.

On Sat, Jul 27, 2013 at 2:22 PM, Wes <wt75@gazeta.pl> wrote:
> Mikael Abrahamsson <swmike <at> swm.pp.se> writes:
>
>> Does seekmark use barriers to assure that data has actually been written?
>> In that case it could be that 2.6.18 has different behaviour from 2.6.32
>> when it comes to barriers and that explains the speed difference.
>>
>
>
>
> Mikael, looks like you were right.
>
> Aside from seekmark I was also testing with random dd to not relay on single
> measurment tool.
>
> I run found out it is not only related to raid but to block devices in
> general. I run 'hdparm -W0 /dev/sda' on cetos5 and got the same poor
> behavior of centos6.
>
> Anyway I cannot still find a way to enable drive write cache on centos6.
> hdparm reports it is enabled but results are the same (poor) no matter
> if after 'hdparm -W0 /dev/sda' or 'hdparm -W1 /dev/sda' so now I am guessing
> write cache must be blocked somewhere in the kernel.
>
> I still cannot find a way to enable write cache in centos 6.
> Booting with 'barriers=off' kernel parameter and 'barrier=0' in fstab does
> not help.
>
>
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Stan Hoeppner]

On 7/27/2013 4:01 PM, Marcus Sorensen wrote:
> Seekmark is very simple. It opens a block device with O_RDWR and
> O_SYNC, divides the disk into block_size chunks, spawns a bunch of
> threads, and each one chooses a random block, seeks there, writes,
> then chooses another, seeks there, writes, etc.  There shouldn't be
> any write barrier issue, since there's no filesystem involved. You can
> also point it at a file on a filesystem and it will do the same with
> that file, the O_SYNC *should* flush on every write.
> 
> There could be IO scheduler differences between the kernels.

~$ cat /sys/block/sda/queue/scheduler
[CFQ] noop deadline

Wes, yours will show CFQ probably as the default on RHEL/CentOS.  You'll
want deadline for best seek and all around performance.  So:

~$ echo deadline > /sys/block/sda/queue/scheduler

Add that to an init script or cron entry so it sets on every boot.

Also, make sure NCQ is working on each drive.  If it is, try disabling
it.  Look in dmesg for 4 lines like this with (depth 31/32), or at least
not zero for the first number.  Post the output for us to see.

ataX.00: xxxxxxxxx sectors, multi 16: LBA48 NCQ (depth 31/32)


>>> Does seekmark use barriers

Barriers are not an issue with this test.

WARNING: never disable filesystem write barriers unless you have a
verified to be working battery/flash backed write cache hardware RAID
controller.  If you disable barriers with individual drives on SATA
controllers and the kernel crashes or you lose power, it can corrupt the
filesystem, sometimes beyond recovery.

-- 
Stan

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Wes]

Stan Hoeppner <stan <at> hardwarefreak.com> writes:


> ~$ cat /sys/block/sda/queue/scheduler
> [CFQ] noop deadline
> Wes, yours will show CFQ probably as the default on RHEL/CentOS.  You'll
> want deadline for best seek and all around performance.  So:
> ~$ echo deadline > /sys/block/sda/queue/scheduler
> Add that to an init script or cron entry so it sets on every boot.
> Barriers are not an issue with this test.
> 

Thank you all. The issue is now closed.
RHEL5 was not doing cache flush right. It was only corrected in 2.6.32+
After removing O_SYNC from seekmark the results are now comparable.
Actually it is hard to find a linux raw device random R/W benchmark tool and
seekmark being the most popular fails when comparing pre and post 2.6.32
systems (unless you remove O_SYNC).

Re: raid10 centos5 vs. centos6 300% worse random write performance

[Stan Hoeppner]

On 8/12/2013 3:43 AM, Wes wrote:
> Stan Hoeppner <stan <at> hardwarefreak.com> writes:
> 
> 
>> ~$ cat /sys/block/sda/queue/scheduler
>> [CFQ] noop deadline
>> Wes, yours will show CFQ probably as the default on RHEL/CentOS.  You'll
>> want deadline for best seek and all around performance.  So:
>> ~$ echo deadline > /sys/block/sda/queue/scheduler
>> Add that to an init script or cron entry so it sets on every boot.
>> Barriers are not an issue with this test.
>>
> 
> Thank you all. The issue is now closed.
> RHEL5 was not doing cache flush right. It was only corrected in 2.6.32+
> After removing O_SYNC from seekmark the results are now comparable.

Glad you got it figured out.

> Actually it is hard to find a linux raw device random R/W benchmark tool and

FIO is good for raw IO benchmarking.  Can do file based IO as well.
Very flexible, but maybe a bit complicated for first time users.

> seekmark being the most popular fails when comparing pre and post 2.6.32
> systems (unless you remove O_SYNC).

Dunno about seekmark being the most popular.  I'd never heard of it
until this thread.  I'd have guessed FIO was most popular.  But then
again I don't get out much. ;)

-- 
Stan