Write-intent bitmap decreases or increase performance of RAID5?

Write-intent bitmap decreases or increase performance of RAID5?

[Shaochun Wang]

From what I read from the internet, write-intent bitmap decreases raid
performance and provides more stability. But in my situation, it also
increases raid performance dramatically!

My RAID5 write speed without write-intent bitmap is 90MB/s, and is
increased to 200MB/s after enabling internal write-intent bitmap. I
use ext4 filesystem on this raid array. The following is my RAID5
array information:

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.0
  Creation Time : Mon Nov 16 02:46:50 2009
     Raid Level : raid5
     Array Size : 10255968576 (9780.85 GiB 10502.11 GB)
  Used Dev Size : 1465138368 (1397.26 GiB 1500.30 GB)
   Raid Devices : 8
  Total Devices : 8
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Wed Jun 30 03:51:56 2010
          State : active
 Active Devices : 8
Working Devices : 8
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 64K

           Name : AOpen:00
           UUID : abb2242c:6a250d29:df65e52b:426f4ed9
         Events : 405636

    Number   Major   Minor   RaidDevice State
       4       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1
       3       8       65        3      active sync   /dev/sde1
       5       8       81        4      active sync   /dev/sdf1
       6       8       97        5      active sync   /dev/sdg1
       7       8      113        6      active sync   /dev/sdh1
       8       8      129        7      active sync   /dev/sdi1

-- 
Shaochun Wang <scwang@ios.ac.cn>

Jabber: fungusw@jabber.org

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 1020 bytes --]

On Wed, 30 Jun 2010 04:11:54 +0800
Shaochun Wang <scwang@ios.ac.cn> wrote:

> From what I read from the internet, write-intent bitmap decreases raid
> performance and provides more stability. But in my situation, it also
> increases raid performance dramatically!
> 
> My RAID5 write speed without write-intent bitmap is 90MB/s, and is
> increased to 200MB/s after enabling internal write-intent bitmap. I
> use ext4 filesystem on this raid array.

How do you measure that speed exactly? The proper (and also quite simple) way
to do it is to run the following test several times (assuming that current dir
is some directory on the array):

  dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc

Also, I see you run an 8-drive RAID5 on 1.5TB drives, so make sure you are
aware of the issues described in:
  http://hardware.slashdot.org/article.pl?sid=08/10/21/2126252
  http://louwrentius.blogspot.com/2008/08/why-raid-1-and-5-will-kill-you-some-day.html

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Mikael Abrahamsson]

On Wed, 30 Jun 2010, Roman Mamedov wrote:

>  dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc

To be a bit more sure about accurate measurement, make sure the amount of 
data you're transferring is twice the amount of RAM in the machine, the 
above is only 2 gigs of data.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On Wed, 30 Jun 2010 07:54:55 +0200 (CEST)
Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 30 Jun 2010, Roman Mamedov wrote:
> 
> >  dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
> 
> To be a bit more sure about accurate measurement, make sure the amount of 
> data you're transferring is twice the amount of RAM in the machine, the 
> above is only 2 gigs of data.

This is not necessary. See the dd man page about the conv=fdatasync switch.
You can even test with just 256 MB this way, and still get an extremely
accurate result.

Three consecutive runs with "count=256" on a machine with 4GB of RAM:
268435456 bytes (268 MB) copied, 4.76846 s, 56.3 MB/s
268435456 bytes (268 MB) copied, 4.79682 s, 56.0 MB/s
268435456 bytes (268 MB) copied, 4.79307 s, 56.0 MB/s

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Mikael Abrahamsson]

On Wed, 30 Jun 2010, Roman Mamedov wrote:

> This is not necessary. See the dd man page about the conv=fdatasync switch.
> You can even test with just 256 MB this way, and still get an extremely
> accurate result.

You're right, for writing this is ok. For read tests it's important to use 
more than ram though, to avoid doing performance testing from block cache 
instead of from the actual drives.

-- 
Mikael Abrahamsson    email: swmike@swm.pp.se

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 696 bytes --]

On Wed, 30 Jun 2010 08:58:28 +0200 (CEST)
Mikael Abrahamsson <swmike@swm.pp.se> wrote:

> On Wed, 30 Jun 2010, Roman Mamedov wrote:
> 
> > This is not necessary. See the dd man page about the conv=fdatasync switch.
> > You can even test with just 256 MB this way, and still get an extremely
> > accurate result.
> 
> You're right, for writing this is ok. For read tests it's important to use 
> more than ram though, to avoid doing performance testing from block cache 
> instead of from the actual drives.

Yeah, though there's also an alternative way of mitigating the effect of
the cache when testing read speed, see http://linux-mm.org/Drop_Caches

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[CoolCold]

flag: direct use direct I/O for data
can be useful too

On Wed, Jun 30, 2010 at 12:23 PM, Roman Mamedov <roman@rm.pp.ru> wrote:
> On Wed, 30 Jun 2010 08:58:28 +0200 (CEST)
> Mikael Abrahamsson <swmike@swm.pp.se> wrote:
>
>> On Wed, 30 Jun 2010, Roman Mamedov wrote:
>>
>> > This is not necessary. See the dd man page about the conv=fdatasync switch.
>> > You can even test with just 256 MB this way, and still get an extremely
>> > accurate result.
>>
>> You're right, for writing this is ok. For read tests it's important to use
>> more than ram though, to avoid doing performance testing from block cache
>> instead of from the actual drives.
>
> Yeah, though there's also an alternative way of mitigating the effect of
> the cache when testing read speed, see http://linux-mm.org/Drop_Caches
>
> --
> With respect,
> Roman
>



-- 
Best regards,
[COOLCOLD-RIPN]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Bill Davidsen]

Mikael Abrahamsson wrote:
> On Wed, 30 Jun 2010, Roman Mamedov wrote:
>
>> This is not necessary. See the dd man page about the conv=fdatasync 
>> switch.
>> You can even test with just 256 MB this way, and still get an extremely
>> accurate result.
>
> You're right, for writing this is ok. For read tests it's important to 
> use more than ram though, to avoid doing performance testing from 
> block cache instead of from the actual drives.
>
See /proc/sys/vm/drop_caches.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Shaochun Wang]

On Wed, Jun 30, 2010 at 07:54:55AM +0200, Mikael Abrahamsson wrote:
> On Wed, 30 Jun 2010, Roman Mamedov wrote:
> 
> >  dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
> 
> To be a bit more sure about accurate measurement, make sure the amount of 
> data you're transferring is twice the amount of RAM in the machine, the 
> above is only 2 gigs of data.
As suggested, I did the test again and got the following result:

	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc
	Password:
	5000+0 records in
	5000+0 records out
	5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s

	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc
	5000+0 records in
	5000+0 records out
	5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
	
I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.

-- 
Shaochun Wang(王绍春) <scwang@ios.ac.cn>
State Key Laboratory of Computer Science,
Institute of Software,
Chinese Academy of Sciences
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Majed B.]

How much RAM does your machine have?

Write to the disk an amount that is larger than the amount of RAM and
see if you get the same result.

Also you can use hdparm to flush the caches before running dd the 2nd time.

2010/7/1 Shaochun Wang <scwang@ios.ac.cn>
>
> On Wed, Jun 30, 2010 at 07:54:55AM +0200, Mikael Abrahamsson wrote:
> > On Wed, 30 Jun 2010, Roman Mamedov wrote:
> >
> > >  dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
> >
> > To be a bit more sure about accurate measurement, make sure the amount of
> > data you're transferring is twice the amount of RAM in the machine, the
> > above is only 2 gigs of data.
> As suggested, I did the test again and got the following result:
>
>        -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc
>        Password:
>        5000+0 records in
>        5000+0 records out
>        5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s
>
>        -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc
>        5000+0 records in
>        5000+0 records out
>        5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
>
> I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
>
> --
> Shaochun Wang(王绍春) <scwang@ios.ac.cn>
> State Key Laboratory of Computer Science,
> Institute of Software,
> Chinese Academy of Sciences
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--
      Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 441 bytes --]

On Thu, 1 Jul 2010 23:30:26 +0300
"Majed B." <majedb@gmail.com> wrote:

> How much RAM does your machine have?
> 
> Write to the disk an amount that is larger than the amount of RAM and
> see if you get the same result.
> 
> Also you can use hdparm to flush the caches before running dd the 2nd time.

Here we go again. Read up on what conv=fdatasync means, and why cache is
irrelevant in this test.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Bill Davidsen]

Majed B. wrote:
> How much RAM does your machine have?
>
> Write to the disk an amount that is larger than the amount of RAM and
> see if you get the same result.
>
> Also you can use hdparm to flush the caches before running dd the 2nd time.
>   

See the meaning of the fdatasync option to dd and the 
/proc/sys/vm/drop_cache control. You are right about hdparm, but it only 
matters if you are using tiny data, drive caches are small.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 886 bytes --]

On Thu, 1 Jul 2010 14:40:44 +0800
Shaochun Wang <scwang@ios.ac.cn> wrote:

> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
> conv=fdatasync,notrunc Password:
> 	5000+0 records in
> 	5000+0 records out
> 	5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s
> 
> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
> conv=fdatasync,notrunc 5000+0 records in
> 	5000+0 records out
> 	5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
> 	
> I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.

That's because the first time the filesystem had to increase the file's size
5000 times by allocating additional 1 MB, and the second time it was just
writing to an already allocated file. If you see such a big difference here,
run that test 3 or more times, and discard the first run's results.


-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Write-intent bitmap decreases or increase performance of RAID5?

[John Robinson]

On 01/07/2010 21:49, Roman Mamedov wrote:
> On Thu, 1 Jul 2010 14:40:44 +0800
> Shaochun Wang <scwang@ios.ac.cn> wrote:
> 
>> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
>> conv=fdatasync,notrunc Password:
>> 	5000+0 records in
>> 	5000+0 records out
>> 	5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s
>>
>> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
>> conv=fdatasync,notrunc 5000+0 records in
>> 	5000+0 records out
>> 	5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
>> 	
>> I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
> 
> That's because the first time the filesystem had to increase the file's size
> 5000 times by allocating additional 1 MB, and the second time it was just
> writing to an already allocated file. If you see such a big difference here,
> run that test 3 or more times, and discard the first run's results.

Or use a proper filesystem benchmarking tool like bonnie++ and read its 
documentation so you know what it's telling you. dd is (in my opinion) 
only really any use for testing raw device streaming write/read speed.

And no, I don't understand why you get better performance with the 
write-intent bitmap turned on, unless you said that because you saw 
something like the above (as Roman says, your initial conditions were 
different so it's not a valid comparison). Usually, you need to tweak 
the write-intent bitmap's chunk size to suit your array and desired 
recovery speed to avoid it killing performance. I use a 16MB chunk on 
arrays with a few cheap drives, others go as high as 128MB on arrays 
with lots of high-performance quality drives.

Cheers,

John.

Re: Write-intent bitmap decreases or increase performance of RAID5?

[Bill Davidsen]

Roman Mamedov wrote:
> On Thu, 1 Jul 2010 14:40:44 +0800
> Shaochun Wang <scwang@ios.ac.cn> wrote:
>
>   
>> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
>> conv=fdatasync,notrunc Password:
>> 	5000+0 records in
>> 	5000+0 records out
>> 	5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s
>>
>> 	-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000
>> conv=fdatasync,notrunc 5000+0 records in
>> 	5000+0 records out
>> 	5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
>> 	
>> I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
>>     
>
> That's because the first time the filesystem had to increase the file's size
> 5000 times by allocating additional 1 MB, and the second time it was just
> writing to an already allocated file. If you see such a big difference here,
> run that test 3 or more times, and discard the first run's results.
>
>   
Unless the notrunc option is used, the file is truncated and allocated 
all over again. I'm pretty sure the inode is reused, that's down in the 
noise.

-- 
Bill Davidsen <davidsen@tmr.com>
  "We can't solve today's problems by using the same thinking we
   used in creating them." - Einstein