Size affects performance?

Size affects performance?

[Andrew Miklas]

Hi,

I've noticed that the size of a device seems to affect its performance.
Is this expected behaviour?

For example, on a device that is exactly 1GB, I can transfer at 16.4  
MB/s:

flanders1:~# dmsetup create test
0 2097152 linear /dev/sda6 0

flanders1:~# dd if=/dev/zero of=/dev/mapper/test bs=1K count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 65.4846 seconds, 16.4 MB/s


Yet, if I resize the device to be one sector larger, the same  
transfer happens at 59.0 MB/s:

flanders1:~# dmsetup create test
0 2097153 linear /dev/sda6 0

flanders1:~# dd if=/dev/zero of=/dev/mapper/test bs=1K count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 18.1887 seconds, 59.0 MB/s


For reference, the underlying device seems capable of ~ 60 MB/s:

flanders1:~# dd if=/dev/zero of=/dev/sda6 bs=1K count=1048576
1048576+0 records in
1048576+0 records out
1073741824 bytes (1.1 GB) copied, 18.1325 seconds, 59.2 MB/s


I've noticed that the problem either doesn't occur or isn't as  
pronounced if different block sizes are used (ie. bs=4K).
I'm running a fairly recent kernel: git tag v2.6.23-rc1-g6a30235


Thanks for any help,


Andrew Miklas

Re: Size affects performance?

[Stefan Bader]

If you use dd for performance tests I would suggest to use oflags=sync. 
Otherwise there might be caching effects.

Mit freundlichem Gruß / Regards,
Stefan Bader

SW Linux on zSeries Development
Stefan.Bader@de.ibm.com

IBM Deutschland Entwicklung GmbH
Vorsitzender des Aufsichtsrats: Martin Jetter
Geschäftsführung: Herbert Kircher
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294
----------------------------------------------------------------------------------
  When all other means of communication fail, try words.

dm-devel-bounces@redhat.com wrote on 01.08.2007 05:03:52:

> Hi,
> 
> I've noticed that the size of a device seems to affect its performance.
> Is this expected behaviour?
> 
> For example, on a device that is exactly 1GB, I can transfer at 16.4 
> MB/s:
> 
> flanders1:~# dmsetup create test
> 0 2097152 linear /dev/sda6 0
> 
> flanders1:~# dd if=/dev/zero of=/dev/mapper/test bs=1K count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes (1.1 GB) copied, 65.4846 seconds, 16.4 MB/s
> 
> 
> Yet, if I resize the device to be one sector larger, the same 
> transfer happens at 59.0 MB/s:
> 
> flanders1:~# dmsetup create test
> 0 2097153 linear /dev/sda6 0
> 
> flanders1:~# dd if=/dev/zero of=/dev/mapper/test bs=1K count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes (1.1 GB) copied, 18.1887 seconds, 59.0 MB/s
> 
> 
> For reference, the underlying device seems capable of ~ 60 MB/s:
> 
> flanders1:~# dd if=/dev/zero of=/dev/sda6 bs=1K count=1048576
> 1048576+0 records in
> 1048576+0 records out
> 1073741824 bytes (1.1 GB) copied, 18.1325 seconds, 59.2 MB/s
> 
> 
> I've noticed that the problem either doesn't occur or isn't as 
> pronounced if different block sizes are used (ie. bs=4K).
> I'm running a fairly recent kernel: git tag v2.6.23-rc1-g6a30235
> 
> 
> Thanks for any help,
> 
> 
> Andrew Miklas
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: Size affects performance?

[Andrew Miklas]

Hi,

On 1-Aug-07, at 5:24 AM, Stefan Bader wrote:

> If you use dd for performance tests I would suggest to use  
> oflags=sync.
> Otherwise there might be caching effects.
>

<snip>

Thanks for the advice.

I considered using sync, but the problem is that I don't think it  
would be a fair evaluation of the system.  Writing slows to a crawl,  
as would be expected.  However, the write speed with sync on doesn't  
reflect how I will be using the system in practice.

In case its useful, turning sync on does eliminate the effect where  
the throughput is dependent on the size of the device.
However, the problem could just be masked by the synchronous I/O  
overhead.

The problem is also noticeable when using the zero target, suggesting  
that the problem isn't in the SCSI driver:

flanders1:~# dmsetup create zero
0 20971520 zero

flanders1:~# dd if=/dev/zero of=/dev/mapper/zero bs=1K count=10485760
10485760+0 records in
10485760+0 records out
10737418240 bytes (11 GB) copied, 36.8242 seconds, 292 MB/s

-----

flanders1:~# dmsetup create zero
0 20971521 zero

flanders1:~# dd if=/dev/zero of=/dev/mapper/zero bs=1K count=10485760
10485760+0 records in
10485760+0 records out
10737418240 bytes (11 GB) copied, 55.7523 seconds, 193 MB/s



-- Andrew

Re: Size affects performance?

[Jun'ichi Nomura]

Andrew Miklas wrote:
> I've noticed that the size of a device seems to affect its performance.
> Is this expected behaviour?

The difference might be caused by internal block size,
which kernel optimizes depending on the device size alignment.
The smaller the block size, the larger the number of I/Os sent
to the device (when you read/write /dev/something).

So you might want to check the block size ("blockdev --getbsz")
and see if it's related to your case.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation of America