why do I get bad disk write performance in the kernel 3.1?

why do I get bad disk write performance in the kernel 3.1?

[Roberto Scudeller]

[-- Attachment #1.1: Type: text/plain, Size: 716 bytes --]

Hi all,

I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
drivers, and receive an amazing result for reading, but the write numbers
are very disappointing.

I run the VM in local disk, and receive these numbers:
Seq read: 244 MB/s
512K read: 239 MB/s
4K read: 27 MB/s
4K QD 32: 90 MB/s

Seq write: 20 MB/s
512K: 20 MB/s
4K: 8 MB/s
4K QD 32: 12MB/s

In older kernel, 2.6.32.x:
Seq read: 189 MB/s
512K read: 169 MB/s
4K read: 11 MB/s
4K QD 32: 34 MB/s

Seq write: 177 MB/s
512K: 166 MB/s
4K: 12 MB/s
4K QD 32: 36 MB/s

Could anyone help me. Why is the new kernel faster in reading, but too slow
writing ?

-- 
Roberto Scudeller

[-- Attachment #1.2: Type: text/html, Size: 888 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why do I get bad disk write performance in the kernel 3.1?

[Konrad Rzeszutek Wilk]

On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:
> Hi all,
> 
> I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
> I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
> drivers, and receive an amazing result for reading, but the write numbers
> are very disappointing.
> 
> I run the VM in local disk, and receive these numbers:
> Seq read: 244 MB/s
> 512K read: 239 MB/s
> 4K read: 27 MB/s
> 4K QD 32: 90 MB/s
> 
> Seq write: 20 MB/s
> 512K: 20 MB/s
> 4K: 8 MB/s
> 4K QD 32: 12MB/s
> 
> In older kernel, 2.6.32.x:
> Seq read: 189 MB/s
> 512K read: 169 MB/s
> 4K read: 11 MB/s
> 4K QD 32: 34 MB/s
> 
> Seq write: 177 MB/s
> 512K: 166 MB/s
> 4K: 12 MB/s
> 4K QD 32: 36 MB/s
> 
> Could anyone help me. Why is the new kernel faster in reading, but too slow
> writing ?

It would help if you gave an idea of what the guest configuration is and
what is the backend. It might be that the backend you are using in 3.1 is
QEMU qdisk, not xen-blkback.

Re: why do I get bad disk write performance in the kernel 3.1?

[Roberto Scudeller]

[-- Attachment #1.1: Type: text/plain, Size: 1784 bytes --]

Konrad,

Thanks for help.

My windows.cfg is:
name='benchCM-windows-2003-64b-std-test'
kernel='/usr/lib/xen/boot/hvmloader'
builder='hvm'
memory=512
vcpus=2
pae=1
acpi=1
apic=1
disk=[ 'tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w' ]
device_model='/usr/lib/xen/bin/qemu-dm'
boot='c'
sdl=0
vnc=1
vncunused=1
vnclisten='0.0.0.0'
vncpasswd=''
stdvga=0
extra=''
on_reboot='restart'
on_shutdown='destroy'
ramdisk=''
image_name=''
on_crash='destroy'
bootloader=''
root=''
platform='xen'
network_mode='tap'
usb = 1
usbdevice = 'tablet'

The xvda disk is a raw disk, created with dd.
I use the windows.cfg in both kernel.



2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>

> On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:
> > Hi all,
> >
> > I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
> > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
> > drivers, and receive an amazing result for reading, but the write numbers
> > are very disappointing.
> >
> > I run the VM in local disk, and receive these numbers:
> > Seq read: 244 MB/s
> > 512K read: 239 MB/s
> > 4K read: 27 MB/s
> > 4K QD 32: 90 MB/s
> >
> > Seq write: 20 MB/s
> > 512K: 20 MB/s
> > 4K: 8 MB/s
> > 4K QD 32: 12MB/s
> >
> > In older kernel, 2.6.32.x:
> > Seq read: 189 MB/s
> > 512K read: 169 MB/s
> > 4K read: 11 MB/s
> > 4K QD 32: 34 MB/s
> >
> > Seq write: 177 MB/s
> > 512K: 166 MB/s
> > 4K: 12 MB/s
> > 4K QD 32: 36 MB/s
> >
> > Could anyone help me. Why is the new kernel faster in reading, but too
> slow
> > writing ?
>
> It would help if you gave an idea of what the guest configuration is and
> what is the backend. It might be that the backend you are using in 3.1 is
> QEMU qdisk, not xen-blkback.
>



-- 
Roberto Scudeller

[-- Attachment #1.2: Type: text/html, Size: 2624 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why do I get bad disk write performance in the kernel 3.1?

[Roberto Scudeller]

[-- Attachment #1.1: Type: text/plain, Size: 2425 bytes --]

Konrad,

# xenstore-ls /local/domain/6/device/vbd
51712 = ""
 backend = "/local/domain/0/backend/qdisk/6/51712"
 backend-id = "0"
 state = "4"
 virtual-device = "51712"
 device-type = "disk"
 ring-ref = "16383"
 event-channel = "7"
51728 = ""
 backend = "/local/domain/0/backend/qdisk/6/51728"
 backend-id = "0"
 state = "4"
 virtual-device = "51728"
 device-type = "disk"
 ring-ref = "16371"
 event-channel = "8"

Thanks for help.



2011/11/9 Roberto Scudeller <beto.rvs@gmail.com>

> Konrad,
>
> Thanks for help.
>
> My windows.cfg is:
> name='benchCM-windows-2003-64b-std-test'
> kernel='/usr/lib/xen/boot/hvmloader'
> builder='hvm'
> memory=512
> vcpus=2
> pae=1
> acpi=1
> apic=1
> disk=[ 'tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w' ]
> device_model='/usr/lib/xen/bin/qemu-dm'
> boot='c'
> sdl=0
> vnc=1
> vncunused=1
> vnclisten='0.0.0.0'
> vncpasswd=''
> stdvga=0
> extra=''
> on_reboot='restart'
> on_shutdown='destroy'
> ramdisk=''
> image_name=''
> on_crash='destroy'
> bootloader=''
> root=''
> platform='xen'
> network_mode='tap'
> usb = 1
> usbdevice = 'tablet'
>
> The xvda disk is a raw disk, created with dd.
> I use the windows.cfg in both kernel.
>
>
>
>
> 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>
>> On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:
>> > Hi all,
>> >
>> > I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
>> > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
>> > drivers, and receive an amazing result for reading, but the write
>> numbers
>> > are very disappointing.
>> >
>> > I run the VM in local disk, and receive these numbers:
>> > Seq read: 244 MB/s
>> > 512K read: 239 MB/s
>> > 4K read: 27 MB/s
>> > 4K QD 32: 90 MB/s
>> >
>> > Seq write: 20 MB/s
>> > 512K: 20 MB/s
>> > 4K: 8 MB/s
>> > 4K QD 32: 12MB/s
>> >
>> > In older kernel, 2.6.32.x:
>> > Seq read: 189 MB/s
>> > 512K read: 169 MB/s
>> > 4K read: 11 MB/s
>> > 4K QD 32: 34 MB/s
>> >
>> > Seq write: 177 MB/s
>> > 512K: 166 MB/s
>> > 4K: 12 MB/s
>> > 4K QD 32: 36 MB/s
>> >
>> > Could anyone help me. Why is the new kernel faster in reading, but too
>> slow
>> > writing ?
>>
>> It would help if you gave an idea of what the guest configuration is and
>> what is the backend. It might be that the backend you are using in 3.1 is
>> QEMU qdisk, not xen-blkback.
>>
>
>
>
> --
> Roberto Scudeller
>
>
>


-- 
Roberto Scudeller

[-- Attachment #1.2: Type: text/html, Size: 3734 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why do I get bad disk write performance in the kernel 3.1?

[Konrad Rzeszutek Wilk]

On Wed, Nov 09, 2011 at 02:52:59PM -0200, Roberto Scudeller wrote:
> Konrad,
> 
> Thanks for help.

OK, so looks like you are using qdisk. Can you try creating an
LVM and blasting the image onto that?

And obviously then using 'phy:/dev/vg_guest/...' ?

You could also try the 'file:/local-disk' which will setup a loopback device
and use that. Try that as well.

> 
> My windows.cfg is:
> name='benchCM-windows-2003-64b-std-test'
> kernel='/usr/lib/xen/boot/hvmloader'
> builder='hvm'
> memory=512
> vcpus=2
> pae=1
> acpi=1
> apic=1
> disk=[ 'tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w' ]
> device_model='/usr/lib/xen/bin/qemu-dm'
> boot='c'
> sdl=0
> vnc=1
> vncunused=1
> vnclisten='0.0.0.0'
> vncpasswd=''
> stdvga=0
> extra=''
> on_reboot='restart'
> on_shutdown='destroy'
> ramdisk=''
> image_name=''
> on_crash='destroy'
> bootloader=''
> root=''
> platform='xen'
> network_mode='tap'
> usb = 1
> usbdevice = 'tablet'
> 
> The xvda disk is a raw disk, created with dd.
> I use the windows.cfg in both kernel.
> 
> 
> 
> 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> 
> > On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:
> > > Hi all,
> > >
> > > I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
> > > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
> > > drivers, and receive an amazing result for reading, but the write numbers
> > > are very disappointing.
> > >
> > > I run the VM in local disk, and receive these numbers:
> > > Seq read: 244 MB/s
> > > 512K read: 239 MB/s
> > > 4K read: 27 MB/s
> > > 4K QD 32: 90 MB/s
> > >
> > > Seq write: 20 MB/s
> > > 512K: 20 MB/s
> > > 4K: 8 MB/s
> > > 4K QD 32: 12MB/s
> > >
> > > In older kernel, 2.6.32.x:
> > > Seq read: 189 MB/s
> > > 512K read: 169 MB/s
> > > 4K read: 11 MB/s
> > > 4K QD 32: 34 MB/s
> > >
> > > Seq write: 177 MB/s
> > > 512K: 166 MB/s
> > > 4K: 12 MB/s
> > > 4K QD 32: 36 MB/s
> > >
> > > Could anyone help me. Why is the new kernel faster in reading, but too
> > slow
> > > writing ?
> >
> > It would help if you gave an idea of what the guest configuration is and
> > what is the backend. It might be that the backend you are using in 3.1 is
> > QEMU qdisk, not xen-blkback.
> >
> 
> 
> 
> -- 
> Roberto Scudeller

Re: why do I get bad disk write performance in the kernel 3.1?

[Ian Campbell]

On Thu, 2011-11-10 at 03:42 +0000, Konrad Rzeszutek Wilk wrote:

> 
> 
> OK, so looks like you are using qdisk. [...]

That depends on whether Roberto is using xm/xend or xl, I think.
Roberto?

I'm not sure what xend does for 'tap2:aio' if blktap isn't available. I
suspect it doesn't fallback to qdisk. Roberto can you report the content
of xenstore under /local/domain/0/backend/ (use xenstore-ls <path>).

> You could also try the 'file:/local-disk' which will setup a loopback device
> and use that. Try that as well.

This is true with xm/xend but with xl you will need to manually setup
the looback and use phy:/dev/loop in order to perform this experiment.
For xl file: turns into qdisk unless blktap is available.

Ian.

> 
> > 
> > My windows.cfg is:
> > name='benchCM-windows-2003-64b-std-test'
> > kernel='/usr/lib/xen/boot/hvmloader'
> > builder='hvm'
> > memory=512
> > vcpus=2
> > pae=1
> > acpi=1
> > apic=1
> > disk=[ 'tap2:aio:/local-disk/benchCM-windows-2003-64b-std/xvda,xvda,w' ]
> > device_model='/usr/lib/xen/bin/qemu-dm'
> > boot='c'
> > sdl=0
> > vnc=1
> > vncunused=1
> > vnclisten='0.0.0.0'
> > vncpasswd=''
> > stdvga=0
> > extra=''
> > on_reboot='restart'
> > on_shutdown='destroy'
> > ramdisk=''
> > image_name=''
> > on_crash='destroy'
> > bootloader=''
> > root=''
> > platform='xen'
> > network_mode='tap'
> > usb = 1
> > usbdevice = 'tablet'
> > 
> > The xvda disk is a raw disk, created with dd.
> > I use the windows.cfg in both kernel.
> > 
> > 
> > 
> > 2011/11/9 Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> > 
> > > On Wed, Nov 09, 2011 at 12:58:27PM -0200, Roberto Scudeller wrote:
> > > > Hi all,
> > > >
> > > > I'm testing the new kernel 3.1 from kernel.org, and xen 4.1.2-rc1-pre.
> > > > I execute a CrystalMark IO Benchmark, in windows 2003 VM with gplpv
> > > > drivers, and receive an amazing result for reading, but the write numbers
> > > > are very disappointing.
> > > >
> > > > I run the VM in local disk, and receive these numbers:
> > > > Seq read: 244 MB/s
> > > > 512K read: 239 MB/s
> > > > 4K read: 27 MB/s
> > > > 4K QD 32: 90 MB/s
> > > >
> > > > Seq write: 20 MB/s
> > > > 512K: 20 MB/s
> > > > 4K: 8 MB/s
> > > > 4K QD 32: 12MB/s
> > > >
> > > > In older kernel, 2.6.32.x:
> > > > Seq read: 189 MB/s
> > > > 512K read: 169 MB/s
> > > > 4K read: 11 MB/s
> > > > 4K QD 32: 34 MB/s
> > > >
> > > > Seq write: 177 MB/s
> > > > 512K: 166 MB/s
> > > > 4K: 12 MB/s
> > > > 4K QD 32: 36 MB/s
> > > >
> > > > Could anyone help me. Why is the new kernel faster in reading, but too
> > > slow
> > > > writing ?
> > >
> > > It would help if you gave an idea of what the guest configuration is and
> > > what is the backend. It might be that the backend you are using in 3.1 is
> > > QEMU qdisk, not xen-blkback.
> > >
> > 
> > 
> > 
> > -- 
> > Roberto Scudeller
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: why do I get bad disk write performance in the kernel 3.1?

[Roberto Scudeller]

[-- Attachment #1.1: Type: text/plain, Size: 3649 bytes --]

Ian,


> That depends on whether Roberto is using xm/xend or xl, I think.
> Roberto?
>

I'm using "xl create windows.cfg"


>
> I'm not sure what xend does for 'tap2:aio' if blktap isn't available. I
> suspect it doesn't fallback to qdisk. Roberto can you report the content
> of xenstore under /local/domain/0/backend/ (use xenstore-ls <path>).
>
> # xenstore-ls /local/domain/0/backend
qdisk = ""
 10 = ""
  51712 = ""
   frontend = "/local/domain/10/device/vbd/51712"
   params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvda"
   frontend-id = "10"
   online = "1"
   removable = "0"
   bootable = "1"
   state = "4"
   dev = "xvda"
   type = "tap"
   mode = "w"
   feature-barrier = "1"
   info = "0"
   sector-size = "512"
   sectors = "67108864"
   hotplug-status = "connected"
  51728 = ""
   frontend = "/local/domain/10/device/vbd/51728"
   params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvdb"
   frontend-id = "10"
   online = "1"
   removable = "0"
   bootable = "1"
   state = "4"
   dev = "xvdb"
   type = "tap"
   mode = "w"
   feature-barrier = "1"
   info = "0"
   sector-size = "512"
   sectors = "614400"
   hotplug-status = "connected"
console = ""
 10 = ""
  0 = ""
   frontend = "/local/domain/10/console"
   frontend-id = "10"
   online = "1"
   state = "1"
   domain = "benchCM-windows-2003-64b-std-test"
   protocol = "vt100"




>  > You could also try the 'file:/local-disk' which will setup a loopback
> device
> > and use that. Try that as well.
>
> This is true with xm/xend but with xl you will need to manually setup
> the looback and use phy:/dev/loop in order to perform this experiment.
> For xl file: turns into qdisk unless blktap is available.
>
> I'm execute with file:/, but in xenstore-list, continue with qdisk. Look:
device = ""
 suspend = ""
  event-channel = "6"
 vbd = ""
  51712 = ""
   backend = "/local/domain/0/backend/qdisk/10/51712"
   backend-id = "0"
   state = "4"
   virtual-device = "51712"
   device-type = "disk"
   ring-ref = "16383"
   event-channel = "7"
  51728 = ""
   backend = "/local/domain/0/backend/qdisk/10/51728"
   backend-id = "0"
   state = "4"
   virtual-device = "51728"
   device-type = "disk"
   ring-ref = "16371"
   event-channel = "8"

Konrad,
I try with "file://" and result was qdisk again. Do you think is LVM test
need?

I testing my windows.cfg in xen 4.1.2-rc1-pre with kernel 2.6.32.43, the
result of xenstore-ls:
# xenstore-ls /local/domain/0/backend
vbd = ""
 2 = ""
  51712 = ""
   frontend = "/local/domain/2/device/vbd/51712"
   tapdisk-params = "aio:/disk-local/benchCM-windows-2003-64b-std/xvda"
   params = "/dev/xen/blktap-2/tapdev2"
   physical-device = "fd:2"
   frontend-id = "2"
   online = "1"
   removable = "0"
   bootable = "1"
   state = "4"
   dev = "xvda"
   type = "phy"
   mode = "w"
   feature-barrier = "1"
   sectors = "67108864"
   info = "0"
   sector-size = "512"
  51728 = ""
   frontend = "/local/domain/2/device/vbd/51728"
   tapdisk-params = "aio:/local-disk/benchCM-windows-2003-64b-std/xvdb"
   params = "/dev/xen/blktap-2/tapdev3"
   physical-device = "fd:3"
   frontend-id = "2"
   online = "1"
   removable = "0"
   bootable = "1"
   state = "4"
   dev = "xvdb"
   type = "phy"
   mode = "w"
   feature-barrier = "1"
   sectors = "614400"
   info = "0"
   sector-size = "512"
console = ""
 2 = ""
  0 = ""
   frontend = "/local/domain/2/console"
   frontend-id = "2"
   online = "1"
   state = "1"
   domain = "benchCM-windows-2003-64b-std-test"
   protocol = "vt100"

Why did kernel 3.1 use the qdisk? I want to use blktap backend
('tap2:aio:/disk-local/xvda,xvda,w').


Thanks for help.

-- 
Roberto Scudeller

[-- Attachment #1.2: Type: text/html, Size: 5825 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: why do I get bad disk write performance in the kernel 3.1?

[Ian Campbell]

On Thu, 2011-11-10 at 14:25 +0000, Roberto Scudeller wrote:
> Ian,
> 
>         
>         That depends on whether Roberto is using xm/xend or xl, I
>         think.
>         Roberto?
> 
> I'm using "xl create windows.cfg"

OK, so you will be getting qdisk (and your logs confirm this).
>  
>         > You could also try the 'file:/local-disk' which will setup a
>         loopback device
>         > and use that. Try that as well.
>         
>         
>         This is true with xm/xend but with xl you will need to
>         manually setup
>         the looback and use phy:/dev/loop in order to perform this
>         experiment.
>         For xl file: turns into qdisk unless blktap is available.
>         
> I'm execute with file:/, but in xenstore-list, continue with qdisk.

Please try using losetup to bind your file to /dev/loopN and specifying
phy:/dev/loopN in your config. That should get you blkback.

[...]
> I try with "file://" and result was qdisk again. Do you think is LVM
> test need?

That would also be interesting.

> Why did kernel 3.1 use the qdisk? I want to use blktap backend
> ('tap2:aio:/disk-local/xvda,xvda,w').

blktap is not available in 3.1. Unfortunately it is not upstreamable so
it is unlikely to become available until someone does a pile of work to
make it a userspace only thing (see the list archives for various
explanations of this).

Since you are using a raw image (rather than e.g. vhd or qcow) there is
no benefit to blktap.

Ian.

> 
> 
> 
> Thanks for help.
> 
> -- 
> Roberto Scudeller
> 
> 

Re: why do I get bad disk write performance in the kernel 3.1?

[Norbert Marx]

Hello,

I have the same problem and filed a bug: 
http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1801

[...]
> > I'm execute with file:/, but in xenstore-list, continue with qdisk.
> 
> Please try using losetup to bind your file to /dev/loopN and specifying
> phy:/dev/loopN in your config. That should get you blkback.
> 
> [...]
> > I try with "file://" and result was qdisk again. Do you think is LVM
> > test need?
> 
> That would also be interesting.

With this hints I modified libxl__try_phy_backend and the result was vbd now. 
But something es does not work with /etc/xen/scripts/block. All details can be 
find at http://bugzilla.xen.org/bugzilla/show_bug.cgi?id=1801.

regards,
Norbert