Limit I/O for individual disks of VM

Limit I/O for individual disks of VM

[Tõnis Bramanis]

Hello!

Will disk i/o limiting be added in any future versions? I see it in the 
4.0 feature requests, but not in any changelogs.

Best regards,

Tõnis Bramanis
Server administrator / client advisor
Elkdata OÜ                    www.veebimajutus.ee
Info:    abi@veebimajutus.ee +372  683 5188
Directly: tonis@elkdata.ee    +372 56 960 910
-------------------------------------------------------

Re: Limit I/O for individual disks of VM

[Jeremy Fitzhardinge]

On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
> Hello!
>
> Will disk i/o limiting be added in any future versions? I see it in
> the 4.0 feature requests, but not in any changelogs.

I think the hope is that we can use the generic Linux mechanisms for
this rather than having to add anything Xen-specific.

    J

Re: Limit I/O for individual disks of VM

[Tõnis Bramanis]

Hello

and thank You for the answer. Has anyone of You tried to limit i/o on 
the VMs?

I have about 40 containers and when the i/o gets too high, the host will 
crash with this error:
[23959.739698] sd 4:0:0:0: rejecting I/O to offline device
5	[23959.771158] Buffer I/O error on device dm-0, logical block 40259

This is not a support request! I am just referring why i would like to 
limit the disks i/o.

Regards,
Tõnis Bramanis

On 27.09.2011 22:58, Jeremy Fitzhardinge wrote:
> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
>> Hello!
>>
>> Will disk i/o limiting be added in any future versions? I see it in
>> the 4.0 feature requests, but not in any changelogs.
>
> I think the hope is that we can use the generic Linux mechanisms for
> this rather than having to add anything Xen-specific.
>
>      J

Re: Limit I/O for individual disks of VM

[Jeremy Fitzhardinge]

On 09/27/2011 11:25 PM, Tõnis Bramanis wrote:
> Hello
>
> and thank You for the answer. Has anyone of You tried to limit i/o on
> the VMs?
>
> I have about 40 containers and when the i/o gets too high, the host
> will crash with this error:
> [23959.739698] sd 4:0:0:0: rejecting I/O to offline device
> 5    [23959.771158] Buffer I/O error on device dm-0, logical block 40259
>
> This is not a support request! I am just referring why i would like to
> limit the disks i/o.

I think that IO error is a real problem that you should try to solve,
rather than work around it by rate-limiting IO.

    J

>
> Regards,
> Tõnis Bramanis
>
> On 27.09.2011 22:58, Jeremy Fitzhardinge wrote:
>> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
>>> Hello!
>>>
>>> Will disk i/o limiting be added in any future versions? I see it in
>>> the 4.0 feature requests, but not in any changelogs.
>>
>> I think the hope is that we can use the generic Linux mechanisms for
>> this rather than having to add anything Xen-specific.
>>
>>      J
>

Re: Limit I/O for individual disks of VM

[Tõnis Bramanis]

I haven't met anyone yet who had ever seen something like that before.
Guess it's time to change the hardware and see what happens then.

Thanks,
Tõnis

On 28.09.2011 9:35, Jeremy Fitzhardinge wrote:
> On 09/27/2011 11:25 PM, Tõnis Bramanis wrote:
>> Hello
>>
>> and thank You for the answer. Has anyone of You tried to limit i/o on
>> the VMs?
>>
>> I have about 40 containers and when the i/o gets too high, the host
>> will crash with this error:
>> [23959.739698] sd 4:0:0:0: rejecting I/O to offline device
>> 5    [23959.771158] Buffer I/O error on device dm-0, logical block 40259
>>
>> This is not a support request! I am just referring why i would like to
>> limit the disks i/o.
>
> I think that IO error is a real problem that you should try to solve,
> rather than work around it by rate-limiting IO.
>
>      J
>
>>
>> Regards,
>> Tõnis Bramanis
>>
>> On 27.09.2011 22:58, Jeremy Fitzhardinge wrote:
>>> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
>>>> Hello!
>>>>
>>>> Will disk i/o limiting be added in any future versions? I see it in
>>>> the 4.0 feature requests, but not in any changelogs.
>>>
>>> I think the hope is that we can use the generic Linux mechanisms for
>>> this rather than having to add anything Xen-specific.
>>>
>>>       J
>>
>

Re: Limit I/O for individual disks of VM

[Joseph Glanville]

[-- Attachment #1.1: Type: text/plain, Size: 2114 bytes --]

Hi,

To add to the above it looks like you have a device driver or hardware issue
with your harddrive or raid controller.
I/O ratelimiting might hide the symptoms but not correct the problem. After
you have addressed that you can look at implementing rate limiting via the
methods described below:

cgroups blk-io controller:
You can find documentation etc in the linux git repo at
/documentation/cgroups/blkio-controller.txt

dm-ioband:
Wiki is here http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband

Both of these are external to Xen for the reasons outlined by previous
posts.

Kind regards,
Joseph.

2011/9/28 Jeremy Fitzhardinge <jeremy@goop.org>

> On 09/27/2011 11:25 PM, Tõnis Bramanis wrote:
> > Hello
> >
> > and thank You for the answer. Has anyone of You tried to limit i/o on
> > the VMs?
> >
> > I have about 40 containers and when the i/o gets too high, the host
> > will crash with this error:
> > [23959.739698] sd 4:0:0:0: rejecting I/O to offline device
> > 5    [23959.771158] Buffer I/O error on device dm-0, logical block 40259
> >
> > This is not a support request! I am just referring why i would like to
> > limit the disks i/o.
>
> I think that IO error is a real problem that you should try to solve,
> rather than work around it by rate-limiting IO.
>
>    J
>
> >
> > Regards,
> > Tõnis Bramanis
> >
> > On 27.09.2011 22:58, Jeremy Fitzhardinge wrote:
> >> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
> >>> Hello!
> >>>
> >>> Will disk i/o limiting be added in any future versions? I see it in
> >>> the 4.0 feature requests, but not in any changelogs.
> >>
> >> I think the hope is that we can use the generic Linux mechanisms for
> >> this rather than having to add anything Xen-specific.
> >>
> >>      J
> >
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>



-- 
*
Founder | Director | VP Research
Orion Virtualisation Solutions* | www.orionvm.com.au | Phone: 1300 56 99 52
| Mobile: 0428 754 846

[-- Attachment #1.2: Type: text/html, Size: 3275 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Limit I/O for individual disks of VM

[Tõnis Bramanis]

If i run the exact same kernel without Xen and test the i/o, it won't crash.

I just tried to look up an old post to refer it to You and seems it has 
been updated just yesterday with a fix:
http://xen.1045712.n5.nabble.com/Xen-4-1-3ware-9690SA-rejecting-I-O-to-offline-device-td3208156.html

Still, i will try to change the raid controller and see what happens then.

Regards,
Tõnis Bramanis

On 28.09.2011 10:38, Joseph Glanville wrote:
> Hi,
>
> To add to the above it looks like you have a device driver or hardware
> issue with your harddrive or raid controller.
> I/O ratelimiting might hide the symptoms but not correct the problem.
> After you have addressed that you can look at implementing rate limiting
> via the methods described below:
>
> cgroups blk-io controller:
> You can find documentation etc in the linux git repo at
> /documentation/cgroups/blkio-controller.txt
>
> dm-ioband:
> Wiki is here http://sourceforge.net/apps/trac/ioband/wiki/dm-ioband
>
> Both of these are external to Xen for the reasons outlined by previous
> posts.
>
> Kind regards,
> Joseph.
>
> 2011/9/28 Jeremy Fitzhardinge <jeremy@goop.org <mailto:jeremy@goop.org>>
>
>     On 09/27/2011 11:25 PM, Tõnis Bramanis wrote:
>      > Hello
>      >
>      > and thank You for the answer. Has anyone of You tried to limit i/o on
>      > the VMs?
>      >
>      > I have about 40 containers and when the i/o gets too high, the host
>      > will crash with this error:
>      > [23959.739698] sd 4:0:0:0: rejecting I/O to offline device
>      > 5    [23959.771158] Buffer I/O error on device dm-0, logical
>     block 40259
>      >
>      > This is not a support request! I am just referring why i would
>     like to
>      > limit the disks i/o.
>
>     I think that IO error is a real problem that you should try to solve,
>     rather than work around it by rate-limiting IO.
>
>         J
>
>      >
>      > Regards,
>      > Tõnis Bramanis
>      >
>      > On 27.09.2011 22 <tel:27.09.2011%2022>:58, Jeremy Fitzhardinge wrote:
>      >> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
>      >>> Hello!
>      >>>
>      >>> Will disk i/o limiting be added in any future versions? I see it in
>      >>> the 4.0 feature requests, but not in any changelogs.
>      >>
>      >> I think the hope is that we can use the generic Linux mechanisms for
>      >> this rather than having to add anything Xen-specific.
>      >>
>      >>      J
>      >
>
>
>     _______________________________________________
>     Xen-devel mailing list
>     Xen-devel@lists.xensource.com <mailto:Xen-devel@lists.xensource.com>
>     http://lists.xensource.com/xen-devel
>
>
>
>
> --
> */
> Founder | Director | VP Research
> Orion Virtualisation Solutions/* | www.orionvm.com.au
> <http://www.orionvm.com.au/> | Phone: 1300 56 99 52 | Mobile: 0428 754 846

Re: Limit I/O for individual disks of VM

[Andrew Cooper]

On 28/09/11 07:35, Jeremy Fitzhardinge wrote:
> On 09/27/2011 11:25 PM, Tõnis Bramanis wrote:
>> Hello
>>
>> and thank You for the answer. Has anyone of You tried to limit i/o on
>> the VMs?
>>
>> I have about 40 containers and when the i/o gets too high, the host
>> will crash with this error:
>> [23959.739698] sd 4:0:0:0: rejecting I/O to offline device
>> 5    [23959.771158] Buffer I/O error on device dm-0, logical block 40259
>>
>> This is not a support request! I am just referring why i would like to
>> limit the disks i/o.
> I think that IO error is a real problem that you should try to solve,
> rather than work around it by rate-limiting IO.
>
>     J

This is looking suspiciously like the line level interrupt migration bug.

Can you confirm the version of Xen, the raid controller you are using,
and paste /proc/interrupts from dom0 please.

~Andrew


>> Regards,
>> Tõnis Bramanis
>>
>> On 27.09.2011 22:58, Jeremy Fitzhardinge wrote:
>>> On 09/27/2011 01:33 AM, Tõnis Bramanis wrote:
>>>> Hello!
>>>>
>>>> Will disk i/o limiting be added in any future versions? I see it in
>>>> the 4.0 feature requests, but not in any changelogs.
>>> I think the hope is that we can use the generic Linux mechanisms for
>>> this rather than having to add anything Xen-specific.
>>>
>>>      J
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com