Monitoring I/O rate for each VMs

Monitoring I/O rate for each VMs

[Thomas Goirand]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I'm one of the 2 authors of dtc-xen. You can see our project page here:

http://www.gplhost.com/software-dtc-xen.html

We do monitoring of network and CPU for each of the domU using RRDTool.
Here are some examples:

http://node6501.gplhost.com/dtc-xen/
http://node0110.gplhost.com/dtc-xen/
http://node3302.gplhost.com/dtc-xen/

We get the CPU information using "xm list"
We get the network usage using /proc/net/dev

It's nice already, but I'd like to add the I/O rate of each of my VMs as
well, to know which of them is taking too much bandwidth or hard drives.
Does one of you know where to get this information?

Thomas Goirand, GPLHost

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFFagvXl4M9yZjvmkkRAvPAAKCewwOdT06KTOVITHgcuG5ApQZ61wCcCzls
qne+DAc6ir1UZvktyvGAgAU=
=dWCo
-----END PGP SIGNATURE-----

Re: Monitoring I/O rate for each VMs

[Keir Fraser]

On 26/11/06 9:49 pm, "Thomas Goirand" <thomas@goirand.fr> wrote:

> It's nice already, but I'd like to add the I/O rate of each of my VMs as
> well, to know which of them is taking too much bandwidth or hard drives.
> Does one of you know where to get this information?

There are per-vbd read and write stats under
/sys/<path-to-vbd>/statistics/{rd_req,wr_req}

Unfortunately these count requests rather than bytes served. This may be
what you want though, as #requests should be proportional to the number of
expensive disc operations (seeking and settling). Long contiguous requests
are not proportionally more expensive than short ones.

 -- Keir

Re: Monitoring I/O rate for each VMs

[Thomas Goirand]

Keir Fraser wrote:
> On 26/11/06 9:49 pm, "Thomas Goirand" <thomas@goirand.fr> wrote:
> 
>> It's nice already, but I'd like to add the I/O rate of each of my VMs as
>> well, to know which of them is taking too much bandwidth or hard drives.
>> Does one of you know where to get this information?
> 
> There are per-vbd read and write stats under
> /sys/<path-to-vbd>/statistics/{rd_req,wr_req}
> 
> Unfortunately these count requests rather than bytes served. This may be
> what you want though, as #requests should be proportional to the number of
> expensive disc operations (seeking and settling). Long contiguous requests
> are not proportionally more expensive than short ones.

Hi!

I didn't find such file where you said. Even a "locate statistics"
didn't help. Note that I'm using lvm partitions, and xen 3.0.2-2 (and
2.0.7 in some older servers), and my LVs are of form /dev/lvm1/xen01, is
<path-to-vdb> for loopback?

Thomas

P.S: No need to write to me, I'm subscribed

Re: Monitoring I/O rate for each VMs

[Keir Fraser]

On 27/11/06 09:25, "Thomas Goirand" <thomas@goirand.fr> wrote:

>> Unfortunately these count requests rather than bytes served. This may be
>> what you want though, as #requests should be proportional to the number of
>> expensive disc operations (seeking and settling). Long contiguous requests
>> are not proportionally more expensive than short ones.
> 
> Hi!
> 
> I didn't find such file where you said. Even a "locate statistics"
> didn't help. Note that I'm using lvm partitions, and xen 3.0.2-2 (and
> 2.0.7 in some older servers), and my LVs are of form /dev/lvm1/xen01, is
> <path-to-vdb> for loopback?

I think the stats were added during 3.0.3 development. There's no way to get
the information you seek with older versions of Xen, although you could try
taking the blkback driver from a 3.0.3 tree and build it against your Linux
dom0 kernel.

 -- Keir

Re: Monitoring I/O rate for each VMs

[Thomas Goirand]

Keir Fraser wrote:
> On 27/11/06 09:25, "Thomas Goirand" <thomas@goirand.fr> wrote:
> 
>>> Unfortunately these count requests rather than bytes served. This may be
>>> what you want though, as #requests should be proportional to the number of
>>> expensive disc operations (seeking and settling). Long contiguous requests
>>> are not proportionally more expensive than short ones.
>> Hi!
>>
>> I didn't find such file where you said. Even a "locate statistics"
>> didn't help. Note that I'm using lvm partitions, and xen 3.0.2-2 (and
>> 2.0.7 in some older servers), and my LVs are of form /dev/lvm1/xen01, is
>> <path-to-vdb> for loopback?
> 
> I think the stats were added during 3.0.3 development. There's no way to get
> the information you seek with older versions of Xen, although you could try
> taking the blkback driver from a 3.0.3 tree and build it against your Linux
> dom0 kernel.

cat /sys/block/dm-21/stat
8        0       96       95       63        0     4432     1608
0      579     1703

What are the meaning of those numbers ???

Thomas

Re: Monitoring I/O rate for each VMs

[Muli Ben-Yehuda]

On Tue, Nov 28, 2006 at 04:04:34PM +0800, Thomas Goirand wrote:
> Keir Fraser wrote:
> > On 27/11/06 09:25, "Thomas Goirand" <thomas@goirand.fr> wrote:
> > 
> >>> Unfortunately these count requests rather than bytes served. This may be
> >>> what you want though, as #requests should be proportional to the number of
> >>> expensive disc operations (seeking and settling). Long contiguous requests
> >>> are not proportionally more expensive than short ones.
> >> Hi!
> >>
> >> I didn't find such file where you said. Even a "locate statistics"
> >> didn't help. Note that I'm using lvm partitions, and xen 3.0.2-2 (and
> >> 2.0.7 in some older servers), and my LVs are of form /dev/lvm1/xen01, is
> >> <path-to-vdb> for loopback?
> > 
> > I think the stats were added during 3.0.3 development. There's no way to get
> > the information you seek with older versions of Xen, although you could try
> > taking the blkback driver from a 3.0.3 tree and build it against your Linux
> > dom0 kernel.
> 
> cat /sys/block/dm-21/stat
> 8        0       96       95       63        0     4432     1608
> 0      579     1703
> 
> What are the meaning of those numbers ???

The best way to know is to check the source code that generates
them...

Cheers,
Muli

Re: Monitoring I/O rate for each VMs

[Nick Craig-Wood]

On Mon, Nov 27, 2006 at 09:41:12AM +0000, Keir Fraser wrote:
> On 27/11/06 09:25, "Thomas Goirand" <thomas@goirand.fr> wrote:
> 
> >> Unfortunately these count requests rather than bytes served. This may be
> >> what you want though, as #requests should be proportional to the number of
> >> expensive disc operations (seeking and settling). Long contiguous requests
> >> are not proportionally more expensive than short ones.
> > 
> > I didn't find such file where you said. Even a "locate statistics"
> > didn't help. Note that I'm using lvm partitions, and xen 3.0.2-2 (and
> > 2.0.7 in some older servers), and my LVs are of form /dev/lvm1/xen01, is
> > <path-to-vdb> for loopback?

Note also, "xm top", followed by 'b' shows the same statistics.

> I think the stats were added during 3.0.3 development. There's no way to get
> the information you seek with older versions of Xen, although you could try
> taking the blkback driver from a 3.0.3 tree and build it against your Linux
> dom0 kernel.

Note that these stats don't work for tap:aio: block devices, eg

# grep disk /etc/xen/sarge1
disk = [ 'tap:aio:/home/sarge1/root_fs,sda1,w', 'tap:aio:/home/sarge1/swap,sda2,w' ]

# xm list sarge1
Name                                      ID Mem(MiB) VCPUs State   Time(s)
sarge1                                     2      128     4 -b----     22.2

# cat /sys/block/dm-2/stat 
       0        0        0        0        0        0        0        0        0        0        0

-- 
Nick Craig-Wood <nick@craig-wood.com> -- http://www.craig-wood.com/nick