Fwd: xentop reporting zero written sectors

Fwd: xentop reporting zero written sectors

[Andy Burns]

No response in xen-users, asking developers ....

---------- Forwarded message ----------

Just moving a chunk of files from one filesysstem on xvba to another
on xvdb, and was monitoring with xentop as it was taking longer than
expected.

The VBD_RD  and VBD_WR counters were both clocking-up as expected, as
was the VBD_RSECT counter, but the VBD_WSECT counter was stuck on
zero, I toggled on the individual VBD device counters and these showed
the same (with the RD and WR counters correctly split between my
source and destination disks)

Is this a long standing "feature" that I've never noticed before or a
problem that should be reported?

Using the following packages from Fedora 16 beta

xen.x86_64                                   4.1.1-3.fc16
en-hypervisor.x86_64                 4.1.1-3.fc16
xen-libs.x86_64                           4.1.1-3.fc16
xen-licenses.x86_64                  4.1.1-3.fc16
xen-runtime.x86_64                    4.1.1-3.fc16
kernel.x86_64                              3.1.0-0.rc8.git0.0.fc16

Re: Fwd: xentop reporting zero written sectors

[Ian Campbell]

On Mon, 2011-10-10 at 16:16 +0100, Andy Burns wrote:
> No response in xen-users, asking developers ....
> 
> ---------- Forwarded message ----------
> 
> Just moving a chunk of files from one filesysstem on xvba to another
> on xvdb, and was monitoring with xentop as it was taking longer than
> expected.
> 
> The VBD_RD  and VBD_WR counters were both clocking-up as expected, as
> was the VBD_RSECT counter, but the VBD_WSECT counter was stuck on
> zero, I toggled on the individual VBD device counters and these showed
> the same (with the RD and WR counters correctly split between my
> source and destination disks)
> 
> Is this a long standing "feature" that I've never noticed before or a
> problem that should be reported?

How would it be a feature?

Scrobbling around under tools/xenstat these statistics appear to be read
from /sys/bus/xen-backend/devices/<vbd>/wr_sect (likewise rd_sect,
wr_req etc etc).

Can you see all of those and do the numbers appear to be doing the right
thing? This will indicate whether this is an issue with the kernel
producing the numbers or the tools consuming them.

Ian.

> 
> Using the following packages from Fedora 16 beta
> 
> xen.x86_64                                   4.1.1-3.fc16
> en-hypervisor.x86_64                 4.1.1-3.fc16
> xen-libs.x86_64                           4.1.1-3.fc16
> xen-licenses.x86_64                  4.1.1-3.fc16
> xen-runtime.x86_64                    4.1.1-3.fc16
> kernel.x86_64                              3.1.0-0.rc8.git0.0.fc16
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Fwd: xentop reporting zero written sectors

[Konrad Rzeszutek Wilk]

On Mon, Oct 10, 2011 at 04:16:04PM +0100, Andy Burns wrote:
> No response in xen-users, asking developers ....
> 
> ---------- Forwarded message ----------
> 
> Just moving a chunk of files from one filesysstem on xvba to another
> on xvdb, and was monitoring with xentop as it was taking longer than
> expected.
> 
> The VBD_RD  and VBD_WR counters were both clocking-up as expected, as
> was the VBD_RSECT counter, but the VBD_WSECT counter was stuck on
> zero, I toggled on the individual VBD device counters and these showed
> the same (with the RD and WR counters correctly split between my
> source and destination disks)

Huh. Imagine that!
> 
> Is this a long standing "feature" that I've never noticed before or a
> problem that should be reported?

Must be a feature. We would never write code with bugs. Never :-)

I think this new "feature" below will fix it for you:


diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
index 0bd7143..3e2ca68 100644
--- a/drivers/block/xen-blkback/blkback.c
+++ b/drivers/block/xen-blkback/blkback.c
@@ -778,7 +778,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
 
 	if (operation == READ)
 		blkif->st_rd_sect += preq.nr_sects;
-	else if (operation == WRITE || operation == WRITE_FLUSH)
+	else if (operation == WRITE_ODIRECT || operation == WRITE_FLUSH)
 		blkif->st_wr_sect += preq.nr_sects;
 
 	return 0;


Thanks for reporting it!

Re: Fwd: xentop reporting zero written sectors

[Andy Burns]

On 10 October 2011 16:29, Ian Campbell <Ian.Campbell@citrix.com> wrote:

> How would it be a feature?

Sorry for the bug <-> feature sarcasm

> Scrobbling around under tools/xenstat these statistics appear to be read
> from /sys/bus/xen-backend/devices/<vbd>/wr_sect (likewise rd_sect,
> wr_req etc etc).
> Can you see all of those and do the numbers appear to be doing the right
> thing?

Yes, just to confirm what Konrad has spotted ...

# cat /sys/bus/xen-backend/devices/*/statistics/wr_sect
0
0
0

# cat /sys/bus/xen-backend/devices/*/statistics/rd_sect
90313
125876
2546

I'll try the patch later, when I've finished pulling my hair out with
a pci-passthrough issue!

Re: Fwd: xentop reporting zero written sectors

[Konrad Rzeszutek Wilk]

On Mon, Oct 10, 2011 at 07:02:08PM +0100, Andy Burns wrote:
> On 10 October 2011 16:29, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> 
> > How would it be a feature?
> 
> Sorry for the bug <-> feature sarcasm
> 
> > Scrobbling around under tools/xenstat these statistics appear to be read
> > from /sys/bus/xen-backend/devices/<vbd>/wr_sect (likewise rd_sect,
> > wr_req etc etc).
> > Can you see all of those and do the numbers appear to be doing the right
> > thing?
> 
> Yes, just to confirm what Konrad has spotted ...
> 
> # cat /sys/bus/xen-backend/devices/*/statistics/wr_sect
> 0
> 0
> 0
> 
> # cat /sys/bus/xen-backend/devices/*/statistics/rd_sect
> 90313
> 125876
> 2546
> 
> I'll try the patch later, when I've finished pulling my hair out with
> a pci-passthrough issue!

HVM or PV?

Re: Fwd: xentop reporting zero written sectors

[Ian Campbell]

On Mon, 2011-10-10 at 17:31 +0100, Konrad Rzeszutek Wilk wrote:
> On Mon, Oct 10, 2011 at 04:16:04PM +0100, Andy Burns wrote:
> > No response in xen-users, asking developers ....
> > 
> > ---------- Forwarded message ----------
> > 
> > Just moving a chunk of files from one filesysstem on xvba to another
> > on xvdb, and was monitoring with xentop as it was taking longer than
> > expected.
> > 
> > The VBD_RD  and VBD_WR counters were both clocking-up as expected, as
> > was the VBD_RSECT counter, but the VBD_WSECT counter was stuck on
> > zero, I toggled on the individual VBD device counters and these showed
> > the same (with the RD and WR counters correctly split between my
> > source and destination disks)
> 
> Huh. Imagine that!
> > 
> > Is this a long standing "feature" that I've never noticed before or a
> > problem that should be reported?
> 
> Must be a feature. We would never write code with bugs. Never :-)
> 
> I think this new "feature" below will fix it for you:
> 
> 
> diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
> index 0bd7143..3e2ca68 100644
> --- a/drivers/block/xen-blkback/blkback.c
> +++ b/drivers/block/xen-blkback/blkback.c
> @@ -778,7 +778,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
>  
>  	if (operation == READ)
>  		blkif->st_rd_sect += preq.nr_sects;
> -	else if (operation == WRITE || operation == WRITE_FLUSH)
> +	else if (operation == WRITE_ODIRECT || operation == WRITE_FLUSH)

Would it be sane to key this off req->operation and BLKIF_OP_* instead
of first encoding them as block layer operations and then decoding back
again? The use of operation here just seems to be adding a layer of
indirection which we don't want or need.

Or maybe (operation & WRITE) instead of == ?

Ian.

>  		blkif->st_wr_sect += preq.nr_sects;
>  
>  	return 0;
> 
> 
> Thanks for reporting it!
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Fwd: xentop reporting zero written sectors

[Konrad Rzeszutek Wilk]

On Tue, Oct 11, 2011 at 08:49:44AM +0100, Ian Campbell wrote:
> On Mon, 2011-10-10 at 17:31 +0100, Konrad Rzeszutek Wilk wrote:
> > On Mon, Oct 10, 2011 at 04:16:04PM +0100, Andy Burns wrote:
> > > No response in xen-users, asking developers ....
> > > 
> > > ---------- Forwarded message ----------
> > > 
> > > Just moving a chunk of files from one filesysstem on xvba to another
> > > on xvdb, and was monitoring with xentop as it was taking longer than
> > > expected.
> > > 
> > > The VBD_RD  and VBD_WR counters were both clocking-up as expected, as
> > > was the VBD_RSECT counter, but the VBD_WSECT counter was stuck on
> > > zero, I toggled on the individual VBD device counters and these showed
> > > the same (with the RD and WR counters correctly split between my
> > > source and destination disks)
> > 
> > Huh. Imagine that!
> > > 
> > > Is this a long standing "feature" that I've never noticed before or a
> > > problem that should be reported?
> > 
> > Must be a feature. We would never write code with bugs. Never :-)
> > 
> > I think this new "feature" below will fix it for you:
> > 
> > 
> > diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
> > index 0bd7143..3e2ca68 100644
> > --- a/drivers/block/xen-blkback/blkback.c
> > +++ b/drivers/block/xen-blkback/blkback.c
> > @@ -778,7 +778,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
> >  
> >  	if (operation == READ)
> >  		blkif->st_rd_sect += preq.nr_sects;
> > -	else if (operation == WRITE || operation == WRITE_FLUSH)
> > +	else if (operation == WRITE_ODIRECT || operation == WRITE_FLUSH)
> 
> Would it be sane to key this off req->operation and BLKIF_OP_* instead
> of first encoding them as block layer operations and then decoding back
> again? The use of operation here just seems to be adding a layer of
> indirection which we don't want or need.
> 
> Or maybe (operation & WRITE) instead of == ?

I am not really sure why we do all of those 'operation' thingies.

I think I will do a lookup function that will do the proper
lookup give it to the bio layer once it is required instead of doing
this ... extra work.

But lets do that in the next set of patches and just treat this
one as a bug-fix.

Re: Fwd: xentop reporting zero written sectors

[Andy Burns]

On 10 October 2011 17:31, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

> I think this new "feature" below will fix it for you:
>
> diff --git a/drivers/block/xen-blkback/blkback.c b/drivers/block/xen-blkback/blkback.c
> index 0bd7143..3e2ca68 100644
> --- a/drivers/block/xen-blkback/blkback.c
> +++ b/drivers/block/xen-blkback/blkback.c
> @@ -778,7 +778,7 @@ static int dispatch_rw_block_io(struct xen_blkif *blkif,
>
>        if (operation == READ)
>                blkif->st_rd_sect += preq.nr_sects;
> -       else if (operation == WRITE || operation == WRITE_FLUSH)
> +       else if (operation == WRITE_ODIRECT || operation == WRITE_FLUSH)
>                blkif->st_wr_sect += preq.nr_sects;
>
>        return 0;

What kernel version is that patch based on?  It won't apply on top of
3.1.0-rc9 for me, but looks like the starting line number is 685
rather than 778 in Fedora's source ... but even by tweaking it I can't
get rpmbuild to apply the patch :-(

Re: Fwd: xentop reporting zero written sectors

[Andy Burns]

On 13 October 2011 22:10, Andy Burns <xen.lists@burns.me.uk> wrote:

> On 10 October 2011 17:31, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
>
>> I think this new "feature" below will fix it for you:
> What kernel version is that patch based on?

I did eventually build a kernel with a lightly tweaked version of that
patch, to allow for Fedora's patched vanilla 3.0 plus rc9 patches

my VBD_WSECT counters are now merricly clocking up, so feel free to add

Acked-by: Andy Burns <andy@burns.net>

--- a/drivers/block/xen-blkback/blkback.c
--- b/drivers/block/xen-blkback/blkback.c
@@ -685,7 +685,7 @@

        if (operation == READ)
                blkif->st_rd_sect += preq.nr_sects;
-       else if (operation == WRITE || operation == WRITE_FLUSH)
+       else if (operation == WRITE_ODIRECT || operation == WRITE_FLUSH)
                blkif->st_wr_sect += preq.nr_sects;

        return 0;