Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Dan Magenheimer]

Has anyone tried using a ramdisk in a PV domain?

I'm trying it in an experiment to compare swapping to
a ramdisk against tmem (frontswap) performance
(suggested as a performance test on lkml).

Writing to the ramdisk appears to be VERY VERY slow,
elapsed time in the guest is several times larger than
user+sys, and xentop shows the guest consuming vcpu
seconds at about the user+sys rate.  Note that
this is when tmem is turned off and there is no
vhd swap disk configured.

I'm suspecting that writing to ramdisk must be causing
some interesting/expensive PV pagetable behavior?
Or maybe somehow /dev/ram0 is getting routed through
qemu?  Or ??

Thanks,
Dan

P.S. Yes, I realize a ramdisk has no real value in
a guest... this is just a performance experiment.

RE: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Dan Magenheimer]

It appears that ramswap is bad for HV domains also.
On one run:

real    11m38.017s
user    0m6.696s
sys     0m27.865s

I tried on bare-metal Linux and there is a
similar pattern but not nearly as bad:

real    0m54.978s
user    0m8.577s
sys     0m3.094s

Maybe there is some horrible TLB thrashing going
on even on bare-metal that is made even worse by
virtualization?

> -----Original Message-----
> From: Dan Magenheimer
> Sent: Wednesday, May 05, 2010 12:14 PM
> To: Xen-Devel (xen-devel@lists.xensource.com)
> Subject: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?
> 
> Has anyone tried using a ramdisk in a PV domain?
> 
> I'm trying it in an experiment to compare swapping to
> a ramdisk against tmem (frontswap) performance
> (suggested as a performance test on lkml).
> 
> Writing to the ramdisk appears to be VERY VERY slow,
> elapsed time in the guest is several times larger than
> user+sys, and xentop shows the guest consuming vcpu
> seconds at about the user+sys rate.  Note that
> this is when tmem is turned off and there is no
> vhd swap disk configured.
> 
> I'm suspecting that writing to ramdisk must be causing
> some interesting/expensive PV pagetable behavior?
> Or maybe somehow /dev/ram0 is getting routed through
> qemu?  Or ??
> 
> Thanks,
> Dan
> 
> P.S. Yes, I realize a ramdisk has no real value in
> a guest... this is just a performance experiment.

Re: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Jeremy Fitzhardinge]

On 05/05/2010 11:14 AM, Dan Magenheimer wrote:
> Has anyone tried using a ramdisk in a PV domain?
>
> I'm trying it in an experiment to compare swapping to
> a ramdisk against tmem (frontswap) performance
> (suggested as a performance test on lkml).
>
> Writing to the ramdisk appears to be VERY VERY slow,
> elapsed time in the guest is several times larger than
> user+sys, and xentop shows the guest consuming vcpu
> seconds at about the user+sys rate.  Note that
> this is when tmem is turned off and there is no
> vhd swap disk configured.
>
> I'm suspecting that writing to ramdisk must be causing
> some interesting/expensive PV pagetable behavior?
> Or maybe somehow /dev/ram0 is getting routed through
> qemu?  Or ??
>   

I haven't looked at ramdisk, but I'm pretty sure there's nothing special
about accessing it.  The only thing I can think of is that if you're
using a 32bit highmem system then you may be being hit by lots of kmap
overhead.  But on a 64-bit system, AFAIK, it should just be memory copies.

    J

RE: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Dan Magenheimer]

> Subject: Re: [Xen-devel] Writing to a ramdisk in a PV domain is
> SLLLOOOWWW?!?
> 
> > Writing to the ramdisk appears to be VERY VERY slow,
> > elapsed time in the guest is several times larger than
> > user+sys, and xentop shows the guest consuming vcpu
> > seconds at about the user+sys rate.  Note that
> > this is when tmem is turned off and there is no
> > vhd swap disk configured.
> >
> > I'm suspecting that writing to ramdisk must be causing
> > some interesting/expensive PV pagetable behavior?
> > Or maybe somehow /dev/ram0 is getting routed through
> > qemu?  Or ??
> >
> 
> I haven't looked at ramdisk, but I'm pretty sure there's nothing
> special
> about accessing it.  The only thing I can think of is that if you're
> using a 32bit highmem system then you may be being hit by lots of kmap
> overhead.  But on a 64-bit system, AFAIK, it should just be memory
> copies.

Thanks for the reply Jeremy.

I tried a 64-bit guest (and bare-metal) and saw the same problem.
I guess I'll start collecting some statistics.

Maybe since ramdisk (via the swap code) is still going
through the blockio layer, there is some kind of cache or
TLB overhead that normally would only be necessary for
memory used for DMA?  (I don't know much about I/O so
this is pure speculation.)  Or maybe there is an assumption
that since blockio is asynchronous, a timer is set to
some minimum value and the write-to-ramdisk isn't completed
until the timer fires?  (Hmmm... but this wouldn't explain
why virtual is much worse than bare-metal.)

Dan

RE: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Santos, Jose Renato G]

This is complete wild guess, but could it be that you are doing multi-page I/O with non contiguous machine pages?
I believe the DMA API for pv guests uses swioltlb which does extra copies to a bounce buffer if you request a dma address to a multi-page region (as the pages may be non contiguous).

Renato 

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com 
> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of 
> Dan Magenheimer
> Sent: Thursday, May 06, 2010 2:02 PM
> To: Jeremy Fitzhardinge
> Cc: Xen-Devel (xen-devel@lists.xensource.com)
> Subject: RE: [Xen-devel] Writing to a ramdisk in a PV domain 
> is SLLLOOOWWW?!?
> 
> > Subject: Re: [Xen-devel] Writing to a ramdisk in a PV domain is 
> > SLLLOOOWWW?!?
> > 
> > > Writing to the ramdisk appears to be VERY VERY slow, 
> elapsed time in 
> > > the guest is several times larger than
> > > user+sys, and xentop shows the guest consuming vcpu
> > > seconds at about the user+sys rate.  Note that this is 
> when tmem is 
> > > turned off and there is no vhd swap disk configured.
> > >
> > > I'm suspecting that writing to ramdisk must be causing some 
> > > interesting/expensive PV pagetable behavior?
> > > Or maybe somehow /dev/ram0 is getting routed through qemu?  Or ??
> > >
> > 
> > I haven't looked at ramdisk, but I'm pretty sure there's nothing
> > special
> > about accessing it.  The only thing I can think of is that if you're
> > using a 32bit highmem system then you may be being hit by 
> lots of kmap
> > overhead.  But on a 64-bit system, AFAIK, it should just be memory
> > copies.
> 
> Thanks for the reply Jeremy.
> 
> I tried a 64-bit guest (and bare-metal) and saw the same problem.
> I guess I'll start collecting some statistics.
> 
> Maybe since ramdisk (via the swap code) is still going
> through the blockio layer, there is some kind of cache or
> TLB overhead that normally would only be necessary for
> memory used for DMA?  (I don't know much about I/O so
> this is pure speculation.)  Or maybe there is an assumption
> that since blockio is asynchronous, a timer is set to
> some minimum value and the write-to-ramdisk isn't completed
> until the timer fires?  (Hmmm... but this wouldn't explain
> why virtual is much worse than bare-metal.)
> 
> Dan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

Re: Writing to a ramdisk in a PV domain is SLLLOOOWWW?!?

[Jeremy Fitzhardinge]

On 05/06/2010 03:04 PM, Santos, Jose Renato G wrote:
> This is complete wild guess, but could it be that you are doing multi-page I/O with non contiguous machine pages?
> I believe the DMA API for pv guests uses swioltlb which does extra copies to a bounce buffer if you request a dma address to a multi-page region (as the pages may be non contiguous).
>   

Ramdisk access shouldn't be going anywhere near DMA API or swiotlb.

    J