HVM domain with write caching going on somewhere to disk

HVM domain with write caching going on somewhere to disk

[James Harper]

I just about have my PV block device drivers booting, except just before
the login screen appears I get a message that the registry couldn't be
written out. I'm pretty sure that this is because some data from the
int13 interface provided by the qemu intel ide driver that has been
written, hasn't actually made it to the block device (lvm volume in my
case).

I've noticed before that any changes made in a Windows HVM domain
shortly before an 'xm shutdown' don't ever get written out...

Anyway, is there a way to make sure that the qemu stuff doesn't do any
write caching at all?

Thanks

James

Re: HVM domain with write caching going on somewhere to disk

[Keir Fraser]

On 8/11/07 10:40, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I've noticed before that any changes made in a Windows HVM domain
> shortly before an 'xm shutdown' don't ever get written out...
> 
> Anyway, is there a way to make sure that the qemu stuff doesn't do any
> write caching at all?

I don't think qemu caches writes internally, and also I believe that
outstanding I/O requests are serviced before a domain is shut down. Could
there be a problem with data ending up in the buffer cache in dom0 kernel?

 -- Keir

RE: HVM domain with write caching going on somewhere to disk

[James Harper]

> On 8/11/07 10:40, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > I've noticed before that any changes made in a Windows HVM domain
> > shortly before an 'xm shutdown' don't ever get written out...
> >
> > Anyway, is there a way to make sure that the qemu stuff doesn't do
any
> > write caching at all?
> 
> I don't think qemu caches writes internally, and also I believe that
> outstanding I/O requests are serviced before a domain is shut down

Are they serviced before a domain is destroyed (eg 'xm destroy')? Maybe
that's what I was using when I noticed it...

> Could
> there be a problem with data ending up in the buffer cache in dom0
kernel?

Hmmm... shouldn't the DomU->qemu->device and DomU->blkbackend->device
path's both have visibility to the same buffer cache though?

Maybe the problem isn't what I thought it was. It seems strange that
windows would boot almost completely using my PV drivers and then barf
at the last minute though.

Thanks

James

Re: HVM domain with write caching going on somewhere to disk

[Keir Fraser]

On 8/11/07 10:51, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> I don't think qemu caches writes internally, and also I believe that
>> outstanding I/O requests are serviced before a domain is shut down
> 
> Are they serviced before a domain is destroyed (eg 'xm destroy')? Maybe
> that's what I was using when I noticed it...

When a domain is shutdown, Xen will not acknowledge the shutdown until all
in-flight I/O requests are serviced. That should be sufficient, although it
might depend on what qemu is using for the storage backend. For example, I'm
not sure how aio will interact with SIGKILL of the requesting process
(that's how we terminate qemu-dm on domain destruction).

>> Could
>> there be a problem with data ending up in the buffer cache in dom0
>> kernel?
> 
> Hmmm... shouldn't the DomU->qemu->device and DomU->blkbackend->device
> path's both have visibility to the same buffer cache though?

No, it's trickier than that. Blkback sends I/O requests direct into the
block subsystem, bypassing the buffer cache. You can see there's potential
for confusion therefore!

 -- Keir

RE: HVM domain with write caching going on somewhere to disk

[James Harper]

> > Hmmm... shouldn't the DomU->qemu->device and
DomU->blkbackend->device
> > path's both have visibility to the same buffer cache though?
> 
> No, it's trickier than that. Blkback sends I/O requests direct into
the
> block subsystem, bypassing the buffer cache. You can see there's
potential
> for confusion therefore!

Ah yes. That would probably do it. So I need to make sure that the
buffer cache is flushed (eg async writes in qemu?)... or maybe get qemu
to talk direct to the block subsystem in the same way... any
suggestions? I've already butchered ioemu to get this working so far
(changed the PCI ID's of the IDE interface so Windows won't detect it)
so I'm not afraid of doing more of it :)

Thanks

James

Re: HVM domain with write caching going on somewhere to disk

[Keir Fraser]

On 8/11/07 11:08, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> No, it's trickier than that. Blkback sends I/O requests direct into
> the
>> block subsystem, bypassing the buffer cache. You can see there's
> potential
>> for confusion therefore!
> 
> Ah yes. That would probably do it. So I need to make sure that the
> buffer cache is flushed (eg async writes in qemu?)... or maybe get qemu
> to talk direct to the block subsystem in the same way... any
> suggestions? I've already butchered ioemu to get this working so far
> (changed the PCI ID's of the IDE interface so Windows won't detect it)
> so I'm not afraid of doing more of it :)

Qemu-dm should probably be specifying O_DIRECT when it opens guest storage
volumes. There was discussion about this quite some time ago, but I don't
think any patch was ever floated.

 -- Keir

RE: HVM domain with write caching going on somewhere to disk

[James Harper]

> 
> Qemu-dm should probably be specifying O_DIRECT when it opens guest
storage
> volumes. There was discussion about this quite some time ago, but I
don't
> think any patch was ever floated.

I just changed the open in block.c to include O_DIRECT, but it fails to
boot. The VNC session flashes up very quickly and I can definitely see
the BIOS there but nothing else.

I'll have a look a bit further tomorrow...

Thanks

James

Nowadays, does xen support multicore , such as cell or not?

[tgh]

hi
   Nowadays,does xen support multicore ,such as cell or others?or does 
xen plan to support it in the future?

Thanks in advance

Re: Nowadays, does xen support multicore , such as cell or not?

[Steven Hu]

What do you mean?
The CPU virtualization in xen is unique VCPU, and when use the cpuid,
xen mask the multi-core return value. We can change the function
vmx_do_cpuid to cheat OS that the multi-vcpus is multi-vcore.

2007/11/8, tgh <wwwwww4187@sina.com.cn>:
> hi
>    Nowadays,does xen support multicore ,such as cell or others?or does
> xen plan to support it in the future?
>
> Thanks in advance
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Nowadays, does xen support multicore , such as cell or not?

[weiming]

Can guest OS use more than 1 cpu in Xen now?

--weiming

On Nov 8, 2007 9:05 AM, Steven Hu <learn.hu@gmail.com> wrote:
> What do you mean?
> The CPU virtualization in xen is unique VCPU, and when use the cpuid,
> xen mask the multi-core return value. We can change the function
> vmx_do_cpuid to cheat OS that the multi-vcpus is multi-vcore.
>
> 2007/11/8, tgh <wwwwww4187@sina.com.cn>:
> > hi
> >    Nowadays,does xen support multicore ,such as cell or others?or does
> > xen plan to support it in the future?
> >
> > Thanks in advance
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Nowadays, does xen support multicore , such as cell or not?

[tgh]

hi
I mean, does xen support hardware architecture such as cell ,or not?or 
does xen have plan for this support in the future?

Thanks

Steven Hu 写道:
> What do you mean?
> The CPU virtualization in xen is unique VCPU, and when use the cpuid,
> xen mask the multi-core return value. We can change the function
> vmx_do_cpuid to cheat OS that the multi-vcpus is multi-vcore.
>
> 2007/11/8, tgh <wwwwww4187@sina.com.cn>:
>   
>> hi
>>    Nowadays,does xen support multicore ,such as cell or others?or does
>> xen plan to support it in the future?
>>
>> Thanks in advance
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
>>     
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>
>
>   

RE: HVM domain with write caching going on somewhere todisk

[James Harper]

> 
> I just changed the open in block.c to include O_DIRECT, but it fails
to
> boot. The VNC session flashes up very quickly and I can definitely see
> the BIOS there but nothing else.
> 
> I'll have a look a bit further tomorrow...
> 

The reason it was failing was that O_DIRECT requires the buffer to the
read() and write() function to be page aligned. I have put some bounce
buffer code into raw_read() and raw_write() in block.c, which does this
(very inefficiently of course), and still get the same error, so maybe
this isn't the problem after all...

Any other suggestions?

Thanks

James

RE: HVM domain with write caching going on somewheretodisk

[James Harper]

More testing... there is one write done by windows via the BIOS
interface on boot, and that is to sector 2958215 on my system. By using
dd, I only finally see the results of that write after I have flushed
out the page cache by doing a few big file transfers. That is even after
I got O_DIRECT working...

Something strange is going on I think.

James

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: Friday, 9 November 2007 12:17
> To: Keir Fraser; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] HVM domain with write caching going on
> somewheretodisk
> 
> >
> > I just changed the open in block.c to include O_DIRECT, but it fails
> to
> > boot. The VNC session flashes up very quickly and I can definitely
see
> > the BIOS there but nothing else.
> >
> > I'll have a look a bit further tomorrow...
> >
> 
> The reason it was failing was that O_DIRECT requires the buffer to the
> read() and write() function to be page aligned. I have put some bounce
> buffer code into raw_read() and raw_write() in block.c, which does
this
> (very inefficiently of course), and still get the same error, so maybe
> this isn't the problem after all...
> 
> Any other suggestions?
> 
> Thanks
> 
> James
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: HVM domain with write caching going on somewheretodisk

[James Harper]

> More testing... there is one write done by windows via the BIOS
> interface on boot, and that is to sector 2958215 on my system. By
using
> dd, I only finally see the results of that write after I have flushed
> out the page cache by doing a few big file transfers. That is even
after
> I got O_DIRECT working...
> 
> Something strange is going on I think.

Found the problem - I wasn't implementing the FLUSH operation in the PV
drivers. D'oh.

It's booting now. I'll put a release up tonight.

James

Re: Nowadays, does xen support multicore , such as cell or not?

[Mark Williamson]

Short answer: I don't know if Xen will run on Cell at the moment, and I 
suspect it wouldn't support all the advanced features of Cell yet.  But on 
x86 multicore it should run OK.

Xen has always supported multiprocessor hardware on the host side.  I'm not 
sure whether old versions of Xen would recognise modern multicore processors 
(because I'm not entirely sure how they're present to the system software) 
but I expect it probably would.

Current versions of Xen run happily on multicore x86 systems.  Since Xen 3.0, 
PV guests can be SMP also so that can take advantage of multiple logical 
processors (but they're not aware of threads vs cores vs packages so they're 
not able to make more targetted process scheduling decisions or memory 
allocation decisions based on this information).  Since a later revision in 
the Xen 3 series, you can also run HVM guests in SMP mode.

Cell hardware is a different case for two reasons:
1) it's PPC based.  The PowerPC port of Xen is not as far along as the x86 
port.  I don't know if the PPC port has been adapted for Cell yet, the last I 
heard they were currently concentrating on PPC970 chips.
2) it's not conventional multicore, because the SPUs are a different 
architecture to the PPC core and require separate management.  The hypervisor 
would have to acquire the ability to manage and swap SPU state across 
multiple virtual machines.  This is definitely doable, and I guess the IBM 
guys may look at it at some point.  But it's a different task to supporting 
SMP-style multipcore as is used on x86.

Hope that helps!

Cheers,
Mark



> I mean, does xen support hardware architecture such as cell ,or not?or
> does xen have plan for this support in the future?
>
> Thanks
>
> Steven Hu 写道:
> > What do you mean?
> > The CPU virtualization in xen is unique VCPU, and when use the cpuid,
> > xen mask the multi-core return value. We can change the function
> > vmx_do_cpuid to cheat OS that the multi-vcpus is multi-vcore.
> >
> > 2007/11/8, tgh <wwwwww4187@sina.com.cn>:
> >> hi
> >>    Nowadays,does xen support multicore ,such as cell or others?or does
> >> xen plan to support it in the future?
> >>
> >> Thanks in advance
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel



-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!

Re: HVM domain with write caching going on somewhere to disk

[Steve Ofsthun]

Keir Fraser wrote:
> On 8/11/07 11:08, "James Harper" <james.harper@bendigoit.com.au> wrote:
> 
>>> No, it's trickier than that. Blkback sends I/O requests direct into
>> the
>>> block subsystem, bypassing the buffer cache. You can see there's
>> potential
>>> for confusion therefore!
>> Ah yes. That would probably do it. So I need to make sure that the
>> buffer cache is flushed (eg async writes in qemu?)... or maybe get qemu
>> to talk direct to the block subsystem in the same way... any
>> suggestions? I've already butchered ioemu to get this working so far
>> (changed the PCI ID's of the IDE interface so Windows won't detect it)
>> so I'm not afraid of doing more of it :)
> 
> Qemu-dm should probably be specifying O_DIRECT when it opens guest storage
> volumes. There was discussion about this quite some time ago, but I don't
> think any patch was ever floated.

We had a patch against the non-AIO version of QEMU that used O_DIRECT.
Initially our motivation was strictly to fix any coherence issues with
PV drivers vs. QEMU.  The patch was somewhat ugly due to the buffer
alignment requirements of using O_DIRECT.  Discussions on the list at
the time indicated that AIO was soon to be integrated in QEMU and any
O_DIRECT work should wait since much of the same code paths were involved.

Further work using the O_DIRECT patch turned up performance concerns.
QEMU tended to generate many small I/Os which O_DIRECT turned into
synchronous I/Os.  This resulted in O_DIRECT performance being
measurably slower than buffered I/O for QEMU emulated disk I/O loads.
For us this translated to slow install performance on HVM guests.

Our current patch (against 3.1.2) uses fsync/fadvise to allow limited
use of the buffer cache.  This improves I/O performance in QEMU (over
O_DIRECT) while still maintaining device block coherence between PV
driver/QEMU disk access.

We are in the process of porting this code to the latest xen-unstable.
When that is ready, we will submit it to the list.

Steve

RE: HVM domain with write caching going on somewhere to disk

[James Harper]

Before I figured out that the problem I was having was nothing to do
with the buffer cache (I was returning an error to windows when it asked
me to flush my buffers, which upset it a bit), I also implemented
O_DIRECT by simply allocating another buffer (with the correct
alignment) in the raw_read and raw_write routines and copying the data
into there. Obviously it was horribly inefficient, but when the problem
I was having still occurred it told me that it must lie elsewhere.

I also tried using /dev/raw/rawX, which has similar restrictions to
O_DIRECT.

Another thing I discovered is that Windows 2003 (and probably others)
only makes two writes via the int13 routines, which I'm guessing is to
set a 'I am booting' flag, so that it can detect a failed boot on the
next startup. Once the windows driver is activated, obviously the int13
routines can no longer be used.

> Our current patch (against 3.1.2) uses fsync/fadvise to allow limited
> use of the buffer cache.  This improves I/O performance in QEMU (over
> O_DIRECT) while still maintaining device block coherence between PV
> driver/QEMU disk access.

That sounds like a good compromise.

Thanks

James