[Qemu-devel] high CPU load / async IO?

[Qemu-devel] high CPU load / async IO?

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]

Hi,

sorry for bothering, but the last time i heard something about asyn IO
was on Feb 2nd. Fabrice said:

[quote]
I have at least 5 items on my TODO list :
1) New kqemu which virtualizes both user and kernel code (currently in
alpha stage but not released yet).
2) DMA block I/O (merged)
3) async block I/O (not merged yet)
4) Network adapter using DMA (not merged yet)
5) VGA acceleration (either improve Cirrus VGA or switch back to Bochs
VGA for Win 2000/XP or Linux)
[/quote]

so 1) and 2) are in HEAD. I saw something concerning 5) too, i think.
I'm not sure about 4) - but - well,

I'm very curious about 3)  :-)

It's not in HEAD yet, isn't it?

Why i'm curious? Well, i'm curious about the improvement it causes. You
people once told me, that the boost will not be that significant. On the
other hand, i see my host CPU usage going towards 100% just because the
guest is doing some IO or ... or is it because of somethine else perhaps?

To be concrete: have you guys ever run windows-update inside qemu? Well,
my win2k guest consumes all CPU on the host for some reason. What might
be the reason?
(qemu is started with -kernel-kqemu -m 256 -soundhw es1370)

Also windows-update's green "progress bar" inside the guest is stopping
for let's say 3 or 5 seconds and not moving continuous.

Is anybody experiencing the same or knows the reason?


Thanks,
  Sven


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[Qemu-devel] Re: high CPU load / async IO?

[Anthony Liguori]

On Mon, 24 Jul 2006 00:47:21 +0200, Sven Köhler wrote:

> 3) async block I/O (not merged yet)

> It's not in HEAD yet, isn't it?

The pthread-based async patch is a band-aid.  No doubt it helps your
particular case, but it's not the right approach long term.

IDE only supports one outstanding request, so having a thread that runs
the synchronous block routines appears reasonable.  However, SATA and SCSI
both support multiple outstanding requests.  The extension to the existing
patch would be simple--increase the number of threads.

A number of Xen hackers (primarily Andy Warfield and Dan Smith) have been
doing a lot of work analyzing userspace block device performance.  As
QEMU's CPU virtualization gets faster (ala kqemu or VT/SVM), it will start
facing the same bottlenecks that we do today in Xen.

To achieve near-native performance, you basically have to be able to
saturate the host's IO scheduler queue.  Using O_DIRECT, you can do
zero-copy meaning that your ability to queue requests is the only limiting
factor.

What's been discovered is that a thread based approach requires a ton of
threads to achieve saturation.  Just imagine the contention of having a
very large number of threads trying to get at a single BDRVState.

The real solution is to modify the block API to be asynchronous and then
provide support for interacting with the host IO scheduler queue via
something like linux-aio (or the win32 equiv).

So the current thread-based async dma patch is really just the wrong long
term solution.  A more long term solution is likely in the works.  It
requires quite a bit of code modification though.

Regards,

Anthony Liguori 

> Why i'm curious? Well, i'm curious about
the improvement it causes. You
> people once told me, that the boost will not be that significant. On the
> other hand, i see my host CPU usage going towards 100% just because the
> guest is doing some IO or ... or is it because of somethine else
> perhaps?
> 
> To be concrete: have you guys ever run windows-update inside qemu? Well,
> my win2k guest consumes all CPU on the host for some reason. What might
> be the reason?
> (qemu is started with -kernel-kqemu -m 256 -soundhw es1370)
> 
> Also windows-update's green "progress bar" inside the guest is stopping
> for let's say 3 or 5 seconds and not moving continuous.
> 
> Is anybody experiencing the same or knows the reason?
> 
> 
> Thanks,
>   Sven
> _______________________________________________ Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

[Qemu-devel] Re: high CPU load / async IO?

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 2051 bytes --]

>> 3) async block I/O (not merged yet)
>> It's not in HEAD yet, isn't it?
> 
> The pthread-based async patch is a band-aid.  No doubt it helps your
> particular case, but it's not the right approach long term.
> 
> IDE only supports one outstanding request, so having a thread that runs
> the synchronous block routines appears reasonable.  However, SATA and SCSI
> both support multiple outstanding requests.  The extension to the existing
> patch would be simple--increase the number of threads.

???

Wasn't there another variant using the async-I/O support of the Host OS
and thereby supporting a larger number of outstanding requests?

> A number of Xen hackers (primarily Andy Warfield and Dan Smith) have been
> doing a lot of work analyzing userspace block device performance.  As
> QEMU's CPU virtualization gets faster (ala kqemu or VT/SVM), it will start
> facing the same bottlenecks that we do today in Xen.
> 
> To achieve near-native performance, you basically have to be able to
> saturate the host's IO scheduler queue.  Using O_DIRECT, you can do
> zero-copy meaning that your ability to queue requests is the only limiting
> factor.
> 
> What's been discovered is that a thread based approach requires a ton of
> threads to achieve saturation.  Just imagine the contention of having a
> very large number of threads trying to get at a single BDRVState.
> 
> The real solution is to modify the block API to be asynchronous and then
> provide support for interacting with the host IO scheduler queue via
> something like linux-aio (or the win32 equiv).

The approch that i mentioned above (using the host's async I/O) is what
you mean with using linux-aio, right?

> So the current thread-based async dma patch is really just the wrong long
> term solution.  A more long term solution is likely in the works.  It
> requires quite a bit of code modification though.

I see. So in other words:

don't ask for simple async I/O now. The more complex and flexible
sollution will follow soon.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

[Qemu-devel] Re: high CPU load / async IO?

[Anthony Liguori]

On Tue, 25 Jul 2006 00:51:23 +0200, Sven Köhler wrote:

>> IDE only supports one outstanding request, so having a thread that runs
>> the synchronous block routines appears reasonable.  However, SATA and SCSI
>> both support multiple outstanding requests.  The extension to the existing
>> patch would be simple--increase the number of threads.
> 
> ???
> 
> Wasn't there another variant using the async-I/O support of the Host OS
> and thereby supporting a larger number of outstanding requests?

Not that I know of.  Do you have a pointer?

> The approch that i mentioned above (using the host's async I/O) is what
> you mean with using linux-aio, right?

It depends on what you mean by the host's async I/O implementation.

>> So the current thread-based async dma patch is really just the wrong long
>> term solution.  A more long term solution is likely in the works.  It
>> requires quite a bit of code modification though.
> 
> I see. So in other words:
> 
> don't ask for simple async I/O now. The more complex and flexible
> sollution will follow soon.

Yes, hopefully really soon.

Regards,

Anthony Liguori

> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

[Qemu-devel] Re: high CPU load / async IO?

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 1023 bytes --]

>>> IDE only supports one outstanding request, so having a thread that runs
>>> the synchronous block routines appears reasonable.  However, SATA and SCSI
>>> both support multiple outstanding requests.  The extension to the existing
>>> patch would be simple--increase the number of threads.
>> ???
>>
>> Wasn't there another variant using the async-I/O support of the Host OS
>> and thereby supporting a larger number of outstanding requests?
> 
> Not that I know of.  Do you have a pointer?

Hmm, perhaps i only heard people talking about it ...
But if i find anything, i post the link.

>>> So the current thread-based async dma patch is really just the wrong long
>>> term solution.  A more long term solution is likely in the works.  It
>>> requires quite a bit of code modification though.
>>
>> I see. So in other words:
>>
>> don't ask for simple async I/O now. The more complex and flexible
>> sollution will follow soon.
> 
> Yes, hopefully really soon.

So i will wait patiently :-)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [Qemu-devel] Re: high CPU load / async IO?

[Jens Axboe]

On Tue, Jul 25 2006, Sven Köhler wrote:
> >>> So the current thread-based async dma patch is really just the wrong long
> >>> term solution.  A more long term solution is likely in the works.  It
> >>> requires quite a bit of code modification though.
> >>
> >> I see. So in other words:
> >>
> >> don't ask for simple async I/O now. The more complex and flexible
> >> sollution will follow soon.
> > 
> > Yes, hopefully really soon.
> 
> So i will wait patiently :-)

Is anyone actively working on this, or is it just speculation? I'd
greatly prefer (and might do, if no one is working on it and Fabrice
would take it) do a libaio version, since that'll for sure perform the
best on Linux. But a posixaio version might be saner, as that should
work on other operating systems as well.

Fabrice, can you let people know what you would prefer?

-- 
Jens Axboe

Re: [Qemu-devel] Re: high CPU load / async IO?

[Fabrice Bellard]

Jens Axboe wrote:
> On Tue, Jul 25 2006, Sven Köhler wrote:
> 
>>>>>So the current thread-based async dma patch is really just the wrong long
>>>>>term solution.  A more long term solution is likely in the works.  It
>>>>>requires quite a bit of code modification though.
>>>>
>>>>I see. So in other words:
>>>>
>>>>don't ask for simple async I/O now. The more complex and flexible
>>>>sollution will follow soon.
>>>
>>>Yes, hopefully really soon.
>>
>>So i will wait patiently :-)
> 
> 
> Is anyone actively working on this, or is it just speculation? I'd
> greatly prefer (and might do, if no one is working on it and Fabrice
> would take it) do a libaio version, since that'll for sure perform the
> best on Linux. But a posixaio version might be saner, as that should
> work on other operating systems as well.
> 
> Fabrice, can you let people know what you would prefer?

I am working on an implementation and the first version will use the 
posix aio and possibly the Windows ReadFile/WriteFile overlapped I/Os. 
Anthony Liguori got a pre version of the code, but it is not commitable yet.

Fabrice.

Re: [Qemu-devel] Re: high CPU load / async IO?

[Jens Axboe]

On Tue, Jul 25 2006, Fabrice Bellard wrote:
> Jens Axboe wrote:
> >On Tue, Jul 25 2006, Sven Köhler wrote:
> >
> >>>>>So the current thread-based async dma patch is really just the
> >>>>>wrong long term solution.  A more long term solution is likely in
> >>>>>the works.  It requires quite a bit of code modification though.
> >>>>
> >>>>I see. So in other words:
> >>>>
> >>>>don't ask for simple async I/O now. The more complex and flexible
> >>>>sollution will follow soon.
> >>>
> >>>Yes, hopefully really soon.
> >>
> >>So i will wait patiently :-)
> >
> >
> >Is anyone actively working on this, or is it just speculation? I'd
> >greatly prefer (and might do, if no one is working on it and Fabrice
> >would take it) do a libaio version, since that'll for sure perform
> >the best on Linux. But a posixaio version might be saner, as that
> >should work on other operating systems as well.
> >
> >Fabrice, can you let people know what you would prefer?
> 
> I am working on an implementation and the first version will use the
> posix aio and possibly the Windows ReadFile/WriteFile overlapped I/Os.
> Anthony Liguori got a pre version of the code, but it is not
> commitable yet.

Sounds good, so at least it's on its way :-)
It's on of those big items left on the TODO, so will be good to see go
in. Then one should implement an ahci host controller for queued command
support next...

-- 
Jens Axboe

Re: [Qemu-devel] Re: high CPU load / async IO?

[Paul Brook]

> Sounds good, so at least it's on its way :-)
> It's on of those big items left on the TODO, so will be good to see go
> in. Then one should implement an ahci host controller for queued command
> support next...

Or use the scsi emulation :-)

Paul

Re: [Qemu-devel] Re: high CPU load / async IO?

[Jens Axboe]

On Wed, Jul 26 2006, Paul Brook wrote:
> > Sounds good, so at least it's on its way :-)
> > It's on of those big items left on the TODO, so will be good to see go
> > in. Then one should implement an ahci host controller for queued command
> > support next...
> 
> Or use the scsi emulation :-)

Ah, did not know that queueing was fully implemented there yet!

-- 
Jens Axboe

Re: [Qemu-devel] Re: high CPU load / async IO?

[Paul Brook]

On Wednesday 26 July 2006 13:23, Jens Axboe wrote:
> On Wed, Jul 26 2006, Paul Brook wrote:
> > > Sounds good, so at least it's on its way :-)
> > > It's on of those big items left on the TODO, so will be good to see go
> > > in. Then one should implement an ahci host controller for queued
> > > command support next...
> >
> > Or use the scsi emulation :-)
>
> Ah, did not know that queueing was fully implemented there yet!

It isn't, but it's nearer than the SATA emulation!

Paul

Re: [Qemu-devel] Re: high CPU load / async IO?

[Jens Axboe]

On Wed, Jul 26 2006, Paul Brook wrote:
> On Wednesday 26 July 2006 13:23, Jens Axboe wrote:
> > On Wed, Jul 26 2006, Paul Brook wrote:
> > > > Sounds good, so at least it's on its way :-)
> > > > It's on of those big items left on the TODO, so will be good to see go
> > > > in. Then one should implement an ahci host controller for queued
> > > > command support next...
> > >
> > > Or use the scsi emulation :-)
> >
> > Ah, did not know that queueing was fully implemented there yet!
> 
> It isn't, but it's nearer than the SATA emulation!

ahci wouldn't be too much work, but definitely more so than finishing
the scsi bits!

-- 
Jens Axboe

[Qemu-devel] Re: high CPU load / async IO?

[Sven Köhler]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

>>>>> Sounds good, so at least it's on its way :-)
>>>>> It's on of those big items left on the TODO, so will be good to see go
>>>>> in. Then one should implement an ahci host controller for queued
>>>>> command support next...
>>>> Or use the scsi emulation :-)
>>> Ah, did not know that queueing was fully implemented there yet!
>> It isn't, but it's nearer than the SATA emulation!
> 
> ahci wouldn't be too much work, but definitely more so than finishing
> the scsi bits!

That sounds great! I feel, like my dreams come true.


BTW: Fabrice said, he will use the POSIX AIO (i guess, he means
http://www.bullopensource.org/posix/ in case of Linux, right?)

Which other OS do also support the POSIX AIO API?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 251 bytes --]

Re: [Qemu-devel] Re: high CPU load / async IO?

[Jens Axboe]

On Wed, Jul 26 2006, Sven Köhler wrote:
> >>>>> Sounds good, so at least it's on its way :-)
> >>>>> It's on of those big items left on the TODO, so will be good to see go
> >>>>> in. Then one should implement an ahci host controller for queued
> >>>>> command support next...
> >>>> Or use the scsi emulation :-)
> >>> Ah, did not know that queueing was fully implemented there yet!
> >> It isn't, but it's nearer than the SATA emulation!
> > 
> > ahci wouldn't be too much work, but definitely more so than finishing
> > the scsi bits!
> 
> That sounds great! I feel, like my dreams come true.
> 
> 
> BTW: Fabrice said, he will use the POSIX AIO (i guess, he means
> http://www.bullopensource.org/posix/ in case of Linux, right?)

Well I would assume that he just would use the glibc posix aio, which is
suboptimal but at least the code can be reused. The bull project looks
like it's trying to mimic posix aio on top of linux aio, so (if they got
the details right) that should be faster. I didn't check their sources,
though. You should be able to use the bull stuff with qemu, it would
most likely overloading the glibc function for posix aio.

> Which other OS do also support the POSIX AIO API?

No idea really, but I would guess any "unixy" OS out there.

-- 
Jens Axboe