Direct I/O performance: direct interrupts to running domU?

Direct I/O performance: direct interrupts to running domU?

[John Byrne]

Hi,

I'm testing performance of a NIC assigned directly to a domU. (xen-3.0.3 
x86_64)

After your help straightening out the problems with my test environment, 
my netperf throughput numbers were  close to base Linux (generally with 
0.5%); however, when I did the netperf transaction tests, the numbers on 
Xen (either dom0 and domU) were 10-20% lower than the base Linux 
depending on the test parameters. Since this linear transaction test 
would be very sensitive to the latency/path-length involved in the 
interrupt delivery, I'm making the assumption, for the moment, that this 
is the problem without cranking up xenoprof/oprofile to prove it.

One of the questions that came up in a discussion was whether it might 
be technically possible to have interrupts delivered directly to a 
running domain without the hypervisor overhead. This would presumably 
involve some additional overhead at the time the hypervisor switched 
domains, but it might be a worthwhile trade-off.

Thanks,

John Byrne

Re: Direct I/O performance: direct interrupts to running domU?

[Mark Williamson]

> One of the questions that came up in a discussion was whether it might
> be technically possible to have interrupts delivered directly to a
> running domain without the hypervisor overhead. This would presumably
> involve some additional overhead at the time the hypervisor switched
> domains, but it might be a worthwhile trade-off.

ISTR Intel's VT containing some extensions to facilitate this sort of thing, 
but can't remember the details so I might be talking rubbish.

Of course, since VT implies fully virt (at the moment) and direct IO implies 
paravirt, this would have to be some kind of weird hybrid...  Seems like it 
ought to be possible.

I'll let more knowledgeable folks comment on the more general case of standard 
hardware...  But would note that if the guest could use polled access for 
lantency-sensitive stuff (for instance by tweaking the behaviour of NAPI) 
then the interrupt dispatch latency wouldn't be necessary at all.  Infiniband 
devices also support polling queues to reduce latency anyhow, so if you were 
to use those the problem could perhaps be reduced.

Just some random thoughts.

Cheers,
Mark

-- 
Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
Mark: To answer a question with a question: What use is a skateboard?
Dave: Skateboards have wheels.
Mark: My wheel has a wheel!