Re: xen_evtchn_do_upcall

[David Xu <davidxu06@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 3160 bytes --]

Hi Lan,

I am trying to improve the TCP/UDP performance in VM. Due the the vCPU
scheduling on a pCPU sharing platform, the TCP receiving delay will be
significant which hurt the TCP/UDP throughput. So I want to offload the
softIRQ context to another Idle pCPU which I call fast-tick CPU. The packet
receiving process is like this: IRQ routine can continue picking packet
from ring buffer and put it to TCP receive buffer ( receive_queue,
prequeue, backlog_queue ) in kernel no matter whether the user process on
another CPU shared with other VMs is running or not. Once the vCPU holding
the user process gets scheduled, user process will fetch all packets from
receive buffer in kernel, which can improve the throughput. This works well
for UDP unfortunately does not work for TCP currently.

I found those  xen_evtchn_do_upcall routines existing when irq context try
to get the spinlock on the socket (Of course, they may happen in other
paths). If this spinlock is held by process context, irq context has to
spin on it and can not put any packet to receive buffer in time. So I doubt
these   xen_evtchn_do_upcall routine are due to the synchronization between
process context and irq context. Since I run process context and irq
context on 2 different vCPU, when they try to get the spinlock on the same
socket there will be interrupts between 2 vCPU which are implemented by
event in Xen.

If there is any error in my description, please correct me. Thanks.

Regards,
Cong

2012/10/24 Ian Campbell <Ian.Campbell@citrix.com>

> On Wed, 2012-10-24 at 14:39 +0100, David Xu wrote:
> > Hi Lan,
> >
> >
> > Thanks for your reply. I did a experiment as follows,
> > I assigned 2 vCPU to a VM, one vCPU (vCPU0) was pinned to a physical
> > CPU shared with several other VMs and the other vCPU (vCPU1) was
> > pinned to an idle physical CPU occupied by this VM only. Then in Guest
> > OS I run iperf server to measure its TCP receiving throughput. In
> > order to shrink the receiving delay caused by vCPU scheduling, I pin
> > the IRQ context of NIC to vCPU1 and run iperf server on the vCPU0.
> > This method works well for UDP, but does not work for TCP. I track the
> > involved function by ftrace and get the following results which
> > contains lots of xen_evtchn_do_upcall routine. What's the meaning of
> > this process (xen_evtchn_do_upcall => handle_irq_event =>
> > xennet_tx_buf_gc =>  gnttab_query_foreign_access)?
>
> Have you looked at the code for any of those functions?
>
> If you had done you'd find it is pretty obviously an interrupt being
> delivered to the network device and the associated work to satisfy that
> interrupt.
>
> It doesn't seem that surprising that an iperf test should involve lots
> of network interrupts.
>
> It's not entirely clear to me what you are expecting to find and/or what
> you are trying to prove.
>
> > When I run both process context of application ( e.g. iperf server )
> > and IRQ context on vCPU1 which is the ''fast" core, no
> > any xen_evtchn_do_upcall routine found.
>
> Perhaps on the fast core NAPI is able to kick in and therefore the NIC
> becomes polled instead of interrupt driven?
>
> Ian.
>
>

[-- Attachment #1.2: Type: text/html, Size: 3837 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel