From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Xu Subject: Re: xen_evtchn_do_upcall Date: Wed, 24 Oct 2012 09:39:35 -0400 Message-ID: References: <1351070623.2237.120.camel@zakaz.uk.xensource.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============3206735932390025830==" Return-path: In-Reply-To: <1351070623.2237.120.camel@zakaz.uk.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ian Campbell Cc: "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org --===============3206735932390025830== Content-Type: multipart/alternative; boundary=20cf3074b0a4a1a52f04ccce3736 --20cf3074b0a4a1a52f04ccce3736 Content-Type: text/plain; charset=ISO-8859-1 Hi Lan, Thanks for your reply. I did a experiment as follows, I assigned 2 vCPU to a VM, one vCPU (vCPU0) was pinned to a physical CPU shared with several other VMs and the other vCPU (vCPU1) was pinned to an idle physical CPU occupied by this VM only. Then in Guest OS I run iperf server to measure its TCP receiving throughput. In order to shrink the receiving delay caused by vCPU scheduling, I pin the IRQ context of NIC to vCPU1 and run iperf server on the vCPU0. This method works well for UDP, but does not work for TCP. I track the involved function by ftrace and get the following results which contains lots of xen_evtchn_do_upcall routine. What's the meaning of this process (xen_evtchn_do_upcall => handle_irq_event => xennet_tx_buf_gc => gnttab_query_foreign_access)? 1) | tcp_v4_rcv() { 1) 0.087 us | __inet_lookup_established(); 1) | sk_filter() { 1) | security_sock_rcv_skb() { 1) 0.049 us | cap_socket_sock_rcv_skb(); 1) 0.374 us | } 1) 0.708 us | } 1) | _raw_spin_lock() { 1) | xen_evtchn_do_upcall() { 1) 0.051 us | exit_idle(); 1) | irq_enter() { 1) | rcu_irq_enter() { 1) 0.094 us | rcu_exit_nohz(); 1) 0.432 us | } 1) 0.055 us | idle_cpu(); 1) 1.166 us | } 1) | __xen_evtchn_do_upcall() { 1) 0.120 us | irq_to_desc(); 1) | handle_edge_irq() { 1) 0.103 us | _raw_spin_lock(); 1) | ack_dynirq() { 1) | evtchn_from_irq() { 1) | info_for_irq() { 1) | irq_get_irq_data() { 1) 0.051 us | irq_to_desc(); 1) 0.400 us | } 1) 0.746 us | } 1) 1.074 us | } 1) 0.050 us | irq_move_irq(); 1) 1.767 us | } 1) | handle_irq_event() { 1) 0.164 us | _raw_spin_unlock(); 1) | handle_irq_event_percpu() { 1) | xennet_interrupt() { 1) 0.125 us | _raw_spin_lock_irqsave(); 1) | xennet_tx_buf_gc() { 1) 0.082 us | gnttab_query_foreign_access(); 1) 0.050 us | gnttab_end_foreign_access_ref(); 1) 0.070 us | gnttab_release_grant_reference(); 1) | dev_kfree_skb_irq() { 1) 0.061 us | raise_softirq_irqoff(); 1) 0.460 us | } 1) 0.058 us | gnttab_query_foreign_access(); 1) 0.050 us | gnttab_end_foreign_access_ref(); 1) 0.050 us | gnttab_release_grant_reference(); 1) | dev_kfree_skb_irq() { 1) 0.059 us | raise_softirq_irqoff(); 1) 0.440 us | } 1) 3.710 us | } 1) 0.092 us | _raw_spin_unlock_irqrestore(); 1) 4.845 us | } 1) 0.075 us | note_interrupt(); 1) 5.567 us | } 1) 0.055 us | _raw_spin_lock(); 1) 6.889 us | } 1) 0.080 us | _raw_spin_unlock(); 1) + 10.081 us | } 1) + 10.965 us | } 1) | irq_exit() { 1) | rcu_irq_exit() { 1) 0.086 us | rcu_enter_nohz(); 1) 0.424 us | } 1) 0.049 us | idle_cpu(); 1) 1.094 us | } 1) + 14.555 us | } 1) 0.120 us | } /* _raw_spin_lock */ 1) | __wake_up_sync_key() { 1) 0.099 us | _raw_spin_lock_irqsave(); 1) | __wake_up_common() { 1) | autoremove_wake_function() { 1) | default_wake_function() { 1) | try_to_wake_up() { 1) 0.103 us | _raw_spin_lock_irqsave(); 1) 0.078 us | task_waking_fair(); 1) 0.102 us | select_task_rq_fair(); 1) | xen_smp_send_reschedule() { 1) | xen_send_IPI_one() { 1) | notify_remote_via_irq() { 1) | evtchn_from_irq() { 1) | info_for_irq() { 1) | irq_get_irq_data() { 1) 0.067 us | irq_to_desc(); 1) 0.396 us | } 1) 0.727 us | } 1) 1.055 us | } 1) 1.699 us | } 1) 2.048 us | } 1) 2.407 us | } 1) 0.066 us | ttwu_stat(); 1) 0.114 us | _raw_spin_unlock_irqrestore(); 1) 4.941 us | } 1) 5.294 us | } 1) 5.645 us | } 1) 6.023 us | } 1) 0.094 us | _raw_spin_unlock_irqrestore(); 1) 7.156 us | } 1) 0.058 us | dst_metric(); 1) | inet_csk_reset_xmit_timer.constprop.34() { 1) | sk_reset_timer() { 1) | mod_timer() { 1) | lock_timer_base.isra.30() { 1) 0.099 us | _raw_spin_lock_irqsave(); 1) 0.436 us | } 1) 0.049 us | idle_cpu(); 1) 0.116 us | _raw_spin_unlock(); 1) 0.074 us | _raw_spin_lock(); 1) 0.072 us | internal_add_timer(); 1) 0.103 us | _raw_spin_unlock_irqrestore(); 1) 2.673 us | } 1) 3.039 us | } 1) 3.397 us | } 1) 0.082 us | _raw_spin_unlock(); 1) 0.061 us | sock_put(); 1) + 48.704 us | } When I run both process context of application ( e.g. iperf server ) and IRQ context on vCPU1 which is the ''fast" core, no any xen_evtchn_do_upcall routine found. 1) | tcp_v4_rcv() { 1) 0.081 us | __inet_lookup_established(); 1) | sk_filter() { 1) | security_sock_rcv_skb() { 1) 0.059 us | cap_socket_sock_rcv_skb(); 1) 0.542 us | } 1) 0.875 us | } 1) 0.060 us | _raw_spin_lock(); 1) 0.117 us | _raw_spin_unlock(); 1) 0.053 us | sock_put(); 1) 2.703 us | } Do you think these xen_evtchn_do_upcall routines are due to the synchronization between process context and softirq context? Thanks. Regards, Cong 2012/10/24 Ian Campbell > On Mon, 2012-10-22 at 02:51 +0100, David Xu wrote: > > Hi, > > > > > > Is anybody know the purpose of this method (xen_evtchn_do_upcall)? > > It is the callback used to inject event channels events (i.e. IRQs) into > the guest. You would expect to see it at the base of any stack trace > taken from interrupt context. > > > When I run a user level application involved in TCP receiving and the > > SoftIRQ for eth0 on the same CPU core, everything is OK. But if I run > > them on 2 different cores, there will be xen_evtchn_do_upcall() > > existing (maybe when the local_bh_disable() or local_bh_enable() is > > called) > > it would not be unusual to get an interrupt immediately after > re-enabling interrupts. > > > in __inet_lookup_established() routine which costs longer time than > > the first scenario. Is it due to the synchronization issue between > > process context and softirq context? Thanks for any reply. > > > > > > 1) | __inet_lookup_established() { > > 1) | xen_evtchn_do_upcall() { > > 1) 0.054 us | exit_idle(); > > 1) | irq_enter() { > > 1) | rcu_irq_enter() { > > 1) 0.102 us | rcu_exit_nohz(); > > 1) 0.431 us | } > > 1) 0.064 us | idle_cpu(); > > 1) 1.152 us | } > > 1) | __xen_evtchn_do_upcall() { > > 1) 0.119 us | irq_to_desc(); > > 1) | handle_edge_irq() { > > 1) 0.107 us | _raw_spin_lock(); > > 1) | ack_dynirq() { > > 1) | evtchn_from_irq() { > > 1) | info_for_irq() { > > 1) | irq_get_irq_data() { > > 1) 0.052 us | irq_to_desc(); > > 1) 0.418 us | } > > 1) 0.782 us | } > > 1) 1.135 us | } > > 1) 0.049 us | irq_move_irq(); > > 1) 1.800 us | } > > 1) | handle_irq_event() { > > 1) 0.161 us | _raw_spin_unlock(); > > 1) | handle_irq_event_percpu() { > > 1) | xennet_interrupt() { > > 1) 0.125 us | _raw_spin_lock_irqsave(); > > 1) | xennet_tx_buf_gc() { > > 1) 0.079 us | gnttab_query_foreign_access(); > > 1) 0.050 us | gnttab_end_foreign_access_ref(); > > 1) 0.069 us | gnttab_release_grant_reference(); > > 1) | dev_kfree_skb_irq() { > > 1) 0.055 us | raise_softirq_irqoff(); > > 1) 0.472 us | } > > 1) 0.049 us | gnttab_query_foreign_access(); > > 1) 0.058 us | gnttab_end_foreign_access_ref(); > > 1) 0.058 us | gnttab_release_grant_reference(); > > 1) | dev_kfree_skb_irq() { > > 1) 0.050 us | raise_softirq_irqoff(); > > 1) 0.456 us | } > > 1) 3.714 us | } > > 1) 0.102 us | _raw_spin_unlock_irqrestore(); > > 1) 4.857 us | } > > 1) 0.061 us | note_interrupt(); > > 1) 5.571 us | } > > 1) 0.054 us | _raw_spin_lock(); > > 1) 6.707 us | } > > 1) 0.083 us | _raw_spin_unlock(); > > 1) + 10.083 us | } > > 1) + 10.985 us | } > > 1) | irq_exit() { > > 1) | rcu_irq_exit() { > > 1) 0.087 us | rcu_enter_nohz(); > > 1) 0.429 us | } > > 1) 0.049 us | idle_cpu(); > > 1) 1.088 us | } > > 1) + 14.551 us | } > > 1) 0.191 us | } /* __inet_lookup_established */ > > > --20cf3074b0a4a1a52f04ccce3736 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Lan,

Thanks for your reply. I did a experiment as fol= lows,=A0
I assigned 2 vCPU to a VM, one vCPU (vCPU0) was pinned t= o a physical CPU shared with several other VMs and the other vCPU (vCPU1) w= as pinned to an idle physical CPU occupied by this VM only. Then in Guest O= S I run iperf server to measure its TCP receiving throughput. In order to s= hrink the receiving delay caused by vCPU scheduling, I pin the IRQ context = of NIC to vCPU1 and run iperf server on the vCPU0. This method works well f= or UDP, but does not work for TCP. I track the involved function by ftrace = and get the following results which contains lots of xen_evtchn_do_upcall r= outine. What's the meaning of this process (xen_evtchn_do_upcall =3D>= ;=A0handle_irq_event =3D>=A0xennet_tx_buf_gc =3D>=A0=A0gnttab_query_f= oreign_access)?

=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0tcp_v4_rcv(= ) {
=A01) =A0 0.087 us =A0 =A0| =A0 =A0__inet_lookup_established(= );
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0sk_filter() {
=
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0security_sock_rcv_skb()= {
=A01) =A0 0.049 us =A0 =A0| =A0 =A0 =A0 =A0cap_socket_sock_rcv_skb();<= /div>
=A01) =A0 0.374 us =A0 =A0| =A0 =A0 =A0}
=A01) =A0 0.70= 8 us =A0 =A0| =A0 =A0}
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 = =A0_raw_spin_lock() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0= xen_evtchn_do_upcall() {
=A01) =A0 0.051 us =A0 =A0| =A0 =A0 =A0exit_idle();
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0irq_enter() {
=A01) =A0 = =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0rcu_irq_enter() {
=A01) = =A0 0.094 us =A0 =A0| =A0 =A0 =A0 =A0 =A0rcu_exit_nohz();
=A01) = =A0 0.432 us =A0 =A0| =A0 =A0 =A0 =A0}
=A01) =A0 0.055 us =A0 =A0| =A0 =A0 =A0 =A0idle_cpu();
=A01)= =A0 1.166 us =A0 =A0| =A0 =A0 =A0}
=A01) =A0 =A0 =A0 =A0 =A0 =A0= =A0 | =A0 =A0 =A0__xen_evtchn_do_upcall() {
=A01) =A0 0.120 us = =A0 =A0| =A0 =A0 =A0 =A0irq_to_desc();
=A01) =A0 =A0 =A0 =A0 =A0 = =A0 =A0 | =A0 =A0 =A0 =A0handle_edge_irq() {
=A01) =A0 0.103 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_lock();
=
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0ack_dynirq() {<= /div>
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0evtchn= _from_irq() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0= =A0 =A0 =A0info_for_irq() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0irq= _get_irq_data() {
=A01) =A0 0.051 us =A0 =A0| =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0irq_to_desc();
=A01) =A0 0.400 us =A0 =A0| =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 0.746 us =A0 =A0| =A0 =A0 = =A0 =A0 =A0 =A0 =A0}
=A01) =A0 1.074 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 0.= 050 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0irq_move_irq();
=A01) =A0 = 1.767 us =A0 =A0| =A0 =A0 =A0 =A0 =A0}
=A01) =A0 =A0 =A0 =A0 =A0 = =A0 =A0 | =A0 =A0 =A0 =A0 =A0handle_irq_event() {
=A01) =A0 0.164= us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_unlock();
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0handle_irq_= event_percpu() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 = =A0 =A0 =A0 =A0xennet_interrupt() {
=A01) =A0 0.125 us =A0 =A0| = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_lock_irqsave();
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0xennet_tx_buf_= gc() {
=A01) =A0 0.082 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= query_foreign_access();
=A01) =A0 0.050 us =A0 =A0| =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0gnttab_end_foreign_access_ref();
=A01) =A0= 0.070 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_release_grant_= reference();
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0dev_kfree_skb_irq() {
=A01) =A0 0.061 us =A0 =A0| =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0raise_softirq_irqoff();
=A01) =A0 0.46= 0 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 0.05= 8 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_query_foreign_acces= s();
=A01) =A0 0.050 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= end_foreign_access_ref();
=A01) =A0 0.050 us =A0 =A0| =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0gnttab_release_grant_reference();
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0dev_kfree_= skb_irq() {
=A01) =A0 0.059 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rai= se_softirq_irqoff();
=A01) =A0 0.440 us =A0 =A0| =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0}
=A01) =A0 3.710 us =A0 =A0| =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0}
=A01) =A0 0.092 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0_raw_spin_unlock_irqrestore();
=A01) =A0 4.845 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A0= 1) =A0 0.075 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0note_interrupt();
=
=A01) =A0 5.567 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0}
=A01) = =A0 0.055 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_lock();
= =A01) =A0 6.889 us =A0 =A0| =A0 =A0 =A0 =A0 =A0}
=A01) =A0 0.080 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_unlock();
=A01) + 10.081 us =A0 | =A0 =A0 =A0 =A0}
=A01) + 10.965 us= =A0 | =A0 =A0 =A0}
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 = =A0irq_exit() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 = =A0rcu_irq_exit() {
=A01) =A0 0.086 us =A0 =A0| =A0 =A0 =A0 =A0 =A0rcu_enter_nohz();
=
=A01) =A0 0.424 us =A0 =A0| =A0 =A0 =A0 =A0}
=A01) =A0 0.049= us =A0 =A0| =A0 =A0 =A0 =A0idle_cpu();
=A01) =A0 1.094 us =A0 = =A0| =A0 =A0 =A0}
=A01) + 14.555 us =A0 | =A0 =A0}
=A01) =A0 0.120 us =A0 =A0| =A0 =A0} /* _raw_spin_lock */
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0__wake_up_sync_key() {
=A01)= =A0 0.099 us =A0 =A0| =A0 =A0 =A0_raw_spin_lock_irqsave();
=A01)= =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0__wake_up_common() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0autoremove_wake_fun= ction() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0= default_wake_function() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0= =A0 =A0 =A0 =A0 =A0try_to_wake_up() {
=A01) =A0 0.103 us =A0 =A0= | =A0 =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_lock_irqsave();
=A01) =A0 0.078 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0task_waking_fai= r();
=A01) =A0 0.102 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0selec= t_task_rq_fair();
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0= =A0 =A0 =A0 =A0xen_smp_send_reschedule() {
=A01) =A0 =A0 =A0 =A0= =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0xen_send_IPI_one() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0notify_remote_via_irq() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0evtchn_from_irq() {
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0in= fo_for_irq() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0irq_get_irq_data() {
=A01) =A0 0.067 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0irq_to_desc();
=A01) =A0 0.396 us =A0 =A0| =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 0.727 us =A0 =A0|= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 1.055 us = =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 1.699 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=
=A01) =A0 2.048 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) =A0 2.407 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0}
=A01) = =A0 0.066 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0ttwu_stat();
=A0= 1) =A0 0.114 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_unlock_irqres= tore();
=A01) =A0 4.941 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0}
=A01) = =A0 5.294 us =A0 =A0| =A0 =A0 =A0 =A0 =A0}
=A01) =A0 5.645 us =A0= =A0| =A0 =A0 =A0 =A0}
=A01) =A0 6.023 us =A0 =A0| =A0 =A0 =A0}
=A01) =A0 0.094 us =A0 =A0| =A0 =A0 =A0_raw_spin_unlock_irqrestore= ();
=A01) =A0 7.156 us =A0 =A0| =A0 =A0}
=A01) =A0 0.058 us =A0 = =A0| =A0 =A0dst_metric();
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0= =A0inet_csk_reset_xmit_timer.constprop.34() {
=A01) =A0 =A0 =A0 = =A0 =A0 =A0 =A0 | =A0 =A0 =A0sk_reset_timer() {
=A01) =A0 =A0 =A0= =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0mod_timer() {
=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0lock_timer_base= .isra.30() {
=A01) =A0 0.099 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0_= raw_spin_lock_irqsave();
=A01) =A0 0.436 us =A0 =A0| =A0 =A0 =A0 = =A0 =A0}
=A01) =A0 0.049 us =A0 =A0| =A0 =A0 =A0 =A0 =A0idle_cpu(= );
=A01) =A0 0.116 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_unlock();
=A01) =A0 0.074 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_lock();
=A01) =A0 0.072 us =A0 =A0| =A0 =A0 =A0 =A0 =A0internal_add_timer();=
=A01) =A0 0.103 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_unlock_= irqrestore();
=A01) =A0 2.673 us =A0 =A0| =A0 =A0 =A0 =A0}
=A01) =A0 3.039= us =A0 =A0| =A0 =A0 =A0}
=A01) =A0 3.397 us =A0 =A0| =A0 =A0}
=A01) =A0 0.082 us =A0 =A0| =A0 =A0_raw_spin_unlock();
= =A01) =A0 0.061 us =A0 =A0| =A0 =A0sock_put();
=A01) + 48.704 us = =A0 | =A0}

When I run both process context of application ( = e.g. iperf server ) and IRQ context on vCPU1 which is the ''fast&qu= ot; core, no any=A0xen_evtchn_do_upcall routine found.=A0

=A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0tcp_v4_rcv() {
= =A01) =A0 0.081 us =A0 =A0| =A0 =A0__inet_lookup_established();
= =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0sk_filter() {
=A01) = =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0security_sock_rcv_skb() {
=A01) =A0 0.059 us =A0 =A0| =A0 =A0 =A0 =A0cap_socket_sock_rcv_skb();<= /div>
=A01) =A0 0.542 us =A0 =A0| =A0 =A0 =A0}
=A01) =A0 0.87= 5 us =A0 =A0| =A0 =A0}
=A01) =A0 0.060 us =A0 =A0| =A0 =A0_raw_sp= in_lock();
=A01) =A0 0.117 us =A0 =A0| =A0 =A0_raw_spin_unlock();=
=A01) =A0 0.053 us =A0 =A0| =A0 =A0sock_put();
=A01) =A0 2.7= 03 us =A0 =A0| =A0}

Do you think these=A0=A0= xen_evtchn_do_upcall routines are due to the synchronization between proces= s context and softirq context? Thanks.

Regards,
Cong


2012/10/24 Ian Campbell <Ian.Campbell@citri= x.com>
On Mon, 2012-10-22 at 02:5= 1 +0100, David Xu wrote:
> Hi,
>
>
> Is anybody know the purpose of this method (xen_evtchn_do_upcall)?

It is the callback used to inject event channels events (i.e. IRQs) i= nto
the guest. You would expect to see it at the base of any stack trace
taken from interrupt context.

> =A0When I run a user level application involved in TCP receiving and t= he
> SoftIRQ for eth0 on the same CPU core, everything is OK. But if I run<= br> > them on 2 different cores, there will be xen_evtchn_do_upcall()
> existing (maybe when the local_bh_disable() or local_bh_enable() is > called)

it would not be unusual to get an interrupt immediately after
re-enabling interrupts.

> =A0in __inet_lookup_established() routine which costs longer time than=
> the first scenario. Is it due to the synchronization issue between
> process context and softirq context? Thanks for any reply.
>
>
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0__inet_lookup_established()= {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0xen_evtchn_do_upcall() { > =A01) =A0 0.054 us =A0 =A0| =A0 =A0 =A0exit_idle();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0irq_enter() {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0rcu_irq_enter() { > =A01) =A0 0.102 us =A0 =A0| =A0 =A0 =A0 =A0 =A0rcu_exit_nohz();
> =A01) =A0 0.431 us =A0 =A0| =A0 =A0 =A0 =A0}
> =A01) =A0 0.064 us =A0 =A0| =A0 =A0 =A0 =A0idle_cpu();
> =A01) =A0 1.152 us =A0 =A0| =A0 =A0 =A0}
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0__xen_evtchn_do_upcall(= ) {
> =A01) =A0 0.119 us =A0 =A0| =A0 =A0 =A0 =A0irq_to_desc();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0handle_edge_irq() {=
> =A01) =A0 0.107 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_lock();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0ack_dynirq() {<= br> > =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0evtchn_from= _irq() {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0info_fo= r_irq() {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0irq= _get_irq_data() {
> =A01) =A0 0.052 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0irq_to_= desc();
> =A01) =A0 0.418 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.782 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 1.135 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.049 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0irq_move_irq();
> =A01) =A0 1.800 us =A0 =A0| =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0handle_irq_even= t() {
> =A01) =A0 0.161 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_unlock();=
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0handle_irq_= event_percpu() {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0xennet_= interrupt() {
> =A01) =A0 0.125 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_l= ock_irqsave();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0xen= net_tx_buf_gc() {
> =A01) =A0 0.079 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= query_foreign_access();
> =A01) =A0 0.050 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= end_foreign_access_ref();
> =A01) =A0 0.069 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= release_grant_reference();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0dev_kfree_skb_irq() {
> =A01) =A0 0.055 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rai= se_softirq_irqoff();
> =A01) =A0 0.472 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.049 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= query_foreign_access();
> =A01) =A0 0.058 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= end_foreign_access_ref();
> =A01) =A0 0.058 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0gnttab_= release_grant_reference();
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0dev_kfree_skb_irq() {
> =A01) =A0 0.050 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0rai= se_softirq_irqoff();
> =A01) =A0 0.456 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 3.714 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.102 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_u= nlock_irqrestore();
> =A01) =A0 4.857 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.061 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0 =A0note_interrupt(= );
> =A01) =A0 5.571 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.054 us =A0 =A0| =A0 =A0 =A0 =A0 =A0 =A0_raw_spin_lock(); > =A01) =A0 6.707 us =A0 =A0| =A0 =A0 =A0 =A0 =A0}
> =A01) =A0 0.083 us =A0 =A0| =A0 =A0 =A0 =A0 =A0_raw_spin_unlock();
> =A01) + 10.083 us =A0 | =A0 =A0 =A0 =A0}
> =A01) + 10.985 us =A0 | =A0 =A0 =A0}
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0irq_exit() {
> =A01) =A0 =A0 =A0 =A0 =A0 =A0 =A0 | =A0 =A0 =A0 =A0rcu_irq_exit() { > =A01) =A0 0.087 us =A0 =A0| =A0 =A0 =A0 =A0 =A0rcu_enter_nohz();
> =A01) =A0 0.429 us =A0 =A0| =A0 =A0 =A0 =A0}
> =A01) =A0 0.049 us =A0 =A0| =A0 =A0 =A0 =A0idle_cpu();
> =A01) =A0 1.088 us =A0 =A0| =A0 =A0 =A0}
> =A01) + 14.551 us =A0 | =A0 =A0}
> =A01) =A0 0.191 us =A0 =A0| =A0 =A0} /* __inet_lookup_established */


--20cf3074b0a4a1a52f04ccce3736-- --===============3206735932390025830== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============3206735932390025830==--