* Re: [Xen-users] Changing netback tx interrupts affinity on Dom0 [not found] ` <1441294496.26292.372.camel@citrix.com> @ 2015-09-03 16:06 ` Jintack Lim 2015-09-03 17:30 ` Jintack Lim 2015-09-03 18:21 ` Julien Grall 0 siblings, 2 replies; 5+ messages in thread From: Jintack Lim @ 2015-09-03 16:06 UTC (permalink / raw) To: xen-devel; +Cc: Ian Campbell [-- Attachment #1.1: Type: text/plain, Size: 2121 bytes --] Hi, I was trying to set irq affinity by writing a value to /proc/irq/<irq>/smp_affinity, but smp_affinity value was not changed at all. Ian suggested to take this to the devel list. I'm working on Xen4.5, ARMv8. Thanks, Jintack On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com> wrote: > On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote: > > Hi, > > > > While I was running Apache server, > > I found that one of Dom0 vcpu is running 100% to handle irqs, > > and those irqs are set to be processed only on that specific vcpu. > > > > Referring to this document, > > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide > > I tried to change smp_affinity by writing a value to /proc/irq/<irq > > -no>/smp_affinity, > > however the smp_affinity value was not changed. > > > > I'm working on Xen 4.5 on ARMv8, > > and the irq is netback tx interrupt. > > > > # cat /proc/irq/106/smp_affinity > > 1 > > > > # cat /proc/interrupts > > ... > > 106: 53849 0 0 0 xen-dyn-event > > vif1.1-q0-tx > > 107: 1 0 0 0 xen-dyn-event > > vif1.1-q0-rx > > 108: 61460 0 0 0 xen-dyn-event > > vif1.1-q1-tx > > 109: 1 0 0 0 xen-dyn-event > > vif1.1-q1-rx > > 110: 67118 0 0 0 xen-dyn-event > > vif1.1-q2-tx > > 111: 1 0 0 0 xen-dyn-event > > vif1.1-q2-rx > > 112: 58273 0 0 0 xen-dyn-event > > vif1.1-q3-tx > > 113: 1 0 0 0 xen-dyn-event > > vif1.1-q3-rx > > ... > > > > What would be the way to change smp_affinity? > > It should be via /proc/irq/<irq>/smp_affinity as you've tried, I asked > Stefano and he things this was already implemented in 4.5 even. That it is > not happening would be a bug, I think. > > I'd suggest you take this to the devel list as a bug. > > > and where is the affinity set initially for netback tx interrupts? > > Not, sure, I think it might just be generic IRQ code. > > Ian. > > [-- Attachment #1.2: Type: text/html, Size: 3459 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-users] Changing netback tx interrupts affinity on Dom0 2015-09-03 16:06 ` [Xen-users] Changing netback tx interrupts affinity on Dom0 Jintack Lim @ 2015-09-03 17:30 ` Jintack Lim 2015-09-03 18:21 ` Julien Grall 1 sibling, 0 replies; 5+ messages in thread From: Jintack Lim @ 2015-09-03 17:30 UTC (permalink / raw) To: xen-devel; +Cc: Ian Campbell [-- Attachment #1.1: Type: text/plain, Size: 2387 bytes --] On Thu, Sep 3, 2015 at 12:06 PM, Jintack Lim <jintack@cs.columbia.edu> wrote: > Hi, > > I was trying to set irq affinity by writing a value > to /proc/irq/<irq>/smp_affinity, > but smp_affinity value was not changed at all. > Ian suggested to take this to the devel list. > I'm working on Xen4.5, ARMv8. > I checked that setting smp_affinity works on Xen4.5, x86. So, probably this is an ARM specific issue. > > Thanks, > Jintack > > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com> > wrote: > >> On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote: >> > Hi, >> > >> > While I was running Apache server, >> > I found that one of Dom0 vcpu is running 100% to handle irqs, >> > and those irqs are set to be processed only on that specific vcpu. >> > >> > Referring to this document, >> > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide >> > I tried to change smp_affinity by writing a value to /proc/irq/<irq >> > -no>/smp_affinity, >> > however the smp_affinity value was not changed. >> > >> > I'm working on Xen 4.5 on ARMv8, >> > and the irq is netback tx interrupt. >> > >> > # cat /proc/irq/106/smp_affinity >> > 1 >> > >> > # cat /proc/interrupts >> > ... >> > 106: 53849 0 0 0 xen-dyn-event >> > vif1.1-q0-tx >> > 107: 1 0 0 0 xen-dyn-event >> > vif1.1-q0-rx >> > 108: 61460 0 0 0 xen-dyn-event >> > vif1.1-q1-tx >> > 109: 1 0 0 0 xen-dyn-event >> > vif1.1-q1-rx >> > 110: 67118 0 0 0 xen-dyn-event >> > vif1.1-q2-tx >> > 111: 1 0 0 0 xen-dyn-event >> > vif1.1-q2-rx >> > 112: 58273 0 0 0 xen-dyn-event >> > vif1.1-q3-tx >> > 113: 1 0 0 0 xen-dyn-event >> > vif1.1-q3-rx >> > ... >> > >> > What would be the way to change smp_affinity? >> >> It should be via /proc/irq/<irq>/smp_affinity as you've tried, I asked >> Stefano and he things this was already implemented in 4.5 even. That it is >> not happening would be a bug, I think. >> >> I'd suggest you take this to the devel list as a bug. >> >> > and where is the affinity set initially for netback tx interrupts? >> >> Not, sure, I think it might just be generic IRQ code. >> >> Ian. >> >> > [-- Attachment #1.2: Type: text/html, Size: 4300 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-users] Changing netback tx interrupts affinity on Dom0 2015-09-03 16:06 ` [Xen-users] Changing netback tx interrupts affinity on Dom0 Jintack Lim 2015-09-03 17:30 ` Jintack Lim @ 2015-09-03 18:21 ` Julien Grall 2015-09-03 21:12 ` Jintack Lim 1 sibling, 1 reply; 5+ messages in thread From: Julien Grall @ 2015-09-03 18:21 UTC (permalink / raw) To: Jintack Lim, xen-devel; +Cc: Ian Campbell On 03/09/15 17:06, Jintack Lim wrote: > Hi, Hi Jintack, > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com > <mailto:ian.campbell@citrix.com>> wrote: > > On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote: > > Hi, > > > > While I was running Apache server, > > I found that one of Dom0 vcpu is running 100% to handle irqs, > > and those irqs are set to be processed only on that specific vcpu. > > > > Referring to this document, > > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide > > I tried to change smp_affinity by writing a value to /proc/irq/<irq > > -no>/smp_affinity, > > however the smp_affinity value was not changed. > > > > I'm working on Xen 4.5 on ARMv8, > > and the irq is netback tx interrupt. > > > > # cat /proc/irq/106/smp_affinity > > 1 > > > > # cat /proc/interrupts > > ... > > 106: 53849 0 0 0 xen-dyn-event > > vif1.1-q0-tx > > 107: 1 0 0 0 xen-dyn-event > > vif1.1-q0-rx > > 108: 61460 0 0 0 xen-dyn-event > > vif1.1-q1-tx > > 109: 1 0 0 0 xen-dyn-event > > vif1.1-q1-rx > > 110: 67118 0 0 0 xen-dyn-event > > vif1.1-q2-tx > > 111: 1 0 0 0 xen-dyn-event > > vif1.1-q2-rx > > 112: 58273 0 0 0 xen-dyn-event > > vif1.1-q3-tx > > 113: 1 0 0 0 xen-dyn-event > > vif1.1-q3-rx > > ... > > > > What would be the way to change smp_affinity? All those interrupts are in-fine event channels. This is a bug in Linux which I sent a fix a month ago and it's queued in xentip for Linux 4.3 [1] The event channel rebinding was not working because we don't have vector callback on ARM and therefore the driver was only allowing event channel to be routed on VCPU0. commit 4a5b69464e51f4a8dd432e8c2a1468630df1a53c Author: Julien Grall <julien.grall@citrix.com> Date: Tue Jul 28 10:10:42 2015 +0100 xen/events: Support event channel rebind on ARM Currently, the event channel rebind code is gated with the presence of the vector callback. The virtual interrupt controller on ARM has the concept of per-CPU interrupt (PPI) which allow us to support per-VCPU event channel.Therefore there is no need of vector callback for ARM. Xen is already using a free PPI to notify the guest VCPU of an event. Furthermore, the xen code initialization in Linux (see arch/arm/xen/enlighten.c) is requesting correctly a per-CPU IRQ. Introduce new helper xen_support_evtchn_rebind to allow architecture decide whether rebind an event is support or not. It will always return true on ARM and keep the same behavior on x86. This is also allow us to drop the usage of xen_have_vector_callback entirely in the ARM code. Signed-off-by: Julien Grall <julien.grall@citrix.com> Signed-off-by: David Vrabel <david.vrabel@citrix.com> Regards, [1] https://lkml.org/lkml/2015/7/28/205 -- Julien Grall ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-users] Changing netback tx interrupts affinity on Dom0 2015-09-03 18:21 ` Julien Grall @ 2015-09-03 21:12 ` Jintack Lim 2015-09-04 13:28 ` Julien Grall 0 siblings, 1 reply; 5+ messages in thread From: Jintack Lim @ 2015-09-03 21:12 UTC (permalink / raw) To: Julien Grall; +Cc: Ian Campbell, xen-devel [-- Attachment #1.1: Type: text/plain, Size: 3949 bytes --] Thanks Julien. On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@citrix.com> wrote: > On 03/09/15 17:06, Jintack Lim wrote: > > Hi, > > Hi Jintack, > > > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com > > <mailto:ian.campbell@citrix.com>> wrote: > > > > On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote: > > > Hi, > > > > > > While I was running Apache server, > > > I found that one of Dom0 vcpu is running 100% to handle irqs, > > > and those irqs are set to be processed only on that specific vcpu. > > > > > > Referring to this document, > > > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide > > > I tried to change smp_affinity by writing a value to /proc/irq/<irq > > > -no>/smp_affinity, > > > however the smp_affinity value was not changed. > > > > > > I'm working on Xen 4.5 on ARMv8, > > > and the irq is netback tx interrupt. > > > > > > # cat /proc/irq/106/smp_affinity > > > 1 > > > > > > # cat /proc/interrupts > > > ... > > > 106: 53849 0 0 0 xen-dyn-event > > > vif1.1-q0-tx > > > 107: 1 0 0 0 xen-dyn-event > > > vif1.1-q0-rx > > > 108: 61460 0 0 0 xen-dyn-event > > > vif1.1-q1-tx > > > 109: 1 0 0 0 xen-dyn-event > > > vif1.1-q1-rx > > > 110: 67118 0 0 0 xen-dyn-event > > > vif1.1-q2-tx > > > 111: 1 0 0 0 xen-dyn-event > > > vif1.1-q2-rx > > > 112: 58273 0 0 0 xen-dyn-event > > > vif1.1-q3-tx > > > 113: 1 0 0 0 xen-dyn-event > > > vif1.1-q3-rx > > > ... > > > > > > What would be the way to change smp_affinity? > > All those interrupts are in-fine event channels. This is a bug in Linux > which I sent a fix a month ago and it's queued in xentip for Linux 4.3 [1] > > The event channel rebinding was not working because we don't have vector > callback on ARM and therefore the driver was only allowing event channel > to be routed on VCPU0. > > After applying this patch, I was able to set smp_affinity. In addition, the default value of smp_affinity for netback tx interrupts on Dom0 becomes "f" (I have four vcpus on Dom0). What I meant by 'netback tx interrupts' is vif1.1-q<num>-tx This is slightly different from x86, because netback tx interrupts are pinned to separate vcpus on x86. I'm not sure this question is appropriate for devel-list, but why the policy to set smp_affinity for netback tx interrupts are different on ARM and x86? > commit 4a5b69464e51f4a8dd432e8c2a1468630df1a53c > Author: Julien Grall <julien.grall@citrix.com> > Date: Tue Jul 28 10:10:42 2015 +0100 > > xen/events: Support event channel rebind on ARM > Currently, the event channel rebind code is gated with the presence of > the vector callback. > > The virtual interrupt controller on ARM has the concept of per-CPU > interrupt (PPI) which allow us to support per-VCPU event > channel.Therefore there is no need of vector callback for ARM. > > Xen is already using a free PPI to notify the guest VCPU of an event. > Furthermore, the xen code initialization in Linux (see > arch/arm/xen/enlighten.c) is requesting correctly a per-CPU IRQ. > > Introduce new helper xen_support_evtchn_rebind to allow architecture > decide whether rebind an event is support or not. It will always return > true on ARM and keep the same behavior on x86. > > This is also allow us to drop the usage of xen_have_vector_callback > entirely in the ARM code. > > Signed-off-by: Julien Grall <julien.grall@citrix.com> > Signed-off-by: David Vrabel <david.vrabel@citrix.com> > > Regards, > > [1] https://lkml.org/lkml/2015/7/28/205 > > -- > Julien Grall > > [-- Attachment #1.2: Type: text/html, Size: 7067 bytes --] [-- Attachment #2: Type: text/plain, Size: 126 bytes --] _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Xen-users] Changing netback tx interrupts affinity on Dom0 2015-09-03 21:12 ` Jintack Lim @ 2015-09-04 13:28 ` Julien Grall 0 siblings, 0 replies; 5+ messages in thread From: Julien Grall @ 2015-09-04 13:28 UTC (permalink / raw) To: Jintack Lim; +Cc: David Vrabel, Ian Campbell, xen-devel Hi, On 03/09/15 22:12, Jintack Lim wrote: > On Thu, Sep 3, 2015 at 2:21 PM, Julien Grall <julien.grall@citrix.com > <mailto:julien.grall@citrix.com>> wrote: > > On 03/09/15 17:06, Jintack Lim wrote: > > Hi, > > Hi Jintack, > > > On Thu, Sep 3, 2015 at 11:34 AM, Ian Campbell <ian.campbell@citrix.com <mailto:ian.campbell@citrix.com> > > <mailto:ian.campbell@citrix.com <mailto:ian.campbell@citrix.com>>> > wrote: > > > > On Thu, 2015-09-03 at 10:59 -0400, Jintack Lim wrote: > > > Hi, > > > > > > While I was running Apache server, > > > I found that one of Dom0 vcpu is running 100% to handle irqs, > > > and those irqs are set to be processed only on that specific > vcpu. > > > > > > Referring to this document, > > > > http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide > > > I tried to change smp_affinity by writing a value to > /proc/irq/<irq > > > -no>/smp_affinity, > > > however the smp_affinity value was not changed. > > > > > > I'm working on Xen 4.5 on ARMv8, > > > and the irq is netback tx interrupt. > > > > > > # cat /proc/irq/106/smp_affinity > > > 1 > > > > > > # cat /proc/interrupts > > > ... > > > 106: 53849 0 0 0 xen-dyn-event > > > vif1.1-q0-tx > > > 107: 1 0 0 0 xen-dyn-event > > > vif1.1-q0-rx > > > 108: 61460 0 0 0 xen-dyn-event > > > vif1.1-q1-tx > > > 109: 1 0 0 0 xen-dyn-event > > > vif1.1-q1-rx > > > 110: 67118 0 0 0 xen-dyn-event > > > vif1.1-q2-tx > > > 111: 1 0 0 0 xen-dyn-event > > > vif1.1-q2-rx > > > 112: 58273 0 0 0 xen-dyn-event > > > vif1.1-q3-tx > > > 113: 1 0 0 0 xen-dyn-event > > > vif1.1-q3-rx > > > ... > > > > > > What would be the way to change smp_affinity? > > All those interrupts are in-fine event channels. This is a bug in Linux > which I sent a fix a month ago and it's queued in xentip for Linux > 4.3 [1] > > The event channel rebinding was not working because we don't have vector > callback on ARM and therefore the driver was only allowing event channel > to be routed on VCPU0. > > > After applying this patch, I was able to set smp_affinity. > > In addition, the default value of smp_affinity for netback tx interrupts > on Dom0 becomes "f" (I have four vcpus on Dom0). > What I meant by 'netback tx interrupts' is vif1.1-q<num>-tx > This is slightly different from x86, because netback tx interrupts are > pinned to separate vcpus on x86. > I'm not sure this question is appropriate for devel-list, > but why the policy to set smp_affinity for netback tx interrupts are > different on ARM and x86? I'm not sure why. By default on both ARM and x86 the IRQ code is setting the affinity to all the vCPU. Note that you are likely using PV for x86 DOM0 which is different than HVM so it may be possible that we differ for setting the affinity. I've CCed David who was working on the event channel code. It may have more insights. Regards, -- Julien Grall ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2015-09-04 13:28 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CAHyh4xj3+si5+TzSGJeN3Wp0VE8MnPV30BFc_b7c+uuW+o+2mw@mail.gmail.com>
[not found] ` <1441294496.26292.372.camel@citrix.com>
2015-09-03 16:06 ` [Xen-users] Changing netback tx interrupts affinity on Dom0 Jintack Lim
2015-09-03 17:30 ` Jintack Lim
2015-09-03 18:21 ` Julien Grall
2015-09-03 21:12 ` Jintack Lim
2015-09-04 13:28 ` Julien Grall
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).