* [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" @ 2015-10-26 19:14 Josh Cartwright 2015-10-27 0:44 ` Paul E. McKenney 0 siblings, 1 reply; 10+ messages in thread From: Josh Cartwright @ 2015-10-26 19:14 UTC (permalink / raw) To: tglx, bigeasy Cc: linux-rt-users, linux-kernel, Eric Dumazet, Paul E. McKenney, David S. Miller This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41. While the use of synchronize_rcu_expedited() might make synchronize_net() "faster", it does so at significant cost on RT systems, as expediting a grace period forcibly preempts any high-priority RT tasks (via the stop_machine() mechanism). Without be3fc413da9e reverted, we can observe a latency spike up to 30us with cyclictest by rapidly unplugging/reestablishing an ethernet link. Cc: Eric Dumazet <eric.dumazet@gmail.com> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: David S. Miller <davem@davemloft.net> Signed-off-by: Josh Cartwright <joshc@ni.com> --- net/core/dev.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/net/core/dev.c b/net/core/dev.c index f8c23de..869ef62 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6969,10 +6969,7 @@ EXPORT_SYMBOL(free_netdev); void synchronize_net(void) { might_sleep(); - if (rtnl_is_locked()) - synchronize_rcu_expedited(); - else - synchronize_rcu(); + synchronize_rcu(); } EXPORT_SYMBOL(synchronize_net); -- 2.5.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-26 19:14 [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" Josh Cartwright @ 2015-10-27 0:44 ` Paul E. McKenney 2015-10-27 12:31 ` Josh Cartwright 0 siblings, 1 reply; 10+ messages in thread From: Paul E. McKenney @ 2015-10-27 0:44 UTC (permalink / raw) To: Josh Cartwright Cc: tglx, bigeasy, linux-rt-users, linux-kernel, Eric Dumazet, David S. Miller On Mon, Oct 26, 2015 at 02:14:55PM -0500, Josh Cartwright wrote: > This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41. > > While the use of synchronize_rcu_expedited() might make > synchronize_net() "faster", it does so at significant cost on RT > systems, as expediting a grace period forcibly preempts any > high-priority RT tasks (via the stop_machine() mechanism). > > Without be3fc413da9e reverted, we can observe a latency spike up to 30us > with cyclictest by rapidly unplugging/reestablishing an ethernet link. > > Cc: Eric Dumazet <eric.dumazet@gmail.com> > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > Cc: David S. Miller <davem@davemloft.net> > Signed-off-by: Josh Cartwright <joshc@ni.com> Hmmm... If I remember correctly, using expedited here resulted in impressive performance improvements in some important cases. Perhaps things have changed (I must defer to Eric), but if not, how about something like this instead? if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREMPT_RT_FULL)) synchronize_rcu_expedited(); else synchronize_rcu(); Alternatively, a boot-time option could be used: if (rtnl_is_locked() && !some_rt_boot_parameter) synchronize_rcu_expedited(); else synchronize_rcu(); I believe that the first alternative is better because it does the right thing without user intervention. The second would be preferred should distros want to offer full RT by default, but I am guessing thta most distros would be reluctant to do this for some time to come. Either way, these approaches have the advantage of giving RT users the latency they need, even in the face of networking configuration changes, while giving non-RT users the required performance of the networking configuration changes themselves. Thanx, Paul > --- > net/core/dev.c | 5 +---- > 1 file changed, 1 insertion(+), 4 deletions(-) > > diff --git a/net/core/dev.c b/net/core/dev.c > index f8c23de..869ef62 100644 > --- a/net/core/dev.c > +++ b/net/core/dev.c > @@ -6969,10 +6969,7 @@ EXPORT_SYMBOL(free_netdev); > void synchronize_net(void) > { > might_sleep(); > - if (rtnl_is_locked()) > - synchronize_rcu_expedited(); > - else > - synchronize_rcu(); > + synchronize_rcu(); > } > EXPORT_SYMBOL(synchronize_net); > > -- > 2.5.1 > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 0:44 ` Paul E. McKenney @ 2015-10-27 12:31 ` Josh Cartwright 2015-10-27 14:18 ` Eric Dumazet 0 siblings, 1 reply; 10+ messages in thread From: Josh Cartwright @ 2015-10-27 12:31 UTC (permalink / raw) To: Paul E. McKenney Cc: tglx, bigeasy, linux-rt-users, linux-kernel, Eric Dumazet, David S. Miller On Mon, Oct 26, 2015 at 05:44:22PM -0700, Paul E. McKenney wrote: > On Mon, Oct 26, 2015 at 02:14:55PM -0500, Josh Cartwright wrote: > > This reverts commit be3fc413da9eb17cce0991f214ab019d16c88c41. > > > > While the use of synchronize_rcu_expedited() might make > > synchronize_net() "faster", it does so at significant cost on RT > > systems, as expediting a grace period forcibly preempts any > > high-priority RT tasks (via the stop_machine() mechanism). > > > > Without be3fc413da9e reverted, we can observe a latency spike up to 30us > > with cyclictest by rapidly unplugging/reestablishing an ethernet link. [..] > > Hmmm... If I remember correctly, using expedited here resulted > in impressive performance improvements in some important cases. > Perhaps things have changed (I must defer to Eric), but if not, how > about something like this instead? > > if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREMPT_RT_FULL)) > synchronize_rcu_expedited(); > else > synchronize_rcu(); > > Alternatively, a boot-time option could be used: > > if (rtnl_is_locked() && !some_rt_boot_parameter) > synchronize_rcu_expedited(); > else > synchronize_rcu(); > > I believe that the first alternative is better because it does the right > thing without user intervention. The second would be preferred should > distros want to offer full RT by default, but I am guessing thta most > distros would be reluctant to do this for some time to come. > > Either way, these approaches have the advantage of giving RT users the > latency they need, even in the face of networking configuration changes, > while giving non-RT users the required performance of the networking > configuration changes themselves. Okay, yes, I like the first suggestion better as well, I've included a patch below that does just that. I hope you don't mind me turning it into a Suggested-by :). Thanks for taking a look! Josh -- 8< -- From: Josh Cartwright <joshc@ni.com> Subject: [PATCH] net: make synchronize_rcu_expedited() conditional on RT_FULL While the use of synchronize_rcu_expedited() might make synchronize_net() "faster", it does so at significant cost on RT systems, as expediting a grace period forcibly preempts any high-priority RT tasks (via the stop_machine() mechanism). Without this change, we can observe a latency spike up to 30us with cyclictest by rapidly unplugging/reestablishing an ethernet link. Cc: Eric Dumazet <eric.dumazet@gmail.com> Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: David S. Miller <davem@davemloft.net> Suggested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Josh Cartwright <joshc@ni.com> --- net/core/dev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/net/core/dev.c b/net/core/dev.c index f8c23de..16fbef8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); void synchronize_net(void) { might_sleep(); - if (rtnl_is_locked()) + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) synchronize_rcu_expedited(); else synchronize_rcu(); -- 2.5.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 12:31 ` Josh Cartwright @ 2015-10-27 14:18 ` Eric Dumazet 2015-10-27 15:02 ` Arnaldo Carvalho de Melo 2015-10-30 9:16 ` David Miller 0 siblings, 2 replies; 10+ messages in thread From: Eric Dumazet @ 2015-10-27 14:18 UTC (permalink / raw) To: Josh Cartwright Cc: Paul E. McKenney, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote: > Okay, yes, I like the first suggestion better as well, I've included a > patch below that does just that. I hope you don't mind me turning it > into a Suggested-by :). > > Thanks for taking a look! > Josh > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); > void synchronize_net(void) > { > might_sleep(); > - if (rtnl_is_locked()) > + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) > synchronize_rcu_expedited(); > else > synchronize_rcu(); No objection from me. Thanks. Acked-by: Eric Dumazet <edumazet@google.com> ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 14:18 ` Eric Dumazet @ 2015-10-27 15:02 ` Arnaldo Carvalho de Melo 2015-10-27 15:27 ` Eric Dumazet 2015-10-30 9:16 ` David Miller 1 sibling, 1 reply; 10+ messages in thread From: Arnaldo Carvalho de Melo @ 2015-10-27 15:02 UTC (permalink / raw) To: Eric Dumazet Cc: Josh Cartwright, Paul E. McKenney, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller, Clark Williams Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu: > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote: > > > Okay, yes, I like the first suggestion better as well, I've included a > > patch below that does just that. I hope you don't mind me turning it > > into a Suggested-by :). > > > > Thanks for taking a look! > > Josh > > > > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); > > void synchronize_net(void) > > { > > might_sleep(); > > - if (rtnl_is_locked()) > > + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) > > synchronize_rcu_expedited(); > > else > > synchronize_rcu(); > > No objection from me. Thanks. > > Acked-by: Eric Dumazet <edumazet@google.com> The first suggestion, with it disabled by default seems to be the most flexible tho, i.e, Paul's original message plus the boot parameter line: Alternatively, a boot-time option could be used: int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT; if (rtnl_is_locked() && !some_rt_boot_parameter) synchronize_rcu_expedited(); else synchronize_rcu(); Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT set to 1, while upstream would have this default to 0. RT oriented kernel users could try using this in some scenarios where networking is not the critical path. - Arnaldo ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 15:02 ` Arnaldo Carvalho de Melo @ 2015-10-27 15:27 ` Eric Dumazet 2015-10-27 23:15 ` Paul E. McKenney 0 siblings, 1 reply; 10+ messages in thread From: Eric Dumazet @ 2015-10-27 15:27 UTC (permalink / raw) To: Arnaldo Carvalho de Melo Cc: Josh Cartwright, Paul E. McKenney, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller, Clark Williams On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote: > Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu: > > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote: > > > > > Okay, yes, I like the first suggestion better as well, I've included a > > > patch below that does just that. I hope you don't mind me turning it > > > into a Suggested-by :). > > > > > > Thanks for taking a look! > > > Josh > > > > > > > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); > > > void synchronize_net(void) > > > { > > > might_sleep(); > > > - if (rtnl_is_locked()) > > > + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) > > > synchronize_rcu_expedited(); > > > else > > > synchronize_rcu(); > > > > No objection from me. Thanks. > > > > Acked-by: Eric Dumazet <edumazet@google.com> > > The first suggestion, with it disabled by default seems to be the most > flexible tho, i.e, Paul's original message plus the boot parameter line: > > Alternatively, a boot-time option could be used: > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT; > > if (rtnl_is_locked() && !some_rt_boot_parameter) > synchronize_rcu_expedited(); > else > synchronize_rcu(); > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT > set to 1, while upstream would have this default to 0. > > RT oriented kernel users could try using this in some scenarios where > networking is not the critical path. Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe a generic solution would make synchronize_rcu_expedited() to fallback synchronize_rcu() after boot time on RT. Not sure why networking use of synchronize_rcu_expedited() would be problematic, and not the others. scripts/checkpatch.pl has this comment about this : # Check for expedited grace periods that interrupt non-idle non-nohz # online CPUs. These expedited can therefore degrade real-time response # if used carelessly, and should be avoided where not absolutely # needed. It is always OK to use synchronize_rcu_expedited() and # synchronize_sched_expedited() at boot time (before real-time applications # start) and in error situations where real-time response is compromised in # any case. Note that synchronize_srcu_expedited() does -not- interrupt # other CPUs, so don't warn on uses of synchronize_srcu_expedited(). # Of course, nothing comes for free, and srcu_read_lock() and # srcu_read_unlock() do contain full memory barriers in payment for # synchronize_srcu_expedited() non-interruption properties. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 15:27 ` Eric Dumazet @ 2015-10-27 23:15 ` Paul E. McKenney 2015-10-28 8:34 ` Josh Cartwright 0 siblings, 1 reply; 10+ messages in thread From: Paul E. McKenney @ 2015-10-27 23:15 UTC (permalink / raw) To: Eric Dumazet Cc: Arnaldo Carvalho de Melo, Josh Cartwright, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller, Clark Williams On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote: > On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote: > > Em Tue, Oct 27, 2015 at 07:18:01AM -0700, Eric Dumazet escreveu: > > > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote: > > > > > > > Okay, yes, I like the first suggestion better as well, I've included a > > > > patch below that does just that. I hope you don't mind me turning it > > > > into a Suggested-by :). > > > > > > > > Thanks for taking a look! > > > > Josh > > > > > > > > > > @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); > > > > void synchronize_net(void) > > > > { > > > > might_sleep(); > > > > - if (rtnl_is_locked()) > > > > + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) > > > > synchronize_rcu_expedited(); > > > > else > > > > synchronize_rcu(); > > > > > > No objection from me. Thanks. > > > > > > Acked-by: Eric Dumazet <edumazet@google.com> > > > > The first suggestion, with it disabled by default seems to be the most > > flexible tho, i.e, Paul's original message plus the boot parameter line: > > > > Alternatively, a boot-time option could be used: > > > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT; > > > > if (rtnl_is_locked() && !some_rt_boot_parameter) > > synchronize_rcu_expedited(); > > else > > synchronize_rcu(); This could be OK, but why not start with something very simple and automatic? We can always add more knobs when and if they actually prove necessary. In contrast, unnecessary knobs can cause confusion and might at the same time get locked into some misbegotten userspace application, which would make the unnecessary knob really hard to get rid of. > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT > > set to 1, while upstream would have this default to 0. > > > > RT oriented kernel users could try using this in some scenarios where > > networking is not the critical path. > > Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe > a generic solution would make synchronize_rcu_expedited() to fallback > synchronize_rcu() after boot time on RT. > > Not sure why networking use of synchronize_rcu_expedited() would be > problematic, and not the others. >From what I can see, their testing just happened to run into this one. Perhaps further testing will run into others, or perhaps the others are off in code paths that should not be exercised while running RT apps. I doubt that it is anything personal. ;-) > scripts/checkpatch.pl has this comment about this : > > # Check for expedited grace periods that interrupt non-idle non-nohz > # online CPUs. These expedited can therefore degrade real-time response > # if used carelessly, and should be avoided where not absolutely > # needed. It is always OK to use synchronize_rcu_expedited() and > # synchronize_sched_expedited() at boot time (before real-time applications > # start) and in error situations where real-time response is compromised in > # any case. Note that synchronize_srcu_expedited() does -not- interrupt > # other CPUs, so don't warn on uses of synchronize_srcu_expedited(). > # Of course, nothing comes for free, and srcu_read_lock() and > # srcu_read_unlock() do contain full memory barriers in payment for > # synchronize_srcu_expedited() non-interruption properties. As the author of that paragraph, I hereby declare that synchronize_net()'s use of synchronize_rcu_expedited() clears the high absolutely-needed bar. The point of that paragraph is to get people to think hard about alternatives, for example, call_rcu(). However, as I understand it, call_rcu() would not be particularly helpful in the synchronize_net() case. Thanx, Paul ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 23:15 ` Paul E. McKenney @ 2015-10-28 8:34 ` Josh Cartwright 2015-10-28 12:27 ` Paul E. McKenney 0 siblings, 1 reply; 10+ messages in thread From: Josh Cartwright @ 2015-10-28 8:34 UTC (permalink / raw) To: Paul E. McKenney Cc: Eric Dumazet, Arnaldo Carvalho de Melo, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller, Clark Williams On Tue, Oct 27, 2015 at 04:15:59PM -0700, Paul E. McKenney wrote: > On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote: > > On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote: [..] > > > The first suggestion, with it disabled by default seems to be the most > > > flexible tho, i.e, Paul's original message plus the boot parameter line: > > > > > > Alternatively, a boot-time option could be used: > > > > > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT; > > > > > > if (rtnl_is_locked() && !some_rt_boot_parameter) > > > synchronize_rcu_expedited(); > > > else > > > synchronize_rcu(); > > This could be OK, but why not start with something very simple and automatic? > We can always add more knobs when and if they actually prove necessary. I suppose the question is if, for acme's usecases the answer to "when it's proven necessary" is "now". > In contrast, unnecessary knobs can cause confusion and might at the same time > get locked into some misbegotten userspace application, which would make the > unnecessary knob really hard to get rid of. I think I would make a stronger statement; the CONFIG_SYNC_NET_DEFAULT proposed option would be a boot/compile time parameter which says "I require networking (and network configuration) in my critical path", why don't we have these flags for other I/O subsystems? What's special about networking? We don't because applications can make use of thread priorities to express exactly which tasks should be more important than others. So perhaps the failure here is that RCU (and networking, by implication) doesn't (can't?) take into consideration the calling thread's priority? (And, there may be a cascade of other problems as well, like deferred work pushed to a waitqueue, and thus losing the callers priority, etc) (I will admit that RCU is a black box to me, so it is entirely possible it's already capable of this, or it's fundamentally impossible, or somewhere in between :) > > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT > > > set to 1, while upstream would have this default to 0. > > > > > > RT oriented kernel users could try using this in some scenarios where > > > networking is not the critical path. > > > > Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe > > a generic solution would make synchronize_rcu_expedited() to fallback > > synchronize_rcu() after boot time on RT. > > > > Not sure why networking use of synchronize_rcu_expedited() would be > > problematic, and not the others. > > From what I can see, their testing just happened to run into this one. > Perhaps further testing will run into others, or perhaps the others are > off in code paths that should not be exercised while running RT apps. I accidentally ran into this issue when I was doing testing with an ethernet cable w/ a broken RJ-45 connector (without the tab, that I was just too lazy to replace), and I kept accidentally knocking it out. :) Regardless, industrial automation environments aren't known for having the most stable network environments; there may be deployed systems doing high priority motion control tasks, we'd want to ensure that the poor network technician sent in to repair a defective network switch wouldn't end up being mangled. > > scripts/checkpatch.pl has this comment about this : Also, Documentation/RCU/checklist.txt mentions: Use of the expedited primitives should be restricted to rare configuration-change operations that would not normally be undertaken while a real-time.. I think it could have been argued at the time, that operations under rtnl_lock() were "configuration-change" operations. However, for our use cases, it's not, as link changes are external events beyond control. Thanks again, Josh ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-28 8:34 ` Josh Cartwright @ 2015-10-28 12:27 ` Paul E. McKenney 0 siblings, 0 replies; 10+ messages in thread From: Paul E. McKenney @ 2015-10-28 12:27 UTC (permalink / raw) To: Josh Cartwright Cc: Eric Dumazet, Arnaldo Carvalho de Melo, tglx, bigeasy, linux-rt-users, linux-kernel, David S. Miller, Clark Williams On Wed, Oct 28, 2015 at 03:34:00AM -0500, Josh Cartwright wrote: > On Tue, Oct 27, 2015 at 04:15:59PM -0700, Paul E. McKenney wrote: > > On Tue, Oct 27, 2015 at 08:27:53AM -0700, Eric Dumazet wrote: > > > On Tue, 2015-10-27 at 12:02 -0300, Arnaldo Carvalho de Melo wrote: > [..] > > > > The first suggestion, with it disabled by default seems to be the most > > > > flexible tho, i.e, Paul's original message plus the boot parameter line: > > > > > > > > Alternatively, a boot-time option could be used: > > > > > > > > int some_rt_boot_parameter = CONFIG_SYNC_NET_DEFAULT; > > > > > > > > if (rtnl_is_locked() && !some_rt_boot_parameter) > > > > synchronize_rcu_expedited(); > > > > else > > > > synchronize_rcu(); > > > > This could be OK, but why not start with something very simple and automatic? > > We can always add more knobs when and if they actually prove necessary. > > I suppose the question is if, for acme's usecases the answer to "when > it's proven necessary" is "now". > > > In contrast, unnecessary knobs can cause confusion and might at the same time > > get locked into some misbegotten userspace application, which would make the > > unnecessary knob really hard to get rid of. > > I think I would make a stronger statement; the CONFIG_SYNC_NET_DEFAULT > proposed option would be a boot/compile time parameter which says "I > require networking (and network configuration) in my critical path", why > don't we have these flags for other I/O subsystems? What's special > about networking? > > We don't because applications can make use of thread priorities to > express exactly which tasks should be more important than others. So > perhaps the failure here is that RCU (and networking, by implication) > doesn't (can't?) take into consideration the calling thread's priority? > (And, there may be a cascade of other problems as well, like deferred > work pushed to a waitqueue, and thus losing the callers priority, etc) > > (I will admit that RCU is a black box to me, so it is entirely possible > it's already capable of this, or it's fundamentally impossible, or > somewhere in between :) CONFIG_RCU_KTHREAD_PRIO=nn, where 0 says SCHED_OTHER and 0 < nn <= 99 says SCHED_FIFO with RT priority nn. > > > > Then RT oriented kernel .config files would have CONFIG_SYNC_NET_DEFAULT > > > > set to 1, while upstream would have this default to 0. > > > > > > > > RT oriented kernel users could try using this in some scenarios where > > > > networking is not the critical path. > > > > > > Well, if synchronize_rcu_expedited() is such a problem on RT, then maybe > > > a generic solution would make synchronize_rcu_expedited() to fallback > > > synchronize_rcu() after boot time on RT. > > > > > > Not sure why networking use of synchronize_rcu_expedited() would be > > > problematic, and not the others. > > > > From what I can see, their testing just happened to run into this one. > > Perhaps further testing will run into others, or perhaps the others are > > off in code paths that should not be exercised while running RT apps. > > I accidentally ran into this issue when I was doing testing with an > ethernet cable w/ a broken RJ-45 connector (without the tab, that I was > just too lazy to replace), and I kept accidentally knocking it out. :) > > Regardless, industrial automation environments aren't known for having > the most stable network environments; there may be deployed systems > doing high priority motion control tasks, we'd want to ensure that the > poor network technician sent in to repair a defective network switch > wouldn't end up being mangled. > > > > scripts/checkpatch.pl has this comment about this : > > Also, Documentation/RCU/checklist.txt mentions: > > Use of the expedited primitives should be restricted to rare > configuration-change operations that would not normally be > undertaken while a real-time.. > > I think it could have been argued at the time, that operations under > rtnl_lock() were "configuration-change" operations. However, for our > use cases, it's not, as link changes are external events beyond control. Certainly the variety of operations that people are willing to run concurrently with real-time applications seems to be steadily growing over time... But much depends on the RT deadlines. Thanx, Paul ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" 2015-10-27 14:18 ` Eric Dumazet 2015-10-27 15:02 ` Arnaldo Carvalho de Melo @ 2015-10-30 9:16 ` David Miller 1 sibling, 0 replies; 10+ messages in thread From: David Miller @ 2015-10-30 9:16 UTC (permalink / raw) To: eric.dumazet; +Cc: joshc, paulmck, tglx, bigeasy, linux-rt-users, linux-kernel From: Eric Dumazet <eric.dumazet@gmail.com> Date: Tue, 27 Oct 2015 07:18:01 -0700 > On Tue, 2015-10-27 at 07:31 -0500, Josh Cartwright wrote: > >> Okay, yes, I like the first suggestion better as well, I've included a >> patch below that does just that. I hope you don't mind me turning it >> into a Suggested-by :). >> >> Thanks for taking a look! >> Josh > > >> @@ -6969,7 +6969,7 @@ EXPORT_SYMBOL(free_netdev); >> void synchronize_net(void) >> { >> might_sleep(); >> - if (rtnl_is_locked()) >> + if (rtnl_is_locked() && !IS_ENABLED(CONFIG_PREEMPT_RT_FULL)) >> synchronize_rcu_expedited(); >> else >> synchronize_rcu(); > > No objection from me. Thanks. > > Acked-by: Eric Dumazet <edumazet@google.com> I agree with the sentiment to do the simple thing here first before adding potentially useless knobs. Signed-off-by: David S. Miller <davem@davemloft.net> ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2015-10-30 9:16 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2015-10-26 19:14 [PATCH -rt] Revert "net: use synchronize_rcu_expedited()" Josh Cartwright 2015-10-27 0:44 ` Paul E. McKenney 2015-10-27 12:31 ` Josh Cartwright 2015-10-27 14:18 ` Eric Dumazet 2015-10-27 15:02 ` Arnaldo Carvalho de Melo 2015-10-27 15:27 ` Eric Dumazet 2015-10-27 23:15 ` Paul E. McKenney 2015-10-28 8:34 ` Josh Cartwright 2015-10-28 12:27 ` Paul E. McKenney 2015-10-30 9:16 ` David Miller
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).