Integration of Open vSwitch

Integration of Open vSwitch

[Jesse Gross]

Hi Herbert and Jamal (and everyone else),

Sorry about starting yet another thread but the other one went in so
many directions that I think a lot of things got lost in it.  As I
mentioned before, I'd like to have a bit of a design discussion of
what it would look like if Open vSwitch were to use some of the
existing components (and really focus on just that).  There were a
number of suggestions made about using parts of the bridge, tc,
netfilter, etc. and some of them overlap or conflict so I don't quite
have a coherent solution in mind.  Would you guys mind walking through
what each of you envision it looking like?

Thanks,
Jesse

Re: Integration of Open vSwitch

[Herbert Xu]

On Tue, Nov 29, 2011 at 10:25:34PM -0800, Jesse Gross wrote:
> Hi Herbert and Jamal (and everyone else),
> 
> Sorry about starting yet another thread but the other one went in so
> many directions that I think a lot of things got lost in it.  As I
> mentioned before, I'd like to have a bit of a design discussion of
> what it would look like if Open vSwitch were to use some of the
> existing components (and really focus on just that).  There were a
> number of suggestions made about using parts of the bridge, tc,
> netfilter, etc. and some of them overlap or conflict so I don't quite
> have a coherent solution in mind.  Would you guys mind walking through
> what each of you envision it looking like?

Personally I think your patches are fine as is.

It would obviously be nice if we could refactor the code as Jamal
suggested into classifiers/actions, which would allow us to reuse
them elsewhere, e.g., the flow cache classifier could be merged
with the GRO mechanism, or something even grander like the unified
flow cache, while the using standard actions would make all
existing actions available and generalise OVS into something
that allows you to direct traffic at will to any destination
in a system, without having to have a data-path object at all.

But really I don't see immediate gains that are big enough to
warrant any actions in that direction right now.  If we really
wanted to do that in future we can always add those classifiers
and actions and migrate things over.

The other factor I considered is scalability.  The OVS code as is
is not really friendly to SMP/NUMA scalability (but as Eric pointed,
neither is the classifier/action layer).  However, if this were to
become a problem in future I'm sure we could extend either the
interface as is (e.g., deploying multiqueue netlink sockets), or
migrate to something else.

So I don't really have any objections to this going into the tree.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Integration of Open vSwitch

[jamal]

On Wed, 2011-11-30 at 15:00 +0800, Herbert Xu wrote:


> The other factor I considered is scalability.  The OVS code as is
> is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> neither is the classifier/action layer). 

Did you mean the qdisc funnel-to-interface part which Eric is hopefully
going to work on now that bql is in? classifier/action happens way
before that.

cheers,
jamal

Re: Integration of Open vSwitch

[Eric Dumazet]

Le mercredi 30 novembre 2011 à 08:14 -0500, jamal a écrit :
> On Wed, 2011-11-30 at 15:00 +0800, Herbert Xu wrote:
> 
> 
> > The other factor I considered is scalability.  The OVS code as is
> > is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> > neither is the classifier/action layer). 
> 
> Did you mean the qdisc funnel-to-interface part which Eric is hopefully
> going to work on now that bql is in? classifier/action happens way
> before that.

Actually I am waiting for John work on this area, before wasting my
time ;)



_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: Integration of Open vSwitch

[John Fastabend]

On 11/30/2011 5:21 AM, Eric Dumazet wrote:
> Le mercredi 30 novembre 2011 à 08:14 -0500, jamal a écrit :
>> On Wed, 2011-11-30 at 15:00 +0800, Herbert Xu wrote:
>>
>>
>>> The other factor I considered is scalability.  The OVS code as is
>>> is not really friendly to SMP/NUMA scalability (but as Eric pointed,
>>> neither is the classifier/action layer). 
>>
>> Did you mean the qdisc funnel-to-interface part which Eric is hopefully
>> going to work on now that bql is in? classifier/action happens way
>> before that.
> 
> Actually I am waiting for John work on this area, before wasting my
> time ;)
> 

I'll try to get these out in the next couple days. Got a few qdiscs
running without the qdisc lock today. After I get some real numbers
and make them at least a bit more presentable I'll post them so
folks can let me know if they think its the right approach.

Thanks,
John.







_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: Integration of Open vSwitch

[Herbert Xu]

On Wed, Nov 30, 2011 at 08:14:51AM -0500, jamal wrote:
> On Wed, 2011-11-30 at 15:00 +0800, Herbert Xu wrote:
> 
> 
> > The other factor I considered is scalability.  The OVS code as is
> > is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> > neither is the classifier/action layer). 
> 
> Did you mean the qdisc funnel-to-interface part which Eric is hopefully
> going to work on now that bql is in? classifier/action happens way
> before that.

I actually meant the scalability with adding/deleting flows,
or in the case with qdiscs, filters.

OVS is quite scalable on the fast-path, but when it comes to
new flows, you have a bottleneck.

Cheers,
-- 
Email: Herbert Xu <herbert-lOAM2aK0SrRLBo1qDEOMRrpzq4S04n8Q@public.gmane.org>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Integration of Open vSwitch

[jamal]

On Wed, 2011-11-30 at 21:40 +0800, Herbert Xu wrote:

> I actually meant the scalability with adding/deleting flows,
> or in the case with qdiscs, filters.

Ah yes, them locks. Unless you start doing per-cpu tables
with lazy synchronization i am not sure how to avoid that.
Note: last time (time flies) i compared against iptables, the tc flow
setup/teardown was pretty consistent regardless of table size
relatively speaking.

cheers,
jamal

Re: Integration of Open vSwitch

[Herbert Xu]

On Wed, Nov 30, 2011 at 09:01:55AM -0500, jamal wrote:
>
> Ah yes, them locks. Unless you start doing per-cpu tables
> with lazy synchronization i am not sure how to avoid that.
> Note: last time (time flies) i compared against iptables, the tc flow
> setup/teardown was pretty consistent regardless of table size
> relatively speaking.

iptables sets a pretty low bar :)

For a flow cache I think going per-cpu or at least per-node will
be unavoidable.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Integration of Open vSwitch

[Simon Horman]

On Wed, Nov 30, 2011 at 09:40:28PM +0800, Herbert Xu wrote:
> On Wed, Nov 30, 2011 at 08:14:51AM -0500, jamal wrote:
> > On Wed, 2011-11-30 at 15:00 +0800, Herbert Xu wrote:
> > 
> > 
> > > The other factor I considered is scalability.  The OVS code as is
> > > is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> > > neither is the classifier/action layer). 
> > 
> > Did you mean the qdisc funnel-to-interface part which Eric is hopefully
> > going to work on now that bql is in? classifier/action happens way
> > before that.
> 
> I actually meant the scalability with adding/deleting flows,
> or in the case with qdiscs, filters.
> 
> OVS is quite scalable on the fast-path, but when it comes to
> new flows, you have a bottleneck.

Having done some testing of OVS in relation to both the
number of flows I can stuff into the system and how fast I can do so,
I agree.

As it stands I don't believe that there has been a whole lot
of work so far on optimising for high rates of new flows. I say
this based on a low-hang optimisations I have found (and discussed
on the OVS dev list).

Obviously at some point the low-hanging fruit can all be picked and
making things faster becomes more difficult. And it seems to me
that the user-space controller is only single threaded and that
is a significant bottle-neck to scaling flow creation beyond some point.

But I also suspect there are lot of use-cases for OVS where the
current code scales sufficiently.

Re: Integration of Open vSwitch

[Simon Horman]

On Wed, Nov 30, 2011 at 03:00:11PM +0800, Herbert Xu wrote:
> On Tue, Nov 29, 2011 at 10:25:34PM -0800, Jesse Gross wrote:
> > Hi Herbert and Jamal (and everyone else),
> > 
> > Sorry about starting yet another thread but the other one went in so
> > many directions that I think a lot of things got lost in it.  As I
> > mentioned before, I'd like to have a bit of a design discussion of
> > what it would look like if Open vSwitch were to use some of the
> > existing components (and really focus on just that).  There were a
> > number of suggestions made about using parts of the bridge, tc,
> > netfilter, etc. and some of them overlap or conflict so I don't quite
> > have a coherent solution in mind.  Would you guys mind walking through
> > what each of you envision it looking like?
> 
> Personally I think your patches are fine as is.
> 
> It would obviously be nice if we could refactor the code as Jamal
> suggested into classifiers/actions, which would allow us to reuse
> them elsewhere, e.g., the flow cache classifier could be merged
> with the GRO mechanism, or something even grander like the unified
> flow cache, while the using standard actions would make all
> existing actions available and generalise OVS into something
> that allows you to direct traffic at will to any destination
> in a system, without having to have a data-path object at all.
> 
> But really I don't see immediate gains that are big enough to
> warrant any actions in that direction right now.  If we really
> wanted to do that in future we can always add those classifiers
> and actions and migrate things over.
> 
> The other factor I considered is scalability.  The OVS code as is
> is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> neither is the classifier/action layer).  However, if this were to
> become a problem in future I'm sure we could extend either the
> interface as is (e.g., deploying multiqueue netlink sockets), or
> migrate to something else.
> 
> So I don't really have any objections to this going into the tree.

I apologies for being rather quiet in this discussion up until now,
the only reason for which being that I have been rather busy with other things.

I have now made a few comments on some other posts with regards to
scaling of flow counts and creation rates as I have done some work
in that area recently.

But as I have been involved in Open vSwitch for a while now I feel it would
remiss of me to not (finally) publicly state that I am supportive of the
current merge effort.

Re: Integration of Open vSwitch

[Jesse Gross]

On Tue, Nov 29, 2011 at 11:00 PM, Herbert Xu
<herbert@gondor.apana.org.au> wrote:
> The other factor I considered is scalability.  The OVS code as is
> is not really friendly to SMP/NUMA scalability (but as Eric pointed,
> neither is the classifier/action layer).  However, if this were to
> become a problem in future I'm sure we could extend either the
> interface as is (e.g., deploying multiqueue netlink sockets), or
> migrate to something else.

I agree that adding new flows is a weakness of the OVS model
(obviously it was a trade off) that has some room to be improved.
But, as you say, that's more of a long-term project that I think can
be done incrementally over time.

> So I don't really have any objections to this going into the tree.

Great, thanks for reviewing.

Re: Integration of Open vSwitch

[jamal]

On Tue, 2011-11-29 at 22:25 -0800, Jesse Gross wrote:
> Hi Herbert and Jamal (and everyone else),
> 
> Sorry about starting yet another thread but the other one went in so
> many directions that I think a lot of things got lost in it.  

Good idea ;->

> As I
> mentioned before, I'd like to have a bit of a design discussion of
> what it would look like if Open vSwitch were to use some of the
> existing components (and really focus on just that).  There were a
> number of suggestions made about using parts of the bridge, tc,
> netfilter, etc. and some of them overlap or conflict so I don't quite
> have a coherent solution in mind.  Would you guys mind walking through
> what each of you envision it looking like?

I'll try my TL;DR version:
My opinion is the classifier action code needs refactoring. I have no
doubt that it goes in as is it will eventually look like the one
we already have. From the evolution of that code i can already see 
that is where it is going. I really dont see need to have two competing
interfaces in that aspect. If you decide to go that way I will be
happy to help review and make suggestions.

My !TL;DR version will include details on other things like:
- bridging, the scalability of kernel->user packet redirect etc.
But youve seen some of those views.

cheers,
jamal

Re: Integration of Open vSwitch

[Jesse Gross]

On Wed, Nov 30, 2011 at 5:11 AM, jamal <hadi@cyberus.ca> wrote:
> On Tue, 2011-11-29 at 22:25 -0800, Jesse Gross wrote:
>> As I
>> mentioned before, I'd like to have a bit of a design discussion of
>> what it would look like if Open vSwitch were to use some of the
>> existing components (and really focus on just that).  There were a
>> number of suggestions made about using parts of the bridge, tc,
>> netfilter, etc. and some of them overlap or conflict so I don't quite
>> have a coherent solution in mind.  Would you guys mind walking through
>> what each of you envision it looking like?
>
> I'll try my TL;DR version:
> My opinion is the classifier action code needs refactoring. I have no
> doubt that it goes in as is it will eventually look like the one
> we already have. From the evolution of that code i can already see
> that is where it is going. I really dont see need to have two competing
> interfaces in that aspect. If you decide to go that way I will be
> happy to help review and make suggestions.

I completely agree with the desire to share code where there's overlap
and it makes sense (I was actually just working on some refactoring to
increase code reuse before this).

I think one of the key things to focus on is the userspace/kernel
interface since we'll have much less opportunity to significantly
change that over time.  Getting both compatibility and performance is
something that we've worked on a fair amount and have arrived at a
solution that meets the needs of OVS (and probably only OVS) pretty
well.  I think it's a nice model but keeping that while refactoring on
top of the tc layer seems challenging to do cleanly because the flow
information is needed by both the flow lookup and userspace
communication.  Stringing that through generic code seems fairly
unappealing.

So I agree with you in principle and if you are right that things tend
to converge there should be more room for code reuse over time.  Right
now though I would just focus on the high level bits.
_______________________________________________
dev mailing list
dev@openvswitch.org
http://openvswitch.org/mailman/listinfo/dev

Re: Integration of Open vSwitch

[Jamal Hadi Salim]

Jesse,

I empathize with effort youve put in, i really do; youve
already created the messaging from user space to kernel
and dammit it works; however, i  dont agree with your reasoning. 
The classifier action code is _exactly_ the same infrastructure.
The user space API/messaging already exists. If there are actions
or a classifier missing - add it.
I wish i had the time to re-write the pieces i think need
changing. If the plan is to migrate to the current tc action
code, I will be fine with it - but you are not even suggesting
that.

I think i will punt this to DaveM to make the call.

cheers,
jamal

On Fri, 2011-12-02 at 15:30 -0800, Jesse Gross wrote:

> 
> I completely agree with the desire to share code where there's overlap
> and it makes sense (I was actually just working on some refactoring to
> increase code reuse before this).
> 
> I think one of the key things to focus on is the userspace/kernel
> interface since we'll have much less opportunity to significantly
> change that over time.  Getting both compatibility and performance is
> something that we've worked on a fair amount and have arrived at a
> solution that meets the needs of OVS (and probably only OVS) pretty
> well.  I think it's a nice model but keeping that while refactoring on
> top of the tc layer seems challenging to do cleanly because the flow
> information is needed by both the flow lookup and userspace
> communication.  Stringing that through generic code seems fairly
> unappealing.
>
> So I agree with you in principle and if you are right that things tend
> to converge there should be more room for code reuse over time.  Right
> now though I would just focus on the high level bits.