[RFC PATCH 0/9] ipvs network name space (netns) aware

[RFC PATCH 0/9] ipvs network name space (netns) aware

[Hans Schillstrom]

This patch series adds network name space (netns) support to the LVS.

REVISION

This is version 1

OVERVIEW

The patch doesn't remove or add any functionality except for netns.
For users that don't use network name space (netns) this patch is
completely transparent.

No it's possible to run LVS in a Linux container (see lxc-tools)
i.e.  a light weight virtualization. For example it's possible to run
one or several lvs on a real server in their own network name spaces.
>From the LVS point of view it looks like it runs on it's own machine.

IMPLEMENTATION
Basic requirements for netns awareness
 - Global variables has to be moved to dyn. allocated memory.

Most global variables now resides in a struct ipvs { } in netns/ip_vs.h.
What is moved and what is not ?

Some cache aligned locks are still in global, module init params and some debug_level.

Algorithm files they are untouched.

QUESTIONS
Drop rate in ip_vs_ctl per netns or grand total ?
Should more lock variables be moved (or less) ?


PATCH SET
This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010
and [patch v4] ipvs: IPv6 tunnel mode

Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

SUMMARY

 include/net/ip_vs.h                     |  136 ++++---
 include/net/net_namespace.h             |    2 +
 include/net/netns/ip_vs.h               |  112 +++++
 net/netfilter/ipvs/ip_vs_app.c          |   96 +++--
 net/netfilter/ipvs/ip_vs_conn.c         |  296 ++++++++-----
 net/netfilter/ipvs/ip_vs_core.c         |  155 ++++---
 net/netfilter/ipvs/ip_vs_ctl.c          |  771 +++++++++++++++++--------------
 net/netfilter/ipvs/ip_vs_est.c          |  127 +++--
 net/netfilter/ipvs/ip_vs_ftp.c          |   64 ++-
 net/netfilter/ipvs/ip_vs_proto.c        |  108 +++++-
 net/netfilter/ipvs/ip_vs_proto_ah_esp.c |   34 +-
 net/netfilter/ipvs/ip_vs_proto_sctp.c   |  126 +++---
 net/netfilter/ipvs/ip_vs_proto_tcp.c    |  112 +++--
 net/netfilter/ipvs/ip_vs_proto_udp.c    |  107 +++--
 net/netfilter/ipvs/ip_vs_sync.c         |  323 +++++++------
 net/netfilter/ipvs/ip_vs_xmit.c         |   12 +-
 16 files changed, 1588 insertions(+), 993 deletions(-)

The patch will be divided for readability into
 1. include files
 2. app
 3. conn
 4. core
 5. ctl
 6. est
 7. ftp
 8. proto files
 9. sync and xmit

Include files,
A new file added include/net/netns/ip_vs.h containg all netns specific data.
include/net/net_namespce.h, pointer to "struct ipvs"  added.
include/net/ip_vs.h a new struct added, and many prototypes changed.

* ip_vs_core.c
All netns init origins from this file - ip_vs_init()

* ip_vs_conn.c
Lock array for conn table is kept due to performance,
(or am I wrong here ?).
"static struct ip_vs_aligned_lock
__ip_vs_conntbl_lock_array[CT_LOCKARRAY_SIZE] __cacheline_aligned;"

* ip_vs_ctl.c
drop_ rate is still global

TESTING
This patch have been running for a month now with three LVS/machine
one in root name-space and two in other name-space.
Both IPv4 & IPv6 have been tested in all three modes DR/TUN and NAT
Only a limited set of algos have been used (read rr).

Backup have been there all the time and a switch has been performed a couple of times.

There is still some BUG_ON(!net) left  ...

Not tested yet:
 Drop level, DOS,  schedulers, performance ....
 Netns exit after usage of LVS (due to a bug in netdev/ipip somewhere tunl0 and


--
Regards
Hans Schillstrom <hans.schillstrom@ericsson.com>

Re: [RFC PATCH 0/9] ipvs network name space (netns) aware

[Simon Horman]

On Fri, Oct 08, 2010 at 01:16:36PM +0200, Hans Schillstrom wrote:
> This patch series adds network name space (netns) support to the LVS.
> 
> REVISION
> 
> This is version 1
> 
> OVERVIEW
> 
> The patch doesn't remove or add any functionality except for netns.
> For users that don't use network name space (netns) this patch is
> completely transparent.
> 
> No it's possible to run LVS in a Linux container (see lxc-tools)
> i.e.  a light weight virtualization. For example it's possible to run
> one or several lvs on a real server in their own network name spaces.
> >From the LVS point of view it looks like it runs on it's own machine.
> 
> IMPLEMENTATION
> Basic requirements for netns awareness
>  - Global variables has to be moved to dyn. allocated memory.
> 
> Most global variables now resides in a struct ipvs { } in netns/ip_vs.h.
> What is moved and what is not ?
> 
> Some cache aligned locks are still in global, module init params and some debug_level.
> 
> Algorithm files they are untouched.
> 
> QUESTIONS
> Drop rate in ip_vs_ctl per netns or grand total ?

My gut-feeling is that per netns makes more sense.

> Should more lock variables be moved (or less) ?

I'm unsure what you are asking here but I will make a general statement
about locking in IPVS: it needs work.

> 
> PATCH SET
> This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010
> and [patch v4] ipvs: IPv6 tunnel mode
> 
> Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"

Unfortunately the patches don't apply with the persistence engine
patches which were recently merged into nf-next-2.6 (although
"[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).

I'm happy to work with you to make the required changes there.

(I realise those patches weren't merged when you made your post.
 But regardless, either your or me will need to update the patches).

Another issue is that your patches seem to be split in a way
where the build breaks along the way. E.g. after applying
patch 1, the build breaks. Could you please split things up
in a manner such that this doesn't happen. The reason being
that it breaks bisection.

Lastly, could you provide a unique subject for each patch.
I know its a bit tedious, but it does make a difference when
browsing the changelog.

Re: [RFC PATCH 0/9] ipvs network name space (netns) aware

[Hans Schillstrom]

On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
> On Fri, Oct 08, 2010 at 01:16:36PM +0200, Hans Schillstrom wrote:
> > This patch series adds network name space (netns) support to the LVS.
> > 
> > REVISION
> > 
> > This is version 1
> > 
> > OVERVIEW
> > 
> > The patch doesn't remove or add any functionality except for netns.
> > For users that don't use network name space (netns) this patch is
> > completely transparent.
> > 
> > No it's possible to run LVS in a Linux container (see lxc-tools)
> > i.e.  a light weight virtualization. For example it's possible to run
> > one or several lvs on a real server in their own network name spaces.
> > >From the LVS point of view it looks like it runs on it's own machine.
> > 
> > IMPLEMENTATION
> > Basic requirements for netns awareness
> >  - Global variables has to be moved to dyn. allocated memory.
> > 
> > Most global variables now resides in a struct ipvs { } in netns/ip_vs.h.
> > What is moved and what is not ?
> > 
> > Some cache aligned locks are still in global, module init params and some debug_level.
> > 
> > Algorithm files they are untouched.
> > 
> > QUESTIONS
> > Drop rate in ip_vs_ctl per netns or grand total ?

This is a tricky one (I think), 
if the interface is shared with root name-space and/or other name-spaces
 - use grand total
if it's an "own interface"  
 - drop rate can/should be in netns...

> 
> My gut-feeling is that per netns makes more sense.
> 
> > Should more lock variables be moved (or less) ?
> 
> I'm unsure what you are asking here but I will make a general statement
> about locking in IPVS: it needs work.

Some locks still resides as global variables, and others in netns_ipvs struct.
Since you have a lot of experience with IPVS locks, 
you might have ideas what to move and what to not move.

> 
> > 
> > PATCH SET
> > This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010
> > and [patch v4] ipvs: IPv6 tunnel mode
> > 
> > Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"
> 
> Unfortunately the patches don't apply with the persistence engine
> patches which were recently merged into nf-next-2.6 (although
> "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).
> 
I do have a patch based on the nf-next without the SIP/PE patch

> I'm happy to work with you to make the required changes there.

I would appreciate that.

> 
> (I realise those patches weren't merged when you made your post.
>  But regardless, either your or me will need to update the patches).
> 
> Another issue is that your patches seem to be split in a way
> where the build breaks along the way. E.g. after applying
> patch 1, the build breaks. Could you please split things up
> in a manner such that this doesn't happen. The reason being
> that it breaks bisection.
> 
Hmm, Daniel also pointed at this,
The Patch is quite large, and will become even larger with pe and sip.
My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it.
I don't know that might be a stupid ?
It's hard to break it up, making the code reentrant causes changes every where.

Daniel L, had another approach break it into many many tiny patches.

> Lastly, could you provide a unique subject for each patch.
> I know its a bit tedious, but it does make a difference when
> browsing the changelog.
> 
Yepp, no problem

> --
> To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Regards
Hans Schillstrom <hans.schillstrom@ericsson.com>

Re: [RFC PATCH 0/9] ipvs network name space (netns) aware

[Daniel Lezcano]

On 10/18/2010 01:55 PM, Hans Schillstrom wrote:
> On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
>    
>> On Fri, Oct 08, 2010 at 01:16:36PM +0200, Hans Schillstrom wrote:
>>      
>>> This patch series adds network name space (netns) support to the LVS.
>>>
>>> REVISION
>>>
>>> This is version 1
>>>
>>> OVERVIEW
>>>
>>> The patch doesn't remove or add any functionality except for netns.
>>> For users that don't use network name space (netns) this patch is
>>> completely transparent.
>>>
>>> No it's possible to run LVS in a Linux container (see lxc-tools)
>>> i.e.  a light weight virtualization. For example it's possible to run
>>> one or several lvs on a real server in their own network name spaces.
>>> > From the LVS point of view it looks like it runs on it's own machine.
>>>
>>> IMPLEMENTATION
>>> Basic requirements for netns awareness
>>>   - Global variables has to be moved to dyn. allocated memory.
>>>
>>> Most global variables now resides in a struct ipvs { } in netns/ip_vs.h.
>>> What is moved and what is not ?
>>>
>>> Some cache aligned locks are still in global, module init params and some debug_level.
>>>
>>> Algorithm files they are untouched.
>>>
>>> QUESTIONS
>>> Drop rate in ip_vs_ctl per netns or grand total ?
>>>        
> This is a tricky one (I think),
> if the interface is shared with root name-space and/or other name-spaces
>   - use grand total
> if it's an "own interface"
>   - drop rate can/should be in netns...
>
>    
>> My gut-feeling is that per netns makes more sense.
>>
>>      
>>> Should more lock variables be moved (or less) ?
>>>        
>> I'm unsure what you are asking here but I will make a general statement
>> about locking in IPVS: it needs work.
>>      
> Some locks still resides as global variables, and others in netns_ipvs struct.
> Since you have a lot of experience with IPVS locks,
> you might have ideas what to move and what to not move.
>
>    
>>      
>>> PATCH SET
>>> This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010
>>> and [patch v4] ipvs: IPv6 tunnel mode
>>>
>>> Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"
>>>        
>> Unfortunately the patches don't apply with the persistence engine
>> patches which were recently merged into nf-next-2.6 (although
>> "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).
>>
>>      
> I do have a patch based on the nf-next without the SIP/PE patch
>
>    
>> I'm happy to work with you to make the required changes there.
>>      
> I would appreciate that.
>
>    
>> (I realise those patches weren't merged when you made your post.
>>   But regardless, either your or me will need to update the patches).
>>
>> Another issue is that your patches seem to be split in a way
>> where the build breaks along the way. E.g. after applying
>> patch 1, the build breaks. Could you please split things up
>> in a manner such that this doesn't happen. The reason being
>> that it breaks bisection.
>>
>>      
> Hmm, Daniel also pointed at this,
> The Patch is quite large, and will become even larger with pe and sip.
> My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it.
> I don't know that might be a stupid ?
> It's hard to break it up, making the code reentrant causes changes every where.
>
> Daniel L, had another approach break it into many many tiny patches.
>    

These patches can be folded if the result is an acceptable patch size. 
The more important is to not break the init_net. Introducing the 
namespace concept step by step, from the bottom-up in the call stack is 
good because we focus first on making the namespace available in the 
code while keeping the init_net functional. The small patches help to 
git bisect and spot the patch which lead to a regression. As the patches 
are small, it is easy to quickly find the culprit and fix it, especially 
for the people who are not author of these patches.

During this step of namespace introduction, we prevent  to unshare the 
network for ipvs, with a non-op when the namespace is not init_net.

When everything is ready and the init_net is running without problem, we 
remove the non-op and consolidate the namespace part of ipvs.

This is how I am used to do, but that does not imply it is the best way 
to do that :)

>> Lastly, could you provide a unique subject for each patch.
>> I know its a bit tedious, but it does make a difference when
>> browsing the changelog.
>>
>>      
> Yepp, no problem
>
>    
>> --
>> To unsubscribe from this list: send the line "unsubscribe lvs-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>      
>    

Re: [RFC PATCH 0/9] ipvs network name space (netns) aware

[Simon Horman]

On Mon, Oct 18, 2010 at 01:55:58PM +0200, Hans Schillstrom wrote:
> On Sunday 17 October 2010 08:47:31 Simon Horman wrote:
> > On Fri, Oct 08, 2010 at 01:16:36PM +0200, Hans Schillstrom wrote:
> > > This patch series adds network name space (netns) support to the LVS.
> > > 
> > > REVISION
> > > 
> > > This is version 1
> > > 
> > > OVERVIEW
> > > 
> > > The patch doesn't remove or add any functionality except for netns.
> > > For users that don't use network name space (netns) this patch is
> > > completely transparent.
> > > 
> > > No it's possible to run LVS in a Linux container (see lxc-tools)
> > > i.e.  a light weight virtualization. For example it's possible to run
> > > one or several lvs on a real server in their own network name spaces.
> > > >From the LVS point of view it looks like it runs on it's own machine.
> > > 
> > > IMPLEMENTATION
> > > Basic requirements for netns awareness
> > >  - Global variables has to be moved to dyn. allocated memory.
> > > 
> > > Most global variables now resides in a struct ipvs { } in netns/ip_vs.h.
> > > What is moved and what is not ?
> > > 
> > > Some cache aligned locks are still in global, module init params and some debug_level.
> > > 
> > > Algorithm files they are untouched.
> > > 
> > > QUESTIONS
> > > Drop rate in ip_vs_ctl per netns or grand total ?
> 
> This is a tricky one (I think), 
> if the interface is shared with root name-space and/or other name-spaces
>  - use grand total
> if it's an "own interface"  
>  - drop rate can/should be in netns...

I hadn't thought about shared devices - yes that is tricky.

> > My gut-feeling is that per netns makes more sense.
> > 
> > > Should more lock variables be moved (or less) ?
> > 
> > I'm unsure what you are asking here but I will make a general statement
> > about locking in IPVS: it needs work.
> 
> Some locks still resides as global variables, and others in netns_ipvs struct.
> Since you have a lot of experience with IPVS locks, 
> you might have ideas what to move and what to not move.

My basic thought is that locks tend to either related to a connection
or the configuration of a service. And it seems to me that if you
have a per-namespace connection hash table then both of these categories
of locks are good candidates to be made per-namespace.

Do you have any particular locks that you are worried about?

> > > PATCH SET
> > > This patch set is based upon net-next-2.6 (2.6.36-rc3) from 4 oct 2010
> > > and [patch v4] ipvs: IPv6 tunnel mode
> > > 
> > > Note: ip_vs_xmit.c will not work without "[patch v4] ipvs: IPv6 tunnel mode"
> > 
> > Unfortunately the patches don't apply with the persistence engine
> > patches which were recently merged into nf-next-2.6 (although
> > "[patch v4.1 ]ipvs: IPv6 tunnel mode" is still unmerged).
> > 
> I do have a patch based on the nf-next without the SIP/PE patch
> 
> > I'm happy to work with you to make the required changes there.
> 
> I would appreciate that.

No problem. I am a bit busy this week as I am attending the Netfilter
Workshop. But I will try to find some time to rebase your changes soon.

> > (I realise those patches weren't merged when you made your post.
> >  But regardless, either your or me will need to update the patches).
> > 
> > Another issue is that your patches seem to be split in a way
> > where the build breaks along the way. E.g. after applying
> > patch 1, the build breaks. Could you please split things up
> > in a manner such that this doesn't happen. The reason being
> > that it breaks bisection.
> > 
> Hmm, Daniel also pointed at this,
> The Patch is quite large, and will become even larger with pe and sip.
> My Idea was to review the patch in pieces and put it together in one or two large patches when submitting it.
> I don't know that might be a stupid ?
> It's hard to break it up, making the code reentrant causes changes every where.
> 
> Daniel L, had another approach break it into many many tiny patches.

I would prefer the tiny patch approach.

> > Lastly, could you provide a unique subject for each patch.
> > I know its a bit tedious, but it does make a difference when
> > browsing the changelog.
> > 
> Yepp, no problem