extending ndo_add_rx_vxlan_port

extending ndo_add_rx_vxlan_port

[Or Gerlitz]

Hi,

So with commit 53cf527513eed6e7170e9dceacd198f9267171b0 "vxlan: Notify 
drivers for listening UDP port changes" drivers that have HW offloads 
for vxlan can be notified on which UDP port to listen. Taking this 
further, some HW may need to know the multicast address and/or the VNID 
used by the vxlan instance/s set above them. In that respect, do we 
prefer to extend ndo_add_rx_vxlan_port() or introduce new ndo?

Or.

Re: extending ndo_add_rx_vxlan_port

[Joseph Gasparakis]

On Sun, 27 Oct 2013, Or Gerlitz wrote:

> Hi,
> 
> So with commit 53cf527513eed6e7170e9dceacd198f9267171b0 "vxlan: Notify drivers
> for listening UDP port changes" drivers that have HW offloads for vxlan can be
> notified on which UDP port to listen. Taking this further, some HW may need to
> know the multicast address and/or the VNID used by the vxlan instance/s set
> above them. In that respect, do we prefer to extend ndo_add_rx_vxlan_port() or
> introduce new ndo?
> 
> Or.
> 

The way this patch works is to notify the drivers when a VXLAN UDP port 
comes up or down. This way drivers do not need to do any sort of accounting. As 
long as this remains, it sounds fine to me to extend the existing ndo. If 
by extedning it, drivers now have to keep track of the udp ports so they 
can determine if a notification is for a new port or not, I would much 
rather go for introducing a new ndo.

Joseph

Re: extending ndo_add_rx_vxlan_port

[Or Gerlitz]

On Sun, Oct 27, 2013 at 7:25 PM, Joseph Gasparakis
<joseph.gasparakis@intel.com> wrote:
>
>
> On Sun, 27 Oct 2013, Or Gerlitz wrote:
>
>> Hi,
>>
>> So with commit 53cf527513eed6e7170e9dceacd198f9267171b0 "vxlan: Notify drivers
>> for listening UDP port changes" drivers that have HW offloads for vxlan can be
>> notified on which UDP port to listen. Taking this further, some HW may need to
>> know the multicast address and/or the VNID used by the vxlan instance/s set
>> above them. In that respect, do we prefer to extend ndo_add_rx_vxlan_port() or
>> introduce new ndo?
>>
>> Or.
>>
>
> The way this patch works is to notify the drivers when a VXLAN UDP port
> comes up or down. This way drivers do not need to do any sort of accounting. As

Could you elaborate why do we want to notify all the netdev instances
in the system (on a certain name-space)
that vxlan instance/s are set to listen on certain UDP port and not
only the device over which the
vxlan device is being set? say the HW can listen limited amount of UDP
ports and few vxlan instances are created
one of top of each "real" netdev in the system and each on different
port. Each netdevice will get all callbacks on port addition
and at some point will start to fail adding them into the HW  when the
HW limit is met.

Or.

Re: extending ndo_add_rx_vxlan_port

[John Fastabend]

On 10/27/2013 01:34 PM, Or Gerlitz wrote:
> On Sun, Oct 27, 2013 at 7:25 PM, Joseph Gasparakis
> <joseph.gasparakis@intel.com> wrote:
>>
>>
>> On Sun, 27 Oct 2013, Or Gerlitz wrote:
>>
>>> Hi,
>>>
>>> So with commit 53cf527513eed6e7170e9dceacd198f9267171b0 "vxlan: Notify drivers
>>> for listening UDP port changes" drivers that have HW offloads for vxlan can be
>>> notified on which UDP port to listen. Taking this further, some HW may need to
>>> know the multicast address and/or the VNID used by the vxlan instance/s set
>>> above them. In that respect, do we prefer to extend ndo_add_rx_vxlan_port() or
>>> introduce new ndo?
>>>
>>> Or.
>>>
>>
>> The way this patch works is to notify the drivers when a VXLAN UDP port
>> comes up or down. This way drivers do not need to do any sort of accounting. As
>
> Could you elaborate why do we want to notify all the netdev instances
> in the system (on a certain name-space)
> that vxlan instance/s are set to listen on certain UDP port and not
> only the device over which the
> vxlan device is being set? say the HW can listen limited amount of UDP
> ports and few vxlan instances are created
> one of top of each "real" netdev in the system and each on different
> port. Each netdevice will get all callbacks on port addition
> and at some point will start to fail adding them into the HW  when the
> HW limit is met.
>
> Or.
> --

The issue is how to determine "the device over which the vlan device is
being set". Because the vxlan device is a layer three device we do not
have a 1:1 binding between vxlan port and l2 net_device. In the
mentioned patches the work around for this was to add the UDP port to
every l2 net_device. I agree it is a bit of a big hammer.

It might be better to only add the UDP port to the default route if it
is specified then add the ndo call to the snooping logic (vxlan_snoop)
and the explicit added netlink fdb entries. This would be a bit more
precise than what we have today.

On the other hand the HW table will eventually be full and even with
this optimization may fail. At this point the user has no way to set
which ports are added and which are failed. Joseph, wanted this to
be automatic so users did not have to configure it so we end up with
this case. The other alternative is to provide a hook into the driver
via rtnetlink I'm not sure its worth the effort though. With this
and a feature flag you could let users manage the device manually.

But is this a problem in practice? My feeling is typically the
number of UDP ports in use is smaller than the number the HW supports.

Thanks,
.John


-- 
John Fastabend         Intel Corporation

Re: extending ndo_add_rx_vxlan_port

[Joseph Gasparakis]

On Sun, 27 Oct 2013, Or Gerlitz wrote:

> On Sun, Oct 27, 2013 at 7:25 PM, Joseph Gasparakis
> <joseph.gasparakis@intel.com> wrote:
> >
> >
> > On Sun, 27 Oct 2013, Or Gerlitz wrote:
> >
> >> Hi,
> >>
> >> So with commit 53cf527513eed6e7170e9dceacd198f9267171b0 "vxlan: Notify drivers
> >> for listening UDP port changes" drivers that have HW offloads for vxlan can be
> >> notified on which UDP port to listen. Taking this further, some HW may need to
> >> know the multicast address and/or the VNID used by the vxlan instance/s set
> >> above them. In that respect, do we prefer to extend ndo_add_rx_vxlan_port() or
> >> introduce new ndo?
> >>
> >> Or.
> >>
> >
> > The way this patch works is to notify the drivers when a VXLAN UDP port
> > comes up or down. This way drivers do not need to do any sort of accounting. As
> 
> Could you elaborate why do we want to notify all the netdev instances
> in the system (on a certain name-space)
> that vxlan instance/s are set to listen on certain UDP port and not
> only the device over which the
> vxlan device is being set? say the HW can listen limited amount of UDP
> ports and few vxlan instances are created
> one of top of each "real" netdev in the system and each on different
> port. Each netdevice will get all callbacks on port addition
> and at some point will start to fail adding them into the HW  when the
> HW limit is met.
> 
> Or.
> 
VXLAN implementation is not done like VLAN. VLANs are stacked on top of 
real interfaces and what you are saying makes sense. VXLAN however is 
using ip[6]_tunnel_xmit, and this is why we need to notify all the 
netdevs in the system that implement the add/del vxlan port ndo (i.e. are 
capable of offloading inner csums). In effect all physical "real" netdevs 
are candidates to receive/transmit VXLAN traffic, subject to routing 
tables changing or not.

Joseph

Re: extending ndo_add_rx_vxlan_port

[Or Gerlitz]

On Mon, Oct 28, 2013 at 1:44 AM, Joseph Gasparakis wrote:

> VXLAN implementation is not done like VLAN. VLANs are stacked on top of
> real interfaces and what you are saying makes sense. VXLAN however is
> using ip[6]_tunnel_xmit, and this is why we need to notify all the

As the name of the ndo you added ndo_add_rx_vxlan_port suggests -- HW
needs that for RX offloads. So basically, what I am saying is: in a
similar manner that we already program the NIC "over which" the vxlan
instance is set to listen on the multicast address which is associated
with that vxlan segement, lets give them a hint that packets arriving
on this group are vxlan ones, so they can use it for programming
steering rules.


> netdevs in the system that implement the add/del vxlan port ndo (i.e. are
> capable of offloading inner csums). In effect all physical "real" netdevs
> are candidates to receive/transmit VXLAN traffic, subject to routing
> tables changing or not.

Re: extending ndo_add_rx_vxlan_port

[Joseph Gasparakis]

On Mon, 28 Oct 2013, Or Gerlitz wrote:

> On Mon, Oct 28, 2013 at 1:44 AM, Joseph Gasparakis wrote:
> 
> > VXLAN implementation is not done like VLAN. VLANs are stacked on top of
> > real interfaces and what you are saying makes sense. VXLAN however is
> > using ip[6]_tunnel_xmit, and this is why we need to notify all the
> 
> As the name of the ndo you added ndo_add_rx_vxlan_port suggests -- HW
> needs that for RX offloads. So basically, what I am saying is: in a
> similar manner that we already program the NIC "over which" the vxlan
> instance is set to listen on the multicast address which is associated
> with that vxlan segement, lets give them a hint that packets arriving
> on this group are vxlan ones, so they can use it for programming
> steering rules.

Why don't you write up some code so everyone has something to look at?
Then we can see what makes sense to do in terms of the existing or new 
ndos.

> 
> 
> > netdevs in the system that implement the add/del vxlan port ndo (i.e. are
> > capable of offloading inner csums). In effect all physical "real" netdevs
> > are candidates to receive/transmit VXLAN traffic, subject to routing
> > tables changing or not.
> 

Re: extending ndo_add_rx_vxlan_port

[Or Gerlitz]

On Mon, Oct 28, 2013 at 7:59 PM, Joseph Gasparakis
> On Mon, 28 Oct 2013, Or Gerlitz wrote:
>> On Mon, Oct 28, 2013 at 1:44 AM, Joseph Gasparakis wrote:

>> > VXLAN implementation is not done like VLAN. VLANs are stacked on top of
>> > real interfaces and what you are saying makes sense. VXLAN however is
>> > using ip[6]_tunnel_xmit, and this is why we need to notify all the
>>
>> As the name of the ndo you added ndo_add_rx_vxlan_port suggests -- HW
>> needs that for RX offloads. So basically, what I am saying is: in a
>> similar manner that we already program the NIC "over which" the vxlan
>> instance is set to listen on the multicast address which is associated
>> with that vxlan segement, lets give them a hint that packets arriving
>> on this group are vxlan ones, so they can use it for programming
>> steering rules.
>
> Why don't you write up some code so everyone has something to look at?
> Then we can see what makes sense to do in terms of the existing or new ndos.

sure, code talks, indeed, still, looking on net-next, for the current
ndo there's no in tree consumer unless I miss anything, did I?

Or.

Re: extending ndo_add_rx_vxlan_port

[Joseph Gasparakis]

On Mon, 28 Oct 2013, Or Gerlitz wrote:

> On Mon, Oct 28, 2013 at 7:59 PM, Joseph Gasparakis
> > On Mon, 28 Oct 2013, Or Gerlitz wrote:
> >> On Mon, Oct 28, 2013 at 1:44 AM, Joseph Gasparakis wrote:
> 
> >> > VXLAN implementation is not done like VLAN. VLANs are stacked on top of
> >> > real interfaces and what you are saying makes sense. VXLAN however is
> >> > using ip[6]_tunnel_xmit, and this is why we need to notify all the
> >>
> >> As the name of the ndo you added ndo_add_rx_vxlan_port suggests -- HW
> >> needs that for RX offloads. So basically, what I am saying is: in a
> >> similar manner that we already program the NIC "over which" the vxlan
> >> instance is set to listen on the multicast address which is associated
> >> with that vxlan segement, lets give them a hint that packets arriving
> >> on this group are vxlan ones, so they can use it for programming
> >> steering rules.
> >
> > Why don't you write up some code so everyone has something to look at?
> > Then we can see what makes sense to do in terms of the existing or new ndos.
> 
> sure, code talks, indeed, still, looking on net-next, for the current
> ndo there's no in tree consumer unless I miss anything, did I?
> 
> Or.
> 

I don't think you missed anything. My patches for the i40e are currently 
in our trees and making their way to netdev. It will be very soon.

Re: extending ndo_add_rx_vxlan_port

[Joseph Gasparakis]

On Mon, 28 Oct 2013, Joseph Gasparakis wrote:

> 
> 
> On Mon, 28 Oct 2013, Or Gerlitz wrote:
> 
> > On Mon, Oct 28, 2013 at 7:59 PM, Joseph Gasparakis
> > > On Mon, 28 Oct 2013, Or Gerlitz wrote:
> > >> On Mon, Oct 28, 2013 at 1:44 AM, Joseph Gasparakis wrote:
> > 
> > >> > VXLAN implementation is not done like VLAN. VLANs are stacked on top of
> > >> > real interfaces and what you are saying makes sense. VXLAN however is
> > >> > using ip[6]_tunnel_xmit, and this is why we need to notify all the
> > >>
> > >> As the name of the ndo you added ndo_add_rx_vxlan_port suggests -- HW
> > >> needs that for RX offloads. So basically, what I am saying is: in a
> > >> similar manner that we already program the NIC "over which" the vxlan
> > >> instance is set to listen on the multicast address which is associated
> > >> with that vxlan segement, lets give them a hint that packets arriving
> > >> on this group are vxlan ones, so they can use it for programming
> > >> steering rules.
> > >
> > > Why don't you write up some code so everyone has something to look at?
> > > Then we can see what makes sense to do in terms of the existing or new ndos.
> > 
> > sure, code talks, indeed, still, looking on net-next, for the current
> > ndo there's no in tree consumer unless I miss anything, did I?
> > 
> > Or.
> > 
> 
> I don't think you missed anything. My patches for the i40e are currently 
> in our trees and making their way to netdev. It will be very soon.
> 
> 

Oh! And I also forgot to mention that if you want to see how it 
will be used, together with the patch mentioned above that introduces 
the ndos, I had added an RFC patch against ixgbe that showcases how the 
driver could use them. The RFC patch was just loging the info, but the 
actual talking to the HW would be vendor specific anyway.