Re: [patch net-next RFC 0/4] introduce infrastructure for support of switch chip datapath

[Roopa Prabhu <roopa@cumulusnetworks.com>]

On 3/26/14, 2:31 PM, Jiri Pirko wrote:
> Wed, Mar 26, 2014 at 10:27:05PM CET, roopa@cumulusnetworks.com wrote:
>> On 3/26/14, 11:03 AM, Jiri Pirko wrote:
>>> Wed, Mar 26, 2014 at 06:47:15PM CET, roopa@cumulusnetworks.com wrote:
>>>> On 3/26/14, 9:59 AM, Jiri Pirko wrote:
>>>>> Wed, Mar 26, 2014 at 05:54:17PM CET, roopa@cumulusnetworks.com wrote:
>>>>>> On 3/26/14, 3:54 AM, Jamal Hadi Salim wrote:
>>>>>>> On 03/26/14 01:37, Roopa Prabhu wrote:
>>>>>>>> On 3/25/14, 1:11 PM, Florian Fainelli wrote:
>>>>>>>>> 2014-03-25 12:35 GMT-07:00 Neil Horman <nhorman@tuxdriver.com>:
>>>>>>>> Sorry about getting on this thread late and possibly in the middle.
>>>>>>>> Agree on the idea of keeping the ports linked to the master switch dev
>>>>>>>> (or the 'conduit' to the switch chip) via private list instead of the
>>>>>>>> master-slave relationship proposed earlier.
>>>>>>>> By private i mean the netdev->priv linkage to the master switch dev and
>>>>>>>> not really keeping the ports from being exposed to the user.
>>>>>>>>
>>>>>>>> We think its better to keep the switch ports exposed as any other netdev
>>>>>>>> on linux.
>>>>>>>>   This approach will make the switch ports look exactly like a nic port
>>>>>>>> and all tools will continue to work seamlessly. The switch port
>>>>>>>> operations could internally be forwarded to the switch netdev (sw1 in
>>>>>>>> the above case).
>>>>>>>>
>>>>>>>> example:
>>>>>>>> $ip link set dev sw1p0 up
>>>>>>>> $ethtool -S sw1p0
>>>>>>>>
>>>>>>> I like the approach. I know the above is a simple version, but i am
>>>>>>> assuming you also mean i can do things like
>>>>>>> ip route add ...
>>>>>>> bridge fdb add ... (and if you like your brctl go ahead)
>>>>>>> bonding ...
>>>>>>>
>>>>>> yes, exactly.  We support this model on our boxes today.
>>>>>> User can bond switch ports on our box in the exact same way as he/she
>>>>>> would bond two nic ports.
>>>>>> Our 'conduit to switch chip' reflects the corresponding lag
>>>>>> configuration in the switch chip.
>>>>>> Same goes for bridging, routing, acls.
>>>>> So you implement bonding netlink api? Or you hook into bonding driver
>>>>> itselt? Can you show us the code?
>>>> We use the netlink API and libnl. In our current model, our switch
>>>> chip driver listens to netlink notifications and programs the switch
>>>> chip. The switch chip driver uses libnl caches and libnl netlink apis
>>>> to reflect the kernel state to switch chip.
>>> So when you configure for example bonding over 2 ports, you actually use
>>> bonding driver to do that. And you userspace app listens to
>>> notifications and programs the switch chip accordingly. Am I close?
>> yes correct.
>>> How about data? Is this new "bonding" interface able to assign ip to is
>>> and send/receive packets.
>> yes
>>> I'm still not sure I understand your concept. Do you have some
>>> documentation for it available?
>>>
>> I think the only documentation available today in this area is the
>> user guide and that in-turn points to native linux command manpages
>> iproute2, sysfs, debian ifupdown etc.
>> I will see if i can find anything else.
> I ment the architecture design documentation. linux manpages are not
> that interesting to me :)
>
yes, i get that and thats why i did not include a pointer to our user 
guide. :).
Sorry, the easiest thing to find right now was a high level marketing 
diagram and here you go: 
http://cumulusnetworks.com/product/architecture/. This is nothing but 
what i mentioned in my emails.
 From here the details involve nothing but programming the broadcom 
asic. This is mostly broadcom details/documentation.

The above is our current working/shipping model.

In our second phase of implementation, We wanted to preserve the above 
user interface model (which people using our boxes are very fond of), 
but introduce the concept of a switchdev and switchports in the kernel.
We had a switchdev api in the works ourselves which we were planning to 
publish on netdev until you beat us to it.
Our version is similar to yours but it reflects some of the points that 
i have brought up in my previous emails.
It probably looks more like your v2 (patch 4/6) without the master/slave 
link.
We can share some code in the comming weeks. It does need some cleanup 
and i am also waiting for scott feldman who is on vacation this week.

I know you are looking for specifics, but we don't have switchdev code 
to create a bond in switch chip asic yet. But we have been thinking 
about the details and the current thought there at a high level was, we 
would add a netdev op which the bonding driver could redirect to the 
switchdev driver when it has slaves with IFF_SWITCH_PORT set.

Thanks,
Roopa