Re: IPoIB child interfaces not working with mlx5

[Leon Romanovsky <leon@kernel.org>]

On Tue, Apr 20, 2021 at 11:14:41AM +0200, Jinpu Wang wrote:
> On Mon, Mar 22, 2021 at 7:56 AM Leon Romanovsky <leon@kernel.org> wrote:
> >
> > On Mon, Mar 22, 2021 at 07:08:01AM +0100, Jinpu Wang wrote:
> > > On Sun, Mar 21, 2021 at 2:07 PM Leon Romanovsky <leon@kernel.org> wrote:
> > > >
> > > > On Sat, Mar 20, 2021 at 02:09:50PM +0100, Jack Wang wrote:
> > > > > Leon Romanovsky <leon@kernel.org>于2021年3月20日 周六12:17写道：
> > > > >
> > > > > > On Fri, Mar 19, 2021 at 08:44:29AM +0100, Jinpu Wang wrote:
> > > > > > > Hi Jason and Leon,
> > > > > > >
> > > > > > > We recently switch to use upstream OFED from MLNX-OFED, and we notice
> > > > > > > IPoIB stop working with upstream kernel 5.4.102 with mellanox CX-5
> > > > > > > HCA, it's working fine on CX-2/CX-3. I tested also on 5.11 kernel it
> > > > > > > behaves the same.
> > > > > >
> > > > > > Are you using "enhanced IPoIB" for CX-5 devices? MLX5_CORE_IPOIB?
> > > > > >
> > > > > > Thanks
> > > > >
> > > > >  Yes.
> > > >
> > > > > Is this expected behavor?
> > > >
> > > > Yes, we wanted to make IPoIB behave like any other netdev interfaces and
> > > > if parent interface isn't enabled, no traffic should pass. More on that,
> > > > in our internal implementation of enhanced IPoIB, we are reusing same
> > > > resources for both parent and child, this requires us to wait for "UP"
> > > > event before allowing traffic.
> > > >
> > > > Thanks
> > > Hi Leon,
> > >
> > > Thanks for the clarification, is this behavior documented somewhere?
> > > is it specific to "enhanced IPoIB" for CX-5?
> >
> > It is specific to "enhanced IPoIB" and not to device. I don't know where
> > we can document it.
> >
> > > Will it work differently if without MLX5_CORE_IPOIB enabled?
> >
> > Yes, without MLX5_CORE_IPOIB, the devices will work in "legacy IPoIB",
> > exactly as cx-3. The best thing will be to change IPoIB ULP to behave
> > like netdev, but we were not comfortable to do it back then due to
> > user visible nature of such change.
> >
> Hi Leon,
> 
> More testing reveals new problems with MLX5_CORE_IPOIB.
> w MLX5_CORE_IPOIB, ping wors on both hosts, but iperf3 doens't send any data.

In our regression, iperf3 works.

Let's take it offline.

Thanks