MLX driver for host-to-ARM communication

MLX driver for host-to-ARM communication

[Tariq Toukan]

Hi,

I plan to submit a patch series introducing a new minimalist MLX 
host-to-ARM communication driver.

Could you confirm whether this series can be submitted in parallel with 
our regular mlx5 feature patches without conflicting with the 15-patch 
limit?

It is to be placed in its own new directory:
drivers/net/ethernet/mellanox/mlx5/nodnic/

Find more details below [1].

Regards,
Tariq


[1]
net/mlx5: Introduce mlx5_nodnic driver for BF4 management PF

BlueField-4 no longer includes the RSHIM hardware that previous
BlueField generations relied on for host-to-ARM communication. In its
place, the firmware exposes two dedicated management PFs — one facing
the host side and one facing the ARM side — and configures steering
rules to loop traffic between them, enabling host-to-ARM communication
through standard networking interfaces.

Add mlx5_nodnic, a lightweight driver for these management PFs.
NODNIC (No DRAM NIC) is a lightweight interface intended to allow
simple packet transmission for minimalistic drivers with a low memory
footprint. The driver communicates with the firmware through PCI
Vendor Specific Capability (VSC) — reads and writes over PCI config
space, unlike mlx5_core which uses a DMA-based command interface.

Each management PF exposes a single SQ and a single RQ that share
one CQ. The interface configures two MSI-Xs: the data MSI-X fires
on CQEs, driving both TX and RX completions; the event MSI-X notifies
on port state changes, where the firmware brings the port up once both
the host and ARM sides have gone up.

Re: MLX driver for host-to-ARM communication

[Jakub Kicinski]

On Thu, 13 Aug 2026 11:33:23 +0300 Tariq Toukan wrote:
> Hi,
> 
> I plan to submit a patch series introducing a new minimalist MLX 
> host-to-ARM communication driver.
> 
> Could you confirm whether this series can be submitted in parallel with 
> our regular mlx5 feature patches without conflicting with the 15-patch 
> limit?

Not really - is there a vendor that's submitting more patches just
because they have more than one driver that you know of?
Doesn't matter how you slice your code.

If you have concerns about patch volume please me reminded that since
you don't review much your patches get the time penalty for coming
from a non-reviewer. Luckily Ido is reviewing a lot so at least you
don't get the company penalty any more.

> It is to be placed in its own new directory:
> drivers/net/ethernet/mellanox/mlx5/nodnic/
> 
> Find more details below [1].
> 
> Regards,
> Tariq
> 
> 
> [1]
> net/mlx5: Introduce mlx5_nodnic driver for BF4 management PF
> 
> BlueField-4 no longer includes the RSHIM hardware that previous
> BlueField generations relied on for host-to-ARM communication. In its
> place, the firmware exposes two dedicated management PFs — one facing
> the host side and one facing the ARM side — and configures steering
> rules to loop traffic between them, enabling host-to-ARM communication
> through standard networking interfaces.
> 
> Add mlx5_nodnic, a lightweight driver for these management PFs.
> NODNIC (No DRAM NIC) is a lightweight interface intended to allow
> simple packet transmission for minimalistic drivers with a low memory
> footprint. The driver communicates with the firmware through PCI
> Vendor Specific Capability (VSC) — reads and writes over PCI config
> space, unlike mlx5_core which uses a DMA-based command interface.
> 
> Each management PF exposes a single SQ and a single RQ that share
> one CQ. The interface configures two MSI-Xs: the data MSI-X fires
> on CQEs, driving both TX and RX completions; the event MSI-X notifies
> on port state changes, where the firmware brings the port up once both
> the host and ARM sides have gone up.

This will be nacked. We nacked multiple such drivers over the years.
Most recent one not more than a year ago.

If you model the device as a network node it should be reachable via
the normal networking path. If it's a control channel - explicit
control channel APIs should be built, rather than doing pretend
networking.

Every IPU vendor gets to this sort of design sooner or later because
it's an easy hack when running a user space SDK. nVidia has clearly
turned from upstream work to pushing your DOCA stack on all customers.
No matter how many shameful lies y'all told about fwctl not being DOCA
related. This is just a next step. But please commence your bullying
campaign. What do I know, right?

Re: MLX driver for host-to-ARM communication

[Gal Pressman]

Hi Jakub,

On 13/08/2026 18:14, Jakub Kicinski wrote:
> This will be nacked. We nacked multiple such drivers over the years.
> Most recent one not more than a year ago.
> 
> If you model the device as a network node it should be reachable via
> the normal networking path. If it's a control channel - explicit
> control channel APIs should be built, rather than doing pretend
> networking.
> 
> Every IPU vendor gets to this sort of design sooner or later because
> it's an easy hack when running a user space SDK. nVidia has clearly
> turned from upstream work to pushing your DOCA stack on all customers.
> No matter how many shameful lies y'all told about fwctl not being DOCA
> related. This is just a next step. But please commence your bullying
> campaign. What do I know, right?
> 

There are some unrelated accusations here, I'll focus on the technical
side of things.

No DRAM NIC (NODNIC) is not a new interface designed for BlueField 4 or
DOCA. It exists in our NIC since 2015, and implemented in other OSs
drivers long before DOCA existed.

It is a lightweight NIC PF intended for simple packet transmission with
a small memory footprint.
It sends and receives ordinary Ethernet frames using SQs, RQs, CQs, and
EQs. There is no private encapsulation or command protocol. You can
connect this PF to any other NIC on the other side of the wire and it
will work.

On BlueField 4, the firmware connects the host side NODNIC PF to a peer
PF on the DPU side using ordinary vport forwarding rules. The Ethernet
packets are forwarded unchanged, both sides are Linux netdevs.

The purpose of this is to provide network connectivity between the host
and the DPU OS, particularly when the DPU’s OOB port is not connected.
Standard networking applications such as ssh and PXE use it.

To be clear, there is no hidden FW involvement or handling of the
packets, these packets are forwarded as is between both sides.
It makes sense to use a networking driver for networking purposes, I
don't see how control channel APIs are better for this.

Re: MLX driver for host-to-ARM communication

[Jakub Kicinski]

On Tue, 25 Aug 2026 11:24:08 +0300 Gal Pressman wrote:
> On 13/08/2026 18:14, Jakub Kicinski wrote:
> > This will be nacked. We nacked multiple such drivers over the years.
> > Most recent one not more than a year ago.
> > 
> > If you model the device as a network node it should be reachable via
> > the normal networking path. If it's a control channel - explicit
> > control channel APIs should be built, rather than doing pretend
> > networking.
> > 
> > Every IPU vendor gets to this sort of design sooner or later because
> > it's an easy hack when running a user space SDK. nVidia has clearly
> > turned from upstream work to pushing your DOCA stack on all customers.
> > No matter how many shameful lies y'all told about fwctl not being DOCA
> > related. This is just a next step. But please commence your bullying
> > campaign. What do I know, right?
> 
> There are some unrelated accusations here, I'll focus on the technical
> side of things.
> 
> No DRAM NIC (NODNIC) is not a new interface designed for BlueField 4 or
> DOCA. It exists in our NIC since 2015, and implemented in other OSs
> drivers long before DOCA existed.
> 
> It is a lightweight NIC PF intended for simple packet transmission with
> a small memory footprint.
> It sends and receives ordinary Ethernet frames using SQs, RQs, CQs, and
> EQs. There is no private encapsulation or command protocol. You can
> connect this PF to any other NIC on the other side of the wire and it
> will work.
> 
> On BlueField 4, the firmware connects the host side NODNIC PF to a peer
> PF on the DPU side using ordinary vport forwarding rules. The Ethernet
> packets are forwarded unchanged, both sides are Linux netdevs.
> 
> The purpose of this is to provide network connectivity between the host
> and the DPU OS, particularly when the DPU’s OOB port is not connected.
> Standard networking applications such as ssh and PXE use it.
> 
> To be clear, there is no hidden FW involvement or handling of the
> packets, these packets are forwarded as is between both sides.
> It makes sense to use a networking driver for networking purposes, I
> don't see how control channel APIs are better for this.

I understand. Having a full networking driver for such point to point
communication between device FW (/OS running on the device, doesn't
matter) complicates the system topology, config/init and mixes very
non-networking requirements with networking. It's a lazy design which
leads to poor, vendor-specific user experience. The NFP had a similar
thing, it obviously never made it upstream, but I'm very familiar with
the implications.

Re: MLX driver for host-to-ARM communication

[Gal Pressman]

On 27/08/2026 18:15, Jakub Kicinski wrote:
> On Tue, 25 Aug 2026 11:24:08 +0300 Gal Pressman wrote:
>> On 13/08/2026 18:14, Jakub Kicinski wrote:
>>> This will be nacked. We nacked multiple such drivers over the years.
>>> Most recent one not more than a year ago.
>>>
>>> If you model the device as a network node it should be reachable via
>>> the normal networking path. If it's a control channel - explicit
>>> control channel APIs should be built, rather than doing pretend
>>> networking.
>>>
>>> Every IPU vendor gets to this sort of design sooner or later because
>>> it's an easy hack when running a user space SDK. nVidia has clearly
>>> turned from upstream work to pushing your DOCA stack on all customers.
>>> No matter how many shameful lies y'all told about fwctl not being DOCA
>>> related. This is just a next step. But please commence your bullying
>>> campaign. What do I know, right?
>>
>> There are some unrelated accusations here, I'll focus on the technical
>> side of things.
>>
>> No DRAM NIC (NODNIC) is not a new interface designed for BlueField 4 or
>> DOCA. It exists in our NIC since 2015, and implemented in other OSs
>> drivers long before DOCA existed.
>>
>> It is a lightweight NIC PF intended for simple packet transmission with
>> a small memory footprint.
>> It sends and receives ordinary Ethernet frames using SQs, RQs, CQs, and
>> EQs. There is no private encapsulation or command protocol. You can
>> connect this PF to any other NIC on the other side of the wire and it
>> will work.
>>
>> On BlueField 4, the firmware connects the host side NODNIC PF to a peer
>> PF on the DPU side using ordinary vport forwarding rules. The Ethernet
>> packets are forwarded unchanged, both sides are Linux netdevs.
>>
>> The purpose of this is to provide network connectivity between the host
>> and the DPU OS, particularly when the DPU’s OOB port is not connected.
>> Standard networking applications such as ssh and PXE use it.
>>
>> To be clear, there is no hidden FW involvement or handling of the
>> packets, these packets are forwarded as is between both sides.
>> It makes sense to use a networking driver for networking purposes, I
>> don't see how control channel APIs are better for this.
> 
> I understand. Having a full networking driver for such point to point
> communication between device FW (/OS running on the device, doesn't
> matter) complicates the system topology, config/init and mixes very
> non-networking requirements with networking. It's a lazy design which
> leads to poor, vendor-specific user experience. The NFP had a similar
> thing, it obviously never made it upstream, but I'm very familiar with
> the implications.

What is the mix of non-networking requirements?

From the networking stack's perspective, this is pretty much the same as
connecting the DPU's OOB port and communicating through that port's netdev.
Also similar to how virtio-net has been used successfully between
hypervisors and VMs for a long time.

I can't comment on other companies' experience, but this driver is
running successfully in our systems, we do not encounter the problems
you described.

What is the basis for the NACK? A technical problem with the networking
driver itself, or a policy objection against this kind of DPU use case?