Re: [net-next V3 15/15] Documentation: networking: Add description for multi-pf netdev

[Tariq Toukan <ttoukan.linux@gmail.com>]

On 16/02/2024 7:23, Jakub Kicinski wrote:
> On Wed, 14 Feb 2024 19:08:14 -0800 Saeed Mahameed wrote:
>> +The advanced Multi-PF NIC technology enables several CPUs within a multi-socket server to

Hi Jakub,

> 
> There are multiple devlink instances, right?

Right.

> In that case we should call out that there may be more than one.
> 

We are combining the PFs in the netdev level.
I did not focus on the parts that we do not touch.
That's why I didn't mention the sysfs for example, until you asked.

For example, irqns for the two PFs are still reachable as they used to, 
under two distinct paths:
ll /sys/bus/pci/devices/0000\:08\:00.0/msi_irqs/
ll /sys/bus/pci/devices/0000\:09\:00.0/msi_irqs/

>> +Currently the sysfs is kept untouched, letting the netdev sysfs point to its primary PF.
>> +Enhancing sysfs to reflect the actual topology is to be discussed and contributed separately.
> 
> I don't anticipate it to be particularly hard, let's not merge
> half-baked code and force users to grow workarounds that are hard
> to remove.
> 

Changing sysfs to expose queues from multiple PFs under one path might 
be misleading and break backward compatibility. IMO it should come as an 
extension to the existing entries.

Anyway, the interesting info exposed in sysfs is now available through 
the netdev genl.

Now, is this sysfs part integral to the feature? IMO, no. This in-driver 
feature is large enough to be completed in stages and not as a one shot.

> Also could you add examples of how the queue and napis look when listed
> via the netdev genl on these devices?
> 

Sure. Example for a 24-cores system:

$ ./cli.py --spec ../../../Documentation/netlink/specs/netdev.yaml 
--dump queue-get --json '{"ifindex": 5}'
[{'id': 0, 'ifindex': 5, 'napi-id': 539, 'type': 'rx'},
  {'id': 1, 'ifindex': 5, 'napi-id': 540, 'type': 'rx'},
  {'id': 2, 'ifindex': 5, 'napi-id': 541, 'type': 'rx'},
  {'id': 3, 'ifindex': 5, 'napi-id': 542, 'type': 'rx'},
  {'id': 4, 'ifindex': 5, 'napi-id': 543, 'type': 'rx'},
  {'id': 5, 'ifindex': 5, 'napi-id': 544, 'type': 'rx'},
  {'id': 6, 'ifindex': 5, 'napi-id': 545, 'type': 'rx'},
  {'id': 7, 'ifindex': 5, 'napi-id': 546, 'type': 'rx'},
  {'id': 8, 'ifindex': 5, 'napi-id': 547, 'type': 'rx'},
  {'id': 9, 'ifindex': 5, 'napi-id': 548, 'type': 'rx'},
  {'id': 10, 'ifindex': 5, 'napi-id': 549, 'type': 'rx'},
  {'id': 11, 'ifindex': 5, 'napi-id': 550, 'type': 'rx'},
  {'id': 12, 'ifindex': 5, 'napi-id': 551, 'type': 'rx'},
  {'id': 13, 'ifindex': 5, 'napi-id': 552, 'type': 'rx'},
  {'id': 14, 'ifindex': 5, 'napi-id': 553, 'type': 'rx'},
  {'id': 15, 'ifindex': 5, 'napi-id': 554, 'type': 'rx'},
  {'id': 16, 'ifindex': 5, 'napi-id': 555, 'type': 'rx'},
  {'id': 17, 'ifindex': 5, 'napi-id': 556, 'type': 'rx'},
  {'id': 18, 'ifindex': 5, 'napi-id': 557, 'type': 'rx'},
  {'id': 19, 'ifindex': 5, 'napi-id': 558, 'type': 'rx'},
  {'id': 20, 'ifindex': 5, 'napi-id': 559, 'type': 'rx'},
  {'id': 21, 'ifindex': 5, 'napi-id': 560, 'type': 'rx'},
  {'id': 22, 'ifindex': 5, 'napi-id': 561, 'type': 'rx'},
  {'id': 23, 'ifindex': 5, 'napi-id': 562, 'type': 'rx'},
  {'id': 0, 'ifindex': 5, 'napi-id': 539, 'type': 'tx'},
  {'id': 1, 'ifindex': 5, 'napi-id': 540, 'type': 'tx'},
  {'id': 2, 'ifindex': 5, 'napi-id': 541, 'type': 'tx'},
  {'id': 3, 'ifindex': 5, 'napi-id': 542, 'type': 'tx'},
  {'id': 4, 'ifindex': 5, 'napi-id': 543, 'type': 'tx'},
  {'id': 5, 'ifindex': 5, 'napi-id': 544, 'type': 'tx'},
  {'id': 6, 'ifindex': 5, 'napi-id': 545, 'type': 'tx'},
  {'id': 7, 'ifindex': 5, 'napi-id': 546, 'type': 'tx'},
  {'id': 8, 'ifindex': 5, 'napi-id': 547, 'type': 'tx'},
  {'id': 9, 'ifindex': 5, 'napi-id': 548, 'type': 'tx'},
  {'id': 10, 'ifindex': 5, 'napi-id': 549, 'type': 'tx'},
  {'id': 11, 'ifindex': 5, 'napi-id': 550, 'type': 'tx'},
  {'id': 12, 'ifindex': 5, 'napi-id': 551, 'type': 'tx'},
  {'id': 13, 'ifindex': 5, 'napi-id': 552, 'type': 'tx'},
  {'id': 14, 'ifindex': 5, 'napi-id': 553, 'type': 'tx'},
  {'id': 15, 'ifindex': 5, 'napi-id': 554, 'type': 'tx'},
  {'id': 16, 'ifindex': 5, 'napi-id': 555, 'type': 'tx'},
  {'id': 17, 'ifindex': 5, 'napi-id': 556, 'type': 'tx'},
  {'id': 18, 'ifindex': 5, 'napi-id': 557, 'type': 'tx'},
  {'id': 19, 'ifindex': 5, 'napi-id': 558, 'type': 'tx'},
  {'id': 20, 'ifindex': 5, 'napi-id': 559, 'type': 'tx'},
  {'id': 21, 'ifindex': 5, 'napi-id': 560, 'type': 'tx'},
  {'id': 22, 'ifindex': 5, 'napi-id': 561, 'type': 'tx'},
  {'id': 23, 'ifindex': 5, 'napi-id': 562, 'type': 'tx'}]

$ ./cli.py --spec ../../../Documentation/netlink/specs/netdev.yaml 
--dump napi-get --json='{"ifindex": 5}'
[{'id': 562, 'ifindex': 5, 'irq': 84},
  {'id': 561, 'ifindex': 5, 'irq': 83},
  {'id': 560, 'ifindex': 5, 'irq': 82},
  {'id': 559, 'ifindex': 5, 'irq': 81},
  {'id': 558, 'ifindex': 5, 'irq': 80},
  {'id': 557, 'ifindex': 5, 'irq': 79},
  {'id': 556, 'ifindex': 5, 'irq': 78},
  {'id': 555, 'ifindex': 5, 'irq': 77},
  {'id': 554, 'ifindex': 5, 'irq': 76},
  {'id': 553, 'ifindex': 5, 'irq': 75},
  {'id': 552, 'ifindex': 5, 'irq': 74},
  {'id': 551, 'ifindex': 5, 'irq': 73},
  {'id': 550, 'ifindex': 5, 'irq': 72},
  {'id': 549, 'ifindex': 5, 'irq': 71},
  {'id': 548, 'ifindex': 5, 'irq': 70},
  {'id': 547, 'ifindex': 5, 'irq': 69},
  {'id': 546, 'ifindex': 5, 'irq': 68},
  {'id': 545, 'ifindex': 5, 'irq': 67},
  {'id': 544, 'ifindex': 5, 'irq': 66},
  {'id': 543, 'ifindex': 5, 'irq': 65},
  {'id': 542, 'ifindex': 5, 'irq': 64},
  {'id': 541, 'ifindex': 5, 'irq': 63},
  {'id': 540, 'ifindex': 5, 'irq': 39},
  {'id': 539, 'ifindex': 5, 'irq': 36}]