Driver and H/W APIs Workshop at netdevconf

Driver and H/W APIs Workshop at netdevconf

[David Ahern]

Alex Duyck and I are co-chairing the "Driver and H/W APIs Workshop" at
netdevconf in July.

The workshop is a forum for discussing issues related to driver
development and APIs (user and kernel) for configuring, monitoring and
debugging hardware. Discussion will be open to anyone to present, though
speakers will need to submit topics ahead of time.

Suggested topics based on recent netdev threads include
- devlink - extensions, shortcomings, ...
- extension to memory pools
- new APIs for managing queues
- challenges of netdev / IB co-existence (e.g., driven by AI workloads)
- fwctl - a proposal for direct firmware access


Please let us know if you have a topic that you would like to discuss
along with a time estimate.

Thanks,

Re: Driver and H/W APIs Workshop at netdevconf

[Jakub Kicinski]

On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:
> Suggested topics based on recent netdev threads include
> - devlink - extensions, shortcomings, ...
> - extension to memory pools
> - new APIs for managing queues
> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> - fwctl - a proposal for direct firmware access

Memory pools and queue API are more of stack features.
Please leave them out of your fwctl session.

Aren't people who are actually working on those things submitting
talks or hosting better scoped discussions? It appears you haven't 
CCed any of them..

Re: Driver and H/W APIs Workshop at netdevconf

[David Ahern]

On 5/6/24 7:06 PM, Jakub Kicinski wrote:
> On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:
>> Suggested topics based on recent netdev threads include
>> - devlink - extensions, shortcomings, ...
>> - extension to memory pools
>> - new APIs for managing queues
>> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
>> - fwctl - a proposal for direct firmware access
> 
> Memory pools and queue API are more of stack features.

That require driver support, no? e.g., There is no way that queue API is
going to work with the Enfabrica device without driver support.

The point of the above is a list to motivate discussion based on recent
topics.


> Please leave them out of your fwctl session.

fwctl is a discussion item not tied to anything else; let's not conflat
topics here. That it is even on this list is because you brought netdev
into a discussion that is not netdev related. Given that, let's give it
proper daylight any topic deserves without undue bias and letting it
dominate the bigger picture.

> 
> Aren't people who are actually working on those things submitting
> talks or hosting better scoped discussions? It appears you haven't 
> CCed any of them..

I have no idea. I started with a list of well known driver contacts and
cc'ed netdev with an explicit statement that it is open to all.

Re: Driver and H/W APIs Workshop at netdevconf

[Jakub Kicinski]

On Mon, 6 May 2024 20:05:36 -0600 David Ahern wrote:
> On 5/6/24 7:06 PM, Jakub Kicinski wrote:
> > On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:  
> >> Suggested topics based on recent netdev threads include
> >> - devlink - extensions, shortcomings, ...
> >> - extension to memory pools
> >> - new APIs for managing queues
> >> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> >> - fwctl - a proposal for direct firmware access  
> > 
> > Memory pools and queue API are more of stack features.  
> 
> That require driver support, no? e.g., There is no way that queue API is
> going to work with the Enfabrica device without driver support.
> 
> The point of the above is a list to motivate discussion based on recent
> topics.

What the point of the above list is is pretty transparent.

> > Please leave them out of your fwctl session.  
> 
> fwctl is a discussion item not tied to anything else; let's not conflat
> topics here. That it is even on this list is because you brought netdev
> into a discussion that is not netdev related. Given that, let's give it
> proper daylight any topic deserves without undue bias and letting it
> dominate the bigger picture.

Obviously no objection to discussions about fwctl or whatever else 
you want. I am looking forward to you presenting about it "without
undue bias".

The other topics deserve their own sessions.

> > Aren't people who are actually working on those things submitting
> > talks or hosting better scoped discussions? It appears you haven't 
> > CCed any of them..  
> 
> I have no idea. I started with a list of well known driver contacts and
> cc'ed netdev with an explicit statement that it is open to all.

Re: Driver and H/W APIs Workshop at netdevconf

[Andy Gospodarek]

On Mon, May 06, 2024 at 08:05:36PM -0600, David Ahern wrote:
> On 5/6/24 7:06 PM, Jakub Kicinski wrote:
> > On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:
> >> Suggested topics based on recent netdev threads include
> >> - devlink - extensions, shortcomings, ...
> >> - extension to memory pools
> >> - new APIs for managing queues
> >> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> >> - fwctl - a proposal for direct firmware access
> > 
> > Memory pools and queue API are more of stack features.
> 
> That require driver support, no? e.g., There is no way that queue API is
> going to work with the Enfabrica device without driver support.

I defintely think that there should be a consumer of the queue API
before it lands upstream, but if it cannot work without a
driver/hardware support that seems a bit odd.

Maybe I've missed it on the list, but do you have something you can
share about this proposed queue API?

> The point of the above is a list to motivate discussion based on recent
> topics.
> 
> 
> > Please leave them out of your fwctl session.
> 
> fwctl is a discussion item not tied to anything else; let's not conflat
> topics here. That it is even on this list is because you brought netdev
> into a discussion that is not netdev related. Given that, let's give it
> proper daylight any topic deserves without undue bias and letting it
> dominate the bigger picture.
> 
> > 
> > Aren't people who are actually working on those things submitting
> > talks or hosting better scoped discussions? It appears you haven't 
> > CCed any of them..
> 
> I have no idea. I started with a list of well known driver contacts and
> cc'ed netdev with an explicit statement that it is open to all.

Re: Driver and H/W APIs Workshop at netdevconf

[David Ahern]

On 5/7/24 10:56 AM, Andy Gospodarek wrote:
>>
>> That require driver support, no? e.g., There is no way that queue API is
>> going to work with the Enfabrica device without driver support.
> 
> I defintely think that there should be a consumer of the queue API
> before it lands upstream, but if it cannot work without a
> driver/hardware support that seems a bit odd.
> 
> Maybe I've missed it on the list, but do you have something you can
> share about this proposed queue API?
> 

commit 087b24de5c825c53f15a9481b94f757223c20610
Author: Mina Almasry <almasrymina@google.com>
Date:   Wed May 1 23:25:40 2024 +0000

    queue_api: define queue api

    This API enables the net stack to reset the queues used for devmem TCP.


and then gve support for it:

commit c93462b914dbf46b0c0256f7784cc79f7c368e45
Author: Shailend Chand <shailend@google.com>
Date:   Wed May 1 23:25:49 2024 +0000

    gve: Implement queue api

    The new netdev queue api is implemented for gve.

Re: Driver and H/W APIs Workshop at netdevconf

[Andy Gospodarek]

On Tue, May 07, 2024 at 11:02:05AM -0600, David Ahern wrote:
> On 5/7/24 10:56 AM, Andy Gospodarek wrote:
> >>
> >> That require driver support, no? e.g., There is no way that queue API is
> >> going to work with the Enfabrica device without driver support.
> > 
> > I defintely think that there should be a consumer of the queue API
> > before it lands upstream, but if it cannot work without a
> > driver/hardware support that seems a bit odd.
> > 
> > Maybe I've missed it on the list, but do you have something you can
> > share about this proposed queue API?
> > 
> 
> commit 087b24de5c825c53f15a9481b94f757223c20610
> Author: Mina Almasry <almasrymina@google.com>
> Date:   Wed May 1 23:25:40 2024 +0000
> 
>     queue_api: define queue api
> 
>     This API enables the net stack to reset the queues used for devmem TCP.
> 
> 
> and then gve support for it:
> 
> commit c93462b914dbf46b0c0256f7784cc79f7c368e45
> Author: Shailend Chand <shailend@google.com>
> Date:   Wed May 1 23:25:49 2024 +0000
> 
>     gve: Implement queue api
> 
>     The new netdev queue api is implemented for gve.

Thanks, good news is I'm aware of those.  I guess I read your statement
a little differently than you probably intended....

Good topic as there are some more things that are worth considering based on
driver/HW nuances that are likely going to be diff for each vendor.

Re: Driver and H/W APIs Workshop at netdevconf

[Mina Almasry]

On Mon, May 6, 2024 at 6:06 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:
> > Suggested topics based on recent netdev threads include
> > - devlink - extensions, shortcomings, ...
> > - extension to memory pools
> > - new APIs for managing queues
> > - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> > - fwctl - a proposal for direct firmware access
>
> Memory pools and queue API are more of stack features.
> Please leave them out of your fwctl session.
>
> Aren't people who are actually working on those things submitting
> talks or hosting better scoped discussions? It appears you haven't
> CCed any of them..
>

Me/Willem/Pavel/David/Shailend (I know, list is long xD), submitted a
Devem TCP + Io_uring joint talk. We don't know if we'll get accepted.
So far we plan to cover netmem + memory pools out of that list. We
didn't plan to cover queue-API yet because we didn't have it accepted
at talk submission time, but we just got it accepted so I was gonna
reach out anyway to see if folks would be OK to have it in our talk.

Any objection to having queue-API discussed as part of our talk? Or
add some of us to yours? I'm fine with whatever. Just thought it fits
well as part of this Devmem TCP + io_uring talk.

-- 
Thanks,
Mina

Re: Driver and H/W APIs Workshop at netdevconf

[Jakub Kicinski]

On Tue, 7 May 2024 11:17:57 -0700 Mina Almasry wrote:
> Me/Willem/Pavel/David/Shailend (I know, list is long xD), submitted a
> Devem TCP + Io_uring joint talk. We don't know if we'll get accepted.
> So far we plan to cover netmem + memory pools out of that list. We
> didn't plan to cover queue-API yet because we didn't have it accepted
> at talk submission time, but we just got it accepted so I was gonna
> reach out anyway to see if folks would be OK to have it in our talk.
> 
> Any objection to having queue-API discussed as part of our talk? Or
> add some of us to yours? I'm fine with whatever. Just thought it fits
> well as part of this Devmem TCP + io_uring talk.

I wonder if Amritha submitted something.

Otherwise it makes sense to cover as part of your session.
Or - if you're submitting a new session, pop my name on the list 
as well, if you don't mind.

Re: Driver and H/W APIs Workshop at netdevconf

[Mina Almasry]

On Tue, May 7, 2024 at 12:21 PM Jakub Kicinski <kuba@kernel.org> wrote:
>
> On Tue, 7 May 2024 11:17:57 -0700 Mina Almasry wrote:
> > Me/Willem/Pavel/David/Shailend (I know, list is long xD), submitted a
> > Devem TCP + Io_uring joint talk. We don't know if we'll get accepted.
> > So far we plan to cover netmem + memory pools out of that list. We
> > didn't plan to cover queue-API yet because we didn't have it accepted
> > at talk submission time, but we just got it accepted so I was gonna
> > reach out anyway to see if folks would be OK to have it in our talk.
> >
> > Any objection to having queue-API discussed as part of our talk? Or
> > add some of us to yours? I'm fine with whatever. Just thought it fits
> > well as part of this Devmem TCP + io_uring talk.
>
> I wonder if Amritha submitted something.
>

Yes please let us know.

> Otherwise it makes sense to cover as part of your session.
> Or - if you're submitting a new session, pop my name on the list
> as well, if you don't mind.

I'm not thinking of submitting a new session tbh, but I could if you
insist. Tbh I'm worried we have too little content for the devmem TCP
+ io_uring talk as-is. Could we dedicate 10/15 mins of that 30 min
talk to queue-API and add you to it? AFAICT I'm allowed to edit the
list of presenters and resubmit the talk.

-- 
Thanks,
Mina

Re: Driver and H/W APIs Workshop at netdevconf

[Jakub Kicinski]

On Tue, 7 May 2024 13:40:08 -0700 Mina Almasry wrote:
> > Otherwise it makes sense to cover as part of your session.
> > Or - if you're submitting a new session, pop my name on the list
> > as well, if you don't mind.  
> 
> I'm not thinking of submitting a new session tbh, but I could if you
> insist.

Not at all.

> Tbh I'm worried we have too little content for the devmem TCP
> + io_uring talk as-is. Could we dedicate 10/15 mins of that 30 min
> talk to queue-API and add you to it? AFAICT I'm allowed to edit the
> list of presenters and resubmit the talk.

SGTM!

Re: Driver and H/W APIs Workshop at netdevconf

[Nambiar, Amritha]

On 5/7/2024 12:21 PM, Jakub Kicinski wrote:
> On Tue, 7 May 2024 11:17:57 -0700 Mina Almasry wrote:
>> Me/Willem/Pavel/David/Shailend (I know, list is long xD), submitted a
>> Devem TCP + Io_uring joint talk. We don't know if we'll get accepted.
>> So far we plan to cover netmem + memory pools out of that list. We
>> didn't plan to cover queue-API yet because we didn't have it accepted
>> at talk submission time, but we just got it accepted so I was gonna
>> reach out anyway to see if folks would be OK to have it in our talk.
>>
>> Any objection to having queue-API discussed as part of our talk? Or
>> add some of us to yours? I'm fine with whatever. Just thought it fits
>> well as part of this Devmem TCP + io_uring talk.
> 
> I wonder if Amritha submitted something.
> 
> Otherwise it makes sense to cover as part of your session.
> Or - if you're submitting a new session, pop my name on the list
> as well, if you don't mind.

I haven't submitted a talk this year. Last year, I covered the queue API 
efforts in 
https://netdevconf.info/0x17/sessions/talk/netlink-apis-to-exposeconfigure-netdev-objects.html. 
Post that, I believe these are the new efforts:
Queue APIs WRT netdev-genl - Jakub's queue stats extensions, Mina's 
netdev dmabuf binding (more like the first queue-set API in netdev-genl)

Netdev queue management ops - the queue APIs for alloc/free/start/stop 
by Mina

I was working on netdev-genl queue-set API for NAPI configuration 
(https://lore.kernel.org/netdev/171234737780.5075.5717254021446469741.stgit@anambiarhost.jf.intel.com/), 
but as discussions progressed, I wasn't sure if that was the solution to 
our problem. As a part of that effort, based on the reviews, I was 
considering generalizing some part of the code into core for queue 
start/stop steps though the driver implementation would still be more 
extensive.

We are also seeking some feedback on devlink-extensions topic and plan 
to bring that as well to the workshop.

Re: Driver and H/W APIs Workshop at netdevconf

[David Ahern]

On 5/7/24 12:17 PM, Mina Almasry wrote:
> On Mon, May 6, 2024 at 6:06 PM Jakub Kicinski <kuba@kernel.org> wrote:
>>
>> On Mon, 6 May 2024 13:59:31 -0600 David Ahern wrote:
>>> Suggested topics based on recent netdev threads include
>>> - devlink - extensions, shortcomings, ...
>>> - extension to memory pools
>>> - new APIs for managing queues
>>> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
>>> - fwctl - a proposal for direct firmware access
>>
>> Memory pools and queue API are more of stack features.
>> Please leave them out of your fwctl session.
>>
>> Aren't people who are actually working on those things submitting
>> talks or hosting better scoped discussions? It appears you haven't
>> CCed any of them..
>>
> 
> Me/Willem/Pavel/David/Shailend (I know, list is long xD), submitted a
> Devem TCP + Io_uring joint talk. We don't know if we'll get accepted.
> So far we plan to cover netmem + memory pools out of that list. We
> didn't plan to cover queue-API yet because we didn't have it accepted
> at talk submission time, but we just got it accepted so I was gonna
> reach out anyway to see if folks would be OK to have it in our talk.
> 
> Any objection to having queue-API discussed as part of our talk? Or
> add some of us to yours? I'm fine with whatever. Just thought it fits
> well as part of this Devmem TCP + io_uring talk.
> 

The queue API is a suggested topic given its newness. The current 4 ndos
and "ndo_queue_mem_size" were created based on gve. Are they sufficient
for other hardware vendors? Are extensions needed? Other use cases?
Discussions items if needed; maybe what exists is fine. Either way if
someone wanted to discuss, the scope for the workshop would be driver
APIs and down (driver code only).

Your use case is a driver capability upward.

To me a clear boundary.

Re: Driver and H/W APIs Workshop at netdevconf

[Jiri Pirko]

Mon, May 06, 2024 at 09:59:31PM CEST, dsahern@kernel.org wrote:
>Alex Duyck and I are co-chairing the "Driver and H/W APIs Workshop" at
>netdevconf in July.
>
>The workshop is a forum for discussing issues related to driver
>development and APIs (user and kernel) for configuring, monitoring and
>debugging hardware. Discussion will be open to anyone to present, though
>speakers will need to submit topics ahead of time.
>
>Suggested topics based on recent netdev threads include
>- devlink - extensions, shortcomings, ...
>- extension to memory pools
>- new APIs for managing queues
>- challenges of netdev / IB co-existence (e.g., driven by AI workloads)
>- fwctl - a proposal for direct firmware access
>
>
>Please let us know if you have a topic that you would like to discuss
>along with a time estimate.

I think that it would be great to discuss multi PF device with shared
resources and possilibity to represent somehow the multi-PF ASIC itself
with struct device entity of some sort.


>
>Thanks,
>

RE: Driver and H/W APIs Workshop at netdevconf

[Keller, Jacob E]

> -----Original Message-----
> From: Jiri Pirko <jiri@resnulli.us>
> Sent: Monday, May 6, 2024 11:47 PM
> To: David Ahern <dsahern@kernel.org>
> Cc: Keller, Jacob E <jacob.e.keller@intel.com>; Lobakin, Aleksander
> <aleksander.lobakin@intel.com>; M, Saeed <saeedm@nvidia.com>; Tariq Toukan
> <tariqt@nvidia.com>; Jason Gunthorpe <jgg@nvidia.com>; Andrew Gospodarek
> <andrew.gospodarek@broadcom.com>; michael.chan@broadcom.com;
> netdev@vger.kernel.org; Jiri Pirko <jiri@nvidia.com>; Alexander Duyck
> <alexander.duyck@gmail.com>
> Subject: Re: Driver and H/W APIs Workshop at netdevconf
> 
> Mon, May 06, 2024 at 09:59:31PM CEST, dsahern@kernel.org wrote:
> >Alex Duyck and I are co-chairing the "Driver and H/W APIs Workshop" at
> >netdevconf in July.
> >
> >The workshop is a forum for discussing issues related to driver
> >development and APIs (user and kernel) for configuring, monitoring and
> >debugging hardware. Discussion will be open to anyone to present, though
> >speakers will need to submit topics ahead of time.
> >
> >Suggested topics based on recent netdev threads include
> >- devlink - extensions, shortcomings, ...
> >- extension to memory pools
> >- new APIs for managing queues
> >- challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> >- fwctl - a proposal for direct firmware access
> >
> >
> >Please let us know if you have a topic that you would like to discuss
> >along with a time estimate.
> 
> I think that it would be great to discuss multi PF device with shared
> resources and possilibity to represent somehow the multi-PF ASIC itself
> with struct device entity of some sort.
> 

+1. Jiri and I have been discussing this due to some work related to the ice driver, and it would be helpful to get some discussion to help figure out a good direction to design towards.

> 
> >
> >Thanks,
> >

Re: Driver and H/W APIs Workshop at netdevconf

[David Ahern]

On 5/6/24 1:59 PM, David Ahern wrote:
> Alex Duyck and I are co-chairing the "Driver and H/W APIs Workshop" at
> netdevconf in July.
> 
> The workshop is a forum for discussing issues related to driver
> development and APIs (user and kernel) for configuring, monitoring and
> debugging hardware. Discussion will be open to anyone to present, though
> speakers will need to submit topics ahead of time.
> 
> Suggested topics based on recent netdev threads include
> - devlink - extensions, shortcomings, ...
> - extension to memory pools
> - new APIs for managing queues
> - challenges of netdev / IB co-existence (e.g., driven by AI workloads)
> - fwctl - a proposal for direct firmware access
> 
> 
> Please let us know if you have a topic that you would like to discuss
> along with a time estimate.
> 
> Thanks,

Reminder about this workshop at the upcoming netdevconf. It is scheduled
for Tuesday, July 16, at 10 a.m. PDT. If anyone has topics to discuss,
please let Alex and I know.

Current responses:
- Jiri Pirko / Jacob Keller
  + multi PF device with shared resources

- Amritha Nambiar
  + NAPI config for queue-set API?
  + devlink-extensions

I have a couple of topics - uapis and possibly fwctl.