[Bpf] Re: Writing into a ring buffer map from user space

[Bpf] Re: Writing into a ring buffer map from user space

[Shankar Seal]

[-- Attachment #1.1: Type: text/plain, Size: 1338 bytes --]

Adding bpf@vger.kernel.org

A common use case of an BPF ring buffer map to use as a queue of events generated by BPF programs that can be read in-order by user space applications. I have a scenario requirement for a user space application to write into a ring buffer (or similar) map, such that events by BPF programs in kernel and user space applications are interleaved in the order they were generated, that can be consumed by another user space application

I would like to implement this new feature in the https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with the implementation, I wanted to check if there is any way to accomplish this in Linux today? If not, is there any reason why this should not be done?


Thanks,
Shankar
শংকর শীল



________________________________
From: Shankar Seal
Sent: Tuesday, May 28, 2024 12:40 AM
To: bpf@ietf.org <bpf@ietf.org>
Subject: Writing into a ring buffer map from user space


I have a scenario requirement for a user space application to write into a ring buffer eBPF map that I would like to implement in the https://github.com/microsoft/ebpf-for-windows project. Is there any way to accomplish this in Linux today? If not, is there any reason why this should not be done?



Thanks,
Shankar
শংকর শীল



[-- Attachment #1.2: Type: text/html, Size: 4227 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

Re: [Bpf] Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

On Tue, May 28, 2024 at 9:32 AM Shankar Seal
<Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
>
> Adding bpf@vger.kernel.org
>
> A common use case of an BPF ring buffer map to use as a queue of events generated by BPF programs that can be read in-order by user space applications. I have a scenario requirement for a user space application to write into a ring buffer (or similar) map, such that events by BPF programs in kernel and user space applications are interleaved in the order they were generated, that can be consumed by another user space application
>
> I would like to implement this new feature in the https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with the implementation, I wanted to check if there is any way to accomplish this in Linux today? If not, is there any reason why this should not be done?

Yes, there is. See user_ring_buffer ([0], [1]).

  [0] https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c
  [1] https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_ringbuf_success.c

>
> Thanks,
> Shankar
> শংকর শীল
>
>
>
> ________________________________
> From: Shankar Seal
> Sent: Tuesday, May 28, 2024 12:40 AM
> To: bpf@ietf.org <bpf@ietf.org>
> Subject: Writing into a ring buffer map from user space
>
>
> I have a scenario requirement for a user space application to write into a ring buffer eBPF map that I would like to implement in the https://github.com/microsoft/ebpf-for-windows project. Is there any way to accomplish this in Linux today? If not, is there any reason why this should not be done?
>
>
> Thanks,
> Shankar
> শংকর শীল
>
>
>
> --
> Bpf mailing list -- bpf@ietf.org
> To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

On Tue, May 28, 2024 at 9:32 AM Shankar Seal
<Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
>
> Adding bpf@vger.kernel.org
>
> A common use case of an BPF ring buffer map to use as a queue of events generated by BPF programs that can be read in-order by user space applications. I have a scenario requirement for a user space application to write into a ring buffer (or similar) map, such that events by BPF programs in kernel and user space applications are interleaved in the order they were generated, that can be consumed by another user space application
>
> I would like to implement this new feature in the https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with the implementation, I wanted to check if there is any way to accomplish this in Linux today? If not, is there any reason why this should not be done?

Yes, there is. See user_ring_buffer ([0], [1]).

  [0] https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/user_ringbuf.c
  [1] https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_ringbuf_success.c

>
> Thanks,
> Shankar
> শংকর শীল
>
>
>
> ________________________________
> From: Shankar Seal
> Sent: Tuesday, May 28, 2024 12:40 AM
> To: bpf@ietf.org <bpf@ietf.org>
> Subject: Writing into a ring buffer map from user space
>
>
> I have a scenario requirement for a user space application to write into a ring buffer eBPF map that I would like to implement in the https://github.com/microsoft/ebpf-for-windows project. Is there any way to accomplish this in Linux today? If not, is there any reason why this should not be done?
>
>
> Thanks,
> Shankar
> শংকর শীল
>
>
>
> --
> Bpf mailing list -- bpf@ietf.org
> To unsubscribe send an email to bpf-leave@ietf.org

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

RE: [Bpf] Re: Writing into a ring buffer map from user space

[dthaler1968]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
> 
> Yes, there is. See user_ring_buffer ([0], [1]).
> 
>   [0]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> user_ringbuf.c
>   [1]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://lwn.net/Articles/907056/

Thanks,
Dave

[Bpf] Re: Writing into a ring buffer map from user space

[dthaler1968=40googlemail.com]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
> 
> Yes, there is. See user_ring_buffer ([0], [1]).
> 
>   [0]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> user_ringbuf.c
>   [1]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://lwn.net/Articles/907056/

Thanks,
Dave



-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Shankar Seal]

[-- Attachment #1.1: Type: text/plain, Size: 4741 bytes --]

Thanks Dave and Andrii.

Per bpf: Add user-space-publisher ring buffer map type [LWN.net]<https://lwn.net/Articles/907056/>, the API that you mentioned
"provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."

But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.

Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in microsoft/ebpf-for-windows: eBPF implementation that runs on top of Windows (github.com)<https://github.com/microsoft/ebpf-for-windows> and eventually provide a Linux implementation as well.


Thanks,
Shankar
শংকর শীল



________________________________
From: dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=40googlemail.com@dmarc.ietf.org>
Sent: Tuesday, May 28, 2024 10:42 AM
To: 'Andrii Nakryiko' <andrii.nakryiko@gmail.com>; Shankar Seal <Shankar.Seal@microsoft.com>
Cc: bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <bpf@vger.kernel.org>
Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space

[You don't often get email from dthaler1968=40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Febpf-for-windows&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885854034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VEDCtAH%2FtGTBOUmRFzf%2BlPbhyVejY3PoJex5yOmmQQE%3D&reserved=0<https://github.com/microsoft/ebpf-for-windows> project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
>
> Yes, there is. See user_ring_buffer ([0], [1]).
>
>   [0]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprog_tests%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885863954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Tu9BXn%2BHuZvMLRRqC6Kvg%2F6V1vVrJ2Chk4jwI21UDWI%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/>
> user_ringbuf.c
>   [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprogs%2Fuser_&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885870894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=J8Zhz8R9j%2BaPnAw6uRvOWiY%2BW%2BExZRV8u5J1y%2BknsJ0%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_>
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F907056%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885875408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZpGIAzP6toavqQM6f63iLFHlM3%2BYc27jCAzaG3lbPMQ%3D&reserved=0<https://lwn.net/Articles/907056/>

Thanks,
Dave




[-- Attachment #1.2: Type: text/html, Size: 7266 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Shankar Seal]

[-- Attachment #1.1: Type: text/plain, Size: 6095 bytes --]

Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.

So I am sharing the signature of the proposed user API.

 ​    /**
    * @brief Write data into the ring buffer map from user space.
    *
    * @param ring_buffer_map_fd ring buffer map file descriptor.
    * @param data Pointer to data to be written.
    * @param data_length Length of data to be written.
  * @retval 0 The operation was successful.
  * @retval <0 An error occured, and errno was set.
    */
   int
   ring_buffer_user__write(
       fd_t ring_buffer_map_fd, const void* data, size_t data_length);

Please let me know if you have any questions about this API.
Thanks,
Shankar
শংকর শীল



________________________________
From: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>
Sent: Wednesday, June 5, 2024 10:01 PM
To: dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <andrii.nakryiko@gmail.com>
Cc: bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <bpf@vger.kernel.org>
Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

You don't often get email from shankar.seal=40microsoft.com@dmarc.ietf.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Thanks Dave and Andrii.

Per bpf: Add user-space-publisher ring buffer map type [LWN.net]<https://lwn.net/Articles/907056/>, the API that you mentioned
"provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."

But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.

Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in microsoft/ebpf-for-windows: eBPF implementation that runs on top of Windows (github.com)<https://github.com/microsoft/ebpf-for-windows> and eventually provide a Linux implementation as well.


Thanks,
Shankar
শংকর শীল



________________________________
From: dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=40googlemail.com@dmarc.ietf.org>
Sent: Tuesday, May 28, 2024 10:42 AM
To: 'Andrii Nakryiko' <andrii.nakryiko@gmail.com>; Shankar Seal <Shankar.Seal@microsoft.com>
Cc: bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <bpf@vger.kernel.org>
Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space

[You don't often get email from dthaler1968=40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Febpf-for-windows&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885854034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VEDCtAH%2FtGTBOUmRFzf%2BlPbhyVejY3PoJex5yOmmQQE%3D&reserved=0<https://github.com/microsoft/ebpf-for-windows> project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
>
> Yes, there is. See user_ring_buffer ([0], [1]).
>
>   [0]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprog_tests%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885863954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Tu9BXn%2BHuZvMLRRqC6Kvg%2F6V1vVrJ2Chk4jwI21UDWI%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/>
> user_ringbuf.c
>   [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprogs%2Fuser_&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885870894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=J8Zhz8R9j%2BaPnAw6uRvOWiY%2BW%2BExZRV8u5J1y%2BknsJ0%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_>
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F907056%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885875408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZpGIAzP6toavqQM6f63iLFHlM3%2BYc27jCAzaG3lbPMQ%3D&reserved=0<https://lwn.net/Articles/907056/>

Thanks,
Dave




[-- Attachment #1.2: Type: text/html, Size: 14736 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

[-- Attachment #1.1: Type: text/plain, Size: 6885 bytes --]

On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <Shankar.Seal@microsoft.com>
wrote:

> Since I have not heard back on this topic, I am assuming that there are no
> strong oppositions to this idea.
>
> So I am sharing the signature of the proposed user API.
>
>      /**
>     * @brief Write data into the ring buffer map from user space.
>     *
>     * @param ring_buffer_map_fd ring buffer map file descriptor.
>     * @param data Pointer to data to be written.
>     * @param data_length Length of data to be written.
>   * @retval 0 The operation was successful.
>   * @retval <0 An error occured, and errno was set.
>     */
>    int
>    ring_buffer_user__write(
>        fd_t ring_buffer_map_fd, const void* data, size_t data_length);
>
> Please let me know if you have any questions about this API.
>

I think the devil will be in the details. API itself makes sense (you can't
simplify it further or make it much different), in the end, you are just
sending an array of bytes into ringbuf.

But the implementation details are what matters. How the notification
works. How user space won't break kernel even if intentionally trying, etc.
It's not clear where you intend to implement this, etc.



> Thanks,
> Shankar
> শংকর শীল
>
>
>
> ------------------------------
> *From:* Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>
> *Sent:* Wednesday, June 5, 2024 10:01 PM
> *To:* dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=
> 40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <
> andrii.nakryiko@gmail.com>
> *Cc:* bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <
> bpf@vger.kernel.org>
> *Subject:* [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map
> from user space
>
> You don't often get email from shankar.seal=40microsoft.com@dmarc.ietf.org.
> Learn why this is important
> <https://aka.ms/LearnAboutSenderIdentification>
> Thanks Dave and Andrii.
>
> Per bpf: Add user-space-publisher ring buffer map type [LWN.net]
> <https://lwn.net/Articles/907056/>, the API that you mentioned
> "provides single-user-space-producer / single-kernel-consumer semantics
> over a ring buffer."
>
> But this is not the desired behavior for our case. We want both bpf
> programs in kernel mode and user application to be able to write to the
> same ring buffer, which can be consumed by a (potentially different) user
> application.
>
> Assuming no such API exists, do you see any strong reason *against* writing
> such an API? If not, we would like to implement one in microsoft/ebpf-for-windows:
> eBPF implementation that runs on top of Windows (github.com)
> <https://github.com/microsoft/ebpf-for-windows> and eventually provide a
> Linux implementation as well.
>
> Thanks,
> Shankar
> শংকর শীল
>
>
> ------------------------------
> *From:* dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=
> 40googlemail.com@dmarc.ietf.org>
> *Sent:* Tuesday, May 28, 2024 10:42 AM
> *To:* 'Andrii Nakryiko' <andrii.nakryiko@gmail.com>; Shankar Seal <
> Shankar.Seal@microsoft.com>
> *Cc:* bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <
> bpf@vger.kernel.org>
> *Subject:* [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from
> user space
>
> [You don't often get email from dthaler1968=
> 40googlemail.com@dmarc.ietf.org. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> > On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> > <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> > >
> > > Adding bpf@vger.kernel.org
> > >
> > > A common use case of an BPF ring buffer map to use as a queue of
> > > events generated by BPF programs that can be read in-order by user
> > > space applications. I have a scenario requirement for a user space
> > > application to write into a ring buffer (or similar) map, such that
> > > events by BPF programs in kernel and user space applications are
> > > interleaved in the order they were generated, that can be consumed by
> > > another user space application
> > >
> > > I would like to implement this new feature in the
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Febpf-for-windows&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885854034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VEDCtAH%2FtGTBOUmRFzf%2BlPbhyVejY3PoJex5yOmmQQE%3D&reserved=0
> <https://github.com/microsoft/ebpf-for-windows> project. But before I go
> ahead with
> > the implementation, I wanted to check if there is any way to accomplish
> this in
> > Linux today? If not, is there any reason why this should not be done?
> >
> > Yes, there is. See user_ring_buffer ([0], [1]).
> >
> >   [0]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprog_tests%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885863954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Tu9BXn%2BHuZvMLRRqC6Kvg%2F6V1vVrJ2Chk4jwI21UDWI%3D&reserved=0
> <https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/>
> > user_ringbuf.c
> >   [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprogs%2Fuser_&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885870894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=J8Zhz8R9j%2BaPnAw6uRvOWiY%2BW%2BExZRV8u5J1y%2BknsJ0%3D&reserved=0
> <https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_>
> > ringbuf_success.c
>
> Both of those links go to GPL code so I suspect Shankar cannot use those
> links.
> I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
> purpose and Shankar can read
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F907056%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885875408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZpGIAzP6toavqQM6f63iLFHlM3%2BYc27jCAzaG3lbPMQ%3D&reserved=0
> <https://lwn.net/Articles/907056/>
>
> Thanks,
> Dave
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 14921 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Shankar Seal]

[-- Attachment #1.1: Type: text/plain, Size: 8640 bytes --]

Here is a brief overview of what we intend to do in the eBPF for Windows code:

The user space app will not directly write into the underlying ring buffer of the eBPF map. Instead, the user app (via the libbpf API) will send the data via an IOCTL<https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-> to the eBPF core (a Windows kernel driver<https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/>) that manages the ring buffer map. The driver will internally invoke the same code that implements the bpf_ringbuf_output helper function to write the user provided data buffer into the ring buffer map.

I am not aware of how the ring buffer map is implemented in the Linux kernel. But presumably a similar approach could be taken in Linux as well?


Thanks,
Shankar
শংকর শীল



________________________________
From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Sent: Monday, June 24, 2024 8:36 PM
To: Shankar Seal <Shankar.Seal@microsoft.com>
Cc: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>; dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=40googlemail.com@dmarc.ietf.org>; bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <bpf@vger.kernel.org>
Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space



On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <Shankar.Seal@microsoft.com<mailto:Shankar.Seal@microsoft.com>> wrote:
Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.

So I am sharing the signature of the proposed user API.

     /**
    * @brief Write data into the ring buffer map from user space.
    *
    * @param ring_buffer_map_fd ring buffer map file descriptor.
    * @param data Pointer to data to be written.
    * @param data_length Length of data to be written.
  * @retval 0 The operation was successful.
  * @retval <0 An error occured, and errno was set.
    */
   int
   ring_buffer_user__write(
       fd_t ring_buffer_map_fd, const void* data, size_t data_length);

Please let me know if you have any questions about this API.

I think the devil will be in the details. API itself makes sense (you can't simplify it further or make it much different), in the end, you are just sending an array of bytes into ringbuf.

But the implementation details are what matters. How the notification works. How user space won't break kernel even if intentionally trying, etc. It's not clear where you intend to implement this, etc.


Thanks,
Shankar
শংকর শীল



________________________________
From: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org<mailto:40microsoft.com@dmarc.ietf.org>>
Sent: Wednesday, June 5, 2024 10:01 PM
To: dthaler1968=40googlemail.com@dmarc.ietf.org<mailto:40googlemail.com@dmarc.ietf.org> <dthaler1968=40googlemail.com@dmarc.ietf.org<mailto:40googlemail.com@dmarc.ietf.org>>; 'Andrii Nakryiko' <andrii.nakryiko@gmail.com<mailto:andrii.nakryiko@gmail.com>>
Cc: bpf@ietf.org<mailto:bpf@ietf.org> <bpf@ietf.org<mailto:bpf@ietf.org>>; bpf@vger.kernel.org<mailto:bpf@vger.kernel.org> <bpf@vger.kernel.org<mailto:bpf@vger.kernel.org>>
Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

You don't often get email from shankar.seal=40microsoft.com@dmarc.ietf.org<mailto:40microsoft.com@dmarc.ietf.org>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Thanks Dave and Andrii.

Per bpf: Add user-space-publisher ring buffer map type [LWN.net]<https://lwn.net/Articles/907056/>, the API that you mentioned
"provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."

But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.

Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in microsoft/ebpf-for-windows: eBPF implementation that runs on top of Windows (github.com)<https://github.com/microsoft/ebpf-for-windows> and eventually provide a Linux implementation as well.


Thanks,
Shankar
শংকর শীল



________________________________
From: dthaler1968=40googlemail.com@dmarc.ietf.org<mailto:40googlemail.com@dmarc.ietf.org> <dthaler1968=40googlemail.com@dmarc.ietf.org<mailto:40googlemail.com@dmarc.ietf.org>>
Sent: Tuesday, May 28, 2024 10:42 AM
To: 'Andrii Nakryiko' <andrii.nakryiko@gmail.com<mailto:andrii.nakryiko@gmail.com>>; Shankar Seal <Shankar.Seal@microsoft.com<mailto:Shankar.Seal@microsoft.com>>
Cc: bpf@ietf.org<mailto:bpf@ietf.org> <bpf@ietf.org<mailto:bpf@ietf.org>>; bpf@vger.kernel.org<mailto:bpf@vger.kernel.org> <bpf@vger.kernel.org<mailto:bpf@vger.kernel.org>>
Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space

[You don't often get email from dthaler1968=40googlemail.com@dmarc.ietf.org<mailto:40googlemail.com@dmarc.ietf.org>. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Andrii Nakryiko <andrii.nakryiko@gmail.com<mailto:andrii.nakryiko@gmail.com>> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=40microsoft.com@dmarc.ietf.org<mailto:40microsoft.com@dmarc.ietf.org>> wrote:
> >
> > Adding bpf@vger.kernel.org<mailto:bpf@vger.kernel.org>
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Febpf-for-windows&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885854034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VEDCtAH%2FtGTBOUmRFzf%2BlPbhyVejY3PoJex5yOmmQQE%3D&reserved=0<https://github.com/microsoft/ebpf-for-windows> project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
>
> Yes, there is. See user_ring_buffer ([0], [1]).
>
>   [0]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprog_tests%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885863954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Tu9BXn%2BHuZvMLRRqC6Kvg%2F6V1vVrJ2Chk4jwI21UDWI%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/>
> user_ringbuf.c
>   [1]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprogs%2Fuser_&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885870894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=J8Zhz8R9j%2BaPnAw6uRvOWiY%2BW%2BExZRV8u5J1y%2BknsJ0%3D&reserved=0<https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_>
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F907056%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885875408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZpGIAzP6toavqQM6f63iLFHlM3%2BYc27jCAzaG3lbPMQ%3D&reserved=0<https://lwn.net/Articles/907056/>

Thanks,
Dave




[-- Attachment #1.2: Type: text/html, Size: 20176 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

[-- Attachment #1.1: Type: text/plain, Size: 8667 bytes --]

On Mon, Jun 24, 2024 at 8:50 PM Shankar Seal <Shankar.Seal@microsoft.com>
wrote:

> Here is a brief overview of what we intend to do in the eBPF for Windows
> code:
>
> The user space app will not directly write into the underlying ring buffer
> of the eBPF map. Instead, the user app (via the libbpf API) will send the
> data via an IOCTL
> <https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-> to
> the eBPF core (a Windows kernel driver
> <https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/>)
> that manages the ring buffer map. The driver will internally invoke the
> same code that implements the bpf_ringbuf_output helper function to write
> the user provided data buffer into the ring buffer map.
>
> I am not aware of how the ring buffer map is implemented in the Linux
> kernel. But presumably a similar approach could be taken in Linux as well?
>

I don't think the Linux side can/should work like that.

Also, keep in mind that your HTML-based messages are not reaching
bpf@vger.kernel.org. So please fix your HTML set up and continue
conversation over bpf@vger.kernel.org.



>
> Thanks,
> Shankar
> শংকর শীল
>
>
> ------------------------------
> *From:* Andrii Nakryiko <andrii.nakryiko@gmail.com>
> *Sent:* Monday, June 24, 2024 8:36 PM
> *To:* Shankar Seal <Shankar.Seal@microsoft.com>
> *Cc:* Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>;
> dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=
> 40googlemail.com@dmarc.ietf.org>; bpf@ietf.org <bpf@ietf.org>;
> bpf@vger.kernel.org <bpf@vger.kernel.org>
> *Subject:* Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer
> map from user space
>
>
>
> On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <Shankar.Seal@microsoft.com>
> wrote:
>
> Since I have not heard back on this topic, I am assuming that there are no
> strong oppositions to this idea.
>
> So I am sharing the signature of the proposed user API.
>
>      /**
>     * @brief Write data into the ring buffer map from user space.
>     *
>     * @param ring_buffer_map_fd ring buffer map file descriptor.
>     * @param data Pointer to data to be written.
>     * @param data_length Length of data to be written.
>   * @retval 0 The operation was successful.
>   * @retval <0 An error occured, and errno was set.
>     */
>    int
>    ring_buffer_user__write(
>        fd_t ring_buffer_map_fd, const void* data, size_t data_length);
>
> Please let me know if you have any questions about this API.
>
>
> I think the devil will be in the details. API itself makes sense (you
> can't simplify it further or make it much different), in the end, you are
> just sending an array of bytes into ringbuf.
>
> But the implementation details are what matters. How the notification
> works. How user space won't break kernel even if intentionally trying, etc.
> It's not clear where you intend to implement this, etc.
>
>
>
> Thanks,
> Shankar
> শংকর শীল
>
>
>
> ------------------------------
> *From:* Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>
> *Sent:* Wednesday, June 5, 2024 10:01 PM
> *To:* dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=
> 40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <
> andrii.nakryiko@gmail.com>
> *Cc:* bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <
> bpf@vger.kernel.org>
> *Subject:* [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map
> from user space
>
> You don't often get email from shankar.seal=40microsoft.com@dmarc.ietf.org.
> Learn why this is important
> <https://aka.ms/LearnAboutSenderIdentification>
> Thanks Dave and Andrii.
>
> Per bpf: Add user-space-publisher ring buffer map type [LWN.net]
> <https://lwn.net/Articles/907056/>, the API that you mentioned
> "provides single-user-space-producer / single-kernel-consumer semantics
> over a ring buffer."
>
> But this is not the desired behavior for our case. We want both bpf
> programs in kernel mode and user application to be able to write to the
> same ring buffer, which can be consumed by a (potentially different) user
> application.
>
> Assuming no such API exists, do you see any strong reason *against* writing
> such an API? If not, we would like to implement one in microsoft/ebpf-for-windows:
> eBPF implementation that runs on top of Windows (github.com)
> <https://github.com/microsoft/ebpf-for-windows> and eventually provide a
> Linux implementation as well.
>
> Thanks,
> Shankar
> শংকর শীল
>
>
> ------------------------------
> *From:* dthaler1968=40googlemail.com@dmarc.ietf.org <dthaler1968=
> 40googlemail.com@dmarc.ietf.org>
> *Sent:* Tuesday, May 28, 2024 10:42 AM
> *To:* 'Andrii Nakryiko' <andrii.nakryiko@gmail.com>; Shankar Seal <
> Shankar.Seal@microsoft.com>
> *Cc:* bpf@ietf.org <bpf@ietf.org>; bpf@vger.kernel.org <
> bpf@vger.kernel.org>
> *Subject:* [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from
> user space
>
> [You don't often get email from dthaler1968=
> 40googlemail.com@dmarc.ietf.org. Learn why this is important at
> https://aka.ms/LearnAboutSenderIdentification ]
>
> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> > On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> > <Shankar.Seal=40microsoft.com@dmarc.ietf.org> wrote:
> > >
> > > Adding bpf@vger.kernel.org
> > >
> > > A common use case of an BPF ring buffer map to use as a queue of
> > > events generated by BPF programs that can be read in-order by user
> > > space applications. I have a scenario requirement for a user space
> > > application to write into a ring buffer (or similar) map, such that
> > > events by BPF programs in kernel and user space applications are
> > > interleaved in the order they were generated, that can be consumed by
> > > another user space application
> > >
> > > I would like to implement this new feature in the
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Febpf-for-windows&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885854034%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=VEDCtAH%2FtGTBOUmRFzf%2BlPbhyVejY3PoJex5yOmmQQE%3D&reserved=0
> <https://github.com/microsoft/ebpf-for-windows> project. But before I go
> ahead with
> > the implementation, I wanted to check if there is any way to accomplish
> this in
> > Linux today? If not, is there any reason why this should not be done?
> >
> > Yes, there is. See user_ring_buffer ([0], [1]).
> >
> >   [0]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprog_tests%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885863954%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=Tu9BXn%2BHuZvMLRRqC6Kvg%2F6V1vVrJ2Chk4jwI21UDWI%3D&reserved=0
> <https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/>
> > user_ringbuf.c
> >   [1]
> >
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftorvalds%2Flinux%2Fblob%2Fmaster%2Ftools%2Ftesting%2Fselftests%2Fbpf%2Fprogs%2Fuser_&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885870894%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=J8Zhz8R9j%2BaPnAw6uRvOWiY%2BW%2BExZRV8u5J1y%2BknsJ0%3D&reserved=0
> <https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_>
> > ringbuf_success.c
>
> Both of those links go to GPL code so I suspect Shankar cannot use those
> links.
> I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
> purpose and Shankar can read
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F907056%2F&data=05%7C02%7CShankar.Seal%40microsoft.com%7C0d269a2ce1364e2be3fd08dc7f3da157%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638525149885875408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=ZpGIAzP6toavqQM6f63iLFHlM3%2BYc27jCAzaG3lbPMQ%3D&reserved=0
> <https://lwn.net/Articles/907056/>
>
> Thanks,
> Dave
>
>
>
>

[-- Attachment #1.2: Type: text/html, Size: 20000 bytes --]

[-- Attachment #2: Type: text/plain, Size: 88 bytes --]

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

RE: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Shankar Seal]

Thanks Andrii.

I am changing the email format to plain text. Hopefully this will work with the vger mailing list.

>> I don't think the Linux side can/should work like that.

Note that the proposal I made for Windows in previous email is *effectively* the same as the following:
1. Load a bpf program that reads data from a user supplied map and then writes the data into a ring buffer.
2. User space app populates the data map and then invoke the program using bpf_prog_test.

Assuming this approach would be considered a valid use of eBPF, I think we can implement the API on Windows as proposed below. I will be happy to work with you to build a solution on Linux that is acceptable to you.

Thanks,
Shankar
শংকর শীল

From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Sent: Wednesday, June 26, 2024 4:44 PM
To: Shankar Seal <Shankar.Seal@microsoft.com>
Cc: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>; dthaler1968=40googlemail.com@dmarc.ietf.org; bpf@ietf.org; bpf@vger.kernel.org
Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space


>> On Mon, Jun 24, 2024 at 8:50 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:

>> Here is a brief overview of what we intend to do in the eBPF for Windows code:

>>  The user space app will not directly write into the underlying ring buffer of the eBPF map. Instead, the user app (via the libbpf API) will send the data via an IOCTL[1] to the eBPF core (a Windows Kernel  Driver[2])  that manages the ring buffer map. The driver will internally invoke the same code that implements the bpf_ringbuf_output helper function to write the user provided data buffer into the ring buffer map.

>>I am not aware of how the ring buffer map is implemented in the Linux kernel. But presumably a similar approach could be taken in Linux as well?

>> [1] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-

>> [2] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-

I don't think the Linux side can/should work like that.

Also, keep in mind that your HTML-based messages are not reaching mailto:bpf@vger.kernel.org. So please fix your HTML set up and continue conversation over mailto:bpf@vger.kernel.org.

Thanks,
Shankar
শংকর শীল

________________________________________
From: Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com>
Sent: Monday, June 24, 2024 8:36 PM
To: Shankar Seal <mailto:Shankar.Seal@microsoft.com>
Cc: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>; dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space



On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.

So I am sharing the signature of the proposed user API.

     /**
    * @brief Write data into the ring buffer map from user space.
    *
    * @param ring_buffer_map_fd ring buffer map file descriptor.
    * @param data Pointer to data to be written.
    * @param data_length Length of data to be written.
  * @retval 0 The operation was successful.
  * @retval <0 An error occured, and errno was set.
    */
   int
   ring_buffer_user__write(
       fd_t ring_buffer_map_fd, const void* data, size_t data_length);

Please let me know if you have any questions about this API.

I think the devil will be in the details. API itself makes sense (you can't simplify it further or make it much different), in the end, you are just sending an array of bytes into ringbuf.

But the implementation details are what matters. How the notification works. How user space won't break kernel even if intentionally trying, etc. It's not clear where you intend to implement this, etc.


Thanks,
Shankar
শংকর শীল


________________________________________
From: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>
Sent: Wednesday, June 5, 2024 10:01 PM
To: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>
Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space


You don't often get email from shankar.seal=mailto:40microsoft.com@dmarc.ietf.org. https://aka.ms/LearnAboutSenderIdentification

Thanks Dave and Andrii.
Per https://lwn.net/Articles/907056/, the API that you mentioned
"provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."

But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.
Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in https://github.com/microsoft/ebpf-for-windows and eventually provide a Linux implementation as well.

Thanks,
Shankar
শংকর শীল

________________________________________
From: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>
Sent: Tuesday, May 28, 2024 10:42 AM
To: 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>; Shankar Seal <mailto:Shankar.Seal@microsoft.com>
Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space

[You don't often get email from dthaler1968=mailto:40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding mailto:bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
>
> Yes, there is. See user_ring_buffer ([0], [1]).
>
>   [0]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> user_ringbuf.c
>   [1]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://lwn.net/Articles/907056/

Thanks,
Dave

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Shankar Seal]

Thanks Andrii.

I am changing the email format to plain text. Hopefully this will work with the vger mailing list.

>> I don't think the Linux side can/should work like that.

Note that the proposal I made for Windows in previous email is *effectively* the same as the following:
1. Load a bpf program that reads data from a user supplied map and then writes the data into a ring buffer.
2. User space app populates the data map and then invoke the program using bpf_prog_test.

Assuming this approach would be considered a valid use of eBPF, I think we can implement the API on Windows as proposed below. I will be happy to work with you to build a solution on Linux that is acceptable to you.

Thanks,
Shankar
শংকর শীল

From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Sent: Wednesday, June 26, 2024 4:44 PM
To: Shankar Seal <Shankar.Seal@microsoft.com>
Cc: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>; dthaler1968=40googlemail.com@dmarc.ietf.org; bpf@ietf.org; bpf@vger.kernel.org
Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space


>> On Mon, Jun 24, 2024 at 8:50 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:

>> Here is a brief overview of what we intend to do in the eBPF for Windows code:

>>  The user space app will not directly write into the underlying ring buffer of the eBPF map. Instead, the user app (via the libbpf API) will send the data via an IOCTL[1] to the eBPF core (a Windows Kernel  Driver[2])  that manages the ring buffer map. The driver will internally invoke the same code that implements the bpf_ringbuf_output helper function to write the user provided data buffer into the ring buffer map.

>>I am not aware of how the ring buffer map is implemented in the Linux kernel. But presumably a similar approach could be taken in Linux as well?

>> [1] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-

>> [2] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-

I don't think the Linux side can/should work like that.

Also, keep in mind that your HTML-based messages are not reaching mailto:bpf@vger.kernel.org. So please fix your HTML set up and continue conversation over mailto:bpf@vger.kernel.org.

Thanks,
Shankar
শংকর শীল

________________________________________
From: Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com>
Sent: Monday, June 24, 2024 8:36 PM
To: Shankar Seal <mailto:Shankar.Seal@microsoft.com>
Cc: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>; dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space



On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.

So I am sharing the signature of the proposed user API.

     /**
    * @brief Write data into the ring buffer map from user space.
    *
    * @param ring_buffer_map_fd ring buffer map file descriptor.
    * @param data Pointer to data to be written.
    * @param data_length Length of data to be written.
  * @retval 0 The operation was successful.
  * @retval <0 An error occured, and errno was set.
    */
   int
   ring_buffer_user__write(
       fd_t ring_buffer_map_fd, const void* data, size_t data_length);

Please let me know if you have any questions about this API.

I think the devil will be in the details. API itself makes sense (you can't simplify it further or make it much different), in the end, you are just sending an array of bytes into ringbuf.

But the implementation details are what matters. How the notification works. How user space won't break kernel even if intentionally trying, etc. It's not clear where you intend to implement this, etc.


Thanks,
Shankar
শংকর শীল


________________________________________
From: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>
Sent: Wednesday, June 5, 2024 10:01 PM
To: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>
Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space


You don't often get email from shankar.seal=mailto:40microsoft.com@dmarc.ietf.org. https://aka.ms/LearnAboutSenderIdentification

Thanks Dave and Andrii.
Per https://lwn.net/Articles/907056/, the API that you mentioned
"provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."

But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.
Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in https://github.com/microsoft/ebpf-for-windows and eventually provide a Linux implementation as well.

Thanks,
Shankar
শংকর শীল

________________________________________
From: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>
Sent: Tuesday, May 28, 2024 10:42 AM
To: 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>; Shankar Seal <mailto:Shankar.Seal@microsoft.com>
Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space

[You don't often get email from dthaler1968=mailto:40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com> wrote:

> On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org> wrote:
> >
> > Adding mailto:bpf@vger.kernel.org
> >
> > A common use case of an BPF ring buffer map to use as a queue of
> > events generated by BPF programs that can be read in-order by user
> > space applications. I have a scenario requirement for a user space
> > application to write into a ring buffer (or similar) map, such that
> > events by BPF programs in kernel and user space applications are
> > interleaved in the order they were generated, that can be consumed by
> > another user space application
> >
> > I would like to implement this new feature in the
> https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> the implementation, I wanted to check if there is any way to accomplish this in
> Linux today? If not, is there any reason why this should not be done?
>
> Yes, there is. See user_ring_buffer ([0], [1]).
>
>   [0]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> user_ringbuf.c
>   [1]
> https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> ringbuf_success.c

Both of those links go to GPL code so I suspect Shankar cannot use those links.
I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
purpose and Shankar can read https://lwn.net/Articles/907056/

Thanks,
Dave


-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org

Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

On Fri, Jun 28, 2024 at 11:44 PM Shankar Seal
<Shankar.Seal@microsoft.com> wrote:
>
> Thanks Andrii.
>
> I am changing the email format to plain text. Hopefully this will work with the vger mailing list.
>
> >> I don't think the Linux side can/should work like that.
>
> Note that the proposal I made for Windows in previous email is *effectively* the same as the following:
> 1. Load a bpf program that reads data from a user supplied map and then writes the data into a ring buffer.
> 2. User space app populates the data map and then invoke the program using bpf_prog_test.
>
> Assuming this approach would be considered a valid use of eBPF, I think we can implement the API on Windows as proposed below. I will be happy to work with you to build a solution on Linux that is acceptable to you.
>

I'm sorry, I don't think I'll have time to work on this. But what you
describe above about triggering a special BPF program to submit data
to ringbuf a) doesn't need any new APIs (we can do that today) but
also b) it's going to be slow and a bit cumbersome to use, probably.


> Thanks,
> Shankar
> শংকর শীল
>
> From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> Sent: Wednesday, June 26, 2024 4:44 PM
> To: Shankar Seal <Shankar.Seal@microsoft.com>
> Cc: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>; dthaler1968=40googlemail.com@dmarc.ietf.org; bpf@ietf.org; bpf@vger.kernel.org
> Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
> >> On Mon, Jun 24, 2024 at 8:50 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
>
> >> Here is a brief overview of what we intend to do in the eBPF for Windows code:
>
> >>  The user space app will not directly write into the underlying ring buffer of the eBPF map. Instead, the user app (via the libbpf API) will send the data via an IOCTL[1] to the eBPF core (a Windows Kernel  Driver[2])  that manages the ring buffer map. The driver will internally invoke the same code that implements the bpf_ringbuf_output helper function to write the user provided data buffer into the ring buffer map.
>
> >>I am not aware of how the ring buffer map is implemented in the Linux kernel. But presumably a similar approach could be taken in Linux as well?
>
> >> [1] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-
>
> >> [2] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-
>
> I don't think the Linux side can/should work like that.
>
> Also, keep in mind that your HTML-based messages are not reaching mailto:bpf@vger.kernel.org. So please fix your HTML set up and continue conversation over mailto:bpf@vger.kernel.org.
>
> Thanks,
> Shankar
> শংকর শীল
>
> ________________________________________
> From: Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com>
> Sent: Monday, June 24, 2024 8:36 PM
> To: Shankar Seal <mailto:Shankar.Seal@microsoft.com>
> Cc: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>; dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
>
> On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
> Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.
>
> So I am sharing the signature of the proposed user API.
>
>      /**
>     * @brief Write data into the ring buffer map from user space.
>     *
>     * @param ring_buffer_map_fd ring buffer map file descriptor.
>     * @param data Pointer to data to be written.
>     * @param data_length Length of data to be written.
>   * @retval 0 The operation was successful.
>   * @retval <0 An error occured, and errno was set.
>     */
>    int
>    ring_buffer_user__write(
>        fd_t ring_buffer_map_fd, const void* data, size_t data_length);
>
> Please let me know if you have any questions about this API.
>
> I think the devil will be in the details. API itself makes sense (you can't simplify it further or make it much different), in the end, you are just sending an array of bytes into ringbuf.
>
> But the implementation details are what matters. How the notification works. How user space won't break kernel even if intentionally trying, etc. It's not clear where you intend to implement this, etc.
>
>
> Thanks,
> Shankar
> শংকর শীল
>
>
> ________________________________________
> From: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>
> Sent: Wednesday, June 5, 2024 10:01 PM
> To: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>
> Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
> You don't often get email from shankar.seal=mailto:40microsoft.com@dmarc.ietf.org. https://aka.ms/LearnAboutSenderIdentification
>
> Thanks Dave and Andrii.
> Per https://lwn.net/Articles/907056/, the API that you mentioned
> "provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."
>
> But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.
> Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in https://github.com/microsoft/ebpf-for-windows and eventually provide a Linux implementation as well.
>
> Thanks,
> Shankar
> শংকর শীল
>
> ________________________________________
> From: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>
> Sent: Tuesday, May 28, 2024 10:42 AM
> To: 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>; Shankar Seal <mailto:Shankar.Seal@microsoft.com>
> Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space
>
> [You don't often get email from dthaler1968=mailto:40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com> wrote:
>
> > On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> > <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org> wrote:
> > >
> > > Adding mailto:bpf@vger.kernel.org
> > >
> > > A common use case of an BPF ring buffer map to use as a queue of
> > > events generated by BPF programs that can be read in-order by user
> > > space applications. I have a scenario requirement for a user space
> > > application to write into a ring buffer (or similar) map, such that
> > > events by BPF programs in kernel and user space applications are
> > > interleaved in the order they were generated, that can be consumed by
> > > another user space application
> > >
> > > I would like to implement this new feature in the
> > https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> > the implementation, I wanted to check if there is any way to accomplish this in
> > Linux today? If not, is there any reason why this should not be done?
> >
> > Yes, there is. See user_ring_buffer ([0], [1]).
> >
> >   [0]
> > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> > user_ringbuf.c
> >   [1]
> > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> > ringbuf_success.c
>
> Both of those links go to GPL code so I suspect Shankar cannot use those links.
> I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
> purpose and Shankar can read https://lwn.net/Articles/907056/
>
> Thanks,
> Dave
>
>

[Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space

[Andrii Nakryiko]

On Fri, Jun 28, 2024 at 11:44 PM Shankar Seal
<Shankar.Seal@microsoft.com> wrote:
>
> Thanks Andrii.
>
> I am changing the email format to plain text. Hopefully this will work with the vger mailing list.
>
> >> I don't think the Linux side can/should work like that.
>
> Note that the proposal I made for Windows in previous email is *effectively* the same as the following:
> 1. Load a bpf program that reads data from a user supplied map and then writes the data into a ring buffer.
> 2. User space app populates the data map and then invoke the program using bpf_prog_test.
>
> Assuming this approach would be considered a valid use of eBPF, I think we can implement the API on Windows as proposed below. I will be happy to work with you to build a solution on Linux that is acceptable to you.
>

I'm sorry, I don't think I'll have time to work on this. But what you
describe above about triggering a special BPF program to submit data
to ringbuf a) doesn't need any new APIs (we can do that today) but
also b) it's going to be slow and a bit cumbersome to use, probably.


> Thanks,
> Shankar
> শংকর শীল
>
> From: Andrii Nakryiko <andrii.nakryiko@gmail.com>
> Sent: Wednesday, June 26, 2024 4:44 PM
> To: Shankar Seal <Shankar.Seal@microsoft.com>
> Cc: Shankar Seal <Shankar.Seal=40microsoft.com@dmarc.ietf.org>; dthaler1968=40googlemail.com@dmarc.ietf.org; bpf@ietf.org; bpf@vger.kernel.org
> Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
> >> On Mon, Jun 24, 2024 at 8:50 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
>
> >> Here is a brief overview of what we intend to do in the eBPF for Windows code:
>
> >>  The user space app will not directly write into the underlying ring buffer of the eBPF map. Instead, the user app (via the libbpf API) will send the data via an IOCTL[1] to the eBPF core (a Windows Kernel  Driver[2])  that manages the ring buffer map. The driver will internally invoke the same code that implements the bpf_ringbuf_output helper function to write the user provided data buffer into the ring buffer map.
>
> >>I am not aware of how the ring buffer map is implemented in the Linux kernel. But presumably a similar approach could be taken in Linux as well?
>
> >> [1] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-
>
> >> [2] https://learn.microsoft.com/en-us/windows/win32/devio/device-input-and-output-control-ioctl-
>
> I don't think the Linux side can/should work like that.
>
> Also, keep in mind that your HTML-based messages are not reaching mailto:bpf@vger.kernel.org. So please fix your HTML set up and continue conversation over mailto:bpf@vger.kernel.org.
>
> Thanks,
> Shankar
> শংকর শীল
>
> ________________________________________
> From: Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com>
> Sent: Monday, June 24, 2024 8:36 PM
> To: Shankar Seal <mailto:Shankar.Seal@microsoft.com>
> Cc: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>; dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: Re: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
>
> On Thu, Jun 20, 2024 at 11:49 PM Shankar Seal <mailto:Shankar.Seal@microsoft.com> wrote:
> Since I have not heard back on this topic, I am assuming that there are no strong oppositions to this idea.
>
> So I am sharing the signature of the proposed user API.
>
>      /**
>     * @brief Write data into the ring buffer map from user space.
>     *
>     * @param ring_buffer_map_fd ring buffer map file descriptor.
>     * @param data Pointer to data to be written.
>     * @param data_length Length of data to be written.
>   * @retval 0 The operation was successful.
>   * @retval <0 An error occured, and errno was set.
>     */
>    int
>    ring_buffer_user__write(
>        fd_t ring_buffer_map_fd, const void* data, size_t data_length);
>
> Please let me know if you have any questions about this API.
>
> I think the devil will be in the details. API itself makes sense (you can't simplify it further or make it much different), in the end, you are just sending an array of bytes into ringbuf.
>
> But the implementation details are what matters. How the notification works. How user space won't break kernel even if intentionally trying, etc. It's not clear where you intend to implement this, etc.
>
>
> Thanks,
> Shankar
> শংকর শীল
>
>
> ________________________________________
> From: Shankar Seal <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org>
> Sent: Wednesday, June 5, 2024 10:01 PM
> To: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>; 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>
> Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: [Bpf] Re: [EXTERNAL] RE: Re: Writing into a ring buffer map from user space
>
>
> You don't often get email from shankar.seal=mailto:40microsoft.com@dmarc.ietf.org. https://aka.ms/LearnAboutSenderIdentification
>
> Thanks Dave and Andrii.
> Per https://lwn.net/Articles/907056/, the API that you mentioned
> "provides single-user-space-producer / single-kernel-consumer semantics over a ring buffer."
>
> But this is not the desired behavior for our case. We want both bpf programs in kernel mode and user application to be able to write to the same ring buffer, which can be consumed by a (potentially different) user application.
> Assuming no such API exists, do you see any strong reason against writing such an API? If not, we would like to implement one in https://github.com/microsoft/ebpf-for-windows and eventually provide a Linux implementation as well.
>
> Thanks,
> Shankar
> শংকর শীল
>
> ________________________________________
> From: dthaler1968=mailto:40googlemail.com@dmarc.ietf.org <dthaler1968=mailto:40googlemail.com@dmarc.ietf.org>
> Sent: Tuesday, May 28, 2024 10:42 AM
> To: 'Andrii Nakryiko' <mailto:andrii.nakryiko@gmail.com>; Shankar Seal <mailto:Shankar.Seal@microsoft.com>
> Cc: mailto:bpf@ietf.org <mailto:bpf@ietf.org>; mailto:bpf@vger.kernel.org <mailto:bpf@vger.kernel.org>
> Subject: [EXTERNAL] RE: [Bpf] Re: Writing into a ring buffer map from user space
>
> [You don't often get email from dthaler1968=mailto:40googlemail.com@dmarc.ietf.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Andrii Nakryiko <mailto:andrii.nakryiko@gmail.com> wrote:
>
> > On Tue, May 28, 2024 at 9:32 AM Shankar Seal
> > <Shankar.Seal=mailto:40microsoft.com@dmarc.ietf.org> wrote:
> > >
> > > Adding mailto:bpf@vger.kernel.org
> > >
> > > A common use case of an BPF ring buffer map to use as a queue of
> > > events generated by BPF programs that can be read in-order by user
> > > space applications. I have a scenario requirement for a user space
> > > application to write into a ring buffer (or similar) map, such that
> > > events by BPF programs in kernel and user space applications are
> > > interleaved in the order they were generated, that can be consumed by
> > > another user space application
> > >
> > > I would like to implement this new feature in the
> > https://github.com/microsoft/ebpf-for-windows project. But before I go ahead with
> > the implementation, I wanted to check if there is any way to accomplish this in
> > Linux today? If not, is there any reason why this should not be done?
> >
> > Yes, there is. See user_ring_buffer ([0], [1]).
> >
> >   [0]
> > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/prog_tests/
> > user_ringbuf.c
> >   [1]
> > https://github.com/torvalds/linux/blob/master/tools/testing/selftests/bpf/progs/user_
> > ringbuf_success.c
>
> Both of those links go to GPL code so I suspect Shankar cannot use those links.
> I think the answer is that BPF_MAP_TYPE_USER_RINGBUF is defined for this
> purpose and Shankar can read https://lwn.net/Articles/907056/
>
> Thanks,
> Dave
>
>

-- 
Bpf mailing list -- bpf@ietf.org
To unsubscribe send an email to bpf-leave@ietf.org