sendfile(2) use with a char driver

sendfile(2) use with a char driver

[hinxx]

I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".

Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.

The device driver provides a character device that can be mmap'ed.

There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?

Thanks! //hinko

Re: sendfile(2) use with a char driver

[Greg KH]

On Tue, Nov 29, 2022 at 09:15:24PM +0000, hinxx wrote:
> I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".
> 
> Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.
> 
> The device driver provides a character device that can be mmap'ed.
> 
> There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?

Please contact the authors of your kernel driver, they can answer this
best.

good luck!

greg k-h

Re: sendfile(2) use with a char driver

[hinxx]

Sent with Proton Mail secure email.

------- Original Message -------
On Wednesday, November 30th, 2022 at 8:05 AM, Greg KH <gregkh@linuxfoundation.org> wrote:


> On Tue, Nov 29, 2022 at 09:15:24PM +0000, hinxx wrote:
> 
> > I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".
> > 
> > Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.
> > 
> > The device driver provides a character device that can be mmap'ed.
> > 
> > There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?
> 
> 
> Please contact the authors of your kernel driver, they can answer this
> best.
> 

That would make sense, sadly they are MIA on their github repo engagement.

But in general, if I write a PCIe, DMA capable char device, what is the general guidance on what it takes to support splice/sendfile from a char device driver? Maybe there is an existing one in the kernel I could look at?

Is it even something worth pursuing? This line make me think it might not work for char devices https://elixir.bootlin.com/linux/v5.19.17/source/fs/splice.c#L827. Is going for block device the only way that such device driver can work with the sendfile?

Thanks! //hinko

Re: sendfile(2) use with a char driver

[Greg KH]

On Wed, Nov 30, 2022 at 08:23:45AM +0000, hinxx wrote:
> 
> 
> 
> 
> 
> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Wednesday, November 30th, 2022 at 8:05 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> 
> > On Tue, Nov 29, 2022 at 09:15:24PM +0000, hinxx wrote:
> > 
> > > I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".
> > > 
> > > Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.
> > > 
> > > The device driver provides a character device that can be mmap'ed.
> > > 
> > > There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?
> > 
> > 
> > Please contact the authors of your kernel driver, they can answer this
> > best.
> > 
> 
> That would make sense, sadly they are MIA on their github repo engagement.

Have a link to that repo?  Again, they are the only ones that can
resolve this, or you can modify the code to support this.

> But in general, if I write a PCIe, DMA capable char device, what is the general guidance on what it takes to support splice/sendfile from a char device driver? Maybe there is an existing one in the kernel I could look at?
> 
> Is it even something worth pursuing? This line make me think it might not work for char devices https://elixir.bootlin.com/linux/v5.19.17/source/fs/splice.c#L827. Is going for block device the only way that such device driver can work with the sendfile?

Step back and ask yourself why you want to try to do this first?  Are
you sure the slowdown of a char driver is the copying of the data?
Usually the hardware is the bottleneck, not the kernel code.

Anyway, this is way off-topic for the stable@vger.kernel.org mailing
list, please work on this on a subsystem-specific list.

good luck!

greg k-h

Re: sendfile(2) use with a char driver

[hinxx]

Sent with Proton Mail secure email.

------- Original Message -------
On Wednesday, November 30th, 2022 at 11:48 AM, Greg KH <gregkh@linuxfoundation.org> wrote:


> On Wed, Nov 30, 2022 at 08:23:45AM +0000, hinxx wrote:
> 
> > Sent with Proton Mail secure email.
> > 
> > ------- Original Message -------
> > On Wednesday, November 30th, 2022 at 8:05 AM, Greg KH gregkh@linuxfoundation.org wrote:
> > 
> > > On Tue, Nov 29, 2022 at 09:15:24PM +0000, hinxx wrote:
> > > 
> > > > I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".
> > > > 
> > > > Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.
> > > > 
> > > > The device driver provides a character device that can be mmap'ed.
> > > > 
> > > > There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?
> > > 
> > > Please contact the authors of your kernel driver, they can answer this
> > > best.
> > 
> > That would make sense, sadly they are MIA on their github repo engagement.
> 
> 
> Have a link to that repo? Again, they are the only ones that can
> resolve this, or you can modify the code to support this.
> 

Thank you for your time Greg!

I'm trying to understand the concepts behind the sendfile and modify the drivers myself.

Here is the Xilinx XDMA driver repo:

https://github.com/Xilinx/dma_ip_drivers/tree/master/XDMA/linux-kernel

A side note: As I noted earlier, I tested XDMA that has mmap()'ed PCIe register space (16 MB size) that the driver provides, just to see if it works. I would need the actual data channel, currently read() based to support sendfile.

I'm also looking to have the same sendfile support for this driver, too, which is also targeting similar hardware:

https://github.com/icshwi/tsc/tree/master/driver

I'm working with hardware / FPGA firmware that utilizes one or the other kernel driver.


> > But in general, if I write a PCIe, DMA capable char device, what is the general guidance on what it takes to support splice/sendfile from a char device driver? Maybe there is an existing one in the kernel I could look at?
> > 
> > Is it even something worth pursuing? This line make me think it might not work for char devices https://elixir.bootlin.com/linux/v5.19.17/source/fs/splice.c#L827. Is going for block device the only way that such device driver can work with the sendfile?
> 
> 
> Step back and ask yourself why you want to try to do this first? Are
> you sure the slowdown of a char driver is the copying of the data?
> Usually the hardware is the bottleneck, not the kernel code.

The hardware that these drivers work with are usually able to deliver data at a very high rate. The hardware contains 2/4 GB of DDR3/4 memory where the FPGA firmware stores the digitized analog signal as "samples". These samples would then be transferred to the CPU for processing, or in case of offline processing, shipped out to the network as-is. The idea is to minimize the CPU involvement in the latter case and my assumption is that would be achieved using sendfile/splice. I'm aware that network bandwidth is going to be the bottleneck later; it is about not using the CPU cycles for filling the NIC buffers.

Here is a note from the README of XDMA:

		Maximum data rate for x8 Gen3 is 8Gbytes/s, so for a x8Gen3
		design value of 0.81 data rate is 0.81*8 = 6.48Gbytes/s.
		Maximum data rate for x16 Gen3 is 16Gbytes/s, so for a x16Gen3
		design value of 0.78 data rate is 0.78*16 = 12.48Gbytes/s.

I've seen x4 Gen3 link ~3.8GB/s rates in practice on the hardware I have at hand.

> 
> Anyway, this is way off-topic for the stable@vger.kernel.org mailing
> list, please work on this on a subsystem-specific list.

No problem. Is the PCI mailing list where this discussion should continue?

Thanks! //hinko

> 
> good luck!
> 
> greg k-h

Re: sendfile(2) use with a char driver

[Greg KH]

On Thu, Dec 01, 2022 at 09:08:27AM +0000, hinxx wrote:
> 
> 
> 
> 
> 
> Sent with Proton Mail secure email.
> 
> ------- Original Message -------
> On Wednesday, November 30th, 2022 at 11:48 AM, Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> 
> > On Wed, Nov 30, 2022 at 08:23:45AM +0000, hinxx wrote:
> > 
> > > Sent with Proton Mail secure email.
> > > 
> > > ------- Original Message -------
> > > On Wednesday, November 30th, 2022 at 8:05 AM, Greg KH gregkh@linuxfoundation.org wrote:
> > > 
> > > > On Tue, Nov 29, 2022 at 09:15:24PM +0000, hinxx wrote:
> > > > 
> > > > > I'm looking to use a sendfile(2) with a Xilinx XDMA kernel driver in order to move data from a PCIe board with Xilinx FPGA to the network card with "zero-copy".
> > > > > 
> > > > > Currently I'm getting EINVAL return status from sendfile(2) when providing opened XDMA device file descriptor as input fd.
> > > > > 
> > > > > The device driver provides a character device that can be mmap'ed.
> > > > > 
> > > > > There seem to be other restrictions. Can anyone provide insight on what would be needed to make this work?
> > > > 
> > > > Please contact the authors of your kernel driver, they can answer this
> > > > best.
> > > 
> > > That would make sense, sadly they are MIA on their github repo engagement.
> > 
> > 
> > Have a link to that repo? Again, they are the only ones that can
> > resolve this, or you can modify the code to support this.
> > 
> 
> Thank you for your time Greg!
> 
> I'm trying to understand the concepts behind the sendfile and modify the drivers myself.
> 
> Here is the Xilinx XDMA driver repo:
> 
> https://github.com/Xilinx/dma_ip_drivers/tree/master/XDMA/linux-kernel

That's a very odd driver, it creates a lot of different char device
nodes, all doing different things.  Please work with the authors on that
to figure out how to get this to go faster if you need it and have
determined that the extra copy is the speed limiter here (for most
processors that is not the case, they can copy memory very fast).

And a char device node can support sendfile if the driver supports it,
so this is up to the driver to do so.

good luck!

greg k-h