realtime ethernet

realtime ethernet

[Stéphane Ancelot]

Hi,

I would like to know if it is possible to use rt preempt for realtime
ethernet purposes ?

Regards,

S.Ancelot

Re: realtime ethernet

[Jeff Epler]

In LinuxCNC, we are successfully using rt preempt for realtime
networking.  You can find the code in our public git:
https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/drivers/mesa-hostmot2/hm2_eth.c
(GPL2+ license).  This talks to an FPGA board from Mesa Electronics,
which is running its own embedded realtime software stack.

Depending on the setup, this generally consists of 2 UDP packets
transmitted + 1 packet received, at a repetitive rate between 500Hz and
4kHz, with most users selecting 1kHz as far as we're aware.  The packets
aren't that big overall, without looking I bet it's under 1kB per period
all told.  (so the bandwidth utilization is not high)

On the Linux end we are generally using a dedicated gigabit NIC, and my
testing personally has been with Intel PCI-E nics like 82574L.  Most
users attach a single FPGA board (that board has a 100-megabit NIC) but
it's also possible to attach several boards via a dedicated gigabit
switch.

(The packet data consists of information from a CNC machine such as
motor position and other machine state, and commands to the machine
consist of things like motor torque commands or step waveform frequency
commands; LinuxCNC closes the position loop and performs other machine
logic)

As usual, not all mainboards and NICs are suitable, but we certainly
have success with specific Intel NICs. (and maybe some realteks?)

There are some tunables like RX IRQ coalesce that need to be, er, tuned
but I don't have that info on hand and it looks like it isn't in our
documentation. :(

I wish the tricks like manual ARP pinning and firewall manipulation
weren't necessary, but in our situation they seem to be somewhere
between helpful and necessary.  In particular, needing to ARP the other
NIC is not great for latency.

I've used hm2_eth with a range of RT kernels from 3.2 to 4.9 (usually
debian -rt kernels).

Disclaimer: I was compensated for the work I'm describing here

Jeff

Re: realtime ethernet

[Stéphane Ancelot]

Hi,

In any cases, I understand that the classic linux nic driver needs  
modifications to be realtime. (refer to rtnet work)

because of the NAPI stack not really suitable for realtime.

So, I  understand that these modifications are not parts of the rt 
preeempt patch today.

for some nics, yes, there are some not well documented timeouts 
registers to be "hacked"  in order being realtime too.

Regards,
S.Ancelot

Le 03/05/2017 à 23:33, Jeff Epler a écrit :
> In LinuxCNC, we are successfully using rt preempt for realtime
> networking.  You can find the code in our public git:
> https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/drivers/mesa-hostmot2/hm2_eth.c
> (GPL2+ license).  This talks to an FPGA board from Mesa Electronics,
> which is running its own embedded realtime software stack.
>
> Depending on the setup, this generally consists of 2 UDP packets
> transmitted + 1 packet received, at a repetitive rate between 500Hz and
> 4kHz, with most users selecting 1kHz as far as we're aware.  The packets
> aren't that big overall, without looking I bet it's under 1kB per period
> all told.  (so the bandwidth utilization is not high)
>
> On the Linux end we are generally using a dedicated gigabit NIC, and my
> testing personally has been with Intel PCI-E nics like 82574L.  Most
> users attach a single FPGA board (that board has a 100-megabit NIC) but
> it's also possible to attach several boards via a dedicated gigabit
> switch.
>
> (The packet data consists of information from a CNC machine such as
> motor position and other machine state, and commands to the machine
> consist of things like motor torque commands or step waveform frequency
> commands; LinuxCNC closes the position loop and performs other machine
> logic)
>
> As usual, not all mainboards and NICs are suitable, but we certainly
> have success with specific Intel NICs. (and maybe some realteks?)
>
> There are some tunables like RX IRQ coalesce that need to be, er, tuned
> but I don't have that info on hand and it looks like it isn't in our
> documentation. :(
>
> I wish the tricks like manual ARP pinning and firewall manipulation
> weren't necessary, but in our situation they seem to be somewhere
> between helpful and necessary.  In particular, needing to ARP the other
> NIC is not great for latency.
>
> I've used hm2_eth with a range of RT kernels from 3.2 to 4.9 (usually
> debian -rt kernels).
>
> Disclaimer: I was compensated for the work I'm describing here
>
> Jeff

Re: realtime ethernet

[Jeff Epler]

On Thu, May 04, 2017 at 09:54:36AM +0200, Stéphane Ancelot wrote:
> Hi,
> 
> In any cases, I understand that the classic linux nic driver needs
> modifications to be realtime. (refer to rtnet work)
> 
> because of the NAPI stack not really suitable for realtime.
> 
> So, I  understand that these modifications are not parts of the rt preeempt
> patch today.

We are working without any additional RT networking patches except those in the
debian -rt kernels.

Jeff