Re: realtime ethernet

[Jeff Epler <jepler@unpythonic.net>]

In LinuxCNC, we are successfully using rt preempt for realtime
networking.  You can find the code in our public git:
https://github.com/LinuxCNC/linuxcnc/blob/master/src/hal/drivers/mesa-hostmot2/hm2_eth.c
(GPL2+ license).  This talks to an FPGA board from Mesa Electronics,
which is running its own embedded realtime software stack.

Depending on the setup, this generally consists of 2 UDP packets
transmitted + 1 packet received, at a repetitive rate between 500Hz and
4kHz, with most users selecting 1kHz as far as we're aware.  The packets
aren't that big overall, without looking I bet it's under 1kB per period
all told.  (so the bandwidth utilization is not high)

On the Linux end we are generally using a dedicated gigabit NIC, and my
testing personally has been with Intel PCI-E nics like 82574L.  Most
users attach a single FPGA board (that board has a 100-megabit NIC) but
it's also possible to attach several boards via a dedicated gigabit
switch.

(The packet data consists of information from a CNC machine such as
motor position and other machine state, and commands to the machine
consist of things like motor torque commands or step waveform frequency
commands; LinuxCNC closes the position loop and performs other machine
logic)

As usual, not all mainboards and NICs are suitable, but we certainly
have success with specific Intel NICs. (and maybe some realteks?)

There are some tunables like RX IRQ coalesce that need to be, er, tuned
but I don't have that info on hand and it looks like it isn't in our
documentation. :(

I wish the tricks like manual ARP pinning and firewall manipulation
weren't necessary, but in our situation they seem to be somewhere
between helpful and necessary.  In particular, needing to ARP the other
NIC is not great for latency.

I've used hm2_eth with a range of RT kernels from 3.2 to 4.9 (usually
debian -rt kernels).

Disclaimer: I was compensated for the work I'm describing here

Jeff