Request for comments on dejittering app

Request for comments on dejittering app

[John Sigler]

Hello,

I'd like to describe what I'm trying to do, so people can comment.

Basically I'm just transfering data between two systems over a network 
(possibly the Internet, but not necessarily).

(Systems A and B are synchronized by NTP.)

System A receives data from an input device on the PCI bus. This input 
device provides data at constant bit rate R, meaning if the driver 
requests B bytes, the DMA will take approximately B/R seconds.

I wrote a 'send' (user space) app on system A which requests 1316 bytes 
from the input device, adds an RTP header, and sends this data in a UDP 
datagram to system B. The RTP header provides, notably, a sequence 
number and a timestamp.

The datagrams travel over several networks, a few get lost, some arrive 
at system B, not necessarily in the order they were sent, and definitely 
not every B/R seconds.

I wrote a 'recv' (user space) app on system B whose job is to receive 
the packets from system A and re-send them (either to an output device 
on the PCI bus, or to another system on the LAN) after having removed 
the network jitter.

Here's my implementation:

Note: Packet P{i} contains a timestamp T{i}

buffer N packets
grab the current (absolute, CLOCK_REALTIME) time H{0}
send P0

for i=1 to infinity
{
   recv() a few packets (e.g. 3 at most)
   compute H{i} = H{i-1} + (T{i} - T{i-1})
   sleep until H{i}
   send packet P_i
}

The packets were timestamped on system A,
thus for every i, T{i} - T{i-1} = ~B/R
Therefore, most of the network jitter will be removed.

Any comments so far?

Now all this seems nice in theory, but I'd also like to make it work in 
practice! (And I'm having some problems.)

R is typically 38 Mbit/s => B/R is typically 277 µs
Obviously, I'm going to need high-resolution timers if I want to sleep 
for so small an interval.

As far as I understand, each packet received will generate one IRQ, and 
each write to the PCI output device will generate one IRQ. This means 
3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI 
device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)

It is easy to reduce the number of IRQs from the PCI device by grouping 
several packets for a single write. It might be worthwhile.

One problem I have is that the PCI device's driver blocks until the 
device has acknowledged the data, and the write operation sometimes 
blocks for 200, 300, even 400 µs (I have not been able to tell why).

I might not need -rt, if I'm willing to handle several packets every 
time I wake up?

buffer N packets
grab the current (absolute) time H{0}
send P0

for i=1 to infinity
{
   sleep for at least X ms
   recv() some packets (how many? as much as I can?)
   grab the current time Hnow
   while H{i} < Hnow send P{i}
}

(Rough pseudo-code.)

I'm eager to hear anyone's comments and suggestions.

Regards.

Re: Request for comments on dejittering app

[Gregory Haskins]

On Fri, 2007-08-10 at 17:39 +0200, John Sigler wrote:

> 
> As far as I understand, each packet received will generate one IRQ

This depends on the driver implementation.  If it uses the optional NAPI
interface the first interrupt received will simply schedule a netif
related thread to poll the device, disable future rx-interrupts for that
NIC, and return.

The packet scheduler thread will then poll your device according to its
configured policy with a specific "budget" of packets.  The device can
netif_rx as many packets from the device in that one invocation until
one of two criteria is met: The budget is exhausted, or the rx-ring is
exhausted.

If the rx-ring is exhausted, rx-interrupts are re-enabled and the return
code tells the netif_rx scheduler to deschedule this device.  If the
budget is exhausted before the ring, the return code indicates that we
still have work to do and interrupts remain off.  The scheduler will
re-poll your device at some point in the future.

There are other nitty gritty details here, but long story short you
generally will not have a 1:1 of interrupts to rx-packets on a busy
link.  This is important from both a latency and DOS perspective because
interrupt storms are mitigated.  So if your driver supports NAPI, I
suggest you use it.  If it doesn't, it might be worth changing it.

I know you had other aspects to your system, but this may help in at
least one dimension.
-Greg

Re: Request for comments on dejittering app

[Steven Rostedt]

--
On Fri, 10 Aug 2007, John Sigler wrote:

> Hello,

Hi John,

>
> I'd like to describe what I'm trying to do, so people can comment.
>

>
> Any comments so far?

Well, some NICs buffer writes, and some even buffer receives. So your NIC
may be buffering underneath you too.

>
> Now all this seems nice in theory, but I'd also like to make it work in
> practice! (And I'm having some problems.)
>
> R is typically 38 Mbit/s => B/R is typically 277 µs
> Obviously, I'm going to need high-resolution timers if I want to sleep
> for so small an interval.
>
> As far as I understand, each packet received will generate one IRQ, and
> each write to the PCI output device will generate one IRQ. This means
> 3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI
> device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)

As mentioned above, the NIC may buffer too, so it may not be sending the
packets out immediately.  Usually, a driver would send out several packets
before sending a transmit IRQ. In fact, that transmit IRQ is usually so
to tell the driver that it can write more after a certain threshold has
been reached. (It's been several years since I wrote a NIC driver, so I'm
going off of my old memory ;-)


>
> It is easy to reduce the number of IRQs from the PCI device by grouping
> several packets for a single write. It might be worthwhile.

May not be needed if the driver already does it.

>
> One problem I have is that the PCI device's driver blocks until the
> device has acknowledged the data, and the write operation sometimes
> blocks for 200, 300, even 400 µs (I have not been able to tell why).

Not sure what you mean by saying the driver "blocks". Is it a sleeping
thread (in -rt) or from an interrupt?  Can you be more specific here.

>
> I might not need -rt, if I'm willing to handle several packets every
> time I wake up?

For any reaction time less than 100 us (or maybe even higher), you will
need -rt.


-- Steve

Re: Request for comments on dejittering app

[John Sigler]

Steven Rostedt wrote:

> John Sigler wrote:
> 
>> Any comments so far?
> 
> Well, some NICs buffer writes, and some even buffer receives. So your NIC
> may be buffering underneath you too.

Now that you mention it, I did notice that /proc/interrupts did not 
report the same numbers whether I send 1 packet every X ms OR N packets 
every N*X ms.

By the way, what does /proc/interrupts report? The number of IRQs that 
were actually serviced by the OS?

>> Now all this seems nice in theory, but I'd also like to make it work in
>> practice! (And I'm having some problems.)
>>
>> R is typically 38 Mbit/s => B/R is typically 277 µs
>> Obviously, I'm going to need high-resolution timers if I want to sleep
>> for so small an interval.
>>
>> As far as I understand, each packet received will generate one IRQ, and
>> each write to the PCI output device will generate one IRQ. This means
>> 3600 IRQs per second from the NIC, and 3600 IRQs per second from the PCI
>> device. Is that reasonable? (Considering a 1267 MHz P3 with no IO-APIC.)
> 
> As mentioned above, the NIC may buffer too, so it may not be sending the
> packets out immediately.  Usually, a driver would send out several packets
> before sending a transmit IRQ. In fact, that transmit IRQ is usually so
> to tell the driver that it can write more after a certain threshold has
> been reached. (It's been several years since I wrote a NIC driver, so I'm
> going off of my old memory ;-)

Is /proc/interrupts a good way to tell whether the NIC is grouping 
transmits before generating an IRQ?

>> It is easy to reduce the number of IRQs from the PCI device by grouping
>> several packets for a single write. It might be worthwhile.
> 
> May not be needed if the driver already does it.
> 
>> One problem I have is that the PCI device's driver blocks until the
>> device has acknowledged the data, and the write operation sometimes
>> blocks for 200, 300, even 400 µs (I have not been able to tell why).
> 
> Not sure what you mean by saying the driver "blocks". Is it a sleeping
> thread (in -rt) or from an interrupt?  Can you be more specific here.

(I may have several misconceptions.) In the case of network drivers, 
when a user space process calls send, the driver copies the user data, 
and schedules the operation to happen "sometime later".

But it "feels" like my PCI device's driver didn't copy the user space 
data, and instead programmed the DMA directly from the user space 
buffer, which means it has to put the user space process to sleep until 
the operation is complete, which sometimes takes 100s of µs.

Or perhaps, there are only a few buffers (I've read something about ping 
pong DMA in the data sheet) in the driver, and because I'm writing only 
small amounts of data, they are sometimes all full, because some DMA 
operations are taking longer than usual.

The driver source code is available here:
http://dektec.com/Products/LinuxSDK/Downloads/LinuxSDK.zip

I will take a hard look at it when I come back at the office.

>> I might not need -rt, if I'm willing to handle several packets every
>> time I wake up?
> 
> For any reaction time less than 100 us (or maybe even higher), you will
> need -rt.

I'm still unsure whether there is a place in my app where I need a quick 
reaction time. What's your opinion?

Regards.