[Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Zhoupeng]

I'm using Rtnet with 82583 ethernet card.
It takes about 2us to send AF_PACKET using rt_dev_sendmsg(), about 200 bytes of data.

It is pretty fast, but I want to find what takes so long.
At last I find that "writel(tx_ring->tail)" spend above 1us to complete.

This is just MMIO write to 82583, of course MMIO is uncache.
And PCIE ASPM is disabled, so L0s is not the problem.

I think PCIE Memory Write TLPs is fast enough, but why spend so much time?

thanks,

Re: [Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Mariusz Janiak]

Dnia Czwartek, 16 Kwietnia 2015 13:30 Zhoupeng <zhoupeng1982@qq.com> napisał(a) 
> I'm using Rtnet with 82583 ethernet card.
> It takes about 2us to send AF_PACKET using rt_dev_sendmsg(), about 200 bytes of data.
> 
> It is pretty fast, but I want to find what takes so long.
> At last I find that "writel(tx_ring-&gt;tail)" spend above 1us to complete.
> 
> This is just MMIO write to 82583, of course MMIO is uncache.
> And PCIE ASPM is disabled, so L0s is not the problem.
> 
> I think PCIE Memory Write TLPs is fast enough, but why spend so much time?

Hi,

There is an interesting article that may be relevant to this issue

"PCI Express as a Killer of Software-based Real-Time Ethernet"

You can find copy in the attachment. 

Best
Mariusz


 
> thanks,
> _______________________________________________
> Xenomai mailing list
> Xenomai@xenomai.org
> http://www.xenomai.org/mailman/listinfo/xenomai

-------------- next part --------------
A non-text attachment was scrubbed...
Name: rteth_hw_eval.pdf
Type: application/x-unknown
Size: 875870 bytes
Desc: not available
URL: <http://www.xenomai.org/pipermail/xenomai/attachments/20150416/f9dc8e97/attachment.bin>

Re: [Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Zhoupeng]

>> This is just MMIO write to 82583, of course MMIO is uncache.
>> And PCIE ASPM is disabled, so L0s is not the problem.
>> 
>> I think PCIE Memory Write TLPs is fast enough, but why spend so much time?

>There is an interesting article that may be relevant to this issue
>
>"PCI Express as a Killer of Software-based Real-Time Ethernet"

I have read the article carefully.
The author don't mention PCIE ASPM, Latency will be better by disable it.

Every rt_dev_sendmsg will trigger one writel(tx_ring->tail) in Rtnet.
As the "writel" is time expensive, improvement may be done to Rtnet:
using rtskb to buffer multiple packets, and using one IOCTL to trigger "writel".

Re: [Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Gilles Chanteperdrix]

On Thu, Apr 16, 2015 at 10:39:59PM +0800, Zhoupeng wrote:
> >> This is just MMIO write to 82583, of course MMIO is uncache.
> >> And PCIE ASPM is disabled, so L0s is not the problem.
> >> 
> >> I think PCIE Memory Write TLPs is fast enough, but why spend so much time?
> 
> >There is an interesting article that may be relevant to this issue
> >
> >"PCI Express as a Killer of Software-based Real-Time Ethernet"
> 
> I have read the article carefully.
> The author don't mention PCIE ASPM, Latency will be better by disable it.
> 
> Every rt_dev_sendmsg will trigger one writel(tx_ring->tail) in Rtnet.
> As the "writel" is time expensive, improvement may be done to Rtnet:
> using rtskb to buffer multiple packets, and using one IOCTL to trigger "writel".

Patches welcome.

-- 
					    Gilles.

Re: [Xenomai] Rtnet e1000e 82583: slow writel(tx_ring->tail), takes above 1us?

[Gilles Chanteperdrix]

On Thu, Apr 16, 2015 at 10:39:59PM +0800, Zhoupeng wrote:
> >> This is just MMIO write to 82583, of course MMIO is uncache.
> >> And PCIE ASPM is disabled, so L0s is not the problem.
> >> 
> >> I think PCIE Memory Write TLPs is fast enough, but why spend so much time?
> 
> >There is an interesting article that may be relevant to this issue
> >
> >"PCI Express as a Killer of Software-based Real-Time Ethernet"
> 
> I have read the article carefully.
> The author don't mention PCIE ASPM, Latency will be better by disable it.
> 
> Every rt_dev_sendmsg will trigger one writel(tx_ring->tail) in Rtnet.
> As the "writel" is time expensive, improvement may be done to Rtnet:
> using rtskb to buffer multiple packets, and using one IOCTL to trigger "writel".

This optimization increases bandwidth at the expense of latency,
exactly the contrary to what RTnet is trying to achieve. So, patch
not welcome in fact. Or would be welcome a very careful patch which
does sends lone packets immediately, not queue them for sending them
later with a timeout or something.

-- 
					    Gilles.