Low latency communication over wifi

Low latency communication over wifi

[Pavel Machek]

Hi!

I'm trying to transfer some audio data over wifi. Latency constrains
are such that if it would took more than 20msec to wait for
transmission, the packet should be dropped instead, and new one should
be transmitted.

My final hardware contains CSR6026 chip, but drivers are way too
ugly, I decided it would be easier to prototype on ath5k-based card.

I hacked qcu.c to limit latencies:

  	  qi->tqi_cw_min = ath5k_cw_validate(0);
          qi->tqi_cw_max = ath5k_cw_validate(20);

...but I don't think it did what I wanted. What am I missing?

(I could use Intel 3945ABG-based card instead, but I figured out ath5k
might be easier to hack?)

Is there way to manipulate type of service from userland to get
similar behaviour without patching kernel?

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Low latency communication over wifi

[Johannes Berg]

On Wed, 2015-09-23 at 13:03 +0200, Pavel Machek wrote:
> Hi!
> 
> I'm trying to transfer some audio data over wifi. Latency constrains
> are such that if it would took more than 20msec to wait for
> transmission, the packet should be dropped instead, and new one 
> should be transmitted.

This is typically not supported, at least not by the drivers right now.
Some hardware might support it, not sure.

>   	  qi->tqi_cw_min = ath5k_cw_validate(0);
>           qi->tqi_cw_max = ath5k_cw_validate(20);
> 
> ...but I don't think it did what I wanted. What am I missing?

That just changes the content window settings, which changes the
probability of getting medium access.

> (I could use Intel 3945ABG-based card instead, but I figured out 
> ath5k might be easier to hack?)
> 
> Is there way to manipulate type of service from userland to get
> similar behaviour without patching kernel?

If your AP has WMM/QoS then you can use setsockopt(SO_PRIORITY) to
change the TID and thus the AC and get your packets classified as voice
(VO) on wifi, which makes them much more likely to get access to the
medium.

johannes

ath5k filling logs (was Re: Low latency communication over wifi)

[Pavel Machek]

Hi!

> I hacked qcu.c to limit latencies:
> 
>   	  qi->tqi_cw_min = ath5k_cw_validate(0);
>           qi->tqi_cw_max = ath5k_cw_validate(20);
> 
> ...but I don't think it did what I wanted. What am I missing?
> 
> (I could use Intel 3945ABG-based card instead, but I figured out ath5k
> might be easier to hack?)

Speaking of ath5k... it fills logs with lots of messages:

[   53.486222] ath5k: ath5k_hw_get_isr: ISR: 0x00000400 IMR: 0x80081035
[   61.217713] ath5k: ath5k_hw_get_isr: ISR: 0x00000400 IMR: 0x80081035

Should I just delete the print from kernel, or is there something I
should debug?

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Low latency communication over wifi

[Bastian Bittorf]

* Pavel Machek <pavel@ucw.cz> [23.09.2015 13:38]:
> Is there way to manipulate type of service from userland to get
> similar behaviour without patching kernel?

yes, iptables DSCP / DiffServ / Differentiated Services Field

# you have to add some magic for matching your data
e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7

also you should consider force the ACK-timing to 450m / Class1
and forbid retransmission in minstrel

bye, bastian

Re: Low latency communication over wifi

[Pavel Machek]

Hi!

> > (I could use Intel 3945ABG-based card instead, but I figured out 
> > ath5k might be easier to hack?)
> > 
> > Is there way to manipulate type of service from userland to get
> > similar behaviour without patching kernel?
> 
> If your AP has WMM/QoS then you can use setsockopt(SO_PRIORITY) to
> change the TID and thus the AC and get your packets classified as voice
> (VO) on wifi, which makes them much more likely to get access to the
> medium.

That would be equivalent to ping -Q, right? It does not seem to have
any effect :-(. I'd expect at least local machine to use shorter waits
for medium, and thus drop packets instead of waiting.

On Wed 2015-09-23 13:42:36, Bastian Bittorf wrote:
> * Pavel Machek <pavel@ucw.cz> [23.09.2015 13:38]:
> > Is there way to manipulate type of service from userland to get
> > similar behaviour without patching kernel?
> 
> yes, iptables DSCP / DiffServ / Differentiated Services Field
> 
> # you have to add some magic for matching your data
> e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7

Again, this is ping -Q equivalent, right? I was doing

ping -c 300 -i .2 -Q $[56*4] -s 500 amd
300 packets transmitted, 300 received, 0% packet loss, time 60989ms
rtt min/avg/max/mdev = 2.155/8.599/44.475/5.677 ms
300 packets transmitted, 300 received, +1 duplicates, 0% packet loss, time 61030ms
rtt min/avg/max/mdev = 2.158/23.809/300.956/49.969 ms, pipe 2

I would expect packet loss, but got long delays instead.

> also you should consider force the ACK-timing to 450m / Class1
> and forbid retransmission in minstrel

Yes, disabling retransmission would be useful. How would I do that?

Thanks and best regards,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Low latency communication over wifi

[Johannes Berg]

On Sat, 2015-09-26 at 12:24 +0200, Pavel Machek wrote:
> 
> That would be equivalent to ping -Q, right? It does not seem to have
> any effect :-(. I'd expect at least local machine to use shorter waits
> for medium, and thus drop packets instead of waiting.

Correct. But it won't *drop* packets, it just increases the chances of
getting medium access.

> > # you have to add some magic for matching your data
> > e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7
> 
> Again, this is ping -Q equivalent, right? I was doing

Yes.

> ping -c 300 -i .2 -Q $[56*4] -s 500 amd
> 300 packets transmitted, 300 received, 0% packet loss, time 60989ms
> rtt min/avg/max/mdev = 2.155/8.599/44.475/5.677 ms
> 300 packets transmitted, 300 received, +1 duplicates, 0% packet loss, time 61030ms
> rtt min/avg/max/mdev = 2.158/23.809/300.956/49.969 ms, pipe 2
> 
> I would expect packet loss, but got long delays instead.

See above :)

> > also you should consider force the ACK-timing to 450m / Class1
> > and forbid retransmission in minstrel
> 
> Yes, disabling retransmission would be useful. How would I do that?
> 
It won't work on Intel devices though since they don't use
minstrel(_ht)

johannes

Re: Low latency communication over wifi

[Bastian Bittorf]

* Johannes Berg <johannes@sipsolutions.net> [28.09.2015 14:00]:
> > > # you have to add some magic for matching your data
> > > e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7
> > 
> > Again, this is ping -Q equivalent, right? I was doing
> 
> Yes.

IMHO this is not the same: you have to do it on both ends.
(also the iptables thingy)

bye, bastian

Re: Low latency communication over wifi

[Johannes Berg]

On Mon, 2015-09-28 at 14:15 +0200, Bastian Bittorf wrote:
> * Johannes Berg <johannes@sipsolutions.net> [28.09.2015 14:00]:
> > > > # you have to add some magic for matching your data
> > > > e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7
> > > 
> > > Again, this is ping -Q equivalent, right? I was doing
> > 
> > Yes.
> 
> IMHO this is not the same: you have to do it on both ends.
> (also the iptables thingy)
> 

Depends what you're measuring, but ping should apply the same DSCP tag
on the response I believe.

johannes

Re: Low latency communication over wifi

[Bastian Bittorf]

* Johannes Berg <johannes@sipsolutions.net> [28.09.2015 14:38]:
> Depends what you're measuring, but ping should apply the same DSCP tag
> on the response I believe.

your are right, I did not know this till yet 8-)
(i applied 0xb8 with 'ping -c1 -Q' just for testing)

14:49:52.491925 IP (tos 0xb8, ttl 64, id 51117, offset 0, flags [+], proto ICMP (1), 
	length 1500) 192.168.2.1 > 192.168.2.103: ICMP echo reply, id 14925, seq 1, length 1480
14:49:52.491993 IP (tos 0xb8, ttl 64, id 51117, offset 1480, flags [none], proto ICMP (1),
	length 48) 192.168.2.1 > 192.168.2.103: icmp

bye, bastian

Re: Low latency communication over wifi

[Pavel Machek]

On Mon 2015-09-28 09:41:43, Johannes Berg wrote:
> On Sat, 2015-09-26 at 12:24 +0200, Pavel Machek wrote:
> > 
> > That would be equivalent to ping -Q, right? It does not seem to have
> > any effect :-(. I'd expect at least local machine to use shorter waits
> > for medium, and thus drop packets instead of waiting.
> 
> Correct. But it won't *drop* packets, it just increases the chances of
> getting medium access.

Increases chances of medium access, but limits number of retries, so
it should drop sooner, no?

> > > # you have to add some magic for matching your data
> > > e.g. $IPT -t mangle -I OUTPUT -j DSCP --set-dscp-class CS7
> > 
> > Again, this is ping -Q equivalent, right? I was doing
> 
> Yes.
> 
> > ping -c 300 -i .2 -Q $[56*4] -s 500 amd
> > 300 packets transmitted, 300 received, 0% packet loss, time 60989ms
> > rtt min/avg/max/mdev = 2.155/8.599/44.475/5.677 ms
> > 300 packets transmitted, 300 received, +1 duplicates, 0% packet loss, time 61030ms
> > rtt min/avg/max/mdev = 2.158/23.809/300.956/49.969 ms, pipe 2
> > 
> > I would expect packet loss, but got long delays instead.
> 
> See above :)
> 
> > > also you should consider force the ACK-timing to 450m / Class1
> > > and forbid retransmission in minstrel
> > 
> > Yes, disabling retransmission would be useful. How would I do that?
> > 
> It won't work on Intel devices though since they don't use
> minstrel(_ht)

I'm using ath5k for testing... that should use minstrel AFAICT.

02:02.0 Ethernet controller: Qualcomm Atheros AR5211 Wireless Network
Adapter [AR5001X 802.11ab] (rev 01)

I'm now trying:

# Background load
sudo ping -f amd -s 8000
# Test
ping -c 30 -i .2 -Q $[0*4] -s 500 amd
ping -c 30 -i .2 -Q $[40*4] -s 500 amd
ping -c 30 -i .2 -Q $[56*4] -s 500 amd

This should send the second  ping to the priority queue based on -Q,
but I don't see an effect against one access point.... and it seems to
work somehow against second one. Good!

avg/maximum latency goes from 8/24 (-Q $[40*4]) to 80/258 (same
settings, every second frame is slow?!) to 8/26 to 7/32 to 6/18. But
this jumps to 92/123, 95/139, 91/128 and 135/678 with (-Q 0). -Q
$[56*4] produces 6/10, 8/23, 6/14, 5/9, 5/14. Good.

Thanks!

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: Low latency communication over wifi

[Bastian Bittorf]

* Pavel Machek <pavel@ucw.cz> [30.09.2015 19:27]:
> This should send the second  ping to the priority queue based on -Q,
> but I don't see an effect against one access point.... and it seems to
> work somehow against second one. Good!

you still need to hack/tune minstrel, because a lot of magic is
happen there including rts/cts on/off switching. a starting point is
e.g. this:
https://github.com/steinwurf/openwrt-patches/blob/master/openwrt-r41097/mac80211/910-minstrel-retry-limit.patch

bye, bastian

Re: Low latency communication over wifi

[Johannes Berg]

On Wed, 2015-09-30 at 19:24 +0200, Pavel Machek wrote:
> On Mon 2015-09-28 09:41:43, Johannes Berg wrote:
> > On Sat, 2015-09-26 at 12:24 +0200, Pavel Machek wrote:
> > > 
> > > That would be equivalent to ping -Q, right? It does not seem to have
> > > any effect :-(. I'd expect at least local machine to use shorter waits
> > > for medium, and thus drop packets instead of waiting.
> > 
> > Correct. But it won't *drop* packets, it just increases the chances of
> > getting medium access.
> 
> Increases chances of medium access, but limits number of retries, so
> it should drop sooner, no?

No, assigning VO AC by itself won't (typically) limit the number of
retries.

> This should send the second  ping to the priority queue based on -Q,
> but I don't see an effect against one access point.... and it seems to
> work somehow against second one. Good!
> 
Well it depends on whether or not your AP supports WMM/QoS to start
with :)

johannes