802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Luis R. Rodriguez]

Folks,

Back in 2009 Intel had put out a wifi-test tree [0] but that seems
deprecated now. The git tree at least is gone. Can someone confirm if
that's dead? We later had Google present at the 2010 San Francisco
wireless summit [1] their test infrastructure using autotest. Last I
checked that stuff was not merged back upstream to autotest. Can
someone confirm ? Lastly we have mac80211_hwsim [2] and also a slew of
internal testing infrastructures that obviously are not open.

If we are to embark on a new journey towards an open testing
infrastructure what should be used? Ben, you have some stuff, and I
know you also report some interesting bugs with insane amount of
stations. Is any of it open?

AFAICT mac80211_hwism should and likely already is used for a slew of
core API changes / tests. Addressing testing using that shoud
hopefully address tons of testings and find a lot of issues. We'd then
just need vendors to replicate behaviour on top of their drivers. The
core test stuff though still needs to be available.

What do we have, anyone have any lofty plans?

[0] http://wireless.kernel.org/en/developers/Testing/wifi-test
[1] http://wireless.kernel.org/en/developers/Summits/SanFranciscoBayArea-2010
[2] http://wireless.kernel.org/en/users/Drivers/mac80211_hwsim

  Luis

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Johannes Berg]

On Tue, 2013-07-30 at 17:53 -0700, Luis R. Rodriguez wrote:
> Folks,
> 
> Back in 2009 Intel had put out a wifi-test tree [0] but that seems
> deprecated now. The git tree at least is gone. Can someone confirm if
> that's dead?

It is indeed dead.

> We later had Google present at the 2010 San Francisco
> wireless summit [1] their test infrastructure using autotest. Last I
> checked that stuff was not merged back upstream to autotest. 

I don't think that's upstream, but it's certainly available as open
source from chromium.

> Can
> someone confirm ? Lastly we have mac80211_hwsim [2] and also a slew of
> internal testing infrastructures that obviously are not open.

hwsim is really orthogonal here - it's a test driver to allow testing in
e.g. virtual machine environments as opposed to a real
testbed/screenroom, but it obviously doesn't provide any test
infrastructure.

johannes

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Jouni Malinen]

On Tue, Jul 30, 2013 at 05:53:31PM -0700, Luis R. Rodriguez wrote:
> AFAICT mac80211_hwism should and likely already is used for a slew of
> core API changes / tests. Addressing testing using that shoud
> hopefully address tons of testings and find a lot of issues. We'd then
> just need vendors to replicate behaviour on top of their drivers. The
> core test stuff though still needs to be available.

I have automated tests running mainly from wpa_supplicant/hostapd
regression testing view point on top of mac80211_hwsim. The test cases
are in hostap.git:
http://w1.fi/gitweb/gitweb.cgi?p=hostap.git;a=tree;f=tests/hwsim;hb=HEAD

These tests are run automatically on all hostap.git commits in a simple
Buildbot setup: http://buildbot.w1.fi:8010/

There's currently 51 test cases covering relatively large set of basic
functionality (and various corner cases, too). I'm hoping to extend this
to cover all common use cases and lot of protocol conformance testing
for that matter. In addition, I use this for development time testing,
so new tests show up at the same time with new functionality.

These tests are currently run on top of a single cfg80211/mac80211
version that I'm updating manually every now and then. I have found
regressions on those components, too, and not just
wpa_supplicant/hostapd, so it would sound useful to get this running
automatically on cfg80211/mac80211 snapshots, too, to get faster reports
on regressions. My buildbot.w1.fi server is a very low-end virtual
server, so it won't be able to do this, but if someone has a bit more
beefier server available, it would be straightforward to extend this
type of Buildbot setup to trigger both on hostap.git and
wireless-testing.git commits to run the tests. The server would need to
be able to run kvm and build kernel.

Obviously mac80211_hwsim tests won't cover everything, but so far, I've
been quite happy with the coverage it can give for testing
wpa_supplicant, hostapd, cfg80211, and mac80211.

-- 
Jouni Malinen                                            PGP id EFC895FA

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Arend van Spriel]

On 07/31/2013 02:53 AM, Luis R. Rodriguez wrote:
> Folks,
>
> Back in 2009 Intel had put out a wifi-test tree [0] but that seems
> deprecated now. The git tree at least is gone. Can someone confirm if
> that's dead? We later had Google present at the 2010 San Francisco
> wireless summit [1] their test infrastructure using autotest. Last I
> checked that stuff was not merged back upstream to autotest. Can
> someone confirm ? Lastly we have mac80211_hwsim [2] and also a slew of
> internal testing infrastructures that obviously are not open.
>
> If we are to embark on a new journey towards an open testing
> infrastructure what should be used? Ben, you have some stuff, and I
> know you also report some interesting bugs with insane amount of
> stations. Is any of it open?
>
> AFAICT mac80211_hwism should and likely already is used for a slew of
> core API changes / tests. Addressing testing using that shoud
> hopefully address tons of testings and find a lot of issues. We'd then
> just need vendors to replicate behaviour on top of their drivers. The
> core test stuff though still needs to be available.
>
> What do we have, anyone have any lofty plans?

We have a large testing infrastructure consisting of shielded boxes, 
attenuators, etc. It is all exercised using tcl scripts and none of it 
is open today. Who knows what tomorrow will bring. The brcm80211 
open-source drivers are tested limited but nightly using it.

I am considering coming up with a new test framework for the brcm80211 
drivers using python scripting, which is why I started py80211 
experiment (available on github), but your mentioning of autotest makes 
me want to revisit that.

Regards,
Arend

> [0] http://wireless.kernel.org/en/developers/Testing/wifi-test
> [1] http://wireless.kernel.org/en/developers/Summits/SanFranciscoBayArea-2010
> [2] http://wireless.kernel.org/en/users/Drivers/mac80211_hwsim
>
>    Luis
> --
> To unsubscribe from this list: send the line "unsubscribe linux-wireless" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Kalle Valo]

"Arend van Spriel" <arend@broadcom.com> writes:

> I am considering coming up with a new test framework for the brcm80211
> drivers using python scripting, which is why I started py80211
> experiment (available on github), but your mentioning of autotest
> makes me want to revisit that.

No no, please forget anything your heard about autotest and just focus
on py80211 ;)

More seriously, my experience with the autotest and any other test
frameworks are that they are so freaking huge and difficult to maintain
that eventually they just die. Small is beautiful and I have been
dreaming about a lean and mean python based nl80211 test harness for
drivers, but never found the time to do anything (my usual story). So
I'm eagerly waiting how your py80211 will workout.

-- 
Kalle Valo

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Arend van Spriel]

On 08/05/2013 09:59 AM, Kalle Valo wrote:
> "Arend van Spriel" <arend@broadcom.com> writes:
>
>> I am considering coming up with a new test framework for the brcm80211
>> drivers using python scripting, which is why I started py80211
>> experiment (available on github), but your mentioning of autotest
>> makes me want to revisit that.
>
> No no, please forget anything your heard about autotest and just focus
> on py80211 ;)
>
> More seriously, my experience with the autotest and any other test
> frameworks are that they are so freaking huge and difficult to maintain
> that eventually they just die. Small is beautiful and I have been
> dreaming about a lean and mean python based nl80211 test harness for
> drivers, but never found the time to do anything (my usual story). So
> I'm eagerly waiting how your py80211 will workout.

I am on vacation now so I may end up working on it. Although, I do have 
an almost 5yr old son claiming most of my time :-)

Anyway, there have been a few patches applied to libnl itself to support 
genl in python. So I can send and receive netlink messages and 
notifications as a python dictionary of netlink attributes. Now need to 
determine the level of abstraction I want to provide to users of 
py80211. Suggestions are more than welcome.

Regards,
Arend

Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Ben Greear]

On 07/30/2013 05:53 PM, Luis R. Rodriguez wrote:
> Folks,
>
> Back in 2009 Intel had put out a wifi-test tree [0] but that seems
> deprecated now. The git tree at least is gone. Can someone confirm if
> that's dead? We later had Google present at the 2010 San Francisco
> wireless summit [1] their test infrastructure using autotest. Last I
> checked that stuff was not merged back upstream to autotest. Can
> someone confirm ? Lastly we have mac80211_hwsim [2] and also a slew of
> internal testing infrastructures that obviously are not open.
>
> If we are to embark on a new journey towards an open testing
> infrastructure what should be used? Ben, you have some stuff, and I
> know you also report some interesting bugs with insane amount of
> stations. Is any of it open?

Our user-space app is not open.  I'm happy to offer free licenses
for good causes, but I understand that is probably not so interesting to most
folks.

Our stuff only really works with ath9k (and possibly ath5k).  Might
work in the future with ath10k if we can get the firmware and other
limitations resolved.  And, maybe some other vendors will start
supporting lots of virtual stations..but I don't know of any others
at this time.

At the least, we'll keep running our internal testing & dev efforts
and will continue to report strange bugs :)

Thanks,
Ben


-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com