Re: 802.11 infrastructure for regression testing - upstream / mac80211 / cfg80211

[Jouni Malinen <j@w1.fi>]

On Tue, Jul 30, 2013 at 05:53:31PM -0700, Luis R. Rodriguez wrote:
> AFAICT mac80211_hwism should and likely already is used for a slew of
> core API changes / tests. Addressing testing using that shoud
> hopefully address tons of testings and find a lot of issues. We'd then
> just need vendors to replicate behaviour on top of their drivers. The
> core test stuff though still needs to be available.

I have automated tests running mainly from wpa_supplicant/hostapd
regression testing view point on top of mac80211_hwsim. The test cases
are in hostap.git:
http://w1.fi/gitweb/gitweb.cgi?p=hostap.git;a=tree;f=tests/hwsim;hb=HEAD

These tests are run automatically on all hostap.git commits in a simple
Buildbot setup: http://buildbot.w1.fi:8010/

There's currently 51 test cases covering relatively large set of basic
functionality (and various corner cases, too). I'm hoping to extend this
to cover all common use cases and lot of protocol conformance testing
for that matter. In addition, I use this for development time testing,
so new tests show up at the same time with new functionality.

These tests are currently run on top of a single cfg80211/mac80211
version that I'm updating manually every now and then. I have found
regressions on those components, too, and not just
wpa_supplicant/hostapd, so it would sound useful to get this running
automatically on cfg80211/mac80211 snapshots, too, to get faster reports
on regressions. My buildbot.w1.fi server is a very low-end virtual
server, so it won't be able to do this, but if someone has a bit more
beefier server available, it would be straightforward to extend this
type of Buildbot setup to trigger both on hostap.git and
wireless-testing.git commits to run the tests. The server would need to
be able to run kvm and build kernel.

Obviously mac80211_hwsim tests won't cover everything, but so far, I've
been quite happy with the coverage it can give for testing
wpa_supplicant, hostapd, cfg80211, and mac80211.

-- 
Jouni Malinen                                            PGP id EFC895FA