Re: Break-it testing for wifi

[Ben Greear <greearb@candelatech.com>]

On 11/22/2016 02:56 AM, Johannes Berg wrote:
> On Mon, 2016-11-21 at 08:10 -0800, Ben Greear wrote:
>
>> I am thinking about adding some sort of framework to wpa_supplicant
>> and/or the mac80211 stack to allow purposefully creating bad station
>> behaviour in order to test robustness of APs.
>
> I'm interested in this.
>
> Have you seen the fuzzer stuff in wpa_s/hostapd?
>
> See
>
> https://w1.fi/cgit/hostap/commit/?id=7d3f18d72c3c883112ee927fc402c0eaed09ff65
>
> for example for something Jouni did after our discussions recently.

I have not seen that yet.  I'll look at that more closely soon.

>
>> Some ideas so far:
>>
>> 1)  Allow supplicant to do bad state-machine transitions (start 4-way
>> before associating, for instance).
>
> Why would you do that? In order to test the AP implementation?

Yes.  And really, you should be able to do similar things on the AP to test stations,
or on IBSS/Mesh devices, etc.  hostapd already has some options to corrupt or
drop a percentage of various management frames.  Supplicant does not as far
as I know.

>
>> 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack
>> and/or supplicant.
>
> I think fuzzing the input path for those frames would be more useful
> than just corrupting things.

Random corruptions, especially by code that had at least some understanding
of management frames should be fast and easy to use.  It would not be as good
as a really clever fuzzer or hand-crafted frames, but for many users, hand-crafting
attacks would be well beyond what they could ever accomplish.

>
>> 3)  Possibly allow user to make specific corruptions.  This would
>> probably be in supplicant
>>       only, and I am not sure how this would be configured.  Maybe
>> allow user to over-ride
>>       existing IEs and add bogus ones of their own choosing.
>
> No idea what you really mean by this :)

Currently, supplicant can (at least with some small patches that I carry),
add custom information elements to probe requests and similar.  But, some things
are built by mac80211 (rate-sets advertised, for instance).  So, I was thinking of slightly extending
the API so that user-space could over-ride whatever mac80211 might normally
build itself.  That lets you more properly fuzz things from user space.

>
>> 4)  Maybe some specific tests like putting in over-flow sized lengths
>> of IEs.
>
> Again, fuzzing would cover this?

Yes, but for ease of use, and to cover frames generated by mac80211, I
was thinking:

echo 0.25 > /debug/.../wlan0/mgt_fuzzer

The fuzzer would then corrupt 25% of the management frames.  And instead of just randomly
scribbling, it could also parse the frames and do some more clever (and still pseudo-random)
modifications to the frames, like re-writing IEs with bad lengths, flipping bits in specific portions of the
frame we feel might find problems, etc.

I think if the tool became useful, then it could grow more clever over time.

>
>> Has anyone done anything similar they would like to share?
>>
>> Johannes:  Any interest in having such a framework in upstream
>> kernels?
>
> I suspect you have something entirely different in mind, like testing a
> (remote) AP implementation?

Yes.

>
> All of the local testing is probably better done via hwsim?

Well, there is a decent chance you could crash some firmware if you sent
corrupted EAPOL frames to it.  And just possibly some drivers inspect packets as well,
so I was thinking that fully transmitting the frames out of the system might
have some use.  And specifically for me, I am trying to test remote systems,
so hwsim would not be useful for that.

But, if local Linux (and local userspace) itself is the test target, then hwsim should give some very good
test coverage.

Thanks,
Ben

>
> johannes
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com