Break-it testing for wifi

Break-it testing for wifi

[Ben Greear]

Hello!

I am thinking about adding some sort of framework to wpa_supplicant and/or the
mac80211 stack to allow purposefully creating bad station behaviour in order to
test robustness of APs.

Some ideas so far:

1)  Allow supplicant to do bad state-machine transitions (start 4-way before associating, for instance).

2)  Randomly corrupt mgt frames in driver and/or mac80211 stack and/or supplicant.

3)  Possibly allow user to make specific corruptions.  This would probably be in supplicant
     only, and I am not sure how this would be configured.  Maybe allow user to over-ride
     existing IEs and add bogus ones of their own choosing.

4)  Maybe some specific tests like putting in over-flow sized lengths of IEs.

Has anyone done anything similar they would like to share?

Johannes:  Any interest in having such a framework in upstream kernels?

Any other ideas for how to improve this feature set?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Break-it testing for wifi

[Mohammed Shafi Shajakhan]

Hi Ben,

just googled out 'wifi fuzzy testing' and found something relevant
as below
https://www.blackhat.com/presentations/bh-europe-07/Butti/Presentation/bh-eu-07-Butti.pdf

regards,
shafi

On Mon, Nov 21, 2016 at 08:10:37AM -0800, Ben Greear wrote:
> Hello!
> 
> I am thinking about adding some sort of framework to wpa_supplicant and/or the
> mac80211 stack to allow purposefully creating bad station behaviour in order to
> test robustness of APs.
> 
> Some ideas so far:
> 
> 1)  Allow supplicant to do bad state-machine transitions (start 4-way before associating, for instance).
> 
> 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack and/or supplicant.
> 
> 3)  Possibly allow user to make specific corruptions.  This would probably be in supplicant
>     only, and I am not sure how this would be configured.  Maybe allow user to over-ride
>     existing IEs and add bogus ones of their own choosing.
> 
> 4)  Maybe some specific tests like putting in over-flow sized lengths of IEs.
> 
> Has anyone done anything similar they would like to share?
> 
> Johannes:  Any interest in having such a framework in upstream kernels?
> 
> Any other ideas for how to improve this feature set?
> 
> Thanks,
> Ben
> 
> -- 
> Ben Greear <greearb@candelatech.com>
> Candela Technologies Inc  http://www.candelatech.com
> 

Re: Break-it testing for wifi

[Ben Greear]

On 11/21/2016 08:55 AM, Steve deRosier wrote:
>
>
> On Mon, Nov 21, 2016 at 8:10 AM, Ben Greear <greearb@candelatech.com <mailto:greearb@candelatech.com>> wrote:
>
>     Hello!
>
>     I am thinking about adding some sort of framework to wpa_supplicant and/or the
>     mac80211 stack to allow purposefully creating bad station behaviour in order to
>     test robustness of APs.
>
>     Some ideas so far:
>
>     1)  Allow supplicant to do bad state-machine transitions (start 4-way before associating, for instance).
>
>     2)  Randomly corrupt mgt frames in driver and/or mac80211 stack and/or supplicant.
>
>     3)  Possibly allow user to make specific corruptions.  This would probably be in supplicant
>         only, and I am not sure how this would be configured.  Maybe allow user to over-ride
>         existing IEs and add bogus ones of their own choosing.
>
>     4)  Maybe some specific tests like putting in over-flow sized lengths of IEs.
>
>     Has anyone done anything similar they would like to share?
>
>     Johannes:  Any interest in having such a framework in upstream kernels?
>
>     Any other ideas for how to improve this feature set?
>
>     Thanks,
>     Ben
>
>
> Hi Ben,
>
> I did something related to this many years ago using the radiotap interface and based on Andy Green's work.  It's very limited but might be interesting.
>
> https://github.com/derosier/packetvector
>
> I would absolutely welcome such a tool. I'd like it to be useful for testing clients and mesh nodes in addition to APs, but I don't think that's a big stretch.
>
> I guess my only comment is if it's possible for this to be in userspace, that's where'd like to see it. If not, some sort of framework we all could use would be
> nice. All in one place would also be nice, but as you've got multiple independent components that cooperate that might be difficult to achieve.

[re-added linux-wireless to CC]

Some things, like probe requests, are created as far down as the firmware (ath10k, for instance).
Some IEs are made in mac80211 as well.

And, ath10k (and likely other thick-firmware implementations) is crap for packet injection.

I think the closer I can be to 'real' supplicant, stack & driver behaviour, the better chance I have of finding
more interesting bugs.

So, I don't think a pure user-space app is that useful for my desired testing scenario.

My specific goal is testing APs, but I think it should work fine for testing other
connection types.

If I made some generic code to mangle management frames, including parsing IEs and re-writing
them and such (as well as randomly bit-flipping as requested), maybe it could be called from
tx logic in mac80211.  Corrupting things in the stack might help harden the drivers (and firmware)
a bit on the sending path, and might work for generic network devices.  Possibly I would need to add hooks in the
driver as well if frames were actually generated there.  For probe requests, I might have to
really hack all the way down into the firmware to have full control for corrupting that.

Since probe requests don't require much state, possibly that is a place where a relatively
dumb user-space packet injection would be best.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Break-it testing for wifi

[Johannes Berg]

On Mon, 2016-11-21 at 08:10 -0800, Ben Greear wrote:

> I am thinking about adding some sort of framework to wpa_supplicant
> and/or the mac80211 stack to allow purposefully creating bad station
> behaviour in order to test robustness of APs.

I'm interested in this.

Have you seen the fuzzer stuff in wpa_s/hostapd?

See

https://w1.fi/cgit/hostap/commit/?id=7d3f18d72c3c883112ee927fc402c0eaed09ff65

for example for something Jouni did after our discussions recently.

> Some ideas so far:
> 
> 1)  Allow supplicant to do bad state-machine transitions (start 4-way 
> before associating, for instance).

Why would you do that? In order to test the AP implementation?

> 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack
> and/or supplicant.

I think fuzzing the input path for those frames would be more useful
than just corrupting things.

> 3)  Possibly allow user to make specific corruptions.  This would
> probably be in supplicant
>      only, and I am not sure how this would be configured.  Maybe
> allow user to over-ride
>      existing IEs and add bogus ones of their own choosing.

No idea what you really mean by this :)

> 4)  Maybe some specific tests like putting in over-flow sized lengths
> of IEs.

Again, fuzzing would cover this?

> Has anyone done anything similar they would like to share?
> 
> Johannes:  Any interest in having such a framework in upstream
> kernels?

I suspect you have something entirely different in mind, like testing a
(remote) AP implementation?

All of the local testing is probably better done via hwsim?

johannes

Re: Break-it testing for wifi

[Ben Greear]

On 11/22/2016 02:56 AM, Johannes Berg wrote:
> On Mon, 2016-11-21 at 08:10 -0800, Ben Greear wrote:
>
>> I am thinking about adding some sort of framework to wpa_supplicant
>> and/or the mac80211 stack to allow purposefully creating bad station
>> behaviour in order to test robustness of APs.
>
> I'm interested in this.
>
> Have you seen the fuzzer stuff in wpa_s/hostapd?
>
> See
>
> https://w1.fi/cgit/hostap/commit/?id=7d3f18d72c3c883112ee927fc402c0eaed09ff65
>
> for example for something Jouni did after our discussions recently.

I have not seen that yet.  I'll look at that more closely soon.

>
>> Some ideas so far:
>>
>> 1)  Allow supplicant to do bad state-machine transitions (start 4-way
>> before associating, for instance).
>
> Why would you do that? In order to test the AP implementation?

Yes.  And really, you should be able to do similar things on the AP to test stations,
or on IBSS/Mesh devices, etc.  hostapd already has some options to corrupt or
drop a percentage of various management frames.  Supplicant does not as far
as I know.

>
>> 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack
>> and/or supplicant.
>
> I think fuzzing the input path for those frames would be more useful
> than just corrupting things.

Random corruptions, especially by code that had at least some understanding
of management frames should be fast and easy to use.  It would not be as good
as a really clever fuzzer or hand-crafted frames, but for many users, hand-crafting
attacks would be well beyond what they could ever accomplish.

>
>> 3)  Possibly allow user to make specific corruptions.  This would
>> probably be in supplicant
>>       only, and I am not sure how this would be configured.  Maybe
>> allow user to over-ride
>>       existing IEs and add bogus ones of their own choosing.
>
> No idea what you really mean by this :)

Currently, supplicant can (at least with some small patches that I carry),
add custom information elements to probe requests and similar.  But, some things
are built by mac80211 (rate-sets advertised, for instance).  So, I was thinking of slightly extending
the API so that user-space could over-ride whatever mac80211 might normally
build itself.  That lets you more properly fuzz things from user space.

>
>> 4)  Maybe some specific tests like putting in over-flow sized lengths
>> of IEs.
>
> Again, fuzzing would cover this?

Yes, but for ease of use, and to cover frames generated by mac80211, I
was thinking:

echo 0.25 > /debug/.../wlan0/mgt_fuzzer

The fuzzer would then corrupt 25% of the management frames.  And instead of just randomly
scribbling, it could also parse the frames and do some more clever (and still pseudo-random)
modifications to the frames, like re-writing IEs with bad lengths, flipping bits in specific portions of the
frame we feel might find problems, etc.

I think if the tool became useful, then it could grow more clever over time.

>
>> Has anyone done anything similar they would like to share?
>>
>> Johannes:  Any interest in having such a framework in upstream
>> kernels?
>
> I suspect you have something entirely different in mind, like testing a
> (remote) AP implementation?

Yes.

>
> All of the local testing is probably better done via hwsim?

Well, there is a decent chance you could crash some firmware if you sent
corrupted EAPOL frames to it.  And just possibly some drivers inspect packets as well,
so I was thinking that fully transmitting the frames out of the system might
have some use.  And specifically for me, I am trying to test remote systems,
so hwsim would not be useful for that.

But, if local Linux (and local userspace) itself is the test target, then hwsim should give some very good
test coverage.

Thanks,
Ben

>
> johannes
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Break-it testing for wifi

[Jouni Malinen]

On Tue, Nov 22, 2016 at 08:59:39AM -0800, Ben Greear wrote:
> On 11/22/2016 02:56 AM, Johannes Berg wrote:
> >On Mon, 2016-11-21 at 08:10 -0800, Ben Greear wrote:
> >>1)  Allow supplicant to do bad state-machine transitions (start 4-way
> >>before associating, for instance).
> >
> >Why would you do that? In order to test the AP implementation?
> 
> Yes.  And really, you should be able to do similar things on the AP to test stations,
> or on IBSS/Mesh devices, etc.  hostapd already has some options to corrupt or
> drop a percentage of various management frames.  Supplicant does not as far
> as I know.

I'm not sure adding a specific option to corrupt or drop a management
frame is the best approach here. Both hostapd and wpa_supplicant have
multiple mechanisms to enable testing in CONFIG_TESTING_OPTIONS=y builds
and the most flexible cases allow the management and EAPOL frame
processing to be handled by a test script through the control interface.
This provides much more control on how to misbehave and allows that to
be done at higher layer components without having to hardcode stuff in
hostapd/wpa_supplicant (or kernel for that matter). In case of
mac80211_hwsim tests, this is done in the python scripts (see
tests/hwsim directory in hostap.git).

> Currently, supplicant can (at least with some small patches that I carry),
> add custom information elements to probe requests and similar.  But, some things
> are built by mac80211 (rate-sets advertised, for instance).  So, I was thinking of slightly extending
> the API so that user-space could over-ride whatever mac80211 might normally
> build itself.  That lets you more properly fuzz things from user space.

For many cases, I'd expect it to be more flexible to override full frame
payload from a test script rather than come up with more detailed (and
more complex to implement) APIs.. As far as cfg80211/mac80211 is
concerned, it might be convenient to have a command that allows user
space to force a STA entry to a specific state when working in station
mode (hostapd can already do this in AP mode). This would allow the
existing offchannel management frame TX/RX operations to be used to
perform arbitrary exchanges and with the forced state changes, this
could cover more cases related to negotiating unexpected combinations or
exchanges frames in unexpected sequence, etc.

> >>4)  Maybe some specific tests like putting in over-flow sized lengths
> >>of IEs.
> >
> >Again, fuzzing would cover this?
> 
> Yes, but for ease of use, and to cover frames generated by mac80211, I
> was thinking:
> 
> echo 0.25 > /debug/.../wlan0/mgt_fuzzer
> 
> The fuzzer would then corrupt 25% of the management frames.  And instead of just randomly
> scribbling, it could also parse the frames and do some more clever (and still pseudo-random)
> modifications to the frames, like re-writing IEs with bad lengths, flipping bits in specific portions of the
> frame we feel might find problems, etc.

Why would this be in mac80211? I would much rather handle this type of
fuzzing operations in user space and just provide generic interfaces in
cfg80211 to allow the messages to be exchanged. For most parts, this
capability exists already today..

> >>Has anyone done anything similar they would like to share?
> >>
> >>Johannes:  Any interest in having such a framework in upstream
> >>kernels?

hostap.git tests/hwsim has quite a few such functions in testing use. I
don't think I'd support kernel changes in the style that you are
thinking of here, but I would very much support providing more generic
interfaces, where required, to allow this type of testing to be
performed from user space scripts.

> Well, there is a decent chance you could crash some firmware if you sent
> corrupted EAPOL frames to it.  And just possibly some drivers inspect packets as well,
> so I was thinking that fully transmitting the frames out of the system might
> have some use.  And specifically for me, I am trying to test remote systems,
> so hwsim would not be useful for that.
> 

Lots of this capability is already there in mac80211, cfg80211, hostapd,
and wpa_supplicant.. While mac80211_hwsim tests are one of the main use
cases for this, this works fine with any mac80211-based driver for
over-the-air testing as well. For AP mode, it is possible to override
most of management frame and EAPOL processing. For STA mode, EAPOL
processing can be fully overridden and quite a few management frame
exchanges can also be handled from test scripts through wpa_supplicant
control interface, but there may still exists some more constraints on
preventing some testing combinations.

> But, if local Linux (and local userspace) itself is the test target, then hwsim should give some very good
> test coverage.

Please keep in mind that almost anything that can tested with
mac80211_hwsim can also be used with real WLAN hardware when using
mac80211-based drivers..

-- 
Jouni Malinen                                            PGP id EFC895FA

Re: Break-it testing for wifi

[Johannes Berg]

On Tue, 2016-11-22 at 08:59 -0800, Ben Greear wrote:
> > Why would you do that? In order to test the AP implementation?
> 
> Yes.  And really, you should be able to do similar things on the AP
> to test stations, or on IBSS/Mesh devices, etc.  hostapd already has
> some options to corrupt or drop a percentage of various management
> frames.  Supplicant does not as far as I know.

Yes, but that's a far different focus. I'm far more interested in
testing *our* implementation(s), at which point I don't need to do it
over the air for most cases, and can be much more efficient that way,
etc.

I also don't really see a point of doing anything over the air to test
our implementations, apart from hitting firmware (but even then ...)

> > > 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack
> > > and/or supplicant.
> > 
> > I think fuzzing the input path for those frames would be more
> > useful than just corrupting things.
> 
> Random corruptions, especially by code that had at least some
> understanding of management frames should be fast and easy to
> use.  It would not be as good as a really clever fuzzer or hand-
> crafted frames, but for many users, hand-crafting attacks would be
> well beyond what they could ever accomplish.

Define "user"?

> Currently, supplicant can (at least with some small patches that I
> carry), add custom information elements to probe requests and
> similar.  But, some things are built by mac80211 (rate-sets
> advertised, for instance).  So, I was thinking of slightly extending
> the API so that user-space could over-ride whatever mac80211 might
> normally build itself.  That lets you more properly fuzz things from
> user space.

Why bother though, at that point? If you hook up some state transitions
you can probably just send the frames from userspace. For these testing
scenarios you can make assumptions about the hardware or query it
beforehand, so there's no need to rely on mac80211 at all?

> Yes, but for ease of use, and to cover frames generated by mac80211,
> I was thinking:
> 
> echo 0.25 > /debug/.../wlan0/mgt_fuzzer
> 
> The fuzzer would then corrupt 25% of the management frames.  And
> instead of just randomly scribbling, it could also parse the frames
> and do some more clever (and still pseudo-random) modifications to
> the frames, like re-writing IEs with bad lengths, flipping bits in
> specific portions of the frame we feel might find problems, etc.
> 
> I think if the tool became useful, then it could grow more clever
> over time.

I'd be far more inclined to just add a tracepoint there at some spot
and allow attaching BPF programs to it ... Or perhaps allow attaching
BPF programs directly, if tracepoint BPF can't modify the data.

Having any kind of logic here in the kernel space seems fairly useless
since you'll want to change it often, etc.

johannes

Re: Break-it testing for wifi

[Ben Greear]

On 11/28/2016 06:28 AM, Johannes Berg wrote:
> On Tue, 2016-11-22 at 08:59 -0800, Ben Greear wrote:
>>> Why would you do that? In order to test the AP implementation?
>>
>> Yes.  And really, you should be able to do similar things on the AP
>> to test stations, or on IBSS/Mesh devices, etc.  hostapd already has
>> some options to corrupt or drop a percentage of various management
>> frames.  Supplicant does not as far as I know.
>
> Yes, but that's a far different focus. I'm far more interested in
> testing *our* implementation(s), at which point I don't need to do it
> over the air for most cases, and can be much more efficient that way,
> etc.
>
> I also don't really see a point of doing anything over the air to test
> our implementations, apart from hitting firmware (but even then ...)

OTA is of primary importance to me, but I think any solution that works
for OTA would work for hwsim as well.

>
>>>> 2)  Randomly corrupt mgt frames in driver and/or mac80211 stack
>>>> and/or supplicant.
>>>
>>> I think fuzzing the input path for those frames would be more
>>> useful than just corrupting things.
>>
>> Random corruptions, especially by code that had at least some
>> understanding of management frames should be fast and easy to
>> use.  It would not be as good as a really clever fuzzer or hand-
>> crafted frames, but for many users, hand-crafting attacks would be
>> well beyond what they could ever accomplish.
>
> Define "user"?

System-test engineer in random company.  Like most of us, I am not working
for pure charity purposes :)

>
>> Currently, supplicant can (at least with some small patches that I
>> carry), add custom information elements to probe requests and
>> similar.  But, some things are built by mac80211 (rate-sets
>> advertised, for instance).  So, I was thinking of slightly extending
>> the API so that user-space could over-ride whatever mac80211 might
>> normally build itself.  That lets you more properly fuzz things from
>> user space.
>
> Why bother though, at that point? If you hook up some state transitions
> you can probably just send the frames from userspace. For these testing
> scenarios you can make assumptions about the hardware or query it
> beforehand, so there's no need to rely on mac80211 at all?

I don't want to re-implement supplicant, nor heavily modify it for this
effort.  I'm not sure exactly how important modifying IEs from user-space
would be for my effort, maybe existing API is enough with in-kernel fuzzer
I am thinking about writing.

>
>> Yes, but for ease of use, and to cover frames generated by mac80211,
>> I was thinking:
>>
>> echo 0.25 > /debug/.../wlan0/mgt_fuzzer
>>
>> The fuzzer would then corrupt 25% of the management frames.  And
>> instead of just randomly scribbling, it could also parse the frames
>> and do some more clever (and still pseudo-random) modifications to
>> the frames, like re-writing IEs with bad lengths, flipping bits in
>> specific portions of the frame we feel might find problems, etc.
>>
>> I think if the tool became useful, then it could grow more clever
>> over time.
>
> I'd be far more inclined to just add a tracepoint there at some spot
> and allow attaching BPF programs to it ... Or perhaps allow attaching
> BPF programs directly, if tracepoint BPF can't modify the data.
>
> Having any kind of logic here in the kernel space seems fairly useless
> since you'll want to change it often, etc.

Well, in-kernel seems best to me.  I will give it a try and see how
it works.

Thanks,
Ben

>
> johannes
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com