Network driver test suite?

Network driver test suite?

[David Hollis]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

Is there any kind of test suite (automated or list of tests) available
for testing Linux network drivers?  I'm completing the addition of a few
new USB ethernet devices and would like to able to test all possible
scenarios instead of coming across them piece-meal in the future.  I'm
not a big fan of the "works on my box" testing that I'm doing now.  I'm
sure there are all kinds of things that I'm not testing that aren't
everyday types of things such as VLANs, multicasting, various
ethtool/mii-tool things, large packets, etc.

If there isn't anything like this, maybe it would be a useful thing to
develop?  At a minimum, maybe to set the treshhold for minimum features
that drivers support and the like.

-- 
David Hollis <dhollis@davehollis.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Network driver test suite?

[Jeff Garzik]

David Hollis wrote:
> Is there any kind of test suite (automated or list of tests) available
> for testing Linux network drivers?  I'm completing the addition of a few


No, but I would love to someone to write one!!!

	Jeff (in a rare case of multi-exclamation-point use)

Re: Network driver test suite?

[David Hollis]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

On Wed, 2005-01-05 at 17:10 -0500, Jeff Garzik wrote:
> David Hollis wrote:
> > Is there any kind of test suite (automated or list of tests) available
> > for testing Linux network drivers?  I'm completing the addition of a few
> 
> 
> No, but I would love to someone to write one!!!
> 
> 	Jeff (in a rare case of multi-exclamation-point use)
> 
> 
What kinds of things do you think should be tested?  What I can think of
in no particular order and certainly not complete:

Simple, standard ping remote host
pings with various crazy large packet sizes
mii-tool
ethtool, all of the various options.  Not all need to be supported
certainly, but there is a set of basic ones that really should be
Changing MTU to various sizes
Configuring VLANs and being able to send/recv traffic

What other types of things should be tested?

-- 
David Hollis <dhollis@davehollis.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fw: Network driver test suite?

[Craig Thomas]

On Wed, 2005-01-05 at 15:26, Stephen Hemminger wrote:
> Begin forwarded message:
> 
> Date: Wed, 05 Jan 2005 15:19:46 -0500
> From: David Hollis <dhollis@davehollis.com>
> To: Netdev <netdev@oss.sgi.com>
> Subject: Network driver test suite?
> 
> 
> Is there any kind of test suite (automated or list of tests) available
> for testing Linux network drivers?  I'm completing the addition of a few
> new USB ethernet devices and would like to able to test all possible
> scenarios instead of coming across them piece-meal in the future.  I'm
> not a big fan of the "works on my box" testing that I'm doing now.  I'm
> sure there are all kinds of things that I'm not testing that aren't
> everyday types of things such as VLANs, multicasting, various
> ethtool/mii-tool things, large packets, etc.

Would there be a desire for someone to collect the tests or at least
create an index to all their locations?  If so, then developers can
scan a library of potential tests to run against newly developed code.

OSDL can start incorporating some of these tests into their test
platform as well.


> 
> If there isn't anything like this, maybe it would be a useful thing to
> develop?  At a minimum, maybe to set the treshhold for minimum features
> that drivers support and the like.

Re: Fw: Network driver test suite?

[David Hollis]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:

> 
> Would there be a desire for someone to collect the tests or at least
> create an index to all their locations?  If so, then developers can
> scan a library of potential tests to run against newly developed code.
> 
> OSDL can start incorporating some of these tests into their test
> platform as well.

I would love to see a collection of the types of tests that should be
performed.  As it appears now, there is nothing defined that a driver
author should do to verify that their driver performs properly, or
supports the right capabilities etc.  Some things may be difficult to
automate, but simply having a checklist would be great.  For the things
that can be automated, that would be even better.

-- 
David Hollis <dhollis@davehollis.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fw: Network driver test suite?

[Craig Thomas]

On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> 
> > 
> > Would there be a desire for someone to collect the tests or at least
> > create an index to all their locations?  If so, then developers can
> > scan a library of potential tests to run against newly developed code.
> > 
> > OSDL can start incorporating some of these tests into their test
> > platform as well.
> 
> I would love to see a collection of the types of tests that should be
> performed.  As it appears now, there is nothing defined that a driver
> author should do to verify that their driver performs properly, or
> supports the right capabilities etc.  Some things may be difficult to
> automate, but simply having a checklist would be great.  For the things
> that can be automated, that would be even better.

Great.  We can do some of this.  I would like to ask, what mimimal
types of tests do you expect to execute for a driver?  If several
can respond to the types of testing they perform, we can start
a checklist.  Then, additional items can be added to fill in the
holes.  I've asked Cliff White of OSDL to help put this together.

Re: Network driver test suite?

[Stephen Hemminger]

On Wed, 12 Jan 2005 09:14:01 -0800
Craig Thomas <craiger@osdl.org> wrote:

> On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> > On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> > 
> > > 
> > > Would there be a desire for someone to collect the tests or at least
> > > create an index to all their locations?  If so, then developers can
> > > scan a library of potential tests to run against newly developed code.
> > > 
> > > OSDL can start incorporating some of these tests into their test
> > > platform as well.
> > 
> > I would love to see a collection of the types of tests that should be
> > performed.  As it appears now, there is nothing defined that a driver
> > author should do to verify that their driver performs properly, or
> > supports the right capabilities etc.  Some things may be difficult to
> > automate, but simply having a checklist would be great.  For the things
> > that can be automated, that would be even better.
> 
> Great.  We can do some of this.  I would like to ask, what mimimal
> types of tests do you expect to execute for a driver?  If several
> can respond to the types of testing they perform, we can start
> a checklist.  Then, additional items can be added to fill in the
> holes.  I've asked Cliff White of OSDL to help put this together.

There are two types of tests that would be easy to set up.
First is a full exercise of all the possible API transitions through
ifconfig, ip link, and ethtool. These could be covered without any
traffic going through.

Then setup a standard test environment with a known good card and a
crossover cable.  The test could then use raw (and/or packet generator)
to send packets down good card to card to be verified.

Also testing, auto negotiation and transitions under load.

Re: Fw: Network driver test suite?

[Ben Greear]

Craig Thomas wrote:
> On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> 
>>On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
>>
>>
>>>Would there be a desire for someone to collect the tests or at least
>>>create an index to all their locations?  If so, then developers can
>>>scan a library of potential tests to run against newly developed code.
>>>
>>>OSDL can start incorporating some of these tests into their test
>>>platform as well.
>>
>>I would love to see a collection of the types of tests that should be
>>performed.  As it appears now, there is nothing defined that a driver
>>author should do to verify that their driver performs properly, or
>>supports the right capabilities etc.  Some things may be difficult to
>>automate, but simply having a checklist would be great.  For the things
>>that can be automated, that would be even better.
> 
> 
> Great.  We can do some of this.  I would like to ask, what mimimal
> types of tests do you expect to execute for a driver?  If several
> can respond to the types of testing they perform, we can start
> a checklist.  Then, additional items can be added to fill in the
> holes.  I've asked Cliff White of OSDL to help put this together.

My wishlist includes:

set and verify all supported link speeds (auto-negotiate, 10Mbps fixed, 100Mbps fixed,
1Gbps, full/half duplex, etc)
set & verify various MTUs

At each speed, generate maximum amount of packets:
    tx only
    rx only
    tx + rx
    Could use pktgen for this as it does not ARP or do other protocol things
    would complicate rx-only and tx-only testing.
    Could also do randomized packet sizes or step through a bunch of different
    sizes.
    Could randomize rates and other things with pktgen as well.
    Determine number of dropped & errored packets at each phase.
      This should be verified by counting the number of packets transmitted
      v/s received and coorelated against any drop/error counters that the
      driver reports.

Generate TCP & UDP traffic at various speeds to make sure it handles
protocols correctly too.

Run a similar battery of tests against 802.1Q VLANS on the
interfaces in question.


I have a proprietary tool and some GPL kernel patches that can do most of this.
A bash/perl script utilizing ethtool and and pktgen (especially a version of
pktgen similar to the one in my patches which also receives packets and reports
statistics on them) and a few other tcp & udp generating tools could also do
this (and in a more transparent manner, probably.)

Please contact me off the list if you want to discuss free use of the
proprietary tool.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network driver test suite?

[Randy.Dunlap]

Stephen Hemminger wrote:
> On Wed, 12 Jan 2005 09:14:01 -0800
> Craig Thomas <craiger@osdl.org> wrote:
> 
> 
>>On Wed, 2005-01-12 at 08:24, David Hollis wrote:
>>
>>>On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
>>>
>>>
>>>>Would there be a desire for someone to collect the tests or at least
>>>>create an index to all their locations?  If so, then developers can
>>>>scan a library of potential tests to run against newly developed code.
>>>>
>>>>OSDL can start incorporating some of these tests into their test
>>>>platform as well.
>>>
>>>I would love to see a collection of the types of tests that should be
>>>performed.  As it appears now, there is nothing defined that a driver
>>>author should do to verify that their driver performs properly, or
>>>supports the right capabilities etc.  Some things may be difficult to
>>>automate, but simply having a checklist would be great.  For the things
>>>that can be automated, that would be even better.
>>
>>Great.  We can do some of this.  I would like to ask, what mimimal
>>types of tests do you expect to execute for a driver?  If several
>>can respond to the types of testing they perform, we can start
>>a checklist.  Then, additional items can be added to fill in the
>>holes.  I've asked Cliff White of OSDL to help put this together.
> 
> 
> There are two types of tests that would be easy to set up.
> First is a full exercise of all the possible API transitions through
> ifconfig, ip link, and ethtool. These could be covered without any
> traffic going through.
> 
> Then setup a standard test environment with a known good card and a
> crossover cable.  The test could then use raw (and/or packet generator)
> to send packets down good card to card to be verified.
> 
> Also testing, auto negotiation and transitions under load.

Other than API testing, stats interface testing, & link/speed 
verification, the most useful test that I ever did in NIC driver
development (Natl Semi & Intel) was just traffic saturation:
copy and compare files for hours, log (and optionally stop on)
compare errors.

-- 
~Randy

Re: Network driver test suite?

[Cliff White]

> On Wed, 12 Jan 2005 09:14:01 -0800
> Craig Thomas <craiger@osdl.org> wrote:
> 
> > On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> > > On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> > > 
> > > > 
> > > > Would there be a desire for someone to collect the tests or at least
> > > > create an index to all their locations?  If so, then developers can
> > > > scan a library of potential tests to run against newly developed code.
> > > > 
> > > > OSDL can start incorporating some of these tests into their test
> > > > platform as well.
> > > 
> > > I would love to see a collection of the types of tests that should be
> > > performed.  As it appears now, there is nothing defined that a driver
> > > author should do to verify that their driver performs properly, or
> > > supports the right capabilities etc.  Some things may be difficult to
> > > automate, but simply having a checklist would be great.  For the things
> > > that can be automated, that would be even better.
> > 
> > Great.  We can do some of this.  I would like to ask, what mimimal
> > types of tests do you expect to execute for a driver?  If several
> > can respond to the types of testing they perform, we can start
> > a checklist.  Then, additional items can be added to fill in the
> > holes.  I've asked Cliff White of OSDL to help put this together.
> 
> There are two types of tests that would be easy to set up.
> First is a full exercise of all the possible API transitions through
> ifconfig, ip link, and ethtool. These could be covered without any
> traffic going through.
> 
> Then setup a standard test environment with a known good card and a
> crossover cable.  The test could then use raw (and/or packet generator)
> to send packets down good card to card to be verified.
> 
> Also testing, auto negotiation and transitions under load.

Okay, I'll see what i can do to start putting together a list of 
tests requirements. 
cliffw

Network driver "test suite"

[Benjamin Herrenschmidt]

Hi folks !

Does anybody knows of an existing kind of automated "test suite" for a
network/ethernet driver ?

IE. Something we could run both on the "tested" driver and a cross-over 
"known good" peer (possibly the latter set to promisc & no offload for
proper analysis), that would out the driver through a whole bunch of
tests, such as verifying the checksum offload on a various combinations
of headers lenghts and encapsulation, vlan stuff, multicast filters,
etc... ?

I've hacking on a driver recently and ended up "manually" testing a
bunch of these things using a palette of tools (iperf, nuttcp, some
multicast hack I have around, etc... along with tcpdump) but it feels
like this is the kind of things that could be greatly automated.

Cheers,
Ben.

Re: Network driver "test suite"

[Florian Fainelli]

Hi,

On 04/11/2017 05:16 PM, Benjamin Herrenschmidt wrote:
> Hi folks !
> 
> Does anybody knows of an existing kind of automated "test suite" for a
> network/ethernet driver ?
> 
> IE. Something we could run both on the "tested" driver and a cross-over 
> "known good" peer (possibly the latter set to promisc & no offload for
> proper analysis), that would out the driver through a whole bunch of
> tests, such as verifying the checksum offload on a various combinations
> of headers lenghts and encapsulation, vlan stuff, multicast filters,
> etc... ?

You could start with using LNST:

https://github.com/jpirko/lnst

and there is also Ostinato which is a great way to get access to
something IXIA-like, but all configurable in software through python
bindings. Andrew's dsa-tests make use of it, but they would not be
directly portable here [1].

[1]: https://github.com/lunn/dsa-tests
-- 
Florian

Re: Network driver "test suite"

[Corentin Labbe]

On Wed, Apr 12, 2017 at 10:16:17AM +1000, Benjamin Herrenschmidt wrote:
> Hi folks !
> 
> Does anybody knows of an existing kind of automated "test suite" for a
> network/ethernet driver ?
> 
> IE. Something we could run both on the "tested" driver and a cross-over 
> "known good" peer (possibly the latter set to promisc & no offload for
> proper analysis), that would out the driver through a whole bunch of
> tests, such as verifying the checksum offload on a various combinations
> of headers lenghts and encapsulation, vlan stuff, multicast filters,
> etc... ?
> 
> I've hacking on a driver recently and ended up "manually" testing a
> bunch of these things using a palette of tools (iperf, nuttcp, some
> multicast hack I have around, etc... along with tcpdump) but it feels
> like this is the kind of things that could be greatly automated.
> 
> Cheers,
> Ben.
> 

I have started to add some tests to kselftests (tools/testing/selftests/net/netdevice.sh)
The major intent is that thoses tests could be run without any user directive. (and so could be usefull in kernelci)
I just need to share the next serie of patch.

Regards

Re: Network driver "test suite"

[Benjamin Herrenschmidt]

On Tue, 2017-04-11 at 17:36 -0700, Florian Fainelli wrote:
> 
> You could start with using LNST:
> 
> https://github.com/jpirko/lnst
> 
> and there is also Ostinato which is a great way to get access to
> something IXIA-like, but all configurable in software through python
> bindings. Andrew's dsa-tests make use of it, but they would not be
> directly portable here [1].
> 
> [1]: https://github.com/lunn/dsa-tests

Thanks. I'll play with these when I have a bit of spare time !

Cheers,
Ben.