Re: Fw: Network driver test suite?

Re: Fw: Network driver test suite?

[Craig Thomas]

On Wed, 2005-01-05 at 15:26, Stephen Hemminger wrote:
> Begin forwarded message:
> 
> Date: Wed, 05 Jan 2005 15:19:46 -0500
> From: David Hollis <dhollis@davehollis.com>
> To: Netdev <netdev@oss.sgi.com>
> Subject: Network driver test suite?
> 
> 
> Is there any kind of test suite (automated or list of tests) available
> for testing Linux network drivers?  I'm completing the addition of a few
> new USB ethernet devices and would like to able to test all possible
> scenarios instead of coming across them piece-meal in the future.  I'm
> not a big fan of the "works on my box" testing that I'm doing now.  I'm
> sure there are all kinds of things that I'm not testing that aren't
> everyday types of things such as VLANs, multicasting, various
> ethtool/mii-tool things, large packets, etc.

Would there be a desire for someone to collect the tests or at least
create an index to all their locations?  If so, then developers can
scan a library of potential tests to run against newly developed code.

OSDL can start incorporating some of these tests into their test
platform as well.


> 
> If there isn't anything like this, maybe it would be a useful thing to
> develop?  At a minimum, maybe to set the treshhold for minimum features
> that drivers support and the like.

Re: Fw: Network driver test suite?

[David Hollis]

[-- Attachment #1: Type: text/plain, Size: 825 bytes --]

On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:

> 
> Would there be a desire for someone to collect the tests or at least
> create an index to all their locations?  If so, then developers can
> scan a library of potential tests to run against newly developed code.
> 
> OSDL can start incorporating some of these tests into their test
> platform as well.

I would love to see a collection of the types of tests that should be
performed.  As it appears now, there is nothing defined that a driver
author should do to verify that their driver performs properly, or
supports the right capabilities etc.  Some things may be difficult to
automate, but simply having a checklist would be great.  For the things
that can be automated, that would be even better.

-- 
David Hollis <dhollis@davehollis.com>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Fw: Network driver test suite?

[Craig Thomas]

On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> 
> > 
> > Would there be a desire for someone to collect the tests or at least
> > create an index to all their locations?  If so, then developers can
> > scan a library of potential tests to run against newly developed code.
> > 
> > OSDL can start incorporating some of these tests into their test
> > platform as well.
> 
> I would love to see a collection of the types of tests that should be
> performed.  As it appears now, there is nothing defined that a driver
> author should do to verify that their driver performs properly, or
> supports the right capabilities etc.  Some things may be difficult to
> automate, but simply having a checklist would be great.  For the things
> that can be automated, that would be even better.

Great.  We can do some of this.  I would like to ask, what mimimal
types of tests do you expect to execute for a driver?  If several
can respond to the types of testing they perform, we can start
a checklist.  Then, additional items can be added to fill in the
holes.  I've asked Cliff White of OSDL to help put this together.

Re: Network driver test suite?

[Stephen Hemminger]

On Wed, 12 Jan 2005 09:14:01 -0800
Craig Thomas <craiger@osdl.org> wrote:

> On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> > On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> > 
> > > 
> > > Would there be a desire for someone to collect the tests or at least
> > > create an index to all their locations?  If so, then developers can
> > > scan a library of potential tests to run against newly developed code.
> > > 
> > > OSDL can start incorporating some of these tests into their test
> > > platform as well.
> > 
> > I would love to see a collection of the types of tests that should be
> > performed.  As it appears now, there is nothing defined that a driver
> > author should do to verify that their driver performs properly, or
> > supports the right capabilities etc.  Some things may be difficult to
> > automate, but simply having a checklist would be great.  For the things
> > that can be automated, that would be even better.
> 
> Great.  We can do some of this.  I would like to ask, what mimimal
> types of tests do you expect to execute for a driver?  If several
> can respond to the types of testing they perform, we can start
> a checklist.  Then, additional items can be added to fill in the
> holes.  I've asked Cliff White of OSDL to help put this together.

There are two types of tests that would be easy to set up.
First is a full exercise of all the possible API transitions through
ifconfig, ip link, and ethtool. These could be covered without any
traffic going through.

Then setup a standard test environment with a known good card and a
crossover cable.  The test could then use raw (and/or packet generator)
to send packets down good card to card to be verified.

Also testing, auto negotiation and transitions under load.

Re: Fw: Network driver test suite?

[Ben Greear]

Craig Thomas wrote:
> On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> 
>>On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
>>
>>
>>>Would there be a desire for someone to collect the tests or at least
>>>create an index to all their locations?  If so, then developers can
>>>scan a library of potential tests to run against newly developed code.
>>>
>>>OSDL can start incorporating some of these tests into their test
>>>platform as well.
>>
>>I would love to see a collection of the types of tests that should be
>>performed.  As it appears now, there is nothing defined that a driver
>>author should do to verify that their driver performs properly, or
>>supports the right capabilities etc.  Some things may be difficult to
>>automate, but simply having a checklist would be great.  For the things
>>that can be automated, that would be even better.
> 
> 
> Great.  We can do some of this.  I would like to ask, what mimimal
> types of tests do you expect to execute for a driver?  If several
> can respond to the types of testing they perform, we can start
> a checklist.  Then, additional items can be added to fill in the
> holes.  I've asked Cliff White of OSDL to help put this together.

My wishlist includes:

set and verify all supported link speeds (auto-negotiate, 10Mbps fixed, 100Mbps fixed,
1Gbps, full/half duplex, etc)
set & verify various MTUs

At each speed, generate maximum amount of packets:
    tx only
    rx only
    tx + rx
    Could use pktgen for this as it does not ARP or do other protocol things
    would complicate rx-only and tx-only testing.
    Could also do randomized packet sizes or step through a bunch of different
    sizes.
    Could randomize rates and other things with pktgen as well.
    Determine number of dropped & errored packets at each phase.
      This should be verified by counting the number of packets transmitted
      v/s received and coorelated against any drop/error counters that the
      driver reports.

Generate TCP & UDP traffic at various speeds to make sure it handles
protocols correctly too.

Run a similar battery of tests against 802.1Q VLANS on the
interfaces in question.


I have a proprietary tool and some GPL kernel patches that can do most of this.
A bash/perl script utilizing ethtool and and pktgen (especially a version of
pktgen similar to the one in my patches which also receives packets and reports
statistics on them) and a few other tcp & udp generating tools could also do
this (and in a more transparent manner, probably.)

Please contact me off the list if you want to discuss free use of the
proprietary tool.

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: Network driver test suite?

[Randy.Dunlap]

Stephen Hemminger wrote:
> On Wed, 12 Jan 2005 09:14:01 -0800
> Craig Thomas <craiger@osdl.org> wrote:
> 
> 
>>On Wed, 2005-01-12 at 08:24, David Hollis wrote:
>>
>>>On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
>>>
>>>
>>>>Would there be a desire for someone to collect the tests or at least
>>>>create an index to all their locations?  If so, then developers can
>>>>scan a library of potential tests to run against newly developed code.
>>>>
>>>>OSDL can start incorporating some of these tests into their test
>>>>platform as well.
>>>
>>>I would love to see a collection of the types of tests that should be
>>>performed.  As it appears now, there is nothing defined that a driver
>>>author should do to verify that their driver performs properly, or
>>>supports the right capabilities etc.  Some things may be difficult to
>>>automate, but simply having a checklist would be great.  For the things
>>>that can be automated, that would be even better.
>>
>>Great.  We can do some of this.  I would like to ask, what mimimal
>>types of tests do you expect to execute for a driver?  If several
>>can respond to the types of testing they perform, we can start
>>a checklist.  Then, additional items can be added to fill in the
>>holes.  I've asked Cliff White of OSDL to help put this together.
> 
> 
> There are two types of tests that would be easy to set up.
> First is a full exercise of all the possible API transitions through
> ifconfig, ip link, and ethtool. These could be covered without any
> traffic going through.
> 
> Then setup a standard test environment with a known good card and a
> crossover cable.  The test could then use raw (and/or packet generator)
> to send packets down good card to card to be verified.
> 
> Also testing, auto negotiation and transitions under load.

Other than API testing, stats interface testing, & link/speed 
verification, the most useful test that I ever did in NIC driver
development (Natl Semi & Intel) was just traffic saturation:
copy and compare files for hours, log (and optionally stop on)
compare errors.

-- 
~Randy

Re: Network driver test suite?

[Cliff White]

> On Wed, 12 Jan 2005 09:14:01 -0800
> Craig Thomas <craiger@osdl.org> wrote:
> 
> > On Wed, 2005-01-12 at 08:24, David Hollis wrote:
> > > On Tue, 2005-01-11 at 17:32 -0800, Craig Thomas wrote:
> > > 
> > > > 
> > > > Would there be a desire for someone to collect the tests or at least
> > > > create an index to all their locations?  If so, then developers can
> > > > scan a library of potential tests to run against newly developed code.
> > > > 
> > > > OSDL can start incorporating some of these tests into their test
> > > > platform as well.
> > > 
> > > I would love to see a collection of the types of tests that should be
> > > performed.  As it appears now, there is nothing defined that a driver
> > > author should do to verify that their driver performs properly, or
> > > supports the right capabilities etc.  Some things may be difficult to
> > > automate, but simply having a checklist would be great.  For the things
> > > that can be automated, that would be even better.
> > 
> > Great.  We can do some of this.  I would like to ask, what mimimal
> > types of tests do you expect to execute for a driver?  If several
> > can respond to the types of testing they perform, we can start
> > a checklist.  Then, additional items can be added to fill in the
> > holes.  I've asked Cliff White of OSDL to help put this together.
> 
> There are two types of tests that would be easy to set up.
> First is a full exercise of all the possible API transitions through
> ifconfig, ip link, and ethtool. These could be covered without any
> traffic going through.
> 
> Then setup a standard test environment with a known good card and a
> crossover cable.  The test could then use raw (and/or packet generator)
> to send packets down good card to card to be verified.
> 
> Also testing, auto negotiation and transitions under load.

Okay, I'll see what i can do to start putting together a list of 
tests requirements. 
cliffw