Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Bryce Harrington]

On Thu, 9 Dec 2004, Tony Reix wrote:
> { > The reason for the question is that a number of companies
> { > (CITI, Polyserv, Novell, NetApp, etc.) we've been talking with are doing
> { > testing of NFSv4, and have expressed interest in sharing of results,
> { > including a public bug tracker (both for posting bugs that are found,
> { > and for reviewing/validating/elaborating bugs that others post).
> {
> { I suspect the best thing would be to post them to the mailinglist, the
> { same way bugs are reported for just about any other subsystem.
>
> I think a mailing list is not appropriate for NFSv4 bug tracking.
> The testers need to have tools for checking that the bugs they have opened
> have make progress, and to add new information.
> The developpers need tools for managing all the bugs they are working on
> and to answer to testers.
> Managing defects is a pain we cannot go without. Just make it easier to live with.

As an example of how a bug tracker can really help in an open source
project, one project I am involved with (Inkscape) originally used
a mailing list for handling bugs, however it was rather infamous for
crashing, losing data, and big problems going unresolved for a long
time.

About a year and a half ago, the development team started making use of
a bug tracker (just the simple one on SourceForge), encouraging users to
report bugs there, and involving a lot of people in testing out the bug
reports both to verify that the bugs are recreatable and that the
patches solved the problem.  This was particularly useful at around
release time because we could track and review all of the critical bugs
and focus on getting those resolved (and verfied fixed) before
releasing.

Today, because of this effort, Inkscape's reputation is that it is very
robust and difficult to make crash.  Also, most of the common problems
people run into are documented (sometimes with work-arounds) in the bug
tracker.

It's true that most open source projects don't use bug trackers, and
certainly a fact that they're more work to manage, however I've seen
them have a very beneficial long term effect on projects and I would
love to see NFSv4 benefit this way as well.  If NFSv4 can gain a
reputation of being extremely robust, that could become a great selling
point for it.

However, I know that not everyone likes having to use bug trackers, and
that's understandable.  Maybe a good compromise would be for those of us
doing testing who want to track bugs, to do so, but continue using the
mailing lists as the primary mechanism.  That way if the bug tracking
approach fizzles out, it doesn't affect normal processes, but if it
succeeds then people can make heavier use of it.  Does this sound like a
feasible approach?

Bryce




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[J. Bruce Fields]

On Thu, Dec 09, 2004 at 01:53:53PM -0800, Bryce Harrington wrote:
> Maybe a good compromise would be for those of us doing testing who
> want to track bugs, to do so, but continue using the mailing lists as
> the primary mechanism.  That way if the bug tracking approach fizzles
> out, it doesn't affect normal processes, but if it succeeds then
> people can make heavier use of it.  Does this sound like a feasible
> approach?

I think that's the only way to go.  Some developers may just never look
at the bug database, so it may be necessary for other people to do
things like retest and close old bugs for them.

A bug database is certainly worth a try, though.

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Lever, Charles]

> On Thu, Dec 09, 2004 at 01:53:53PM -0800, Bryce Harrington wrote:
> > Maybe a good compromise would be for those of us doing testing who
> > want to track bugs, to do so, but continue using the=20
> mailing lists as
> > the primary mechanism.  That way if the bug tracking=20
> approach fizzles
> > out, it doesn't affect normal processes, but if it succeeds then
> > people can make heavier use of it.  Does this sound like a feasible
> > approach?
>=20
> I think that's the only way to go.  Some developers may just=20
> never look
> at the bug database, so it may be necessary for other people to do
> things like retest and close old bugs for them.
>=20
> A bug database is certainly worth a try, though.

we tried this once already.  i set up a cvstrac bug database here at
CITI.  you and i were the only ones who actually touched the thing, and
now it is long since dead.

will something like this work if some people use it and others don't?
there's no way i can assign a bug to someone who won't use the database.
that means we have to have two separate processes for tracking bugs
(email and database) to accomodate both types of developers.  and we
have to remember which process to use when submitting a bug report for
each developer.

i'd rather have a single common bug tracking system.  that common
denominator seems to be e-mail.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[J. Bruce Fields]

On Thu, Dec 09, 2004 at 02:24:27PM -0800, Lever, Charles wrote:
> we tried this once already.  i set up a cvstrac bug database here at
> CITI.  you and i were the only ones who actually touched the thing, and
> now it is long since dead.

Part of the problem was just that we didn't have the time to maintain
the database.  (It was crashing periodically at one point).

> will something like this work if some people use it and others don't?

I don't know.  We might be able to deal with a few who didn't if there
were testers willing to do some of the work for them.  E.g. retest old
bugs and email the list if necessary.

To me it seems at least worth a try, if there's people willing to spend
the time.

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Bryce Harrington]

On Thu, 9 Dec 2004, J. Bruce Fields wrote:

> On Thu, Dec 09, 2004 at 02:24:27PM -0800, Lever, Charles wrote:
> > we tried this once already.  i set up a cvstrac bug database here at
> > CITI.  you and i were the only ones who actually touched the thing, and
> > now it is long since dead.

Do you think any of the bugs in that system still valid?  If so would
it be possible for us to get a copy in some form (db dump or whatever)?

> Part of the problem was just that we didn't have the time to maintain
> the database.  (It was crashing periodically at one point).

The two options we have both have dedicated sysadmin staff attached.
The existing nfs bug tracker on SourceForge has the SF administration;
there's outages every once and a while but it tends to be good enough.
For the other option, Bugzilla hosted at OSDL, we can get the normal
admin support as we do for the Linux kernel Bugzilla.

So I think that keeping the bugs themselves up to date (updating status,
etc.) would be our principle challenge.  Keeping the system running
shouldn't be a problem.

> > will something like this work if some people use it and others don't?
>
> I don't know.  We might be able to deal with a few who didn't if there
> were testers willing to do some of the work for them.  E.g. retest old
> bugs and email the list if necessary.

I also don't know how it'll work, however at Inkscape we definitely have
some people who are more attentive to using it than others, and they
tend to make up for those who aren't.  Also, a lot of bugs don't ever
make it to the tracker before getting fixed, and that ends up being
fine; the main value of the tracker is for bugs that can't (or won't) be
fixed right away, so they don't get lost.

> To me it seems at least worth a try, if there's people willing to spend
> the time.

Same here.  I can dedicate time to getting it organized over the coming
months, if we have enough testers willing to make use of it.

Bryce



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[J. Bruce Fields]

On Thu, Dec 09, 2004 at 05:07:45PM -0800, Bryce Harrington wrote:
> On Thu, 9 Dec 2004, J. Bruce Fields wrote:
> > On Thu, Dec 09, 2004 at 02:24:27PM -0800, Lever, Charles wrote:
> > > we tried this once already.  i set up a cvstrac bug database here at
> > > CITI.  you and i were the only ones who actually touched the thing, and
> > > now it is long since dead.
> 
> Do you think any of the bugs in that system still valid?  If so would
> it be possible for us to get a copy in some form (db dump or whatever)?

It's pretty old, and would be a fair amount of work to work out what to
do with each bug.  Definitely not worth the trouble.

--b.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Lever, Charles]

> On Thu, 9 Dec 2004, J. Bruce Fields wrote:
>=20
> > On Thu, Dec 09, 2004 at 02:24:27PM -0800, Lever, Charles wrote:
> > > we tried this once already.  i set up a cvstrac bug
> > > database here at=20
> > > CITI.  you and i were the only ones who actually touched=20
> > > the thing, and now it is long since dead.
>=20
> Do you think any of the bugs in that system still valid?  If
> so would it be possible for us to get a copy in some form (db=20
> dump or whatever)?

many of those bugs were against a 2.4 backport of the NFSv4 code, and
are thus not relevant.  of the remaining bugs, only a few are of
interest, and those are widely known issues (for example, the
requirement for keyring support for session-based authentication, or the
requirement for ACL side-band protocol support in upstream kernels).
most or all of that information is documented in the priorities list at
client.linux-nfs.org.

i used the cvstrac database primarily for release planning.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Lever, Charles]

> So I think that keeping the bugs themselves up to date=20
> (updating status, etc.) would be our principle challenge.

my concern exactly.  i wonder about the effectiveness v. overhead ratio
because we won't have 100% developer participation.  i wonder who will
define and enforce the tracking process.  ie. bug databases are great in
theory, but in practice, sometimes there are big problems...

we may not "lose" bugs if we record them in the database, but we may
"lose" them because they get ignored after they are added.  what kind of
resources are available to manage (police) the process?  do we have a
triage mechanism for incoming bugs?

is this for just development testers, or do we accept bug reports from
users, NFS implementation partners, or the Linux distributors?  are we
interested solely in NFSv4 issues, or do we want reports on
v2/3-specific problems, side band protocol issues, problems occuring in
advanced development projects, or RPC SEC issues?  do we have a good
reason to exclude any of these bug types?

my position is that i would like this to work, as bruce would.  but i
would also like not to waste time if there is a good chance that a bug
database will be ineffective and eventually abandoned, as both citi's
NFSv4 cvstrac and SF's NFS bug database were.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Bryce Harrington]

On Fri, 10 Dec 2004, Lever, Charles wrote:
> > So I think that keeping the bugs themselves up to date
> > (updating status, etc.) would be our principle challenge.
>
> my concern exactly.  i wonder about the effectiveness v. overhead ratio
> because we won't have 100% developer participation.  i wonder who will
> define and enforce the tracking process.  ie. bug databases are great in
> theory, but in practice, sometimes there are big problems...

Well, anyway, I'm willing to put some time to help in this area if it
would be needed.  I had seen this listed on the NFS priorities page, and
we've heard the need expressed by several others, so am of the
assumption that there are people interested in using and participating
with it.  I've seen it work well elsewhere, like I mentioned, and feel
like it could be an effective tool for the NFS project.  It does work
best if developers view it as a useful tool, though.

A lot of open source projects do use bug trackers and find them handy,
but there's also a lot that don't use them, or that only use them
partially (like the Linux kernel).

> we may not "lose" bugs if we record them in the database, but we may
> "lose" them because they get ignored after they are added.  what kind of
> resources are available to manage (police) the process?  do we have a
> triage mechanism for incoming bugs?

Well, the approach I've used in the past is to establish four
criticalities (high, med, low, and unsorted).  Stuff that causes crashes
or data loss go into high.  Stuff that is really minor is set to low,
everything else is medium.  In the run-up to a release, developers try
to fix most of the high criticality items (with the rest left as 'known
issues').  The bug tracker isn't used for prioritization of which bugs
to fix (that's up to whomever is wrangling the releases to decide).

With the SourceForge bug tracker, it's also possible to CC all bugs and
bug reports to a mailing list, so I typically configure a
foo-tracking@lists.sf.net list for people to get notified about bugs.  I
think Bugzilla can be likewise configured.

For managing the process, I personally prefer treating it less as
something that needs 'policing' and more like a collective whiteboard
that the developers have strong ownership of and use to assist in
preparing for the release.  Most of the time it's left fairly informal,
with users, testers, and developers using it as a place to share info
about the bugs.  In the weeks leading up to the release, the list is
more carefully analyzed and tracked by the release coordinator.

> is this for just development testers, or do we accept bug reports from
> users, NFS implementation partners, or the Linux distributors?  are we
> interested solely in NFSv4 issues, or do we want reports on
> v2/3-specific problems, side band protocol issues, problems occuring in
> advanced development projects, or RPC SEC issues?  do we have a good
> reason to exclude any of these bug types?

OSDL's mainly interested in NFS on linux.  My own area of focus is v4.
However, I would think that a general purpose bug tracker with
categories for the different areas would could be logical.

The area I personally have seen bug trackers pay off the most is for
users reporting issues.  This can be especially useful with obscure or
hard to reproduce bugs, since it can be entered, and then someone else
(such as a tester) can later validate it and add more info, until it
becomes clear enough to a developer that it can be fixed.

I know that they can also be used for tracking day-to-day development
bugs, but since you can just email the developer directly this probably
wouldn't be an area where you'd be using a bug tracker.

> my position is that i would like this to work, as bruce would.  but i
> would also like not to waste time if there is a good chance that a bug
> database will be ineffective and eventually abandoned, as both citi's
> NFSv4 cvstrac and SF's NFS bug database were.

*Nod* Well, I'm willing to help with it if folks want to give it a shot.
I agree with you that if it's considered to be not worth the effort,
then time would be better spent elsewhere.  I'm optimistic that it could
be made to work, but I'm still just a newbie in this project so would
appreciate if others would make the call of whether it's worth trying.

Thanks,
Bryce





-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

RE: [Storage_sig] Re: Question on bug tracking for NFSv4 work

[Lever, Charles]

> On Fri, 10 Dec 2004, Lever, Charles wrote:
> > we may not "lose" bugs if we record them in the database, but we may
> > "lose" them because they get ignored after they are added. =20
> what kind of
> > resources are available to manage (police) the process?  do=20
> we have a
> > triage mechanism for incoming bugs?
>=20
> Well, the approach I've used in the past is to establish four
> criticalities (high, med, low, and unsorted).  Stuff that=20
> causes crashes
> or data loss go into high.  Stuff that is really minor is set to low,
> everything else is medium.  In the run-up to a release, developers try
> to fix most of the high criticality items (with the rest left=20
> as 'known
> issues').  The bug tracker isn't used for prioritization of which bugs
> to fix (that's up to whomever is wrangling the releases to decide).
>=20
> With the SourceForge bug tracker, it's also possible to CC=20
> all bugs and
> bug reports to a mailing list, so I typically configure a
> foo-tracking@lists.sf.net list for people to get notified=20
> about bugs.  I
> think Bugzilla can be likewise configured.
>=20
> For managing the process, I personally prefer treating it less as
> something that needs 'policing' and more like a collective whiteboard
> that the developers have strong ownership of and use to assist in
> preparing for the release.  Most of the time it's left fairly=20
> informal,
> with users, testers, and developers using it as a place to share info
> about the bugs.  In the weeks leading up to the release, the list is
> more carefully analyzed and tracked by the release coordinator.

there are several processes a bug tracker can help us with.  i think
some of these issues are more tractable than others.  to wit:

1.  personal bug to-do lists

    this will work well, i think, and is what most folks
    would like to have.  each tester and developer can
    chose to use or not use the bug tracker.  it will
    provide a venue for publishing bug lists.  and each
    developer or tester can track problems in his or her
    own source trees.

2.  release management

    this is probably impractical unless we have 100%
    participation and the maintainers want to use the
    tool this way.  we don't need or want group
    process here yet (do we?).  but it could work quite
    well if we require bugs & patches to be in the
    database as part of the process of submitting a
    patch to one of the maintainers.

3.  user/outside bug submission

    this is also probably impractical.  testers and
    developers can assign and chase their own bugs;
    users need an area expert to do this.  i expect
    that if a user reports a problem on the mailing
    list and a tester or developer wants to register
    it and track it themselves, then that's fine.
    (see below)

4.  others?

i think it would be best to track all bugs in the Linux NFS/RPC clients,
servers, and user-level utilities in one database.

> The area I personally have seen bug trackers pay off the most is for
> users reporting issues.  This can be especially useful with obscure or
> hard to reproduce bugs, since it can be entered, and then someone else
> (such as a tester) can later validate it and add more info, until it
> becomes clear enough to a developer that it can be fixed.

i think this is impractical without defining human resources who can
assign and track these problems, as we are making participation
optional, who can help users refine the problem description if it is
lacking, and who can filter out duplicates and pilot error.  also i
would rather that the distributors handle as much customer support as
possible.

imho the e-mail list is still the best place for this.


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. 
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs