* Re: [LTP] [PATCH 1/2] Add latest LTP test in autotest
2009-07-08 10:19 ` [LTP] [PATCH 1/2] Add latest LTP test in autotest Subrata Modak
@ 2009-07-08 15:05 ` Martin Bligh
2009-07-08 23:05 ` Mike Frysinger
1 sibling, 0 replies; 3+ messages in thread
From: Martin Bligh @ 2009-07-08 15:05 UTC (permalink / raw)
To: subrata
Cc: ltp-list, Mike Frysinger, kvm-devel, Lucas Meneghel Rodrigues,
Autotest mailing list, Uri Lublin, sudhir kumar
>> Yup, we can pass an excluded test list. I really wish they'd fix their
>> tests, but I've been saying that for 6 years now, and it hasn't happened
>> yet ;-(
>
> I would slightly disagree to that. 6 years is history. But, have you
> recently checked with LTP ?
I hate to be completely cynical about this, but that's exactly the same
message I get every year.
Yes, absolute, the best thing would be for someone to run all the tests,
work through all the problems, categorize them as kernel / library / distro,
and get each of them fixed. However, it's a fair chunk of work that I don't
have time to do.
So all I'm saying is that I know which of the current tests we have issues
with, and I don't want to upgrade LTP without a new set of data, and that
work being done. From previous experience, I would be extremely
surprised if there's not at least one new problem, and I'm not just going
to dump that on users.
Does the LTP project do this itself on a regular basis ... ie are you running
LTP against the latest kernel (or even some known stable kernel) and
seeing which tests are broken? If you can point me to that, I'd have much
more faith about picking this up ...
Up until this point we've not even managed to agree that PASS means
"ran as expected" and FAIL meant "something is wrong". LTP always
had "expected failures" which seems like a completely broken model
to me.
M.
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [LTP] [PATCH 1/2] Add latest LTP test in autotest
2009-07-08 10:19 ` [LTP] [PATCH 1/2] Add latest LTP test in autotest Subrata Modak
2009-07-08 15:05 ` Martin Bligh
@ 2009-07-08 23:05 ` Mike Frysinger
1 sibling, 0 replies; 3+ messages in thread
From: Mike Frysinger @ 2009-07-08 23:05 UTC (permalink / raw)
To: subrata
Cc: ltp-list, Uri Lublin, kvm-devel, Lucas Meneghel Rodrigues,
Martin Bligh, Autotest mailing list, sudhir kumar
[-- Attachment #1.1: Type: text/plain, Size: 5023 bytes --]
On Wednesday 08 July 2009 06:19:27 Subrata Modak wrote:
> On Tue, 2009-07-07 at 10:45 -0700, Martin Bligh wrote:
> > On Tue, Jul 7, 2009 at 12:24 AM, sudhir kumar<smalikphy@gmail.com> wrote:
> > > On Tue, Jul 7, 2009 at 12:07 AM, Martin Bligh<mbligh@google.com> wrote:
> > >>>> Issues: LTP has a history of some of the testcases getting broken.
> > >>
> > >> Right, that's always the concern with doing this.
> > >>
> > >>>> Anyways
> > >>>> that has nothing to worry about with respect to autotest. One of the
> > >>>> known issue is broken memory controller issue with latest
> > >>>> kernels(cgroups and memory resource controller enabled kernels). The
> > >>>> workaround for them I use is to disable or delete those tests from
> > >>>> ltp source and tar it again with the same name. Though people might
> > >>>> use different workarounds for it.
> > >>
> > >> OK, Can we encapsulate this into the wrapper though, rather than
> > >> making people do it manually? in the existing ltp.patch or something?
> > >
> > > definitely we can do that, but that needs to know about all the corner
> > > cases of failure. So may be we can continue enhancing the patch as per
> > > the failure reports on different OSes.
> > >
> > > 1 more thing I wanted to start a discussion on LTP mailing list is to
> > > make aware the testcase if it is running on a physical host or on a
> > > guest(say KVM guest). Testcases like power management, group
> > > scheduling fairness etc do not make much sense to run on a guest(as
> > > they will fail or break). So It is better for the test to recognise
> > > the environment and not execute if it is under virtualization and it
> > > is supposed to fail or break under that environment. Does that make
> > > sense to you also ?
> >
> > Yup, we can pass an excluded test list. I really wish they'd fix their
> > tests, but I've been saying that for 6 years now, and it hasn't happened
> > yet ;-(
>
> I would slightly disagree to that. 6 years is history. But, have you
> recently checked with LTP ?
>
> Yes, there were tests in LTP which were broken by design. And many of
> them were fixed in last course of years. And probably this is the first
> time in LTPś history when people fixed all build/broken issues in a
> month. Recently people stopped complaining about the so-called broken
> issues. But i am hearing it again. Could you please point to this
> mailing list the issues that you still face. I am sure there are now
> very active members who would help you fix them.
>
> Few more observations. If the test cases design is broken, the reason
> for it reporting BROKEN, then this is a genuine issues. We would need to
> fix them. But, when a test case reports BROKEN, is it justified to put
> the whole blame on the test case. Did we verify whether the issue can
> also be with:
>
> 1. Libraries,
> 2. Kernel (the very purpose for which test case is designed),
>
> What will be the point if we just want the test case to PASS ?? So, do
> we want to point that the kernel is always OK, and we just want the test
> cases to report PASS for it, else the test case is broken ?
>
> Fixing build issues:
> LTP cannot stop inheriting new test cases with each passing day. But,
> how do we guarantee that the test suite build will succeed on:
> 1. every Distro,
> 2. every arch,
> 3. every kernel
>
> Unlike the Linux Kernel, which carries all the stuff needed to build
> itself on any arch, an user land project like LTP is completely
> dependant on the system headers/libraries to complete itś build. So,
> when these stuff are not consistent across the architecture and Distro
> geography, so, how do we solve the problem. And in most of the instance
> we find that 50% of our effort/code goes in fixing this dependencies,
> rather than into the actual kernel test code. And, yes, to cater to the
> need, LTP has embraced the AUTOCONF framework.
>
> Fixing Real test case BROKEN issues:
> Here plays one of the unfortunate drawbacks of the Open Source Projects.
> Everything does not get funded for eternity. When test cases were
> checked in for some feature in the kernel, the project should have been
> funded. Now, when the ABIs/functionality in the kernel changes, the
> original author of those test cases is no more funded to make the
> corresponding changes in the concerned tests. So, this can be solved
> only when somebody reports such issues with a patch rather than sending
> repeated reminders that such-and-such test cases are broken. I do not
> have any other idea of how this can be solved.
>
> I am not sure, if i am able to resolve all your doubts/concerns
> completely. Mike/others from LTP mailing list will may also be of some
> help.
not much else to say. it's an open source project and if you arent willing to
contribute to fixing it but rather just sit back and complain, then you cant
really be surprised when things dont move.
-mike
[-- Attachment #1.2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
[-- Attachment #2: Type: text/plain, Size: 389 bytes --]
------------------------------------------------------------------------------
Enter the BlackBerry Developer Challenge
This is your chance to win up to $100,000 in prizes! For a limited time,
vendors submitting new applications to BlackBerry App World(TM) will have
the opportunity to enter the BlackBerry Developer Challenge. See full prize
details at: http://p.sf.net/sfu/Challenge
[-- Attachment #3: Type: text/plain, Size: 155 bytes --]
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
^ permalink raw reply [flat|nested] 3+ messages in thread