From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from sfi-mx-2.v28.ch3.sourceforge.com ([172.29.28.122] helo=mx.sourceforge.net) by 335xhf1.ch3.sourceforge.com with esmtp (Exim 4.69) (envelope-from ) id 1MOUFg-0005Ev-91 for ltp-list@lists.sourceforge.net; Wed, 08 Jul 2009 10:19:48 +0000 Received: from e6.ny.us.ibm.com ([32.97.182.146]) by 72vjzd1.ch3.sourceforge.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69) id 1MOUFY-0005Sq-EX for ltp-list@lists.sourceforge.net; Wed, 08 Jul 2009 10:19:48 +0000 From: Subrata Modak In-Reply-To: <33307c790907071045v72e19614i571c36ad8af8062c@mail.gmail.com> References: <1246863519.2865.18.camel@localhost.localdomain> <33307c790907061137h3da12536q47517b1662498793@mail.gmail.com> <33307c790907071045v72e19614i571c36ad8af8062c@mail.gmail.com> Date: Wed, 08 Jul 2009 15:49:27 +0530 Message-Id: <1247048369.5405.35.camel@subratamodak.linux.ibm.com> Mime-Version: 1.0 Subject: Re: [LTP] [PATCH 1/2] Add latest LTP test in autotest Reply-To: subrata@linux.vnet.ibm.com List-Id: Linux Test Project General Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Errors-To: ltp-list-bounces@lists.sourceforge.net To: Martin Bligh , Mike Frysinger , ltp-list Cc: Autotest mailing list , Uri Lublin , Lucas Meneghel Rodrigues , sudhir kumar , kvm-devel T24gVHVlLCAyMDA5LTA3LTA3IGF0IDEwOjQ1IC0wNzAwLCBNYXJ0aW4gQmxpZ2ggd3JvdGU6Cj4g T24gVHVlLCBKdWwgNywgMjAwOSBhdCAxMjoyNCBBTSwgc3VkaGlyIGt1bWFyPHNtYWxpa3BoeUBn bWFpbC5jb20+IHdyb3RlOgo+ID4gT24gVHVlLCBKdWwgNywgMjAwOSBhdCAxMjowNyBBTSwgTWFy dGluIEJsaWdoPG1ibGlnaEBnb29nbGUuY29tPiB3cm90ZToKPiA+Pj4+IElzc3VlczogTFRQIGhh cyBhIGhpc3Rvcnkgb2Ygc29tZSBvZiB0aGUgdGVzdGNhc2VzIGdldHRpbmcgYnJva2VuLgo+ID4+ Cj4gPj4gUmlnaHQsIHRoYXQncyBhbHdheXMgdGhlIGNvbmNlcm4gd2l0aCBkb2luZyB0aGlzLgo+ ID4+Cj4gPj4+PiBBbnl3YXlzCj4gPj4+PiB0aGF0IGhhcyBub3RoaW5nIHRvIHdvcnJ5IGFib3V0 IHdpdGggcmVzcGVjdCB0byBhdXRvdGVzdC4gT25lIG9mIHRoZSBrbm93biBpc3N1ZQo+ID4+Pj4g aXMgYnJva2VuIG1lbW9yeSBjb250cm9sbGVyIGlzc3VlIHdpdGggbGF0ZXN0IGtlcm5lbHMoY2dy b3VwcyBhbmQgbWVtb3J5Cj4gPj4+PiByZXNvdXJjZSBjb250cm9sbGVyIGVuYWJsZWQga2VybmVs cykuIFRoZSB3b3JrYXJvdW5kIGZvciB0aGVtIEkgdXNlIGlzIHRvCj4gPj4+PiBkaXNhYmxlIG9y IGRlbGV0ZSB0aG9zZSB0ZXN0cyBmcm9tIGx0cCBzb3VyY2UgYW5kIHRhciBpdCBhZ2FpbiB3aXRo IHRoZSBzYW1lCj4gPj4+PiBuYW1lLiBUaG91Z2ggcGVvcGxlIG1pZ2h0IHVzZSBkaWZmZXJlbnQg d29ya2Fyb3VuZHMgZm9yIGl0Lgo+ID4+Cj4gPj4gT0ssIENhbiB3ZSBlbmNhcHN1bGF0ZSB0aGlz IGludG8gdGhlIHdyYXBwZXIgdGhvdWdoLCByYXRoZXIgdGhhbiBtYWtpbmcKPiA+PiBwZW9wbGUg ZG8gaXQgbWFudWFsbHk/IGluIHRoZSBleGlzdGluZyBsdHAucGF0Y2ggb3Igc29tZXRoaW5nPwo+ ID4+Cj4gPiBkZWZpbml0ZWx5IHdlIGNhbiBkbyB0aGF0LCBidXQgdGhhdCBuZWVkcyB0byBrbm93 IGFib3V0IGFsbCB0aGUgY29ybmVyCj4gPiBjYXNlcyBvZiBmYWlsdXJlLiBTbyBtYXkgYmUgd2Ug Y2FuIGNvbnRpbnVlIGVuaGFuY2luZyB0aGUgcGF0Y2ggYXMgcGVyCj4gPiB0aGUgZmFpbHVyZSBy ZXBvcnRzIG9uIGRpZmZlcmVudCBPU2VzLgo+ID4KPiA+IDEgbW9yZSB0aGluZyBJIHdhbnRlZCB0 byBzdGFydCBhIGRpc2N1c3Npb24gb24gTFRQIG1haWxpbmcgbGlzdCBpcyB0bwo+ID4gbWFrZSBh d2FyZSB0aGUgdGVzdGNhc2UgaWYgaXQgaXMgcnVubmluZyBvbiBhIHBoeXNpY2FsIGhvc3Qgb3Ig b24gYQo+ID4gZ3Vlc3Qoc2F5IEtWTSBndWVzdCkuIFRlc3RjYXNlcyBsaWtlIHBvd2VyIG1hbmFn ZW1lbnQsIGdyb3VwCj4gPiBzY2hlZHVsaW5nIGZhaXJuZXNzIGV0YyBkbyBub3QgbWFrZSBtdWNo IHNlbnNlIHRvIHJ1biBvbiBhIGd1ZXN0KGFzCj4gPiB0aGV5IHdpbGwgZmFpbCBvciBicmVhayku IFNvIEl0IGlzIGJldHRlciBmb3IgdGhlIHRlc3QgdG8gcmVjb2duaXNlCj4gPiB0aGUgZW52aXJv bm1lbnQgYW5kIG5vdCBleGVjdXRlIGlmIGl0IGlzIHVuZGVyIHZpcnR1YWxpemF0aW9uIGFuZCBp dAo+ID4gaXMgc3VwcG9zZWQgdG8gZmFpbCBvciBicmVhayB1bmRlciB0aGF0IGVudmlyb25tZW50 LiBEb2VzIHRoYXQgbWFrZQo+ID4gc2Vuc2UgdG8geW91IGFsc28gPwo+IAo+IFl1cCwgd2UgY2Fu IHBhc3MgYW4gZXhjbHVkZWQgdGVzdCBsaXN0LiBJIHJlYWxseSB3aXNoIHRoZXknZCBmaXggdGhl aXIKPiB0ZXN0cywgYnV0IEkndmUgYmVlbiBzYXlpbmcgdGhhdCBmb3IgNiB5ZWFycyBub3csIGFu ZCBpdCBoYXNuJ3QgaGFwcGVuZWQKPiB5ZXQgOy0oCgpJIHdvdWxkIHNsaWdodGx5IGRpc2FncmVl IHRvIHRoYXQuIDYgeWVhcnMgaXMgaGlzdG9yeS4gQnV0LCBoYXZlIHlvdQpyZWNlbnRseSBjaGVj a2VkIHdpdGggTFRQID8KClllcywgdGhlcmUgd2VyZSB0ZXN0cyBpbiBMVFAgd2hpY2ggd2VyZSBi cm9rZW4gYnkgZGVzaWduLiBBbmQgbWFueSBvZgp0aGVtIHdlcmUgZml4ZWQgaW4gbGFzdCBjb3Vy c2Ugb2YgeWVhcnMuIEFuZCBwcm9iYWJseSB0aGlzIGlzIHRoZSBmaXJzdAp0aW1lIGluIExUUMWb IGhpc3Rvcnkgd2hlbiBwZW9wbGUgZml4ZWQgYWxsIGJ1aWxkL2Jyb2tlbiBpc3N1ZXMgaW4gYQpt b250aC4gUmVjZW50bHkgcGVvcGxlIHN0b3BwZWQgY29tcGxhaW5pbmcgYWJvdXQgdGhlIHNvLWNh bGxlZCBicm9rZW4KaXNzdWVzLiBCdXQgaSBhbSBoZWFyaW5nIGl0IGFnYWluLiBDb3VsZCB5b3Ug cGxlYXNlIHBvaW50IHRvIHRoaXMKbWFpbGluZyBsaXN0IHRoZSBpc3N1ZXMgdGhhdCB5b3Ugc3Rp bGwgZmFjZS4gSSBhbSBzdXJlIHRoZXJlIGFyZSBub3cKdmVyeSBhY3RpdmUgbWVtYmVycyB3aG8g d291bGQgaGVscCB5b3UgZml4IHRoZW0uCgpGZXcgbW9yZSBvYnNlcnZhdGlvbnMuIElmIHRoZSB0 ZXN0IGNhc2VzIGRlc2lnbiBpcyBicm9rZW4sIHRoZSByZWFzb24KZm9yIGl0IHJlcG9ydGluZyBC Uk9LRU4sIHRoZW4gdGhpcyBpcyBhIGdlbnVpbmUgaXNzdWVzLiBXZSB3b3VsZCBuZWVkIHRvCmZp eCB0aGVtLiBCdXQsIHdoZW4gYSB0ZXN0IGNhc2UgcmVwb3J0cyBCUk9LRU4sIGlzIGl0IGp1c3Rp ZmllZCB0byBwdXQKdGhlIHdob2xlIGJsYW1lIG9uIHRoZSB0ZXN0IGNhc2UuIERpZCB3ZSB2ZXJp Znkgd2hldGhlciB0aGUgaXNzdWUgY2FuCmFsc28gYmUgd2l0aDoKCiAgICAgMS4gTGlicmFyaWVz LAogICAgIDIuIEtlcm5lbCAodGhlIHZlcnkgcHVycG9zZSBmb3Igd2hpY2ggdGVzdCBjYXNlIGlz IGRlc2lnbmVkKSwKCldoYXQgd2lsbCBiZSB0aGUgcG9pbnQgaWYgd2UganVzdCB3YW50IHRoZSB0 ZXN0IGNhc2UgdG8gUEFTUyA/PyBTbywgZG8Kd2Ugd2FudCB0byBwb2ludCB0aGF0IHRoZSBrZXJu ZWwgaXMgYWx3YXlzIE9LLCBhbmQgd2UganVzdCB3YW50IHRoZSB0ZXN0CmNhc2VzIHRvIHJlcG9y dCBQQVNTIGZvciBpdCwgZWxzZSB0aGUgdGVzdCBjYXNlIGlzIGJyb2tlbiA/CgpGaXhpbmcgYnVp bGQgaXNzdWVzOgpMVFAgY2Fubm90IHN0b3AgaW5oZXJpdGluZyBuZXcgdGVzdCBjYXNlcyB3aXRo IGVhY2ggcGFzc2luZyBkYXkuIEJ1dCwKaG93IGRvIHdlIGd1YXJhbnRlZSB0aGF0IHRoZSB0ZXN0 IHN1aXRlIGJ1aWxkIHdpbGwgc3VjY2VlZCBvbjoKICAgICAxLiBldmVyeSBEaXN0cm8sCiAgICAg Mi4gZXZlcnkgYXJjaCwKICAgICAzLiBldmVyeSBrZXJuZWwKClVubGlrZSB0aGUgTGludXggS2Vy bmVsLCB3aGljaCBjYXJyaWVzIGFsbCB0aGUgc3R1ZmYgbmVlZGVkIHRvIGJ1aWxkCml0c2VsZiBv biBhbnkgYXJjaCwgYW4gdXNlciBsYW5kIHByb2plY3QgbGlrZSBMVFAgaXMgY29tcGxldGVseQpk ZXBlbmRhbnQgb24gdGhlIHN5c3RlbSBoZWFkZXJzL2xpYnJhcmllcyB0byBjb21wbGV0ZSBpdMWb IGJ1aWxkLiBTbywKd2hlbiB0aGVzZSBzdHVmZiBhcmUgbm90IGNvbnNpc3RlbnQgYWNyb3NzIHRo ZSBhcmNoaXRlY3R1cmUgYW5kIERpc3RybwpnZW9ncmFwaHksIHNvLCBob3cgZG8gd2Ugc29sdmUg dGhlIHByb2JsZW0uIEFuZCBpbiBtb3N0IG9mIHRoZSBpbnN0YW5jZQp3ZSBmaW5kIHRoYXQgNTAl IG9mIG91ciBlZmZvcnQvY29kZSBnb2VzIGluIGZpeGluZyB0aGlzIGRlcGVuZGVuY2llcywKcmF0 aGVyIHRoYW4gaW50byB0aGUgYWN0dWFsIGtlcm5lbCB0ZXN0IGNvZGUuIEFuZCwgeWVzLCB0byBj YXRlciB0byB0aGUKbmVlZCwgTFRQIGhhcyBlbWJyYWNlZCB0aGUgQVVUT0NPTkYgZnJhbWV3b3Jr LgoKRml4aW5nIFJlYWwgdGVzdCBjYXNlIEJST0tFTiBpc3N1ZXM6CkhlcmUgcGxheXMgb25lIG9m IHRoZSB1bmZvcnR1bmF0ZSBkcmF3YmFja3Mgb2YgdGhlIE9wZW4gU291cmNlIFByb2plY3RzLgpF dmVyeXRoaW5nIGRvZXMgbm90IGdldCBmdW5kZWQgZm9yIGV0ZXJuaXR5LiBXaGVuIHRlc3QgY2Fz ZXMgd2VyZQpjaGVja2VkIGluIGZvciBzb21lIGZlYXR1cmUgaW4gdGhlIGtlcm5lbCwgdGhlIHBy b2plY3Qgc2hvdWxkIGhhdmUgYmVlbgpmdW5kZWQuIE5vdywgd2hlbiB0aGUgQUJJcy9mdW5jdGlv bmFsaXR5IGluIHRoZSBrZXJuZWwgY2hhbmdlcywgdGhlCm9yaWdpbmFsIGF1dGhvciBvZiB0aG9z ZSB0ZXN0IGNhc2VzIGlzIG5vIG1vcmUgZnVuZGVkIHRvIG1ha2UgdGhlCmNvcnJlc3BvbmRpbmcg Y2hhbmdlcyBpbiB0aGUgY29uY2VybmVkIHRlc3RzLiBTbywgdGhpcyBjYW4gYmUgc29sdmVkCm9u bHkgd2hlbiBzb21lYm9keSByZXBvcnRzIHN1Y2ggaXNzdWVzIHdpdGggYSBwYXRjaCByYXRoZXIg dGhhbiBzZW5kaW5nCnJlcGVhdGVkIHJlbWluZGVycyB0aGF0IHN1Y2gtYW5kLXN1Y2ggdGVzdCBj YXNlcyBhcmUgYnJva2VuLiBJIGRvIG5vdApoYXZlIGFueSBvdGhlciBpZGVhIG9mIGhvdyB0aGlz IGNhbiBiZSBzb2x2ZWQuCgpJIGFtIG5vdCBzdXJlLCBpZiBpIGFtIGFibGUgdG8gcmVzb2x2ZSBh bGwgeW91ciBkb3VidHMvY29uY2VybnMKY29tcGxldGVseS4gTWlrZS9vdGhlcnMgZnJvbSBMVFAg bWFpbGluZyBsaXN0IHdpbGwgbWF5IGFsc28gYmUgb2Ygc29tZQpoZWxwLgoKTWlrZSwKCldoYXQg ZG8geW91IHNheSA/CgpSZWdhcmRzLS0KU3VicmF0YQoKPiBfX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fXwo+IEF1dG90ZXN0IG1haWxpbmcgbGlzdAo+IEF1dG90 ZXN0QHRlc3Qua2VybmVsLm9yZwo+IGh0dHA6Ly90ZXN0Lmtlcm5lbC5vcmcvY2dpLWJpbi9tYWls bWFuL2xpc3RpbmZvL2F1dG90ZXN0CgoKLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCkVudGVyIHRoZSBC bGFja0JlcnJ5IERldmVsb3BlciBDaGFsbGVuZ2UgIApUaGlzIGlzIHlvdXIgY2hhbmNlIHRvIHdp biB1cCB0byAkMTAwLDAwMCBpbiBwcml6ZXMhIEZvciBhIGxpbWl0ZWQgdGltZSwgCnZlbmRvcnMg c3VibWl0dGluZyBuZXcgYXBwbGljYXRpb25zIHRvIEJsYWNrQmVycnkgQXBwIFdvcmxkKFRNKSB3 aWxsIGhhdmUKdGhlIG9wcG9ydHVuaXR5IHRvIGVudGVyIHRoZSBCbGFja0JlcnJ5IERldmVsb3Bl ciBDaGFsbGVuZ2UuIFNlZSBmdWxsIHByaXplICAKZGV0YWlscyBhdDogaHR0cDovL3Auc2YubmV0 L3NmdS9DaGFsbGVuZ2UKX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX18KTHRwLWxpc3QgbWFpbGluZyBsaXN0Ckx0cC1saXN0QGxpc3RzLnNvdXJjZWZvcmdlLm5l dApodHRwczovL2xpc3RzLnNvdXJjZWZvcmdlLm5ldC9saXN0cy9saXN0aW5mby9sdHAtbGlzdAo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: Subrata Modak Subject: Re: [PATCH 1/2] Add latest LTP test in autotest Date: Wed, 08 Jul 2009 15:49:27 +0530 Message-ID: <1247048369.5405.35.camel@subratamodak.linux.ibm.com> References: <1246863519.2865.18.camel@localhost.localdomain> <33307c790907061137h3da12536q47517b1662498793@mail.gmail.com> <33307c790907071045v72e19614i571c36ad8af8062c@mail.gmail.com> Reply-To: subrata@linux.vnet.ibm.com Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Cc: sudhir kumar , Autotest mailing list , Lucas Meneghel Rodrigues , Uri Lublin , kvm-devel To: Martin Bligh , Mike Frysinger , ltp-list Return-path: Received: from e8.ny.us.ibm.com ([32.97.182.138]:34606 "EHLO e8.ny.us.ibm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752958AbZGHKTm (ORCPT ); Wed, 8 Jul 2009 06:19:42 -0400 Received: from d01relay02.pok.ibm.com (d01relay02.pok.ibm.com [9.56.227.234]) by e8.ny.us.ibm.com (8.13.1/8.13.1) with ESMTP id n68A8mFN024511 for ; Wed, 8 Jul 2009 06:08:48 -0400 Received: from d01av02.pok.ibm.com (d01av02.pok.ibm.com [9.56.224.216]) by d01relay02.pok.ibm.com (8.13.8/8.13.8/NCO v9.2) with ESMTP id n68AJeUD169354 for ; Wed, 8 Jul 2009 06:19:40 -0400 Received: from d01av02.pok.ibm.com (loopback [127.0.0.1]) by d01av02.pok.ibm.com (8.12.11.20060308/8.13.3) with ESMTP id n68AH91S019700 for ; Wed, 8 Jul 2009 06:17:10 -0400 In-Reply-To: <33307c790907071045v72e19614i571c36ad8af8062c@mail.gmail.com> Sender: kvm-owner@vger.kernel.org List-ID: On Tue, 2009-07-07 at 10:45 -0700, Martin Bligh wrote: > On Tue, Jul 7, 2009 at 12:24 AM, sudhir kumar wr= ote: > > On Tue, Jul 7, 2009 at 12:07 AM, Martin Bligh wr= ote: > >>>> Issues: LTP has a history of some of the testcases getting broke= n. > >> > >> Right, that's always the concern with doing this. > >> > >>>> Anyways > >>>> that has nothing to worry about with respect to autotest. One of= the known issue > >>>> is broken memory controller issue with latest kernels(cgroups an= d memory > >>>> resource controller enabled kernels). The workaround for them I = use is to > >>>> disable or delete those tests from ltp source and tar it again w= ith the same > >>>> name. Though people might use different workarounds for it. > >> > >> OK, Can we encapsulate this into the wrapper though, rather than m= aking > >> people do it manually? in the existing ltp.patch or something? > >> > > definitely we can do that, but that needs to know about all the cor= ner > > cases of failure. So may be we can continue enhancing the patch as = per > > the failure reports on different OSes. > > > > 1 more thing I wanted to start a discussion on LTP mailing list is = to > > make aware the testcase if it is running on a physical host or on a > > guest(say KVM guest). Testcases like power management, group > > scheduling fairness etc do not make much sense to run on a guest(as > > they will fail or break). So It is better for the test to recognise > > the environment and not execute if it is under virtualization and i= t > > is supposed to fail or break under that environment. Does that make > > sense to you also ? >=20 > Yup, we can pass an excluded test list. I really wish they'd fix thei= r > tests, but I've been saying that for 6 years now, and it hasn't happe= ned > yet ;-( I would slightly disagree to that. 6 years is history. But, have you recently checked with LTP ? Yes, there were tests in LTP which were broken by design. And many of them were fixed in last course of years. And probably this is the first time in LTP=C5=9B history when people fixed all build/broken issues in = a month. Recently people stopped complaining about the so-called broken issues. But i am hearing it again. Could you please point to this mailing list the issues that you still face. I am sure there are now very active members who would help you fix them. =46ew more observations. If the test cases design is broken, the reason for it reporting BROKEN, then this is a genuine issues. We would need t= o fix them. But, when a test case reports BROKEN, is it justified to put the whole blame on the test case. Did we verify whether the issue can also be with: 1. Libraries, 2. Kernel (the very purpose for which test case is designed), What will be the point if we just want the test case to PASS ?? So, do we want to point that the kernel is always OK, and we just want the tes= t cases to report PASS for it, else the test case is broken ? =46ixing build issues: LTP cannot stop inheriting new test cases with each passing day. But, how do we guarantee that the test suite build will succeed on: 1. every Distro, 2. every arch, 3. every kernel Unlike the Linux Kernel, which carries all the stuff needed to build itself on any arch, an user land project like LTP is completely dependant on the system headers/libraries to complete it=C5=9B build. S= o, when these stuff are not consistent across the architecture and Distro geography, so, how do we solve the problem. And in most of the instance we find that 50% of our effort/code goes in fixing this dependencies, rather than into the actual kernel test code. And, yes, to cater to the need, LTP has embraced the AUTOCONF framework. =46ixing Real test case BROKEN issues: Here plays one of the unfortunate drawbacks of the Open Source Projects= =2E Everything does not get funded for eternity. When test cases were checked in for some feature in the kernel, the project should have been funded. Now, when the ABIs/functionality in the kernel changes, the original author of those test cases is no more funded to make the corresponding changes in the concerned tests. So, this can be solved only when somebody reports such issues with a patch rather than sending repeated reminders that such-and-such test cases are broken. I do not have any other idea of how this can be solved. I am not sure, if i am able to resolve all your doubts/concerns completely. Mike/others from LTP mailing list will may also be of some help. Mike, What do you say ? Regards-- Subrata > _______________________________________________ > Autotest mailing list > Autotest@test.kernel.org > http://test.kernel.org/cgi-bin/mailman/listinfo/autotest