* [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. @ 2011-03-02 11:40 Shubham Goyal 2011-03-02 12:06 ` Paolo Ciarrocchi 0 siblings, 1 reply; 14+ messages in thread From: Shubham Goyal @ 2011-03-02 11:40 UTC (permalink / raw) To: ltp-list, linux-kernel; +Cc: torvalds, akpm, yanegomi, vapier, subrata, chrubis Hi, The Linux Test Project test suite has been released for the month of FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has been multiple changes for building/installing the test suite after the recent changes in Makefile infrastructure. The latest version of the test-suite contains 3000+ tests for the Linux OS and can be found at: http://ltp.sourceforge.net/, Latest happenings in LTP can also be found at: http://ltp.sourceforge.net/wiki/, http://ltp.sourceforge.net/wikiArchives.php, and, IRC: irc.freenode.org #ltp. Our web site also contains other information such as: - A Linux test tools matrix - Technical papers - How To's on Linux testing - Code coverage analysis tool. We would encourage the community to post results to ltp-results@lists.sf.net, patches, new tests, bugs or comments/questions toltp-list@lists.sf.net, http://sourceforge.net/tracker/?func=add&group_id=3382&atid=103382 (for New Bug(s)), http://sourceforge.net/tracker/?func=add&group_id=3382&atid=303382 (for New Patch(s)), http://sourceforge.net/tracker/?func=add&group_id=3382&atid=353382 (for New Feature Request(s)) **Note: Some pending patches will be checked in for next monthś release. Happy testing, Regards, Shubham ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-02 11:40 [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011 Shubham Goyal @ 2011-03-02 12:06 ` Paolo Ciarrocchi 2011-03-02 12:23 ` Subrata Modak 0 siblings, 1 reply; 14+ messages in thread From: Paolo Ciarrocchi @ 2011-03-02 12:06 UTC (permalink / raw) To: Shubham Goyal Cc: ltp-list, linux-kernel, torvalds, akpm, yanegomi, vapier, subrata, chrubis On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal <shubham@linux.vnet.ibm.com> wrote: > Hi, > > The Linux Test Project test suite has been released for the month of > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has > been multiple changes for building/installing the test suite after the > recent changes in Makefile infrastructure. Wouldn't make sense to integrate this test suite in the kernel source tree? Regards, Paolo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-02 12:06 ` Paolo Ciarrocchi @ 2011-03-02 12:23 ` Subrata Modak 2011-03-02 19:45 ` Garrett Cooper 2011-03-03 1:52 ` [LTP] " CAI Qian 0 siblings, 2 replies; 14+ messages in thread From: Subrata Modak @ 2011-03-02 12:23 UTC (permalink / raw) To: Paolo Ciarrocchi Cc: Shubham Goyal, ltp-list, linux-kernel, torvalds, akpm, yanegomi, vapier, chrubis On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote: > On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal > <shubham@linux.vnet.ibm.com> wrote: > > Hi, > > > > The Linux Test Project test suite has been released for the month of > > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has > > been multiple changes for building/installing the test suite after the > > recent changes in Makefile infrastructure. > > Wouldn't make sense to integrate this test suite in the kernel source tree? There was discussion like this some few years back. The idea was to get some core tests from LTP to the kernel source tree. But then the idea was dropped probably to avoid maintenance overhead ;-) Regards-- Subrata > > Regards, > Paolo ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-02 12:23 ` Subrata Modak @ 2011-03-02 19:45 ` Garrett Cooper 2011-03-02 21:49 ` Mike Frysinger 2011-03-03 1:52 ` [LTP] " CAI Qian 1 sibling, 1 reply; 14+ messages in thread From: Garrett Cooper @ 2011-03-02 19:45 UTC (permalink / raw) To: subrata Cc: Paolo Ciarrocchi, Shubham Goyal, ltp-list, linux-kernel, torvalds, akpm, vapier, chrubis On Wed, Mar 2, 2011 at 4:23 AM, Subrata Modak <subrata@linux.vnet.ibm.com> wrote: > On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote: >> On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal >> <shubham@linux.vnet.ibm.com> wrote: >> > Hi, >> > >> > The Linux Test Project test suite has been released for the month of >> > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has >> > been multiple changes for building/installing the test suite after the >> > recent changes in Makefile infrastructure. >> >> Wouldn't make sense to integrate this test suite in the kernel source tree? > > There was discussion like this some few years back. The idea was to get > some core tests from LTP to the kernel source tree. But then the idea > was dropped probably to avoid maintenance overhead ;-) Putting LTP in the kernel.org sources really doesn't make sense for the following reasons: 1. LTP isn't really tied to a single kernel release. 2. LTP isn't the only test project out there for Linux. 3. LTP has more stuff than it needs to have for testing out the kernel (well, it did more in the past before I started cleaning it up in the past couple of months). 4. Maintaining it will become a political bloodbath for both parties as Linux is loosely managed by Linus et all, and LTP has been largely developed by SGI and maintained by IBM and a few other parties like Fujitsu, Nokia, Redhat, etc. 5. Integrating LTP into Kbuild, etc would probably be non-trivial due to the size of LTP (but it might be easier after the Makefile restructuring I did a year and a half ago). That being said, if Linux devs took the initiative and submitted testcases that either illustrated past regressions in the kernel, feature tested enhancements, and submitted documentation that actually described their changes to the Linux kernel and the manpages project, I would take this over having LTP in the linux sources because right now things largely work because the Linux sources haven't really been rototilled since 2.4 -> 2.6, but they are somewhat bitrotted and when Linux rototills its sources again, we'll have to go through rototilling our stuff as well. Right now multiple QA engineers are sort of playing whack-a-mole trying to figure out requirements and submit tests to LTP, and since they don't have 100% context into the actual problem, some information is lost in translation when the tests are submitted. This isn't desirable. Thanks, -Garrett ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-02 19:45 ` Garrett Cooper @ 2011-03-02 21:49 ` Mike Frysinger 0 siblings, 0 replies; 14+ messages in thread From: Mike Frysinger @ 2011-03-02 21:49 UTC (permalink / raw) To: Garrett Cooper Cc: subrata, Paolo Ciarrocchi, Shubham Goyal, ltp-list, linux-kernel, torvalds, akpm, chrubis [-- Attachment #1: Type: Text/Plain, Size: 1999 bytes --] On Wednesday, March 02, 2011 14:45:38 Garrett Cooper wrote: > On Wed, Mar 2, 2011 at 4:23 AM, Subrata Modak > > <subrata@linux.vnet.ibm.com> wrote: > > On Wed, 2011-03-02 at 13:06 +0100, Paolo Ciarrocchi wrote: > >> On Wed, Mar 2, 2011 at 12:40 PM, Shubham Goyal > >> > >> <shubham@linux.vnet.ibm.com> wrote: > >> > Hi, > >> > > >> > The Linux Test Project test suite has been released for the month of > >> > FEBRUARY 2011. Please see ltp/INSTALL file carefully, as there has > >> > been multiple changes for building/installing the test suite after the > >> > recent changes in Makefile infrastructure. > >> > >> Wouldn't make sense to integrate this test suite in the kernel source > >> tree? > > > > There was discussion like this some few years back. The idea was to get > > some core tests from LTP to the kernel source tree. But then the idea > > was dropped probably to avoid maintenance overhead ;-) > > Putting LTP in the kernel.org sources really doesn't make sense for > the following reasons: > > 1. LTP isn't really tied to a single kernel release. > 2. LTP isn't the only test project out there for Linux. > 3. LTP has more stuff than it needs to have for testing out the kernel > (well, it did more in the past before I started cleaning it up in the > past couple of months). > 4. Maintaining it will become a political bloodbath for both parties > as Linux is loosely managed by Linus et all, and LTP has been largely > developed by SGI and maintained by IBM and a few other parties like > Fujitsu, Nokia, Redhat, etc. > 5. Integrating LTP into Kbuild, etc would probably be non-trivial due > to the size of LTP (but it might be easier after the Makefile > restructuring I did a year and a half ago). these are all very good reasons. additional points: - ltp is pretty fsckin huge - ltp often times tests both sides of the userspace API/ABI -- between the kernel and the C library, and the C library and end applications -mike [-- Attachment #2: This is a digitally signed message part. --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-02 12:23 ` Subrata Modak 2011-03-02 19:45 ` Garrett Cooper @ 2011-03-03 1:52 ` CAI Qian 2011-03-03 14:24 ` Cyril Hrubis 2011-03-04 18:58 ` Theodore Tso 1 sibling, 2 replies; 14+ messages in thread From: CAI Qian @ 2011-03-03 1:52 UTC (permalink / raw) To: subrata; +Cc: ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi > There was discussion like this some few years back. The idea was to get > some core tests from LTP to the kernel source tree. But then the idea > was dropped probably to avoid maintenance overhead ;-) Where are the places in kernel source tree for those? Those days, there just too many tests and testing projects for kernel like LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and then to extract the best? LTP has so many goals and focus which isn't going to be only to test kernel any more and it is increasing difficult to support so many distros, kernel versions, and so on. There are some CORE tests like memory management tests, ksm, oom etc have benefit from the developers' bless and review. It also need to be updated to keep the tests relevant to the current git tree, since the features/specs are changing consistent inside the kernel. This could be also useful to improve the kernel quality by providing test code inside the kernel tree that to be used during the code review process that for example, a ksm patchset needs to pass that particular sanity tests in order to catch the regression. It provide benefit that when the changelog said that it passed the ksm tests inside the kernel, we knew exactly what it is without needing to sync up with another project like LTP. In term of maintenance, it needs to be selectively which tests need to be inside the kernel. There should ideally have a dedicated maintainer from the testing point of view to review them. The criteria can be something like, 1) purely purpose of the tests are to test kernel written in C with the kernel coding style. Userspace and integration tests should be better to put into LTP and other projects. 2) tests need to pass sub-system maintainers' review that for example, ksm tests need MM sub-system maintainers' review-by and sign-off-by and alike. 3) they need to be working with and sync up to the latest git version. 4) they have to be proved to be the best tests we can have to test those particular kernel code. There are many tests in LTP, but there are also many duplicated tests as well. Those need to be solved when considering to be moved inside the kernel source. 5) they should really be functional testing. Non-functional tests like stress or performance tests are usually more complex to setup hence defeat the purpose of quick regression checking. Once we have those tests in-place, the next step to improve the kernel quality is to have more patches had Tested-by tags before been accepted in the kernel git tree. Those testers can simply use the non-ambiguious references for tests provided by those in-kernel tests. In addition, those could be a follow-up items with the kernel regression reports that after fixed/analyzed a regression in kernel, the next natural thing to do is to fix/add missing tests to close this gap in the future by providing efficient tests to our users if all possible. CAI Qian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-03 1:52 ` [LTP] " CAI Qian @ 2011-03-03 14:24 ` Cyril Hrubis 2011-03-04 8:21 ` CAI Qian 2011-03-04 18:58 ` Theodore Tso 1 sibling, 1 reply; 14+ messages in thread From: Cyril Hrubis @ 2011-03-03 14:24 UTC (permalink / raw) To: CAI Qian Cc: subrata, ltp-list, vapier, linux-kernel, Paolo Ciarrocchi, akpm, torvalds Hi! I'm not completely against this. At least LTP and possibly other testsuites would gain by attention from developers (it seems that the QA people themselves are the only ones interested). But some things are not just as simple as you see them (at least from my point of view). > Those days, there just too many tests and testing projects for kernel like > LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and > then to extract the best? That is IMHO just too much work. You would need somebody to extract it and then somebody to keep things in sync. > LTP has so many goals and focus which isn't going to be only to test kernel > any more and it is increasing difficult to support so many distros, kernel > versions, and so on. Frankly, LTP has maybe quite a lot of goals, but has a very little manpower, so just now it's mostly broken and rotten code. I would rather focus on cleaning up and fixing up the LTP, dropping ambiguous tests and so. We already did some progress in that area (I tend to say "we" but besides random contributions we have like three or five people). > There are some CORE tests like memory management tests, ksm, oom etc have > benefit from the developers' bless and review. It also need to be updated > to keep the tests relevant to the current git tree, since the features/specs > are changing consistent inside the kernel. That's true indeed. > This could be also useful to improve the kernel quality by providing test > code inside the kernel tree that to be used during the code review process > that for example, a ksm patchset needs to pass that particular sanity tests > in order to catch the regression. It provide benefit that when the changelog > said that it passed the ksm tests inside the kernel, we knew exactly what it is > without needing to sync up with another project like LTP. Well, I don't see what would be gained by merging parts of the LTP into kernel tree. As I said before, this would probably lead to splitting of the forces (and not that we have a lot to split anyway). LTP already has directory called testcases/kernel/, LTP is in the git repository and we have a mailing list. All that is needed is people start noticing that we are here. > In term of maintenance, it needs to be selectively which tests need to be > inside the kernel. There should ideally have a dedicated maintainer from the > testing point of view to review them. The criteria can be something like, > > 1) purely purpose of the tests are to test kernel written in C with the kernel > coding style. Userspace and integration tests should be better to put into > LTP and other projects. I don't think that it's easy to say if some tests are testing kernel/userspace. Sometimes the line isn't that clear. > 2) tests need to pass sub-system maintainers' review that for example, ksm > tests need MM sub-system maintainers' review-by and sign-off-by and alike. Well, requiring maintainers to sign-off your tests is kind of dull. That would probably block the tests from being accepted just because maintainers don't care too much/have different things to do. > 3) they need to be working with and sync up to the latest git version. > 4) they have to be proved to be the best tests we can have to test those > particular kernel code. There are many tests in LTP, but there are also > many duplicated tests as well. Those need to be solved when considering to > be moved inside the kernel source. You can't easily prove that something is best ;). > 5) they should really be functional testing. Non-functional tests like stress > or performance tests are usually more complex to setup hence defeat the > purpose of quick regression checking. > > Once we have those tests in-place, the next step to improve the kernel quality > is to have more patches had Tested-by tags before been accepted in the kernel > git tree. Those testers can simply use the non-ambiguious references for tests > provided by those in-kernel tests. Once again, LTP does exist so reference to LTP is not ambiguous. Yes, it's, for historical reasons, hosted on sourceforge rather than kernel.org. But there it is. > In addition, those could be a follow-up items with the kernel regression reports > that after fixed/analyzed a regression in kernel, the next natural thing to do > is to fix/add missing tests to close this gap in the future by providing efficient > tests to our users if all possible. That's right thing to do, but once more, LTP is here for you ;). So my conclusion is that the major point here is make LTP more visible among linux hackers. On that note, I'm planing to prepare some presentation to let the people know what is the current state and what we are doing to get it better. -- Cyril Hrubis chrubis@suse.cz ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-03 14:24 ` Cyril Hrubis @ 2011-03-04 8:21 ` CAI Qian 2011-03-04 9:04 ` Garrett Cooper 0 siblings, 1 reply; 14+ messages in thread From: CAI Qian @ 2011-03-04 8:21 UTC (permalink / raw) To: Cyril Hrubis Cc: subrata, ltp-list, vapier, linux-kernel, Paolo Ciarrocchi, akpm, torvalds > Well, I don't see what would be gained by merging parts of the LTP into > kernel tree. As I said before, this would probably lead to splitting of > the forces (and not that we have a lot to split anyway). LTP already has > directory called testcases/kernel/, LTP is in the git repository and we > have a mailing list. All that is needed is people start noticing that > we are here. Then, the approach to merge parts of LTP to kernel is to say "Here we are, please accept our best". On the other hand, I have noticed that there are many developers tend to have test code in their kernel submit changelog which isn't it better to make life easier for them to add those testing code in a proper place in kernel which in-turn to benefit in a long run. > I don't think that it's easy to say if some tests are testing > kernel/userspace. Sometimes the line isn't that clear. There are C code as in kernel coding style. Scripting code like Bash, Perl better to re-written in C that in a long run when there are something like thousands of tests to run that performance/scalabitlies/maintenence is going to matter just like to write an OS. > Well, requiring maintainers to sign-off your tests is kind of dull. That > would probably block the tests from being accepted just because > maintainers don't care too much/have different things to do. The idea is to raise a bar to get the best out of it. If maintainers don't care too much about the testing right now that is fine. There are many people they do care. A particular subsystem maintainer and its tests maintainer aren't necessary to be the same person because subsystem maintainer isn't necessary to be the best one to find/acknowledge defeats for code he maintained. > You can't easily prove that something is best ;). The best will at lest be reviewed by eyes from the kernel community and experts, and will be the one to be accepted by the community. > Once again, LTP does exist so reference to LTP is not ambiguous. Yes, > it's, for historical reasons, hosted on sourceforge rather than > kernel.org. But there it is. By accepted into the kernel, it certainly make it easier to reference without dealing with two projects and trees. CAI Qian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-04 8:21 ` CAI Qian @ 2011-03-04 9:04 ` Garrett Cooper 0 siblings, 0 replies; 14+ messages in thread From: Garrett Cooper @ 2011-03-04 9:04 UTC (permalink / raw) To: CAI Qian Cc: Cyril Hrubis, ltp-list, vapier, torvalds, linux-kernel, Paolo Ciarrocchi, akpm On Fri, Mar 4, 2011 at 12:21 AM, CAI Qian <caiqian@redhat.com> wrote: > >> Well, I don't see what would be gained by merging parts of the LTP into >> kernel tree. As I said before, this would probably lead to splitting of >> the forces (and not that we have a lot to split anyway). LTP already has >> directory called testcases/kernel/, LTP is in the git repository and we >> have a mailing list. All that is needed is people start noticing that >> we are here. > Then, the approach to merge parts of LTP to kernel is to say "Here we are, > please accept our best". On the other hand, I have noticed that there are > many developers tend to have test code in their kernel submit changelog > which isn't it better to make life easier for them to add those testing > code in a proper place in kernel which in-turn to benefit in a long run. > >> I don't think that it's easy to say if some tests are testing >> kernel/userspace. Sometimes the line isn't that clear. > There are C code as in kernel coding style. Scripting code like Bash, Perl > better to re-written in C that in a long run when there are something like > thousands of tests to run that performance/scalabitlies/maintenence is going > to matter just like to write an OS. > >> Well, requiring maintainers to sign-off your tests is kind of dull. That >> would probably block the tests from being accepted just because >> maintainers don't care too much/have different things to do. > The idea is to raise a bar to get the best out of it. If maintainers don't > care too much about the testing right now that is fine. There are many people > they do care. A particular subsystem maintainer and its tests maintainer > aren't necessary to be the same person because subsystem maintainer isn't > necessary to be the best one to find/acknowledge defeats for code he maintained. > >> You can't easily prove that something is best ;). > The best will at lest be reviewed by eyes from the kernel community and > experts, and will be the one to be accepted by the community. > >> Once again, LTP does exist so reference to LTP is not ambiguous. Yes, >> it's, for historical reasons, hosted on sourceforge rather than >> kernel.org. But there it is. > By accepted into the kernel, it certainly make it easier to reference > without dealing with two projects and trees. I'm sorry for even starting this bikeshed discussion. Let's just bury the hatchet and get back to work on more fruitful things. Thanks, -Garrett ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-03 1:52 ` [LTP] " CAI Qian 2011-03-03 14:24 ` Cyril Hrubis @ 2011-03-04 18:58 ` Theodore Tso 2011-03-07 17:59 ` Steven Rostedt 1 sibling, 1 reply; 14+ messages in thread From: Theodore Tso @ 2011-03-04 18:58 UTC (permalink / raw) To: CAI Qian Cc: subrata, ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi On Mar 2, 2011, at 8:52 PM, CAI Qian wrote: > Those days, there just too many tests and testing projects for kernel like > LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and > then to extract the best? Part of the problem is that every single testing project has different goals and priorities. For example xfstests is maintained by the XFS folks, as well as people from some of the other file system development efforts (the ext4 one in particular, thanks to people like Eric Sandeen), to test file systems. At least at one point, I had heard a complaint that LTP was more focused on increasing test coverage as measured by a code coverage tool in the kernel than it was about about covering edge conditions, or races. There's nothing wrong with that, per se, and I don't know if it was true then or now, but it's a very different focus from one which is focused increasing the data reliability of file systems, quickly and efficiently. And then there's the LSB test suites, which is really code at testing correctness from a standards perspective, which is a different focus yet again from the LTP and xfstests approach. Bottom line, I'm a big fan of having different test suites, with different philosophies. Each philosophy has its strengths and blind spots, and so a problem that might be missed by one test suite might get caught by another. The only real problem is an operational one. There are some programs which are used by both LTP and xfstests, and changes that is made in one, don't necessarily get propagated to the other unless someone manually does it. But I think we can solve that without trying to merge all of these tests into a single Grand Unified Test Suite. -- Ted ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-04 18:58 ` Theodore Tso @ 2011-03-07 17:59 ` Steven Rostedt 2011-03-08 2:33 ` CAI Qian 0 siblings, 1 reply; 14+ messages in thread From: Steven Rostedt @ 2011-03-07 17:59 UTC (permalink / raw) To: Theodore Tso Cc: CAI Qian, subrata, ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi On Fri, Mar 04, 2011 at 01:58:45PM -0500, Theodore Tso wrote: > > On Mar 2, 2011, at 8:52 PM, CAI Qian wrote: > > > Those days, there just too many tests and testing projects for kernel like > > LTP, autotest, xfstests and so on. Why not have somewhere to collabrate and > > then to extract the best? > > Part of the problem is that every single testing project has different goals and > priorities. For example xfstests is maintained by the XFS folks, as well as people > from some of the other file system development efforts (the ext4 one in particular, > thanks to people like Eric Sandeen), to test file systems. Perhaps we need to get developer's tests into the kernel. We now have a tools/testing directory lets use it. > > At least at one point, I had heard a complaint that LTP was more focused on > increasing test coverage as measured by a code coverage tool in the kernel > than it was about about covering edge conditions, or races. There's nothing > wrong with that, per se, and I don't know if it was true then or now, but it's a very > different focus from one which is focused increasing the data reliability of file > systems, quickly and efficiently. > > And then there's the LSB test suites, which is really code at testing correctness > from a standards perspective, which is a different focus yet again from the LTP > and xfstests approach. > > Bottom line, I'm a big fan of having different test suites, with different philosophies. > Each philosophy has its strengths and blind spots, and so a problem that might > be missed by one test suite might get caught by another. > > The only real problem is an operational one. There are some programs which > are used by both LTP and xfstests, and changes that is made in one, don't > necessarily get propagated to the other unless someone manually does it. > But I think we can solve that without trying to merge all of these tests into a > single Grand Unified Test Suite. > How about having the developers write tests and place them in the tools/testing directory and then the folks at LTP and xfstests et al. can update their code with the code from the kernel. Currently only ktest.pl exists in this directory. I use it constantly and post bugs that it finds. It focuses on just the building and booting of a kernel. It can run several randconfigs, do bisects and such, but it just has a single command to do any tests. This command is just a shell command the user can put in. I leave what tests to run to the user. Thus LTP could be the test that gets kicked off. For example: I have this script I run on my x86 box: TEST_START ITERATE 10 TEST_TYPE = test BUILD_TYPE = randconfig MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-net CHECKOUT = origin/master TEST = ssh root@mitest /work/c/hackbench_32 50 TEST_START ITERATE 10 TEST_TYPE = boot BUILD_TYPE = randconfig MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min MAKE_CMD = make ARCH=i386 DEFAULTS REBOOT_ON_ERROR = 0 POWEROFF_ON_ERROR = 1 POWEROFF_ON_SUCCESS = 1 REBOOT_ON_SUCCESS = 0 DIE_ON_FAILURE = 0 STORE_FAILURES = /home/rostedt/work/autotest/nobackup/failures POWEROFF_AFTER_HALT = 60 CLEAR_LOG = 1 MIN_CONFIG = /home/rostedt/work/autotest/configs/mitest/config-mitest-min SSH_USER = root BUILD_DIR = /home/rostedt/work/autotest/nobackup/linux-test.git OUTPUT_DIR = /home/rostedt/work/autotest/nobackup/mitest BUILD_TARGET = arch/x86/boot/bzImage TARGET_IMAGE = /boot/vmlinuz-test POWER_CYCLE = /home/rostedt/work/autotest/cycle-mxtest CONSOLE = nc -d fedora 3001 LOCALVERSION = -test GRUB_MENU = Test Kernel MAKE_CMD = distmake-32 ARCH=i386 POWER_OFF = /home/rostedt/work/autotest/poweroff-mxtest BUILD_OPTIONS = -j20 LOG_FILE = /home/rostedt/work/autotest/nobackup/mitest/mitest.log TEST = ssh root@mitest cat /debug/tracing/trace ADD_CONFIG = /home/rostedt/work/autotest/configs/config-broken /home/rostedt/work/autotest/config-general Each command is documented in the samples.conf that is also in that directory. The above test runs two sets of tests. The first runs randconfig 10 times with a minimum config that will allow my box to have a network connection, and after it boots, it runs hackbench. The second test runs ten randconfig builds 10 times and just makes sure the box can boot. I'm working on getting to run commands via the console so I do not need the network active to run tests. Once could change the TEST = ... to run LTP tests, or anything else. Having tests to run that I can add to my automated testing would be helpful. I could build randconfigs and run these tests making sure that they work with different configurations. Maybe it would also be helpful to have the CONFIGs that are needed by the tests. It would not make sense to test iptables if iptables is not configured ;) -- Steve ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-07 17:59 ` Steven Rostedt @ 2011-03-08 2:33 ` CAI Qian 2011-03-08 15:27 ` Steven Rostedt 0 siblings, 1 reply; 14+ messages in thread From: CAI Qian @ 2011-03-08 2:33 UTC (permalink / raw) To: Steven Rostedt Cc: subrata, ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi, Theodore Tso > Perhaps we need to get developer's tests into the kernel. We now have a > tools/testing directory lets use it. There are a few test code span over other directories too. Documentation/vm/hugepage-mmap.c Documentation/vm/hugepage-shm.c Documentation/vm/map_hugetlb.c ... CAI Qian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-08 2:33 ` CAI Qian @ 2011-03-08 15:27 ` Steven Rostedt 2011-03-09 1:33 ` CAI Qian 0 siblings, 1 reply; 14+ messages in thread From: Steven Rostedt @ 2011-03-08 15:27 UTC (permalink / raw) To: CAI Qian Cc: subrata, ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi, Theodore Tso On Mon, 2011-03-07 at 21:33 -0500, CAI Qian wrote: > > Perhaps we need to get developer's tests into the kernel. We now have a > > tools/testing directory lets use it. > There are a few test code span over other directories too. > Documentation/vm/hugepage-mmap.c > Documentation/vm/hugepage-shm.c > Documentation/vm/map_hugetlb.c These all look like "Examples", which means they should probably be moved to samples? As "samples" can be compiled on boot. But perhaps because they are user space programs its fine to keep them here? -- Steve > ... ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [LTP] [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011. 2011-03-08 15:27 ` Steven Rostedt @ 2011-03-09 1:33 ` CAI Qian 0 siblings, 0 replies; 14+ messages in thread From: CAI Qian @ 2011-03-09 1:33 UTC (permalink / raw) To: Steven Rostedt Cc: subrata, ltp-list, vapier, linux-kernel, akpm, torvalds, Paolo Ciarrocchi, Theodore Tso > > > Perhaps we need to get developer's tests into the kernel. We now > > > have a > > > tools/testing directory lets use it. > > There are a few test code span over other directories too. > > Documentation/vm/hugepage-mmap.c > > Documentation/vm/hugepage-shm.c > > Documentation/vm/map_hugetlb.c > > These all look like "Examples", which means they should probably be > moved to samples? As "samples" can be compiled on boot. But perhaps > because they are user space programs its fine to keep them here? Yes, but also test cases to check if basic hugetlb code is working. Well, the problem is that they are all over the place, and some of good test cases are within commit logs as well which makes things harder to maintain, use... CAI Qian ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2011-03-09 1:34 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-03-02 11:40 [ANNOUNCE] The Linux Test Project has been released for FEBRUARY 2011 Shubham Goyal 2011-03-02 12:06 ` Paolo Ciarrocchi 2011-03-02 12:23 ` Subrata Modak 2011-03-02 19:45 ` Garrett Cooper 2011-03-02 21:49 ` Mike Frysinger 2011-03-03 1:52 ` [LTP] " CAI Qian 2011-03-03 14:24 ` Cyril Hrubis 2011-03-04 8:21 ` CAI Qian 2011-03-04 9:04 ` Garrett Cooper 2011-03-04 18:58 ` Theodore Tso 2011-03-07 17:59 ` Steven Rostedt 2011-03-08 2:33 ` CAI Qian 2011-03-08 15:27 ` Steven Rostedt 2011-03-09 1:33 ` CAI Qian
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox