[LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Petr Vorel]

Test run mostly below 1 sec per filesystem, only NTFS is very slow
~ 10-15 sec depending on architecture.

Also this is the only tests in syscalls which fails on due timeout on
emulated risc-v with LTP_TIMEOUT_MUL=2,LTP_RUNTIME_MUL=2 (with the
default 30s runtime => 1 min due LTP_RUNTIME_MUL=2).

Therefore set runtime to 45s, that should be enough even for slow
risc-v and not that high for other archs.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
---
Changes v1->v2:
* 60 => 45

 testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
index 314630267d..ef75c92de0 100644
--- a/testcases/kernel/syscalls/swapoff/swapoff01.c
+++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
@@ -53,5 +53,6 @@ static struct tst_test test = {
 	.all_filesystems = 1,
 	.needs_root = 1,
 	.test_all = verify_swapoff,
+	.max_runtime = 45,
 	.setup = setup
 };
-- 
2.45.2


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Cyril Hrubis]

Hi!
> Test run mostly below 1 sec per filesystem, only NTFS is very slow
> ~ 10-15 sec depending on architecture.

If part of the slowdown is caused by the fact that we format the
filesystems the real question is if all the tests that format and mount
a device should have a max_runtime defined. It may make sense to
increment the max_runtime in the test library to compensate for the
mkfs, but it looks that it's not the mkfs that is slow.

I did some measurements for the swapoff01 case and the whole testrun
runs for about 10 seconds for me. If I change the test to do
tst_brk(TCONF, ...) as the first thing in the test setup its 1s. So
writing out the 1MB of swapfile and running mkswap and swapon() on that
takes about 90% of the runtime.

So yes it looks like the actual test needs some runtime defined,
although I'm not sure how long it should be. We probably need a
reference machine to callibrate this on. I would avoid any emulated CPU
and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
be a good choice since they are widely available and slow by any modern
standards.

> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> ---
> Changes v1->v2:
> * 60 => 45
> 
>  testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
> index 314630267d..ef75c92de0 100644
> --- a/testcases/kernel/syscalls/swapoff/swapoff01.c
> +++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
> @@ -53,5 +53,6 @@ static struct tst_test test = {
>  	.all_filesystems = 1,
>  	.needs_root = 1,
>  	.test_all = verify_swapoff,
> +	.max_runtime = 45,
>  	.setup = setup
>  };
> -- 
> 2.45.2
> 

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Petr Vorel]

> Hi!
> > Test run mostly below 1 sec per filesystem, only NTFS is very slow
> > ~ 10-15 sec depending on architecture.

> If part of the slowdown is caused by the fact that we format the
> filesystems the real question is if all the tests that format and mount
> a device should have a max_runtime defined. It may make sense to
> increment the max_runtime in the test library to compensate for the
> mkfs, but it looks that it's not the mkfs that is slow.

> I did some measurements for the swapoff01 case and the whole testrun
> runs for about 10 seconds for me. If I change the test to do
> tst_brk(TCONF, ...) as the first thing in the test setup its 1s. So
> writing out the 1MB of swapfile and running mkswap and swapon() on that
> takes about 90% of the runtime.

Thanks for your investigation.

> So yes it looks like the actual test needs some runtime defined,
> although I'm not sure how long it should be. We probably need a
> reference machine to callibrate this on. I would avoid any emulated CPU
> and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
> be a good choice since they are widely available and slow by any modern
> standards.

This makes sense.

Kind regards,
Petr

> > Signed-off-by: Petr Vorel <pvorel@suse.cz>
> > ---
> > Changes v1->v2:
> > * 60 => 45

> >  testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
> >  1 file changed, 1 insertion(+)

> > diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
> > index 314630267d..ef75c92de0 100644
> > --- a/testcases/kernel/syscalls/swapoff/swapoff01.c
> > +++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
> > @@ -53,5 +53,6 @@ static struct tst_test test = {
> >  	.all_filesystems = 1,
> >  	.needs_root = 1,
> >  	.test_all = verify_swapoff,
> > +	.max_runtime = 45,
> >  	.setup = setup
> >  };
> > -- 
> > 2.45.2

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Cyril Hrubis]

Hi!
> > So yes it looks like the actual test needs some runtime defined,
> > although I'm not sure how long it should be. We probably need a
> > reference machine to callibrate this on. I would avoid any emulated CPU
> > and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
> > be a good choice since they are widely available and slow by any modern
> > standards.
> 
> This makes sense.

So I dusted off my Pi Zero and indeed the test timeouts there as well.
The ntfs filesystem takes 78s, ext2 29s, exfat 20s. So I would say that
max_runtime should be at least a minute. I guess I will try to run the
whole run there before the release and check for timeouts so that we are
sure that the runtime is set reasonably even for very slow hardware.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Li Wang]

On Wed, Sep 4, 2024 at 4:53 PM Cyril Hrubis <chrubis@suse.cz> wrote:
>
> Hi!
> > > So yes it looks like the actual test needs some runtime defined,
> > > although I'm not sure how long it should be. We probably need a
> > > reference machine to callibrate this on. I would avoid any emulated CPU
> > > and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
> > > be a good choice since they are widely available and slow by any modern
> > > standards.
> >
> > This makes sense.
>
> So I dusted off my Pi Zero and indeed the test timeouts there as well.
> The ntfs filesystem takes 78s, ext2 29s, exfat 20s. So I would say that

I'm wondering which distribution (and kernel version) did you use on raspi-0?

My raspi4(ubuntu-24, 6.8.0-1010, Cortex-A72) takes a little faster.

ext2: 12s
ext3: 12s
ext4: 0.4s
ntfs: 46s
exfat: skip

> max_runtime should be at least a minute. I guess I will try to run the

Agree.

> whole run there before the release and check for timeouts so that we are
> sure that the runtime is set reasonably even for very slow hardware.

+1



-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Li Wang]

On Wed, Sep 4, 2024 at 5:26 PM Li Wang <liwang@redhat.com> wrote:
>
> On Wed, Sep 4, 2024 at 4:53 PM Cyril Hrubis <chrubis@suse.cz> wrote:
> >
> > Hi!
> > > > So yes it looks like the actual test needs some runtime defined,
> > > > although I'm not sure how long it should be. We probably need a
> > > > reference machine to callibrate this on. I would avoid any emulated CPU
> > > > and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
> > > > be a good choice since they are widely available and slow by any modern
> > > > standards.
> > >
> > > This makes sense.
> >
> > So I dusted off my Pi Zero and indeed the test timeouts there as well.
> > The ntfs filesystem takes 78s, ext2 29s, exfat 20s. So I would say that
>
> I'm wondering which distribution (and kernel version) did you use on raspi-0?

I ask this because we might need to consider the oldest supported systems:
- https://linux-test-project.readthedocs.io/en/latest/users/supported_systems.html

-- 
Regards,
Li Wang


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Petr Vorel]

> On Wed, Sep 4, 2024 at 5:26 PM Li Wang <liwang@redhat.com> wrote:

> > On Wed, Sep 4, 2024 at 4:53 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> > > Hi!
> > > > > So yes it looks like the actual test needs some runtime defined,
> > > > > although I'm not sure how long it should be. We probably need a
> > > > > reference machine to callibrate this on. I would avoid any emulated CPU
> > > > > and choose a real hardware. I guess one of the RPis e.g. Pi Zero would
> > > > > be a good choice since they are widely available and slow by any modern
> > > > > standards.

Pi Zero is good that it's slow enough that even emulated risc-v will work.

But OTOH might be more practical to measure on RPI4 or RPI3 (maybe more of us
has RPI4 or RPI3 personally or in CI). I mean unless others have/will buy Pi
Zero you will be forced to do the reference. Also Pi Zero is slow (you need to
compile, thus I will not love to do the measurements.

Also emulated risc-v and Pi Zero will always need to set LTP_RUNTIME_MUL, right?
Or do we want to do measurements that even these will run without
LTP_RUNTIME_MUL? That would help users not having to bother, but increasing
limits for fast devices might not be a good idea.

> > > > This makes sense.

> > > So I dusted off my Pi Zero and indeed the test timeouts there as well.
> > > The ntfs filesystem takes 78s, ext2 29s, exfat 20s. So I would say that

> > I'm wondering which distribution (and kernel version) did you use on raspi-0?

Yes, 78s, ext2 29s, exfat 20s is faster, but that's due Raspberry Pi Zero using
just Broadcom BCM2835 SoC as the first generation Raspberry Pi [1] (32 bit arm).
RPI4 uses Broadcom BCM2711 SoC (64-bit quad-core).

Kind regards,
Petr

[1] https://en.wikipedia.org/wiki/Raspberry_Pi#Processor

> I ask this because we might need to consider the oldest supported systems:
> - https://linux-test-project.readthedocs.io/en/latest/users/supported_systems.html

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Cyril Hrubis]

Hi!
> I'm wondering which distribution (and kernel version) did you use on raspi-0?

The default raspbian. The difference I guess is that RPi Zero is a
single ARMv6 if I'm looking right it's the same CPU as on the original
RPi. That's way slower than quad Cortex-A72 on RPi 4.

> My raspi4(ubuntu-24, 6.8.0-1010, Cortex-A72) takes a little faster.
> 
> ext2: 12s
> ext3: 12s
> ext4: 0.4s
> ntfs: 46s
> exfat: skip

I suppose that this is stil slow on RPi 4 because the rootfs is on a SD
card, since the CPU on RPi 4 is way faster than RPi Zero.

-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v2 1/1] swapoff01: Define max_runtime 45s

[Li Wang]

On Wed, Sep 4, 2024 at 10:00 PM Cyril Hrubis <chrubis@suse.cz> wrote:

> Hi!
> > I'm wondering which distribution (and kernel version) did you use on
> raspi-0?
>
> The default raspbian. The difference I guess is that RPi Zero is a
> single ARMv6 if I'm looking right it's the same CPU as on the original
> RPi. That's way slower than quad Cortex-A72 on RPi 4.
>
> > My raspi4(ubuntu-24, 6.8.0-1010, Cortex-A72) takes a little faster.
> >
> > ext2: 12s
> > ext3: 12s
> > ext4: 0.4s
> > ntfs: 46s
> > exfat: skip
>
> I suppose that this is stil slow on RPi 4 because the rootfs is on a SD
> card, since the CPU on RPi 4 is way faster than RPi Zero.
>

Yes, exactly.

@Petr, if you planning to send out V3, I think it is better to set a proper
value
for max_runtime in swapon01 simultaneously. Since our CI jobs catch that one
timeout out sporadically too.

     7	tst_device.c:96: TINFO: Found free device 0 '/dev/loop0'
     8	tst_test.c:1733: TINFO: LTP version: 20240524

     9	tst_test.c:1617: TINFO: Timeout per run is 0h 01m 00s

    10	tst_supported_fs_types.c:96: TINFO: Kernel supports ext2
    11	tst_supported_fs_types.c:61: TINFO: mkfs.ext2 does exist


    61	tst_test.c:1693: TINFO: === Testing on vfat ===
    62	tst_test.c:1106: TINFO: Formatting /dev/loop0 with vfat opts=''
extra opts=''
    63	tst_test.c:1120: TINFO: Mounting /dev/loop0 to
/mnt/testarea/ltp-ukK7X0nhcs/LTP_swakCd462/mntpoint fstyp=vfat flags=0
    64	libswap.c:198: TINFO: create a swapfile size of 1 megabytes (MB)
    65	tst_ioctl.c:26: TINFO: FIBMAP ioctl is supported
    66	swapon01.c:41: TINFO: create a swapfile size of 128 megabytes (MB)
    67	swapon01.c:27: TPASS: tst_syscall(__NR_swapon, SWAP_FILE, 0) passed
    68	tst_test.c:1671: TINFO: If you are running on slow machine, try
exporting LTP_TIMEOUT_MUL > 1
    69	tst_test.c:1673: TBROK: Test killed! (timeout?)



-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

[LTP] [PATCH v3] swapoff01/swapon01: Define max_runtime 1 mins

[Li Wang]

From: Petr Vorel <pvorel@suse.cz>

Test run mostly below 1 sec per filesystem, only NTFS is very slow
~ 10-15 sec depending on architecture.

Also this is the only tests in syscalls which fails on due timeout on
emulated risc-v with LTP_TIMEOUT_MUL=2,LTP_RUNTIME_MUL=2 (with the
default 30s runtime => 1 min due LTP_RUNTIME_MUL=2).

Based on the test consumed time on RPi0 and RPi4, we set runtime to 60s,
that should be enough even for slow risc-v and not that high for other archs.

Signed-off-by: Petr Vorel <pvorel@suse.cz>
Signed-off-by: Li Wang <liwang@redhat.com>
---
 testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
 testcases/kernel/syscalls/swapon/swapon01.c   | 1 +
 2 files changed, 2 insertions(+)

diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
index 314630267..965d17cc8 100644
--- a/testcases/kernel/syscalls/swapoff/swapoff01.c
+++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
@@ -53,5 +53,6 @@ static struct tst_test test = {
 	.all_filesystems = 1,
 	.needs_root = 1,
 	.test_all = verify_swapoff,
+	.max_runtime = 60,
 	.setup = setup
 };
diff --git a/testcases/kernel/syscalls/swapon/swapon01.c b/testcases/kernel/syscalls/swapon/swapon01.c
index 49915c32d..eddcd5788 100644
--- a/testcases/kernel/syscalls/swapon/swapon01.c
+++ b/testcases/kernel/syscalls/swapon/swapon01.c
@@ -51,5 +51,6 @@ static struct tst_test test = {
 	.all_filesystems = 1,
 	.needs_cgroup_ctrls = (const char *const []){ "memory", NULL },
 	.test_all = verify_swapon,
+	.max_runtime = 60,
 	.setup = setup
 };
-- 
2.46.0


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3] swapoff01/swapon01: Define max_runtime 1 mins

[Avinesh Kumar]

Hi,

Reviewed-by: Avinesh Kumar <akumar@suse.de>

On Monday, September 9, 2024 4:52:43 AM GMT+2 Li Wang wrote:
> From: Petr Vorel <pvorel@suse.cz>
> 
> Test run mostly below 1 sec per filesystem, only NTFS is very slow
> ~ 10-15 sec depending on architecture.
> 
> Also this is the only tests in syscalls which fails on due timeout on
> emulated risc-v with LTP_TIMEOUT_MUL=2,LTP_RUNTIME_MUL=2 (with the
> default 30s runtime => 1 min due LTP_RUNTIME_MUL=2).
> 
> Based on the test consumed time on RPi0 and RPi4, we set runtime to 60s,
> that should be enough even for slow risc-v and not that high for other archs.
> 
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> Signed-off-by: Li Wang <liwang@redhat.com>
> ---
>  testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
>  testcases/kernel/syscalls/swapon/swapon01.c   | 1 +
>  2 files changed, 2 insertions(+)
> 
> diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
> index 314630267..965d17cc8 100644
> --- a/testcases/kernel/syscalls/swapoff/swapoff01.c
> +++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
> @@ -53,5 +53,6 @@ static struct tst_test test = {
>  	.all_filesystems = 1,
>  	.needs_root = 1,
>  	.test_all = verify_swapoff,
> +	.max_runtime = 60,
>  	.setup = setup
>  };
> diff --git a/testcases/kernel/syscalls/swapon/swapon01.c b/testcases/kernel/syscalls/swapon/swapon01.c
> index 49915c32d..eddcd5788 100644
> --- a/testcases/kernel/syscalls/swapon/swapon01.c
> +++ b/testcases/kernel/syscalls/swapon/swapon01.c
> @@ -51,5 +51,6 @@ static struct tst_test test = {
>  	.all_filesystems = 1,
>  	.needs_cgroup_ctrls = (const char *const []){ "memory", NULL },
>  	.test_all = verify_swapon,
> +	.max_runtime = 60,
>  	.setup = setup
>  };
> 

Regards,
Avinesh



-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3] swapoff01/swapon01: Define max_runtime 1 mins

[Jan Stancek]

On Mon, Sep 9, 2024 at 4:53 AM Li Wang <liwang@redhat.com> wrote:
>
> From: Petr Vorel <pvorel@suse.cz>
>
> Test run mostly below 1 sec per filesystem, only NTFS is very slow
> ~ 10-15 sec depending on architecture.
>
> Also this is the only tests in syscalls which fails on due timeout on
> emulated risc-v with LTP_TIMEOUT_MUL=2,LTP_RUNTIME_MUL=2 (with the
> default 30s runtime => 1 min due LTP_RUNTIME_MUL=2).
>
> Based on the test consumed time on RPi0 and RPi4, we set runtime to 60s,
> that should be enough even for slow risc-v and not that high for other archs.
>
> Signed-off-by: Petr Vorel <pvorel@suse.cz>
> Signed-off-by: Li Wang <liwang@redhat.com>

Acked-by: Jan Stancek <jstancek@redhat.com>

> ---
>  testcases/kernel/syscalls/swapoff/swapoff01.c | 1 +
>  testcases/kernel/syscalls/swapon/swapon01.c   | 1 +
>  2 files changed, 2 insertions(+)
>
> diff --git a/testcases/kernel/syscalls/swapoff/swapoff01.c b/testcases/kernel/syscalls/swapoff/swapoff01.c
> index 314630267..965d17cc8 100644
> --- a/testcases/kernel/syscalls/swapoff/swapoff01.c
> +++ b/testcases/kernel/syscalls/swapoff/swapoff01.c
> @@ -53,5 +53,6 @@ static struct tst_test test = {
>         .all_filesystems = 1,
>         .needs_root = 1,
>         .test_all = verify_swapoff,
> +       .max_runtime = 60,
>         .setup = setup
>  };
> diff --git a/testcases/kernel/syscalls/swapon/swapon01.c b/testcases/kernel/syscalls/swapon/swapon01.c
> index 49915c32d..eddcd5788 100644
> --- a/testcases/kernel/syscalls/swapon/swapon01.c
> +++ b/testcases/kernel/syscalls/swapon/swapon01.c
> @@ -51,5 +51,6 @@ static struct tst_test test = {
>         .all_filesystems = 1,
>         .needs_cgroup_ctrls = (const char *const []){ "memory", NULL },
>         .test_all = verify_swapon,
> +       .max_runtime = 60,
>         .setup = setup
>  };
> --
> 2.46.0
>
>
> --
> Mailing list info: https://lists.linux.it/listinfo/ltp
>


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3] swapoff01/swapon01: Define max_runtime 1 mins

[Cyril Hrubis]

Hi!
Reviewed-by: Cyril Hrubis <chrubis@suse.cz>


-- 
Cyril Hrubis
chrubis@suse.cz

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Re: [LTP] [PATCH v3] swapoff01/swapon01: Define max_runtime 1 mins

[Li Wang]

Patch merged, thank you all.

-- 
Regards,
Li Wang

-- 
Mailing list info: https://lists.linux.it/listinfo/ltp