performance regression of migrate_pages system call

performance regression of migrate_pages system call

[Shijith Thotton]

Observed migrate_pages03 of LTP is failing on ARM with v4.19 kernel. Test was 
run on ThunderX2 99xx with 2 sockets. migrate_pages syscall was taking 1+ 
seconds and caused a timeout in the test.

strace output:
<...>
[pid 35146] 14:11:14 migrate_pages(0, 64, [0x0000000000000001], 
[0x0000000000000002]) = 0 <1.099427>
[pid 35146] 14:11:15 migrate_pages(0, 64, [0x0000000000000002], 
[0x0000000000000001]) = 0 <1.145143>
<..>

Git bisect points to following change as the cause of regression.
commit 3b8c9f
arm64: IPI each CPU after invalidating the I-cache for kernel mappings

Regression happens with 4K page size and not with 64K pages.
Please let me know if anyone is experiencing similar issue.

Thanks,
Shijith


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: performance regression of migrate_pages system call

[Shijith Thotton]

Hi Will,

On 01/02/2019 07:22 PM, Shijith Thotton wrote:
> Observed migrate_pages03 of LTP is failing on ARM with v4.19 kernel. Test was
> run on ThunderX2 99xx with 2 sockets. migrate_pages syscall was taking 1+
> seconds and caused a timeout in the test.
> 
> strace output:
> <...>
> [pid 35146] 14:11:14 migrate_pages(0, 64, [0x0000000000000001],
> [0x0000000000000002]) = 0 <1.099427>
> [pid 35146] 14:11:15 migrate_pages(0, 64, [0x0000000000000002],
> [0x0000000000000001]) = 0 <1.145143>
> <..>
> 
> Git bisect points to following change as the cause of regression.
> commit 3b8c9f
> arm64: IPI each CPU after invalidating the I-cache for kernel mappings
> 
> Regression happens with 4K page size and not with 64K pages.
> Please let me know if anyone is experiencing similar issue.
> 

Please comment if a solution like below (by modifying sync_icache_aliases) is 
possible for fixing the regression.
http://lists.infradead.org/pipermail/linux-arm-kernel/2018-December/618866.html

Function trace:
migrate_pages()
   + unmap_and_move()
     + __unmap_and_move()
       + remove_migration_ptes()
         + set_pte_at()
           + __sync_icache_dcache()
             + sync_icache_aliases()
               + flush_icache_range()
                 + kick_all_cpus_sync()

Thanks,
Shijith



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: performance regression of migrate_pages system call

[Will Deacon]

On Mon, Jan 07, 2019 at 08:44:47AM +0000, Shijith Thotton wrote:
> On 01/02/2019 07:22 PM, Shijith Thotton wrote:
> > Observed migrate_pages03 of LTP is failing on ARM with v4.19 kernel. Test was
> > run on ThunderX2 99xx with 2 sockets. migrate_pages syscall was taking 1+
> > seconds and caused a timeout in the test.
> > 
> > strace output:
> > <...>
> > [pid 35146] 14:11:14 migrate_pages(0, 64, [0x0000000000000001],
> > [0x0000000000000002]) = 0 <1.099427>
> > [pid 35146] 14:11:15 migrate_pages(0, 64, [0x0000000000000002],
> > [0x0000000000000001]) = 0 <1.145143>
> > <..>
> > 
> > Git bisect points to following change as the cause of regression.
> > commit 3b8c9f
> > arm64: IPI each CPU after invalidating the I-cache for kernel mappings
> > 
> > Regression happens with 4K page size and not with 64K pages.
> > Please let me know if anyone is experiencing similar issue.
> > 
> 
> Please comment if a solution like below (by modifying sync_icache_aliases) is 
> possible for fixing the regression.
> http://lists.infradead.org/pipermail/linux-arm-kernel/2018-December/618866.html
> 
> Function trace:
> migrate_pages()
>    + unmap_and_move()
>      + __unmap_and_move()
>        + remove_migration_ptes()
>          + set_pte_at()
>            + __sync_icache_dcache()
>              + sync_icache_aliases()
>                + flush_icache_range()
>                  + kick_all_cpus_sync()

Unfortunately, we can't drop the IPI in this case because we really do need
to ensure that other CPUs have drained any instructions previously fetched
from the new physical page.

Adding Anshuman to cc, since I think he's been looking at other ways to
improve this.

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel