[linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[PINTU KUMAR]

Hi All,

For one of our ARM embedded product, we recently updated the Kernel version from
3.4 to 3.18 and we noticed that the same application memory usage (PSS value)
gone up by ~10% and for some cases it even crossed ~50%.
There is no change in platform part. All platform component was built with ARM
32-bit toolchain.
However, the Kernel is changed from 32-bit to 64-bit.

Is upgrading Kernel version and moving from 32-bit to 64-bit is such a risk ?
After the upgrade, what can we do further to reduce the application memory usage
?
Is there any other factor that will help us to improve without major
modifications in platform ?

As a proof, we did a small experiment on our Ubuntu-32 bit machine.
We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we observed the
following:
--------------------------------------------------------------------------------
-------------
|UBUNTU-32 bit		|Kernel 3.13	|Kernel 4.03	|DIFF	|
|CALCULATOR PSS	|6057 KB	|6466 KB	|409 KB	|
--------------------------------------------------------------------------------
-------------
So, just by upgrading the Kernel version: PSS value for calculator is increased
by 409KB.

If anybody knows any in-sight about it please point out more details about the
root cause.

Thank You!

Regards,
Pintu

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[Minchan Kim]

On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
> Hi All,
> 
> For one of our ARM embedded product, we recently updated the Kernel version from
> 3.4 to 3.18 and we noticed that the same application memory usage (PSS value)
> gone up by ~10% and for some cases it even crossed ~50%.
> There is no change in platform part. All platform component was built with ARM
> 32-bit toolchain.
> However, the Kernel is changed from 32-bit to 64-bit.
> 
> Is upgrading Kernel version and moving from 32-bit to 64-bit is such a risk ?
> After the upgrade, what can we do further to reduce the application memory usage
> ?
> Is there any other factor that will help us to improve without major
> modifications in platform ?
> 
> As a proof, we did a small experiment on our Ubuntu-32 bit machine.
> We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we observed the
> following:
> --------------------------------------------------------------------------------
> -------------
> |UBUNTU-32 bit		|Kernel 3.13	|Kernel 4.03	|DIFF	|
> |CALCULATOR PSS	|6057 KB	|6466 KB	|409 KB	|
> --------------------------------------------------------------------------------
> -------------
> So, just by upgrading the Kernel version: PSS value for calculator is increased
> by 409KB.
> 
> If anybody knows any in-sight about it please point out more details about the
> root cause.

One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[PINTU KUMAR]

Hi,
> -----Original Message-----
> From: Minchan Kim [mailto:minchan@kernel.org]
> Sent: Friday, August 05, 2016 1:50 PM
> To: PINTU KUMAR
> Cc: linux-kernel@vger.kernel.org; linux-mm@kvack.org;
> jaejoon.seo@samsung.com; jy0.jeon@samsung.com; vishnu.ps@samsung.com
> Subject: Re: [linux-mm] Drastic increase in application memory usage with
Kernel
> version upgrade
> 
> On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
> > Hi All,
> >
> > For one of our ARM embedded product, we recently updated the Kernel
> > version from
> > 3.4 to 3.18 and we noticed that the same application memory usage (PSS
> > value) gone up by ~10% and for some cases it even crossed ~50%.
> > There is no change in platform part. All platform component was built
> > with ARM 32-bit toolchain.
> > However, the Kernel is changed from 32-bit to 64-bit.
> >
> > Is upgrading Kernel version and moving from 32-bit to 64-bit is such a risk
?
> > After the upgrade, what can we do further to reduce the application
> > memory usage ?
> > Is there any other factor that will help us to improve without major
> > modifications in platform ?
> >
> > As a proof, we did a small experiment on our Ubuntu-32 bit machine.
> > We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we observed
> > the
> > following:
> > ----------------------------------------------------------------------
> > ----------
> > -------------
> > |UBUNTU-32 bit		|Kernel 3.13	|Kernel 4.03	|DIFF	|
> > |CALCULATOR PSS	|6057 KB	|6466 KB	|409 KB	|
> > ----------------------------------------------------------------------
> > ----------
> > -------------
> > So, just by upgrading the Kernel version: PSS value for calculator is
> > increased by 409KB.
> >
> > If anybody knows any in-sight about it please point out more details
> > about the root cause.
> 
> One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].
Ok. Thank you for your reply.
So, if I revert this patch, will the memory usage be decreased for the processes
with Kernel 3.18 ?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[Pavel Machek]

On Fri 2016-08-05 20:17:36, PINTU KUMAR wrote:
> Hi,

> > On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
> > > Hi All,
> > >
> > > For one of our ARM embedded product, we recently updated the Kernel
> > > version from
> > > 3.4 to 3.18 and we noticed that the same application memory usage (PSS
> > > value) gone up by ~10% and for some cases it even crossed ~50%.
> > > There is no change in platform part. All platform component was built
> > > with ARM 32-bit toolchain.
> > > However, the Kernel is changed from 32-bit to 64-bit.
> > >
> > > Is upgrading Kernel version and moving from 32-bit to 64-bit is such a risk
> ?
> > > After the upgrade, what can we do further to reduce the application
> > > memory usage ?
> > > Is there any other factor that will help us to improve without major
> > > modifications in platform ?
> > >
> > > As a proof, we did a small experiment on our Ubuntu-32 bit machine.
> > > We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we observed
> > > the
> > > following:
> > > ----------------------------------------------------------------------
> > > ----------
> > > -------------
> > > |UBUNTU-32 bit		|Kernel 3.13	|Kernel 4.03	|DIFF	|
> > > |CALCULATOR PSS	|6057 KB	|6466 KB	|409 KB	|
> > > ----------------------------------------------------------------------
> > > ----------
> > > -------------
> > > So, just by upgrading the Kernel version: PSS value for calculator is
> > > increased by 409KB.
> > >
> > > If anybody knows any in-sight about it please point out more details
> > > about the root cause.
> > 
> > One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].
> Ok. Thank you for your reply.
> So, if I revert this patch, will the memory usage be decreased for the processes
> with Kernel 3.18 ?

I guess you should try it...

You may want to try the same kernel version, once in 32-bit and once
in 64-bit version. And you may consider moving to recent kernel.

Yes, 64-bit kernel will increase memory usage _of kernel_, but...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[PINTU KUMAR]

Hi,

> -----Original Message-----
> From: Pavel Machek [mailto:pavel@ucw.cz]
> Sent: Saturday, August 06, 2016 2:20 AM
> To: PINTU KUMAR
> Cc: 'Minchan Kim'; linux-kernel@vger.kernel.org; linux-mm@kvack.org;
> jaejoon.seo@samsung.com; jy0.jeon@samsung.com; vishnu.ps@samsung.com
> Subject: Re: [linux-mm] Drastic increase in application memory usage with
Kernel
> version upgrade
> 
> On Fri 2016-08-05 20:17:36, PINTU KUMAR wrote:
> > Hi,
> 
> > > On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
> > > > Hi All,
> > > >
> > > > For one of our ARM embedded product, we recently updated the
> > > > Kernel version from 3.4 to 3.18 and we noticed that the same 
> > > > application memory usage  (PSS value) gone up by ~10% and for 
> > > > some cases it even crossed ~50%. There is no change in platform 
> > > > part. All platform component was  built with ARM 32-bit toolchain.
> > > > However, the Kernel is changed from 32-bit to 64-bit.
> > > >
> > > > Is upgrading Kernel version and moving from 32-bit to 64-bit is
> > > > such a risk?
> > > > After the upgrade, what can we do further to reduce the
> > > > application memory usage ?
> > > > Is there any other factor that will help us to improve without
> > > > major modifications in platform ?
> > > >
> > > > As a proof, we did a small experiment on our Ubuntu-32 bit machine.
> > > > We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we
> > > > observed the following:
> > > > ------------------------------------------------------------------
> > > > |UBUNTU-32 bit	|Kernel 3.13	|Kernel 4.03	|DIFF	|
> > > > |CALCULATOR PSS	|6057 KB	|6466 KB	|409 KB	|
> > > > ------------------------------------------------------------------
> > > > So, just by upgrading the Kernel version: PSS value for calculator
> > > > is increased by 409KB.
> > > >
> > > > If anybody knows any in-sight about it please point out more
> > > > details about the root cause.
> > >
> > > One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].
> > Ok. Thank you for your reply.
> > So, if I revert this patch, will the memory usage be decreased for the
> > processes with Kernel 3.18 ?
> 
> I guess you should try it...
> 
Thanks for the reply and confirmation.
Our exact kernel version is: 3.18.14
And, we already have this patch:
/*
mm: do not call do_fault_around for non-linear fault
Ingo Korb reported that "repeated mapping of the same file on tmpfs
using remap_file_pages sometimes triggers a BUG at mm/filemap.c:202 when
the process exits".
He bisected the bug to d7c1755179b8 ("mm: implement ->map_pages for
shmem/tmpfs"), although the bug was actually added by commit
8c6e50b0290c ("mm: introduce vm_ops->map_pages()").
*/

So, I guess, reverting this patch (8c6e50b0290c), is not required ?
But, still we have memory usage issue.

> You may want to try the same kernel version, once in 32-bit and once in 64-bit
> version. And you may consider moving to recent kernel.
> 
Sorry, but moving to higher kernel version is not possible as of now. 
As it involves other BSP changes.
We can only upgrades the relevant patches.
However, as I said earlier, we even found the difference on Ubuntu 32-bit 
after moving from Kernel-3.13 to 4.0.
So, I guess, this problem exists even in higher kernel version too.

> Yes, 64-bit kernel will increase memory usage _of kernel_, but...
> 
Ok, But ?
Will it be major culprit ? Or higher kernel version is major culprit ?

> 
> 	Pavel
> --
> (english) http://www.livejournal.com/~pavelmachek
> (cesky, pictures)
http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[vinayak menon]

On Wed, Aug 10, 2016 at 6:56 PM, PINTU KUMAR <pintu.k@samsung.com> wrote:
> Hi,
>
>> -----Original Message-----
>> From: Pavel Machek [mailto:pavel@ucw.cz]
>> Sent: Saturday, August 06, 2016 2:20 AM
>> To: PINTU KUMAR
>> Cc: 'Minchan Kim'; linux-kernel@vger.kernel.org; linux-mm@kvack.org;
>> jaejoon.seo@samsung.com; jy0.jeon@samsung.com; vishnu.ps@samsung.com
>> Subject: Re: [linux-mm] Drastic increase in application memory usage with
> Kernel
>> version upgrade
>>
>> On Fri 2016-08-05 20:17:36, PINTU KUMAR wrote:
>> > Hi,
>>
>> > > On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
>> > > > Hi All,
>> > > >
>> > > > For one of our ARM embedded product, we recently updated the
>> > > > Kernel version from 3.4 to 3.18 and we noticed that the same
>> > > > application memory usage  (PSS value) gone up by ~10% and for
>> > > > some cases it even crossed ~50%. There is no change in platform
>> > > > part. All platform component was  built with ARM 32-bit toolchain.
>> > > > However, the Kernel is changed from 32-bit to 64-bit.
>> > > >
>> > > > Is upgrading Kernel version and moving from 32-bit to 64-bit is
>> > > > such a risk?
>> > > > After the upgrade, what can we do further to reduce the
>> > > > application memory usage ?
>> > > > Is there any other factor that will help us to improve without
>> > > > major modifications in platform ?
>> > > >
>> > > > As a proof, we did a small experiment on our Ubuntu-32 bit machine.
>> > > > We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we
>> > > > observed the following:
>> > > > ------------------------------------------------------------------
>> > > > |UBUNTU-32 bit  |Kernel 3.13    |Kernel 4.03    |DIFF   |
>> > > > |CALCULATOR PSS |6057 KB        |6466 KB        |409 KB |
>> > > > ------------------------------------------------------------------
>> > > > So, just by upgrading the Kernel version: PSS value for calculator
>> > > > is increased by 409KB.
>> > > >
>> > > > If anybody knows any in-sight about it please point out more
>> > > > details about the root cause.
>> > >
>> > > One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].
>> > Ok. Thank you for your reply.
>> > So, if I revert this patch, will the memory usage be decreased for the
>> > processes with Kernel 3.18 ?
>>
>> I guess you should try it...
>>
> Thanks for the reply and confirmation.
> Our exact kernel version is: 3.18.14
> And, we already have this patch:
> /*
> mm: do not call do_fault_around for non-linear fault
> Ingo Korb reported that "repeated mapping of the same file on tmpfs
> using remap_file_pages sometimes triggers a BUG at mm/filemap.c:202 when
> the process exits".
> He bisected the bug to d7c1755179b8 ("mm: implement ->map_pages for
> shmem/tmpfs"), although the bug was actually added by commit
> 8c6e50b0290c ("mm: introduce vm_ops->map_pages()").
> */
>
> So, I guess, reverting this patch (8c6e50b0290c), is not required ?
> But, still we have memory usage issue.
>
I had observed the PSS increase with 3.18, and that was because of the
faultaround patch which MInchan mentioned.
Without reverting the patch you can just try reducing
fault_around_bytes (mm/memory.c) to PAGE_SIZE. That should
bring down the PSS.

Thanks,
Vinayak

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: [linux-mm] Drastic increase in application memory usage with Kernel version upgrade

[PINTU KUMAR]

Hi,
> -----Original Message-----
> From: vinayak menon [mailto:vinayakm.list@gmail.com]
> Sent: Thursday, August 11, 2016 10:23 AM
> To: PINTU KUMAR
> Cc: Pavel Machek; Konstantin Khlebnikov; Minchan Kim; linux-
> kernel@vger.kernel.org; linux-mm@kvack.org; jaejoon.seo@samsung.com;
> jy0.jeon@samsung.com; vishnu.ps@samsung.com; chulspro.kim@samsung.com
> Subject: Re: [linux-mm] Drastic increase in application memory usage with Kernel
> version upgrade
> 
> On Wed, Aug 10, 2016 at 6:56 PM, PINTU KUMAR <pintu.k@samsung.com> wrote:
> > Hi,
> >
> >> -----Original Message-----
> >> From: Pavel Machek [mailto:pavel@ucw.cz]
> >> Sent: Saturday, August 06, 2016 2:20 AM
> >> To: PINTU KUMAR
> >> Cc: 'Minchan Kim'; linux-kernel@vger.kernel.org; linux-mm@kvack.org;
> >> jaejoon.seo@samsung.com; jy0.jeon@samsung.com;
> vishnu.ps@samsung.com
> >> Subject: Re: [linux-mm] Drastic increase in application memory usage
> >> with
> > Kernel
> >> version upgrade
> >>
> >> On Fri 2016-08-05 20:17:36, PINTU KUMAR wrote:
> >> > Hi,
> >>
> >> > > On Fri, Aug 05, 2016 at 10:26:37AM +0530, PINTU KUMAR wrote:
> >> > > > Hi All,
> >> > > >
> >> > > > For one of our ARM embedded product, we recently updated the
> >> > > > Kernel version from 3.4 to 3.18 and we noticed that the same
> >> > > > application memory usage  (PSS value) gone up by ~10% and for
> >> > > > some cases it even crossed ~50%. There is no change in platform
> >> > > > part. All platform component was  built with ARM 32-bit toolchain.
> >> > > > However, the Kernel is changed from 32-bit to 64-bit.
> >> > > >
> >> > > > Is upgrading Kernel version and moving from 32-bit to 64-bit is
> >> > > > such a risk?
> >> > > > After the upgrade, what can we do further to reduce the
> >> > > > application memory usage ?
> >> > > > Is there any other factor that will help us to improve without
> >> > > > major modifications in platform ?
> >> > > >
> >> > > > As a proof, we did a small experiment on our Ubuntu-32 bit machine.
> >> > > > We upgraded Ubuntu Kernel version from 3.13 to 4.01 and we
> >> > > > observed the following:
> >> > > > ---------------------------------------------------------------
> >> > > > ---
> >> > > > |UBUNTU-32 bit  |Kernel 3.13    |Kernel 4.03    |DIFF   |
> >> > > > |CALCULATOR PSS |6057 KB        |6466 KB        |409 KB |
> >> > > > ---------------------------------------------------------------
> >> > > > --- So, just by upgrading the Kernel version: PSS value for
> >> > > > calculator is increased by 409KB.
> >> > > >
> >> > > > If anybody knows any in-sight about it please point out more
> >> > > > details about the root cause.
> >> > >
> >> > > One of culprit is [8c6e50b0290c, mm: introduce vm_ops->map_pages()].
> >> > Ok. Thank you for your reply.
> >> > So, if I revert this patch, will the memory usage be decreased for
> >> > the processes with Kernel 3.18 ?
> >>
> >> I guess you should try it...
> >>
> > Thanks for the reply and confirmation.
> > Our exact kernel version is: 3.18.14
> > And, we already have this patch:
> > /*
> > mm: do not call do_fault_around for non-linear fault Ingo Korb
> > reported that "repeated mapping of the same file on tmpfs using
> > remap_file_pages sometimes triggers a BUG at mm/filemap.c:202 when the
> > process exits".
> > He bisected the bug to d7c1755179b8 ("mm: implement ->map_pages for
> > shmem/tmpfs"), although the bug was actually added by commit
> > 8c6e50b0290c ("mm: introduce vm_ops->map_pages()").
> > */
> >
> > So, I guess, reverting this patch (8c6e50b0290c), is not required ?
> > But, still we have memory usage issue.
> >
> I had observed the PSS increase with 3.18, and that was because of the
> faultaround patch which MInchan mentioned.
> Without reverting the patch you can just try reducing fault_around_bytes
> (mm/memory.c) to PAGE_SIZE. That should bring down the PSS.
> 
Thanks for your reply.
I tried changing fault_around_bytes value from 65536 to 4096.
But, still there is no change in PSS.
Please let me know if anything is missing.
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -2776,7 +2776,8 @@ void do_set_pte(struct vm_area_struct *vma, unsigned long address,
 }
 static unsigned long fault_around_bytes __read_mostly =
-       rounddown_pow_of_two(65536);
+       rounddown_pow_of_two(PAGE_SIZE);

> Thanks,
> Vinayak


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>