PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

This is better explained here:
http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
(The
highest-voted answer believes this to be a kernel bug.)

Summary: I have set up a system where I am using LXC to create multiple
virtualized containers on my system with limited resources. Unfortunately, I'm
running into a troublesome scenario where the OOM killer is hard killing
processes in my LXC container when I write a file with size exceeding the
memory limitation (set to 300MB). There appears to be some issue with the
file buffering respecting the containers memory limit.


Reproducing:

/done on a c1.xlarge instance running on Amazon EC2

Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
ubuntu -- -r precise)

Modify the configuration of each container to set lxc.cgroup.memory.
limit_in_bytes = 300M

Within each container run:
dd if=/dev/zero of=test2 bs=100k count=5010
parallel

This will with high probability activate the OOM (as seen in demsg); often
the dd processes themselves will be killed.

This has been verified to have problems on:
Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
x86_64 GNU/Linux

Please let me know your thoughts.

Regards,
Aaron Staley

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> This is better explained here:
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> (The
> highest-voted answer believes this to be a kernel bug.)

Yeah, sorry I haven't had time to look more into it, but I'm pretty
that's the case.  When you sent the previous email I looked quickly at
the dd source.  I had always assumed that dd looked at available memory
and malloced as much as it thought it could - but looking at the source,
it does not in fact do that.  So yes, I think the kernel is simply
leaving it all in page cache and accounting that to the process which
then gets OOMed.

Instead, the kernel should be throttling the task while it waits for
the page cache to be written to disk (since blkio might also be
slowed down).

-serge

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

Hi Serge,

Thanks a lot. Would you know of any workarounds outside of forcing every
write to sync to disk (which kills performance)? Perhaps some settings in
the container I can set?  Unfortunately, modifying dirty_background_ratio
 and dirty_expire_centiseconds globally (/etc/sysctl.conf) as suggested by
the serverfault answer will not stop the OOM kills.

Regards,
Aaron


On Tue, Jun 25, 2013 at 6:24 AM, Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>wrote:

> Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > This is better explained here:
> >
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > (The
> > highest-voted answer believes this to be a kernel bug.)
>
> Yeah, sorry I haven't had time to look more into it, but I'm pretty
> that's the case.  When you sent the previous email I looked quickly at
> the dd source.  I had always assumed that dd looked at available memory
> and malloced as much as it thought it could - but looking at the source,
> it does not in fact do that.  So yes, I think the kernel is simply
> leaving it all in page cache and accounting that to the process which
> then gets OOMed.
>
> Instead, the kernel should be throttling the task while it waits for
> the page cache to be written to disk (since blkio might also be
> slowed down).
>
> -serge
>



-- 
Aaron Staley
*PiCloud, Inc.*

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Zhu Yanhai]

Hi,
Please check this patch, it could fix your problem,
http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=e62e384e9da8d9a0c599795464a7e76fd490931c

--
Thanks,
Zhu Yanhai


2013/6/26 Aaron Staley <aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org>

> Hi Serge,
>
> Thanks a lot. Would you know of any workarounds outside of forcing every
> write to sync to disk (which kills performance)? Perhaps some settings in
> the container I can set?  Unfortunately, modifying dirty_background_ratio
>  and dirty_expire_centiseconds globally (/etc/sysctl.conf) as suggested by
> the serverfault answer will not stop the OOM kills.
>
> Regards,
> Aaron
>
>
> On Tue, Jun 25, 2013 at 6:24 AM, Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org
> >wrote:
>
> > Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > > This is better explained here:
> > >
> >
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> >
> > Yeah, sorry I haven't had time to look more into it, but I'm pretty
> > that's the case.  When you sent the previous email I looked quickly at
> > the dd source.  I had always assumed that dd looked at available memory
> > and malloced as much as it thought it could - but looking at the source,
> > it does not in fact do that.  So yes, I think the kernel is simply
> > leaving it all in page cache and accounting that to the process which
> > then gets OOMed.
> >
> > Instead, the kernel should be throttling the task while it waits for
> > the page cache to be written to disk (since blkio might also be
> > slowed down).
> >
> > -serge
> >
>
>
>
> --
> Aaron Staley
> *PiCloud, Inc.*
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers
>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

The behavior it fixes sounds similar to what I'm seeing. However, if I read
the logs correctly, wasn't this committed into Linux 3.5? If so, wouldn't
Linux 3.8.0-25-generic #37-Ubuntu SMP (where I can reproduce the problem)
already have this fix?

Thanks,
Aaron


On Wed, Jun 26, 2013 at 10:12 PM, Zhu Yanhai <zhu.yanhai-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Hi,
> Please check this patch, it could fix your problem,
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=e62e384e9da8d9a0c599795464a7e76fd490931c
>
> --
> Thanks,
> Zhu Yanhai
>
>
> 2013/6/26 Aaron Staley <aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org>
>
>> Hi Serge,
>>
>> Thanks a lot. Would you know of any workarounds outside of forcing every
>> write to sync to disk (which kills performance)? Perhaps some settings in
>> the container I can set?  Unfortunately, modifying dirty_background_ratio
>>  and dirty_expire_centiseconds globally (/etc/sysctl.conf) as suggested by
>> the serverfault answer will not stop the OOM kills.
>>
>> Regards,
>> Aaron
>>
>>
>> On Tue, Jun 25, 2013 at 6:24 AM, Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org
>> >wrote:
>>
>> > Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
>> > > This is better explained here:
>> > >
>> >
>> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
>> > > (The
>> > > highest-voted answer believes this to be a kernel bug.)
>> >
>> > Yeah, sorry I haven't had time to look more into it, but I'm pretty
>> > that's the case.  When you sent the previous email I looked quickly at
>> > the dd source.  I had always assumed that dd looked at available memory
>> > and malloced as much as it thought it could - but looking at the source,
>> > it does not in fact do that.  So yes, I think the kernel is simply
>> > leaving it all in page cache and accounting that to the process which
>> > then gets OOMed.
>> >
>> > Instead, the kernel should be throttling the task while it waits for
>> > the page cache to be written to disk (since blkio might also be
>> > slowed down).
>> >
>> > -serge
>> >
>>
>>
>>
>> --
>> Aaron Staley
>> *PiCloud, Inc.*
>> _______________________________________________
>> Containers mailing list
>> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
>> https://lists.linuxfoundation.org/mailman/listinfo/containers
>>
>
>


-- 
Aaron Staley
*PiCloud, Inc.*

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aristeu Rozanski]

On Wed, Jun 26, 2013 at 10:19:51PM -0700, Aaron Staley wrote:
> The behavior it fixes sounds similar to what I'm seeing. However, if I read
> the logs correctly, wasn't this committed into Linux 3.5? If so, wouldn't
> Linux 3.8.0-25-generic #37-Ubuntu SMP (where I can reproduce the problem)
> already have this fix?

3.6, but yes, it should be in your kernel.

-- 
Aristeu

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge E. Hallyn]

Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> Hi Serge,
> 
> Thanks a lot. Would you know of any workarounds outside of forcing every
> write to sync to disk (which kills performance)? Perhaps some settings in
> the container I can set?  Unfortunately, modifying dirty_background_ratio
>  and dirty_expire_centiseconds globally (/etc/sysctl.conf) as suggested by
> the serverfault answer will not stop the OOM kills.

Sorry i haven't replied yet have I - no, I'm afraid I don't have any
good ideas offhand.  I think this is something that's going to have
to be fixed in the kernel, and there's really nothing you can do
in the container.  I could be wrong.

Over the next few days I'm going to be taking a fresh look at the
behaviors of the various controllers (at least as documented) for
the sake of thinking about configuration api - perhaps the answer
will become clear to me as I do that.

-serge

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> The behavior it fixes sounds similar to what I'm seeing. However, if I read
> the logs correctly, wasn't this committed into Linux 3.5? If so, wouldn't
> Linux 3.8.0-25-generic #37-Ubuntu SMP (where I can reproduce the problem)
> already have this fix?

Hi,

I've been trying to reproduce this in ubuntu raring, but couldn't.  I
started a shell as unprivileged user, and stuck it in a memory cgroup
with memory.limit_in_bytes set to 10M.  Did dd if=/dev/zero of=/tmp/xxx
bs=1M count=100M - it was slow, and memory.failcnt hit 1612526, but it
did in the end succeed.

kernel here is 3.8.0-23-generic #34-Ubuntu, slightly older than you.
Maybe you're onto a new regression?

-serge

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

Hi Serge,

To reproduce, I need to run ~6 containers in parallel each running that
command. I generally cannot reproduce it with just one running. (Repro
instructions are in original email).

Regards,
Aaron


On Mon, Jul 1, 2013 at 8:21 AM, Serge Hallyn <serge.hallyn-GeWIH/nMZzLQT0dZR+AlfA@public.gmane.org>wrote:

> Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > The behavior it fixes sounds similar to what I'm seeing. However, if I
> read
> > the logs correctly, wasn't this committed into Linux 3.5? If so, wouldn't
> > Linux 3.8.0-25-generic #37-Ubuntu SMP (where I can reproduce the problem)
> > already have this fix?
>
> Hi,
>
> I've been trying to reproduce this in ubuntu raring, but couldn't.  I
> started a shell as unprivileged user, and stuck it in a memory cgroup
> with memory.limit_in_bytes set to 10M.  Did dd if=/dev/zero of=/tmp/xxx
> bs=1M count=100M - it was slow, and memory.failcnt hit 1612526, but it
> did in the end succeed.
>
> kernel here is 3.8.0-23-generic #34-Ubuntu, slightly older than you.
> Maybe you're onto a new regression?
>
> -serge
>



-- 
Aaron Staley
*PiCloud, Inc.*

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> This is better explained here:
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> (The
> highest-voted answer believes this to be a kernel bug.)

Hi,

in irc it has been suggested that indeed the kernel should be slowing
down new page creates while waiting for old page cache entries to be
written out to disk, rather than ooming.

With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
count=100 is immediately killed.  In contrast, doing the same from a
3.0.8 kernel did the right thing for me.  But I did reproduce your
experiment below on ec2 with the same result.

So, cc:ing linux-mm in the hopes someone can tell us whether this
is expected behavior, known mis-behavior, or an unknown bug.

> Summary: I have set up a system where I am using LXC to create multiple
> virtualized containers on my system with limited resources. Unfortunately, I'm
> running into a troublesome scenario where the OOM killer is hard killing
> processes in my LXC container when I write a file with size exceeding the
> memory limitation (set to 300MB). There appears to be some issue with the
> file buffering respecting the containers memory limit.
> 
> 
> Reproducing:
> 
> /done on a c1.xlarge instance running on Amazon EC2
> 
> Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> ubuntu -- -r precise)
> 
> Modify the configuration of each container to set lxc.cgroup.memory.
> limit_in_bytes = 300M
> 
> Within each container run:
> dd if=/dev/zero of=test2 bs=100k count=5010
> parallel
> 
> This will with high probability activate the OOM (as seen in demsg); often
> the dd processes themselves will be killed.
> 
> This has been verified to have problems on:
> Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> x86_64 GNU/Linux
> 
> Please let me know your thoughts.
> 
> Regards,
> Aaron Staley
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Aaron Staley (aaron@picloud.com):
> This is better explained here:
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> (The
> highest-voted answer believes this to be a kernel bug.)

Hi,

in irc it has been suggested that indeed the kernel should be slowing
down new page creates while waiting for old page cache entries to be
written out to disk, rather than ooming.

With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
count=100 is immediately killed.  In contrast, doing the same from a
3.0.8 kernel did the right thing for me.  But I did reproduce your
experiment below on ec2 with the same result.

So, cc:ing linux-mm in the hopes someone can tell us whether this
is expected behavior, known mis-behavior, or an unknown bug.

> Summary: I have set up a system where I am using LXC to create multiple
> virtualized containers on my system with limited resources. Unfortunately, I'm
> running into a troublesome scenario where the OOM killer is hard killing
> processes in my LXC container when I write a file with size exceeding the
> memory limitation (set to 300MB). There appears to be some issue with the
> file buffering respecting the containers memory limit.
> 
> 
> Reproducing:
> 
> /done on a c1.xlarge instance running on Amazon EC2
> 
> Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> ubuntu -- -r precise)
> 
> Modify the configuration of each container to set lxc.cgroup.memory.
> limit_in_bytes = 300M
> 
> Within each container run:
> dd if=/dev/zero of=test2 bs=100k count=5010
> parallel
> 
> This will with high probability activate the OOM (as seen in demsg); often
> the dd processes themselves will be killed.
> 
> This has been verified to have problems on:
> Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> x86_64 GNU/Linux
> 
> Please let me know your thoughts.
> 
> Regards,
> Aaron Staley
> _______________________________________________
> Containers mailing list
> Containers@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Aaron Staley (aaron@picloud.com):
> This is better explained here:
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> (The
> highest-voted answer believes this to be a kernel bug.)

Hi,

in irc it has been suggested that indeed the kernel should be slowing
down new page creates while waiting for old page cache entries to be
written out to disk, rather than ooming.

With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
count=100 is immediately killed.  In contrast, doing the same from a
3.0.8 kernel did the right thing for me.  But I did reproduce your
experiment below on ec2 with the same result.

So, cc:ing linux-mm in the hopes someone can tell us whether this
is expected behavior, known mis-behavior, or an unknown bug.

> Summary: I have set up a system where I am using LXC to create multiple
> virtualized containers on my system with limited resources. Unfortunately, I'm
> running into a troublesome scenario where the OOM killer is hard killing
> processes in my LXC container when I write a file with size exceeding the
> memory limitation (set to 300MB). There appears to be some issue with the
> file buffering respecting the containers memory limit.
> 
> 
> Reproducing:
> 
> /done on a c1.xlarge instance running on Amazon EC2
> 
> Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> ubuntu -- -r precise)
> 
> Modify the configuration of each container to set lxc.cgroup.memory.
> limit_in_bytes = 300M
> 
> Within each container run:
> dd if=/dev/zero of=test2 bs=100k count=5010
> parallel
> 
> This will with high probability activate the OOM (as seen in demsg); often
> the dd processes themselves will be killed.
> 
> This has been verified to have problems on:
> Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> x86_64 GNU/Linux
> 
> Please let me know your thoughts.
> 
> Regards,
> Aaron Staley
> _______________________________________________
> Containers mailing list
> Containers@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/containers

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Johannes Weiner]

On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > This is better explained here:
> > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > (The
> > highest-voted answer believes this to be a kernel bug.)
> 
> Hi,
> 
> in irc it has been suggested that indeed the kernel should be slowing
> down new page creates while waiting for old page cache entries to be
> written out to disk, rather than ooming.
> 
> With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> count=100 is immediately killed.  In contrast, doing the same from a
> 3.0.8 kernel did the right thing for me.  But I did reproduce your
> experiment below on ec2 with the same result.
>
> So, cc:ing linux-mm in the hopes someone can tell us whether this
> is expected behavior, known mis-behavior, or an unknown bug.

It's a known issue that was fixed/improved in e62e384 'memcg: prevent
OOM with too many dirty pages', included in 3.6+.

> > Summary: I have set up a system where I am using LXC to create multiple
> > virtualized containers on my system with limited resources. Unfortunately, I'm
> > running into a troublesome scenario where the OOM killer is hard killing
> > processes in my LXC container when I write a file with size exceeding the
> > memory limitation (set to 300MB). There appears to be some issue with the
> > file buffering respecting the containers memory limit.
> > 
> > 
> > Reproducing:
> > 
> > /done on a c1.xlarge instance running on Amazon EC2
> > 
> > Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> > ubuntu -- -r precise)
> > 
> > Modify the configuration of each container to set lxc.cgroup.memory.
> > limit_in_bytes = 300M
> > 
> > Within each container run:
> > dd if=/dev/zero of=test2 bs=100k count=5010
> > parallel
> > 
> > This will with high probability activate the OOM (as seen in demsg); often
> > the dd processes themselves will be killed.
> > 
> > This has been verified to have problems on:
> > Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> > 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> > x86_64 GNU/Linux
> > 
> > Please let me know your thoughts.
> > 
> > Regards,
> > Aaron Staley
> > _______________________________________________
> > Containers mailing list
> > Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> > https://lists.linuxfoundation.org/mailman/listinfo/containers
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo-Bw31MaZKKs0EbZ0PF+XxCw@public.gmane.org  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org"> email-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Johannes Weiner]

On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> Quoting Aaron Staley (aaron@picloud.com):
> > This is better explained here:
> > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > (The
> > highest-voted answer believes this to be a kernel bug.)
> 
> Hi,
> 
> in irc it has been suggested that indeed the kernel should be slowing
> down new page creates while waiting for old page cache entries to be
> written out to disk, rather than ooming.
> 
> With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> count=100 is immediately killed.  In contrast, doing the same from a
> 3.0.8 kernel did the right thing for me.  But I did reproduce your
> experiment below on ec2 with the same result.
>
> So, cc:ing linux-mm in the hopes someone can tell us whether this
> is expected behavior, known mis-behavior, or an unknown bug.

It's a known issue that was fixed/improved in e62e384 'memcg: prevent
OOM with too many dirty pages', included in 3.6+.

> > Summary: I have set up a system where I am using LXC to create multiple
> > virtualized containers on my system with limited resources. Unfortunately, I'm
> > running into a troublesome scenario where the OOM killer is hard killing
> > processes in my LXC container when I write a file with size exceeding the
> > memory limitation (set to 300MB). There appears to be some issue with the
> > file buffering respecting the containers memory limit.
> > 
> > 
> > Reproducing:
> > 
> > /done on a c1.xlarge instance running on Amazon EC2
> > 
> > Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> > ubuntu -- -r precise)
> > 
> > Modify the configuration of each container to set lxc.cgroup.memory.
> > limit_in_bytes = 300M
> > 
> > Within each container run:
> > dd if=/dev/zero of=test2 bs=100k count=5010
> > parallel
> > 
> > This will with high probability activate the OOM (as seen in demsg); often
> > the dd processes themselves will be killed.
> > 
> > This has been verified to have problems on:
> > Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> > 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> > x86_64 GNU/Linux
> > 
> > Please let me know your thoughts.
> > 
> > Regards,
> > Aaron Staley
> > _______________________________________________
> > Containers mailing list
> > Containers@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/containers
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Johannes Weiner]

On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> Quoting Aaron Staley (aaron@picloud.com):
> > This is better explained here:
> > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > (The
> > highest-voted answer believes this to be a kernel bug.)
> 
> Hi,
> 
> in irc it has been suggested that indeed the kernel should be slowing
> down new page creates while waiting for old page cache entries to be
> written out to disk, rather than ooming.
> 
> With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> count=100 is immediately killed.  In contrast, doing the same from a
> 3.0.8 kernel did the right thing for me.  But I did reproduce your
> experiment below on ec2 with the same result.
>
> So, cc:ing linux-mm in the hopes someone can tell us whether this
> is expected behavior, known mis-behavior, or an unknown bug.

It's a known issue that was fixed/improved in e62e384 'memcg: prevent
OOM with too many dirty pages', included in 3.6+.

> > Summary: I have set up a system where I am using LXC to create multiple
> > virtualized containers on my system with limited resources. Unfortunately, I'm
> > running into a troublesome scenario where the OOM killer is hard killing
> > processes in my LXC container when I write a file with size exceeding the
> > memory limitation (set to 300MB). There appears to be some issue with the
> > file buffering respecting the containers memory limit.
> > 
> > 
> > Reproducing:
> > 
> > /done on a c1.xlarge instance running on Amazon EC2
> > 
> > Create 6 empty lxc containers (in my case I did lxc-create -n testcon -t
> > ubuntu -- -r precise)
> > 
> > Modify the configuration of each container to set lxc.cgroup.memory.
> > limit_in_bytes = 300M
> > 
> > Within each container run:
> > dd if=/dev/zero of=test2 bs=100k count=5010
> > parallel
> > 
> > This will with high probability activate the OOM (as seen in demsg); often
> > the dd processes themselves will be killed.
> > 
> > This has been verified to have problems on:
> > Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> > 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64 x86_64
> > x86_64 GNU/Linux
> > 
> > Please let me know your thoughts.
> > 
> > Regards,
> > Aaron Staley
> > _______________________________________________
> > Containers mailing list
> > Containers@lists.linux-foundation.org
> > https://lists.linuxfoundation.org/mailman/listinfo/containers
> 
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

Hi Johannes,

It does appear to still be happening on Linux 3.8.  Does it remain an open
issue?

Regards,
Aaron


On Mon, Jul 1, 2013 at 11:45 AM, Johannes Weiner <hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org> wrote:

> On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> > Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > > This is better explained here:
> > >
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> >
> > Hi,
> >
> > in irc it has been suggested that indeed the kernel should be slowing
> > down new page creates while waiting for old page cache entries to be
> > written out to disk, rather than ooming.
> >
> > With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> > count=100 is immediately killed.  In contrast, doing the same from a
> > 3.0.8 kernel did the right thing for me.  But I did reproduce your
> > experiment below on ec2 with the same result.
> >
> > So, cc:ing linux-mm in the hopes someone can tell us whether this
> > is expected behavior, known mis-behavior, or an unknown bug.
>
> It's a known issue that was fixed/improved in e62e384 'memcg: prevent
> OOM with too many dirty pages', included in 3.6+.
>
> > > Summary: I have set up a system where I am using LXC to create multiple
> > > virtualized containers on my system with limited resources.
> Unfortunately, I'm
> > > running into a troublesome scenario where the OOM killer is hard
> killing
> > > processes in my LXC container when I write a file with size exceeding
> the
> > > memory limitation (set to 300MB). There appears to be some issue with
> the
> > > file buffering respecting the containers memory limit.
> > >
> > >
> > > Reproducing:
> > >
> > > /done on a c1.xlarge instance running on Amazon EC2
> > >
> > > Create 6 empty lxc containers (in my case I did lxc-create -n testcon
> -t
> > > ubuntu -- -r precise)
> > >
> > > Modify the configuration of each container to set lxc.cgroup.memory.
> > > limit_in_bytes = 300M
> > >
> > > Within each container run:
> > > dd if=/dev/zero of=test2 bs=100k count=5010
> > > parallel
> > >
> > > This will with high probability activate the OOM (as seen in demsg);
> often
> > > the dd processes themselves will be killed.
> > >
> > > This has been verified to have problems on:
> > > Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> > > 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64
> x86_64
> > > x86_64 GNU/Linux
> > >
> > > Please let me know your thoughts.
> > >
> > > Regards,
> > > Aaron Staley
> > > _______________________________________________
> > > Containers mailing list
> > > Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/containers
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo-Bw31MaZKKs0EbZ0PF+XxCw@public.gmane.org  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org"> email-Bw31MaZKKs3YtjvyW6yDsg@public.gmane.org </a>
>



-- 
Aaron Staley
*PiCloud, Inc.*

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Aaron Staley]

[-- Attachment #1: Type: text/plain, Size: 3116 bytes --]

Hi Johannes,

It does appear to still be happening on Linux 3.8.  Does it remain an open
issue?

Regards,
Aaron


On Mon, Jul 1, 2013 at 11:45 AM, Johannes Weiner <hannes@cmpxchg.org> wrote:

> On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> > Quoting Aaron Staley (aaron@picloud.com):
> > > This is better explained here:
> > >
> http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> >
> > Hi,
> >
> > in irc it has been suggested that indeed the kernel should be slowing
> > down new page creates while waiting for old page cache entries to be
> > written out to disk, rather than ooming.
> >
> > With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> > count=100 is immediately killed.  In contrast, doing the same from a
> > 3.0.8 kernel did the right thing for me.  But I did reproduce your
> > experiment below on ec2 with the same result.
> >
> > So, cc:ing linux-mm in the hopes someone can tell us whether this
> > is expected behavior, known mis-behavior, or an unknown bug.
>
> It's a known issue that was fixed/improved in e62e384 'memcg: prevent
> OOM with too many dirty pages', included in 3.6+.
>
> > > Summary: I have set up a system where I am using LXC to create multiple
> > > virtualized containers on my system with limited resources.
> Unfortunately, I'm
> > > running into a troublesome scenario where the OOM killer is hard
> killing
> > > processes in my LXC container when I write a file with size exceeding
> the
> > > memory limitation (set to 300MB). There appears to be some issue with
> the
> > > file buffering respecting the containers memory limit.
> > >
> > >
> > > Reproducing:
> > >
> > > /done on a c1.xlarge instance running on Amazon EC2
> > >
> > > Create 6 empty lxc containers (in my case I did lxc-create -n testcon
> -t
> > > ubuntu -- -r precise)
> > >
> > > Modify the configuration of each container to set lxc.cgroup.memory.
> > > limit_in_bytes = 300M
> > >
> > > Within each container run:
> > > dd if=/dev/zero of=test2 bs=100k count=5010
> > > parallel
> > >
> > > This will with high probability activate the OOM (as seen in demsg);
> often
> > > the dd processes themselves will be killed.
> > >
> > > This has been verified to have problems on:
> > > Linux 3.8.0-25-generic #37-Ubuntu SMP and Linux ip-10-8-139-98
> > > 3.2.0-29-virtual #46-Ubuntu SMP Fri Jul 27 17:23:50 UTC 2012 x86_64
> x86_64
> > > x86_64 GNU/Linux
> > >
> > > Please let me know your thoughts.
> > >
> > > Regards,
> > > Aaron Staley
> > > _______________________________________________
> > > Containers mailing list
> > > Containers@lists.linux-foundation.org
> > > https://lists.linuxfoundation.org/mailman/listinfo/containers
> >
> > --
> > To unsubscribe, send a message with 'unsubscribe linux-mm' in
> > the body to majordomo@kvack.org.  For more info on Linux MM,
> > see: http://www.linux-mm.org/ .
> > Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
>



-- 
Aaron Staley
*PiCloud, Inc.*

[-- Attachment #2: Type: text/html, Size: 4730 bytes --]

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Johannes Weiner (hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org):
> On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> > Quoting Aaron Staley (aaron-ZNH+RosXeVlBDgjK7y7TUQ@public.gmane.org):
> > > This is better explained here:
> > > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> > 
> > Hi,
> > 
> > in irc it has been suggested that indeed the kernel should be slowing
> > down new page creates while waiting for old page cache entries to be
> > written out to disk, rather than ooming.
> > 
> > With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> > count=100 is immediately killed.  In contrast, doing the same from a
> > 3.0.8 kernel did the right thing for me.  But I did reproduce your
> > experiment below on ec2 with the same result.
> >
> > So, cc:ing linux-mm in the hopes someone can tell us whether this
> > is expected behavior, known mis-behavior, or an unknown bug.
> 
> It's a known issue that was fixed/improved in e62e384 'memcg: prevent

Ah ok, I see the commit says:

    The solution is far from being ideal - long term solution is memcg aware
    dirty throttling - but it is meant to be a band aid until we have a real
    fix.  We are seeing this happening during nightly backups which are placed

... and ...

    The issue is more visible with slower devices for output.

I'm guessing we see it on ec2 because of slowed fs.

> OOM with too many dirty pages', included in 3.6+.

Is anyone actively working on the long term solution?

thanks,
-serge

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Johannes Weiner (hannes@cmpxchg.org):
> On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> > Quoting Aaron Staley (aaron@picloud.com):
> > > This is better explained here:
> > > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> > 
> > Hi,
> > 
> > in irc it has been suggested that indeed the kernel should be slowing
> > down new page creates while waiting for old page cache entries to be
> > written out to disk, rather than ooming.
> > 
> > With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> > count=100 is immediately killed.  In contrast, doing the same from a
> > 3.0.8 kernel did the right thing for me.  But I did reproduce your
> > experiment below on ec2 with the same result.
> >
> > So, cc:ing linux-mm in the hopes someone can tell us whether this
> > is expected behavior, known mis-behavior, or an unknown bug.
> 
> It's a known issue that was fixed/improved in e62e384 'memcg: prevent

Ah ok, I see the commit says:

    The solution is far from being ideal - long term solution is memcg aware
    dirty throttling - but it is meant to be a band aid until we have a real
    fix.  We are seeing this happening during nightly backups which are placed

... and ...

    The issue is more visible with slower devices for output.

I'm guessing we see it on ec2 because of slowed fs.

> OOM with too many dirty pages', included in 3.6+.

Is anyone actively working on the long term solution?

thanks,
-serge

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Serge Hallyn]

Quoting Johannes Weiner (hannes@cmpxchg.org):
> On Mon, Jul 01, 2013 at 01:01:01PM -0500, Serge Hallyn wrote:
> > Quoting Aaron Staley (aaron@picloud.com):
> > > This is better explained here:
> > > http://serverfault.com/questions/516074/why-are-applications-in-a-memory-limited-lxc-container-writing-large-files-to-di
> > > (The
> > > highest-voted answer believes this to be a kernel bug.)
> > 
> > Hi,
> > 
> > in irc it has been suggested that indeed the kernel should be slowing
> > down new page creates while waiting for old page cache entries to be
> > written out to disk, rather than ooming.
> > 
> > With a 3.0.27-1-ac100 kernel, doing dd if=/dev/zero of=xxx bs=1M
> > count=100 is immediately killed.  In contrast, doing the same from a
> > 3.0.8 kernel did the right thing for me.  But I did reproduce your
> > experiment below on ec2 with the same result.
> >
> > So, cc:ing linux-mm in the hopes someone can tell us whether this
> > is expected behavior, known mis-behavior, or an unknown bug.
> 
> It's a known issue that was fixed/improved in e62e384 'memcg: prevent

Ah ok, I see the commit says:

    The solution is far from being ideal - long term solution is memcg aware
    dirty throttling - but it is meant to be a band aid until we have a real
    fix.  We are seeing this happening during nightly backups which are placed

... and ...

    The issue is more visible with slower devices for output.

I'm guessing we see it on ec2 because of slowed fs.

> OOM with too many dirty pages', included in 3.6+.

Is anyone actively working on the long term solution?

thanks,
-serge

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Michal Hocko]

On Mon 01-07-13 14:02:22, Serge Hallyn wrote:
> Quoting Johannes Weiner (hannes-druUgvl0LCNAfugRpC6u6w@public.gmane.org):
[...]
> > OOM with too many dirty pages', included in 3.6+.
> 
> Is anyone actively working on the long term solution?

Patches for memcg dirty pages accounted were posted quite some time ago.
I plan to look at the at some point but I am rather busy with other
stuff right now. That would be just a first step though. Then we need to
hook into dirty pages throttling and make it memcg aware which sounds
like a bigger challenge.

-- 
Michal Hocko
SUSE Labs

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Michal Hocko]

On Mon 01-07-13 14:02:22, Serge Hallyn wrote:
> Quoting Johannes Weiner (hannes@cmpxchg.org):
[...]
> > OOM with too many dirty pages', included in 3.6+.
> 
> Is anyone actively working on the long term solution?

Patches for memcg dirty pages accounted were posted quite some time ago.
I plan to look at the at some point but I am rather busy with other
stuff right now. That would be just a first step though. Then we need to
hook into dirty pages throttling and make it memcg aware which sounds
like a bigger challenge.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: PROBLEM: Processes writing large files in memory-limited LXC container are killed by OOM

[Michal Hocko]

On Mon 01-07-13 14:02:22, Serge Hallyn wrote:
> Quoting Johannes Weiner (hannes@cmpxchg.org):
[...]
> > OOM with too many dirty pages', included in 3.6+.
> 
> Is anyone actively working on the long term solution?

Patches for memcg dirty pages accounted were posted quite some time ago.
I plan to look at the at some point but I am rather busy with other
stuff right now. That would be just a first step though. Then we need to
hook into dirty pages throttling and make it memcg aware which sounds
like a bigger challenge.

-- 
Michal Hocko
SUSE Labs