[PATCH v2 0/2] fix RLIMIT_CORE accounting for sparse dumps

[Omar Sandoval <osandov@osandov.com>]

From: Omar Sandoval <osandov@fb.com>

Hi, Al,

I'm resending these patches because I realized that I made a subtle typo
in my first version. I only just noticed that I inadvertently changed
this in `dump_emit()`.

----
-		cprm->written += n;
+		cprm->written += nr;
----

That's fixed now, this applies to v4.6-rc3. Please apply or let me know
if there's anything else I messed up.

Original cover letter below:

Hi,

Someone here reported that they were getting truncated core dumps even
when RLIMIT_CORE was larger than the physical memory of the machine. It
looks some cleanup patches back in v3.13 [1] changed the behaviour of
the limit to also charge for sparse areas of a file. Here's an example
in 4.5-rc4, where a.out is:

----
#include <signal.h>

int main(int argc, char **argv)
{
	raise(SIGQUIT);
	return 0;
}
----

Make sure that your sysctl kernel.core_pattern isn't piping anywhere,
and be aware that there's some stupidity about the units used for ulimit
-c in different shells (bash uses 1024-byte blocks, same as what's shown
by du by default).

----
# ulimit -c unlimited
# ./a.out
Quit (core dumped)
# du core.248
88      core.248
# du --apparent-size core.248
232     core.248
# ulimit -c 128
# ./a.out
Quit (core dumped)
# du core.252
64      core.252
# du --apparent-size core.252
72      core.252
----

These 2 patches restore the original behavior:

----
# ulimit -c 128
# ./a.out
Quit (core dumped)
# du core.245
88      core.245
# du --apparent-size core.245
232     core.245
----

Patch 1 gets rid of cprm->written, since as far as I could tell, it's
always going to be equal to cprm->file->f_pos. Patch 2 reintroduces
cprm->written as the number of bytes actually written to the file, not
including what we seek over.

This series is based on 4.5-rc4. Al, could you apply these?

Thanks!

1: http://lkml.iu.edu/hypermail/linux/kernel/1310.1/00758.html

Appendix A: a quick sanity test with the patches applied

----
# ulimit -c unlimited
# python -c 'import os, signal; l = [0] * 1024 * 1024; os.kill(os.getpid(), signal.SIGQUIT)'
Quit (core dumped)
# du core.262
12268   core.262
# du --apparent-size core.262
12628   core.262
# ulimit -c 1024
# python -c 'import os, signal; l = [0] * 1024 * 1024; os.kill(os.getpid(), signal.SIGQUIT)'
Quit (core dumped)
# du core.266
1024    core.266
# du --apparent-size core.266
1024    core.266
----

Omar Sandoval (2):
  coredump: get rid of coredump_params->written
  coredump: only charge written data against RLIMIT_CORE

 arch/powerpc/platforms/cell/spufs/coredump.c | 5 +++--
 fs/binfmt_elf.c                              | 2 +-
 fs/binfmt_elf_fdpic.c                        | 2 +-
 fs/coredump.c                                | 5 +----
 4 files changed, 6 insertions(+), 8 deletions(-)

-- 
2.8.0