sparse triggers OOM killer

sparse triggers OOM killer

[Sam Ravnborg]

[-- Attachment #1: Type: text/plain, Size: 2430 bytes --]

I have played a little with attached patch.
It teach kbuild to check all files in one go in a directory when
sparsing the kernel.

I did a make allmodconfig and then ran:
make fs/xfs/ C=A

On my machine with 1 GB RAM it started a swap storm and later the OOM
triggered.
xfs is one of the bigger modules but still I was suprised to see it
causing such dramatic effect on my box.
[amd64 3.7 GHz  UP  1 GB RAM]

There was no oops or similar and sparse just exited after a while with
an errorcode (137).
Now I wonder if I have hit a bug in sparse or this is what I should
expect.

An easy way to try it out would be to run the sparse command from
attached file (it's long..).
If I can do anything to find out if this is a sparse bug or just
expected behaviour please say so.

	Sam

diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index c33e62b..7e1b40a 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -85,16 +85,23 @@ __build: $(if $(KBUILD_BUILTIN),$(builti
 	@:
 
 # Linus' kernel sanity checking tool
-ifneq ($(KBUILD_CHECKSRC),0)
-  ifeq ($(KBUILD_CHECKSRC),2)
-    quiet_cmd_force_checksrc = CHECK   $<
-          cmd_force_checksrc = $(CHECK) $(CHECKFLAGS) $(c_flags) $< ;
-  else
-      quiet_cmd_checksrc     = CHECK   $<
-            cmd_checksrc     = $(CHECK) $(CHECKFLAGS) $(c_flags) $< ;
-  endif
+ifeq ($(KBUILD_CHECKSRC),1)
+  quiet_cmd_checksrc     = CHECK   $<
+        cmd_checksrc     = $(CHECK) $(CHECKFLAGS) $(c_flags) $< ;
+endif
+ifeq ($(KBUILD_CHECKSRC),2)
+  quiet_cmd_force_checksrc = CHECK   $<
+        cmd_force_checksrc = $(CHECK) $(CHECKFLAGS) $(c_flags) $< ;
+endif
+ifeq ($(KBUILD_CHECKSRC),A)
+  all-c-files = $(wildcard $(patsubst %.o, %.c, $(real-objs-y) $(real-objs-m)))
+  all-o-files = $(patsubst %.c, %.o, $(all-c-files))
+  all-o-flags = $(strip $(foreach o, $(all-o-files), $(CFLAGS_$(notdir $(o)))))
+  quiet_cmd_sparse-all = CHECK $(notdir $(all-c-files))
+        cmd_sparse-all = $(CHECK) $(CHECKFLAGS) \
+                                  $(c_flags) $(all-o-flags) $(all-c-files)
+  sparse-all = $(call cmd,sparse-all)
 endif
-
 
 # Compile C sources (.c)
 # ---------------------------------------------------------------------------
@@ -256,6 +263,7 @@ cmd_link_o_target = $(if $(strip $(obj-y
 		      rm -f $@; $(AR) rcs $@)
 
 $(builtin-target): $(obj-y) FORCE
+	$(call sparse-all)
 	$(call if_changed,link_o_target)
 
 targets += $(builtin-target)

[-- Attachment #2: sparse-xfs --]
[-- Type: text/plain, Size: 2511 bytes --]

sparse -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ -Wbitwise -D__x86_64__ -m64 -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.4/include -Wp,-MD,fs/xfs/.built-in.o.d -nostdinc -isystem /usr/lib/gcc/x86_64-pc-linux-gnu/3.4.4/include -D__KERNEL__ -Iinclude -include include/linux/autoconf.h -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -ffreestanding -Os -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -march=k8 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wdeclaration-after-statement -Ifs/xfs -Ifs/xfs/linux-2.6 -funsigned-char -DKBUILD_STR(s)=#s -DKBUILD_BASENAME=KBUILD_STR(built_in) -DKBUILD_MODNAME=KBUILD_STR(built_in) fs/xfs/quota/xfs_dquot.c fs/xfs/quota/xfs_dquot_item.c fs/xfs/quota/xfs_trans_dquot.c fs/xfs/quota/xfs_qm_syscalls.c fs/xfs/quota/xfs_qm_bhv.c fs/xfs/quota/xfs_qm.c fs/xfs/quota/xfs_qm_stats.c fs/xfs/xfs_rtalloc.c fs/xfs/xfs_acl.c fs/xfs/linux-2.6/xfs_stats.c fs/xfs/linux-2.6/xfs_sysctl.c fs/xfs/linux-2.6/xfs_ioctl32.c fs/xfs/linux-2.6/xfs_export.c fs/xfs/xfs_alloc.c fs/xfs/xfs_alloc_btree.c fs/xfs/xfs_attr.c fs/xfs/xfs_attr_leaf.c fs/xfs/xfs_behavior.c fs/xfs/xfs_bit.c fs/xfs/xfs_bmap.c fs/xfs/xfs_bmap_btree.c fs/xfs/xfs_btree.c fs/xfs/xfs_buf_item.c fs/xfs/xfs_da_btree.c fs/xfs/xfs_dir.c fs/xfs/xfs_dir2.c fs/xfs/xfs_dir2_block.c fs/xfs/xfs_dir2_data.c fs/xfs/xfs_dir2_leaf.c fs/xfs/xfs_dir2_node.c fs/xfs/xfs_dir2_sf.c fs/xfs/xfs_dir_leaf.c fs/xfs/xfs_error.c fs/xfs/xfs_extfree_item.c fs/xfs/xfs_fsops.c fs/xfs/xfs_ialloc.c fs/xfs/xfs_ialloc_btree.c fs/xfs/xfs_iget.c fs/xfs/xfs_inode.c fs/xfs/xfs_inode_item.c fs/xfs/xfs_iocore.c fs/xfs/xfs_iomap.c fs/xfs/xfs_itable.c fs/xfs/xfs_dfrag.c fs/xfs/xfs_log.c fs/xfs/xfs_log_recover.c fs/xfs/xfs_mount.c fs/xfs/xfs_rename.c fs/xfs/xfs_trans.c fs/xfs/xfs_trans_ail.c fs/xfs/xfs_trans_buf.c fs/xfs/xfs_trans_extfree.c fs/xfs/xfs_trans_inode.c fs/xfs/xfs_trans_item.c fs/xfs/xfs_utils.c fs/xfs/xfs_vfsops.c fs/xfs/xfs_vnodeops.c fs/xfs/xfs_rw.c fs/xfs/xfs_dmops.c fs/xfs/xfs_qmops.c fs/xfs/linux-2.6/kmem.c fs/xfs/linux-2.6/xfs_aops.c fs/xfs/linux-2.6/xfs_buf.c fs/xfs/linux-2.6/xfs_file.c fs/xfs/linux-2.6/xfs_fs_subr.c fs/xfs/linux-2.6/xfs_globals.c fs/xfs/linux-2.6/xfs_ioctl.c fs/xfs/linux-2.6/xfs_iops.c fs/xfs/linux-2.6/xfs_lrw.c fs/xfs/linux-2.6/xfs_super.c fs/xfs/linux-2.6/xfs_vfs.c fs/xfs/linux-2.6/xfs_vnode.c fs/xfs/support/debug.c fs/xfs/support/move.c fs/xfs/support/uuid.c

Re: sparse triggers OOM killer

[Linus Torvalds]

On Sat, 7 Jan 2006, Sam Ravnborg wrote:
> 
> There was no oops or similar and sparse just exited after a while with
> an errorcode (137).

That's just SIGKILL (128+9). Which is normal for the OOM killer.

> Now I wonder if I have hit a bug in sparse or this is what I should
> expect.

Well, sparse does keep a _lot_ of stuff in memory, and the "do many files 
at once" will basically keep every single one (with full types, full 
linearization etc) in memory at the same time.

It's probably fairly easy to fix: I should just make sparse release all 
the linearizations and symbols when they go out of file scope.

The "do many files at once" thing really was just a quick hack, so the 
lack of memory release is not that susprising.

I'll see what I can do.

		Linus

Re: sparse triggers OOM killer

[Sam Ravnborg]

Hi Linus.

On Sat, Jan 07, 2006 at 10:07:10AM -0800, Linus Torvalds wrote:
> 
> 
> On Sat, 7 Jan 2006, Sam Ravnborg wrote:
> > 
> > There was no oops or similar and sparse just exited after a while with
> > an errorcode (137).
> 
> That's just SIGKILL (128+9). Which is normal for the OOM killer.
> 
> > Now I wonder if I have hit a bug in sparse or this is what I should
> > expect.
> 
> Well, sparse does keep a _lot_ of stuff in memory, and the "do many files 
> at once" will basically keep every single one (with full types, full 
> linearization etc) in memory at the same time.
> 
> It's probably fairly easy to fix: I should just make sparse release all 
> the linearizations and symbols when they go out of file scope.
> 
> The "do many files at once" thing really was just a quick hack, so the 
> lack of memory release is not that susprising.
> 
> I'll see what I can do.

Any news on this (call this a polite remineder)?
I have not tried to dig into it myself - assuming that it will take me
considerably time to do right..

	Sam