* NTFS - Kernel memory leak in driver for kernel 2.4.28?
@ 2005-02-16 16:28 Martin Bogomolni
2005-02-16 16:55 ` NTFS - Kernel memory leak in driver for kernel 2.4.28 (update) Martin Bogomolni
2005-02-16 17:00 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? linux-os
0 siblings, 2 replies; 7+ messages in thread
From: Martin Bogomolni @ 2005-02-16 16:28 UTC (permalink / raw)
To: linux-kernel
I am maintaining an embedded linux environment that is used for
disaster data recovery for Xpoint Technologies. The environment is
based on kernel 2.4.28, and a 32megabyte ramdisk containing bash, a
minimal busybox environment, and the backup software with no swap.
During software development to add a new feature to restore individual
files from an NTFS partition, the software started to exhibit out of
memory errors.
The following procedure can be used to replicate the error :
Boot a 2.4.28 kernel, with a 32meg ramdisk and minimal busybox
environment in the initrd ramdisk with the "find" utility. The NTFS
driver can be compiled in, or can be a module. This has also been
experimentally verified using a small program that replicates "find"
with careful attention to malloc( ) and free ( ). However, the
standard "find" utility suffices to show the problem :
Note the available memory in /proc/meminfo, and look at /proc/slabinfo
Mount a large NTFS filesystem, and then run : find /path/to/ntfs -name *
Note the available memory in /proc/meminfo, and look at /proc/slabinfo
Unmounting the ntfs filesystem, and unloading the kernel module (when
compiled as such) had no effect on the amount of free ram available
after the 'find' command is run. malloc( ) fails to allocate
afterwards. so the kernel does not free any of the missing RAM for
malloc( ).
-------------------------------------------------------------------------
On a testbed machine (P4, 256Mb RAM = IBM Netvista) :
Free Memory before : 209768448 (~200 Mb)
Free Memory after : 3579904 (~35 Mb)
Linux (none) 2.4.28 #1 Tue Feb 15 10:25:25 EST 2005 i686 unknown
(/proc/meminfo before)
total: used: free: shared: buffers: cached:
Mem: 252907520 43139072 209768448 0 385024 38862848
Swap: 0 0 0
MemTotal: 246980 kB
MemFree: 204852 kB
MemShared: 0 kB
Buffers: 376 kB
Cached: 37952 kB
SwapCached: 0 kB
Active: 2784 kB
Inactive: 35552 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 246980 kB
LowFree: 204852 kB
SwapTotal: 0 kB
SwapFree: 0 kB
(/proc/slabinfo before)
slabinfo - version: 1.1
kmem_cache 61 72 108 2 2 1
tcp_tw_bucket 0 0 96 0 0 1
tcp_bind_bucket 0 0 32 0 0 1
tcp_open_request 0 0 64 0 0 1
inet_peer_cache 0 0 64 0 0 1
ip_fib_hash 4 113 32 1 1 1
ip_dst_cache 0 0 160 0 0 1
arp_cache 0 0 96 0 0 1
blkdev_requests 3072 4120 96 77 103 1
nfs_write_data 0 0 352 0 0 1
nfs_read_data 0 0 352 0 0 1
nfs_page 0 0 96 0 0 1
journal_head 0 0 48 0 0 1
revoke_table 0 0 12 0 0 1
revoke_record 0 0 32 0 0 1
dnotify_cache 0 0 20 0 0 1
file_lock_cache 0 0 92 0 0 1
fasync_cache 0 0 16 0 0 1
uid_cache 0 0 32 0 0 1
skbuff_head_cache 1 24 160 1 1 1
sock 5 9 864 1 1 2
sigqueue 0 29 132 0 1 1
kiobuf 0 0 64 0 0 1
cdev_cache 6 59 64 1 1 1
bdev_cache 3 59 64 1 1 1
mnt_cache 11 59 64 1 1 1
inode_cache 167 168 480 21 21 1
dentry_cache 224 240 128 8 8 1
filp 37 60 128 2 2 1
names_cache 0 2 4096 0 2 1
buffer_head 5941 5960 96 149 149 1
mm_struct 7 24 160 1 1 1
vm_area_struct 69 120 96 2 3 1
fs_cache 6 113 32 1 1 1
files_cache 7 18 416 1 2 1
signal_act 8 12 1312 3 4 1
size-131072(DMA) 0 0 131072 0 0 32
size-131072 0 0 131072 0 0 32
size-65536(DMA) 0 0 65536 0 0 16
size-65536 0 0 65536 0 0 16
size-32768(DMA) 0 0 32768 0 0 8
size-32768 0 1 32768 0 1 8
size-16384(DMA) 0 0 16384 0 0 4
size-16384 1 2 16384 1 2 4
size-8192(DMA) 0 0 8192 0 0 2
size-8192 4 5 8192 4 5 2
size-4096(DMA) 0 0 4096 0 0 1
size-4096 6 6 4096 6 6 1
size-2048(DMA) 0 0 2048 0 0 1
size-2048 5 6 2048 3 3 1
size-1024(DMA) 0 4 1024 0 1 1
size-1024 95 96 1024 24 24 1
size-512(DMA) 0 0 512 0 0 1
size-512 28 32 512 4 4 1
size-256(DMA) 0 0 256 0 0 1
size-256 6 15 256 1 1 1
size-128(DMA) 0 0 128 0 0 1
size-128 490 510 128 17 17 1
size-64(DMA) 0 0 64 0 0 1
size-64 108 118 64 2 2 1
size-32(DMA) 0 0 32 0 0 1
size-32 206 339 32 2 3 1
----------- run : find /mnt/ntfs -name \*
(/proc/meminfo after)
total: used: free: shared: buffers: cached:
Mem: 252907520 249327616 3579904 0 39251968 37748736
Swap: 0 0 0
MemTotal: 246980 kB
MemFree: 3496 kB
MemShared: 0 kB
Buffers: 38332 kB
Cached: 36864 kB
SwapCached: 0 kB
Active: 67084 kB
Inactive: 8120 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 246980 kB
LowFree: 3496 kB
SwapTotal: 0 kB
SwapFree: 0 kB
(/proc/slabinfo after)
slabinfo - version: 1.1
kmem_cache 61 72 108 2 2 1
tcp_tw_bucket 0 0 96 0 0 1
tcp_bind_bucket 0 0 32 0 0 1
tcp_open_request 0 0 64 0 0 1
inet_peer_cache 0 0 64 0 0 1
ip_fib_hash 4 113 32 1 1 1
ip_dst_cache 0 0 160 0 0 1
arp_cache 0 0 96 0 0 1
blkdev_requests 3072 3600 96 77 90 1
nfs_write_data 0 0 352 0 0 1
nfs_read_data 0 0 352 0 0 1
nfs_page 0 0 96 0 0 1
journal_head 0 0 48 0 0 1
revoke_table 0 0 12 0 0 1
revoke_record 0 0 32 0 0 1
dnotify_cache 0 0 20 0 0 1
file_lock_cache 0 0 92 0 0 1
fasync_cache 0 0 16 0 0 1
uid_cache 0 0 32 0 0 1
skbuff_head_cache 1 24 160 1 1 1
sock 5 9 864 1 1 2
sigqueue 0 29 132 0 1 1
kiobuf 0 0 64 0 0 1
cdev_cache 5 59 64 1 1 1
bdev_cache 3 59 64 1 1 1
mnt_cache 12 59 64 1 1 1
inode_cache 26560 26560 480 3320 3320 1
dentry_cache 20721 21840 128 728 728 1
filp 42 60 128 2 2 1
names_cache 0 1 4096 0 1 1
buffer_head 80852 80880 96 2022 2022 1
mm_struct 7 24 160 1 1 1
vm_area_struct 69 120 96 2 3 1
fs_cache 6 113 32 1 1 1
files_cache 7 18 416 1 2 1
signal_act 8 12 1312 3 4 1
size-131072(DMA) 0 0 131072 0 0 32
size-131072 0 0 131072 0 0 32
size-65536(DMA) 0 0 65536 0 0 16
size-65536 0 0 65536 0 0 16
size-32768(DMA) 0 0 32768 0 0 8
size-32768 0 1 32768 0 1 8
size-16384(DMA) 0 0 16384 0 0 4
size-16384 1 2 16384 1 2 4
size-8192(DMA) 0 0 8192 0 0 2
size-8192 4 5 8192 4 5 2
size-4096(DMA) 0 0 4096 0 0 1
size-4096 22252 22288 4096 22252 22288 1
size-2048(DMA) 0 0 2048 0 0 1
size-2048 5 6 2048 3 3 1
size-1024(DMA) 0 4 1024 0 1 1
size-1024 26781 26992 1024 6696 6748 1
size-512(DMA) 0 0 512 0 0 1
size-512 27801 27960 512 3476 3495 1
size-256(DMA) 0 0 256 0 0 1
size-256 1718 1725 256 115 115 1
size-128(DMA) 0 0 128 0 0 1
size-128 57680 58050 128 1923 1935 1
size-64(DMA) 0 0 64 0 0 1
size-64 12190 12213 64 207 207 1
size-32(DMA) 0 0 32 0 0 1
size-32 55206 55483 32 491 491 1
^ permalink raw reply [flat|nested] 7+ messages in thread* NTFS - Kernel memory leak in driver for kernel 2.4.28 (update) 2005-02-16 16:28 NTFS - Kernel memory leak in driver for kernel 2.4.28? Martin Bogomolni @ 2005-02-16 16:55 ` Martin Bogomolni 2005-02-16 17:00 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? linux-os 1 sibling, 0 replies; 7+ messages in thread From: Martin Bogomolni @ 2005-02-16 16:55 UTC (permalink / raw) To: linux-kernel I noticed that the inode_cache is quite high. No matter what I do to the tunable system parameters in /proc (below) I can't get the amount of memory taken by the inode cache to reduce and become freed for userspace programs. /proc/sys/vm/vm_mapped_ratio ( tried 20 - 100 ) /proc/sys/vm/vm_vfs_scan_ratio ( tried 2 - 6 ) How often does the dcache/icache shrinking take place in 2.4? On vfat/fat32, ext2/3 filesystems I don't see anywhere near the amount of lost memory than when using the NTFS filesystem. This is what led me to think that the NTFS driver was leaking, rather than this being an inode cache problem. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NTFS - Kernel memory leak in driver for kernel 2.4.28? 2005-02-16 16:28 NTFS - Kernel memory leak in driver for kernel 2.4.28? Martin Bogomolni 2005-02-16 16:55 ` NTFS - Kernel memory leak in driver for kernel 2.4.28 (update) Martin Bogomolni @ 2005-02-16 17:00 ` linux-os 2005-02-16 17:16 ` Martin Bogomolni 1 sibling, 1 reply; 7+ messages in thread From: linux-os @ 2005-02-16 17:00 UTC (permalink / raw) To: Martin Bogomolni; +Cc: Linux kernel On Wed, 16 Feb 2005, Martin Bogomolni wrote: [SNIPPED...] > after the 'find' command is run. malloc( ) fails to allocate > afterwards. so the kernel does not free any of the missing RAM for > malloc( ). > Whatever program is using malloc() is either corrupting its buffers or has a memory leak. Malloc() will always succeed even if the kernel has no memory available. This is because the actual allocation only occurs when the program attempts to write to one of those pages malloc() "promised". When you look at kernel memory after using `find` everything, the directory of everything you have accessed remains in memory until the kernel needs page(s) to give to processes. So, the bottom line is, if malloc() returns NULL, you have a problem with your program. It has nothing to do with the kernel and "discovering" that the kernel has used all available RAM for temporary buffers is not interesting. [SNIPPED...] Cheers, Dick Johnson Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips). Notice : All mail here is now cached for review by Dictator Bush. 98.36% of all statistics are fiction. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NTFS - Kernel memory leak in driver for kernel 2.4.28? 2005-02-16 17:00 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? linux-os @ 2005-02-16 17:16 ` Martin Bogomolni 2005-02-16 18:03 ` kernel 2.4 inode/dentry cache not clearing on umount? Martin Bogomolni 2005-02-17 11:40 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? Anton Altaparmakov 0 siblings, 2 replies; 7+ messages in thread From: Martin Bogomolni @ 2005-02-16 17:16 UTC (permalink / raw) To: linux-os Dick, I should say that the malloc() succeeds, but the 16mb I need for the buffer are not available. Since there is no swap/page file in the embedded environment, there isn't enough memory left afterwards for the buffer. After taking another look at the problem, the kernel has a lot of memory tied up in the inode and dentry cache. I've tuned /proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio with no real success in shrinking the amount of memory used by these caches. Is there a way to tune and shring the overall amount of memory the kernel attempts to use for the dentry/inode cache, or make it much, much more aggressive at clearing it? -Martin On Wed, 16 Feb 2005 12:00:53 -0500 (EST), linux-os <linux-os@analogic.com> wrote: > On Wed, 16 Feb 2005, Martin Bogomolni wrote: > > [SNIPPED...] > > > after the 'find' command is run. malloc( ) fails to allocate > > afterwards. so the kernel does not free any of the missing RAM for > > malloc( ). > > > > Whatever program is using malloc() is either corrupting > its buffers or has a memory leak. > > Malloc() will always succeed even if the kernel has no > memory available. This is because the actual allocation > only occurs when the program attempts to write to one > of those pages malloc() "promised". > > When you look at kernel memory after using `find` everything, > the directory of everything you have accessed remains in > memory until the kernel needs page(s) to give to processes. > > So, the bottom line is, if malloc() returns NULL, you have > a problem with your program. It has nothing to do with > the kernel and "discovering" that the kernel has used > all available RAM for temporary buffers is not interesting. > > [SNIPPED...] > > Cheers, > Dick Johnson > Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips). > Notice : All mail here is now cached for review by Dictator Bush. > 98.36% of all statistics are fiction. > ^ permalink raw reply [flat|nested] 7+ messages in thread
* kernel 2.4 inode/dentry cache not clearing on umount? 2005-02-16 17:16 ` Martin Bogomolni @ 2005-02-16 18:03 ` Martin Bogomolni 2005-02-16 18:10 ` Martin Bogomolni 2005-02-17 11:40 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? Anton Altaparmakov 1 sibling, 1 reply; 7+ messages in thread From: Martin Bogomolni @ 2005-02-16 18:03 UTC (permalink / raw) To: linux-os Also .. David : Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file.... char *p; p = (char *) malloc( 64*1024*1024 ); I assure you that under these conditions, the malloc( ) will fail with NULL. --------------------------------- Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache. However, if I unmount the filesystem then run: cat /dev/hda > /dev/null This causes the inode/dentry cache to finally shrink and the amount of available free memory increases back to ~200Mb. However, this reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory. On Wed, 16 Feb 2005 09:16:40 -0800, Martin Bogomolni <martinbogo@gmail.com> wrote: > Dick, > > I should say that the malloc() succeeds, but the 16mb I need for the > buffer are not available. Since there is no swap/page file in the > embedded environment, there isn't enough memory left afterwards for > the buffer. > > After taking another look at the problem, the kernel has a lot of > memory tied up in the inode and dentry cache. I've tuned > /proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio > with no real success in shrinking the amount of memory used by these > caches. > > Is there a way to tune and shring the overall amount of memory the > kernel attempts to use for the dentry/inode cache, or make it much, > much more aggressive at clearing it? > > -Martin > > > On Wed, 16 Feb 2005 12:00:53 -0500 (EST), linux-os > <linux-os@analogic.com> wrote: > > On Wed, 16 Feb 2005, Martin Bogomolni wrote: > > > > [SNIPPED...] > > > > > after the 'find' command is run. malloc( ) fails to allocate > > > afterwards. so the kernel does not free any of the missing RAM for > > > malloc( ). > > > > > > > Whatever program is using malloc() is either corrupting > > its buffers or has a memory leak. > > > > Malloc() will always succeed even if the kernel has no > > memory available. This is because the actual allocation > > only occurs when the program attempts to write to one > > of those pages malloc() "promised". > > > > When you look at kernel memory after using `find` everything, > > the directory of everything you have accessed remains in > > memory until the kernel needs page(s) to give to processes. > > > > So, the bottom line is, if malloc() returns NULL, you have > > a problem with your program. It has nothing to do with > > the kernel and "discovering" that the kernel has used > > all available RAM for temporary buffers is not interesting. > > > > [SNIPPED...] > > > > Cheers, > > Dick Johnson > > Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips). > > Notice : All mail here is now cached for review by Dictator Bush. > > 98.36% of all statistics are fiction. > > > ^ permalink raw reply [flat|nested] 7+ messages in thread
* kernel 2.4 inode/dentry cache not clearing on umount? 2005-02-16 18:03 ` kernel 2.4 inode/dentry cache not clearing on umount? Martin Bogomolni @ 2005-02-16 18:10 ` Martin Bogomolni 0 siblings, 0 replies; 7+ messages in thread From: Martin Bogomolni @ 2005-02-16 18:10 UTC (permalink / raw) To: linux-kernel Also .. David : Are you saying that, on a system with 256Megs of ram, of which the kernel is reporting only 3-4Mb free because the inode/dentry caches are taking up most of the memory, and NO page/swap file.... char *p; p = (char *) malloc( 64*1024*1024 ); I assure you that under these conditions, the malloc( ) will fail with NULL. --------------------------------- Now, in the meantime I have discovered that merely unmounting the filesystem is not enough to clear the dcache and icache. However, if I unmount the filesystem then run: cat /dev/hda > /dev/null This causes the inode/dentry cache to finally shrink and the amount of available free memory increases back to ~200Mb. However, this reduction does not immediately take place when the filesystem is unmounted, and while the filesystem is mounted .. the inode/dentry cache does not shrink and leaves only 3Mb of available free memory. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: NTFS - Kernel memory leak in driver for kernel 2.4.28? 2005-02-16 17:16 ` Martin Bogomolni 2005-02-16 18:03 ` kernel 2.4 inode/dentry cache not clearing on umount? Martin Bogomolni @ 2005-02-17 11:40 ` Anton Altaparmakov 1 sibling, 0 replies; 7+ messages in thread From: Anton Altaparmakov @ 2005-02-17 11:40 UTC (permalink / raw) To: Martin Bogomolni; +Cc: linux-os, Linux kernel On Wed, 2005-02-16 at 09:16 -0800, Martin Bogomolni wrote: > I should say that the malloc() succeeds, but the 16mb I need for the > buffer are not available. Since there is no swap/page file in the > embedded environment, there isn't enough memory left afterwards for > the buffer. > > After taking another look at the problem, the kernel has a lot of > memory tied up in the inode and dentry cache. I've tuned > /proc/sys/vm/vm_cache_scan_ratio, vm_mapped_ratio, vm_vfs_scan_ratio > with no real success in shrinking the amount of memory used by these > caches. > > Is there a way to tune and shring the overall amount of memory the > kernel attempts to use for the dentry/inode cache, or make it much, > much more aggressive at clearing it? Are you using the old (1.x) or new (2.x) ntfs driver? If you are using the old one you could try using the new driver. That should be a lot better than the old one in terms of how much memory it uses. You can get an outdated patch (but should still work) for the new driver here: http://sourceforge.net/project/showfiles.php?group_id=13956&package_id=21892 Best regards, Anton -- Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/ ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2005-02-17 11:40 UTC | newest] Thread overview: 7+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2005-02-16 16:28 NTFS - Kernel memory leak in driver for kernel 2.4.28? Martin Bogomolni 2005-02-16 16:55 ` NTFS - Kernel memory leak in driver for kernel 2.4.28 (update) Martin Bogomolni 2005-02-16 17:00 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? linux-os 2005-02-16 17:16 ` Martin Bogomolni 2005-02-16 18:03 ` kernel 2.4 inode/dentry cache not clearing on umount? Martin Bogomolni 2005-02-16 18:10 ` Martin Bogomolni 2005-02-17 11:40 ` NTFS - Kernel memory leak in driver for kernel 2.4.28? Anton Altaparmakov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox