Inactive memory keep growing and how to release it?

Inactive memory keep growing and how to release it?

[Lenky Gao]

Hi,

When i just run a test on Centos 6.2 as follows:
################################
#!/bin/bash

while true
do

        file="/tmp/filetest"

        echo $file

        dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null

        sleep 5
done
################################

the inactive memory keep growing:

#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   420144 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   554662 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   882824 kB

and i cannot reclaim it:

# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   882824 kB
# sync;
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   777664 kB

My file system type is ext3, mounted with the option data=journal and
it is easy to reproduce. If the mount option is data=ordered,
reproducing this problom need more time. Because of the problom, when
in the final situation, i cannot kmalloc a larger contiguous memory,
especially in interrupt context. I have tested on other version
kernel, such as 2.6.30 and 3.6.11, the problom also exists.

Is this a bug?

Or can you give some tips to avoid this?

-- 
Regards,

Lenky

a problem about ext4.

[zhuyj]

Hi, all

I install Ubuntu 11.04 on Dell-Latitude-D630.
Linux kernel is 2.6.39.4(I compiled)

uname -a
Linux Dell-Latitude-D630 2.6.39.4-zhuyj26394 #2 SMP Tue Mar 12 13:23:06 
CST 2013 i686 i686 i386 GNU/Linux

I followed these steps:

****************************Begin***************************************
1. get ltp-full-20130109.bz2 and run "./configure;make", then we will 
get fsstress;

2. mount -t tmpfs -o size=512m tmpfs /tmp

3. dd if=/dev/zero of=/tmp/img bs=1024 count=$((500*1024))

4. ls -lah /tmp/img

total 501M
drwxrwxrwt  2 root root   60 2013-03-12 15:18 .
drwxr-xr-x 23 root root 4.0K 2013-03-12 13:09 ..
-rw-r--r--  1 root root 500M 2013-03-12 15:18 img

5. df

Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            113317544   6994416 100566868   7% /
none                   1023772       696   1023076   1% /dev
none                   1030440        80   1030360   1% /dev/shm
none                   1030440        92   1030348   1% /var/run
none                   1030440         0   1030440   0% /var/lock
tmpfs                   524288    512504     11784  98% /tmp

6. mkfs.ext4 /tmp/img

mke2fs 1.41.14 (22-Dec-2010)
/tmp/img is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
     8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 30 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

7. mkdir /tmp/mnt

8. mount -t ext4 -o loop /tmp/img /tmp/mnt

9. df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1            113317544   6994424 100566860   7% /
none                   1023772       696   1023076   1% /dev
none                   1030440        80   1030360   1% /dev/shm
none                   1030440        92   1030348   1% /var/run
none                   1030440         0   1030440   0% /var/lock
tmpfs                   524288    512504     11784  98% /tmp
/dev/loop0              495844     10510    459734   3% /tmp/mnt

10. cd /tmp/mnt; mkdir /tmp/mnt/tmp1 /tmp/mnt/tmp2; cp ~/fsstress /tmp/mnt
./fsstress -c -p 5 -n 10000 -d . -l 0 -s 10M &
./fsstress -c -p 5 -n 10000 -d ./tmp1 -l 0 -s 10M &
./fsstress -c -p 5 -n 10000 -d ./tmp2 -l 0 -s 10M &

11.when the following appears, run "ps;killall fsstress;ps"
./tmp1: No such file or directory
./tmp2: No such file or directory

[2]-  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp1 -l 0 
-s 10M
[3]+  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp2 -l 0 
-s 10M

These messages will appear:

ps;killall fsstress;ps
   PID TTY          TIME CMD
  1366 pts/0    00:00:01 bash
  1600 pts/0    00:00:00 fsstress
  1633 pts/0    00:00:07 fsstress
  1634 pts/0    00:00:06 fsstress
  1635 pts/0    00:00:06 fsstress
  1636 pts/0    00:00:07 fsstress
  1637 pts/0    00:00:07 fsstress
  1640 pts/0    00:00:00 ps
[1]+  Terminated              ./fsstress -c -p 5 -n 10000 -d . -l 0 -s 10M
   PID TTY          TIME CMD
  1366 pts/0    00:00:01 bash
  1642 pts/0    00:00:00 ps

12. /bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches

13. rm -Rf p*;rm -Rf tmp*

rm: cannot remove `p0': Directory not empty
rm: cannot remove `p1/d99d/d9b4/da40/da71/da74': Directory not empty
rm: cannot remove `p2/dc/d79/f56f': Input/output error
rm: cannot remove `p2/dc/d79/f362': Input/output error
rm: cannot remove `p2/dc/d79/c460': Input/output error
rm: cannot remove `p2/dc/d79/cf4': Input/output error
rm: cannot remove `p2/dc/d79/f3b0': Input/output error
rm: cannot remove `p2/dc/d79/d107/d95e/d7f7/fb50': Input/output error
rm: cannot remove `p2/dc/d79/d107/d95e/f99e': Input/output error
rm: cannot remove `p2/dc/d79/d107/d95e/fc40': Input/output error
rm: cannot remove `p2/dc/d79/d107/f7ab': Input/output error
rm: cannot remove `p2/dc/d79/d107/d547/c843': Input/output error
rm: cannot remove `p2/dc/d79/d107/d547/c871': Input/output error
rm: cannot remove `p2/dc/d79/d107/fa77': Input/output error
rm: cannot remove `p2/dc/d79/f12e': Input/output error
rm: cannot remove `p2/dc/d79/c623': Input/output error
rm: cannot remove `p2/dc/d79/f207': Input/output error
rm: cannot remove `p2/dc/d79/c216': Input/output error
rm: cannot remove `p2/dc/d79/f23d': Input/output error
rm: cannot remove `p2/dc/d79/c250': Input/output error
rm: cannot remove `p2/dc/d79/f274': Input/output error
rm: cannot remove `p2/dc/d79/f285': Input/output error
rm: cannot remove `p2/dc/d79/f2af': Input/output error
rm: cannot remove `p2/dc/d79/c2d1': Input/output error
rm: cannot remove `p2/dc/d79/c2ee': Input/output error
rm: cannot remove `p2/dc/d79/c35a': Input/output error
.....

14. run "dmesg"

[  469.002632] EXT4-fs warning (device loop0): dx_probe:364: 
Unrecognised inode hash code 12
[  469.002647] EXT4-fs warning (device loop0): dx_probe:472: Corrupt dir 
inode 8423, running e2fsck is recommended.
[  469.018830] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8926
[  469.019792] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8816
[  469.019950] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8570
[  469.020729] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 7814
[  469.021227] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 9259
[  469.048610] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#10178: comm rm: deleted inode referenced: 7791
[  469.049919] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#10177: comm rm: deleted inode referenced: 8100
[  469.053826] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#10177: comm rm: deleted inode referenced: 8822
[  469.060481] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8417: comm rm: deleted inode referenced: 8113
[  469.062336] EXT4-fs warning (device loop0): dx_probe:364: 
Unrecognised inode hash code 12
[  469.062349] EXT4-fs warning (device loop0): dx_probe:472: Corrupt dir 
inode 10163, running e2fsck is recommended.
[  469.065721] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#10163: comm rm: deleted inode referenced: 9027
[  469.066138] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#10163: comm rm: deleted inode referenced: 9090
[  469.067526] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8417: comm rm: deleted inode referenced: 8443
[  469.068584] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8067
[  469.069280] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 9694
[  469.069893] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8100
[  469.070123] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8635
[  469.070509] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
#8423: comm rm: deleted inode referenced: 8929
......

***************************End**********************************************

But if I use ext3, this will not occur.
If I do not run "/bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches", this 
will not occur.
If I use kernel>=3.0, this will not occur.

But I use ext4 and run "/bin/sync; /bin/echo 1 > 
/proc/sys/vm/drop_caches", these error messages.

So is this is a bug?

Re: a problem about ext4.

[zhuyj]

Hi, all

Almost 2.6.3x has this problem. Is it a problem?

Any reply is appreciated.
Zhuyj

On 03/12/2013 03:41 PM, zhuyj wrote:
> Hi, all
>
> I install Ubuntu 11.04 on Dell-Latitude-D630.
> Linux kernel is 2.6.39.4(I compiled)
>
> uname -a
> Linux Dell-Latitude-D630 2.6.39.4-zhuyj26394 #2 SMP Tue Mar 12 
> 13:23:06 CST 2013 i686 i686 i386 GNU/Linux
>
> I followed these steps:
>
> ****************************Begin***************************************
> 1. get ltp-full-20130109.bz2 and run "./configure;make", then we will 
> get fsstress;
>
> 2. mount -t tmpfs -o size=512m tmpfs /tmp
>
> 3. dd if=/dev/zero of=/tmp/img bs=1024 count=$((500*1024))
>
> 4. ls -lah /tmp/img
>
> total 501M
> drwxrwxrwt  2 root root   60 2013-03-12 15:18 .
> drwxr-xr-x 23 root root 4.0K 2013-03-12 13:09 ..
> -rw-r--r--  1 root root 500M 2013-03-12 15:18 img
>
> 5. df
>
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda1            113317544   6994416 100566868   7% /
> none                   1023772       696   1023076   1% /dev
> none                   1030440        80   1030360   1% /dev/shm
> none                   1030440        92   1030348   1% /var/run
> none                   1030440         0   1030440   0% /var/lock
> tmpfs                   524288    512504     11784  98% /tmp
>
> 6. mkfs.ext4 /tmp/img
>
> mke2fs 1.41.14 (22-Dec-2010)
> /tmp/img is not a block special device.
> Proceed anyway? (y,n) y
> Filesystem label=
> OS type: Linux
> Block size=1024 (log=0)
> Fragment size=1024 (log=0)
> Stride=0 blocks, Stripe width=0 blocks
> 128016 inodes, 512000 blocks
> 25600 blocks (5.00%) reserved for the super user
> First data block=1
> Maximum filesystem blocks=67633152
> 63 block groups
> 8192 blocks per group, 8192 fragments per group
> 2032 inodes per group
> Superblock backups stored on blocks:
>     8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
>
> Writing inode tables: done
> Creating journal (8192 blocks): done
> Writing superblocks and filesystem accounting information: done
>
> This filesystem will be automatically checked every 30 mounts or
> 180 days, whichever comes first.  Use tune2fs -c or -i to override.
>
> 7. mkdir /tmp/mnt
>
> 8. mount -t ext4 -o loop /tmp/img /tmp/mnt
>
> 9. df
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sda1            113317544   6994424 100566860   7% /
> none                   1023772       696   1023076   1% /dev
> none                   1030440        80   1030360   1% /dev/shm
> none                   1030440        92   1030348   1% /var/run
> none                   1030440         0   1030440   0% /var/lock
> tmpfs                   524288    512504     11784  98% /tmp
> /dev/loop0              495844     10510    459734   3% /tmp/mnt
>
> 10. cd /tmp/mnt; mkdir /tmp/mnt/tmp1 /tmp/mnt/tmp2; cp ~/fsstress 
> /tmp/mnt
> ./fsstress -c -p 5 -n 10000 -d . -l 0 -s 10M &
> ./fsstress -c -p 5 -n 10000 -d ./tmp1 -l 0 -s 10M &
> ./fsstress -c -p 5 -n 10000 -d ./tmp2 -l 0 -s 10M &
>
> 11.when the following appears, run "ps;killall fsstress;ps"
> ./tmp1: No such file or directory
> ./tmp2: No such file or directory
>
> [2]-  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp1 -l 
> 0 -s 10M
> [3]+  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp2 -l 
> 0 -s 10M
>
> These messages will appear:
>
> ps;killall fsstress;ps
>   PID TTY          TIME CMD
>  1366 pts/0    00:00:01 bash
>  1600 pts/0    00:00:00 fsstress
>  1633 pts/0    00:00:07 fsstress
>  1634 pts/0    00:00:06 fsstress
>  1635 pts/0    00:00:06 fsstress
>  1636 pts/0    00:00:07 fsstress
>  1637 pts/0    00:00:07 fsstress
>  1640 pts/0    00:00:00 ps
> [1]+  Terminated              ./fsstress -c -p 5 -n 10000 -d . -l 0 -s 
> 10M
>   PID TTY          TIME CMD
>  1366 pts/0    00:00:01 bash
>  1642 pts/0    00:00:00 ps
>
> 12. /bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches
>
> 13. rm -Rf p*;rm -Rf tmp*
>
> rm: cannot remove `p0': Directory not empty
> rm: cannot remove `p1/d99d/d9b4/da40/da71/da74': Directory not empty
> rm: cannot remove `p2/dc/d79/f56f': Input/output error
> rm: cannot remove `p2/dc/d79/f362': Input/output error
> rm: cannot remove `p2/dc/d79/c460': Input/output error
> rm: cannot remove `p2/dc/d79/cf4': Input/output error
> rm: cannot remove `p2/dc/d79/f3b0': Input/output error
> rm: cannot remove `p2/dc/d79/d107/d95e/d7f7/fb50': Input/output error
> rm: cannot remove `p2/dc/d79/d107/d95e/f99e': Input/output error
> rm: cannot remove `p2/dc/d79/d107/d95e/fc40': Input/output error
> rm: cannot remove `p2/dc/d79/d107/f7ab': Input/output error
> rm: cannot remove `p2/dc/d79/d107/d547/c843': Input/output error
> rm: cannot remove `p2/dc/d79/d107/d547/c871': Input/output error
> rm: cannot remove `p2/dc/d79/d107/fa77': Input/output error
> rm: cannot remove `p2/dc/d79/f12e': Input/output error
> rm: cannot remove `p2/dc/d79/c623': Input/output error
> rm: cannot remove `p2/dc/d79/f207': Input/output error
> rm: cannot remove `p2/dc/d79/c216': Input/output error
> rm: cannot remove `p2/dc/d79/f23d': Input/output error
> rm: cannot remove `p2/dc/d79/c250': Input/output error
> rm: cannot remove `p2/dc/d79/f274': Input/output error
> rm: cannot remove `p2/dc/d79/f285': Input/output error
> rm: cannot remove `p2/dc/d79/f2af': Input/output error
> rm: cannot remove `p2/dc/d79/c2d1': Input/output error
> rm: cannot remove `p2/dc/d79/c2ee': Input/output error
> rm: cannot remove `p2/dc/d79/c35a': Input/output error
> .....
>
> 14. run "dmesg"
>
> [  469.002632] EXT4-fs warning (device loop0): dx_probe:364: 
> Unrecognised inode hash code 12
> [  469.002647] EXT4-fs warning (device loop0): dx_probe:472: Corrupt 
> dir inode 8423, running e2fsck is recommended.
> [  469.018830] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8926
> [  469.019792] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8816
> [  469.019950] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8570
> [  469.020729] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 7814
> [  469.021227] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 9259
> [  469.048610] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #10178: comm rm: deleted inode referenced: 7791
> [  469.049919] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #10177: comm rm: deleted inode referenced: 8100
> [  469.053826] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #10177: comm rm: deleted inode referenced: 8822
> [  469.060481] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8417: comm rm: deleted inode referenced: 8113
> [  469.062336] EXT4-fs warning (device loop0): dx_probe:364: 
> Unrecognised inode hash code 12
> [  469.062349] EXT4-fs warning (device loop0): dx_probe:472: Corrupt 
> dir inode 10163, running e2fsck is recommended.
> [  469.065721] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #10163: comm rm: deleted inode referenced: 9027
> [  469.066138] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #10163: comm rm: deleted inode referenced: 9090
> [  469.067526] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8417: comm rm: deleted inode referenced: 8443
> [  469.068584] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8067
> [  469.069280] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 9694
> [  469.069893] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8100
> [  469.070123] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8635
> [  469.070509] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
> #8423: comm rm: deleted inode referenced: 8929
> ......
>
> ***************************End********************************************** 
>
>
> But if I use ext3, this will not occur.
> If I do not run "/bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches", 
> this will not occur.
> If I use kernel>=3.0, this will not occur.
>
> But I use ext4 and run "/bin/sync; /bin/echo 1 > 
> /proc/sys/vm/drop_caches", these error messages.
>
> So is this is a bug?
>

Re: a problem about ext4.

[zhuyj]

But if I use ext3, this will not occur.
If I do not run "/bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches", this 
will not occur.
If I use kernel>=3.0, this will not occur.

If I use ext4 and run "/bin/sync; /bin/echo 1 > 
/proc/sys/vm/drop_caches", these error messages will appear.

So is this is a bug?


On 03/12/2013 03:49 PM, zhuyj wrote:
> Hi, all
>
> Almost 2.6.3x has this problem. Is it a problem?
>
> Any reply is appreciated.
> Zhuyj
>
> On 03/12/2013 03:41 PM, zhuyj wrote:
>> Hi, all
>>
>> I install Ubuntu 11.04 on Dell-Latitude-D630.
>> Linux kernel is 2.6.39.4(I compiled)
>>
>> uname -a
>> Linux Dell-Latitude-D630 2.6.39.4-zhuyj26394 #2 SMP Tue Mar 12 
>> 13:23:06 CST 2013 i686 i686 i386 GNU/Linux
>>
>> I followed these steps:
>>
>> ****************************Begin***************************************
>> 1. get ltp-full-20130109.bz2 and run "./configure;make", then we will 
>> get fsstress;
>>
>> 2. mount -t tmpfs -o size=512m tmpfs /tmp
>>
>> 3. dd if=/dev/zero of=/tmp/img bs=1024 count=$((500*1024))
>>
>> 4. ls -lah /tmp/img
>>
>> total 501M
>> drwxrwxrwt  2 root root   60 2013-03-12 15:18 .
>> drwxr-xr-x 23 root root 4.0K 2013-03-12 13:09 ..
>> -rw-r--r--  1 root root 500M 2013-03-12 15:18 img
>>
>> 5. df
>>
>> Filesystem           1K-blocks      Used Available Use% Mounted on
>> /dev/sda1            113317544   6994416 100566868   7% /
>> none                   1023772       696   1023076   1% /dev
>> none                   1030440        80   1030360   1% /dev/shm
>> none                   1030440        92   1030348   1% /var/run
>> none                   1030440         0   1030440   0% /var/lock
>> tmpfs                   524288    512504     11784  98% /tmp
>>
>> 6. mkfs.ext4 /tmp/img
>>
>> mke2fs 1.41.14 (22-Dec-2010)
>> /tmp/img is not a block special device.
>> Proceed anyway? (y,n) y
>> Filesystem label=
>> OS type: Linux
>> Block size=1024 (log=0)
>> Fragment size=1024 (log=0)
>> Stride=0 blocks, Stripe width=0 blocks
>> 128016 inodes, 512000 blocks
>> 25600 blocks (5.00%) reserved for the super user
>> First data block=1
>> Maximum filesystem blocks=67633152
>> 63 block groups
>> 8192 blocks per group, 8192 fragments per group
>> 2032 inodes per group
>> Superblock backups stored on blocks:
>>     8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
>>
>> Writing inode tables: done
>> Creating journal (8192 blocks): done
>> Writing superblocks and filesystem accounting information: done
>>
>> This filesystem will be automatically checked every 30 mounts or
>> 180 days, whichever comes first.  Use tune2fs -c or -i to override.
>>
>> 7. mkdir /tmp/mnt
>>
>> 8. mount -t ext4 -o loop /tmp/img /tmp/mnt
>>
>> 9. df
>> Filesystem           1K-blocks      Used Available Use% Mounted on
>> /dev/sda1            113317544   6994424 100566860   7% /
>> none                   1023772       696   1023076   1% /dev
>> none                   1030440        80   1030360   1% /dev/shm
>> none                   1030440        92   1030348   1% /var/run
>> none                   1030440         0   1030440   0% /var/lock
>> tmpfs                   524288    512504     11784  98% /tmp
>> /dev/loop0              495844     10510    459734   3% /tmp/mnt
>>
>> 10. cd /tmp/mnt; mkdir /tmp/mnt/tmp1 /tmp/mnt/tmp2; cp ~/fsstress 
>> /tmp/mnt
>> ./fsstress -c -p 5 -n 10000 -d . -l 0 -s 10M &
>> ./fsstress -c -p 5 -n 10000 -d ./tmp1 -l 0 -s 10M &
>> ./fsstress -c -p 5 -n 10000 -d ./tmp2 -l 0 -s 10M &
>>
>> 11.when the following appears, run "ps;killall fsstress;ps"
>> ./tmp1: No such file or directory
>> ./tmp2: No such file or directory
>>
>> [2]-  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp1 
>> -l 0 -s 10M
>> [3]+  Exit 1                  ./fsstress -c -p 5 -n 10000 -d ./tmp2 
>> -l 0 -s 10M
>>
>> These messages will appear:
>>
>> ps;killall fsstress;ps
>>   PID TTY          TIME CMD
>>  1366 pts/0    00:00:01 bash
>>  1600 pts/0    00:00:00 fsstress
>>  1633 pts/0    00:00:07 fsstress
>>  1634 pts/0    00:00:06 fsstress
>>  1635 pts/0    00:00:06 fsstress
>>  1636 pts/0    00:00:07 fsstress
>>  1637 pts/0    00:00:07 fsstress
>>  1640 pts/0    00:00:00 ps
>> [1]+  Terminated              ./fsstress -c -p 5 -n 10000 -d . -l 0 
>> -s 10M
>>   PID TTY          TIME CMD
>>  1366 pts/0    00:00:01 bash
>>  1642 pts/0    00:00:00 ps
>>
>> 12. /bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches
>>
>> 13. rm -Rf p*;rm -Rf tmp*
>>
>> rm: cannot remove `p0': Directory not empty
>> rm: cannot remove `p1/d99d/d9b4/da40/da71/da74': Directory not empty
>> rm: cannot remove `p2/dc/d79/f56f': Input/output error
>> rm: cannot remove `p2/dc/d79/f362': Input/output error
>> rm: cannot remove `p2/dc/d79/c460': Input/output error
>> rm: cannot remove `p2/dc/d79/cf4': Input/output error
>> rm: cannot remove `p2/dc/d79/f3b0': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/d95e/d7f7/fb50': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/d95e/f99e': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/d95e/fc40': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/f7ab': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/d547/c843': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/d547/c871': Input/output error
>> rm: cannot remove `p2/dc/d79/d107/fa77': Input/output error
>> rm: cannot remove `p2/dc/d79/f12e': Input/output error
>> rm: cannot remove `p2/dc/d79/c623': Input/output error
>> rm: cannot remove `p2/dc/d79/f207': Input/output error
>> rm: cannot remove `p2/dc/d79/c216': Input/output error
>> rm: cannot remove `p2/dc/d79/f23d': Input/output error
>> rm: cannot remove `p2/dc/d79/c250': Input/output error
>> rm: cannot remove `p2/dc/d79/f274': Input/output error
>> rm: cannot remove `p2/dc/d79/f285': Input/output error
>> rm: cannot remove `p2/dc/d79/f2af': Input/output error
>> rm: cannot remove `p2/dc/d79/c2d1': Input/output error
>> rm: cannot remove `p2/dc/d79/c2ee': Input/output error
>> rm: cannot remove `p2/dc/d79/c35a': Input/output error
>> .....
>>
>> 14. run "dmesg"
>>
>> [  469.002632] EXT4-fs warning (device loop0): dx_probe:364: 
>> Unrecognised inode hash code 12
>> [  469.002647] EXT4-fs warning (device loop0): dx_probe:472: Corrupt 
>> dir inode 8423, running e2fsck is recommended.
>> [  469.018830] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8926
>> [  469.019792] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8816
>> [  469.019950] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8570
>> [  469.020729] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 7814
>> [  469.021227] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 9259
>> [  469.048610] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #10178: comm rm: deleted inode referenced: 7791
>> [  469.049919] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #10177: comm rm: deleted inode referenced: 8100
>> [  469.053826] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #10177: comm rm: deleted inode referenced: 8822
>> [  469.060481] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8417: comm rm: deleted inode referenced: 8113
>> [  469.062336] EXT4-fs warning (device loop0): dx_probe:364: 
>> Unrecognised inode hash code 12
>> [  469.062349] EXT4-fs warning (device loop0): dx_probe:472: Corrupt 
>> dir inode 10163, running e2fsck is recommended.
>> [  469.065721] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #10163: comm rm: deleted inode referenced: 9027
>> [  469.066138] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #10163: comm rm: deleted inode referenced: 9090
>> [  469.067526] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8417: comm rm: deleted inode referenced: 8443
>> [  469.068584] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8067
>> [  469.069280] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 9694
>> [  469.069893] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8100
>> [  469.070123] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8635
>> [  469.070509] EXT4-fs error (device loop0): ext4_lookup:1044: inode 
>> #8423: comm rm: deleted inode referenced: 8929
>> ......
>>
>> ***************************End********************************************** 
>>
>>
>> But if I use ext3, this will not occur.
>> If I do not run "/bin/sync;/bin/echo 1 > /proc/sys/vm/drop_caches", 
>> this will not occur.
>> If I use kernel>=3.0, this will not occur.
>>
>> But I use ext4 and run "/bin/sync; /bin/echo 1 > 
>> /proc/sys/vm/drop_caches", these error messages.
>>
>> So is this is a bug?
>>
>

Re: a problem about ext4.

[Theodore Ts'o]

On Tue, Mar 12, 2013 at 03:41:31PM +0800, zhuyj wrote:
> If I use kernel>=3.0, this will not occur.

Sounds like this is a problem in 2.6.39 that has since been fixed in
newer kernels.

> So is this is a bug?

Sure looks like it's a bug.  :-)

There are so many distributions and other old embedded systems, etc.,
using older kernels that there's just no way that upstream developers
can try to debug every single older kernel --- and 2.6.39 isn't even a
kernel that is being supported by a volunteer as a long-term supported
kernel.

See http://www.kernel.org for a list of kernels which are supported as
long-term kernels, and even then, please remember that unless fixes
are automatically backported, or someone manually backports a fix that
doesn't automatically apply to an older kernel, it's not going to
happen.....

Regards,

					- Ted

P.S.  Your problem appears to be completely unrelated to the thread
which you replied to.  This makes it hard for us to keep track of
questions/bug reports which users submit.

Ext4:can not rm directories on 2.6.3x

[zhuyj]

OK. Thanks a lot.
This bug can happen on Ubuntu 11.04,10.10 with kernel 2.6.3x.
Follow the above steps, this bug can be reproduced.

Anyone has the same experience?

On 03/12/2013 09:42 PM, Theodore Ts'o wrote:
> On Tue, Mar 12, 2013 at 03:41:31PM +0800, zhuyj wrote:
>> If I use kernel>=3.0, this will not occur.
> Sounds like this is a problem in 2.6.39 that has since been fixed in
> newer kernels.
>
>> So is this is a bug?
> Sure looks like it's a bug.  :-)
>
> There are so many distributions and other old embedded systems, etc.,
> using older kernels that there's just no way that upstream developers
> can try to debug every single older kernel --- and 2.6.39 isn't even a
> kernel that is being supported by a volunteer as a long-term supported
> kernel.
>
> See http://www.kernel.org for a list of kernels which are supported as
> long-term kernels, and even then, please remember that unless fixes
> are automatically backported, or someone manually backports a fix that
> doesn't automatically apply to an older kernel, it's not going to
> happen.....
>
> Regards,
>
> 					- Ted
>
> P.S.  Your problem appears to be completely unrelated to the thread
> which you replied to.  This makes it hard for us to keep track of
> questions/bug reports which users submit.
>

Re: Ext4:can not rm directories on 2.6.3x

[zhuyj]

Hi, all

The two patches can fix this problem.
ext4: Fix fs corruption when make_indexed_dir() fails
ext4: don't dereference null pointer when make_indexed_dir() fails

If any one comes across this problem, he can apply the above two patches.

Zhuyj

On 03/13/2013 11:24 AM, zhuyj wrote:
> OK. Thanks a lot.
> This bug can happen on Ubuntu 11.04,10.10 with kernel 2.6.3x.
> Follow the above steps, this bug can be reproduced.
>
> Anyone has the same experience?
>
> On 03/12/2013 09:42 PM, Theodore Ts'o wrote:
>> On Tue, Mar 12, 2013 at 03:41:31PM +0800, zhuyj wrote:
>>> If I use kernel>=3.0, this will not occur.
>> Sounds like this is a problem in 2.6.39 that has since been fixed in
>> newer kernels.
>>
>>> So is this is a bug?
>> Sure looks like it's a bug.  :-)
>>
>> There are so many distributions and other old embedded systems, etc.,
>> using older kernels that there's just no way that upstream developers
>> can try to debug every single older kernel --- and 2.6.39 isn't even a
>> kernel that is being supported by a volunteer as a long-term supported
>> kernel.
>>
>> See http://www.kernel.org for a list of kernels which are supported as
>> long-term kernels, and even then, please remember that unless fixes
>> are automatically backported, or someone manually backports a fix that
>> doesn't automatically apply to an older kernel, it's not going to
>> happen.....
>>
>> Regards,
>>
>>                     - Ted
>>
>> P.S.  Your problem appears to be completely unrelated to the thread
>> which you replied to.  This makes it hard for us to keep track of
>> questions/bug reports which users submit.
>>
>
>

Re: Ext4:can not rm directories on 2.6.3x

[Theodore Ts'o]

[-- Attachment #1: Type: text/plain, Size: 748 bytes --]

On Fri, Mar 15, 2013 at 03:16:52PM +0800, zhuyj wrote:
> Hi, all
> 
> The two patches can fix this problem.
> ext4: Fix fs corruption when make_indexed_dir() fails
> ext4: don't dereference null pointer when make_indexed_dir() fails

Thanks, Zhujh for figuring this out.  Those two patches weren't
taggled with cc: stable@vger.kernel.org, but they probably should have
been.  Since they first appeared in v3.0, the two long-term stable
kernels that would benefit with these patches are v3.0.x and v3.2.x.

Willy, Paul, would you consider cherry-picking the following commits
for your respective stable kernels?

git cherry-pick 7ad8e4e6ae2a7c95445ee1715b1714106fb95037
git cherry-pick 6976a6f2acde2b0443cd64f1d08af90630e4ce81

Thanks!!

				- Ted


[-- Attachment #2: 0001-ext4-Fix-fs-corruption-when-make_indexed_dir-fails.patch --]
[-- Type: text/x-diff, Size: 1573 bytes --]

>From 7ad8e4e6ae2a7c95445ee1715b1714106fb95037 Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Tue, 3 May 2011 11:05:55 -0400
Subject: [PATCH] ext4: Fix fs corruption when make_indexed_dir() fails

When make_indexed_dir() fails (e.g. because of ENOSPC) after it has
allocated block for index tree root, we did not properly mark all
changed buffers dirty.  This lead to only some of these buffers being
written out and thus effectively corrupting the directory.

Fix the issue by marking all changed data dirty even in the error
failure case.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
---
 fs/ext4/namei.c | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index 67fd0b0..cadf04b 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -1414,9 +1414,19 @@ static int make_indexed_dir(handle_t *handle, struct dentry *dentry,
 	frame->bh = bh;
 	bh = bh2;
 	de = do_split(handle,dir, &bh, frame, &hinfo, &retval);
-	dx_release (frames);
-	if (!(de))
+	if (!de) {
+		/*
+		 * Even if the block split failed, we have to properly write
+		 * out all the changes we did so far. Otherwise we can end up
+		 * with corrupted filesystem.
+		 */
+		ext4_mark_inode_dirty(handle, dir);
+		ext4_handle_dirty_metadata(handle, dir, frame->bh);
+		ext4_handle_dirty_metadata(handle, dir, bh);
+		dx_release(frames);
 		return retval;
+	}
+	dx_release(frames);
 
 	retval = add_dirent_to_buf(handle, dentry, inode, de, bh);
 	brelse(bh);
-- 
1.7.12.rc0.22.gcdd159b


[-- Attachment #3: 0001-ext4-don-t-dereference-null-pointer-when-make_indexe.patch --]
[-- Type: text/x-diff, Size: 1293 bytes --]

>From 6976a6f2acde2b0443cd64f1d08af90630e4ce81 Mon Sep 17 00:00:00 2001
From: Allison Henderson <achender@linux.vnet.ibm.com>
Date: Sun, 15 May 2011 00:19:41 -0400
Subject: [PATCH] ext4: don't dereference null pointer when make_indexed_dir()
 fails

Fix for a null pointer bug found while running punch hole tests

Signed-off-by: Allison Henderson <achender@us.ibm.com>
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
---
 fs/ext4/namei.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index 3c7a06e..b754b77 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -1413,6 +1413,10 @@ static int make_indexed_dir(handle_t *handle, struct dentry *dentry,
 	frame->at = entries;
 	frame->bh = bh;
 	bh = bh2;
+
+	ext4_handle_dirty_metadata(handle, dir, frame->bh);
+	ext4_handle_dirty_metadata(handle, dir, bh);
+
 	de = do_split(handle,dir, &bh, frame, &hinfo, &retval);
 	if (!de) {
 		/*
@@ -1421,8 +1425,6 @@ static int make_indexed_dir(handle_t *handle, struct dentry *dentry,
 		 * with corrupted filesystem.
 		 */
 		ext4_mark_inode_dirty(handle, dir);
-		ext4_handle_dirty_metadata(handle, dir, frame->bh);
-		ext4_handle_dirty_metadata(handle, dir, bh);
 		dx_release(frames);
 		return retval;
 	}
-- 
1.7.12.rc0.22.gcdd159b

Re: Ext4:can not rm directories on 2.6.3x

[Willy Tarreau]

On Fri, Mar 15, 2013 at 10:53:14AM -0400, Theodore Ts'o wrote:
> On Fri, Mar 15, 2013 at 03:16:52PM +0800, zhuyj wrote:
> > Hi, all
> > 
> > The two patches can fix this problem.
> > ext4: Fix fs corruption when make_indexed_dir() fails
> > ext4: don't dereference null pointer when make_indexed_dir() fails
> 
> Thanks, Zhujh for figuring this out.  Those two patches weren't
> taggled with cc: stable@vger.kernel.org, but they probably should have
> been.  Since they first appeared in v3.0, the two long-term stable
> kernels that would benefit with these patches are v3.0.x and v3.2.x.
> 
> Willy, Paul, would you consider cherry-picking the following commits
> for your respective stable kernels?
> 
> git cherry-pick 7ad8e4e6ae2a7c95445ee1715b1714106fb95037
> git cherry-pick 6976a6f2acde2b0443cd64f1d08af90630e4ce81

Yes of course !

Thank you Ted!
Willy

Re: Ext4:can not rm directories on 2.6.3x

[Theodore Ts'o]

On Fri, Mar 15, 2013 at 03:56:04PM +0100, Willy Tarreau wrote:
> > Thanks, Zhujh for figuring this out.  Those two patches weren't
> > taggled with cc: stable@vger.kernel.org, but they probably should have
> > been.  Since they first appeared in v3.0, the two long-term stable
> > kernels that would benefit with these patches are v3.0.x and v3.2.x.

Oops, sorry, that should have read 2.6.32 and 2.6.34.  :-)

> > Willy, Paul, would you consider cherry-picking the following commits
> > for your respective stable kernels?
> > 
> > git cherry-pick 7ad8e4e6ae2a7c95445ee1715b1714106fb95037
> > git cherry-pick 6976a6f2acde2b0443cd64f1d08af90630e4ce81
> 
> Yes of course !

Great, thanks!!

						- Ted

Re: Ext4:can not rm directories on 2.6.3x

[Paul Gortmaker]

On 13-03-15 10:53 AM, Theodore Ts'o wrote:
> On Fri, Mar 15, 2013 at 03:16:52PM +0800, zhuyj wrote:
>> Hi, all
>>
>> The two patches can fix this problem.
>> ext4: Fix fs corruption when make_indexed_dir() fails
>> ext4: don't dereference null pointer when make_indexed_dir() fails
> 
> Thanks, Zhujh for figuring this out.  Those two patches weren't
> taggled with cc: stable@vger.kernel.org, but they probably should have
> been.  Since they first appeared in v3.0, the two long-term stable
> kernels that would benefit with these patches are v3.0.x and v3.2.x.
> 
> Willy, Paul, would you consider cherry-picking the following commits
> for your respective stable kernels?

Both queued for 2.6.34.15

Thanks,
Paul.
--

> 
> git cherry-pick 7ad8e4e6ae2a7c95445ee1715b1714106fb95037
> git cherry-pick 6976a6f2acde2b0443cd64f1d08af90630e4ce81
> 
> Thanks!!
> 
> 				- Ted
> 

Re: Inactive memory keep growing and how to release it?

[Zheng Liu]

Hi Gao,

On Tue, Mar 12, 2013 at 02:58:24PM +0800, Lenky Gao wrote:
> Hi,
> 
> When i just run a test on Centos 6.2 as follows:
> ################################
> #!/bin/bash
> 
> while true
> do
> 
>         file="/tmp/filetest"
> 
>         echo $file
> 
>         dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
> 
>         sleep 5
> done
> ################################
> 
> the inactive memory keep growing:
> 
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   554662 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   882824 kB
> 
> and i cannot reclaim it:
> 
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   882824 kB
> # sync;
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   777664 kB
> 
> My file system type is ext3, mounted with the option data=journal and
> it is easy to reproduce. If the mount option is data=ordered,
> reproducing this problom need more time. Because of the problom, when
> in the final situation, i cannot kmalloc a larger contiguous memory,
> especially in interrupt context. I have tested on other version
> kernel, such as 2.6.30 and 3.6.11, the problom also exists.
> 
> Is this a bug?

I don't think it is a bug.  If I understand correctly, when you try to
reclaim memory (sync and drop caches), the test script is still running,
right?  If so, that means that some memory will be re-dirtied again.  So
the number of inactive pages could not reduce as you expected.

Regards,
                                                - Zheng

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

On Tue, Mar 12, 2013 at 4:37 PM, Zheng Liu <gnehzuil.liu@gmail.com> wrote:
> Hi Gao,
> I don't think it is a bug.  If I understand correctly, when you try to
> reclaim memory (sync and drop caches), the test script is still running,
> right?  If so, that means that some memory will be re-dirtied again.  So
> the number of inactive pages could not reduce as you expected.

The test script has stopped.

Some log messages came from my UML Linux as follows:

...
/sdb/file_test
             total       used       free     shared    buffers     cached
Mem:           182        179          2          0         10         13
-/+ buffers/cache:        155         27
Swap:            0          0          0
/sdb/file_test
^C
[root@localhost ~]#
[root@localhost ~]# sync;
[root@localhost ~]# echo 3 > /proc/sys/vm/drop_caches
[root@localhost ~]#
[root@localhost ~]# echo 3 > /proc/sys/vm/drop_caches
[root@localhost ~]# sync;
[root@localhost ~]# echo 3 > /proc/sys/vm/drop_caches
[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:           182        157         25          0          0          3
-/+ buffers/cache:        153         28
Swap:            0          0          0

[root@localhost ~]# cat /proc/meminfo | grep Inac
Inactive:         124756 kB
Inactive(anon):     1408 kB
Inactive(file):   123348 kB
[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.30.8 #20 Mon Mar 11 07:04:06 EDT 2013
x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# mount | grep sdb
/dev/ubdb1 on /sdb type ext3 (rw,data=journal)
[root@localhost ~]# cat ./test_write_disk.sh
#!/bin/bash

while true
do

#       file="/sdb/file_`date +%y%m%d%k%m%s`"
        file="/sdb/file_test"

        echo $file

        dd if=/dev/zero of=${file} bs=512 count=20480 &> /dev/null

#	sync
#	cat /proc/meminfo | grep Inactive\(file\)
	free -m

#        rm -fr ${file}
done

-- 
Regards,

Lenky

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

Hi Everyone,

Maybe i have found the answer for this question. The author of the JBD
have explained in the comments:

/*
 * When an ext3-ordered file is truncated, it is possible that many pages are
 * not successfully freed, because they are attached to a committing
transaction.
 * After the transaction commits, these pages are left on the LRU, with no
 * ->mapping, and with attached buffers.  These pages are trivially reclaimable
 * by the VM, but their apparent absence upsets the VM accounting, and it makes
 * the numbers in /proc/meminfo look odd.
...
 */
static void release_buffer_page(struct buffer_head *bh)
{
	struct page *page;
...

But my new question is why not free those pages after the transaction commits?



-- 
Regards,

Lenky

Inactive memory keep growing and how to release it?

[Lenky Gao]

Hi,

When i just run a test on Centos 6.2 as follows:

#!/bin/bash

while true
do

	file="/tmp/filetest"

	echo $file

	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null

	sleep 5
done

the inactive memory keep growing:

#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   420144 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   911912 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1547484 kB
...

and i cannot reclaim it:

# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1557684 kB
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1520832 kB

I have tested on other version kernel, such as 2.6.30 and .6.11, the
problom also exists.

When in the final situation, i cannot kmalloc a larger contiguous
memory, especially in interrupt context.
Can you give some tips to avoid this?

PS:
# uname -a
Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux



--
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Inactive memory keep growing and how to release it?

[Lenky Gao]

Hi,

When i just run a test on Centos 6.2 as follows:

#!/bin/bash

while true
do

	file="/tmp/filetest"

	echo $file

	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null

	sleep 5
done

the inactive memory keep growing:

#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   420144 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):   911912 kB
...
#cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1547484 kB
...

and i cannot reclaim it:

# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1557684 kB
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(fi
Inactive(file):  1520832 kB

I have tested on other version kernel, such as 2.6.30 and .6.11, the
problom also exists.

When in the final situation, i cannot kmalloc a larger contiguous
memory, especially in interrupt context.
Can you give some tips to avoid this?

PS:
# uname -a
Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux



--
Regards,

Lenky

Re: Inactive memory keep growing and how to release it?

[Zlatko Calusic]

On 04.03.2013 10:52, Lenky Gao wrote:
> Hi,
>
> When i just run a test on Centos 6.2 as follows:
>
> #!/bin/bash
>
> while true
> do
>
> 	file="/tmp/filetest"
>
> 	echo $file
>
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>
> 	sleep 5
> done
>
> the inactive memory keep growing:
>
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
>
> and i cannot reclaim it:
>
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
>
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
>
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.
> Can you give some tips to avoid this?
>

The drop_caches mechanism doesn't free dirty page cache pages. And your 
bash script is creating a lot of dirty pages. Run it like this and see 
if it helps your case:

sync; echo 3 > /proc/sys/vm/drop_caches

Regards,
-- 
Zlatko

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Zlatko Calusic]

On 04.03.2013 10:52, Lenky Gao wrote:
> Hi,
>
> When i just run a test on Centos 6.2 as follows:
>
> #!/bin/bash
>
> while true
> do
>
> 	file="/tmp/filetest"
>
> 	echo $file
>
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>
> 	sleep 5
> done
>
> the inactive memory keep growing:
>
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
>
> and i cannot reclaim it:
>
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
>
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
>
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.
> Can you give some tips to avoid this?
>

The drop_caches mechanism doesn't free dirty page cache pages. And your 
bash script is creating a lot of dirty pages. Run it like this and see 
if it helps your case:

sync; echo 3 > /proc/sys/vm/drop_caches

Regards,
-- 
Zlatko

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>
> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
> script is creating a lot of dirty pages. Run it like this and see if it
> helps your case:
>
> sync; echo 3 > /proc/sys/vm/drop_caches

Thanks for your advice.

The inactive memory still cannot be reclaimed after i execute the sync command:

# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   882824 kB
# sync;
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   777664 kB

I find these page becomes orphaned in this function, but do not understand why:

/*
 * If truncate cannot remove the fs-private metadata from the page, the page
 * becomes orphaned.  It will be left on the LRU and may even be mapped into
 * user pagetables if we're racing with filemap_fault().
 *
 * We need to bale out if page->mapping is no longer equal to the original
 * mapping.  This happens a) when the VM reclaimed the page while we waited on
 * its lock, b) when a concurrent invalidate_mapping_pages got there first and
 * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
 */
static int
truncate_complete_page(struct address_space *mapping, struct page *page)
{
...

My file system type is ext3, mounted with the opteion data=journal and
it is easy to reproduce.


-- 
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>
> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
> script is creating a lot of dirty pages. Run it like this and see if it
> helps your case:
>
> sync; echo 3 > /proc/sys/vm/drop_caches

Thanks for your advice.

The inactive memory still cannot be reclaimed after i execute the sync command:

# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   882824 kB
# sync;
# echo 3 > /proc/sys/vm/drop_caches
# cat /proc/meminfo | grep Inactive\(file\);
Inactive(file):   777664 kB

I find these page becomes orphaned in this function, but do not understand why:

/*
 * If truncate cannot remove the fs-private metadata from the page, the page
 * becomes orphaned.  It will be left on the LRU and may even be mapped into
 * user pagetables if we're racing with filemap_fault().
 *
 * We need to bale out if page->mapping is no longer equal to the original
 * mapping.  This happens a) when the VM reclaimed the page while we waited on
 * its lock, b) when a concurrent invalidate_mapping_pages got there first and
 * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
 */
static int
truncate_complete_page(struct address_space *mapping, struct page *page)
{
...

My file system type is ext3, mounted with the opteion data=journal and
it is easy to reproduce.


-- 
Regards,

Lenky

Re: Inactive memory keep growing and how to release it?

[Will Huck]

Cc experts. Hugh, Johannes,

On 03/04/2013 08:21 PM, Lenky Gao wrote:
> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
>> script is creating a lot of dirty pages. Run it like this and see if it
>> helps your case:
>>
>> sync; echo 3 > /proc/sys/vm/drop_caches
> Thanks for your advice.
>
> The inactive memory still cannot be reclaimed after i execute the sync command:
>
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   882824 kB
> # sync;
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   777664 kB
>
> I find these page becomes orphaned in this function, but do not understand why:
>
> /*
>   * If truncate cannot remove the fs-private metadata from the page, the page
>   * becomes orphaned.  It will be left on the LRU and may even be mapped into
>   * user pagetables if we're racing with filemap_fault().
>   *
>   * We need to bale out if page->mapping is no longer equal to the original
>   * mapping.  This happens a) when the VM reclaimed the page while we waited on
>   * its lock, b) when a concurrent invalidate_mapping_pages got there first and
>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>   */
> static int
> truncate_complete_page(struct address_space *mapping, struct page *page)
> {
> ...
>
> My file system type is ext3, mounted with the opteion data=journal and
> it is easy to reproduce.
>
>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Will Huck]

Cc experts. Hugh, Johannes,

On 03/04/2013 08:21 PM, Lenky Gao wrote:
> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>> The drop_caches mechanism doesn't free dirty page cache pages. And your bash
>> script is creating a lot of dirty pages. Run it like this and see if it
>> helps your case:
>>
>> sync; echo 3 > /proc/sys/vm/drop_caches
> Thanks for your advice.
>
> The inactive memory still cannot be reclaimed after i execute the sync command:
>
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   882824 kB
> # sync;
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(file\);
> Inactive(file):   777664 kB
>
> I find these page becomes orphaned in this function, but do not understand why:
>
> /*
>   * If truncate cannot remove the fs-private metadata from the page, the page
>   * becomes orphaned.  It will be left on the LRU and may even be mapped into
>   * user pagetables if we're racing with filemap_fault().
>   *
>   * We need to bale out if page->mapping is no longer equal to the original
>   * mapping.  This happens a) when the VM reclaimed the page while we waited on
>   * its lock, b) when a concurrent invalidate_mapping_pages got there first and
>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>   */
> static int
> truncate_complete_page(struct address_space *mapping, struct page *page)
> {
> ...
>
> My file system type is ext3, mounted with the opteion data=journal and
> it is easy to reproduce.
>
>

Re: Inactive memory keep growing and how to release it?

[Hillf Danton]

On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
> Cc experts. Hugh, Johannes,
>
> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>
>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>
>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>> bash
>>> script is creating a lot of dirty pages. Run it like this and see if it
>>> helps your case:
>>>
>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>
>> Thanks for your advice.
>>
>> The inactive memory still cannot be reclaimed after i execute the sync
>> command:
>>
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   882824 kB
>> # sync;
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   777664 kB
>>
>> I find these page becomes orphaned in this function, but do not understand
>> why:
>>
>> /*
>>   * If truncate cannot remove the fs-private metadata from the page, the
>> page
>>   * becomes orphaned.  It will be left on the LRU and may even be mapped
>> into
>>   * user pagetables if we're racing with filemap_fault().
>>   *
>>   * We need to bale out if page->mapping is no longer equal to the
>> original
>>   * mapping.  This happens a) when the VM reclaimed the page while we
>> waited on
>>   * its lock, b) when a concurrent invalidate_mapping_pages got there
>> first and
>>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>   */
>> static int
>> truncate_complete_page(struct address_space *mapping, struct page *page)
>> {
>> ...
>>
>> My file system type is ext3, mounted with the opteion data=journal and
>> it is easy to reproduce.
>>

Perhaps we have to consider page count for orphan page if it
could be reproduced with mainline.

Hillf
---
--- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
+++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
@@ -315,14 +315,14 @@ out:
 	return ret;
 }

-static inline int is_page_cache_freeable(struct page *page)
+static inline int is_page_cache_freeable(struct page *page, int has_mapping)
 {
 	/*
 	 * A freeable page cache page is referenced only by the caller
 	 * that isolated the page, the page cache radix tree and
 	 * optional buffer heads at page->private.
 	 */
-	return page_count(page) - page_has_private(page) == 2;
+	return page_count(page) - page_has_private(page) == has_mapping + 1;
 }

 static int may_write_to_queue(struct backing_dev_info *bdi,
@@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
 	 * swap_backing_dev_info is bust: it doesn't reflect the
 	 * congestion state of the swapdevs.  Easy to fix, if needed.
 	 */
-	if (!is_page_cache_freeable(page))
+	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
 		return PAGE_KEEP;
 	if (!mapping) {
 		/*
--

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Hillf Danton]

On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
> Cc experts. Hugh, Johannes,
>
> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>
>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>
>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>> bash
>>> script is creating a lot of dirty pages. Run it like this and see if it
>>> helps your case:
>>>
>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>
>> Thanks for your advice.
>>
>> The inactive memory still cannot be reclaimed after i execute the sync
>> command:
>>
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   882824 kB
>> # sync;
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(file\);
>> Inactive(file):   777664 kB
>>
>> I find these page becomes orphaned in this function, but do not understand
>> why:
>>
>> /*
>>   * If truncate cannot remove the fs-private metadata from the page, the
>> page
>>   * becomes orphaned.  It will be left on the LRU and may even be mapped
>> into
>>   * user pagetables if we're racing with filemap_fault().
>>   *
>>   * We need to bale out if page->mapping is no longer equal to the
>> original
>>   * mapping.  This happens a) when the VM reclaimed the page while we
>> waited on
>>   * its lock, b) when a concurrent invalidate_mapping_pages got there
>> first and
>>   * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>   */
>> static int
>> truncate_complete_page(struct address_space *mapping, struct page *page)
>> {
>> ...
>>
>> My file system type is ext3, mounted with the opteion data=journal and
>> it is easy to reproduce.
>>

Perhaps we have to consider page count for orphan page if it
could be reproduced with mainline.

Hillf
---
--- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
+++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
@@ -315,14 +315,14 @@ out:
 	return ret;
 }

-static inline int is_page_cache_freeable(struct page *page)
+static inline int is_page_cache_freeable(struct page *page, int has_mapping)
 {
 	/*
 	 * A freeable page cache page is referenced only by the caller
 	 * that isolated the page, the page cache radix tree and
 	 * optional buffer heads at page->private.
 	 */
-	return page_count(page) - page_has_private(page) == 2;
+	return page_count(page) - page_has_private(page) == has_mapping + 1;
 }

 static int may_write_to_queue(struct backing_dev_info *bdi,
@@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
 	 * swap_backing_dev_info is bust: it doesn't reflect the
 	 * congestion state of the swapdevs.  Easy to fix, if needed.
 	 */
-	if (!is_page_cache_freeable(page))
+	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
 		return PAGE_KEEP;
 	if (!mapping) {
 		/*
--

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

On Thu, Mar 14, 2013 at 6:14 PM, Michal Hocko <mhocko@suse.cz> wrote:
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>

Good tip thanks. :)

> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and
> up-to-date kernel and report here.

I have tested on other version vanilla kernel, such as 2.6.30 and 3.6.11, the
issue also exist and it is easy to reproduce.

Maybe i have found the answer for this question:

On Thu, Mar 14, 2013 at 4:00 PM, Lenky Gao <lenky.gao@gmail.com> wrote:
> Hi Everyone,
>
> Maybe i have found the answer for this question. The author of the JBD
> have explained in the comments:
>
> /*
>  * When an ext3-ordered file is truncated, it is possible that many pages are
>  * not successfully freed, because they are attached to a committing
> transaction.
>  * After the transaction commits, these pages are left on the LRU, with no
>  * ->mapping, and with attached buffers.  These pages are trivially reclaimable
>  * by the VM, but their apparent absence upsets the VM accounting, and it makes
>  * the numbers in /proc/meminfo look odd.
> ...
>  */
> static void release_buffer_page(struct buffer_head *bh)
> {
>         struct page *page;
> ...

But my new question is why not free those pages directly after the
transaction commits?

On Thu, Mar 14, 2013 at 8:39 PM, Hillf Danton <dhillf@gmail.com> wrote:
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.
>
> Hillf
> ---
> --- a/mm/vmscan.c       Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c       Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>         return ret;
>  }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>  {
>         /*
>          * A freeable page cache page is referenced only by the caller
>          * that isolated the page, the page cache radix tree and
>          * optional buffer heads at page->private.
>          */
> -       return page_count(page) - page_has_private(page) == 2;
> +       return page_count(page) - page_has_private(page) == has_mapping + 1;
>  }
>
>  static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>          * swap_backing_dev_info is bust: it doesn't reflect the
>          * congestion state of the swapdevs.  Easy to fix, if needed.
>          */
> -       if (!is_page_cache_freeable(page))
> +       if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>                 return PAGE_KEEP;
>         if (!mapping) {
>                 /*

Thanks, i'll test it.

I am totally a newbie regarding VMM and EXT/JBD, thanks to everyone
for your kind attention and help.

-- 
Regards,

Lenky

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Lenky Gao]

On Thu, Mar 14, 2013 at 6:14 PM, Michal Hocko <mhocko@suse.cz> wrote:
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>

Good tip thanks. :)

> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and
> up-to-date kernel and report here.

I have tested on other version vanilla kernel, such as 2.6.30 and 3.6.11, the
issue also exist and it is easy to reproduce.

Maybe i have found the answer for this question:

On Thu, Mar 14, 2013 at 4:00 PM, Lenky Gao <lenky.gao@gmail.com> wrote:
> Hi Everyone,
>
> Maybe i have found the answer for this question. The author of the JBD
> have explained in the comments:
>
> /*
>  * When an ext3-ordered file is truncated, it is possible that many pages are
>  * not successfully freed, because they are attached to a committing
> transaction.
>  * After the transaction commits, these pages are left on the LRU, with no
>  * ->mapping, and with attached buffers.  These pages are trivially reclaimable
>  * by the VM, but their apparent absence upsets the VM accounting, and it makes
>  * the numbers in /proc/meminfo look odd.
> ...
>  */
> static void release_buffer_page(struct buffer_head *bh)
> {
>         struct page *page;
> ...

But my new question is why not free those pages directly after the
transaction commits?

On Thu, Mar 14, 2013 at 8:39 PM, Hillf Danton <dhillf@gmail.com> wrote:
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.
>
> Hillf
> ---
> --- a/mm/vmscan.c       Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c       Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>         return ret;
>  }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>  {
>         /*
>          * A freeable page cache page is referenced only by the caller
>          * that isolated the page, the page cache radix tree and
>          * optional buffer heads at page->private.
>          */
> -       return page_count(page) - page_has_private(page) == 2;
> +       return page_count(page) - page_has_private(page) == has_mapping + 1;
>  }
>
>  static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>          * swap_backing_dev_info is bust: it doesn't reflect the
>          * congestion state of the swapdevs.  Easy to fix, if needed.
>          */
> -       if (!is_page_cache_freeable(page))
> +       if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>                 return PAGE_KEEP;
>         if (!mapping) {
>                 /*

Thanks, i'll test it.

I am totally a newbie regarding VMM and EXT/JBD, thanks to everyone
for your kind attention and help.

-- 
Regards,

Lenky

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 08:39 PM, Hillf Danton wrote:
> On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
>> Cc experts. Hugh, Johannes,
>>
>> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>>> bash
>>>> script is creating a lot of dirty pages. Run it like this and see if it
>>>> helps your case:
>>>>
>>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>> Thanks for your advice.
>>>
>>> The inactive memory still cannot be reclaimed after i execute the sync
>>> command:
>>>
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   882824 kB
>>> # sync;
>>> # echo 3 > /proc/sys/vm/drop_caches
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   777664 kB
>>>
>>> I find these page becomes orphaned in this function, but do not understand
>>> why:
>>>
>>> /*
>>>    * If truncate cannot remove the fs-private metadata from the page, the
>>> page
>>>    * becomes orphaned.  It will be left on the LRU and may even be mapped
>>> into
>>>    * user pagetables if we're racing with filemap_fault().
>>>    *
>>>    * We need to bale out if page->mapping is no longer equal to the
>>> original
>>>    * mapping.  This happens a) when the VM reclaimed the page while we
>>> waited on
>>>    * its lock, b) when a concurrent invalidate_mapping_pages got there
>>> first and
>>>    * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>>    */
>>> static int
>>> truncate_complete_page(struct address_space *mapping, struct page *page)
>>> {
>>> ...
>>>
>>> My file system type is ext3, mounted with the opteion data=journal and
>>> it is easy to reproduce.
>>>
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.

Why? /proc/sys/vm/drop_caches will call invalidate_mapping_pages() 
instead of truncate_complete_page().

>
> Hillf
> ---
> --- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>   	return ret;
>   }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>   {
>   	/*
>   	 * A freeable page cache page is referenced only by the caller
>   	 * that isolated the page, the page cache radix tree and
>   	 * optional buffer heads at page->private.
>   	 */
> -	return page_count(page) - page_has_private(page) == 2;
> +	return page_count(page) - page_has_private(page) == has_mapping + 1;
>   }

page count 2 is for page cache and isolator, why you check mapping 
separately?

>   static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>   	 * swap_backing_dev_info is bust: it doesn't reflect the
>   	 * congestion state of the swapdevs.  Easy to fix, if needed.
>   	 */
> -	if (!is_page_cache_freeable(page))
> +	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>   		return PAGE_KEEP;
>   	if (!mapping) {
>   		/*
> --
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 08:39 PM, Hillf Danton wrote:
> On Sat, Mar 9, 2013 at 10:14 AM, Will Huck <will.huckk@gmail.com> wrote:
>> Cc experts. Hugh, Johannes,
>>
>> On 03/04/2013 08:21 PM, Lenky Gao wrote:
>>> 2013/3/4 Zlatko Calusic <zlatko.calusic@iskon.hr>:
>>>> The drop_caches mechanism doesn't free dirty page cache pages. And your
>>>> bash
>>>> script is creating a lot of dirty pages. Run it like this and see if it
>>>> helps your case:
>>>>
>>>> sync; echo 3 > /proc/sys/vm/drop_caches
>>> Thanks for your advice.
>>>
>>> The inactive memory still cannot be reclaimed after i execute the sync
>>> command:
>>>
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   882824 kB
>>> # sync;
>>> # echo 3 > /proc/sys/vm/drop_caches
>>> # cat /proc/meminfo | grep Inactive\(file\);
>>> Inactive(file):   777664 kB
>>>
>>> I find these page becomes orphaned in this function, but do not understand
>>> why:
>>>
>>> /*
>>>    * If truncate cannot remove the fs-private metadata from the page, the
>>> page
>>>    * becomes orphaned.  It will be left on the LRU and may even be mapped
>>> into
>>>    * user pagetables if we're racing with filemap_fault().
>>>    *
>>>    * We need to bale out if page->mapping is no longer equal to the
>>> original
>>>    * mapping.  This happens a) when the VM reclaimed the page while we
>>> waited on
>>>    * its lock, b) when a concurrent invalidate_mapping_pages got there
>>> first and
>>>    * c) when tmpfs swizzles a page between a tmpfs inode and swapper_space.
>>>    */
>>> static int
>>> truncate_complete_page(struct address_space *mapping, struct page *page)
>>> {
>>> ...
>>>
>>> My file system type is ext3, mounted with the opteion data=journal and
>>> it is easy to reproduce.
>>>
> Perhaps we have to consider page count for orphan page if it
> could be reproduced with mainline.

Why? /proc/sys/vm/drop_caches will call invalidate_mapping_pages() 
instead of truncate_complete_page().

>
> Hillf
> ---
> --- a/mm/vmscan.c	Sun Mar 10 13:36:26 2013
> +++ b/mm/vmscan.c	Thu Mar 14 20:29:40 2013
> @@ -315,14 +315,14 @@ out:
>   	return ret;
>   }
>
> -static inline int is_page_cache_freeable(struct page *page)
> +static inline int is_page_cache_freeable(struct page *page, int has_mapping)
>   {
>   	/*
>   	 * A freeable page cache page is referenced only by the caller
>   	 * that isolated the page, the page cache radix tree and
>   	 * optional buffer heads at page->private.
>   	 */
> -	return page_count(page) - page_has_private(page) == 2;
> +	return page_count(page) - page_has_private(page) == has_mapping + 1;
>   }

page count 2 is for page cache and isolator, why you check mapping 
separately?

>   static int may_write_to_queue(struct backing_dev_info *bdi,
> @@ -393,7 +393,7 @@ static pageout_t pageout(struct page *pa
>   	 * swap_backing_dev_info is bust: it doesn't reflect the
>   	 * congestion state of the swapdevs.  Easy to fix, if needed.
>   	 */
> -	if (!is_page_cache_freeable(page))
> +	if (!is_page_cache_freeable(page, mapping ? 1 : 0))
>   		return PAGE_KEEP;
>   	if (!mapping) {
>   		/*
> --
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Michal Hocko]

On Mon 04-03-13 17:52:22, Lenky Gao wrote:
> Hi,
> 
> When i just run a test on Centos 6.2 as follows:
> 
> #!/bin/bash
> 
> while true
> do
> 
> 	file="/tmp/filetest"
> 
> 	echo $file
> 
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
> 
> 	sleep 5
> done
> 
> the inactive memory keep growing:
> 
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
> 
> and i cannot reclaim it:

How did you try to reclaim the memory? How much memory is still free?
Are you above watermaks (/proc/zoneinfo will tell you more)

> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
> 
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
> 
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.

This could be related to the memory fragmentation and your kernel seem
to be too large to have memory compaction which helps a lot in that
area.

> Can you give some tips to avoid this?

One way would be to increase /proc/sys/vm/min_free_kbytes which will
enlarge watermaks so the reclaim starts sooner.
 
> PS:
> # uname -a
> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

This is really an old kernel and also a distribution one which might
contain a lot of patches on top of the core kernel. I would suggest to
contact Redhat or try to reproduce the issue with the vanilla and
up-to-date kernel and report here.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Michal Hocko]

On Mon 04-03-13 17:52:22, Lenky Gao wrote:
> Hi,
> 
> When i just run a test on Centos 6.2 as follows:
> 
> #!/bin/bash
> 
> while true
> do
> 
> 	file="/tmp/filetest"
> 
> 	echo $file
> 
> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
> 
> 	sleep 5
> done
> 
> the inactive memory keep growing:
> 
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   420144 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):   911912 kB
> ...
> #cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1547484 kB
> ...
> 
> and i cannot reclaim it:

How did you try to reclaim the memory? How much memory is still free?
Are you above watermaks (/proc/zoneinfo will tell you more)

> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1557684 kB
> # echo 3 > /proc/sys/vm/drop_caches
> # cat /proc/meminfo | grep Inactive\(fi
> Inactive(file):  1520832 kB
> 
> I have tested on other version kernel, such as 2.6.30 and .6.11, the
> problom also exists.
> 
> When in the final situation, i cannot kmalloc a larger contiguous
> memory, especially in interrupt context.

This could be related to the memory fragmentation and your kernel seem
to be too large to have memory compaction which helps a lot in that
area.

> Can you give some tips to avoid this?

One way would be to increase /proc/sys/vm/min_free_kbytes which will
enlarge watermaks so the reclaim starts sooner.
 
> PS:
> # uname -a
> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux

This is really an old kernel and also a distribution one which might
contain a lot of patches on top of the core kernel. I would suggest to
contact Redhat or try to reproduce the issue with the vanilla and
up-to-date kernel and report here.
-- 
Michal Hocko
SUSE Labs

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 06:14 PM, Michal Hocko wrote:
> On Mon 04-03-13 17:52:22, Lenky Gao wrote:
>> Hi,
>>
>> When i just run a test on Centos 6.2 as follows:
>>
>> #!/bin/bash
>>
>> while true
>> do
>>
>> 	file="/tmp/filetest"
>>
>> 	echo $file
>>
>> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>>
>> 	sleep 5
>> done
>>
>> the inactive memory keep growing:
>>
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   420144 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   911912 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1547484 kB
>> ...
>>
>> and i cannot reclaim it:
> How did you try to reclaim the memory? How much memory is still free?
> Are you above watermaks (/proc/zoneinfo will tell you more)
>
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1557684 kB
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1520832 kB
>>
>> I have tested on other version kernel, such as 2.6.30 and .6.11, the
>> problom also exists.
>>
>> When in the final situation, i cannot kmalloc a larger contiguous
>> memory, especially in interrupt context.
> This could be related to the memory fragmentation and your kernel seem
> to be too large to have memory compaction which helps a lot in that
> area.
>
>> Can you give some tips to avoid this?
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>   
>> PS:
>> # uname -a
>> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
>> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and

What's the meaning of vanilla?

> up-to-date kernel and report here.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Simon Jeons]

On 03/14/2013 06:14 PM, Michal Hocko wrote:
> On Mon 04-03-13 17:52:22, Lenky Gao wrote:
>> Hi,
>>
>> When i just run a test on Centos 6.2 as follows:
>>
>> #!/bin/bash
>>
>> while true
>> do
>>
>> 	file="/tmp/filetest"
>>
>> 	echo $file
>>
>> 	dd if=/dev/zero of=${file} bs=512 count=204800 &> /dev/null
>>
>> 	sleep 5
>> done
>>
>> the inactive memory keep growing:
>>
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   420144 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):   911912 kB
>> ...
>> #cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1547484 kB
>> ...
>>
>> and i cannot reclaim it:
> How did you try to reclaim the memory? How much memory is still free?
> Are you above watermaks (/proc/zoneinfo will tell you more)
>
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1557684 kB
>> # echo 3 > /proc/sys/vm/drop_caches
>> # cat /proc/meminfo | grep Inactive\(fi
>> Inactive(file):  1520832 kB
>>
>> I have tested on other version kernel, such as 2.6.30 and .6.11, the
>> problom also exists.
>>
>> When in the final situation, i cannot kmalloc a larger contiguous
>> memory, especially in interrupt context.
> This could be related to the memory fragmentation and your kernel seem
> to be too large to have memory compaction which helps a lot in that
> area.
>
>> Can you give some tips to avoid this?
> One way would be to increase /proc/sys/vm/min_free_kbytes which will
> enlarge watermaks so the reclaim starts sooner.
>   
>> PS:
>> # uname -a
>> Linux localhost.localdomain 2.6.32-220.el6.x86_64 #1 SMP Tue Dec 6
>> 19:48:22 GMT 2011 x86_64 x86_64 x86_64 GNU/Linux
> This is really an old kernel and also a distribution one which might
> contain a lot of patches on top of the core kernel. I would suggest to
> contact Redhat or try to reproduce the issue with the vanilla and

What's the meaning of vanilla?

> up-to-date kernel and report here.

Re: Inactive memory keep growing and how to release it?

[Theodore Ts'o]

On Fri, Mar 15, 2013 at 04:41:41PM +0800, Simon Jeons wrote:
> >This is really an old kernel and also a distribution one which might
> >contain a lot of patches on top of the core kernel. I would suggest to
> >contact Redhat or try to reproduce the issue with the vanilla and
> 
> What's the meaning of vanilla?

Vanilla means an up-to-date (i.e., non-prehistoric) kernel from
kernel.org, without any "Value Added" patches from a distribution.

See: https://www.kernel.org/

Regards,

					- Ted

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Inactive memory keep growing and how to release it?

[Theodore Ts'o]

On Fri, Mar 15, 2013 at 04:41:41PM +0800, Simon Jeons wrote:
> >This is really an old kernel and also a distribution one which might
> >contain a lot of patches on top of the core kernel. I would suggest to
> >contact Redhat or try to reproduce the issue with the vanilla and
> 
> What's the meaning of vanilla?

Vanilla means an up-to-date (i.e., non-prehistoric) kernel from
kernel.org, without any "Value Added" patches from a distribution.

See: https://www.kernel.org/

Regards,

					- Ted