[Bug 76821] New: jbd2 write on disk each 5/6 seconds

[Bug 76821] New: jbd2 write on disk each 5/6 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

            Bug ID: 76821
           Summary: jbd2 write on disk each 5/6 seconds
           Product: File System
           Version: 2.5
    Kernel Version: 3.14.4-1-ARCH
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
          Assignee: fs_ext4@kernel-bugs.osdl.org
          Reporter: abdelhak@cryptolab.net
        Regression: No

➜  ~  sudo iotop -obtqqq | grep jbd2             
[sudo] password for hacko: 
13:12:05   390 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.87 %
[jbd2/sda6-8]
13:12:06   390 be/3 root        0.00 B/s   31.45 K/s  0.00 %  2.48 %
[jbd2/sda6-8]
13:12:12   111 be/3 root        0.00 B/s   11.53 K/s  0.00 %  4.99 %
[jbd2/sda3-8]
13:12:19   390 be/3 root        0.00 B/s    3.84 K/s  0.00 %  2.98 %
[jbd2/sda6-8]
13:12:25   390 be/3 root        0.00 B/s    3.84 K/s  0.00 %  5.91 %
[jbd2/sda6-8]
13:12:25   111 be/3 root        0.00 B/s   23.03 K/s  0.00 %  5.71 %
[jbd2/sda3-8]
13:12:31   390 be/3 root        0.00 B/s    3.84 K/s  0.00 %  4.42 %
[jbd2/sda6-8]
13:12:32   390 be/3 root        0.00 B/s    0.00 B/s  0.00 %  0.03 %
[jbd2/sda6-8]
13:12:34   390 be/3 root        0.00 B/s   23.05 K/s  0.00 %  7.62 %
[jbd2/sda6-8]
13:12:39   390 be/3 root        0.00 B/s   26.86 K/s  0.00 %  4.40 %
[jbd2/sda6-8]
13:12:46   390 be/3 root        0.00 B/s    3.83 K/s  0.00 %  4.42 %
[jbd2/sda6-8]
13:12:53   390 be/3 root        0.00 B/s   11.49 K/s  0.00 %  4.31 %
[jbd2/sda6-8]
13:13:01   390 be/3 root        0.00 B/s   11.53 K/s  0.00 %  4.46 %
[jbd2/sda6-8]
13:13:07   390 be/3 root        0.00 B/s    7.69 K/s  0.00 %  4.55 %
[jbd2/sda6-8]
13:13:12   390 be/3 root        0.00 B/s   11.56 K/s  0.00 %  4.85 %
[jbd2/sda6-8]
13:13:19   390 be/3 root        0.00 B/s   11.56 K/s  0.00 %  4.27 %
[jbd2/sda6-8]
13:13:23   390 be/3 root        0.00 B/s   46.13 K/s  0.00 %  9.26 %
[jbd2/sda6-8]
13:13:26   390 be/3 root        0.00 B/s    7.54 K/s  0.00 %  6.05 %
[jbd2/sda6-8]
13:13:31   390 be/3 root        0.00 B/s   30.01 K/s  0.00 %  4.60 %
[jbd2/sda6-8]

-- 
You are receiving this mail because:
You are watching the assignee of the bug.--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Bug 76821] jbd2 write on disk each 5/6 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Abdelhak Bougouffa <abdelhak@cryptolab.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                URL|                            |https://bbs.archlinux.org/v
                   |                            |iewtopic.php?pid=1393346

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 5/6 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Abdelhak Bougouffa <abdelhak@cryptolab.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
     Kernel Version|3.14.4-1-ARCH               |3.14.4-1-ARCH,
                   |                            |3.10.40-1-lts

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Abdelhak Bougouffa <abdelhak@cryptolab.net> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
            Summary|jbd2 write on disk each 5/6 |jbd2 write on disk each 2/3
                   |seconds                     |seconds

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

--- Comment #1 from Abdelhak Bougouffa <abdelhak@cryptolab.net> ---
Hi,
I have observed that "kworker" is writing to the disk even when no program
opened >.<, when I ran `iotop -Po` I see that "kworker" write more than jbd2
:-(

I have no idea how to solve this!

I searched the web and I found that many users have the same problem

https://bbs.archlinux.org/viewtopic.php?id=171608

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Theodore Tso <tytso@mit.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |tytso@mit.edu

--- Comment #2 from Theodore Tso <tytso@mit.edu> ---
If your kernel doesn't have ftrace enabled, please enable it by enabling the
following kernel config options:

    CONFIG_FUNCTION_TRACER
    CONFIG_FUNCTION_GRAPH_TRACER
    CONFIG_STACK_TRACER
    CONFIG_DYNAMIC_FTRACE

Once you do that, make sure debugfs is mounted:

mount -t debugfs debugfs /sys/kernel/debug

Now enable the ext4_mark_inode_dirty tracepoint and start tracing:

cd /sys/kernel/debug/tracing
echo 1 > events/ext4/ext4_mark_inode_dirty/enable
cat trace_pipe

You will start seeing lines like this:

   kworker/u16:2-14718 [006] ...1 616239.342464: ext4_mark_inode_dirty: dev
252,0 ino 4203428 caller ext4_writepages+0x743/0x93e
 BrowserBlocking-5875  [006] ...1 616241.972533: ext4_mark_inode_dirty: dev 8,3
ino 3545316 caller ext4_dirty_inode+0x40/0x59


The first line indicates that process 14718, which is a kernel thread named
kworking/u16:2, dirtied inode 4203428 on device 252,0, which is /dev/dm-0
(which on my system has major number 252, minor 0, which you can see by the
command ls -l /dev/dm-0).   This is probably dirty inode writeback, so that
doesn't help tell you who dirtied the file.  You can find the filename using
debugfs:

debugfs /dev/dm-0
debugfs: ncheck 4203428
Inode    Pathname
4203428    /build/e2fsprogs-maint/tests/f_eofblocks.1.log

(and since /dev/dm-0 is mounted on /u1, I know the filename is
/u1/build/e2fsprogs-maint/....)  

The second line indicates the task id was 5875, and with a name like
BrowserBlocking, it's almost certainly one of the Chrome threads.  The ps
program won't show individual threads by default, but you can also use the
inode number to find which process has the file open by using lsof:

lsof | grep 3545316

Once you're done, don't forget to disable the tracing to save a bit on CPU and
battery life:

cat /dev/null > /sys/kernel/debug/tracing/set_event

Anyway, I hope this helps.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

--- Comment #3 from Abdelhak Bougouffa <abdelhak@cryptolab.net> ---
Hi, 
Thank you Mr. Theodore

So, I tried this commands, and I get a big number of process! 

I observed that the kworker is not really the cause! the systemd-journal is
writing all times to the disk!, When I traced one of his used inodes, I found
it writing to /var/log/journal/****.journal, each time a file of 8MB.

http://pastebin.com/W3yu3SVH

It's notable that the debugging options:
    CONFIG_FUNCTION_TRACER
    CONFIG_FUNCTION_GRAPH_TRACER
    CONFIG_STACK_TRACER
    CONFIG_DYNAMIC_FTRACE

Are enabled in my working kernel (I installed it using the Arch repository)

Thanks!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

--- Comment #4 from Abdelhak Bougouffa <abdelhak@cryptolab.net> ---
I observed also that all the journal files are binary files and the total size
of /var/log/journal directory is 2.0GB O.o ... so what can I do!!

-- 
You are receiving this mail because:
You are watching the assignee of the bug.

[Bug 76821] jbd2 write on disk each 2/3 seconds

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Theodore Tso <tytso@mit.edu> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #5 from Theodore Tso <tytso@mit.edu> ---
You can try running "journalctl -f" and see what kind of log messages you are
seeing.  But in any case, this isn't an ext4 bug, but rather a systemd,
journald, or some system daemon or distro-level bug.

If userspace shovels stuff for us to write, we have to write it!   So for that
reason, I've closed out this bug.  If this isn't a configuration issue, please
open a bug with the appropriate distribution or systemd bug tracking system.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.