systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

[Szőts Ákos]

Dear list,

I have two openSUSE 12.3 systems with kernel 3.9. On one of them there's an 
ext4 partition, while on the other there's a btrfs.

I issued a "time journalctl -b --no-pager" command on both systems. This shows 
the logs from the current boot without passing them to "less".

On ext4 (3.9.3):
real    0m1.898s
user    0m0.291s
sys     0m0.105s

On btrfs (3.9.2):
real    1m49.698s
user    0m0.102s
sys     0m0.470s

Journalctl on btrfs was always this slow, some btrfsck were made on the file 
system too, but I don't think it was corrupted. On just the first run it's 
sluggish, after it's fast as the ext4 one.

Is it a known issue or can I help somehow debugging this further?

Best regards,

Ákos Szőts

Re: systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

[Clemens Eisserer]

Hi,

>> Journalctl on btrfs was always this slow, some btrfsck were made on the file
>> system too, but I don't think it was corrupted. On just the first run it's
>> sluggish, after it's fast as the ext4 one.

Most likely the file is heavily fragmented due to COW.
Have you tried to manually defrag the filesystem and turn on autodefrag?

Regards, Clemens

Re: systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

[George Mitchell]

I have gotten what appear to be large increases in speed out of btrfs by 
defragmentation of meta data. The manual defragmentation process takes 
forever as you have to defragment incrementally directory by directory.  
I was at the point where KDE startup times were getting abysmal (along 
with journalctl, etc) and the multiple drives would churn incessantly on 
startup.  In the case of KDE, I found almost magical improvement with 
one operation:  `btrfs filesystem defrag /usr/share`.  I am currently 
going through the whole system deframenting directory by directory.  Its 
amazing, it proceeds quite quickly and then hits a directory at random 
where it sits and plods away seemingly forever before moving on.  I am 
convinced that there is something going on here with meta data 
fragmentation that, at times, is seriously affecting performance.  I 
*think* that autodefrag, once its out the door will hopefully solve 
this, in the mean time I am trying to come up with some sort of way to 
schedule an anacron job to deal with this issue.  But my suggestion 
would be that you try defragging your /var filesystem as thoroughly as 
possible on the meta data side.

On 05/27/2013 09:21 AM, Szőts Ákos wrote:
> Dear list,
>
> I have two openSUSE 12.3 systems with kernel 3.9. On one of them there's an
> ext4 partition, while on the other there's a btrfs.
>
> I issued a "time journalctl -b --no-pager" command on both systems. This shows
> the logs from the current boot without passing them to "less".
>
> On ext4 (3.9.3):
> real    0m1.898s
> user    0m0.291s
> sys     0m0.105s
>
> On btrfs (3.9.2):
> real    1m49.698s
> user    0m0.102s
> sys     0m0.470s
>
> Journalctl on btrfs was always this slow, some btrfsck were made on the file
> system too, but I don't think it was corrupted. On just the first run it's
> sluggish, after it's fast as the ext4 one.
>
> Is it a known issue or can I help somehow debugging this further?
>
> Best regards,
>
> Ákos Szőts
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

[Sergei Trofimovich]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

On Mon, 27 May 2013 18:21:21 +0200
Szőts Ákos <szotsaki@gmail.com> wrote:

> Dear list,
> 
> I have two openSUSE 12.3 systems with kernel 3.9. On one of them there's an 
> ext4 partition, while on the other there's a btrfs.
> 
> I issued a "time journalctl -b --no-pager" command on both systems. This shows 
> the logs from the current boot without passing them to "less".
> 
> On ext4 (3.9.3):
> real    0m1.898s
> user    0m0.291s
> sys     0m0.105s
> 
> On btrfs (3.9.2):
> real    1m49.698s
> user    0m0.102s
> sys     0m0.470s
> 
> Journalctl on btrfs was always this slow, some btrfsck were made on the file 
> system too, but I don't think it was corrupted. On just the first run it's 
> sluggish, after it's fast as the ext4 one.
> 
> Is it a known issue or can I help somehow debugging this further?

Let's look at your
    /usr/sbin/filefrag /var/log/journal/*/*
on both systems

-- 

  Sergei

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: systemd "journalctl" is 1.89sec on ext4, 1.49 min on btrfs

[Szőts Ákos]

Thank you very much for you suggestions!

I ran `filefrag` on /var/log/journal/ and the most fragmented files were:
/var/log/journal/[...]/faa9e41c6.journal: 11136 extents found
/var/log/journal/[...]/e9f8e1346c.journal: 633 extents found
/var/log/journal/[...]/59ec50b631.journal: 3585 extents found
/var/log/journal/[...]/59ec51d1b4.journal: 710 extents found
/var/log/journal/[...]/f74afadee9.journal: 484 extents found

Then I ran `btrfs fi defrag -v -f -clzo *` in the aforementioned directory 
three times and restarted the machine. Now, the `journalctl -b --no-pager` 
command runs in less than two seconds.

I tried to further optimize my file system (MySQL big storage files are highly 
affected), but somewhere (for me it looks like, on random places) there are 
kernel errors, so I suspended the further defragmentation until those errors 
are tracked down.

Thank you very much again.

Ákos