High latency with ext4 / jbd2 kernel thread

High latency with ext4 / jbd2 kernel thread

[Andy Campbell]

Hi ext4 mailing list members,

I hope you can help me with a problem.

I recently upgraded my main audio PC to Fedora FC15 and, at the same
time, reformatted by root drive to ext4.  Since this upgrade this, my
system has high system latency which is causing my audio hardware
(firewire) to experience buffer under-runs (XRUNs).  This causes
"unhandled xrun" errors and the audio software fails.

My audio software (jackd) is running in realtime mode with SCHED_FIFO priority.

I have been using latencytop (http://www.latencytop.org/) to try to
understand what is causing the system latency.
Typically my system runs with a max latency around 5ms.  If i run a
disk heavy operation (such as updatedb) I see a max latency of around
20ms.

Around the time that jackd fails, I can see events like this with a
latency of 40 - 120 ms:
"writing buffer to disk (synchronous)"  = "jbd2_journal_commit_transaction"
Screenshot here
http://dl.dropbox.com/u/3289905/latencytop.png

My kernel version is
Linux speedy.local 2.6.38.8-32.fc15.i686.PAE #1 SMP Mon Jun 13
19:55:27 UTC 2011 i686 i686 i386 GNU/Linu

The ext4 drive is mounted with these options:
/dev/sda4 on / type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)
I also tried mounting using using noatime and also using
data=writeback options.  Unfortunately neither of these parameters
resolved the problem

Could you give me any advice or ideas about how I could minimize this latency?

Thanks!

Andy

Re: High latency with ext4 / jbd2 kernel thread

[Tao Ma]

Hi Andy,
On 07/12/2011 08:07 AM, Andy Campbell wrote:
> Hi ext4 mailing list members,
> 
> I hope you can help me with a problem.
> 
> I recently upgraded my main audio PC to Fedora FC15 and, at the same
> time, reformatted by root drive to ext4.  Since this upgrade this, my
> system has high system latency which is causing my audio hardware
> (firewire) to experience buffer under-runs (XRUNs).  This causes
> "unhandled xrun" errors and the audio software fails.
> 
> My audio software (jackd) is running in realtime mode with SCHED_FIFO priority.
> 
> I have been using latencytop (http://www.latencytop.org/) to try to
> understand what is causing the system latency.
> Typically my system runs with a max latency around 5ms.  If i run a
> disk heavy operation (such as updatedb) I see a max latency of around
> 20ms.
> 
> Around the time that jackd fails, I can see events like this with a
> latency of 40 - 120 ms:
> "writing buffer to disk (synchronous)"  = "jbd2_journal_commit_transaction"
> Screenshot here
> http://dl.dropbox.com/u/3289905/latencytop.png
I guess the reason may be some regression.
I have sent a patch to linux-ext4 with the subject
"jbd/2[stable only]: Use WRITE_SYNC_PLUG in journal_commit_transaction."

Can you try it to see if it helps.
> 
> My kernel version is
> Linux speedy.local 2.6.38.8-32.fc15.i686.PAE #1 SMP Mon Jun 13
> 19:55:27 UTC 2011 i686 i686 i386 GNU/Linu
> 
> The ext4 drive is mounted with these options:
> /dev/sda4 on / type ext4 (rw,relatime,seclabel,barrier=1,data=ordered)
> I also tried mounting using using noatime and also using
> data=writeback options.  Unfortunately neither of these parameters
> resolved the problem

Thanks
Tao

Re: High latency with ext4 / jbd2 kernel thread

[Theodore Tso]

On Jul 11, 2011, at 8:07 PM, Andy Campbell wrote:

> Hi ext4 mailing list members,
> 
> I hope you can help me with a problem.
> 
> I recently upgraded my main audio PC to Fedora FC15 and, at the same
> time, reformatted by root drive to ext4.  Since this upgrade this, my
> system has high system latency which is causing my audio hardware
> (firewire) to experience buffer under-runs (XRUNs).  This causes
> "unhandled xrun" errors and the audio software fails.

What were you using before?  ext3 ?   It may be that ext4's default use of
barriers (for safety reasons) is causing a difference.   Try mounting your
file systems with barrier=0.   For the root filesystem, add "rootflags=barrier=0".
This is an unsafe way to run, in that if you have a power failure while you
are writing to the disk, your file system(s) could get corrupted.   Ext3 
defaulted to this for historical reasons (although the enterprise Linux systems,
RHEL and SuSE changed the default for safety reasons).

If this solves the problem, I'd suggest that you look into finding ways to
increase the buffer size of your audio system (were you recording or
doing playback at the time).   You might also try to see if there was 
some amount of disk activity (especially write activity) that could be
avoided or find some way of avoiding the use of fsync() so that you
can re-enable write barriers without causing your latency skips.

Regards,

-- Ted

Re: High latency with ext4 / jbd2 kernel thread

[Andy Campbell]

Hi Ted, thanks very much for getting back to me

The problem started when I upgraded from Fedora 14 to Fedora 15.
Previously I was using ext3.  However, I was also using a different
firewire stack (the "old stack", instead of the new "Juju stack").
So, there have been several changes around the time that the problem
started, not only a change to Ext4.

I have tried many different buffer sizes on the audio software.
Previously this hardware worked perfectly with 3 * 256 buffers.  I've
tried up to 3 * 1024 buffers, but I still get the same problem.

I tried using barrier=0.
This stopped the "writing buffer to disk (synchronous)"  =
"jbd2_journal_commit_transaction" latency events from showing, which
is great.  However, I still get high latency from other events in the
system including "kworker/1:1 Executing raw SCSI command" and
"kworker/u:7 - Fork() system call".

These still cause drop-outs on my system.  It looks like either the
new stack is more sensitive to latency, or  something else changed in
the kernel between Fedora 14 and Fedora 15.

I think my conclusion is that my problems are not caused by Ext4, but
are something else... I will keep looking :O)

Thanks for your help!

A

On Tue, Jul 12, 2011 at 12:17 PM, Theodore Tso <tytso@mit.edu> wrote:
>
> On Jul 11, 2011, at 8:07 PM, Andy Campbell wrote:
>
>> Hi ext4 mailing list members,
>>
>> I hope you can help me with a problem.
>>
>> I recently upgraded my main audio PC to Fedora FC15 and, at the same
>> time, reformatted by root drive to ext4.  Since this upgrade this, my
>> system has high system latency which is causing my audio hardware
>> (firewire) to experience buffer under-runs (XRUNs).  This causes
>> "unhandled xrun" errors and the audio software fails.
>
> What were you using before?  ext3 ?   It may be that ext4's default use of
> barriers (for safety reasons) is causing a difference.   Try mounting your
> file systems with barrier=0.   For the root filesystem, add "rootflags=barrier=0".
> This is an unsafe way to run, in that if you have a power failure while you
> are writing to the disk, your file system(s) could get corrupted.   Ext3
> defaulted to this for historical reasons (although the enterprise Linux systems,
> RHEL and SuSE changed the default for safety reasons).
>
> If this solves the problem, I'd suggest that you look into finding ways to
> increase the buffer size of your audio system (were you recording or
> doing playback at the time).   You might also try to see if there was
> some amount of disk activity (especially write activity) that could be
> avoided or find some way of avoiding the use of fsync() so that you
> can re-enable write barriers without causing your latency skips.
>
> Regards,
>
> -- Ted
>
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html