Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

Hey Guys,
I seem to be hitting my first kernel bug in the kernel stated in my
subject line.
When I transfer a lot of music for a brtfs external hard drive I have the music
in clementine I am listening to is stopped for a few seconds and then continues.
I just using basic intel audio, nothing amazing. When someone replies to this
I will give more information as asked for :).
Cheers Nick
P.S. Thanks Guys for putting up with me a bit before I did something useful

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Matt Joras]

Nick,

I could be wrong but I doubt this is symptomatic of a "kernel bug" as 
you seem to think. Are you saying that the music stutters during a 
transfer operation to your external hard drive? I.e. the music playback 
in Clementine stutters while you are writing to the disk the music files 
are on? If so, that doesn't really sound like a bug to me, especially 
not a sound driver bug. That sounds like a pretty typical instance of IO 
waiting; i.e. the drive is busy and Clementine has to wait its turn to 
fill up its buffer again with data. Most of the time applications like 
that will let you increase their "buffer size" which is essentially just 
the amount of data the application will load at a time from the disk.

If that's not the situation you were describing, feel free to clarify.

Regards,
Matt Joras

On 07/26/2014 08:41 PM, Nick Krause wrote:
> Hey Guys,
> I seem to be hitting my first kernel bug in the kernel stated in my
> subject line.
> When I transfer a lot of music for a brtfs external hard drive I have the music
> in clementine I am listening to is stopped for a few seconds and then continues.
> I just using basic intel audio, nothing amazing. When someone replies to this
> I will give more information as asked for :).
> Cheers Nick
> P.S. Thanks Guys for putting up with me a bit before I did something useful
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Mike Galbraith]

On Sat, 2014-07-26 at 22:07 -0700, Matt Joras wrote: 
> Nick,
> 
> I could be wrong but I doubt this is symptomatic of a "kernel bug" as 
> you seem to think. Are you saying that the music stutters during a 
> transfer operation to your external hard drive? I.e. the music playback 
> in Clementine stutters while you are writing to the disk the music files 
> are on? If so, that doesn't really sound like a bug to me, especially 
> not a sound driver bug.

Not a sound driver bug, no, but is perhaps a latency issue if buffering
is not ridiculously tiny.  It's a difficult to explore, but worth it.

-Mike

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Mike Galbraith]

On Sat, 2014-07-26 at 23:41 -0400, Nick Krause wrote: 
> Hey Guys,
> I seem to be hitting my first kernel bug in the kernel stated in my
> subject line.
> When I transfer a lot of music for a brtfs external hard drive I have the music
> in clementine I am listening to is stopped for a few seconds and then continues.

The thing to do now is to collect solid data for a proper bug report.
How to reproduce etc.  See REPORTING-BUGS.

For a guy interested in kernel hacking, this is the perfect opportunity
to sink your teeth into a bug that you can reproduce on your box.

If you go there...

You could use perf sched record/lat and/or ftrace to trace/identify the
event, analyze what's going on in the kernel while the bad thing is
happening.  If you manage to get the root cause properly nailed down,
you will have dug out highly useful information.

Um, don't be surprised and appalled if you find that you're not up to
fixing this.  A more realistic goal is to capture some high quality
information, and learn how to use the tools to _chase_ bugs.  Once you
get a firm grip on a bug, you can always holler "Help, it's got me!" if
it's a serious bug.

-Mike

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Sun, Jul 27, 2014 at 4:37 AM, Heinz Diehl <htd@fancy-poultry.org> wrote:
> On 27.07.2014, Nick Krause wrote:
>
>> When I transfer a lot of music for a brtfs external hard drive I have the music
>> in clementine I am listening to is stopped for a few seconds and then continues.
>
> Most probably, this is not a bug, but caused by high disk I/O. It's
> not clear to me if the music you are listening to is located on the
> same disk which you transfer to/from. This would make things even
> worse.
>
> However, there are a few thing you can do to try to minimize the observed
> stalls:
>
> 1. Compile your kernel with CONFIG_PREEMPT enabled
> 2. Reduce the amount of memory content sync'ed to disk
> 3. If your disk is a SSD: change the elevator to "noop"
> 4. Enable writeback on your disk
> 5. Increase the disks readahead sectors
> 6. Increase the audio buffers in your player
>
> 1. CONFIG_PREEMPT=y
>
> 2. Sysctl: vm.dirty_ratio = 10
>            vm.dirty_background_ratio = 5
>
> 3. echo "noop" > /sys/block/sda/queue/scheduler
>
> 4. /sbin/hdparm -W1 /dev/sda
>
> 5. /sbin/blockdev --setra 8192 /dev/sda
>
> Regards,
>  Heinz.
Heinz,
The audio buffer is not going to work as this was never a issue with
default Ubuntu kernels.
However I will try your other ideas and report back :). I am
transferring into the same directory
for the music I am listening to. In addition , my cpu usage when doing
this is half of my older
Ubuntu distro kernel, 6 vs 3 percent for upstream.
Cheers Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Sun, Jul 27, 2014 at 4:18 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Sun, Jul 27, 2014 at 4:37 AM, Heinz Diehl <htd@fancy-poultry.org> wrote:
>> On 27.07.2014, Nick Krause wrote:
>>
>>> When I transfer a lot of music for a brtfs external hard drive I have the music
>>> in clementine I am listening to is stopped for a few seconds and then continues.
>>
>> Most probably, this is not a bug, but caused by high disk I/O. It's
>> not clear to me if the music you are listening to is located on the
>> same disk which you transfer to/from. This would make things even
>> worse.
>>
>> However, there are a few thing you can do to try to minimize the observed
>> stalls:
>>
>> 1. Compile your kernel with CONFIG_PREEMPT enabled
>> 2. Reduce the amount of memory content sync'ed to disk
>> 3. If your disk is a SSD: change the elevator to "noop"
>> 4. Enable writeback on your disk
>> 5. Increase the disks readahead sectors
>> 6. Increase the audio buffers in your player
>>
>> 1. CONFIG_PREEMPT=y
>>
>> 2. Sysctl: vm.dirty_ratio = 10
>>            vm.dirty_background_ratio = 5
>>
>> 3. echo "noop" > /sys/block/sda/queue/scheduler
>>
>> 4. /sbin/hdparm -W1 /dev/sda
>>
>> 5. /sbin/blockdev --setra 8192 /dev/sda
>>
>> Regards,
>>  Heinz.
> Heinz,
> The audio buffer is not going to work as this was never a issue with
> default Ubuntu kernels.
> However I will try your other ideas and report back :). I am
> transferring into the same directory
> for the music I am listening to. In addition , my cpu usage when doing
> this is half of my older
> Ubuntu distro kernel, 6 vs 3 percent for upstream.
> Cheers Nick

Seems to be an issue after updating my settings as stated.
I am now using kernel 3.16 r7 as I updated my git tree before
compiling my kernel again.
Regards Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Heinz Diehl]

First: my email you are quoting here was sent to you directly, off-list,
because it only contains common stuff and has nothing to do with
kernel development. In other words to avoid annoying lkml people. 
In addition, I placed an "Reply-to" header pointing back to me. 
Despite, you *manually* redirected your answer to my private mail 
back to the list, thus annoying people and breaking the informational
flow of this thread.


On 27.07.2014, Nick Krause wrote: 

> I am transferring into the same directory for the music 
> I am listening to.

So you produce a lot of disk I/O by writing/reading from the same disk
at the same time, and as you encounter stalls, you take this for a
kernel bug (which clearly is not). These stalls are most probably
caused by fsync() flushing your data/buffers, which is blocking.

So moving the music data you are listening to to another disk would
make things a lot easier.

> In addition , my cpu usage when doing this is half of my older

DMA transfer is mostly done be the controller itself, and not the CPU.

Btw, EOT for me.

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Mon, Jul 28, 2014 at 5:54 AM, Heinz Diehl <htd@fancy-poultry.org> wrote:
> First: my email you are quoting here was sent to you directly, off-list,
> because it only contains common stuff and has nothing to do with
> kernel development. In other words to avoid annoying lkml people.
> In addition, I placed an "Reply-to" header pointing back to me.
> Despite, you *manually* redirected your answer to my private mail
> back to the list, thus annoying people and breaking the informational
> flow of this thread.
>
>
> On 27.07.2014, Nick Krause wrote:
>
>> I am transferring into the same directory for the music
>> I am listening to.
>
> So you produce a lot of disk I/O by writing/reading from the same disk
> at the same time, and as you encounter stalls, you take this for a
> kernel bug (which clearly is not). These stalls are most probably
> caused by fsync() flushing your data/buffers, which is blocking.
>
> So moving the music data you are listening to to another disk would
> make things a lot easier.
>
>> In addition , my cpu usage when doing this is half of my older
>
> DMA transfer is mostly done be the controller itself, and not the CPU.
>
> Btw, EOT for me.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/


Then explain to me why this is only happening on recent rc kernels and not
the Ubuntu distro kernels I was using before. Seems weird that only
recent rc kernels are triggering this.
Regards Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Heinz Diehl]

On 28.07.2014, Nick Krause wrote: 

> Then explain to me why this is only happening on recent rc kernels and not
> the Ubuntu distro kernels I was using before. Seems weird that only
> recent rc kernels are triggering this.

The, if you actually can reproduce it with a vanilla kernel (non-
distro) and you're still convinced it must be a bug, you could
consider bisecting the offending patch. Here is how it works:

http://tinyurl.com/yo533k

Good luck!

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Mon, Jul 28, 2014 at 4:13 PM, Heinz Diehl <htd@fancy-poultry.org> wrote:
> On 28.07.2014, Nick Krause wrote:
>
>> Then explain to me why this is only happening on recent rc kernels and not
>> the Ubuntu distro kernels I was using before. Seems weird that only
>> recent rc kernels are triggering this.
>
> The, if you actually can reproduce it with a vanilla kernel (non-
> distro) and you're still convinced it must be a bug, you could
> consider bisecting the offending patch. Here is how it works:
>
> http://tinyurl.com/yo533k
>
> Good luck!
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Thanks Heinz,
I will bisect tonight and see if I can find the offending commit.
Regards Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Mon, Jul 28, 2014 at 6:24 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Mon, Jul 28, 2014 at 4:13 PM, Heinz Diehl <htd@fancy-poultry.org> wrote:
>> On 28.07.2014, Nick Krause wrote:
>>
>>> Then explain to me why this is only happening on recent rc kernels and not
>>> the Ubuntu distro kernels I was using before. Seems weird that only
>>> recent rc kernels are triggering this.
>>
>> The, if you actually can reproduce it with a vanilla kernel (non-
>> distro) and you're still convinced it must be a bug, you could
>> consider bisecting the offending patch. Here is how it works:
>>
>> http://tinyurl.com/yo533k
>>
>> Good luck!
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>
> Thanks Heinz,
> I will bisect tonight and see if I can find the offending commit.
> Regards Nick

Heinz,
No luck with git still appears in 3.12 r6 as far as I can tell seems to be
from an older kernel then I thought. Doesn't show up on a newer 3.13
Ubuntu kernel through.
Regards Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Tue, Jul 29, 2014 at 1:37 AM, Nick Krause <xerofoify@gmail.com> wrote:
> On Mon, Jul 28, 2014 at 6:24 PM, Nick Krause <xerofoify@gmail.com> wrote:
>> On Mon, Jul 28, 2014 at 4:13 PM, Heinz Diehl <htd@fancy-poultry.org> wrote:
>>> On 28.07.2014, Nick Krause wrote:
>>>
>>>> Then explain to me why this is only happening on recent rc kernels and not
>>>> the Ubuntu distro kernels I was using before. Seems weird that only
>>>> recent rc kernels are triggering this.
>>>
>>> The, if you actually can reproduce it with a vanilla kernel (non-
>>> distro) and you're still convinced it must be a bug, you could
>>> consider bisecting the offending patch. Here is how it works:
>>>
>>> http://tinyurl.com/yo533k
>>>
>>> Good luck!
>>>
>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>> Thanks Heinz,
>> I will bisect tonight and see if I can find the offending commit.
>> Regards Nick
>
> Heinz,
> No luck with git still appears in 3.12 r6 as far as I can tell seems to be
> from an older kernel then I thought. Doesn't show up on a newer 3.13
> Ubuntu kernel through.
> Regards Nick

Seems after changing my settings in Clementine it works but it
now seems to need to buffer 10 seconds versus 4 seconds
from before. Seems like a regression in either block io or
something related to the vfs.
Cheers Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Tue, Jul 29, 2014 at 1:02 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Tue, Jul 29, 2014 at 11:42 AM,  <dev@mjoras.com> wrote:
>> On 2014-07-28 22:57, Nick Krause wrote:
>>>r the hang, so doing a bisect might lead us to some innocent commit.
> I have run the rsync + snapshot job several times here now, and no hang.
>
>>>
>>> Seems after changing my settings in Clementine it works but it
>>> now seems to need to buffer 10 seconds versus 4 seconds
>>> from before.
>>
>>
>> Nick,
>> For the record you can also use something like ionice to avoid these sorts
>> of annoyances. See: https://bbs.archlinux.org/viewtopic.php?id=176711
>> Matt Joras
>
> Thanks Matt,
> Seems weird through that sound buffering is needed more, thanks for
> the advice through.
> Regards Nick

Matt,
After doing more testing this is definitely a bug as no matter what
buffer size this
seems to be an issue my music stalls when moving music into the directory
of the music I am listening to and this didn't happen when I was running stock
Ubuntu kernels.
Regards Nick

Re: Bug on Kernel 3.16 r6: Sound and Buffering in Clementine with Files are Transferring to Music Directory

[Nick Krause]

On Tue, Jul 29, 2014 at 1:34 PM, Nick Krause <xerofoify@gmail.com> wrote:
> On Tue, Jul 29, 2014 at 1:02 PM, Nick Krause <xerofoify@gmail.com> wrote:
>> On Tue, Jul 29, 2014 at 11:42 AM,  <dev@mjoras.com> wrote:
>>> On 2014-07-28 22:57, Nick Krause wrote:
>>>>r the hang, so doing a bisect might lead us to some innocent commit.
>> I have run the rsync + snapshot job several times here now, and no hang.
>>
>>>>
>>>> Seems after changing my settings in Clementine it works but it
>>>> now seems to need to buffer 10 seconds versus 4 seconds
>>>> from before.
>>>
>>>
>>> Nick,
>>> For the record you can also use something like ionice to avoid these sorts
>>> of annoyances. See: https://bbs.archlinux.org/viewtopic.php?id=176711
>>> Matt Joras
>>
>> Thanks Matt,
>> Seems weird through that sound buffering is needed more, thanks for
>> the advice through.
>> Regards Nick
>
> Matt,
> After doing more testing this is definitely a bug as no matter what
> buffer size this
> seems to be an issue my music stalls when moving music into the directory
> of the music I am listening to and this didn't happen when I was running stock
> Ubuntu kernels.
> Regards Nick

Seems to be an issue with Ubuntu.
Regards Nick