Bad fs performance, IO freezes

Bad fs performance, IO freezes

[cheater00 .]

Hi guys,
I am running into really bad performance. Here's my setup:

WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
32-bit with kernel 4.0.4-040004-generic #201505171336.

Single btrfs partition covering whole disk.

Autodefrag is on.

fstab line:
UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0

Sometimes when files are being modified or removed, I see
btrfs-transacti eat 100% cpu; during this time no io operations
succeed, that is, they're all stalled. You can't even ls on that fs.
This happens for several minutes then normal operation resumes. There
doesn't seem to be a rule to what will trigger this, other than
opening a single file and reading usually works quite well. (say,
watching a movie while all other programs are closed). But even moving
files off the disks triggers some sort of bug. Just now I am moving a
few files (just 30gb worth) onto another disk, and the bug triggers.
So btrfs-transacti was eating my cpu for over 5 minutes and according
to mv's output after this was done and cpu usage went back to normal
what I was waiting for was for a tiny png file to be removed. This is
pretty bad.

I have tried defragmenting directories where files are being accessed
and moved. This hasn't helped.

This happens whether the FS is near full or not. It currently is near
full but it wasn't before and it still did that. It still has about ~
100GB free space now.

The more things are happening the more often this bug gets triggered.
So if I have utorrent running and its temporary downloads directory is
there, its download speed graph will be a few spikes of running at
several MB/sec separated by durations of 0 download speed.

Nothing seems to show up in dmesg or syslog.

I have asked in #btrfs but the suggestions ended up not fixing the
issue (autodefrag, defrag dirs).

Please advise what I should do with this issue.

Re: Bad fs performance, IO freezes

[Donald Pearson]

Accidentally didn't reply to the list the 1st time.

I see the same issue when I have quotas enabled.  If you have quotas
on, see if turning them off helps.

On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
> Hi guys,
> I am running into really bad performance. Here's my setup:
>
> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>
> Single btrfs partition covering whole disk.
>
> Autodefrag is on.
>
> fstab line:
> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>
> Sometimes when files are being modified or removed, I see
> btrfs-transacti eat 100% cpu; during this time no io operations
> succeed, that is, they're all stalled. You can't even ls on that fs.
> This happens for several minutes then normal operation resumes. There
> doesn't seem to be a rule to what will trigger this, other than
> opening a single file and reading usually works quite well. (say,
> watching a movie while all other programs are closed). But even moving
> files off the disks triggers some sort of bug. Just now I am moving a
> few files (just 30gb worth) onto another disk, and the bug triggers.
> So btrfs-transacti was eating my cpu for over 5 minutes and according
> to mv's output after this was done and cpu usage went back to normal
> what I was waiting for was for a tiny png file to be removed. This is
> pretty bad.
>
> I have tried defragmenting directories where files are being accessed
> and moved. This hasn't helped.
>
> This happens whether the FS is near full or not. It currently is near
> full but it wasn't before and it still did that. It still has about ~
> 100GB free space now.
>
> The more things are happening the more often this bug gets triggered.
> So if I have utorrent running and its temporary downloads directory is
> there, its download speed graph will be a few spikes of running at
> several MB/sec separated by durations of 0 download speed.
>
> Nothing seems to show up in dmesg or syslog.
>
> I have asked in #btrfs but the suggestions ended up not fixing the
> issue (autodefrag, defrag dirs).
>
> Please advise what I should do with this issue.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[cheater00 .]

There are no quotas. I haven't enabled them. I believe the fstab says
that - could they be enabled in another way? How do I check for sure?
The man page doesn't say how to check the status:
https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota

On Mon, Oct 26, 2015 at 2:32 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> Accidentally didn't reply to the list the 1st time.
>
> I see the same issue when I have quotas enabled.  If you have quotas
> on, see if turning them off helps.
>
> On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
>> Hi guys,
>> I am running into really bad performance. Here's my setup:
>>
>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>
>> Single btrfs partition covering whole disk.
>>
>> Autodefrag is on.
>>
>> fstab line:
>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>
>> Sometimes when files are being modified or removed, I see
>> btrfs-transacti eat 100% cpu; during this time no io operations
>> succeed, that is, they're all stalled. You can't even ls on that fs.
>> This happens for several minutes then normal operation resumes. There
>> doesn't seem to be a rule to what will trigger this, other than
>> opening a single file and reading usually works quite well. (say,
>> watching a movie while all other programs are closed). But even moving
>> files off the disks triggers some sort of bug. Just now I am moving a
>> few files (just 30gb worth) onto another disk, and the bug triggers.
>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>> to mv's output after this was done and cpu usage went back to normal
>> what I was waiting for was for a tiny png file to be removed. This is
>> pretty bad.
>>
>> I have tried defragmenting directories where files are being accessed
>> and moved. This hasn't helped.
>>
>> This happens whether the FS is near full or not. It currently is near
>> full but it wasn't before and it still did that. It still has about ~
>> 100GB free space now.
>>
>> The more things are happening the more often this bug gets triggered.
>> So if I have utorrent running and its temporary downloads directory is
>> there, its download speed graph will be a few spikes of running at
>> several MB/sec separated by durations of 0 download speed.
>>
>> Nothing seems to show up in dmesg or syslog.
>>
>> I have asked in #btrfs but the suggestions ended up not fixing the
>> issue (autodefrag, defrag dirs).
>>
>> Please advise what I should do with this issue.
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Donald Pearson]

AFAIK quotas aren't a mount option, but if you never enabled them and
created the qgroups by hand that's your answer and the issue must be
something else.

On Mon, Oct 26, 2015 at 8:36 AM, cheater00 . <cheater00@gmail.com> wrote:
> There are no quotas. I haven't enabled them. I believe the fstab says
> that - could they be enabled in another way? How do I check for sure?
> The man page doesn't say how to check the status:
> https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota
>
> On Mon, Oct 26, 2015 at 2:32 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> Accidentally didn't reply to the list the 1st time.
>>
>> I see the same issue when I have quotas enabled.  If you have quotas
>> on, see if turning them off helps.
>>
>> On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
>>> Hi guys,
>>> I am running into really bad performance. Here's my setup:
>>>
>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>
>>> Single btrfs partition covering whole disk.
>>>
>>> Autodefrag is on.
>>>
>>> fstab line:
>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>
>>> Sometimes when files are being modified or removed, I see
>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>> This happens for several minutes then normal operation resumes. There
>>> doesn't seem to be a rule to what will trigger this, other than
>>> opening a single file and reading usually works quite well. (say,
>>> watching a movie while all other programs are closed). But even moving
>>> files off the disks triggers some sort of bug. Just now I am moving a
>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>> to mv's output after this was done and cpu usage went back to normal
>>> what I was waiting for was for a tiny png file to be removed. This is
>>> pretty bad.
>>>
>>> I have tried defragmenting directories where files are being accessed
>>> and moved. This hasn't helped.
>>>
>>> This happens whether the FS is near full or not. It currently is near
>>> full but it wasn't before and it still did that. It still has about ~
>>> 100GB free space now.
>>>
>>> The more things are happening the more often this bug gets triggered.
>>> So if I have utorrent running and its temporary downloads directory is
>>> there, its download speed graph will be a few spikes of running at
>>> several MB/sec separated by durations of 0 download speed.
>>>
>>> Nothing seems to show up in dmesg or syslog.
>>>
>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>> issue (autodefrag, defrag dirs).
>>>
>>> Please advise what I should do with this issue.
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[cheater00 .]

I don't remember doing that, but just to exclude everything, how do I check?

On Mon, Oct 26, 2015 at 2:45 PM, Donald Pearson
<donaldwhpearson@gmail.com> wrote:
> AFAIK quotas aren't a mount option, but if you never enabled them and
> created the qgroups by hand that's your answer and the issue must be
> something else.
>
> On Mon, Oct 26, 2015 at 8:36 AM, cheater00 . <cheater00@gmail.com> wrote:
>> There are no quotas. I haven't enabled them. I believe the fstab says
>> that - could they be enabled in another way? How do I check for sure?
>> The man page doesn't say how to check the status:
>> https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota
>>
>> On Mon, Oct 26, 2015 at 2:32 PM, Donald Pearson
>> <donaldwhpearson@gmail.com> wrote:
>>> Accidentally didn't reply to the list the 1st time.
>>>
>>> I see the same issue when I have quotas enabled.  If you have quotas
>>> on, see if turning them off helps.
>>>
>>> On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
>>>> Hi guys,
>>>> I am running into really bad performance. Here's my setup:
>>>>
>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>
>>>> Single btrfs partition covering whole disk.
>>>>
>>>> Autodefrag is on.
>>>>
>>>> fstab line:
>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>
>>>> Sometimes when files are being modified or removed, I see
>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>> This happens for several minutes then normal operation resumes. There
>>>> doesn't seem to be a rule to what will trigger this, other than
>>>> opening a single file and reading usually works quite well. (say,
>>>> watching a movie while all other programs are closed). But even moving
>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>> to mv's output after this was done and cpu usage went back to normal
>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>> pretty bad.
>>>>
>>>> I have tried defragmenting directories where files are being accessed
>>>> and moved. This hasn't helped.
>>>>
>>>> This happens whether the FS is near full or not. It currently is near
>>>> full but it wasn't before and it still did that. It still has about ~
>>>> 100GB free space now.
>>>>
>>>> The more things are happening the more often this bug gets triggered.
>>>> So if I have utorrent running and its temporary downloads directory is
>>>> there, its download speed graph will be a few spikes of running at
>>>> several MB/sec separated by durations of 0 download speed.
>>>>
>>>> Nothing seems to show up in dmesg or syslog.
>>>>
>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>> issue (autodefrag, defrag dirs).
>>>>
>>>> Please advise what I should do with this issue.
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[cheater00 .]

fwiw, I did this:

sudo btrfs qgroup show /media/X
ERROR: can't perform the search - No such file or directory
ERROR: can't list qgroups: No such file or directory

I assume this means no qgroups present, which means no quotas present.
Please correct me if I'm wrong.
So yes, the issue must lie elsewhere.

On Mon, Oct 26, 2015 at 2:46 PM, cheater00 . <cheater00@gmail.com> wrote:
> I don't remember doing that, but just to exclude everything, how do I check?
>
> On Mon, Oct 26, 2015 at 2:45 PM, Donald Pearson
> <donaldwhpearson@gmail.com> wrote:
>> AFAIK quotas aren't a mount option, but if you never enabled them and
>> created the qgroups by hand that's your answer and the issue must be
>> something else.
>>
>> On Mon, Oct 26, 2015 at 8:36 AM, cheater00 . <cheater00@gmail.com> wrote:
>>> There are no quotas. I haven't enabled them. I believe the fstab says
>>> that - could they be enabled in another way? How do I check for sure?
>>> The man page doesn't say how to check the status:
>>> https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota
>>>
>>> On Mon, Oct 26, 2015 at 2:32 PM, Donald Pearson
>>> <donaldwhpearson@gmail.com> wrote:
>>>> Accidentally didn't reply to the list the 1st time.
>>>>
>>>> I see the same issue when I have quotas enabled.  If you have quotas
>>>> on, see if turning them off helps.
>>>>
>>>> On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
>>>>> Hi guys,
>>>>> I am running into really bad performance. Here's my setup:
>>>>>
>>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>>
>>>>> Single btrfs partition covering whole disk.
>>>>>
>>>>> Autodefrag is on.
>>>>>
>>>>> fstab line:
>>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>>
>>>>> Sometimes when files are being modified or removed, I see
>>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>>> This happens for several minutes then normal operation resumes. There
>>>>> doesn't seem to be a rule to what will trigger this, other than
>>>>> opening a single file and reading usually works quite well. (say,
>>>>> watching a movie while all other programs are closed). But even moving
>>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>>> to mv's output after this was done and cpu usage went back to normal
>>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>>> pretty bad.
>>>>>
>>>>> I have tried defragmenting directories where files are being accessed
>>>>> and moved. This hasn't helped.
>>>>>
>>>>> This happens whether the FS is near full or not. It currently is near
>>>>> full but it wasn't before and it still did that. It still has about ~
>>>>> 100GB free space now.
>>>>>
>>>>> The more things are happening the more often this bug gets triggered.
>>>>> So if I have utorrent running and its temporary downloads directory is
>>>>> there, its download speed graph will be a few spikes of running at
>>>>> several MB/sec separated by durations of 0 download speed.
>>>>>
>>>>> Nothing seems to show up in dmesg or syslog.
>>>>>
>>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>>> issue (autodefrag, defrag dirs).
>>>>>
>>>>> Please advise what I should do with this issue.
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Donald Pearson]

I get the same kind of muddy errors when I do a quota rescan on the
filesystem or qgroup show on any subvolume on mine, and I know I don't
have them enabled.

On Mon, Oct 26, 2015 at 8:56 AM, cheater00 . <cheater00@gmail.com> wrote:
> fwiw, I did this:
>
> sudo btrfs qgroup show /media/X
> ERROR: can't perform the search - No such file or directory
> ERROR: can't list qgroups: No such file or directory
>
> I assume this means no qgroups present, which means no quotas present.
> Please correct me if I'm wrong.
> So yes, the issue must lie elsewhere.
>
> On Mon, Oct 26, 2015 at 2:46 PM, cheater00 . <cheater00@gmail.com> wrote:
>> I don't remember doing that, but just to exclude everything, how do I check?
>>
>> On Mon, Oct 26, 2015 at 2:45 PM, Donald Pearson
>> <donaldwhpearson@gmail.com> wrote:
>>> AFAIK quotas aren't a mount option, but if you never enabled them and
>>> created the qgroups by hand that's your answer and the issue must be
>>> something else.
>>>
>>> On Mon, Oct 26, 2015 at 8:36 AM, cheater00 . <cheater00@gmail.com> wrote:
>>>> There are no quotas. I haven't enabled them. I believe the fstab says
>>>> that - could they be enabled in another way? How do I check for sure?
>>>> The man page doesn't say how to check the status:
>>>> https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-quota
>>>>
>>>> On Mon, Oct 26, 2015 at 2:32 PM, Donald Pearson
>>>> <donaldwhpearson@gmail.com> wrote:
>>>>> Accidentally didn't reply to the list the 1st time.
>>>>>
>>>>> I see the same issue when I have quotas enabled.  If you have quotas
>>>>> on, see if turning them off helps.
>>>>>
>>>>> On Mon, Oct 26, 2015 at 7:16 AM, cheater00 . <cheater00@gmail.com> wrote:
>>>>>> Hi guys,
>>>>>> I am running into really bad performance. Here's my setup:
>>>>>>
>>>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>>>
>>>>>> Single btrfs partition covering whole disk.
>>>>>>
>>>>>> Autodefrag is on.
>>>>>>
>>>>>> fstab line:
>>>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>>>
>>>>>> Sometimes when files are being modified or removed, I see
>>>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>>>> This happens for several minutes then normal operation resumes. There
>>>>>> doesn't seem to be a rule to what will trigger this, other than
>>>>>> opening a single file and reading usually works quite well. (say,
>>>>>> watching a movie while all other programs are closed). But even moving
>>>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>>>> to mv's output after this was done and cpu usage went back to normal
>>>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>>>> pretty bad.
>>>>>>
>>>>>> I have tried defragmenting directories where files are being accessed
>>>>>> and moved. This hasn't helped.
>>>>>>
>>>>>> This happens whether the FS is near full or not. It currently is near
>>>>>> full but it wasn't before and it still did that. It still has about ~
>>>>>> 100GB free space now.
>>>>>>
>>>>>> The more things are happening the more often this bug gets triggered.
>>>>>> So if I have utorrent running and its temporary downloads directory is
>>>>>> there, its download speed graph will be a few spikes of running at
>>>>>> several MB/sec separated by durations of 0 download speed.
>>>>>>
>>>>>> Nothing seems to show up in dmesg or syslog.
>>>>>>
>>>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>>>> issue (autodefrag, defrag dirs).
>>>>>>
>>>>>> Please advise what I should do with this issue.
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Liu Bo]

On 10/26/2015 08:16 PM, cheater00 . wrote:
> Hi guys,
> I am running into really bad performance. Here's my setup:
>
> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>
> Single btrfs partition covering whole disk.
>
> Autodefrag is on.
>
> fstab line:
> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>
> Sometimes when files are being modified or removed, I see
> btrfs-transacti eat 100% cpu; during this time no io operations
> succeed, that is, they're all stalled. You can't even ls on that fs.
> This happens for several minutes then normal operation resumes. There
> doesn't seem to be a rule to what will trigger this, other than
> opening a single file and reading usually works quite well. (say,
> watching a movie while all other programs are closed). But even moving
> files off the disks triggers some sort of bug. Just now I am moving a
> few files (just 30gb worth) onto another disk, and the bug triggers.
> So btrfs-transacti was eating my cpu for over 5 minutes and according
> to mv's output after this was done and cpu usage went back to normal
> what I was waiting for was for a tiny png file to be removed. This is
> pretty bad.
>
> I have tried defragmenting directories where files are being accessed
> and moved. This hasn't helped.
>
> This happens whether the FS is near full or not. It currently is near
> full but it wasn't before and it still did that. It still has about ~
> 100GB free space now.
>
> The more things are happening the more often this bug gets triggered.
> So if I have utorrent running and its temporary downloads directory is
> there, its download speed graph will be a few spikes of running at
> several MB/sec separated by durations of 0 download speed.
>
> Nothing seems to show up in dmesg or syslog.
>
> I have asked in #btrfs but the suggestions ended up not fixing the
> issue (autodefrag, defrag dirs).
>
> Please advise what I should do with this issue.

It might be related to delayed ref rework, the last time I saw this kind 
of hanging problem about btrfs-transaction eating cpu is that because 
btrfs doesn't merge delayed refs, it'd be best to try the lastest kernel 
and if the issue is not resolved, then we can work out a reproducer and 
provide debugging.

Thanks,

Liubo

Re: Bad fs performance, IO freezes

[cheater00 .]

Thanks for the reply. What version did this go into? I'll try getting
a prebuilt backport of the kernel, building source could slow things
down considerably, but debs will not be available for the latest few
minor versions I guess. So if you can tell me a min version, I'll try
to find the latest deb newer than that, or I'll build if that's not
available.

On Mon, Oct 26, 2015 at 3:25 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
> On 10/26/2015 08:16 PM, cheater00 . wrote:
>>
>> Hi guys,
>> I am running into really bad performance. Here's my setup:
>>
>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>
>> Single btrfs partition covering whole disk.
>>
>> Autodefrag is on.
>>
>> fstab line:
>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>
>> Sometimes when files are being modified or removed, I see
>> btrfs-transacti eat 100% cpu; during this time no io operations
>> succeed, that is, they're all stalled. You can't even ls on that fs.
>> This happens for several minutes then normal operation resumes. There
>> doesn't seem to be a rule to what will trigger this, other than
>> opening a single file and reading usually works quite well. (say,
>> watching a movie while all other programs are closed). But even moving
>> files off the disks triggers some sort of bug. Just now I am moving a
>> few files (just 30gb worth) onto another disk, and the bug triggers.
>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>> to mv's output after this was done and cpu usage went back to normal
>> what I was waiting for was for a tiny png file to be removed. This is
>> pretty bad.
>>
>> I have tried defragmenting directories where files are being accessed
>> and moved. This hasn't helped.
>>
>> This happens whether the FS is near full or not. It currently is near
>> full but it wasn't before and it still did that. It still has about ~
>> 100GB free space now.
>>
>> The more things are happening the more often this bug gets triggered.
>> So if I have utorrent running and its temporary downloads directory is
>> there, its download speed graph will be a few spikes of running at
>> several MB/sec separated by durations of 0 download speed.
>>
>> Nothing seems to show up in dmesg or syslog.
>>
>> I have asked in #btrfs but the suggestions ended up not fixing the
>> issue (autodefrag, defrag dirs).
>>
>> Please advise what I should do with this issue.
>
>
> It might be related to delayed ref rework, the last time I saw this kind of
> hanging problem about btrfs-transaction eating cpu is that because btrfs
> doesn't merge delayed refs, it'd be best to try the lastest kernel and if
> the issue is not resolved, then we can work out a reproducer and provide
> debugging.
>
> Thanks,
>
> Liubo

Re: Bad fs performance, IO freezes

[cheater00 .]

I have located 4.3.0-rc7 binaries which I will now try.

On Mon, Oct 26, 2015 at 3:38 PM, cheater00 . <cheater00@gmail.com> wrote:
> Thanks for the reply. What version did this go into? I'll try getting
> a prebuilt backport of the kernel, building source could slow things
> down considerably, but debs will not be available for the latest few
> minor versions I guess. So if you can tell me a min version, I'll try
> to find the latest deb newer than that, or I'll build if that's not
> available.
>
> On Mon, Oct 26, 2015 at 3:25 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>> On 10/26/2015 08:16 PM, cheater00 . wrote:
>>>
>>> Hi guys,
>>> I am running into really bad performance. Here's my setup:
>>>
>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>
>>> Single btrfs partition covering whole disk.
>>>
>>> Autodefrag is on.
>>>
>>> fstab line:
>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>
>>> Sometimes when files are being modified or removed, I see
>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>> This happens for several minutes then normal operation resumes. There
>>> doesn't seem to be a rule to what will trigger this, other than
>>> opening a single file and reading usually works quite well. (say,
>>> watching a movie while all other programs are closed). But even moving
>>> files off the disks triggers some sort of bug. Just now I am moving a
>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>> to mv's output after this was done and cpu usage went back to normal
>>> what I was waiting for was for a tiny png file to be removed. This is
>>> pretty bad.
>>>
>>> I have tried defragmenting directories where files are being accessed
>>> and moved. This hasn't helped.
>>>
>>> This happens whether the FS is near full or not. It currently is near
>>> full but it wasn't before and it still did that. It still has about ~
>>> 100GB free space now.
>>>
>>> The more things are happening the more often this bug gets triggered.
>>> So if I have utorrent running and its temporary downloads directory is
>>> there, its download speed graph will be a few spikes of running at
>>> several MB/sec separated by durations of 0 download speed.
>>>
>>> Nothing seems to show up in dmesg or syslog.
>>>
>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>> issue (autodefrag, defrag dirs).
>>>
>>> Please advise what I should do with this issue.
>>
>>
>> It might be related to delayed ref rework, the last time I saw this kind of
>> hanging problem about btrfs-transaction eating cpu is that because btrfs
>> doesn't merge delayed refs, it'd be best to try the lastest kernel and if
>> the issue is not resolved, then we can work out a reproducer and provide
>> debugging.
>>
>> Thanks,
>>
>> Liubo

Re: Bad fs performance, IO freezes

[cheater00 .]

So far I cannot reproduce. If I don't post again this means the issue
has been fixed by updating the kernel.

On Mon, Oct 26, 2015 at 4:40 PM, cheater00 . <cheater00@gmail.com> wrote:
> I have located 4.3.0-rc7 binaries which I will now try.
>
> On Mon, Oct 26, 2015 at 3:38 PM, cheater00 . <cheater00@gmail.com> wrote:
>> Thanks for the reply. What version did this go into? I'll try getting
>> a prebuilt backport of the kernel, building source could slow things
>> down considerably, but debs will not be available for the latest few
>> minor versions I guess. So if you can tell me a min version, I'll try
>> to find the latest deb newer than that, or I'll build if that's not
>> available.
>>
>> On Mon, Oct 26, 2015 at 3:25 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>>> On 10/26/2015 08:16 PM, cheater00 . wrote:
>>>>
>>>> Hi guys,
>>>> I am running into really bad performance. Here's my setup:
>>>>
>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>
>>>> Single btrfs partition covering whole disk.
>>>>
>>>> Autodefrag is on.
>>>>
>>>> fstab line:
>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>
>>>> Sometimes when files are being modified or removed, I see
>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>> This happens for several minutes then normal operation resumes. There
>>>> doesn't seem to be a rule to what will trigger this, other than
>>>> opening a single file and reading usually works quite well. (say,
>>>> watching a movie while all other programs are closed). But even moving
>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>> to mv's output after this was done and cpu usage went back to normal
>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>> pretty bad.
>>>>
>>>> I have tried defragmenting directories where files are being accessed
>>>> and moved. This hasn't helped.
>>>>
>>>> This happens whether the FS is near full or not. It currently is near
>>>> full but it wasn't before and it still did that. It still has about ~
>>>> 100GB free space now.
>>>>
>>>> The more things are happening the more often this bug gets triggered.
>>>> So if I have utorrent running and its temporary downloads directory is
>>>> there, its download speed graph will be a few spikes of running at
>>>> several MB/sec separated by durations of 0 download speed.
>>>>
>>>> Nothing seems to show up in dmesg or syslog.
>>>>
>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>> issue (autodefrag, defrag dirs).
>>>>
>>>> Please advise what I should do with this issue.
>>>
>>>
>>> It might be related to delayed ref rework, the last time I saw this kind of
>>> hanging problem about btrfs-transaction eating cpu is that because btrfs
>>> doesn't merge delayed refs, it'd be best to try the lastest kernel and if
>>> the issue is not resolved, then we can work out a reproducer and provide
>>> debugging.
>>>
>>> Thanks,
>>>
>>> Liubo

Re: Bad fs performance, IO freezes

[cheater00 .]

I do not experience btrfs-transacti going up to 100% for minutes at a
time now (not reproduced yet) but I have it spiking up to say 30% for
a short while and everything jags during that time. So, say, if I am
watching youtube, the sound cuts out and the video drops out for a
bit. And if I'm typing, then what I typed during that time gets lost,
like if I never typed that.

I have also connected the same HDD bay with a USB3 cable instead of
USB2. It's on an USB3 port. So it's running via USB3 now.


On Mon, Oct 26, 2015 at 6:43 PM, cheater00 . <cheater00@gmail.com> wrote:
> So far I cannot reproduce. If I don't post again this means the issue
> has been fixed by updating the kernel.
>
> On Mon, Oct 26, 2015 at 4:40 PM, cheater00 . <cheater00@gmail.com> wrote:
>> I have located 4.3.0-rc7 binaries which I will now try.
>>
>> On Mon, Oct 26, 2015 at 3:38 PM, cheater00 . <cheater00@gmail.com> wrote:
>>> Thanks for the reply. What version did this go into? I'll try getting
>>> a prebuilt backport of the kernel, building source could slow things
>>> down considerably, but debs will not be available for the latest few
>>> minor versions I guess. So if you can tell me a min version, I'll try
>>> to find the latest deb newer than that, or I'll build if that's not
>>> available.
>>>
>>> On Mon, Oct 26, 2015 at 3:25 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>>>> On 10/26/2015 08:16 PM, cheater00 . wrote:
>>>>>
>>>>> Hi guys,
>>>>> I am running into really bad performance. Here's my setup:
>>>>>
>>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>>
>>>>> Single btrfs partition covering whole disk.
>>>>>
>>>>> Autodefrag is on.
>>>>>
>>>>> fstab line:
>>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>>
>>>>> Sometimes when files are being modified or removed, I see
>>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>>> This happens for several minutes then normal operation resumes. There
>>>>> doesn't seem to be a rule to what will trigger this, other than
>>>>> opening a single file and reading usually works quite well. (say,
>>>>> watching a movie while all other programs are closed). But even moving
>>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>>> to mv's output after this was done and cpu usage went back to normal
>>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>>> pretty bad.
>>>>>
>>>>> I have tried defragmenting directories where files are being accessed
>>>>> and moved. This hasn't helped.
>>>>>
>>>>> This happens whether the FS is near full or not. It currently is near
>>>>> full but it wasn't before and it still did that. It still has about ~
>>>>> 100GB free space now.
>>>>>
>>>>> The more things are happening the more often this bug gets triggered.
>>>>> So if I have utorrent running and its temporary downloads directory is
>>>>> there, its download speed graph will be a few spikes of running at
>>>>> several MB/sec separated by durations of 0 download speed.
>>>>>
>>>>> Nothing seems to show up in dmesg or syslog.
>>>>>
>>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>>> issue (autodefrag, defrag dirs).
>>>>>
>>>>> Please advise what I should do with this issue.
>>>>
>>>>
>>>> It might be related to delayed ref rework, the last time I saw this kind of
>>>> hanging problem about btrfs-transaction eating cpu is that because btrfs
>>>> doesn't merge delayed refs, it'd be best to try the lastest kernel and if
>>>> the issue is not resolved, then we can work out a reproducer and provide
>>>> debugging.
>>>>
>>>> Thanks,
>>>>
>>>> Liubo

Re: Bad fs performance, IO freezes

[cheater00 .]

Hello,
currently my computer freezes every several seconds for half a second
or so. Using it feels like I'm playing musical chairs with the kernel.
I have just one download happening on utorrent right now - this is
what the graph looks like:
http://i.imgur.com/LqhMtrJ.png
and every time a new spike happens, a freeze happens just before
that... that's the only time those freezes happen, too.

Please advise.

On Mon, Oct 26, 2015 at 7:31 PM, cheater00 . <cheater00@gmail.com> wrote:
> I do not experience btrfs-transacti going up to 100% for minutes at a
> time now (not reproduced yet) but I have it spiking up to say 30% for
> a short while and everything jags during that time. So, say, if I am
> watching youtube, the sound cuts out and the video drops out for a
> bit. And if I'm typing, then what I typed during that time gets lost,
> like if I never typed that.
>
> I have also connected the same HDD bay with a USB3 cable instead of
> USB2. It's on an USB3 port. So it's running via USB3 now.
>
>
> On Mon, Oct 26, 2015 at 6:43 PM, cheater00 . <cheater00@gmail.com> wrote:
>> So far I cannot reproduce. If I don't post again this means the issue
>> has been fixed by updating the kernel.
>>
>> On Mon, Oct 26, 2015 at 4:40 PM, cheater00 . <cheater00@gmail.com> wrote:
>>> I have located 4.3.0-rc7 binaries which I will now try.
>>>
>>> On Mon, Oct 26, 2015 at 3:38 PM, cheater00 . <cheater00@gmail.com> wrote:
>>>> Thanks for the reply. What version did this go into? I'll try getting
>>>> a prebuilt backport of the kernel, building source could slow things
>>>> down considerably, but debs will not be available for the latest few
>>>> minor versions I guess. So if you can tell me a min version, I'll try
>>>> to find the latest deb newer than that, or I'll build if that's not
>>>> available.
>>>>
>>>> On Mon, Oct 26, 2015 at 3:25 PM, Liu Bo <bo.li.liu@oracle.com> wrote:
>>>>> On 10/26/2015 08:16 PM, cheater00 . wrote:
>>>>>>
>>>>>> Hi guys,
>>>>>> I am running into really bad performance. Here's my setup:
>>>>>>
>>>>>> WD Red 6 TB connected over USB2 to my core i7 laptop, running Ubuntu
>>>>>> 32-bit with kernel 4.0.4-040004-generic #201505171336.
>>>>>>
>>>>>> Single btrfs partition covering whole disk.
>>>>>>
>>>>>> Autodefrag is on.
>>>>>>
>>>>>> fstab line:
>>>>>> UUID=... /media/X btrfs rw,nosuid,nodev,autodefrag 0 0
>>>>>>
>>>>>> Sometimes when files are being modified or removed, I see
>>>>>> btrfs-transacti eat 100% cpu; during this time no io operations
>>>>>> succeed, that is, they're all stalled. You can't even ls on that fs.
>>>>>> This happens for several minutes then normal operation resumes. There
>>>>>> doesn't seem to be a rule to what will trigger this, other than
>>>>>> opening a single file and reading usually works quite well. (say,
>>>>>> watching a movie while all other programs are closed). But even moving
>>>>>> files off the disks triggers some sort of bug. Just now I am moving a
>>>>>> few files (just 30gb worth) onto another disk, and the bug triggers.
>>>>>> So btrfs-transacti was eating my cpu for over 5 minutes and according
>>>>>> to mv's output after this was done and cpu usage went back to normal
>>>>>> what I was waiting for was for a tiny png file to be removed. This is
>>>>>> pretty bad.
>>>>>>
>>>>>> I have tried defragmenting directories where files are being accessed
>>>>>> and moved. This hasn't helped.
>>>>>>
>>>>>> This happens whether the FS is near full or not. It currently is near
>>>>>> full but it wasn't before and it still did that. It still has about ~
>>>>>> 100GB free space now.
>>>>>>
>>>>>> The more things are happening the more often this bug gets triggered.
>>>>>> So if I have utorrent running and its temporary downloads directory is
>>>>>> there, its download speed graph will be a few spikes of running at
>>>>>> several MB/sec separated by durations of 0 download speed.
>>>>>>
>>>>>> Nothing seems to show up in dmesg or syslog.
>>>>>>
>>>>>> I have asked in #btrfs but the suggestions ended up not fixing the
>>>>>> issue (autodefrag, defrag dirs).
>>>>>>
>>>>>> Please advise what I should do with this issue.
>>>>>
>>>>>
>>>>> It might be related to delayed ref rework, the last time I saw this kind of
>>>>> hanging problem about btrfs-transaction eating cpu is that because btrfs
>>>>> doesn't merge delayed refs, it'd be best to try the lastest kernel and if
>>>>> the issue is not resolved, then we can work out a reproducer and provide
>>>>> debugging.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Liubo

Re: Bad fs performance, IO freezes

[Duncan]

cheater00 . posted on Tue, 27 Oct 2015 03:00:05 +0100 as excerpted:

> currently my computer freezes every several seconds for half a second or
> so. Using it feels like I'm playing musical chairs with the kernel.
> I have just one download happening on utorrent right now - this is what
> the graph looks like:
> http://i.imgur.com/LqhMtrJ.png and every time a new spike happens, a
> freeze happens just before that... that's the only time those freezes
> happen, too.

Is it perchance every 30 seconds?  (The graph seems to indicate more like 
50 second cycles, but...)

Because that's btrfs' normal commit time.  If it is, there's a mount 
option to change the commit time (the wiki says commit=N, N=30 by 
default, since kernel 3.12).  You could try fiddling with that, say 
setting it to 15 or 60, not necessarily to fix the problem (tho 60 
seconds might increase the problem while making it less frequent, while 
15 might make it more frequent but less of a problem), but to see if the 
cycle changes with the commit time option, or stays @ 30 seconds.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: Bad fs performance, IO freezes

[cheater00 .]

No, sadly the intervals don't seem to be regular like that.

On Tue, Oct 27, 2015 at 7:39 AM, Duncan <1i5t5.duncan@cox.net> wrote:
> cheater00 . posted on Tue, 27 Oct 2015 03:00:05 +0100 as excerpted:
>
>> currently my computer freezes every several seconds for half a second or
>> so. Using it feels like I'm playing musical chairs with the kernel.
>> I have just one download happening on utorrent right now - this is what
>> the graph looks like:
>> http://i.imgur.com/LqhMtrJ.png and every time a new spike happens, a
>> freeze happens just before that... that's the only time those freezes
>> happen, too.
>
> Is it perchance every 30 seconds?  (The graph seems to indicate more like
> 50 second cycles, but...)
>
> Because that's btrfs' normal commit time.  If it is, there's a mount
> option to change the commit time (the wiki says commit=N, N=30 by
> default, since kernel 3.12).  You could try fiddling with that, say
> setting it to 15 or 60, not necessarily to fix the problem (tho 60
> seconds might increase the problem while making it less frequent, while
> 15 might make it more frequent but less of a problem), but to see if the
> cycle changes with the commit time option, or stays @ 30 seconds.
>
> --
> Duncan - List replies preferred.   No HTML msgs.
> "Every nonfree program has a lord, a master --
> and if you use the program, he is your master."  Richard Stallman
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 1015 bytes --]

On 2015-10-26 22:00, cheater00 . wrote:
> Hello,
> currently my computer freezes every several seconds for half a second
> or so. Using it feels like I'm playing musical chairs with the kernel.
> I have just one download happening on utorrent right now - this is
> what the graph looks like:
> http://i.imgur.com/LqhMtrJ.png
> and every time a new spike happens, a freeze happens just before
> that... that's the only time those freezes happen, too.
>
Do you have the 'autodefrag' mount option enabled?  If it is turned on, 
then that may be the problem.  Most bittorrent clients pre-allocate the 
space for a download, then write each block directly into the location 
it's supposed to be in the resultant download, which means depending on 
how it's pre-allocating the space, that you end up with a large number 
of randomly ordered writes into a single file, which in turn will 
trigger the autodefrag code, which can cause latency spikes when you're 
also hitting the disk at the same time.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Bad fs performance, IO freezes

[Henk Slager]

I don't have a lot experience with autodefrag, but as indicated by
Austin, expect a lot of full rewrites of files that are relatively
slowly filled up by a torrent client, starting with a sparse file. So
1st advice would be to remove this option and run it as crontask at
particular times.

What SATA-USB bridge is between the harddisk and the PC motherboard ?
Also what USB host chipset is on the PC motherboard ?
Why don't you run 64-bit Ubuntu on this core i7 ?


On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-10-26 22:00, cheater00 . wrote:
>>
>> Hello,
>> currently my computer freezes every several seconds for half a second
>> or so. Using it feels like I'm playing musical chairs with the kernel.
>> I have just one download happening on utorrent right now - this is
>> what the graph looks like:
>> http://i.imgur.com/LqhMtrJ.png
>> and every time a new spike happens, a freeze happens just before
>> that... that's the only time those freezes happen, too.
>>
> Do you have the 'autodefrag' mount option enabled?  If it is turned on, then
> that may be the problem.  Most bittorrent clients pre-allocate the space for
> a download, then write each block directly into the location it's supposed
> to be in the resultant download, which means depending on how it's
> pre-allocating the space, that you end up with a large number of randomly
> ordered writes into a single file, which in turn will trigger the autodefrag
> code, which can cause latency spikes when you're also hitting the disk at
> the same time.
>

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 3253 bytes --]

On 2015-10-27 09:00, Henk Slager wrote:
> I don't have a lot experience with autodefrag, but as indicated by
> Austin, expect a lot of full rewrites of files that are relatively
> slowly filled up by a torrent client, starting with a sparse file. So
> 1st advice would be to remove this option and run it as crontask at
> particular times.
>
> What SATA-USB bridge is between the harddisk and the PC motherboard ?
I hadn't thought of this, but the specific adapter being used for the 
disk can have a lot of impact on how it preforms.  I've personally had 
lots of issues with JMicron chipsets (ranging from latency issues like 
what you are seeing to sever data corruption), but have found that 
ASMedia ones tend to be pretty much rock solid reliable and have good 
performance (although I think they only do USB 3.0 adapters).
> Also what USB host chipset is on the PC motherboard ?
If it's a Intel motherboard, the USB 2.0 ports are probably routed 
through on-board hubs to the ports provided by whatever Intel calls 
their equivalent of the south bridge these days, and the USB 3.0 ports 
are probably a mix of Intel and ASMedia XHCI controllers (ASMedia was 
one of the first companies to do an inexpensive standalone XHCI chip, so 
they're relatively popular for extra USB 3.0 ports).  FWIW, the first 
generation of Intel XHCI chips had some issues with older Linux kernels, 
although IIRC those issues were along the lines of a port just 
disappearing after disconnecting whatever was connected to it.
> Why don't you run 64-bit Ubuntu on this core i7 ?
64 versus 32 bit shouldn't cause anything like this to happen (although, 
if it can be proven that it does, then that is a serious bug that needs 
to be fixed).  That said, unless you have some desperate need to be 
running 32-bit only, you should seriously look into updating to a 64-bit 
version, your whole system should run faster, and Ubuntu has really good 
32-bit compatibility in the 64-bit version (which is part of why it's 
popular as a support target for third party software like Steam).
>
>
> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 2015-10-26 22:00, cheater00 . wrote:
>>>
>>> Hello,
>>> currently my computer freezes every several seconds for half a second
>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>> I have just one download happening on utorrent right now - this is
>>> what the graph looks like:
>>> http://i.imgur.com/LqhMtrJ.png
>>> and every time a new spike happens, a freeze happens just before
>>> that... that's the only time those freezes happen, too.
>>>
>> Do you have the 'autodefrag' mount option enabled?  If it is turned on, then
>> that may be the problem.  Most bittorrent clients pre-allocate the space for
>> a download, then write each block directly into the location it's supposed
>> to be in the resultant download, which means depending on how it's
>> pre-allocating the space, that you end up with a large number of randomly
>> ordered writes into a single file, which in turn will trigger the autodefrag
>> code, which can cause latency spikes when you're also hitting the disk at
>> the same time.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Bad fs performance, IO freezes

[cheater00 .]

The (dual) HDD bay and the chipset are, according to lsusb:
Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
USA Technology Corp.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

Not sure how to find out specific model numbers? I could open up the
bay. OK I'll open up the bay.
Good thing I have just the right screwdriver. It's a JMS551, and just
for records sake, here's the manufacture info:

JMS551
1120 LGAA2 A
572QV0024

The laptop manual says it's either "Intel HM65 Express chipset with
NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
Here are technical documents for my model:
Manual: http://docdro.id/hG627JM
"Intel chipset datasheet": http://docdro.id/yKRupYO
Service guide: http://docdro.id/AuDgUdE
Service guide, alt. ver.: http://docdro.id/WwQRpsH

FWIW I'm using one of the USB3 ports on the left. The ones on the
right are USB2.

I've never used docdro.id so if it's not good let me know where to
upload the PDFs to.

autodefrag is on, yes. But I have been having issues before turning it
on - I turned it on as a measure towards fixing the issues. I will
turn it off and remount, then report. But I don't think that should be
it. As you see the transfer speeds are minimal. They're *all* that's
happening on the disk. Right now that's under 100 KB/sec and I'm still
getting freezes albeit less. Also why would I be getting freezes when
the transfer speeds jump up - just for them to drop again? Hmm, maybe
utorrent has some sort of scheduler that gets preempted while the
spike is happening, and some algorithm in it gets the wrong idea and
turns some sort of flow control on, because it thinks it's hit some
sort of physical transfer speed barrier. Also notice the upload and
download both scale together, but that just might be how torrent
works, maybe it just tries to be fair i.e. only uploads as much as it
downloaded (scaled by a constant).

The system is 32 bit because I installed ubuntu 6 one day and just
kept on upgrading. I keep on telling myself I'll update to 64 bits,
one of these days. But this laptop only has 8 gigs of ram, so no real
reason to upgrade to 64 bit anyways. It's not like I need firefox to
be able to eat 8 gb of ram whereas right now it can only eat 4. There
is no simple upgrade path that I know of so it's either a fresh
install or doing something like this: http://www.ewan.cc/?q=node/132
-- I keep telling myself /one of these days/...

On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-10-27 09:00, Henk Slager wrote:
>>
>> I don't have a lot experience with autodefrag, but as indicated by
>> Austin, expect a lot of full rewrites of files that are relatively
>> slowly filled up by a torrent client, starting with a sparse file. So
>> 1st advice would be to remove this option and run it as crontask at
>> particular times.
>>
>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>
> I hadn't thought of this, but the specific adapter being used for the disk
> can have a lot of impact on how it preforms.  I've personally had lots of
> issues with JMicron chipsets (ranging from latency issues like what you are
> seeing to sever data corruption), but have found that ASMedia ones tend to
> be pretty much rock solid reliable and have good performance (although I
> think they only do USB 3.0 adapters).
>>
>> Also what USB host chipset is on the PC motherboard ?
>
> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
> on-board hubs to the ports provided by whatever Intel calls their equivalent
> of the south bridge these days, and the USB 3.0 ports are probably a mix of
> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
> to do an inexpensive standalone XHCI chip, so they're relatively popular for
> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
> some issues with older Linux kernels, although IIRC those issues were along
> the lines of a port just disappearing after disconnecting whatever was
> connected to it.
>>
>> Why don't you run 64-bit Ubuntu on this core i7 ?
>
> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
> it can be proven that it does, then that is a serious bug that needs to be
> fixed).  That said, unless you have some desperate need to be running 32-bit
> only, you should seriously look into updating to a 64-bit version, your
> whole system should run faster, and Ubuntu has really good 32-bit
> compatibility in the 64-bit version (which is part of why it's popular as a
> support target for third party software like Steam).
>
>>
>>
>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
>>>
>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>
>>>>
>>>> Hello,
>>>> currently my computer freezes every several seconds for half a second
>>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>>> I have just one download happening on utorrent right now - this is
>>>> what the graph looks like:
>>>> http://i.imgur.com/LqhMtrJ.png
>>>> and every time a new spike happens, a freeze happens just before
>>>> that... that's the only time those freezes happen, too.
>>>>
>>> Do you have the 'autodefrag' mount option enabled?  If it is turned on,
>>> then
>>> that may be the problem.  Most bittorrent clients pre-allocate the space
>>> for
>>> a download, then write each block directly into the location it's
>>> supposed
>>> to be in the resultant download, which means depending on how it's
>>> pre-allocating the space, that you end up with a large number of randomly
>>> ordered writes into a single file, which in turn will trigger the
>>> autodefrag
>>> code, which can cause latency spikes when you're also hitting the disk at
>>> the same time.
>
>
>

Re: Bad fs performance, IO freezes

[cheater00 .]

If you can suggest a dual (or better yet quad) USB3 bay that can be
bought on Amazon, I'll buy it now, and once that arrives, we can be
sure it's not the JMicron chipset.

On Tue, Oct 27, 2015 at 3:22 PM, cheater00 . <cheater00@gmail.com> wrote:
> The (dual) HDD bay and the chipset are, according to lsusb:
> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
> USA Technology Corp.
> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>
> Not sure how to find out specific model numbers? I could open up the
> bay. OK I'll open up the bay.
> Good thing I have just the right screwdriver. It's a JMS551, and just
> for records sake, here's the manufacture info:
>
> JMS551
> 1120 LGAA2 A
> 572QV0024
>
> The laptop manual says it's either "Intel HM65 Express chipset with
> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
> Here are technical documents for my model:
> Manual: http://docdro.id/hG627JM
> "Intel chipset datasheet": http://docdro.id/yKRupYO
> Service guide: http://docdro.id/AuDgUdE
> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>
> FWIW I'm using one of the USB3 ports on the left. The ones on the
> right are USB2.
>
> I've never used docdro.id so if it's not good let me know where to
> upload the PDFs to.
>
> autodefrag is on, yes. But I have been having issues before turning it
> on - I turned it on as a measure towards fixing the issues. I will
> turn it off and remount, then report. But I don't think that should be
> it. As you see the transfer speeds are minimal. They're *all* that's
> happening on the disk. Right now that's under 100 KB/sec and I'm still
> getting freezes albeit less. Also why would I be getting freezes when
> the transfer speeds jump up - just for them to drop again? Hmm, maybe
> utorrent has some sort of scheduler that gets preempted while the
> spike is happening, and some algorithm in it gets the wrong idea and
> turns some sort of flow control on, because it thinks it's hit some
> sort of physical transfer speed barrier. Also notice the upload and
> download both scale together, but that just might be how torrent
> works, maybe it just tries to be fair i.e. only uploads as much as it
> downloaded (scaled by a constant).
>
> The system is 32 bit because I installed ubuntu 6 one day and just
> kept on upgrading. I keep on telling myself I'll update to 64 bits,
> one of these days. But this laptop only has 8 gigs of ram, so no real
> reason to upgrade to 64 bit anyways. It's not like I need firefox to
> be able to eat 8 gb of ram whereas right now it can only eat 4. There
> is no simple upgrade path that I know of so it's either a fresh
> install or doing something like this: http://www.ewan.cc/?q=node/132
> -- I keep telling myself /one of these days/...
>
> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
> <ahferroin7@gmail.com> wrote:
>> On 2015-10-27 09:00, Henk Slager wrote:
>>>
>>> I don't have a lot experience with autodefrag, but as indicated by
>>> Austin, expect a lot of full rewrites of files that are relatively
>>> slowly filled up by a torrent client, starting with a sparse file. So
>>> 1st advice would be to remove this option and run it as crontask at
>>> particular times.
>>>
>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>
>> I hadn't thought of this, but the specific adapter being used for the disk
>> can have a lot of impact on how it preforms.  I've personally had lots of
>> issues with JMicron chipsets (ranging from latency issues like what you are
>> seeing to sever data corruption), but have found that ASMedia ones tend to
>> be pretty much rock solid reliable and have good performance (although I
>> think they only do USB 3.0 adapters).
>>>
>>> Also what USB host chipset is on the PC motherboard ?
>>
>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>> some issues with older Linux kernels, although IIRC those issues were along
>> the lines of a port just disappearing after disconnecting whatever was
>> connected to it.
>>>
>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>
>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>> it can be proven that it does, then that is a serious bug that needs to be
>> fixed).  That said, unless you have some desperate need to be running 32-bit
>> only, you should seriously look into updating to a 64-bit version, your
>> whole system should run faster, and Ubuntu has really good 32-bit
>> compatibility in the 64-bit version (which is part of why it's popular as a
>> support target for third party software like Steam).
>>
>>>
>>>
>>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>>>>
>>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>>
>>>>>
>>>>> Hello,
>>>>> currently my computer freezes every several seconds for half a second
>>>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>>>> I have just one download happening on utorrent right now - this is
>>>>> what the graph looks like:
>>>>> http://i.imgur.com/LqhMtrJ.png
>>>>> and every time a new spike happens, a freeze happens just before
>>>>> that... that's the only time those freezes happen, too.
>>>>>
>>>> Do you have the 'autodefrag' mount option enabled?  If it is turned on,
>>>> then
>>>> that may be the problem.  Most bittorrent clients pre-allocate the space
>>>> for
>>>> a download, then write each block directly into the location it's
>>>> supposed
>>>> to be in the resultant download, which means depending on how it's
>>>> pre-allocating the space, that you end up with a large number of randomly
>>>> ordered writes into a single file, which in turn will trigger the
>>>> autodefrag
>>>> code, which can cause latency spikes when you're also hitting the disk at
>>>> the same time.
>>
>>
>>

Re: Bad fs performance, IO freezes

[cheater00 .]

Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
the unknowns.

On Tue, Oct 27, 2015 at 3:26 PM, cheater00 . <cheater00@gmail.com> wrote:
> If you can suggest a dual (or better yet quad) USB3 bay that can be
> bought on Amazon, I'll buy it now, and once that arrives, we can be
> sure it's not the JMicron chipset.
>
> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 . <cheater00@gmail.com> wrote:
>> The (dual) HDD bay and the chipset are, according to lsusb:
>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>> USA Technology Corp.
>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>
>> Not sure how to find out specific model numbers? I could open up the
>> bay. OK I'll open up the bay.
>> Good thing I have just the right screwdriver. It's a JMS551, and just
>> for records sake, here's the manufacture info:
>>
>> JMS551
>> 1120 LGAA2 A
>> 572QV0024
>>
>> The laptop manual says it's either "Intel HM65 Express chipset with
>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>> Here are technical documents for my model:
>> Manual: http://docdro.id/hG627JM
>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>> Service guide: http://docdro.id/AuDgUdE
>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>>
>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>> right are USB2.
>>
>> I've never used docdro.id so if it's not good let me know where to
>> upload the PDFs to.
>>
>> autodefrag is on, yes. But I have been having issues before turning it
>> on - I turned it on as a measure towards fixing the issues. I will
>> turn it off and remount, then report. But I don't think that should be
>> it. As you see the transfer speeds are minimal. They're *all* that's
>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>> getting freezes albeit less. Also why would I be getting freezes when
>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>> utorrent has some sort of scheduler that gets preempted while the
>> spike is happening, and some algorithm in it gets the wrong idea and
>> turns some sort of flow control on, because it thinks it's hit some
>> sort of physical transfer speed barrier. Also notice the upload and
>> download both scale together, but that just might be how torrent
>> works, maybe it just tries to be fair i.e. only uploads as much as it
>> downloaded (scaled by a constant).
>>
>> The system is 32 bit because I installed ubuntu 6 one day and just
>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>> one of these days. But this laptop only has 8 gigs of ram, so no real
>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>> is no simple upgrade path that I know of so it's either a fresh
>> install or doing something like this: http://www.ewan.cc/?q=node/132
>> -- I keep telling myself /one of these days/...
>>
>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
>>> On 2015-10-27 09:00, Henk Slager wrote:
>>>>
>>>> I don't have a lot experience with autodefrag, but as indicated by
>>>> Austin, expect a lot of full rewrites of files that are relatively
>>>> slowly filled up by a torrent client, starting with a sparse file. So
>>>> 1st advice would be to remove this option and run it as crontask at
>>>> particular times.
>>>>
>>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>>
>>> I hadn't thought of this, but the specific adapter being used for the disk
>>> can have a lot of impact on how it preforms.  I've personally had lots of
>>> issues with JMicron chipsets (ranging from latency issues like what you are
>>> seeing to sever data corruption), but have found that ASMedia ones tend to
>>> be pretty much rock solid reliable and have good performance (although I
>>> think they only do USB 3.0 adapters).
>>>>
>>>> Also what USB host chipset is on the PC motherboard ?
>>>
>>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>>> some issues with older Linux kernels, although IIRC those issues were along
>>> the lines of a port just disappearing after disconnecting whatever was
>>> connected to it.
>>>>
>>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>>
>>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>>> it can be proven that it does, then that is a serious bug that needs to be
>>> fixed).  That said, unless you have some desperate need to be running 32-bit
>>> only, you should seriously look into updating to a 64-bit version, your
>>> whole system should run faster, and Ubuntu has really good 32-bit
>>> compatibility in the 64-bit version (which is part of why it's popular as a
>>> support target for third party software like Steam).
>>>
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>>> <ahferroin7@gmail.com> wrote:
>>>>>
>>>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>>>
>>>>>>
>>>>>> Hello,
>>>>>> currently my computer freezes every several seconds for half a second
>>>>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>>>>> I have just one download happening on utorrent right now - this is
>>>>>> what the graph looks like:
>>>>>> http://i.imgur.com/LqhMtrJ.png
>>>>>> and every time a new spike happens, a freeze happens just before
>>>>>> that... that's the only time those freezes happen, too.
>>>>>>
>>>>> Do you have the 'autodefrag' mount option enabled?  If it is turned on,
>>>>> then
>>>>> that may be the problem.  Most bittorrent clients pre-allocate the space
>>>>> for
>>>>> a download, then write each block directly into the location it's
>>>>> supposed
>>>>> to be in the resultant download, which means depending on how it's
>>>>> pre-allocating the space, that you end up with a large number of randomly
>>>>> ordered writes into a single file, which in turn will trigger the
>>>>> autodefrag
>>>>> code, which can cause latency spikes when you're also hitting the disk at
>>>>> the same time.
>>>
>>>
>>>

Re: Bad fs performance, IO freezes

[cheater00 .]

I have remounted without autodefrag and the issue keeps on happening.

On Tue, Oct 27, 2015 at 3:30 PM, cheater00 . <cheater00@gmail.com> wrote:
> Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
> the unknowns.
>
> On Tue, Oct 27, 2015 at 3:26 PM, cheater00 . <cheater00@gmail.com> wrote:
>> If you can suggest a dual (or better yet quad) USB3 bay that can be
>> bought on Amazon, I'll buy it now, and once that arrives, we can be
>> sure it's not the JMicron chipset.
>>
>> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 . <cheater00@gmail.com> wrote:
>>> The (dual) HDD bay and the chipset are, according to lsusb:
>>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>>> USA Technology Corp.
>>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>>
>>> Not sure how to find out specific model numbers? I could open up the
>>> bay. OK I'll open up the bay.
>>> Good thing I have just the right screwdriver. It's a JMS551, and just
>>> for records sake, here's the manufacture info:
>>>
>>> JMS551
>>> 1120 LGAA2 A
>>> 572QV0024
>>>
>>> The laptop manual says it's either "Intel HM65 Express chipset with
>>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>>> Here are technical documents for my model:
>>> Manual: http://docdro.id/hG627JM
>>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>>> Service guide: http://docdro.id/AuDgUdE
>>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>>>
>>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>>> right are USB2.
>>>
>>> I've never used docdro.id so if it's not good let me know where to
>>> upload the PDFs to.
>>>
>>> autodefrag is on, yes. But I have been having issues before turning it
>>> on - I turned it on as a measure towards fixing the issues. I will
>>> turn it off and remount, then report. But I don't think that should be
>>> it. As you see the transfer speeds are minimal. They're *all* that's
>>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>>> getting freezes albeit less. Also why would I be getting freezes when
>>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>>> utorrent has some sort of scheduler that gets preempted while the
>>> spike is happening, and some algorithm in it gets the wrong idea and
>>> turns some sort of flow control on, because it thinks it's hit some
>>> sort of physical transfer speed barrier. Also notice the upload and
>>> download both scale together, but that just might be how torrent
>>> works, maybe it just tries to be fair i.e. only uploads as much as it
>>> downloaded (scaled by a constant).
>>>
>>> The system is 32 bit because I installed ubuntu 6 one day and just
>>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>>> one of these days. But this laptop only has 8 gigs of ram, so no real
>>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>>> is no simple upgrade path that I know of so it's either a fresh
>>> install or doing something like this: http://www.ewan.cc/?q=node/132
>>> -- I keep telling myself /one of these days/...
>>>
>>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>>> <ahferroin7@gmail.com> wrote:
>>>> On 2015-10-27 09:00, Henk Slager wrote:
>>>>>
>>>>> I don't have a lot experience with autodefrag, but as indicated by
>>>>> Austin, expect a lot of full rewrites of files that are relatively
>>>>> slowly filled up by a torrent client, starting with a sparse file. So
>>>>> 1st advice would be to remove this option and run it as crontask at
>>>>> particular times.
>>>>>
>>>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>>>
>>>> I hadn't thought of this, but the specific adapter being used for the disk
>>>> can have a lot of impact on how it preforms.  I've personally had lots of
>>>> issues with JMicron chipsets (ranging from latency issues like what you are
>>>> seeing to sever data corruption), but have found that ASMedia ones tend to
>>>> be pretty much rock solid reliable and have good performance (although I
>>>> think they only do USB 3.0 adapters).
>>>>>
>>>>> Also what USB host chipset is on the PC motherboard ?
>>>>
>>>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>>>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>>>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>>>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>>>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>>>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>>>> some issues with older Linux kernels, although IIRC those issues were along
>>>> the lines of a port just disappearing after disconnecting whatever was
>>>> connected to it.
>>>>>
>>>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>>>
>>>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>>>> it can be proven that it does, then that is a serious bug that needs to be
>>>> fixed).  That said, unless you have some desperate need to be running 32-bit
>>>> only, you should seriously look into updating to a 64-bit version, your
>>>> whole system should run faster, and Ubuntu has really good 32-bit
>>>> compatibility in the 64-bit version (which is part of why it's popular as a
>>>> support target for third party software like Steam).
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>>>> <ahferroin7@gmail.com> wrote:
>>>>>>
>>>>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>> currently my computer freezes every several seconds for half a second
>>>>>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>>>>>> I have just one download happening on utorrent right now - this is
>>>>>>> what the graph looks like:
>>>>>>> http://i.imgur.com/LqhMtrJ.png
>>>>>>> and every time a new spike happens, a freeze happens just before
>>>>>>> that... that's the only time those freezes happen, too.
>>>>>>>
>>>>>> Do you have the 'autodefrag' mount option enabled?  If it is turned on,
>>>>>> then
>>>>>> that may be the problem.  Most bittorrent clients pre-allocate the space
>>>>>> for
>>>>>> a download, then write each block directly into the location it's
>>>>>> supposed
>>>>>> to be in the resultant download, which means depending on how it's
>>>>>> pre-allocating the space, that you end up with a large number of randomly
>>>>>> ordered writes into a single file, which in turn will trigger the
>>>>>> autodefrag
>>>>>> code, which can cause latency spikes when you're also hitting the disk at
>>>>>> the same time.
>>>>
>>>>
>>>>

Re: Bad fs performance, IO freezes

[Holger Hoffstätte]

On 10/27/15 15:43, cheater00 . wrote:
> I have remounted without autodefrag and the issue keeps on happening.

At the risk of stating the obvious, the simplest thing is to stop using
COW with compression for torrents. It's fundamentally not useful to have
many small in-place writes, irregular compression and not encounter insane
levels of fragmentation, which - at least in the current state of btrfs -
is known to cause long stalls. Some fixes for that will be in 4.4.

Until then just do yourself a favor, stop doing the wrong thing and put
torrent downloads into a directory where they won't get COWed.

See the wiki at:
https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
for more.

-h

Re: Bad fs performance, IO freezes

[cheater00 .]

I'll just add nodatacow to the fstab, because the whole disk is for downloads.
There actually is compression happening?

On Tue, Oct 27, 2015 at 4:01 PM, Holger Hoffstätte
<holger.hoffstaette@googlemail.com> wrote:
> On 10/27/15 15:43, cheater00 . wrote:
>> I have remounted without autodefrag and the issue keeps on happening.
>
> At the risk of stating the obvious, the simplest thing is to stop using
> COW with compression for torrents. It's fundamentally not useful to have
> many small in-place writes, irregular compression and not encounter insane
> levels of fragmentation, which - at least in the current state of btrfs -
> is known to cause long stalls. Some fixes for that will be in 4.4.
>
> Until then just do yourself a favor, stop doing the wrong thing and put
> torrent downloads into a directory where they won't get COWed.
>
> See the wiki at:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
> for more.
>
> -h
>

Re: Bad fs performance, IO freezes

[cheater00 .]

Can I have nodatacow but still have checksumming?

On Tue, Oct 27, 2015 at 4:05 PM, cheater00 . <cheater00@gmail.com> wrote:
> I'll just add nodatacow to the fstab, because the whole disk is for downloads.
> There actually is compression happening?
>
> On Tue, Oct 27, 2015 at 4:01 PM, Holger Hoffstätte
> <holger.hoffstaette@googlemail.com> wrote:
>> On 10/27/15 15:43, cheater00 . wrote:
>>> I have remounted without autodefrag and the issue keeps on happening.
>>
>> At the risk of stating the obvious, the simplest thing is to stop using
>> COW with compression for torrents. It's fundamentally not useful to have
>> many small in-place writes, irregular compression and not encounter insane
>> levels of fragmentation, which - at least in the current state of btrfs -
>> is known to cause long stalls. Some fixes for that will be in 4.4.
>>
>> Until then just do yourself a favor, stop doing the wrong thing and put
>> torrent downloads into a directory where they won't get COWed.
>>
>> See the wiki at:
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Can_copy-on-write_be_turned_off_for_data_blocks.3F
>> for more.
>>
>> -h
>>

Re: Bad fs performance, IO freezes

[Holger Hoffstätte]

On 10/27/15 16:07, cheater00 . wrote:
> Can I have nodatacow but still have checksumming?

No, see "nodatacow":
https://btrfs.wiki.kernel.org/index.php/Mount_options

Torrents are typically rehashed both on a per-block basis and after
completion. That won't protect you from bitflips or fs corruption, but
then again you don't have data redundancy anyway, so it doesn't really
matter.

Also any silently bitflipped blocks will be rejected by peers since
they don't match they block hash, so even in that case you won't be
distributing corrupt data.

-h

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 9889 bytes --]

On 2015-10-27 10:43, cheater00 . wrote:
> I have remounted without autodefrag and the issue keeps on happening.
OK, that at least narrows things down further.  My guess is the spikes 
are utorrent getting a bunch of blocks at once from one place, and then 
trying to write all of them at the same time, which could theoretically 
cause a latency spike on any filesystem, and BTRFS may just be making it 
worse.
>
> On Tue, Oct 27, 2015 at 3:30 PM, cheater00 . <cheater00@gmail.com> wrote:
>> Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
>> the unknowns.
When it comes to external cables, I've had really good success with 
Amazon's 'Amazon Basics' branded stuff.  It's usually some of the best 
quality you can find for the price.  The 'Cable Matters' and 'Pluggable' 
brands also tend to be really good quality for the price.
>> On Tue, Oct 27, 2015 at 3:26 PM, cheater00 . <cheater00@gmail.com> wrote:
>>> If you can suggest a dual (or better yet quad) USB3 bay that can be
>>> bought on Amazon, I'll buy it now, and once that arrives, we can be
>>> sure it's not the JMicron chipset.
I don't really have any suggestions here.  Usually when I hook up an 
external drive, it's to recover data from a friends computer, so I don't 
typically use a enclosure, but just use a simple adapter cable.  I would 
suggest looking for one advertising 'UAS' or 'UASP' support, as that's a 
relatively new standard for USB storage devices, and newer hardware 
should be more reliable.  It's also notoriously hard to determine what 
chipset a given model of external drive bay has (there are people I know 
who bought multiples of the same model and each one had a different 
chipset internally), and to complicate matters, quite often the exact 
same hardware gets marketed under half a dozen different names.  JMicron 
is popular because their chips are comparatively inexpensive, and while 
I've not had good results with them, that doesn't mean that they are all 
bad (especially considering that they are highly configurable based on 
how they are wired into the device, and not everyone who designs 
hardware around them properly understands the implications of some of 
the features).
>>>
>>> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 . <cheater00@gmail.com> wrote:
>>>> The (dual) HDD bay and the chipset are, according to lsusb:
>>>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>>>> USA Technology Corp.
>>>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>>>
>>>> Not sure how to find out specific model numbers? I could open up the
>>>> bay. OK I'll open up the bay.
>>>> Good thing I have just the right screwdriver. It's a JMS551, and just
>>>> for records sake, here's the manufacture info:
>>>>
>>>> JMS551
>>>> 1120 LGAA2 A
>>>> 572QV0024
>>>>
>>>> The laptop manual says it's either "Intel HM65 Express chipset with
>>>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>>>> Here are technical documents for my model:
>>>> Manual: http://docdro.id/hG627JM
>>>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>>>> Service guide: http://docdro.id/AuDgUdE
>>>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
 From what I can tell, you've got the one with the NEC USB 3.0 chip, I'm 
pretty cure that the HM65 doesn't have USB 3.0 itself.  FWIW, I've never 
personally had issues with NEC's USB 3.0 chips, but I've not had much 
experience using systems with them either.
>>>>
>>>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>>>> right are USB2.
>>>>
>>>> I've never used docdro.id so if it's not good let me know where to
>>>> upload the PDFs to.
>>>>
>>>> autodefrag is on, yes. But I have been having issues before turning it
>>>> on - I turned it on as a measure towards fixing the issues. I will
>>>> turn it off and remount, then report. But I don't think that should be
>>>> it. As you see the transfer speeds are minimal. They're *all* that's
>>>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>>>> getting freezes albeit less. Also why would I be getting freezes when
>>>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>>>> utorrent has some sort of scheduler that gets preempted while the
>>>> spike is happening, and some algorithm in it gets the wrong idea and
>>>> turns some sort of flow control on, because it thinks it's hit some
>>>> sort of physical transfer speed barrier. Also notice the upload and
>>>> download both scale together, but that just might be how torrent
>>>> works, maybe it just tries to be fair i.e. only uploads as much as it
>>>> downloaded (scaled by a constant).
Yeah, utorrent defaults to trying for a 1:1 ratio of uploads to 
downloads (so in terms of viewing the group of clients downloading a 
torrent as a network, it defaults to contributing as much bandwidth as 
it uses).  This is pretty typical behavior for most torrent clients, and 
in fact downloading more than you upload for a torrent is generally 
considered bad etiquette.
>>>>
>>>> The system is 32 bit because I installed ubuntu 6 one day and just
>>>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>>>> one of these days. But this laptop only has 8 gigs of ram, so no real
>>>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>>>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>>>> is no simple upgrade path that I know of so it's either a fresh
>>>> install or doing something like this: http://www.ewan.cc/?q=node/132
>>>> -- I keep telling myself /one of these days/...
That's entirely understandable.  It's never been easy to do an in-place 
upgrade from 32 to 64 bit.  It's worth noting that while it's not easy 
to do a full upgrade to 64-bit, it is relatively easy to run a 32-bit 
userspace on a 64-bit kernel (at least, it should be, it's been so long 
since I used Ubuntu for anything that I really don't have much frame of 
reference regarding it beyond the fact that it's based on Debian).
>>>>
>>>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>>>> <ahferroin7@gmail.com> wrote:
>>>>> On 2015-10-27 09:00, Henk Slager wrote:
>>>>>>
>>>>>> I don't have a lot experience with autodefrag, but as indicated by
>>>>>> Austin, expect a lot of full rewrites of files that are relatively
>>>>>> slowly filled up by a torrent client, starting with a sparse file. So
>>>>>> 1st advice would be to remove this option and run it as crontask at
>>>>>> particular times.
>>>>>>
>>>>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>>>>
>>>>> I hadn't thought of this, but the specific adapter being used for the disk
>>>>> can have a lot of impact on how it preforms.  I've personally had lots of
>>>>> issues with JMicron chipsets (ranging from latency issues like what you are
>>>>> seeing to sever data corruption), but have found that ASMedia ones tend to
>>>>> be pretty much rock solid reliable and have good performance (although I
>>>>> think they only do USB 3.0 adapters).
>>>>>>
>>>>>> Also what USB host chipset is on the PC motherboard ?
>>>>>
>>>>> If it's a Intel motherboard, the USB 2.0 ports are probably routed through
>>>>> on-board hubs to the ports provided by whatever Intel calls their equivalent
>>>>> of the south bridge these days, and the USB 3.0 ports are probably a mix of
>>>>> Intel and ASMedia XHCI controllers (ASMedia was one of the first companies
>>>>> to do an inexpensive standalone XHCI chip, so they're relatively popular for
>>>>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips had
>>>>> some issues with older Linux kernels, although IIRC those issues were along
>>>>> the lines of a port just disappearing after disconnecting whatever was
>>>>> connected to it.
>>>>>>
>>>>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>>>>
>>>>> 64 versus 32 bit shouldn't cause anything like this to happen (although, if
>>>>> it can be proven that it does, then that is a serious bug that needs to be
>>>>> fixed).  That said, unless you have some desperate need to be running 32-bit
>>>>> only, you should seriously look into updating to a 64-bit version, your
>>>>> whole system should run faster, and Ubuntu has really good 32-bit
>>>>> compatibility in the 64-bit version (which is part of why it's popular as a
>>>>> support target for third party software like Steam).
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>>>>> <ahferroin7@gmail.com> wrote:
>>>>>>>
>>>>>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>> currently my computer freezes every several seconds for half a second
>>>>>>>> or so. Using it feels like I'm playing musical chairs with the kernel.
>>>>>>>> I have just one download happening on utorrent right now - this is
>>>>>>>> what the graph looks like:
>>>>>>>> http://i.imgur.com/LqhMtrJ.png
>>>>>>>> and every time a new spike happens, a freeze happens just before
>>>>>>>> that... that's the only time those freezes happen, too.
>>>>>>>>
>>>>>>> Do you have the 'autodefrag' mount option enabled?  If it is turned on,
>>>>>>> then
>>>>>>> that may be the problem.  Most bittorrent clients pre-allocate the space
>>>>>>> for
>>>>>>> a download, then write each block directly into the location it's
>>>>>>> supposed
>>>>>>> to be in the resultant download, which means depending on how it's
>>>>>>> pre-allocating the space, that you end up with a large number of randomly
>>>>>>> ordered writes into a single file, which in turn will trigger the
>>>>>>> autodefrag
>>>>>>> code, which can cause latency spikes when you're also hitting the disk at
>>>>>>> the same time.



[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Bad fs performance, IO freezes

[cheater00 .]

Hi Liu,
after talking with Holger I believe turning off COW on this FS will
work to alleviate this issue. However, even with COW on, btrfs
shouldn't be making my computer freeze every 5 seconds... especially
while the disk is written to at mere tens of kilobytes per second.
It's not even the disk holding the system. I consider this a pretty
bad bug... should we go on with trying to reproduce a minimum case?
How would I go about this?

Thanks

On Tue, Oct 27, 2015 at 4:26 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-10-27 10:43, cheater00 . wrote:
>>
>> I have remounted without autodefrag and the issue keeps on happening.
>
> OK, that at least narrows things down further.  My guess is the spikes are
> utorrent getting a bunch of blocks at once from one place, and then trying
> to write all of them at the same time, which could theoretically cause a
> latency spike on any filesystem, and BTRFS may just be making it worse.
>>
>>
>> On Tue, Oct 27, 2015 at 3:30 PM, cheater00 . <cheater00@gmail.com> wrote:
>>>
>>> Feel free to suggest a good 1.5m USB3 cable, too. Let's get rid of all
>>> the unknowns.
>
> When it comes to external cables, I've had really good success with Amazon's
> 'Amazon Basics' branded stuff.  It's usually some of the best quality you
> can find for the price.  The 'Cable Matters' and 'Pluggable' brands also
> tend to be really good quality for the price.
>>>
>>> On Tue, Oct 27, 2015 at 3:26 PM, cheater00 . <cheater00@gmail.com> wrote:
>>>>
>>>> If you can suggest a dual (or better yet quad) USB3 bay that can be
>>>> bought on Amazon, I'll buy it now, and once that arrives, we can be
>>>> sure it's not the JMicron chipset.
>
> I don't really have any suggestions here.  Usually when I hook up an
> external drive, it's to recover data from a friends computer, so I don't
> typically use a enclosure, but just use a simple adapter cable.  I would
> suggest looking for one advertising 'UAS' or 'UASP' support, as that's a
> relatively new standard for USB storage devices, and newer hardware should
> be more reliable.  It's also notoriously hard to determine what chipset a
> given model of external drive bay has (there are people I know who bought
> multiples of the same model and each one had a different chipset
> internally), and to complicate matters, quite often the exact same hardware
> gets marketed under half a dozen different names.  JMicron is popular
> because their chips are comparatively inexpensive, and while I've not had
> good results with them, that doesn't mean that they are all bad (especially
> considering that they are highly configurable based on how they are wired
> into the device, and not everyone who designs hardware around them properly
> understands the implications of some of the features).
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 3:22 PM, cheater00 . <cheater00@gmail.com>
>>>> wrote:
>>>>>
>>>>> The (dual) HDD bay and the chipset are, according to lsusb:
>>>>> Bus 002 Device 005: ID 152d:0551 JMicron Technology Corp. / JMicron
>>>>> USA Technology Corp.
>>>>> Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
>>>>>
>>>>> Not sure how to find out specific model numbers? I could open up the
>>>>> bay. OK I'll open up the bay.
>>>>> Good thing I have just the right screwdriver. It's a JMS551, and just
>>>>> for records sake, here's the manufacture info:
>>>>>
>>>>> JMS551
>>>>> 1120 LGAA2 A
>>>>> 572QV0024
>>>>>
>>>>> The laptop manual says it's either "Intel HM65 Express chipset with
>>>>> NEC USB 3.0 (select models only)" or "Intel HM65 Express chipset".
>>>>> Here are technical documents for my model:
>>>>> Manual: http://docdro.id/hG627JM
>>>>> "Intel chipset datasheet": http://docdro.id/yKRupYO
>>>>> Service guide: http://docdro.id/AuDgUdE
>>>>> Service guide, alt. ver.: http://docdro.id/WwQRpsH
>
> From what I can tell, you've got the one with the NEC USB 3.0 chip, I'm
> pretty cure that the HM65 doesn't have USB 3.0 itself.  FWIW, I've never
> personally had issues with NEC's USB 3.0 chips, but I've not had much
> experience using systems with them either.
>>>>>
>>>>>
>>>>> FWIW I'm using one of the USB3 ports on the left. The ones on the
>>>>> right are USB2.
>>>>>
>>>>> I've never used docdro.id so if it's not good let me know where to
>>>>> upload the PDFs to.
>>>>>
>>>>> autodefrag is on, yes. But I have been having issues before turning it
>>>>> on - I turned it on as a measure towards fixing the issues. I will
>>>>> turn it off and remount, then report. But I don't think that should be
>>>>> it. As you see the transfer speeds are minimal. They're *all* that's
>>>>> happening on the disk. Right now that's under 100 KB/sec and I'm still
>>>>> getting freezes albeit less. Also why would I be getting freezes when
>>>>> the transfer speeds jump up - just for them to drop again? Hmm, maybe
>>>>> utorrent has some sort of scheduler that gets preempted while the
>>>>> spike is happening, and some algorithm in it gets the wrong idea and
>>>>> turns some sort of flow control on, because it thinks it's hit some
>>>>> sort of physical transfer speed barrier. Also notice the upload and
>>>>> download both scale together, but that just might be how torrent
>>>>> works, maybe it just tries to be fair i.e. only uploads as much as it
>>>>> downloaded (scaled by a constant).
>
> Yeah, utorrent defaults to trying for a 1:1 ratio of uploads to downloads
> (so in terms of viewing the group of clients downloading a torrent as a
> network, it defaults to contributing as much bandwidth as it uses).  This is
> pretty typical behavior for most torrent clients, and in fact downloading
> more than you upload for a torrent is generally considered bad etiquette.
>>>>>
>>>>>
>>>>> The system is 32 bit because I installed ubuntu 6 one day and just
>>>>> kept on upgrading. I keep on telling myself I'll update to 64 bits,
>>>>> one of these days. But this laptop only has 8 gigs of ram, so no real
>>>>> reason to upgrade to 64 bit anyways. It's not like I need firefox to
>>>>> be able to eat 8 gb of ram whereas right now it can only eat 4. There
>>>>> is no simple upgrade path that I know of so it's either a fresh
>>>>> install or doing something like this: http://www.ewan.cc/?q=node/132
>>>>> -- I keep telling myself /one of these days/...
>
> That's entirely understandable.  It's never been easy to do an in-place
> upgrade from 32 to 64 bit.  It's worth noting that while it's not easy to do
> a full upgrade to 64-bit, it is relatively easy to run a 32-bit userspace on
> a 64-bit kernel (at least, it should be, it's been so long since I used
> Ubuntu for anything that I really don't have much frame of reference
> regarding it beyond the fact that it's based on Debian).
>
>>>>>
>>>>> On Tue, Oct 27, 2015 at 2:30 PM, Austin S Hemmelgarn
>>>>> <ahferroin7@gmail.com> wrote:
>>>>>>
>>>>>> On 2015-10-27 09:00, Henk Slager wrote:
>>>>>>>
>>>>>>>
>>>>>>> I don't have a lot experience with autodefrag, but as indicated by
>>>>>>> Austin, expect a lot of full rewrites of files that are relatively
>>>>>>> slowly filled up by a torrent client, starting with a sparse file. So
>>>>>>> 1st advice would be to remove this option and run it as crontask at
>>>>>>> particular times.
>>>>>>>
>>>>>>> What SATA-USB bridge is between the harddisk and the PC motherboard ?
>>>>>>
>>>>>>
>>>>>> I hadn't thought of this, but the specific adapter being used for the
>>>>>> disk
>>>>>> can have a lot of impact on how it preforms.  I've personally had lots
>>>>>> of
>>>>>> issues with JMicron chipsets (ranging from latency issues like what
>>>>>> you are
>>>>>> seeing to sever data corruption), but have found that ASMedia ones
>>>>>> tend to
>>>>>> be pretty much rock solid reliable and have good performance (although
>>>>>> I
>>>>>> think they only do USB 3.0 adapters).
>>>>>>>
>>>>>>>
>>>>>>> Also what USB host chipset is on the PC motherboard ?
>>>>>>
>>>>>>
>>>>>> If it's a Intel motherboard, the USB 2.0 ports are probably routed
>>>>>> through
>>>>>> on-board hubs to the ports provided by whatever Intel calls their
>>>>>> equivalent
>>>>>> of the south bridge these days, and the USB 3.0 ports are probably a
>>>>>> mix of
>>>>>> Intel and ASMedia XHCI controllers (ASMedia was one of the first
>>>>>> companies
>>>>>> to do an inexpensive standalone XHCI chip, so they're relatively
>>>>>> popular for
>>>>>> extra USB 3.0 ports).  FWIW, the first generation of Intel XHCI chips
>>>>>> had
>>>>>> some issues with older Linux kernels, although IIRC those issues were
>>>>>> along
>>>>>> the lines of a port just disappearing after disconnecting whatever was
>>>>>> connected to it.
>>>>>>>
>>>>>>>
>>>>>>> Why don't you run 64-bit Ubuntu on this core i7 ?
>>>>>>
>>>>>>
>>>>>> 64 versus 32 bit shouldn't cause anything like this to happen
>>>>>> (although, if
>>>>>> it can be proven that it does, then that is a serious bug that needs
>>>>>> to be
>>>>>> fixed).  That said, unless you have some desperate need to be running
>>>>>> 32-bit
>>>>>> only, you should seriously look into updating to a 64-bit version,
>>>>>> your
>>>>>> whole system should run faster, and Ubuntu has really good 32-bit
>>>>>> compatibility in the 64-bit version (which is part of why it's popular
>>>>>> as a
>>>>>> support target for third party software like Steam).
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 27, 2015 at 12:44 PM, Austin S Hemmelgarn
>>>>>>> <ahferroin7@gmail.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2015-10-26 22:00, cheater00 . wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>> currently my computer freezes every several seconds for half a
>>>>>>>>> second
>>>>>>>>> or so. Using it feels like I'm playing musical chairs with the
>>>>>>>>> kernel.
>>>>>>>>> I have just one download happening on utorrent right now - this is
>>>>>>>>> what the graph looks like:
>>>>>>>>> http://i.imgur.com/LqhMtrJ.png
>>>>>>>>> and every time a new spike happens, a freeze happens just before
>>>>>>>>> that... that's the only time those freezes happen, too.
>>>>>>>>>
>>>>>>>> Do you have the 'autodefrag' mount option enabled?  If it is turned
>>>>>>>> on,
>>>>>>>> then
>>>>>>>> that may be the problem.  Most bittorrent clients pre-allocate the
>>>>>>>> space
>>>>>>>> for
>>>>>>>> a download, then write each block directly into the location it's
>>>>>>>> supposed
>>>>>>>> to be in the resultant download, which means depending on how it's
>>>>>>>> pre-allocating the space, that you end up with a large number of
>>>>>>>> randomly
>>>>>>>> ordered writes into a single file, which in turn will trigger the
>>>>>>>> autodefrag
>>>>>>>> code, which can cause latency spikes when you're also hitting the
>>>>>>>> disk at
>>>>>>>> the same time.
>
>
>

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2955 bytes --]

On 2015-10-29 09:03, cheater00 . wrote:
> Hi Liu,
> after talking with Holger I believe turning off COW on this FS will
> work to alleviate this issue. However, even with COW on, btrfs
> shouldn't be making my computer freeze every 5 seconds... especially
> while the disk is written to at mere tens of kilobytes per second.
> It's not even the disk holding the system. I consider this a pretty
> bad bug... should we go on with trying to reproduce a minimum case?
> How would I go about this?

Well, COW can cause some pretty unexpected behavior for some use cases. 
  If you have a big disk (I think I remember you saying it was larger 
than 1TB), then COW can cause some pretty significant seek times because 
of how it works.  With the current state of BTRFS, I wouldn't personally 
consider running BTRFS on anything bigger than 256G with a non-zero seek 
time with COW turned on, because large rewrites would have the potential 
to cause horrifically long seek times just for a RMW cycle on a single 
block, and this is in turn part of why database files and 
virtual-machine images tend to be pathological use cases for BTRFS.

I do agree that this kind of thing is a bug, but it's not something that 
causes data corruption, which means that it is slightly lower priority 
as far as most people are concerned.  Reproducing it might be tricky 
also, because I'd be willing to bet that things get better to the point 
of it being almost unnoticeable with an internal disk (USB is horrible 
when it comes to block storage performance, and has all kinds of 
potential reliability issues).

Normally, when I try to go about reproducing something like this, I use 
a virtual machine running the most recent stable version of the Linux 
kernel, usually with a minimalistic Gentoo installation (although a 
clean install of pretty much any distro works fine).  There are a couple 
of reasons I use such a setup:
1. Using a clean install provides a well defined initial state, making 
it easier for other people to reproduce any results.
2. Using the most recent stable kernel available (usually) eliminates 
the chances of old bugs causing issues.
3. Using a VM means that your disk access will be slower, which will 
visibly accentuate any kind of performance issues.
4. Using a VM also means that it is very easy to safely generate crash 
dumps and simulate data corruption for testing purposes, and makes it 
easier to experiment with different parameters (for example, UP versus 
SMP, or different amounts of RAM).

If you do decide to go this route, my suggestion would be to use 
VirtualBox unless you have significant experience with some other 
hypervisor, as it's one of the easiest to learn to use (I usually use 
Xen or QEMU, but both require significant effort to set up initially, 
and are decidedly non-trivial to learn), and learning to debug stuff 
like this is itself not an easy task.


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Bad fs performance, IO freezes

[cheater00 .]

Hi Austin,
seek times are fine, but this literally freezes my computer for a
split second. I've had to re-type this email twice because the freezes
meant letters I typed would not arrive on the screen.
USB disks are so common they should not be having issues.
I have 4.3.0-040300rc7-generic #201510260712 which is just three days old.

Please advise. Isn't it better to *not* use a vm to debug this?
BTW, if we are talking about slow speed making things worse, I could
try downgrading the cable to usb2.
Is there a standard virtualbox VM that I could use?
I'll download Gentoo in the meantime. I have never used it. I'm
getting the "minimal installation cd" from 29th september.
http://distfiles.gentoo.org/releases/x86/autobuilds/20150929/install-x86-minimal-20150929.iso

On Thu, Oct 29, 2015 at 3:00 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-10-29 09:03, cheater00 . wrote:
>>
>> Hi Liu,
>> after talking with Holger I believe turning off COW on this FS will
>> work to alleviate this issue. However, even with COW on, btrfs
>> shouldn't be making my computer freeze every 5 seconds... especially
>> while the disk is written to at mere tens of kilobytes per second.
>> It's not even the disk holding the system. I consider this a pretty
>> bad bug... should we go on with trying to reproduce a minimum case?
>> How would I go about this?
>
>
> Well, COW can cause some pretty unexpected behavior for some use cases.  If
> you have a big disk (I think I remember you saying it was larger than 1TB),
> then COW can cause some pretty significant seek times because of how it
> works.  With the current state of BTRFS, I wouldn't personally consider
> running BTRFS on anything bigger than 256G with a non-zero seek time with
> COW turned on, because large rewrites would have the potential to cause
> horrifically long seek times just for a RMW cycle on a single block, and
> this is in turn part of why database files and virtual-machine images tend
> to be pathological use cases for BTRFS.
>
> I do agree that this kind of thing is a bug, but it's not something that
> causes data corruption, which means that it is slightly lower priority as
> far as most people are concerned.  Reproducing it might be tricky also,
> because I'd be willing to bet that things get better to the point of it
> being almost unnoticeable with an internal disk (USB is horrible when it
> comes to block storage performance, and has all kinds of potential
> reliability issues).
>
> Normally, when I try to go about reproducing something like this, I use a
> virtual machine running the most recent stable version of the Linux kernel,
> usually with a minimalistic Gentoo installation (although a clean install of
> pretty much any distro works fine).  There are a couple of reasons I use
> such a setup:
> 1. Using a clean install provides a well defined initial state, making it
> easier for other people to reproduce any results.
> 2. Using the most recent stable kernel available (usually) eliminates the
> chances of old bugs causing issues.
> 3. Using a VM means that your disk access will be slower, which will visibly
> accentuate any kind of performance issues.
> 4. Using a VM also means that it is very easy to safely generate crash dumps
> and simulate data corruption for testing purposes, and makes it easier to
> experiment with different parameters (for example, UP versus SMP, or
> different amounts of RAM).
>
> If you do decide to go this route, my suggestion would be to use VirtualBox
> unless you have significant experience with some other hypervisor, as it's
> one of the easiest to learn to use (I usually use Xen or QEMU, but both
> require significant effort to set up initially, and are decidedly
> non-trivial to learn), and learning to debug stuff like this is itself not
> an easy task.
>

Re: Bad fs performance, IO freezes

[Henk Slager]

The graph uploaded shows 'Up speed' and 'Down speed', I just assumed
that this is network speed and not disk I/O. Is this correct
assumption? If so, how does the traffic to/from the /dev/sdX look like
(e.g. iostat or ksysguard) ?

W.r.t. USB: I had quite some trouble with NEC/Renesas USB3 host
controller 2 years back. I got it working once under windows7 after
quite some drivers version trials. Under linux it is listed, but I
don't use is anymore; On the same PC under windows7 it doesn't work
anymore. For btrfs on WD elements and Sandisk Extreme on ASUS H87M-Pro
with kernels 3.x kernels (64-bit) I got similar freezes/perfomance
issues as mentioned here. The same fs configurations and tests on sata
cabled (2TB HDD) did not show these hickups. I have now mostly ext4 on
those USB connected disk, so with 4.3-rcX kernels I cant tell more.

There might be many configuration issues if a drive is connected via
removable USB, like writeback-caching etc. I have not looked into all
possible issues any further.
A month ago I got quite some random I/O and timeout errors on a 4TB
raw dd_rescue copy action (1 HDD in USB3 bay), kernel 4.1.6. Then
hooked-up all 3 disks, including rootfs, to another motherboard via
sata and not a single error. On the other hand, I have also a 2TB disk
connected via USB2 formatted btrfs and online 24/7 for over a year and
it works fine.

So I would (temporary) connect this WB 6TB via sata and use some
latest 64-bit liveCD/DVD linux distro and see how the disktraffic is
for some file copy or defrag action. And then, step by step go back to
your original configuration. Also make sure the disk is not filled
much more than 95%, as that could easily lead to the situation that
your latest active files will easily be very scattered, so not
beneficial for fs performance.

Re: Bad fs performance, IO freezes

[Austin S Hemmelgarn]

[-- Attachment #1: Type: text/plain, Size: 2606 bytes --]

On 2015-10-29 11:49, cheater00 . wrote:
> Hi Austin,
> seek times are fine, but this literally freezes my computer for a
> split second. I've had to re-type this email twice because the freezes
> meant letters I typed would not arrive on the screen.
> USB disks are so common they should not be having issues.
That's debatable.  USB is commonly used because it's almost impossible 
to find a system that doesn't have it, not because it's reliable.  The 
original intent was for it to be used for stuff like mice and keyboards, 
so it was designed with low-latency and fair scheduling in mind, both of 
which really hurt performance of bulk data storage devices.
> I have 4.3.0-040300rc7-generic #201510260712 which is just three days old.
That should be perfectly recent enough, although FWIW, the official 
version of 4.3 should be out this Sunday.
>
> Please advise. Isn't it better to *not* use a vm to debug this?
That depends.  For something like this, it could go either way.  I just 
use a VM because that's what I always use, because it's nice not 
crashing your system when trying to debug a kernel panic.
> BTW, if we are talking about slow speed making things worse, I could
> try downgrading the cable to usb2.
> Is there a standard virtualbox VM that I could use?
In general, it's pretty easy to set something like Ubuntu up in 
VirtualBox, the install is essentially identical to regular hardware 
aside from the initial setup of the VM itself.  The documentation for 
VirtualBox is really good, if you've never used virtualization before, 
it's definitely worth reading.
> I'll download Gentoo in the meantime. I have never used it. I'm
> getting the "minimal installation cd" from 29th september.
> http://distfiles.gentoo.org/releases/x86/autobuilds/20150929/install-x86-minimal-20150929.iso
I meant by no means that you needed to use Gentoo, I only mentioned it 
because it's what I use (which in turn is because that's what I use on 
just about everything except stuff like the Raspberry Pi or the 
BeagleBoard).  If you just want to debug this and then be done with it, 
I would actually advise against using Gentoo, it takes a lot of effort 
to get a system up and running with it, and it's very involved to 
maintain compared to Ubuntu.  On the other hand though, if you are 
willing to learn to use it, it's one of the most highly customizable 
Linux distros out there, and can have noticeably better performance than 
more generic distros (FWIW, it's also one of the last big distros that 
doesn't force systemd on it's users by default).


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]

Re: Bad fs performance, IO freezes

[cheater00 .]

I am getting a sata dock for my laptop next week. Until then, is it
possible to perform an action in btrfs (like rm which seems to trigger
the issue) and make it log what exactly it's doing?


On Thu, Oct 29, 2015 at 9:01 PM, Austin S Hemmelgarn
<ahferroin7@gmail.com> wrote:
> On 2015-10-29 11:49, cheater00 . wrote:
>>
>> Hi Austin,
>> seek times are fine, but this literally freezes my computer for a
>> split second. I've had to re-type this email twice because the freezes
>> meant letters I typed would not arrive on the screen.
>> USB disks are so common they should not be having issues.
>
> That's debatable.  USB is commonly used because it's almost impossible to
> find a system that doesn't have it, not because it's reliable.  The original
> intent was for it to be used for stuff like mice and keyboards, so it was
> designed with low-latency and fair scheduling in mind, both of which really
> hurt performance of bulk data storage devices.
>>
>> I have 4.3.0-040300rc7-generic #201510260712 which is just three days old.
>
> That should be perfectly recent enough, although FWIW, the official version
> of 4.3 should be out this Sunday.
>>
>>
>> Please advise. Isn't it better to *not* use a vm to debug this?
>
> That depends.  For something like this, it could go either way.  I just use
> a VM because that's what I always use, because it's nice not crashing your
> system when trying to debug a kernel panic.
>>
>> BTW, if we are talking about slow speed making things worse, I could
>> try downgrading the cable to usb2.
>> Is there a standard virtualbox VM that I could use?
>
> In general, it's pretty easy to set something like Ubuntu up in VirtualBox,
> the install is essentially identical to regular hardware aside from the
> initial setup of the VM itself.  The documentation for VirtualBox is really
> good, if you've never used virtualization before, it's definitely worth
> reading.
>>
>> I'll download Gentoo in the meantime. I have never used it. I'm
>> getting the "minimal installation cd" from 29th september.
>>
>> http://distfiles.gentoo.org/releases/x86/autobuilds/20150929/install-x86-minimal-20150929.iso
>
> I meant by no means that you needed to use Gentoo, I only mentioned it
> because it's what I use (which in turn is because that's what I use on just
> about everything except stuff like the Raspberry Pi or the BeagleBoard).  If
> you just want to debug this and then be done with it, I would actually
> advise against using Gentoo, it takes a lot of effort to get a system up and
> running with it, and it's very involved to maintain compared to Ubuntu.  On
> the other hand though, if you are willing to learn to use it, it's one of
> the most highly customizable Linux distros out there, and can have
> noticeably better performance than more generic distros (FWIW, it's also one
> of the last big distros that doesn't force systemd on it's users by
> default).
>