BitTorrent+Reiser4: curiouser and curiouser

BitTorrent+Reiser4: curiouser and curiouser

[David Masover]

Azureus had a problem.  Once it got up to a good clip downloading, it 
would thrash the disk.  It would thrash the disk, and the system, so 
hard that even web browsing was difficult, due to disk access being 
many, many times slower than Internet access, even an Internet which is 
being hogged by BitTorrent.

After changing Azureus' cache to 32 megs, and telling it not to write 
files immediately, I thought I had the problem solved -- no thrashing at 
all.  Until the cache got full.  Then:  Thrashing.  Less freqent, but 
much more vigorous -- Azureus becomes extremely unresponsive for a few 
minutes.

It shouldn't be touching the disk AT ALL when there's over a gig of FREE 
RAM (as in, neither buffer nor cache nor actually used yet), and the 
file I'm attempting to download is less than 200 megs.  I tried an 
strace, but as I am not at all skilled in the ways of debugging or 
reverse engineering, I got syscall spam -- a 200 meg log file, and when 
I finally found a decent way to analyze it, I found most of Azureus' 
system call wall time is spent in futex().  Huh?

Looked up "futex" on Wikipedia, and I still have no clue how this makes 
any sense.  Either futex was somehow thrashing the disk, or Azureus has 
somehow managed to fork completely out of strace's control.  Or maybe 
it's somehow something that the kernel is doing on its own, which is 
somehow forcing azureus to block, but somehow not tripping strace's 
timers while doing so.

This problem did not always happen with my Reiser4, but unfortunately, I 
can't pin down exactly when it started doing this.  It might have been a 
kernel upgrade, a Reiser4 upgrade, or an Azureus upgrade.

Here's the catch, though -- when I finally tried another client 
(BitTornado, on the same file), I have had absolutely no thrashing yet. 
  It's hardly touched the disk.  I was thinking maybe Azureus synced 
somehow, and BT didn't, but running "sync" on the commandline took about 
2 seconds.  Which means that, with BitTornado, everything works exactly 
the way it's supposed to.

So I'm happy it works, but I'm still curious why Azureus thrashed so 
much, and BitTornado doesn't thrash at all.  Maybe it's the apps?  Or 
Python vs Java?  Or maybe it's something like Evolution and column 
resizing -- something so embarrassingly, retardedly inefficient as 
flushing the column width information to disk every couple of pixels, 
that went unnoticed for so long because fsync performs well enough on 
other filesystems.

That's what it seems like to me, but one thing's sure -- it is neither 
fsync nor fdatasync.  I've disabled those at the kernel level.  I've 
still got no clue as to what it is, but I'll be glad to be rid of 
Azureus just as soon as I can actually find the features I like from it 
in other BitTorrent clients.

Re: BitTorrent+Reiser4: curiouser and curiouser

[Alexander Zarochentsev]

Hi,

On 22 September 2006 10:28, David Masover wrote:
> Azureus had a problem.  Once it got up to a good clip downloading, it
> would thrash the disk.  It would thrash the disk, and the system, so
> hard that even web browsing was difficult, due to disk access being
> many, many times slower than Internet access, even an Internet which
> is being hogged by BitTorrent.
>
> After changing Azureus' cache to 32 megs, and telling it not to write
> files immediately, I thought I had the problem solved -- no thrashing
> at all.  Until the cache got full.  Then:  Thrashing.  Less freqent,
> but much more vigorous -- Azureus becomes extremely unresponsive for
> a few minutes.
>
> It shouldn't be touching the disk AT ALL when there's over a gig of
> FREE RAM (as in, neither buffer nor cache nor actually used yet), and
> the file I'm attempting to download is less than 200 megs.  I tried
> an strace, but as I am not at all skilled in the ways of debugging or
> reverse engineering, I got syscall spam -- a 200 meg log file, and
> when I finally found a decent way to analyze it, I found most of
> Azureus' system call wall time is spent in futex().  Huh?

I guess futex (, FUTEX_WAIT, ) calls can be ignored in this analysis. 
They just wait another threat to call futex(, FUTEX_WAKE, ).  
Interesting to find that thread and look what it was doing before 
FUTEX_WAKE? Or FUTEX_WAIT returns ETIMEDOUT?

> Looked up "futex" on Wikipedia, and I still have no clue how this
> makes any sense.  Either futex was somehow thrashing the disk, or
> Azureus has somehow managed to fork completely out of strace's
> control.  Or maybe it's somehow something that the kernel is doing on
> its own, which is somehow forcing azureus to block, but somehow not
> tripping strace's timers while doing so.
>

[ ... ]

-- 
Alex.

Re: BitTorrent+Reiser4: curiouser and curiouser

[Konstantin Münning]

David Masover wrote:

(snip)
> It shouldn't be touching the disk AT ALL when there's over a gig of FREE
> RAM (as in, neither buffer nor cache nor actually used yet), and the
> file I'm attempting to download is less than 200 megs.  I tried an
> strace, but as I am not at all skilled in the ways of debugging or
> reverse engineering, I got syscall spam -- a 200 meg log file, and when
> I finally found a decent way to analyze it, I found most of Azureus'
> system call wall time is spent in futex().  Huh?
> 
> Looked up "futex" on Wikipedia, and I still have no clue how this makes
> any sense.  Either futex was somehow thrashing the disk, or Azureus has
> somehow managed to fork completely out of strace's control.  Or maybe
> it's somehow something that the kernel is doing on its own, which is
> somehow forcing azureus to block, but somehow not tripping strace's
> timers while doing so.

Have you used -f or -ff with strace? Without it you would see only the
initial process and not the forked processes. Having the futex call
indicates that there should be child processes, so -f or -ff is a must.

Just my 2 cents.
-- 
Konstantin Münning

Re: BitTorrent+Reiser4: curiouser and curiouser

[David Masover]

Konstantin Münning wrote:
> David Masover wrote:
> 
> (snip)
>> It shouldn't be touching the disk AT ALL when there's over a gig of FREE
>> RAM (as in, neither buffer nor cache nor actually used yet), and the
>> file I'm attempting to download is less than 200 megs.  I tried an
>> strace, but as I am not at all skilled in the ways of debugging or
>> reverse engineering, I got syscall spam -- a 200 meg log file, and when
>> I finally found a decent way to analyze it, I found most of Azureus'
>> system call wall time is spent in futex().  Huh?
>>
>> Looked up "futex" on Wikipedia, and I still have no clue how this makes
>> any sense.  Either futex was somehow thrashing the disk, or Azureus has
>> somehow managed to fork completely out of strace's control.  Or maybe
>> it's somehow something that the kernel is doing on its own, which is
>> somehow forcing azureus to block, but somehow not tripping strace's
>> timers while doing so.
> 
> Have you used -f or -ff with strace?

I used -f.  What's the difference between that and -ff?

Re: BitTorrent+Reiser4: curiouser and curiouser

[David Masover]

Alexander Zarochentsev wrote:

> I guess futex (, FUTEX_WAIT, ) calls can be ignored in this analysis. 
> They just wait another threat to call futex(, FUTEX_WAKE, ).  
> Interesting to find that thread and look what it was doing before 
> FUTEX_WAKE? Or FUTEX_WAIT returns ETIMEDOUT?

It probably would be interesting, but I'm a complete newbie at strace. 
You'll have to walk me through this one step by step...

Re: BitTorrent+Reiser4: curiouser and curiouser

[Milan Holzäpfel]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On Fri, 22 Sep 2006 12:20:51 -0500
David Masover <ninja@slaphack.com> wrote:

> Konstantin Münning wrote:
> > Have you used -f or -ff with strace?
> 
> I used -f.  What's the difference between that and -ff?

-ff puts messages for process n into filen <name>.n, where <name> is
given to -o. -o can be given with -f too, then all syscalls are written
in one file prefixed with the process ids.  See also strace(1)

As mentioned already, futex() is usually used for interprocess
communication, e. g. waiting for another thread / process. 

Regards,
Milan

[-- Attachment #2: Type: application/pgp-signature, Size: 827 bytes --]

Re: BitTorrent+Reiser4: curiouser and curiouser

[Nicolas STRANSKY]

Le 09/22/2006 08:28 AM, David Masover a écrit :
> Azureus had a problem.  Once it got up to a good clip downloading, it
> would thrash the disk.  It would thrash the disk, and the system, so
> hard that even web browsing was difficult, due to disk access being
> many, many times slower than Internet access, even an Internet which is
> being hogged by BitTorrent.

Just to add my 2¢, I have exactly the same problem with azureus + r4.
Azureus starts to make constant disk writes at ~500-700kB/s when
downloading a file on a r4 partition.
I had opened a bugreport for azureus, but it now appears that this
problem is reiser4 specific as I cannot reproduce it with other filesystems.
https://sourceforge.net/tracker/?func=detail&atid=575154&aid=1558227&group_id=84122

-- 
Nico
On ne trouve guère d'ingrats tant qu'on est en état de faire du bien.
	-+- François de La Rochefoucauld (1613-1680), Maximes 306 -+-

Re: BitTorrent+Reiser4: curiouser and curiouser

[Craig Shelley]

[-- Attachment #1: Type: text/plain, Size: 1043 bytes --]

On Mon, 2006-09-25 at 21:27 +0200, Nicolas STRANSKY wrote:
> Le 09/22/2006 08:28 AM, David Masover a écrit :
> > Azureus had a problem.  Once it got up to a good clip downloading, it
> > would thrash the disk.  It would thrash the disk, and the system, so
> > hard that even web browsing was difficult, due to disk access being
> > many, many times slower than Internet access, even an Internet which is
> > being hogged by BitTorrent.
> 
> Just to add my 2¢, I have exactly the same problem with azureus + r4.
> Azureus starts to make constant disk writes at ~500-700kB/s when
> downloading a file on a r4 partition.
> I had opened a bugreport for azureus, but it now appears that this
> problem is reiser4 specific as I cannot reproduce it with other filesystems.
> https://sourceforge.net/tracker/?func=detail&atid=575154&aid=1558227&group_id=84122
> 

I get this problem too. I believe I first noticed it when the fsync
trouble started.

-- 
Craig Shelley
EMail: craig@microtron.org.uk
Jabber: shell@jabber.earth.li

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: BitTorrent+Reiser4: curiouser and curiouser

[rvalles]

On Fri, Sep 22, 2006 at 01:28:10AM -0500, David Masover wrote:
> Azureus had a problem.  Once it got up to a good clip downloading, it 
> would thrash the disk.  It would thrash the disk, and the system, so 
> hard that even web browsing was difficult, due to disk access being 
> many, many times slower than Internet access, even an Internet which is 
> being hogged by BitTorrent.
> 
> After changing Azureus' cache to 32 megs, and telling it not to write 
> files immediately, I thought I had the problem solved -- no thrashing at 
> all.  Until the cache got full.  Then:  Thrashing.  Less freqent, but 
> much more vigorous -- Azureus becomes extremely unresponsive for a few 
> minutes.

I think this happens with any filesystem when using Azureus; that client
insists on doing its own caching, ignoring the cache linux has, and also
does waste hundreds of mb in buffers. Too bad it's the only client with
the so popular "azureus DHT" which works on GNU/Linux.

I personally recommend this one:
http://libtorrent.rakshasa.no

It does use mmap() for file IO, being very fast and using little RAM and
CPU.

Next time I need Azureus DHT I'll try if I can get to run some Windows
client on Wine.
-- 
- Do you study theories? - Oh yes! Theories are fun. - How do you use them? - When I must act, theories are too slow. To act, I must know.

Re: BitTorrent+Reiser4: curiouser and curiouser

[Christopher Sawtell]

On Tuesday 03 October 2006 04:39, rvalles wrote:
> Too bad it's the only client with
> the so popular "azureus DHT" which works on GNU/Linux.
KDE's ktorrent has DHT also.

-- 
CS

Re: BitTorrent+Reiser4: curiouser and curiouser

[rvalles]

[-- Attachment #1: Type: text/plain, Size: 555 bytes --]

On Tue, Oct 03, 2006 at 04:47:49PM +1300, Christopher Sawtell wrote:
> On Tuesday 03 October 2006 04:39, rvalles wrote:
> > Too bad it's the only client with
> > the so popular "azureus DHT" which works on GNU/Linux.
> KDE's ktorrent has DHT also.
Sure, but not "azureus DHT", which is a different thing (works as a
fallback when trackers fail, the torrent doesn't have to be a
"trackerless torrent").
-- 
- Do you study theories? - Oh yes! Theories are fun. - How do you use them? - When I must act, theories are too slow. To act, I must know.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]